From 70b6022c8baba3d0a52de6bd9fa11866bdbd3b3f Mon Sep 17 00:00:00 2001 From: forhappy Date: Wed, 28 Oct 2026 23:59:02 -0700 Subject: [PATCH 001/568] docs: select shared cell groups as the Raft replacement --- plans/multi-raft/README.md | 7 + plans/multi-raft/density-decision.md | 263 ++++++++++++++++++++++++ plans/multi-raft/implementation-plan.md | 6 + plans/multi-raft/scaling.md | 39 ++-- plans/multi-raft/technical-design.md | 6 + 5 files changed, 305 insertions(+), 16 deletions(-) create mode 100644 plans/multi-raft/density-decision.md diff --git a/plans/multi-raft/README.md b/plans/multi-raft/README.md index 10e1d5fb5..05b75c153 100644 --- a/plans/multi-raft/README.md +++ b/plans/multi-raft/README.md @@ -5,6 +5,13 @@ Status: MR-01 through MR-12 are complete at their recorded native boundaries. MR-13 through MR-17 remain open. The complete delivery goal is not achieved. +The [shared cell group decision](density-decision.md) selects multiple independent +SQLite cells per Raft group as the replacement architecture, based on inspection +of TiKV's Region implementation and Cellar's idle-group costs. A dedicated cell +uses the same group machinery with one cell. The runtime conversion, +complete group recovery manifests and qualification remain pending; current +evidence below still describes the one-cell-per-group implementation. + The [fleet balancing and maintenance delivery package](fleet-operations/README.md) defines the next CAS-coordinated fleet design, executable task plan and operator workflows for automatic repair, redistribution, scale-out, scale-in and node diff --git a/plans/multi-raft/density-decision.md b/plans/multi-raft/density-decision.md new file mode 100644 index 000000000..64f941481 --- /dev/null +++ b/plans/multi-raft/density-decision.md @@ -0,0 +1,263 @@ + + +# Shared cell groups: architecture decision + +Status: selected replacement architecture; runtime conversion and qualification +are pending. This document does not claim that Cellar supports shared groups yet. +It supersedes one-cell-per-group as the target architecture, while the existing +technical design and qualification records describe the implemented baseline. + +## Decision + +Make a **cell group** the unit of Raft consensus, placement, membership, snapshot +and disaster recovery. A group contains multiple cells. Each cell retains its own +SQLite database, LTX chain, command receipts, query versions and application +identity. Every replica of a group owns the group's complete committed state. + +Use one implementation for every group size. A large or hot cell can occupy a +group alone; it does not select a second consensus architecture. Remove the +assumption that creating a cell creates a Raft instance. Keep the shared node log +engine, worker pools, transport and resource limits already implemented. + +Groups are independent of physical nodes. A three-voter group spans three +placement domains; one node can lead several groups and follow others. Grouping +all cells currently on a node would bind ownership to a host and produce an +unbounded failure and scheduling unit. It is not the intended design. + +The decision targets the fixed consensus overhead of many small cells. It does +not promise lower latency for every workload, remove per-cell storage work, or +provide cross-cell transactions. Busy cells in the same group compete for a +leader, log bandwidth and the contiguous apply prefix. + +Grouping also does not split a hot cell's SQLite transaction stream across leaders. +The application retains its existing per-cell atomicity boundary. + +## TiKV verification + +Inspected the user-provided TiKV checkout at commit +`b3995dab28308f09cbb7c125bf32f789dbc203fe` (2026-10-22). No TiKV files were changed +and no TiKV benchmark was run. The paths below are relative to that checkout. + +| Source | Verified behavior | Cellar consequence | +| --- | --- | --- | +| `components/raftstore/src/store/peer.rs`, `Peer` | A Region peer owns a `RawNode>`. | One Raft machine per group replica, rather than per application object. | +| `components/raftstore-v2/src/raft/peer.rs`, `Peer` | The tablet path also owns one `RawNode>` per peer. | The consensus boundary survives a change of storage layout. | +| `components/raftstore/src/store/fsm/store.rs`, `StoreMeta` | Maps Region IDs and key-range boundaries to Region metadata. | A node hosts many groups and routes to the selected group. | +| `components/raftstore/src/store/util.rs`, `check_key_in_region`; `store/fsm/apply.rs`, `handle_put` | Writes validate their key against the Region's half-open key range. Many distinct keys can use the same Region. | Every Cellar entry must identify and validate its target cell inside its group. | +| `components/raftstore/src/store/fsm/apply.rs`, `exec_batch_split`, `exec_prepare_merge`, `exec_commit_merge` | Splits and merges change Region boundaries and epochs through replicated administrative operations. | Moving a cell between groups requires a protocol and routing fencing, not a local map edit. | +| `components/raftstore/src/store/async_io/write.rs`, `WriteTaskBatch::add_write_task` | Batches log writes while retaining their Region IDs. | Shared physical persistence and shared consensus groups are separate optimizations. Cellar already has the former. | +| `components/raftstore/src/store/fsm/peer.rs`, `quorum_agree_to_hibernate`, `on_hibernate_request`, tick handling | Coordinates hibernation votes and checks pending history before stopping ticks. | Group hibernation is a separate future protocol; an idle cell cannot stop a busy group's Raft ticks. | + +TiKV's [multi-Raft explanation](https://tikv.org/deep-dive/scalability/multi-raft/) +and [current Region glossary](https://docs.pingcap.com/tidb/stable/glossary/#regionpeerraft-group) +confirm that a Region holds a range of keys and its peers form one Raft group. +Cellar's analogous partition contains whole cells, not individual SQLite keys or +pages. Keep the name **cell group** to avoid confusing it with a geographic +region. Do not copy TiKV's byte thresholds or historical event-loop defaults: +Cellar has different image, transaction and recovery costs. + +## Evidence and expected savings + +Cellar source inspected: `1be27d234d4674d4d014785c60cecbfe394bdab9`. +The local idle experiment at that source measured 32 and 128 three-voter groups: +1.437 and 5.794 MB/s outgoing control traffic, respectively, or approximately +44 KiB/s per logical group across the three embedded nodes. Settled ten-second +windows recorded no engine writes or object-store reads/puts. These are debug +build diagnostics on one M2 Max with TLS loopback proxies and an in-memory object +store, not production costs, RustFS results or a shared-group benchmark. +Probe sources, exact environment and run metadata remain in the local ignored +`.cellar/cell-cost-report.md` and `.cellar/cell-cost-metadata.json`. + +For C cells with approximately K cells per group, G is approximately ceil(C/K). +With R voters, no learners and heartbeat interval H, baseline leader-to-follower +logical deliveries are G*(R-1)/H per second, before replies and batching. + +| Cells | Target cells/group | Groups at ideal packing | Group replicas with three voters | +| ---: | ---: | ---: | ---: | +| 1,000,000 | 1 | 1,000,000 | 3,000,000 | +| 1,000,000 | 32 | 31,250 | 93,750 | +| 1,000,000 | 64 | 15,625 | 46,875 | +| 1,000,000 | 128 | 7,813 | 23,439 | + +The number of cell database replicas remains 3,000,000 in every row. Grouping +reduces consensus state, timers, election and membership bookkeeping. It does not +divide cell data, receipts, LTX bytes, SQLite apply work or archive payload by K. +These counts are arithmetic, not a supported fleet size or a measured speedup. +Partially filled groups, dedicated cells and placement constraints increase G. + +Benchmark occupancies 1, 8, 32, 64 and 128 before choosing a default. Limit group +size by total bytes, write rate, apply latency and recovery budget as well as cell +count. Cell count alone cannot distinguish 64 tiny counters from 64 large cells. + +## Identity, assignment and admission + +Separate physical consensus history from the target cell: + +| Contract | Scope | +| --- | --- | +| `GroupScope` | Cluster, namespace, group ID, group incarnation; no cell name | +| `CellScope` | Group scope, cell identity and immutable assignment identity | +| Group birth / membership | Voter identities, disk claims, placement and initial protocol/schema binding | +| Cell enrollment | Cell scope, schema/application digest and verified initial image | +| Command / receipt / served version | Cell scope plus the exact group term/index and cell image identity | + +The names above describe the replacement contracts, not types already delivered. +Start with cells from one namespace and its immutable application/schema contract +in a group. Do not combine tenants or incompatible durability/placement policies +merely because they currently reside on the same node. + +Initially, assignment is immutable for an enrolled cell. A directory selects a +group with capacity for a new cell; adding nodes or groups does not rehash existing +cells. Concurrent creation uses a single conditional cell-assignment reservation. +Only its confirmed winner may enroll the cell, and the replicated enrollment binds +that reservation and the group incarnation. An uncertain reservation is reconciled +before proceeding. Route publication follows committed, durable enrollment; a +stale route cannot grant authority. Reservations and retries are bounded and +recoverable across controller/leader crashes. + +New groups can take new cells without moving existing ones. Group capacity is +rechecked at enrollment, so concurrent reservations cannot exceed its hard bound. +A refused reservation cannot be silently reassigned after it might have enrolled. +Birth records and permanent voter-disk claims occur per group. Cell-directory and +enrollment metadata still scale with the cell population. + +Online cell transfer, split and merge are separate work. They require source +quiescence committed through its quorum, an exact transfer cut, destination +installation and a fenced routing transition. No destination becomes writable +while the source might still accept commands. Receipts and effect identities must +survive transfer without replaying acknowledged work. Until that protocol passes +fault tests, operations move whole groups or create new groups for new cells. + +## Write, apply and read invariants + +1. Admit a command under current group leadership and the exact cell assignment. + Prepare it in that cell's private SQLite transaction, including its receipt. + Preserve one unresolved data proposal per cell in the first implementation. +2. Propose an envelope containing the group history, cell identity/assignment, + proposal identity, application digest and that cell's base/result LTX images. + Never propose an unscoped LTX update or replay application callbacks on followers. +3. Different cells may prepare concurrently through bounded shared SQL workers. + Fair cell queues feed group and node byte budgets. A shared log still orders + all entries; initial apply publication remains ordered by group index. +4. Apply an entry only to its addressed cell. Fsync immutable image data before + atomically advancing the durable group manifest and contiguous applied prefix. + Publish the cell view and settle output only after that marker is durable. +5. Quorum and S3 acknowledgements retain their existing meanings. Timeouts remain + ambiguous after proposal; duplicate receipt lookup remains scoped to the cell. + +If entries 40 and 41 update A and B, group applied=41 means both entries are +applied. A's latest image still corresponds to entry 40. B's progress cannot satisfy +an A receipt, replace A's base image or manufacture a newer A version. Advancing a +global applied index past an unfinished earlier entry is forbidden. + +Linearizable reads obtain a current-term group barrier, wait for the group apply +prefix to cover it, then pin the target cell's image from a coherent published +manifest. The cell need not have a mutation at the barrier index. AtLeast compares +the target cell's verified version within the same assignment and incarnation. +Stale reads remain explicitly stale. No read treats directory ownership or another +cell's newer image as a quorum proof. + +## Durable group manifest and crash recovery + +Replace the single-image `DurableApply` with a bounded committed manifest root. +The root authenticates the group history, applied position, enrollment state and +each cell's image commitment, LTX position and latest mutation position. Include +deletions/tombstones if deletion is exposed; absence must not resurrect a cell. +Use incremental immutable manifest storage so updating one cell does not copy +every SQLite image or serialize an unbounded group map on each command. + +Recover only a complete manifest selected by the durable engine marker. A crash +before the marker may leave orphan image/manifest objects; it cannot publish them. +A crash after the marker must recover every referenced cell without repeating its +callback. Missing, corrupt or mismatched referenced state refuses group admission. +Retention must protect the previous recoverable root until the new root is durable, +plus all snapshot, archive, reader and in-flight apply pins. + +Raft hard state, membership and apply state remain group scoped. Preserve the +existing shared-engine failure quarantine and permanent voter identity checks. +Closing one idle execution cache leaves its committed image and group membership +intact. It does not stop elections, change ownership or erase its cell manifest. + +## Snapshots, S3 and reclamation + +A group checkpoint at index K contains the complete cell manifest as of K, +membership/enrollment state and all referenced image data or extent roots. Cell +images may have different last mutation indexes, each no greater than K. Pin that +consistent set before streaming. Install it atomically at the group boundary; +receiving only a subset cannot make the group queryable or writable. + +S3 recovery sets contain a complete checkpoint plus an optional contiguous, +cell-addressed committed log tail through a declared group cut. Verify every +dependency, checksum, base/result chain and membership transition before publishing +coverage. Subsequent checkpoints can reuse unchanged immutable cell objects; they +need not upload every database again. A hot cell's archive failure can delay the +group's archive prefix, so track group lag and bound retained bytes. + +An S3 acknowledgement for entry N requires a retained complete group recovery set +covering N. Archive coverage for one cell cannot cover unrelated earlier entries +in the group. Keep quorum-only acknowledgements distinct: a zero-quorum disaster +can lose their unarchived suffix even though the live quorum had acknowledged it. + +Local log truncation requires a valid covering group snapshot and the existing +recovery/retention conditions. Where reclamation relies on S3, require confirmed +complete group coverage too. Uploaded bytes alone establish neither condition. +Physical shared-engine file reclamation must respect every group's live log ranges. +Reclaiming entries never deletes current cell images or dependencies pinned by a +retained recovery set. Preserve snapshot availability for lagging peers. + +Zero-quorum recovery fences every old voter of the group using the existing +fencing-authority contract, then creates one new group incarnation with fresh voter +identities. Restore every cell from the selected group recovery set, preserve its +receipts and require a fresh quorum before admission. Old tokens/assignments cannot +be used to write in the new incarnation. Old-origin outbox ambiguity still refuses +admission unless explicitly reconciled; grouping does not solve external effects +or make different groups' recovery cuts a fleet-wide consistent snapshot. + +## Replacement and delivery + +Replace the current one-cell/group path instead of adding another production mode. +Keep format markers at 1, with distinct protocol/domain bindings that reject old +layouts and mismatched peers. Use a fresh development namespace/storage root for +the hard cutover. Do not reinterpret a one-cell image as a multi-cell manifest, +silently migrate existing state, or delete existing disks/archives. A group with +one enrolled cell uses the exact same replacement code as a larger group. + +| Step | Owning code | Required evidence before completion | +| --- | --- | --- | +| 1. Separate identities and enrollment | `cellar-logic/consensus.rs`; `application/consensus/protocol.rs`; Raft catalog, routing, version and receipt adapters | Cross-cell/namespace/assignment tokens and LTX updates refuse; concurrent enrollment has one winner; old wire/storage bindings refuse. | +| 2. Group manifest and ordered apply | Raft cache/storage, apply, extent roots, host and SQL owners | Three distinct SQLite cells in one actual `RawNode` group; interleaved writes; durable marker/image crash cuts; duplicate receipts; missing manifest dependencies refuse. | +| 3. Group snapshots and peer rebuild | Snapshot/checkpoint transfer, membership and startup | Follower catch-up and lost-disk replacement restore every cell at one cut; no partial group admission; bounded streaming and pins. | +| 4. Group S3 archive and reclamation | Archive codec/publisher/recovery, extent objects, log purge | Real RustFS checkpoint/tail restore; delayed cell dependency blocks coverage; acknowledged history survives actual file reclamation and restart. | +| 5. Fenced group disaster recovery | Catalog disaster API, fencing authority and Docker fixture | Delete all voter disks; restore all cells into one new incarnation; reject old tokens; preserve receipts and new writes; quorum-only unarchived suffix reported separately. | +| 6. Fleet and application integration | Public SDK, native/Axum ingress, fleet planner, alarms, outbox and operator views | One group leader/placement for many cells; per-cell authorization, schema, effects and fairness; whole-group repair/drain; all owners join on shutdown. | +| 7. Remove baseline assumptions and qualify | CLI/examples/docs, all native Raft harnesses, capacity profiles | No production fallback allocating a group for each ordinary cell; occupancy-one uses shared code; equal-workload performance and fault evidence below. | + +These are dependent implementation slices of one replacement, not separate +supported architectures. They are not marked complete by adding types, changing +an ID, grouping rows in one SQLite database, or counting fewer catalog records. +Existing MR-16 physical-tier release gates remain open. + +## Qualification matrix + +Compare the existing revision against the replacement at equal logical cell +population, per-cell data sizes, offered arrival rate, active fraction, request +mix, durability and hardware. Within the replacement, compare occupancies +1/8/32/64/128. Use independent SQLite files and LTX chains in every cohort. + +Measure idle CPU/RSS/FD/control bytes, real Raft instance count, fsyncs, log and +image write amplification, archive requests/bytes/lag, p50/p95/p99, refusal/error +counts and recovery time to successful business reads/writes. Include a hot cell +beside cold cells, concurrent cells in one group, separate groups, large images, +slow apply, a slow minority, leader loss, object-store outage and mixed durability. +Record shared-group head-of-line delay rather than averaging it away. + +Fault tests must cover interleaved cell history across every new durable boundary, +stale assignment/term completions, enrollment crash/retry, snapshots while another +cell writes, corrupt/missing cell dependencies, reclamation with lagging peers, +retained-disk restart and the destructive real RustFS drill above. Strong-read and +receipt assertions must use the individual cell's history. Preserve failure and +ambiguous outcomes in the workload ledger. + +Group hibernation follows as separate work with safe wake/read authority, retained +state, peer-triggered wake, pending work, alarms/outbox and membership coverage. +No full-group wake latency or production density claim follows from this decision. diff --git a/plans/multi-raft/implementation-plan.md b/plans/multi-raft/implementation-plan.md index f16fe80f5..0e06bcbc1 100644 --- a/plans/multi-raft/implementation-plan.md +++ b/plans/multi-raft/implementation-plan.md @@ -1554,6 +1554,12 @@ shown by `plan.py show MR-15`; the manifest is authoritative for arguments. Dependencies: MR-15. Phase: scale. Initial status: TODO. +The [density decision](density-decision.md) selects shared cell groups as the +replacement for one-cell-per-group consensus and defines the dependent runtime, +manifest, archive, recovery and SDK conversion slices. The decision is recorded; +implementation and qualification are pending. This does not complete MR-16 or +relax its MR-15 dependency and physical-tier release gates. + Modify: `crates/cellar/durability/raft/runtime.rs`, `crates/cellar/durability/raft/placement.rs`, `crates/cellar/durability/raft/apply.rs`, `tools/perf/raft_profiles.json`. Create: `plans/multi-raft/density-decision.md`, `crates/cellar/tests/native_raft_density.rs`. diff --git a/plans/multi-raft/scaling.md b/plans/multi-raft/scaling.md index 5b33028b6..0d23b925f 100644 --- a/plans/multi-raft/scaling.md +++ b/plans/multi-raft/scaling.md @@ -2,8 +2,10 @@ # Cellar multi-Raft scaling plan -Multi-Raft distributes many small consensus groups. The first release uses one -cell per group and three voters. A node may lead one cell and follow another; +Multi-Raft distributes many small consensus groups. The current implementation +uses one cell per group and three voters. The [density decision](density-decision.md) +selects shared cell groups as its replacement; conversion and qualification are +pending. A node may lead one group and follow another; adding nodes does not enlarge every group's quorum. Aggregate capacity grows only while storage, network, routing, apply and workload distribution permit it. @@ -104,20 +106,25 @@ logical bytes, encoded LTX bytes, replicated bytes, SQLite copy/apply bytes and S3 bytes separately. Full-image publication is expected to constrain larger cells; do not extrapolate a small-counter benchmark to large databases. -## Later group and storage refinements - -MR-16 first measures which resource limits density. It then selects and qualifies -one improvement at a time: page-versioned committed images, safe group hibernation, -or several small cells in one group. Every choice requires an updated protocol -and crash tests before implementation. - -Shared groups retain separate SQLite databases and LTX chains. Their log entries -identify a cell; leadership/membership and consensus ordering are shared. This -reduces group count but couples placement, failure recovery and throughput. It -does not create cross-cell atomicity. Large/hot cells may merit dedicated groups, -but an online transfer must fence the source, commit a transfer boundary, install -the destination and atomically change routing without admitting two writers. -Until that protocol is qualified, group assignment is immutable. +## Shared group replacement and later refinements + +The [density decision](density-decision.md) selects shared cell groups and specifies +their identity, apply, snapshot, archive and recovery boundaries before runtime +conversion. Separate SQLite databases and LTX chains remain per cell. Leadership, +membership, placement and the contiguous apply/archive prefixes become group +scoped. Dedicated hot cells use the same implementation at occupancy one. + +With C cells and average occupancy K, group count is approximately C/K; physical +cell database replicas remain C*R. The capacity profiles above still describe +the one-cell baseline. Shared groups reduce fixed consensus overhead, while +adding queueing and recovery coupling that must be measured under equal logical +cell workloads. They provide no cross-cell transaction contract. + +Initial cell assignment is immutable. Online transfer, split and merge require +source fencing, a committed transfer cut, destination installation and a fenced +routing transition. Group hibernation is also a separate protocol and cannot be +inferred from closing an idle cell's execution cache. MR-16's physical-tier gates +remain unchanged; the architecture decision is not scale qualification. ## Federation for very large fleets diff --git a/plans/multi-raft/technical-design.md b/plans/multi-raft/technical-design.md index d3cd865b0..296acb89a 100644 --- a/plans/multi-raft/technical-design.md +++ b/plans/multi-raft/technical-design.md @@ -6,6 +6,12 @@ Status: proposed implementation contract. Baseline and source hashes are in [baseline.json](baseline.json); execution order is in [implementation-plan.md](implementation-plan.md). +The [shared cell group decision](density-decision.md) supersedes the one-cell-per- +group target below. This document retains the implemented baseline's contracts; +the replacement separates cell identity from group consensus and requires new +group manifests, snapshots, archives and recovery qualification before cutover. +Shared groups are not implemented by this documentation change. + ## 1 Scope and architectural decision Introduce an opt-in Raft namespace. Every cell in that namespace has an independent From e8e7ff1a824b4a9d36091ebce3f6ab5037397df5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 01:17:31 -0700 Subject: [PATCH 002/568] docs: add executable cell-group multi-Raft delivery plan --- plans/multi-raft/README.md | 5 + plans/multi-raft/cell-groups/README.md | 104 + plans/multi-raft/cell-groups/baseline.json | 153 ++ plans/multi-raft/cell-groups/design.md | 217 ++ .../cell-groups/implementation-plan.md | 37 + plans/multi-raft/cell-groups/plan.py | 557 +++++ plans/multi-raft/cell-groups/profiles.json | 630 ++++++ plans/multi-raft/cell-groups/protocols.md | 217 ++ plans/multi-raft/cell-groups/qualification.md | 225 ++ plans/multi-raft/cell-groups/tasks.json | 1986 +++++++++++++++++ plans/multi-raft/cell-groups/test_plan.py | 377 ++++ plans/multi-raft/density-decision.md | 7 + plans/multi-raft/scaling.md | 4 +- 13 files changed, 4518 insertions(+), 1 deletion(-) create mode 100644 plans/multi-raft/cell-groups/README.md create mode 100644 plans/multi-raft/cell-groups/baseline.json create mode 100644 plans/multi-raft/cell-groups/design.md create mode 100644 plans/multi-raft/cell-groups/implementation-plan.md create mode 100644 plans/multi-raft/cell-groups/plan.py create mode 100644 plans/multi-raft/cell-groups/profiles.json create mode 100644 plans/multi-raft/cell-groups/protocols.md create mode 100644 plans/multi-raft/cell-groups/qualification.md create mode 100644 plans/multi-raft/cell-groups/tasks.json create mode 100644 plans/multi-raft/cell-groups/test_plan.py diff --git a/plans/multi-raft/README.md b/plans/multi-raft/README.md index 05b75c153..1f5038eb1 100644 --- a/plans/multi-raft/README.md +++ b/plans/multi-raft/README.md @@ -12,6 +12,11 @@ uses the same group machinery with one cell. The runtime conversion, complete group recovery manifests and qualification remain pending; current evidence below still describes the one-cell-per-group implementation. +The [cell group delivery package](cell-groups/README.md) turns that decision into +16 dependent implementation tasks with source preflight, exact test oracles, +split/merge and hibernation protocols, and source-bound release evidence gates. +It is a validated implementation plan; all runtime conversion tasks remain pending. + The [fleet balancing and maintenance delivery package](fleet-operations/README.md) defines the next CAS-coordinated fleet design, executable task plan and operator workflows for automatic repair, redistribution, scale-out, scale-in and node diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md new file mode 100644 index 000000000..fcda07fa2 --- /dev/null +++ b/plans/multi-raft/cell-groups/README.md @@ -0,0 +1,104 @@ + + +# Cell group delivery package + +Status: executable design and implementation plan, requested on 2026-10-29. +The production runtime is unchanged. All CG implementation tasks are TODO. +Package validation is not implementation, benchmark or release qualification. + +This package implements the direction selected in the +[density decision](../density-decision.md). The target is TiKV-style partitioned +multi-Raft: many cells in a group, many groups on a node, group replication across +nodes, dynamic split/merge, placement and coordinated hibernation. Cellar keeps +independent SQLite/LTX cell state and its Quorum/S3 durability contracts. +It does not implement TiKV's KV API, MVCC, transactions, wire compatibility or PD. + +## Read and execute + +The plan runner requires Python 3.11 or newer and Git; it has no Python package dependencies. + +1. Read [design](design.md), [protocols](protocols.md) and + [qualification](qualification.md). These define the target, including crash + behavior and final release gates; they supersede fixed assignment as the final + target. Fixed assignment is an intermediate milestone only. +2. Run the commands below from the repository root. `ready` reports dependency + eligibility; `show CG-01` emits a complete executor brief, source excerpts, + scope, steps, exact test commands and completion rules. It never runs them. +3. Implement each task in dependency order on the active PR branch or an explicitly + assigned branch. Preserve working intermediate commits. Do not expose a second + production mode or reinterpret existing data as the new format. +4. Record actual test outputs and source/artifact hashes. A task can become DONE + only with verified evidence and completed dependencies. Final qualification + repeats the full capability matrix at one release revision. + +```sh +python3 plans/multi-raft/cell-groups/plan.py validate +python3 plans/multi-raft/cell-groups/plan.py preflight +python3 plans/multi-raft/cell-groups/plan.py ready +python3 plans/multi-raft/cell-groups/plan.py show CG-01 +python3 -m unittest discover -s plans/multi-raft/cell-groups -p 'test_*.py' +``` + +`validate` checks the task graph, coverage, source inventory, links and evidence +for DONE tasks. `preflight` checks source drift against `baseline.json`; it does +not silently refresh hashes. After a dependency changes an input, review that +change and update the baseline with its new revision and review note before +executing the dependent task. `show` labels planned test targets that are absent; +an absent target is never a passing test. `release` exits nonzero until every +task and required deployment gate is evidenced at the final source revision. + +The runner accepts no arbitrary shell commands and launches no processes or +containers apart from read-only Git source inspection. Commands displayed in +executor briefs are explicitly run by the implementation agent. Destructive +qualification belongs only to the harness's recorded, owned disposable resources. + +## Deliverables and source of truth + +| Artifact | Responsibility | +| --- | --- | +| `design.md` | Identities, routing, storage, APIs, compatibility and bounded ownership | +| `protocols.md` | Enrollment, apply, snapshot/archive, transfer, hibernation and disaster state machines | +| `tasks.json` | Ordered implementation tasks, file ownership, steps, tests and invariant mapping | +| `implementation-plan.md` | Human index generated from the same tasks; `show` supplies complete briefs | +| `qualification.md`, `profiles.json` | Fault oracles, fixed workload contracts, numerical candidate gates and evidence format | +| `baseline.json` | Reviewed Cellar source hashes/excerpts and TiKV investigation pin | +| `plan.py`, `test_plan.py` | Executable package validator, drift checks, task briefs and negative validation tests | + +The complete scope includes managed fleet integration, effect identity preservation +through split/merge, and disaster recovery while a transfer is incomplete. These +cannot be dropped to make a fixed-group demonstration count as full delivery. +One-cell groups remain available through the same implementation for hot cells. + +The initial independent review corrected three delivery hazards: effects must be +adapted before public cutover; catalog admission tests precede real enrollment +integration in the host task; and remotely verified transfer closure is mandatory +before publishing split/merge ownership. The validator also binds benchmark cohorts +to exact binaries, controls and profiles instead of accepting aggregate pass flags. + +No supported density, performance improvement or hibernation latency is claimed +until the corresponding profile passes. Local containers do not prove independent +host or zone resilience. Historical one-cell measurements remain baseline evidence. + +## Execution conventions + +Use Rust 1.98.1 and locked Cargo builds at this baseline. Read current root and +subtree AGENTS.md instructions. This checkout uses a private workspace-volume +target directory, for example: + +```sh +export PATH="$HOME/.cargo/bin:$PATH" +export CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/cellar-3617-cell-groups" +``` + +Keep `cellar-logic` free of I/O, async and dependencies. Use execution-domain +services and join owners on shutdown. Use a component `mod.rs` with named children, +or a leaf with sibling test files; never combine `feature.rs` and a Rust-filled +`feature/` directory. Match `native_raft_snapshot.rs` and `support/raft_cluster.rs` +for real-engine, native SQLite fixtures, rather than scheduler tests with stub +application markers. Do not enable `cellar_internal_tests` without its external +harness. Preserve license notices and keep credentials and raw cell data private. + +The parent `plans/multi-raft/plan.py validate` already rejects a repository-relative +component link in `010-large-cell-recovery-time.md`. This package has its own +repository-aware link validation; it does not suppress or claim to fix that +unrelated parent failure. Run normal repository docs checks as well. diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json new file mode 100644 index 000000000..2495c47c4 --- /dev/null +++ b/plans/multi-raft/cell-groups/baseline.json @@ -0,0 +1,153 @@ +{ + "format": 1, + "source_revision": "70b6022c8baba3d0a52de6bd9fa11866bdbd3b3f", + "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "review_note": "2026-10-29: inspected current owners, callers, native fixtures and CI; planning-only branch at 70b6022c. Reconcile changed source explicitly before implementation.", + "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", + "anchors": { + "identity": { + "path": "crates/cellar-logic/consensus.rs", + "sha256": "4003a743cae358bc29d1b4f5f527d7685e9c468c5cfaaa8d9fcded18b5194912", + "line": 83, + "excerpt": "pub struct GroupScope {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub cell: String,\n pub group: GroupId,\n pub incarnation: IncarnationId,\n}\n\nimpl GroupScope {" + }, + "codec": { + "path": "crates/cellar/application/consensus/protocol.rs", + "sha256": "cb2f16cf0c8f900b32c8c33ee504870539c8a7fd4ad282af27d4b0a28dce9229", + "line": 160, + "excerpt": "pub struct CellUpdate {\n pub scope: GroupScope,\n pub application_digest: [u8; 32],\n pub proposal: ProposalId,\n pub base: LtxImage,\n pub result: LtxImage,\n pub files: Vec,\n}\n" + }, + "manifest": { + "path": "crates/cellar/durability/raft/cache.rs", + "sha256": "c10a9c6eb84c3b7ba3866de68ca331456a926ad219bc276b6d767a4d7b27f770", + "line": 82, + "excerpt": "pub struct DurableApply {\n pub position: LogPosition,\n /// None denotes the canonical empty state before the schema command.\n pub image: Option,\n}\n\n#[derive(Clone, Debug)]\npub enum SnapshotChange {\n /// A verified local snapshot; retain entries beyond its included index." + }, + "apply": { + "path": "crates/cellar/durability/raft/apply.rs", + "sha256": "22dacc21f18449945d61ef2553621500512691faaf2da87c697262f6b4dcc3a1", + "line": 80, + "excerpt": "pub struct ImageApplier {\n group: GroupLog,\n images: GenerationStore,\n application_digest: [u8; 32],\n limits: EnvelopeLimits,\n current: Option,\n snapshot_pin: Option,\n pending: Option,\n}" + }, + "extent": { + "path": "crates/cellar/durability/raft/rooted/marker.rs", + "sha256": "b354ad8c64957569d21e8002b4f26082c41581140513fd31827e703c15076d43", + "line": 59, + "excerpt": "pub(super) fn adopt(\n group: &GroupLog,\n store: &NodeExtentStore,\n schema: [u8; 32],\n marker: &CommittedImage,\n max_image_bytes: u64,\n buffer: &mut [u8],\n) -> Result {\n marker.validate().map_err(ImageApplyError::Log)?;" + }, + "host": { + "path": "crates/cellar/durability/raft/host.rs", + "sha256": "94d5b62a99129b057377243a737c9e8ac642438eb677ad50ed78ccc15379bcc2", + "line": 1313, + "excerpt": "struct Cell {\n extents: Option,\n birth: GroupBirth,\n route_revision: LogIndex,\n configuration: Option,\n retired: bool,\n archive: CellArchive,\n archived: LogIndex,\n archive_retained: admission::Retained," + }, + "sql": { + "path": "crates/cellar/durability/raft/host_sql/mod.rs", + "sha256": "afab717be888dc826ff558bca6898abeb7406a9c531baf36da1a644d2218a923", + "line": 802, + "excerpt": "struct Cell {\n group: GroupLog,\n class: Arc,\n images: ImageOwner,\n snapshot: Option<(ApplyTicket, PreparedSnapshot, Permit)>,\n private: Option,\n active: Option,\n}\n" + }, + "runtime": { + "path": "crates/cellar/durability/raft/runtime.rs", + "sha256": "96b2b60c65333f0a16c8e2a538399ecdbc1c4456c14e504e904feb1bc41f655d", + "line": 727, + "excerpt": "struct Group {\n replica: Replica,\n scheduled: bool,\n write_scheduled: bool,\n deferred: VecDeque,\n inbound: Arc,\n hints: BTreeMap,\n tick: bool,\n failed: Option," + }, + "ticks": { + "path": "crates/cellar/durability/raft/runtime.rs", + "sha256": "96b2b60c65333f0a16c8e2a538399ecdbc1c4456c14e504e904feb1bc41f655d", + "line": 1055, + "excerpt": " fn tick(&mut self) {\n for (scope, group) in &mut self.groups {\n group.tick = true;\n group.wake(scope, &mut self.order);\n }\n }\n\n fn progress(&mut self) -> Result {\n let mut made = false;" + }, + "catalog": { + "path": "crates/cellar/durability/raft/catalog.rs", + "sha256": "65053cb9c70931f8234041d99e43ccd748c8f64b7479ed299888a57030850e1a", + "line": 104, + "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." + }, + "birth": { + "path": "crates/cellar/durability/raft/catalog.rs", + "sha256": "65053cb9c70931f8234041d99e43ccd748c8f64b7479ed299888a57030850e1a", + "line": 173, + "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." + }, + "version": { + "path": "crates/cellar/durability/raft/version.rs", + "sha256": "950d023f70e7b3ab1242e9b9d2a02475da4097f8aed0e9bf888e21c35a214c37", + "line": 20, + "excerpt": "pub enum VersionError {\n Invalid,\n Expired,\n WrongHistory,\n WrongSchema,\n KeyRetention,\n Limit,\n}\n" + }, + "snapshot": { + "path": "crates/cellar/durability/raft/snapshot.rs", + "sha256": "1882395d832cc76b2233d3e9d223105b2ee03da44e5c241f04f339238aa145a3", + "line": 296, + "excerpt": "pub struct SnapshotPackage {\n group: GroupLog,\n snapshot: Snapshot,\n bytes: Option,\n root: Option,\n pin: Option,\n limits: SnapshotLimits,\n membership: super::member_state::GroupMembership,\n}" + }, + "archive": { + "path": "crates/cellar/durability/raft/archive/mod.rs", + "sha256": "ef082373601af208a5ece0a73b5653874459f69abd24ed0b0fe1fcbd6a32e202", + "line": 346, + "excerpt": "pub struct CellArchive {\n coordination: Option,\n activity: Option,\n snapshot_scratch: stream_snapshot::ScratchPool,\n scratch_directory: Option,\n store: Arc,\n prefix: String,\n bundle_prefix: String,\n scope: GroupScope," + }, + "disaster": { + "path": "crates/cellar/durability/raft/disaster.rs", + "sha256": "8214156072b8fe4296869223596defbf75660f676635a85071b41d843a71c81c", + "line": 17, + "excerpt": "pub struct S3DisasterRecovery {\n namespace: NamespaceCatalog,\n host: cellar_ltx::LtxHost,\n limits: ArchiveLimits,\n}\n\nimpl S3DisasterRecovery {\n pub async fn open(\n store: Arc," + }, + "membership": { + "path": "crates/cellar/durability/raft/host_membership.rs", + "sha256": "ece36f2f8d1f0f099cfa3b56cc6214d60070bc9e9bc42469cdbc90c42b15521d", + "line": 15, + "excerpt": "pub(super) enum Goal {\n Fleet {\n binding: Box,\n fence: Option>,\n },\n Learner(JoiningMember),\n Repair {\n plan: RepairPlan,\n phase: RepairPhase," + }, + "effects": { + "path": "crates/cellar/durability/raft/host_effects.rs", + "sha256": "292eb1b101eeb0c250a1e095d8be7d3c345166af335b9139eedc10a6c710dda0", + "line": 8, + "excerpt": "pub(super) struct Scheduler {\n options: RaftEffectOptions,\n cursor: Option,\n next_at: u64,\n sequence: u64,\n busy: Option,\n enabled: bool,\n memory: Arc,\n}" + }, + "placement": { + "path": "crates/cellar/durability/raft/placement.rs", + "sha256": "f0ccb5cb44b7d98e8b5a6df507196cb8f5bc4da078f8176edcefa8e91ea9e7d4", + "line": 11, + "excerpt": "pub(super) struct Context<'a> {\n pub cells: &'a BTreeMap,\n pub peers: &'a BTreeMap,\n pub local: NodeId,\n pub rng: &'a mut crate::asyncrt::rng::Stream,\n}\n\nstruct Seen<'a> {\n view: &'a CellInspection," + }, + "fleet": { + "path": "crates/cellar/durability/raft/fleet_catalog/mod.rs", + "sha256": "66f72bfc0791d0080313419f82075eff6f1f480c3dc65fddffb0e7f8b6eb3522", + "line": 4, + "excerpt": "mod boots;\nmod capacity;\nmod deadline;\nmod disruption;\nmod docker_isolation;\nmod evacuation;\nmod execution;\nmod failure_recovery;\nmod finalization;" + }, + "public": { + "path": "crates/cellar/durability/raft/host_node.rs", + "sha256": "4b351c8f15a4c117e03ce3d46c69c40c86ab2747bdb379fde4687c3a5913c90d", + "line": 39, + "excerpt": "pub struct RaftHostObservation {\n pub node: super::super::inspection::NodeInspection,\n /// Synced physical boot, independent of a caller-provided process hint.\n pub fleet_boot: Option,\n pub startup_bounds: RaftHostStartupBounds,\n /// Rooted observation used by this boot, never a fresh eligibility proof.\n pub startup_enrollment: Option,\n /// Enrollment in the installed current trust cut. It may differ from boot.\n pub current_enrollment: Option," + }, + "fixture": { + "path": "crates/cellar/tests/native_raft_snapshot.rs", + "sha256": "3de9090b47be8aac228b071cba44668c12c6e4562472bbdb361563826dc1b67f", + "line": 91, + "excerpt": "async fn native_extent_fresh_learner_peer_seed_resumes_and_catches_up() -> Result<()> {\n native_seed_drill(\n false,\n false,\n false,\n SeedScenario::Ordinary,\n RaftImageStorage::Extents,\n )\n .await" + }, + "harness": { + "path": "tools/perf/raft_fleet.py", + "sha256": "417078701099601536e4bb1c4044dab05ceb4290257cb7d42de75c2df350bd7b", + "line": 246, + "excerpt": "def main():\n parser = argparse.ArgumentParser(description=__doc__)\n commands = parser.add_subparsers(dest='mode', required=True)\n setup = commands.add_parser('provision')\n setup.add_argument('--context', default='colima-cellar-raft-perf')\n setup.add_argument('--image', required=True)\n setup.add_argument('--policy', type=Path, required=True)\n setup.add_argument('--output', type=Path, required=True)\n setup.add_argument('--backing-disk', type=Path, default=Path.home()" + }, + "recovery": { + "path": "tools/perf/process_recovery.py", + "sha256": "f95cc6a69ef88f214314a50a59428e98140258a0d9644476eb0e78da3af9c9c4", + "line": 130, + "excerpt": "def main():\n parser = argparse.ArgumentParser(description=__doc__)\n parser.add_argument('--scenario', choices=['owner-loss', 'quorum-loss'], default='owner-loss')\n parser.add_argument('--profile', choices=['272mib', '10gb'], required=True)\n parser.add_argument('--output', type=Path, required=True)\n parser.add_argument('--work-directory', type=Path, required=True)\n parser.add_argument('--store-data-directory', type=Path, required=True)\n parser.add_argument('--fixture', type=Path)\n parser.add_argument('--binary', type=Path)" + }, + "ci": { + "path": ".github/workflows/native-sdk.yml", + "sha256": "05c5dc73b4ed7c2a29af7ada4765db3dbd6ec7645c8d2c997223417b18145de0", + "line": 67, + "excerpt": " RUST_VERSION: 1.98.1\n # Process tests publish executable artifacts; DWARF makes those fixtures huge.\n # Keep debug assertions and test behavior while omitting symbol information.\n CARGO_PROFILE_DEV_DEBUG: \"0\"\n CARGO_PROFILE_TEST_DEBUG: \"0\"\n\njobs:\n filesystem_release:\n if: github.event_name != 'workflow_dispatch' || inputs.mode == 'qualification'" + } + } +} diff --git a/plans/multi-raft/cell-groups/design.md b/plans/multi-raft/cell-groups/design.md new file mode 100644 index 000000000..fade0eb12 --- /dev/null +++ b/plans/multi-raft/cell-groups/design.md @@ -0,0 +1,217 @@ + + +# Cell group design + +Status: target contract for CG-01 through CG-16; not implemented. +Cellar baseline: `1be27d234d4674d4d014785c60cecbfe394bdab9`. +TiKV investigation: `b3995dab28308f09cbb7c125bf32f789dbc203fe` in the supplied +checkout. The [decision](../density-decision.md) records the inspected source. + +## Architecture and scope + +A cell group owns one Raft state machine on each of its replicas. The group +contains whole cells; every serving replica has each cell's committed SQLite +state. Only the leader executes cell commands. Followers apply cell-addressed LTX +updates. Independent groups share node workers, persistent log batches, transport +and resource budgets. Three voting replicas remain the production default. + +```mermaid +flowchart LR + Route[Cell router] --> L[Group 7 leader on node A] + L --> A[Cell X SQLite] + L --> B[Cell Y SQLite] + L --> C[Cell Z SQLite] + L <-->|one group log| F[Group 7 follower on node B] + L <-->|one group log| G[Group 7 follower on node C] + F --> FC[SQLite replicas X Y Z] + G --> GC[SQLite replicas X Y Z] + L --> S3[Complete group checkpoint and tail in S3] +``` + +Cells share placement, leadership and log order, but keep separate transactions, +receipt tables, LTX chains, schema checks, authorization and query versions. +Neither cross-cell transactions nor intra-cell sharding are introduced. One hot +cell can occupy one group; all group sizes use identical consensus machinery. +The feature-off Fleet/bucket protocol and non-Raft applications keep their contract. + +Full delivery includes split/merge, automatic placement, coordinated hibernation, +materialized and extent images, managed and unmanaged namespaces, native/Axum +ingress, alarms/outbox/inbox, group S3 reclamation and fenced disaster recovery. +TiKV transaction/MVCC APIs, SQL over multiple cells, geographic replication, online +conversion of old one-cell storage and a claim of TiKV-equivalent performance are +outside this package. No new consensus algorithm or raft-rs fork is needed. + +## Identities and versions + +| Type | Fields and lifetime | +| --- | --- | +| `CellHistory` | Cluster, namespace, validated cell ID, cell incarnation; stable through elections, replica moves, split and merge | +| `GroupHistory` | Cluster, namespace, opaque group ID, group incarnation; one physical log history | +| `GroupRoute` | Group history, half-open cell-key range, routing epoch, birth/schema binding | +| `GroupMembership` | Group history, committed membership revision, member/node/disk identities; independent of routing epoch | +| `CellVersion` | Cell history, nonzero monotone mutation sequence, LTX epoch/txid/checksum and image commitment | +| `CommitWitness` | Group history, term/index and proposal identity; locates the decision without defining the cell's lifetime | +| `TransitionId` | Non-reusable operation ID plus pinned source/destination histories and expected catalog root | + +Increment the cell mutation sequence for every committed cell image update, +including internal receipt/effect maintenance. Group no-op or membership entries +do not increment it. Overflow refuses before proposal. Enrollment creates sequence +1. Preserve the sequence and `CellHistory` through normal group transfer. +Disaster recovery creates new group and cell incarnations, retaining receipt +contents/client floors but rejecting tokens from the prior cell history. + +Version tokens authenticate cell history/version, schema, expiry/key identity and +the commit witness. They grant no access. AtLeast after normal transfer compares +the same cell history's mutation sequence, using installed transfer provenance; +it does not compare log indexes from unrelated groups. A token for another cell, +incarnation or schema refuses. Preserve original command-result bytes and receipt +identity through transfer. A recovery-seeded duplicate returns the retained result +without callback execution and a version covering that effect in the new history; +the old witness remains provenance, never new-incarnation authority. + +Image roots bind logical cell history, schema and LTX position. Group manifests +bind those roots to physical group history and a committed prefix. Extent payloads +remain content addressed. A normal transfer can reuse verified immutable cell +roots locally; it cannot borrow another group's authority or an unsealed local +reference. Disaster recovery rebinds the cell root's history, retaining payload +objects only after full verification. Materialized images follow the same rules. + +## Partitioning and directory + +Use a stable ordered cell routing key: + +```text +digest = SHA256("cellar-cell-route-1\0" || namespace_id[16] + || cell_id_length_u32_be || cell_id_utf8) +CellKey = (digest[32], cell_id_utf8) +``` + +Compare the tuple lexicographically by bytes. The full cell ID breaks digest ties; +a hash collision never aliases cells. Validate IDs before hashing, without locale +normalization. Ranges are `[lower, upper)` with explicit negative/positive infinity +sentinels. Split boundaries always lie between whole cell keys. A namespace's +active range map covers the complete domain without overlap or gaps. + +This hash ordering distributes application IDs and differs intentionally from +TiKV's user-key byte ordering. Cellar exposes no cross-cell ordered scan contract. +A dedicated hot-cell interval can use actual neighbor boundaries; it need not +split a cell or special-case the consensus engine. + +The namespace root conditionally publishes a digest of an immutable paged range +tree and its mutation journal. Pages have fixed byte/count limits; readers descend +only the relevant path and cache hints. No per-request full namespace scan, no +monolithic million-cell map and no environment-selected alternate routing hash. +Updates copy bounded tree paths and CAS one root, serializing namespace topology +changes only. Independent namespaces can progress independently. The final load +profile must measure this catalog write bottleneck. + +Initially provision a bounded explicit set of non-overlapping ranges; an empty +range record may remain unmaterialized until first enrollment. Lazy group birth +uses the existing conditional creation and permanent member claims. A node move +changes peer placement, not cell partitioning. An enrollment request goes to the +current group leader, which commits the cell's initial image. Concurrent creators +converge on that log decision and receipt. The previous per-cell reservation idea +is replaced by unique range ownership plus replicated enrollment; do not build two +assignment authorities. While a range is sealed, enrollment is refused/retried. + +Cached routes carry group history and routing epoch; every destination verifies +the complete binding. Redirects are authenticated hints, bounded by the original +deadline and hop count. An ambiguous submitted command is retried only with its +original cell-scoped command identity. An unavailable catalog may prevent births +and topology changes; it does not turn a known group's write path into S3 quorum. + +## Runtime and persistence + +Split the current host's combined `Cell` owner into a group owner and bounded cell +execution owners. Only the group owns RawNode, hard state, membership, read barriers, +apply prefix, snapshot/archive jobs and topology transitions. A cell owns receipt +admission, private execution, pending proposal, image/query pins and effect state. +The node owns log engine, extent store, transport, SQL worker pool and global quotas. + +One unresolved data proposal per cell is sufficient initially. Prepare different +cells concurrently through fair queues; serialize publication by group log index. +No slow callback or image preparation runs on the consensus tick loop. Group/node +byte quotas include queued, prepared, in-flight and ambiguous work; charge physical +owners until they release it, even after the request waiter is cancelled. + +Data entries carry cell history, route binding, schema, mutation sequence, +base/result image, receipt/proposal identity and bounded LTX payload. Administrative +entries are explicitly typed: enroll, membership, checkpoint, seal, install, +activate, park. Unknown tags, unsupported domains or inconsistent lengths refuse +before allocation. Batch independent proposals where useful, retaining each +cell's identity and base chain; batching is never a cross-cell transaction. + +The durable group root contains group history, applied position, routing/transition +state, membership binding and an authenticated map of all cell image leaves. +Each leaf contains cell history/version, last group mutation position and effect +summary. Use a content-addressed paged tree with bounded copy-on-write paths, not +an O(cell-count) full manifest rewrite for every update. An engine transaction +publishes the durable root and contiguous apply position after referenced image +and manifest pages are synced. Readers pin a published root and exact cell image. + +Configuration/no-op entries advance the group prefix without changing cell versions. +A missing earlier cell update prevents later group publication. Parallel staging +may finish out of order but cannot advance a hole in the durable applied prefix. +Retain exact term/index proof for each pending output; an unrelated cell's progress +cannot settle it. Storage sync failure quarantines the owning engine as today. + +## Reads, caches and effects + +Linearizable reads obtain a fresh current-term group ReadIndex, wait for contiguous +apply through it, then select the target cell from a coherent root. An unchanged +cell's last mutation can precede the barrier. AtLeast uses cell version lineage; +Stale explicitly permits an older coherent image. No clock lease optimization is +introduced. Transfer-sealed groups refuse new strong reads and mutations for their +range; requests already in progress retain their original deadlines/proof rules. + +Idle cell eviction closes disposable execution/SQLite caches only when no proposal, +effect callback or unsafe pin is active. Reopening reconstructs the private base +from the committed cell root. Group membership, data and receipt floors remain. +File-descriptor and cache byte limits must apply to thousands of enrolled cells, +not just an active-cell metric. Cache misses, image downloads and application work +are separately measured from consensus wake latency. + +Alarms and outbox/inbox IDs bind `CellHistory`, not current group or node. Transfers +preserve delivery identity and receiver dedupe; they must drain/journal already +dispatched work and never mint new IDs for an unknown outcome. Durable summaries +let node schedulers find the next due effect without opening every SQLite DB. +Control ownership follows the current group leader. A pending old-origin delivery +after disaster recovery still needs reconciliation; the existing refusal remains. + +## Resource policy and operational API + +Expose distinct limits for cells/group, group logical bytes, resident cell bytes, +queued LTX bytes, unarchived bytes, snapshot/transition slots, open readers and +group replicas/node. Existing validated envelope/image limits remain in force. +Initial candidate occupancies are 8, 32, 64 and 128; no numeric default becomes a +capacity claim before qualification. Oversized single cells use dedicated groups +and explicit per-cell quotas. Never raise a hard cap to hide a failed test. + +Scheduler inputs include cell count, bytes, write/apply rate, queue age, archive +lag, leader CPU and failure-domain placement. Split/merge use hysteresis, cooldown +and bounded concurrent work. Only compatible adjacent ranges can merge. Replica +movement uses learners, verified catch-up and committed configuration changes; +membership changes cannot race a seal, park or disaster operation. + +The SDK continues to address cells. Add operator group list/inspect, split/merge, +rebalance, park/wake and transition status/resume operations. Status is read-only; +waiting does not advance a mutation. CLI/admin/native and Axum enforce the same +identity and authorization contract. Metrics use bounded labels and distinguish +cells, group replicas, leaders, parked groups, unresolved operations and archive +lag. Shutdown seals admission, drains/records ambiguity, joins all owned work and +preserves transition pins/debt for restart. + +## Cutover and delivery boundaries + +Use the existing raft-rs and Raft Engine integration; no TiKV source copying or +dependency replacement. Add private contracts during staged development, switch +all production consumers together, then delete superseded one-cell assumptions. +Do not ship a toggle selecting legacy per-cell consensus. Occupancy one exercises +the same group path as larger occupancies. + +All Cellar-owned markers stay at 1, with new semantic domain/protocol digests and +fresh namespace/storage roots. Old wire, image and archive layouts refuse. There +is no automatic data migration or rollback by pointing an old binary at new data. +Retain existing disks/archives; any developer reset is an explicit separate action. +The final release sweep covers catalog, tokens, public SDK, CLI/examples, effects, +managed fleet and destructive recovery, not just the new host fixture. diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md new file mode 100644 index 000000000..a79427dde --- /dev/null +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -0,0 +1,37 @@ + + +# Implementation order + +Generated from `tasks.json`; print again with `plan.py index` after status changes. +Every implementation task is pending until source-bound evidence passes. + +Use `python3 plans/multi-raft/cell-groups/plan.py show CG-01` for a complete +executor brief. Replace the ID to select another task. Commands are displayed, +never executed by the plan runner. Read the [design](design.md), +[protocols](protocols.md) and [qualification](qualification.md) first. + +| Task | Deliverable | Dependencies | Status | Required cases | +| --- | --- | --- | --- | ---: | +| CG-01 | Separate logical cell history from physical group consensus | None | TODO | 4 | +| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | TODO | 5 | +| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | +| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | +| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | +| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | +| CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | +| CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | TODO | 7 | +| CG-10 | Split hot groups and merge compatible adjacent ranges | CG-09 | TODO | 5 | +| CG-11 | Schedule group placement and adaptive partition size | CG-07, CG-10 | TODO | 5 | +| CG-12 | Coordinate group hibernation and authority-checked wakeup | CG-08, CG-10, CG-11 | TODO | 5 | +| CG-14 | Build reproducible shared-group workload and RustFS harnesses | CG-07, CG-08 | TODO | 5 | +| CG-13 | Recover managed groups and incomplete transitions from S3 | CG-06, CG-09, CG-10, CG-11, CG-14 | TODO | 6 | +| CG-15 | Qualify crashes, split merge, wake and zero-quorum recovery | CG-10, CG-11, CG-12, CG-13, CG-14 | TODO | 6 | +| CG-16 | Publish the unified group runtime and qualified release envelope | CG-15 | TODO | 4 | + +CG-01 through CG-08 establish the shared runtime and public cutover. CG-09 through +CG-13 add movement, adaptive scheduling, hibernation and complete managed recovery. +CG-08 adapts effects before CG-07 public cutover. CG-14 can proceed after CG-07; +CG-13 requires its real-infrastructure harness. +CG-15 and CG-16 qualify the combined artifact. These milestones are not separate +production architectures. No full delivery claim precedes the final gate. diff --git a/plans/multi-raft/cell-groups/plan.py b/plans/multi-raft/cell-groups/plan.py new file mode 100644 index 000000000..5d276da85 --- /dev/null +++ b/plans/multi-raft/cell-groups/plan.py @@ -0,0 +1,557 @@ +#!/usr/bin/env python3 +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Validate delivery contracts and print executor briefs; never execute their commands.""" + +import argparse +import hashlib +import json +import math +from pathlib import Path +import re +import shlex +import subprocess +import sys +from urllib.parse import unquote, urlsplit + +PACKAGE = Path(__file__).resolve().parent +ROOT = PACKAGE.parents[2] +STATUSES = {"TODO", "IN_PROGRESS", "BLOCKED", "DONE"} +SHA256 = re.compile(r"[0-9a-f]{64}") +REVISION = re.compile(r"[0-9a-f]{40}") +CASE = re.compile(r"[a-z][a-z0-9_]*") +TASK_IDS = {f"CG-{number:02}" for number in range(1, 17)} +INVARIANTS = {f"I{number:02}" for number in range(1, 15)} +PROFILE_IDS = {"idle_density", "mixed_quorum", "mixed_s3", "hot_cold", "split_merge", + "hibernate_wake", "rustfs_disaster", "independent_domain_loss", "soak_24h"} +# Include examples, vendored crates, build scripts and future inputs by default. +# Plan bookkeeping may advance without changing the qualified executable. +RUNTIME_PATHS = [".", ":(exclude)plans/**"] +SAFETY_METRICS = {"lost_acknowledged_effects", "duplicate_mutations", "stale_strong_reads", + "double_writers", "unexpected_errors", "unjoined_owners", + "old_token_acceptances", "old_host_restart_acceptances", + "restored_receipt_callbacks", "missed_due_effects", "writes_without_quorum"} + + +class PlanError(ValueError): + pass + + +def require(condition, message): + if not condition: + raise PlanError(message) + + +def nonempty(value): + return isinstance(value, str) and bool(value.strip()) + + +def read_json(path): + def unique(pairs): + result = {} + for key, value in pairs: + require(key not in result, f"Duplicate JSON key: {key}") + result[key] = value + return result + + def constant(value): + raise PlanError(f"Non-finite JSON value: {value}") + + return json.loads(path.read_text(), object_pairs_hook=unique, parse_constant=constant) + + +def within(root, name): + require(nonempty(name) and not Path(name).is_absolute(), f"Not a relative path: {name}") + path = (root / name).resolve() + require(path.is_relative_to(root.resolve()), f"Path escapes repository: {name}") + return path + + +def digest(path): + with path.open("rb") as stream: + return hashlib.file_digest(stream, "sha256").hexdigest() + + +def artifact(root, item): + require(isinstance(item, dict), "Artifact must be an object") + path = within(root, item.get("path")) + require(path.is_file() and path.stat().st_size > 0, f"Missing/empty artifact: {path}") + require(isinstance(item.get("sha256"), str) and SHA256.fullmatch(item["sha256"]), + f"Invalid artifact digest: {path}") + require(digest(path) == item["sha256"], f"Artifact hash mismatch: {path}") + return path + + +def git_bytes(root, revision, path): + require(isinstance(revision, str) and REVISION.fullmatch(revision), "Invalid source revision") + within(root, path) + result = subprocess.run(["git", "show", f"{revision}:{path}"], cwd=root, + capture_output=True, check=False) + require(result.returncode == 0, f"Source absent at {revision}: {path}") + return result.stdout + + +def ordered(tasks): + by_id = {task["id"]: task for task in tasks} + require(len(by_id) == len(tasks), "Duplicate task ID") + visiting, visited, result = set(), set(), [] + + def visit(tid): + require(tid in by_id, f"Unknown dependency: {tid}") + require(tid not in visiting, f"Dependency cycle: {tid}") + if tid in visited: + return + visiting.add(tid) + for dep in by_id[tid]["depends_on"]: + visit(dep) + visiting.remove(tid) + visited.add(tid) + result.append(tid) + + for tid in by_id: + visit(tid) + return result + + +def cargo(toolchain, *args): + return ["cargo", f"+{toolchain}", *args] + + +def task_commands(task, toolchain): + test = cargo(toolchain, "test", "-p", "cellar", "--test", task["test_target"], "--locked", "--") + commands = [cargo(toolchain, "check", "-p", "cellar", "-p", "cellar-logic", "--all-targets", "--locked"), + test + ["--list"]] + commands.extend(test + [case["name"], "--exact", "--test-threads=1"] for case in task["test_cases"]) + commands.extend([cargo(toolchain, "format", "--check"), cargo(toolchain, "check-layout"), + cargo(toolchain, "clippy", "-p", "cellar", "-p", "cellar-logic", "--all-targets", + "--locked", "--", "-D", "warnings")]) + if task["id"] == "CG-01": + commands.append(cargo(toolchain, "test", "-p", "cellar-logic", "--lib", "--locked")) + if task["id"] == "CG-14": + commands.append(["python3", "-m", "unittest", "discover", "-s", "tools/perf", "-p", "test_cell_groups.py"]) + return commands + + +def release_commands(toolchain): + return [cargo(toolchain, "format", "--check"), cargo(toolchain, "check-layout"), + ["python3", "tools/check_native_dependencies.py"], ["python3", "tools/docs/check.py"], + cargo(toolchain, "check", "--workspace", "--all-targets", "--all-features", "--locked"), + cargo(toolchain, "test", "--workspace", "--all-features", "--lib", "--tests", "--locked", "--", "--test-threads=1"), + cargo(toolchain, "test", "--workspace", "--all-features", "--doc", "--locked"), + cargo(toolchain, "clippy", "--workspace", "--all-targets", "--all-features", "--locked", "--", "-D", "warnings"), + ["python3", "-m", "unittest", "discover", "-s", "tools/perf", "-p", "test_*.py"]] + + +def commands_for(task, manifest): + tasks = manifest["tasks"] if task["id"] == "CG-16" else [task] + commands = [command for item in tasks for command in task_commands(item, manifest["toolchain"])] + if task["id"] == "CG-16": + commands.extend(release_commands(manifest["toolchain"])) + return [list(command) for command in dict.fromkeys(tuple(command) for command in commands)] + + +def validate_contract(manifest, baseline, profiles, root): + require(manifest.get("format") == baseline.get("format") == profiles.get("format") == 1, "Wrong format") + require(REVISION.fullmatch(manifest.get("planned_at", "")), "Missing planned-at revision") + require(REVISION.fullmatch(baseline.get("source_revision", "")), "Missing reviewed source revision") + require(nonempty(baseline.get("review_note")), "Missing baseline review note") + require(manifest.get("toolchain") == "1.98.1", "Review CI/toolchain contract before changing it") + require(set(manifest.get("invariants", [])) == INVARIANTS, "Incomplete invariant inventory") + anchors = baseline.get("anchors", {}) + require(isinstance(anchors, dict) and anchors, "Missing source anchors") + for name, anchor in anchors.items(): + require(within(root, anchor.get("path")).is_file(), f"Missing source anchor: {name}") + require(SHA256.fullmatch(anchor.get("sha256", "")), f"Invalid anchor digest: {name}") + require(type(anchor.get("line")) is int and anchor["line"] > 0 and nonempty(anchor.get("excerpt")), + f"Missing source excerpt: {name}") + tasks = manifest.get("tasks", []) + require(isinstance(tasks, list) and tasks, "Missing tasks") + require({task.get("id") for task in tasks} == TASK_IDS, "Incomplete task inventory") + covered, names, producers = set(), set(), {} + for task in tasks: + tid = task["id"] + require(task.get("status") in STATUSES, f"Unknown task status: {tid}") + for field in ("title", "priority", "effort", "risk", "capability"): + require(nonempty(task.get(field)), f"{tid}: missing {field}") + for field in ("depends_on", "anchors", "scope", "creates", "steps", "test_cases", "invariants", + "evidence", "stop_conditions", "out_of_scope"): + require(isinstance(task.get(field), list), f"{tid}: {field} must be a list") + require(len(set(task["depends_on"])) == len(task["depends_on"]), f"{tid}: duplicate dependency") + require(task["anchors"] and set(task["anchors"]) <= set(anchors), f"{tid}: unknown source anchor") + for path in task["scope"]: + require(within(root, path).exists(), f"{tid}: missing source scope: {path}") + require(task["scope"] and task["stop_conditions"] and task["out_of_scope"], f"{tid}: incomplete scope") + require(CASE.fullmatch(task.get("test_target", "")), f"{tid}: invalid test target") + require(f'crates/cellar/tests/{task["test_target"]}.rs' in task["creates"], f"{tid}: missing target output") + for path in task["creates"]: + within(root, path) + require(path not in producers, f"Duplicate output: {path}") + require(path not in {a["path"] for a in anchors.values()}, f"Output collides with baseline: {path}") + producers[path] = tid + local_names, local_invariants = set(), set() + for case in task["test_cases"]: + require(CASE.fullmatch(case.get("name", "")), f"{tid}: invalid case") + require(case["name"] not in names, f'Duplicate test: {case["name"]}') + require(nonempty(case.get("oracle")), f"{tid}: empty oracle") + require(case.get("invariants") and set(case["invariants"]) <= INVARIANTS, f"{tid}: invalid invariant") + names.add(case["name"]) + local_names.add(case["name"]) + local_invariants.update(case["invariants"]) + require(local_names and local_invariants == set(task["invariants"]), f"{tid}: invariant coverage differs") + step_cases = set() + for step in task["steps"]: + require(nonempty(step.get("action")) and step.get("tests"), f"{tid}: empty step") + require(set(step["tests"]) <= local_names, f"{tid}: unknown step test") + step_cases.update(step["tests"]) + require(step_cases == local_names, f"{tid}: cases missing from steps") + covered.update(local_invariants) + require(covered == INVARIANTS, "Missing invariant coverage") + order = ordered(tasks) + by_id = {task["id"]: task for task in tasks} + for task in tasks: + if task["status"] == "DONE": + require(all(by_id[d]["status"] == "DONE" for d in task["depends_on"]), f'{task["id"]}: DONE before dependency') + require(task["evidence"], f'{task["id"]}: DONE without evidence') + profile_ids = set() + require(profiles.get("profiles"), "Missing qualification profiles") + for profile in profiles["profiles"]: + pid = profile.get("id") + require(isinstance(pid, str) and CASE.fullmatch(pid) and pid not in profile_ids, "Invalid/duplicate profile") + profile_ids.add(pid) + require(profile.get("required") is True, f"{pid}: required gate was disabled") + require(profile.get("topology") in {"local_containers", "independent_hosts"}, f"{pid}: invalid topology") + require(profile.get("checks"), f"{pid}: empty profile checks") + for field in ("cells", "nodes", "voters", "duration_seconds", "repetitions"): + require(type(profile.get(field)) is int and profile[field] > 0, f"{pid}: invalid {field}") + require(profile.get("occupancies") and all(type(v) is int and v > 0 for v in profile["occupancies"]), + f"{pid}: invalid occupancy matrix") + require(set(profile.get("image_modes", [])) == {"materialized", "extents"}, f"{pid}: image mode missing") + require(profile.get("gate_occupancies") and set(profile["gate_occupancies"]) <= set(profile["occupancies"]), + f"{pid}: invalid gated cohorts") + require(type(profile.get("historical_control")) is bool, f"{pid}: missing control contract") + if profile["historical_control"]: + require(REVISION.fullmatch(profile.get("historical_revision", "")), f"{pid}: missing historical pin") + for name, ratio in profile.get("ratios", {}).items(): + require(name in profile["checks"] and nonempty(ratio.get("numerator")) and nonempty(ratio.get("denominator")), + f"{pid}: invalid ratio contract") + require("control_occupancy" not in ratio or ratio["control_occupancy"] in profile["occupancies"], + f"{pid}: missing ratio control") + for name, bounds in profile["checks"].items(): + require(bounds and set(bounds) <= {"eq", "min", "max"}, f"{pid}: invalid metric bound {name}") + require(all(type(v) in (int, float) and math.isfinite(v) for v in bounds.values()), f"{pid}: nonfinite bound") + require(profile_ids == PROFILE_IDS, "Required qualification profile inventory changed") + require(any(p["required"] and p["topology"] == "independent_hosts" for p in profiles["profiles"]), + "Release requires independent-domain qualification") + return {"tasks": len(tasks), "required_cases": len(names), "invariants": len(covered), + "profiles": len(profile_ids), "order": order} + + +def metric(value, name): + require(type(value) in (int, float) and math.isfinite(value) and value >= 0, f"Missing/invalid metric: {name}") + return value + + +def check_profile(report, profile, profile_digest, revision, root, executable): + require(report.get("format") == 1 and report.get("profile_id") == profile["id"], "Wrong profile identity") + require(report.get("source_revision") == revision, "Profile source differs from release source") + require(report.get("profiles_sha256") == profile_digest, "Profile configuration changed") + require(report.get("executable_sha256") == executable and SHA256.fullmatch(executable), "Profile executable differs") + require(nonempty(report.get("environment")), "Missing actual profile environment") + require(report.get("topology") == profile["topology"], "Profile topology differs") + metrics = report.get("metrics", {}) + for name, bounds in profile["checks"].items(): + value = metrics.get(name) + metric(value, name) + for op, bound in bounds.items(): + passed = value == bound if op == "eq" else value >= bound if op == "min" else value <= bound + require(passed, f'{profile["id"]}: {name}={value} violates {op} {bound}') + artifacts = report.get("artifacts", {}) + for role in ("ledger", "measurements", "topology", "fault_schedule", "verification"): + require(role in artifacts, f"Missing profile artifact: {role}") + artifact(root, artifacts[role]) + measured = read_json(artifact(root, artifacts["measurements"])) + require(measured.get("metrics") == metrics, "Summary differs from retained measurements") + expected = {("replacement", o, mode, rep) for o in profile["occupancies"] + for mode in profile["image_modes"] for rep in range(1, profile["repetitions"] + 1)} + if profile["historical_control"]: + expected.update(("historical", 1, mode, rep) for mode in profile["image_modes"] + for rep in range(1, profile["repetitions"] + 1)) + rows = {} + for row in measured.get("cohorts", []): + key = (row.get("variant"), row.get("occupancy"), row.get("image_mode"), row.get("repetition")) + require(key in expected and key not in rows, "Unknown/duplicate measured cohort") + source = revision if key[0] == "replacement" else profile["historical_revision"] + require(row.get("source_revision") == source, "Cohort source mismatch") + require(SHA256.fullmatch(row.get("executable_sha256", "")), "Missing cohort executable") + if key[0] == "replacement": + require(row["executable_sha256"] == executable, "Cohort executable differs") + for name in ("cells", "nodes", "voters", "offered_requests_per_second"): + require(type(row.get(name)) is int and row[name] == profile[name], f"Measured cohort changed {name}") + require(metric(row.get("measured_seconds"), "measured_seconds") >= profile["duration_seconds"], "Cohort too short") + require(row.get("ledger_sha256") == artifacts["ledger"]["sha256"], "Cohort ledger not bound") + require(isinstance(row.get("metrics"), dict), "Missing cohort metrics") + rows[key] = row + require(set(rows) == expected, "Required cohort matrix incomplete") + candidates = [(key, row) for key, row in rows.items() if key[0] == "replacement" and key[1] in profile["gate_occupancies"]] + for key, row in candidates: + values = row["metrics"] + for name, ratio in profile.get("ratios", {}).items(): + control = rows[("replacement", ratio["control_occupancy"], key[2], key[3])] if "control_occupancy" in ratio else row + numerator = metric(values.get(ratio["numerator"]), ratio["numerator"]) + denominator = metric(control["metrics"].get(ratio["denominator"]), ratio["denominator"]) + require(denominator > 0, "Ratio control denominator is zero") + require(math.isclose(metric(values.get(name), name), numerator / denominator, rel_tol=1e-9), "Ratio is not derived from matched control") + for name, bounds in profile["checks"].items(): + selected = list(rows.items()) if name in SAFETY_METRICS else candidates + values = [metric(row["metrics"].get(name), name) for _, row in selected] + for op, bound in bounds.items(): + aggregate = max(values) if op == "max" else min(values) if op == "min" else bound + require(metrics[name] == aggregate, f"Metric does not retain worst cohort: {name}") + if op == "eq": + require(all(value == bound for value in values), f"Cohort equality gate failed: {name}") + topology = read_json(artifact(root, artifacts["topology"])) + require(topology.get("kind") == profile["topology"], "Actual topology mismatch") + nodes = topology.get("nodes", []) + require(len(nodes) == profile["nodes"] and len({n["id"] for n in nodes}) == len(nodes), "Incomplete topology inventory") + if profile["topology"] == "independent_hosts": + require(len({n["physical_host"] for n in nodes}) == len(nodes), "Containers do not prove independent hosts") + require(len({n["failure_domain"] for n in nodes}) >= profile["voters"], "Independent failure domains missing") + verified = read_json(artifact(root, artifacts["verification"])) + require(verified.get("source_revision") == revision and verified.get("complete") is True, "History verification incomplete") + require(verified.get("ledger_sha256") == artifacts["ledger"]["sha256"], "History verification used another ledger") + require(verified.get("measurements_sha256") == artifacts["measurements"]["sha256"], "History verification used another measurement set") + + +def check_evidence(task, record, manifest, baseline, profiles, profile_digest, root): + tid = task["id"] + require(record.get("format") == 1 and record.get("task_id") == tid, "Wrong evidence task identity") + revision = record.get("source_revision", "") + require(REVISION.fullmatch(revision), "Missing evidence source revision") + require(SHA256.fullmatch(record.get("executable_sha256", "")), "Missing evidence executable digest") + require(nonempty(record.get("environment")) and nonempty(record.get("limitations")), "Missing evidence scope") + cohort = manifest["tasks"] if tid == "CG-16" else [task] + required_paths = {baseline["anchors"][a]["path"] for t in cohort for a in t["anchors"]} + required_paths.update(f'crates/cellar/tests/{t["test_target"]}.rs' for t in cohort) + inputs = record.get("source_inputs", []) + require(isinstance(inputs, list) and inputs, "Missing source input inventory") + paths = set() + for item in inputs: + path = item.get("path") + require(path not in paths, "Duplicate source input") + paths.add(path) + content = git_bytes(root, revision, path) + require(hashlib.sha256(content).hexdigest() == item.get("sha256"), f"Source hash mismatch: {path}") + require(required_paths <= paths, f"Evidence omits source inputs: {sorted(required_paths - paths)}") + changed = subprocess.check_output(["git", "diff", "--name-only", "--diff-filter=ACMRT", "-z", + baseline["source_revision"], revision, "--", *RUNTIME_PATHS], cwd=root).decode().split("\0") + require(set(changed) - {""} <= paths, "Evidence omits changed runtime inputs") + outputs = {} + for run in record.get("commands", []): + require(type(run.get("exit_code")) is int and run["exit_code"] == 0, "Failed/unconfirmed evidence command") + argv = run.get("argv") + require(isinstance(argv, list) and all(nonempty(a) for a in argv), "Invalid recorded command") + key = tuple(argv) + require(key not in outputs, "Duplicate command evidence") + path = artifact(root, {"path": run.get("output"), "sha256": run.get("sha256")}) + outputs[key] = path + expected = commands_for(task, manifest) + require(all(tuple(command) in outputs for command in expected), "Missing required command evidence") + for item in cohort: + prefix = cargo(manifest["toolchain"], "test", "-p", "cellar", "--test", item["test_target"], "--locked", "--") + inventory = outputs[tuple(prefix + ["--list"])].read_text() + for case in item["test_cases"]: + name = case["name"] + require(re.search(rf"^{re.escape(name)}: test$", inventory, re.M), f"Case absent from inventory: {name}") + output = outputs[tuple(prefix + [name, "--exact", "--test-threads=1"])].read_text() + require(re.search(rf"^test {re.escape(name)} \.\.\. ok$", output, re.M), f"Case did not pass: {name}") + require("test result: ok. 1 passed; 0 failed; 0 ignored;" in output, f"Not exactly one passing case: {name}") + required_profiles = {p["id"] for p in profiles["profiles"] if p["required"]} if tid in {"CG-15", "CG-16"} else set() + if tid == "CG-13": + required_profiles.add("rustfs_disaster") + seen = set() + by_id = {p["id"]: p for p in profiles["profiles"]} + for item in record.get("profiles", []): + report = read_json(artifact(root, item)) + pid = report.get("profile_id") + require(pid in by_id and pid not in seen, "Unknown/duplicate profile evidence") + qualified = record.get("qualified_executable_sha256", "") + require(SHA256.fullmatch(qualified), "Missing qualified deployment executable digest") + check_profile(report, by_id[pid], profile_digest, revision, root, qualified) + seen.add(pid) + require(required_profiles <= seen, f"Missing required profile evidence: {sorted(required_profiles - seen)}") + return revision + + +def check_links(package, root): + checked = 0 + for source in package.glob("*.md"): + prose = re.sub(r"```.*?```", "", source.read_text(), flags=re.S) + for raw in re.findall(r"!?\[[^\]]*\]\(([^)]+)\)", prose): + url = urlsplit(raw.strip().strip("<>")) + if url.scheme or url.netloc: + continue + target = (source.parent / unquote(url.path)).resolve() if url.path else source + require(target.is_relative_to(root.resolve()) and target.exists(), f"Broken repository link: {source.name}: {raw}") + if url.fragment and target.suffix == ".md": + headings = re.findall(r"^#+\s+(.+)$", target.read_text(), re.M) + ids = {re.sub(r"[^\w\- ]", "", h.lower().replace("`", "")).replace(" ", "-") for h in headings} + require(unquote(url.fragment) in ids, f"Missing anchor: {raw}") + checked += 1 + return checked + + +def index_text(manifest): + lines = ["", "", + "# Implementation order", "", "Generated from `tasks.json`; print again with `plan.py index` after status changes.", + "Every implementation task is pending until source-bound evidence passes.", "", + "Use `python3 plans/multi-raft/cell-groups/plan.py show CG-01` for a complete", + "executor brief. Replace the ID to select another task. Commands are displayed,", + "never executed by the plan runner. Read the [design](design.md),", + "[protocols](protocols.md) and [qualification](qualification.md) first.", "", + "| Task | Deliverable | Dependencies | Status | Required cases |", "| --- | --- | --- | --- | ---: |"] + by_id = {task["id"]: task for task in manifest["tasks"]} + for tid in ordered(manifest["tasks"]): + task = by_id[tid] + lines.append(f'| {task["id"]} | {task["title"]} | {", ".join(task["depends_on"]) or "None"} | {task["status"]} | {len(task["test_cases"])} |') + lines.extend(["", "CG-01 through CG-08 establish the shared runtime and public cutover. CG-09 through", + "CG-13 add movement, adaptive scheduling, hibernation and complete managed recovery.", + "CG-08 adapts effects before CG-07 public cutover. CG-14 can proceed after CG-07;", + "CG-13 requires its real-infrastructure harness.", + "CG-15 and CG-16 qualify the combined artifact. These milestones are not separate", + "production architectures. No full delivery claim precedes the final gate.", ""]) + return "\n".join(lines) + + +def brief(task, manifest, baseline, root): + lines = [f'# {task["id"]}: {task["title"]}', "", f'Planned at `{manifest["planned_at"]}`; reviewed source `{baseline["source_revision"]}`.', + f'Status {task["status"]}; priority {task["priority"]}; effort {task["effort"]}; risk {task["risk"]}.', + f'Dependencies: {", ".join(task["depends_on"]) or "none"}.', "", + "Implement TiKV-style groups containing whole independent SQLite cells. One physical group owns", + "consensus, placement, contiguous apply, checkpoint/archive and recovery; each cell retains", + "identity, transactions, receipt dedupe, LTX and query version. Normal transfers preserve cell", + "history; disaster recovery fences old writers and creates fresh incarnations. No cross-cell", + "transactions, old-format migration or quorum bypass is allowed.", "", + "Read this repository's AGENTS.md and plans/multi-raft/cell-groups/{design,protocols,qualification}.md.", + "Use Rust 1.98.1, locked Cargo and a private writable workspace-volume CARGO_TARGET_DIR.", + "Keep cellar-logic pure; use execution-domain I/O/timers, typed failures and joined owners.", + "Use leaf.rs with sibling leaf_tests.rs or component/mod.rs. Markers remain 1 with fresh domains.", + "Use native_raft_snapshot.rs/support/raft_cluster.rs real-image fixtures, not stub markers.", + "Do not enable cellar_internal_tests without its external harness. Preserve licenses and secrets.", "", + "## Drift and source context", "", "Run `python3 plans/multi-raft/cell-groups/plan.py preflight --task " + task["id"] + "`.", + "Expected exit 0. A changed input requires review and a recorded successor baseline, not blind rehashing.", ""] + for name in task["anchors"]: + anchor = baseline["anchors"][name] + lines.extend([f'`{anchor["path"]}:{anchor["line"]}` at the reviewed revision:', "", "```", anchor["excerpt"], "```", ""]) + lines.extend(["## Scope", "", "Edit only these owner paths/subtrees and the listed new outputs:", ""]) + lines.extend(f'- `{path}`' for path in task["scope"] + task["creates"]) + lines.extend(["", "Changes to callers outside this scope require a recorded scope update and matching regression tests.", + "Keep unrelated work. Use codex/ branches and the active PR; no merge is implied.", "", "Out of scope:", ""]) + lines.extend(f'- {item}' for item in task["out_of_scope"]) + lines.extend(["", "## Steps and verification", ""]) + target = root / f'crates/cellar/tests/{task["test_target"]}.rs' + lines.append("Test target: " + ("present; inspect its real assertions." if target.is_file() else "PLANNED OUTPUT, currently absent. Create it; absence is not a pass.")) + prefix = cargo(manifest["toolchain"], "test", "-p", "cellar", "--test", task["test_target"], "--locked", "--") + for n, step in enumerate(task["steps"], 1): + lines.extend(["", f'### Step {n}', "", step["action"], "", "After implementation, verify these exact cases (each must select and pass one test):", "", "```sh"]) + lines.extend(shlex.join(prefix + [name, "--exact", "--test-threads=1"]) for name in step["tests"]) + lines.append("```") + lines.extend(["", "## Required behavior", ""]) + for case in task["test_cases"]: + lines.append(f'- `{case["name"]}` ({", ".join(case["invariants"])}): {case["oracle"]}') + lines.extend(["", "## Complete command/evidence gate", "", "First check inventory contains every named case; no zero-test success is accepted.", "", "```sh"]) + lines.extend(shlex.join(command) for command in commands_for(task, manifest)) + lines.extend(["```", "", "Every command must exit 0 with retained hashed output at the exact source revision.", + "Record environment, limitations, executable/source hashes and required deployment profiles", + "using qualification.md. Set DONE only after evidence validation and dependency completion.", + "Regenerate implementation-plan.md with plan.py index. A plan-only result is not DONE.", "", "## Stop and reconcile", ""]) + lines.extend(f'- {item}' for item in task["stop_conditions"]) + lines.extend(["", "Review protocol changes at their owning boundary and update source anchors, tests and docs together.", + "Do not weaken a failed safety assertion or discard failed load runs. CG-16 reruns every capability", + "at one release source; earlier isolated passes do not qualify a later combined artifact.", ""]) + return "\n".join(lines) + + +def load(package=PACKAGE, root=ROOT): + manifest, baseline, profiles = (read_json(package / name) for name in ("tasks.json", "baseline.json", "profiles.json")) + stats = validate_contract(manifest, baseline, profiles, root) + return manifest, baseline, profiles, stats + + +def preflight(baseline, root, names=None): + for name, anchor in baseline["anchors"].items(): + if names is not None and name not in names: + continue + content = git_bytes(root, baseline["source_revision"], anchor["path"]) + require(hashlib.sha256(content).hexdigest() == anchor["sha256"], f"Baseline is not pinned to source: {name}") + require(anchor["excerpt"] in content.decode(), f"Excerpt differs at source: {name}") + lines = content.decode().splitlines() + start = anchor["line"] - 1 + excerpt_lines = anchor["excerpt"].split("\n") + require(lines[start:start + len(excerpt_lines)] == excerpt_lines, + f"Excerpt line differs at source: {name}") + require(digest(within(root, anchor["path"])) == anchor["sha256"], f"Source drift requires review: {anchor['path']}") + + +def require_release_inputs(root, revision): + require(REVISION.fullmatch(revision), "Invalid qualified source revision") + compared = subprocess.run(["git", "diff", "--quiet", revision, "HEAD", "--", *RUNTIME_PATHS], + cwd=root, capture_output=True, check=False) + require(compared.returncode == 0, "Current executable inputs differ from qualified source") + changed = subprocess.check_output(["git", "status", "--porcelain", "--untracked-files=all", "--", *RUNTIME_PATHS], + cwd=root, text=True) + require(not changed, "Release executable inputs are dirty") + + +def validate(package=PACKAGE, root=ROOT): + manifest, baseline, profiles, stats = load(package, root) + require((package / "implementation-plan.md").read_text() == index_text(manifest), "Implementation index is stale; regenerate with plan.py index") + stats["local_links"] = check_links(package, root) + profile_digest = digest(package / "profiles.json") + for task in manifest["tasks"]: + for path in task["evidence"]: + check_evidence(task, read_json(within(root, path)), manifest, baseline, profiles, profile_digest, root) + stats["done"] = sum(t["status"] == "DONE" for t in manifest["tasks"]) + return manifest, baseline, profiles, stats + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + sub = parser.add_subparsers(dest="command", required=True) + for name in ("validate", "ready", "index", "release"): + sub.add_parser(name) + sub.add_parser("show").add_argument("task") + sub.add_parser("preflight").add_argument("--task") + args = parser.parse_args() + if args.command == "index": + manifest, _, _, _ = load() + print(index_text(manifest), end="") + return 0 + manifest, baseline, profiles, stats = validate() + by_id = {task["id"]: task for task in manifest["tasks"]} + if args.command == "validate": + print(json.dumps(stats, indent=2)) + elif args.command == "preflight": + require(not args.task or args.task in by_id, "Unknown task") + preflight(baseline, ROOT, by_id[args.task]["anchors"] if args.task else None) + print("Reviewed source hashes and excerpts match. Runtime tasks remain pending.") + elif args.command == "show": + require(args.task in by_id, "Unknown task") + print(brief(by_id[args.task], manifest, baseline, ROOT)) + elif args.command == "ready": + ready = [t["id"] for t in manifest["tasks"] if t["status"] == "TODO" + and all(by_id[d]["status"] == "DONE" for d in t["depends_on"])] + print(json.dumps({"dependency_ready": ready, "done": stats["done"], "note": "Ready means eligible to implement, not implemented."}, indent=2)) + elif args.command == "release": + require(stats["done"] == len(manifest["tasks"]), "Release blocked: implementation tasks/evidence incomplete") + revision = read_json(within(ROOT, by_id["CG-16"]["evidence"][-1]))["source_revision"] + require_release_inputs(ROOT, revision) + print(f"Release evidence verified for {revision}; only the recorded topology/load envelope is qualified.") + return 0 + + +if __name__ == "__main__": + try: + sys.exit(main()) + except (PlanError, OSError, KeyError, TypeError, json.JSONDecodeError) as error: + print(f"ERROR: {error}", file=sys.stderr) + sys.exit(1) diff --git a/plans/multi-raft/cell-groups/profiles.json b/plans/multi-raft/cell-groups/profiles.json new file mode 100644 index 000000000..1f9aa2e16 --- /dev/null +++ b/plans/multi-raft/cell-groups/profiles.json @@ -0,0 +1,630 @@ +{ + "format": 1, + "status": "unqualified_candidate_gates", + "note": "All numbers are proposed acceptance targets, not measurements. Freeze profile digest before each campaign; retain misses and obtain a reviewed revision before changing targets. All expected fault refusals remain in the ledger.", + "profiles": [ + { + "id": "idle_density", + "required": true, + "description": "Matched 4096-cell idle cohorts; compare occupancy 64 with occupancy 1, including control costs and complete physical inventory.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 300, + "offered_requests_per_second": 0, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "control_bytes_ratio": { + "max": 0.1 + }, + "cpu_core_ratio": { + "max": 0.65 + }, + "baseline_cpu_cores": { + "min": 0.05 + }, + "baseline_control_bytes": { + "min": 1 + }, + "actual_cells": { + "eq": 4096 + }, + "measured_groups_occupancy64": { + "max": 128 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 64 + ], + "ratios": { + "control_bytes_ratio": { + "numerator": "control_bytes_per_second", + "denominator": "control_bytes_per_second", + "control_occupancy": 1 + }, + "cpu_core_ratio": { + "numerator": "cpu_cores", + "denominator": "cpu_cores", + "control_occupancy": 1 + } + } + }, + { + "id": "mixed_quorum", + "required": true, + "description": "Fixed 100 requests/s, 80% writes and 20% strong/AtLeast reads, uniform cells, Quorum acknowledgements.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "refused_requests": { + "max": 0 + }, + "ambiguous_requests": { + "max": 0 + }, + "latency_samples": { + "min": 60000 + }, + "p99_ratio_to_occupancy1": { + "max": 1.2 + }, + "max_archive_lag_seconds": { + "max": 60 + }, + "p99_ms": { + "max": 200 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 8, + 32, + 64, + 128 + ], + "ratios": { + "p99_ratio_to_occupancy1": { + "numerator": "p99_ms", + "denominator": "p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "mixed_s3", + "required": true, + "description": "Identical trace to mixed_quorum with S3 acknowledgements against real RustFS.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 8, + 32, + 64, + 128 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "refused_requests": { + "max": 0 + }, + "ambiguous_requests": { + "max": 0 + }, + "latency_samples": { + "min": 60000 + }, + "p99_ratio_to_occupancy1": { + "max": 1.2 + }, + "max_archive_lag_seconds": { + "max": 60 + }, + "p99_ms": { + "max": 1000 + } + }, + "historical_control": true, + "historical_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", + "gate_occupancies": [ + 8, + 32, + 64, + 128 + ], + "ratios": { + "p99_ratio_to_occupancy1": { + "numerator": "p99_ms", + "denominator": "p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "hot_cold", + "required": true, + "description": "One cell receives 50% of writes; compare adapted grouping with occupancy-one replacement at identical offered traffic.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 1, + 64 + ], + "duration_seconds": 600, + "offered_requests_per_second": 100, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "cold_p99_ms": { + "max": 250 + }, + "cold_p99_ratio_to_control": { + "max": 1.25 + }, + "refused_requests": { + "max": 0 + }, + "hot_cell_isolated": { + "eq": 1 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": { + "cold_p99_ratio_to_control": { + "numerator": "cold_p99_ms", + "denominator": "cold_p99_ms", + "control_occupancy": 1 + } + } + }, + { + "id": "split_merge", + "required": true, + "description": "Foreground commands/strong reads during repeated split, merge, learner movement and controller restarts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 900, + "offered_requests_per_second": 50, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "completed_transitions": { + "min": 30 + }, + "max_transition_pause_ms": { + "max": 10000 + }, + "peak_transition_slot_ratio": { + "max": 1 + }, + "history_checks": { + "min": 45000 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "hibernate_wake", + "required": true, + "description": "Separate healthy retained leader, cold execution cache, parked leader loss, minority wake and alarm-triggered wake cohorts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 600, + "offered_requests_per_second": 10, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "healthy_wake_samples": { + "min": 1000 + }, + "healthy_cached_wake_p99_ms": { + "max": 500 + }, + "leader_loss_wake_max_ms": { + "max": 10000 + }, + "missed_due_effects": { + "max": 0 + }, + "parked_control_ratio": { + "max": 0.1 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": { + "parked_control_ratio": { + "numerator": "parked_control_bytes_per_second", + "denominator": "active_control_bytes_per_second" + } + } + }, + { + "id": "rustfs_disaster", + "required": true, + "description": "Owned all-voter erasure after reclaim, before/after topology publication, managed and unmanaged group restore from real RustFS.", + "topology": "local_containers", + "cells": 256, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 900, + "offered_requests_per_second": 20, + "repetitions": 3, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "erased_voter_disks": { + "min": 3 + }, + "restored_cells": { + "eq": 256 + }, + "old_token_acceptances": { + "max": 0 + }, + "old_host_restart_acceptances": { + "max": 0 + }, + "peer_seed_bytes": { + "eq": 0 + }, + "restored_receipt_callbacks": { + "eq": 0 + }, + "scenarios_completed": { + "min": 6 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "independent_domain_loss", + "required": true, + "description": "Actual five-node deployment across at least three independent failure domains; isolate one domain, then demonstrate two-domain quorum refusal.", + "topology": "independent_hosts", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 1800, + "offered_requests_per_second": 50, + "repetitions": 1, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "independent_failure_domains": { + "min": 3 + }, + "actual_hosts": { + "min": 5 + }, + "successful_single_domain_failover": { + "eq": 1 + }, + "writes_without_quorum": { + "eq": 0 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + }, + { + "id": "soak_24h", + "required": true, + "description": "24-hour mixed traffic, effect scheduling, bounded S3 outages, transfers, hibernation and rolling process restarts.", + "topology": "local_containers", + "cells": 4096, + "nodes": 5, + "voters": 3, + "occupancies": [ + 64 + ], + "duration_seconds": 86400, + "offered_requests_per_second": 50, + "repetitions": 1, + "image_modes": [ + "materialized", + "extents" + ], + "checks": { + "lost_acknowledged_effects": { + "max": 0 + }, + "duplicate_mutations": { + "max": 0 + }, + "stale_strong_reads": { + "max": 0 + }, + "double_writers": { + "max": 0 + }, + "unexpected_errors": { + "max": 0 + }, + "peak_rss_limit_ratio": { + "max": 0.85 + }, + "unjoined_owners": { + "max": 0 + }, + "measured_seconds": { + "min": 86400 + }, + "max_archive_lag_after_recovery_seconds": { + "max": 60 + }, + "steady_rss_growth_ratio": { + "max": 1.1 + }, + "completed_history_checks": { + "min": 4320000 + } + }, + "historical_control": false, + "historical_revision": null, + "gate_occupancies": [ + 64 + ], + "ratios": {} + } + ] +} diff --git a/plans/multi-raft/cell-groups/protocols.md b/plans/multi-raft/cell-groups/protocols.md new file mode 100644 index 000000000..a4930dbe7 --- /dev/null +++ b/plans/multi-raft/cell-groups/protocols.md @@ -0,0 +1,217 @@ + + +# Durable protocols and crash cuts + +Status: required target behavior. Names below are contracts to implement. +The [design](design.md) defines identity, partitioning and storage. A catalog +record is never a vote, a majority proof or infrastructure fencing evidence. + +## Invariants + +| ID | Required invariant | +| --- | --- | +| I01 | Cell, group, incarnation, route and schema identities are checked independently. | +| I02 | Every acknowledged Quorum output has a real persisted majority and a published exact cell effect. | +| I03 | Receipts and cell versions cannot be borrowed across cells; retries preserve identity through movement. | +| I04 | Durable group apply is contiguous and references a complete synced image/manifest closure. | +| I05 | Strong reads use current-term quorum authority and a coherent target cell image. | +| I06 | S3 acknowledgements require a retained complete group checkpoint/tail closure. | +| I07 | Reclamation preserves every live group, snapshot, reader, archive and transition dependency. | +| I08 | Membership changes preserve quorum intersection, disk lineage and failure-domain constraints. | +| I09 | Split/merge admits at most one writable owner of every cell, including after lost replies. | +| I10 | Hibernation never manufactures authority or bypasses a fresh wake/read barrier. | +| I11 | Zero-quorum recovery fences all possible old writers and creates fresh incarnations. | +| I12 | Work, memory, file descriptors, catalog scans and shutdown remain bounded and owned. | +| I13 | Alarms/outbox/inbox preserve committed intent and stable effect identity across movement. | +| I14 | Release claims are tied to exact source, actual tests, workloads and infrastructure. | + +## Enrollment + +Resolve the current range, materialize its group birth if absent, and obtain +current-term leadership. Propose `Enroll(CellHistory, schema, initial_image)` +under that route epoch. The group checks the key is in its active range and no +other live incarnation has enrolled that ID. The command is idempotent by its +stable creation identity. Only committed/published enrollment becomes visible. + +| Crash/collision | Recovery | +| --- | --- | +| Birth CAS result unknown | Read the canonical birth/mutation journal; never allocate competing voter history. | +| Schema image synced, enrollment uncommitted | Keep it unqueryable; replay/reconcile the same request or collect an orphan after all pins expire. | +| Enrollment committed, response lost | Return the recorded cell history/result on retry; do not run initialization twice. | +| Concurrent creators | One committed enrollment wins; mismatch refuses rather than changing class/incarnation. | +| Split seals during preparation | Revalidate before proposal; an accepted-before-seal entry belongs to the frozen prefix. | + +## Apply, checkpoint and archive + +For entry N, verify the cell base/sequence, stage the result, sync its immutable +image closure, sync changed manifest pages, then atomically persist the group root +and applied=N in the engine. Publish and acknowledge only after the marker succeeds. +Old roots remain pinned until no recovery/reader/transfer owner needs them. + +| Crash cut | Required outcome | +| --- | --- | +| Before image sync | Recover prior root; no output credited. | +| Image durable, manifest incomplete | Prior root remains authoritative; staged image is not visible. | +| Manifest durable, engine marker absent | Prior root remains authoritative; replay the same log entry without callback execution. | +| Marker durable, response/publication lost | Recover the new complete root; receipt retry returns the committed effect. | +| Any referenced image/page missing | Refuse group admission; never substitute an empty cell. | +| Later cell staged before earlier cell finishes | No applied-prefix hole and no early output settlement. | + +A checkpoint pins the whole root at K, including empty enrollments, membership, +range/transition state, receipt floors and effect summaries. Stream bounded chunks +and verify the receiver's own synced closure before atomic install. A successful +transfer is not proof that another replica has installed it. A new voter receives +fresh member/disk identity and joins only through committed membership changes. + +An archive publishes a complete checkpoint plus a contiguous committed tail to K. +The tail addresses cells and validates each base chain; administrative entries +also replay. Do not rerun commands. Publish coverage only after all dependencies +are durable and independently verifiable. A dependency failure in one cell blocks +group coverage. Unchanged immutable cell objects can be reused between cuts. + +Quorum writes may proceed during an S3 outage within explicit disk/unarchived-byte +budgets. S3-mode outputs wait for confirmed coverage. Log reclamation requires the +covering local checkpoint and configured archive retention proofs, plus transfer +pins. Shared engine purge must preserve other groups' live ranges. Readers and +in-flight installs retain object pins until their physical owners finish. + +## Split and merge + +Use the same transition state machine for one parent to multiple child ranges +(split) and compatible adjacent parents to one child (merge). Create fresh child +group IDs and retire every parent; this avoids reusing a partially changed parent +as a second activation path. Initially seed children on the parents' qualified +voter set where possible; ordinary learner movement can rebalance afterward. +Pre-copy immutable images while parents are live, then send the final delta after +sealing. Hibernation and membership changes must be resolved before sealing. + +Every transition pins its exact parents, range epochs, child births, final cell +partition and resource reservations in a durable catalog journal. There is a +namespace topology CAS, but no catalog I/O on an ordinary already-routed command. + +| State | Durable action and admission rule | +| --- | --- | +| Planned | CAS a unique intent against the exact current range tree; reserve child/transfer resources. Parents still serve. | +| Prepared | Children are non-serving learners of transfer state; copy immutable bytes. No child has business admission. | +| SealAuthorized | CAS the journal from Prepared to an irreversible seal decision before issuing any parent Seal request. The exact retained decision grants permission to ask the parents to seal, not a quorum proof. Cancellation can no longer win. | +| Sealing | Every parent commits `Seal(transition, children, ranges)` after all earlier accepted entries. Seal covers enrollment, commands and strong reads. | +| Sealed | All parent quorums have durably applied their seal cuts. Capture and pin the exact cell/effect/receipt closure. The operation remains roll-forward only. | +| Installed | Every child's own quorum commits and applies its seed from the sealed cuts, with fresh Raft term/vote state. Children remain non-serving. | +| Archived | Publish and independently verify an immutable remote transfer closure containing every sealed parent cut, child partition/seed root and all cell dependencies. Pin it in the transition journal before range publication. | +| Published | CAS the namespace root from all exact parent ranges to all exact child ranges at once. Validate complete equal union and no gaps/overlap. Unknown CAS results reconcile by transition ID. | +| Active | Each child commits activation binding the retained publication certificate. Serving requires that activation plus current-term quorum authority. | +| Retired | Parents persist retirement/redirect tombstones. Release old data only after child archive coverage and all dependency pins are safe. | + +Child install receipts bind the source seals, own committed seed, membership and +manifest digest. Publication also requires the exact verified remote transfer +closure, so losing every voter immediately afterward can still recover all prior +S3-protected effects under the child partition. A controller observation without +those proofs cannot publish. S3 outage may stall topology changes; it does not +change ordinary Quorum acknowledgement behavior. +Publication certificates are immutable journal entries, not a mutable head URL. +Children validate their exact transition and refuse replay after later retirement. +Source tombstones remain effective across restart, delayed messages and catalog +unavailability. The whole transition is one fenced owner operation; concurrent +controllers reconcile the same journal instead of selecting different children. + +| Failure | Required behavior | +| --- | --- | +| Controller lost before seal authorization | Resume or CAS Prepared to Cancelled; seal authorization and cancellation compete on the same journal state. Revoke child activation permits before releasing reservations. | +| Seal reply lost | Treat outcome as unknown; reconcile committed log/receipts. Do not reopen a parent based on timeout. | +| One merge parent sealed, another unavailable | Sealed parent remains fenced; unsealed parent's earlier writes join its eventual seal cut. No destination admission. Resume when quorum returns or use the disaster protocol. | +| Child install incomplete | Keep parents sealed and all retained pins. Repair/finish the same child or use authorized disaster recovery. | +| Transfer closure upload/verification incomplete | Do not publish child ranges. Keep parent and child pins; retry the same immutable closure. | +| Directory CAS response lost | Read journal/root and confirm the exact winner. Never roll back a possibly published transition. | +| Publication succeeds, some children not active | Only activated children serve their disjoint ranges; finish activation. Parent never resumes. | +| All disks lost immediately after publication | Recover from the journal-pinned remote transfer closure plus any archived child tails; no newly acknowledged S3 effect may disappear. | +| Delayed old proposal/read/activation | Check history, epoch, transition and local sealed/retired state; refuse stale authority. | +| Parent disks fail after publication | Children serve from installed state. Parent deletion does not establish archive safety. | + +After SealAuthorized, automatic cancellation is forbidden, even if no Seal reply +has arrived. Parent Seal admission requires that exact authorization certificate. +Availability can pause while completing a transition; preserving single-writer +ownership takes precedence. Pre-seal cancellation also needs a durable controller +decision in the same journal. Cancelled intents can never mint seal/activation +permits. Unknown authorization-versus-cancellation CAS results reconcile the +recorded winner before issuing work; timeout cannot establish that cancellation won. + +Normal transfer preserves cell incarnation, mutation sequence, exact receipt +contents/client floors and effect identities. Seed provenance authenticates the +original commit witnesses. New group positions do not replace per-cell ordering. +In-flight outbox delivery is settled or journaled with the original ID; a lost +delivery response never authorizes a new identity. The target may retry that ID +under the receiver's existing dedupe contract. + +## Coordinated hibernation + +Closing an execution cache is independent from parking the Raft group. For the +first hibernation implementation require all configured voters to participate; +a healthy majority can continue active work if another voter is unavailable, but +cannot park. This is a conservative availability tradeoff, not a Raft safety rule. +Learners must have an explicit quiesced/catch-up state; no unbounded transfer may +be abandoned merely to park. + +1. The current leader checks no unresolved proposal/read, transition, membership + change, snapshot install or unsafe effect dispatch exists. Freeze new admission + while negotiating. Publish a bounded next-due-effect summary to a node wake owner. +2. Commit `PreparePark(generation, term, membership_revision)` and apply through it. + Every voter syncs a local acknowledgement for that exact generation and cut. +3. The leader assembles all acknowledgements, syncs its park certificate and sends + authenticated park messages. Each recipient persists its parked generation + before suppressing routine group ticks. Partial delivery is safe: an active + peer's messages wake parked peers. Node-level transport remains available. +4. API request, peer message, due effect, topology operation or local restart + wakes the group. Persist closure of that park generation before new admission; + late park messages for a closed generation cannot re-park it. +5. Resume ordinary Raft using retained term/vote/log. Send wake hints to peers and + obtain current-term commit/read authority before serving. Wake messages and + retained leadership are not quorum proofs. Missing a quorum refuses/waits. + +Use execution-domain timers and a bounded node-level due index; do not allocate a +thread per cell/group. A restart may conservatively wake all locally parked groups +in bounded batches. Never reset term/vote or initialize a missing disk. Include +park state/tombstones in local recovery and checkpoint contracts. The scheduler +must reconcile alarm changes after leader loss, and an overdue alarm wakes the +group immediately rather than waiting for a client. + +Required fault cuts include lost prepare/ack/park/wake messages, delayed duplicate +park after a new command, leader loss while parked, minority wake, mismatched +membership revision, alarm changes during negotiation and all-voter restart. +Measure healthy retained-state wake separately from election, image-cache restore +and S3 disaster recovery. Do not claim wake percentiles from a few successful trials. + +## Zero-quorum recovery and transition closure + +Recovering a live range changes authority, so serialize it with namespace topology +and fleet lifecycle operations. Pin an exact recovery set; verify its full remote +closure before requesting infrastructure fencing. Never choose a newer head after +the operator request was admitted. Existing Quorum versus S3 recovery-point limits +continue to apply. + +If the group participates in an incomplete split/merge, recover the connected +transition component, including every parent/child that could still activate. +Resolve the durable publication journal first. Fence the launch authority and all +possible voting lineages in that component, including removed/replacement hosts +whose restart could violate the selected boundary. Partial fencing cannot publish +new ownership. A list of current online nodes is not a complete fencing inventory. + +For an unpublished transition, choose complete archived parent cuts and reconstruct +their non-overlapping domain under new groups. For a published transition, require +child recovery sets or a retained transfer closure plus each child's archived +tail; a parent checkpoint alone cannot cover child acknowledgements. Refuse if the +selected closure cannot account for every S3-acknowledged effect in the retained +recovery horizon. An intentionally older rollback needs explicit operator scope. + +Create fresh group and cell incarnations, node/member/disk/process identities and +Raft genesis state. Restore SQLite, receipts and client floors without callbacks. +Publish replacement ranges through one fenced topology mutation, then require a +fresh quorum and committed activation. Managed fleet import/maintenance/recovery +must share that same mutation journal and resource debt; no unmanaged bypass. +Unknown fencing/publication outcomes resume the identical request. They never +unfence old hosts or select another cut. + +The mandatory RustFS drill deletes all owned voter disks after independently +checking the ledger and archive cut. It verifies every cell, old-token rejection, +receipt retries, new writes and retained-disk restart. Repeat during split/merge +crash cuts and after physical log reclamation. This remains group/component +recovery, not a globally consistent snapshot of unrelated groups or external effects. diff --git a/plans/multi-raft/cell-groups/qualification.md b/plans/multi-raft/cell-groups/qualification.md new file mode 100644 index 000000000..642279729 --- /dev/null +++ b/plans/multi-raft/cell-groups/qualification.md @@ -0,0 +1,225 @@ + + +# Qualification and evidence contract + +Status: proposed workload contracts and release gates, not measured results. +`profiles.json` freezes candidate requirements. Change a missed target only through +an explicit reviewed profile revision, retaining the old run and its failure. +Do not silently raise budgets, remove failures, retry mutations with new IDs, or +compare different logical cell populations to manufacture an improvement. + +## Tests and release levels + +Every task has exact named native cases. Its executor brief first lists tests, +then runs each required case with `--exact --test-threads=1`. The evidence validator +requires the name in the inventory and a successful one-test result; Cargo exiting +zero after selecting no tests is a failure. These test targets are planned outputs, +not available commands until their owning task creates them. + +Use real SQLite/LTX and Raft Engine for state assertions. Scheduler-only fixtures +with stub image markers can test fairness but cannot qualify durability. Both +materialized and extent state, empty enrollment, managed and unmanaged groups, +all read modes and mixed Quorum/S3 commands must appear in the matrix. Native +test execution is serial between fixtures, concurrent within each fixture. + +| Level | Evidence | Claim allowed | +| --- | --- | --- | +| Package | Python validator/tests, source preflight and local links | Executable plan only | +| Native | All task cases on real engines plus injected crash/process cuts | Exact tested functional boundaries | +| Local deployment | Five owned Cellar containers/processes, actual RustFS, enforced resource limits and complete ledgers | Local measured workload only | +| Independent domains | Actual separate machines/failure domains and observed physical isolation under load | Only the measured topology/domain-loss envelope | +| Release | All tasks, regressions and required profiles at one immutable source | Complete cell-group capability within that envelope | + +The final release requires all profiles marked required in `profiles.json`. +Container counts alone cannot satisfy the independent-domain profile. Million-cell +and Q2–Q4 scale claims remain separate larger experiments from the parent roadmap. + +## Planned harness interface + +CG-14 implements `tools/perf/cell_groups.py`; it does not exist at plan delivery. +The following is its required CLI, not a command claimed to work today: + +```text +python3 tools/perf/cell_groups.py preflight --config PRIVATE_CONFIG +python3 tools/perf/cell_groups.py run --profile PROFILE --config PRIVATE_CONFIG --output FRESH_RUN_DIRECTORY +python3 tools/perf/cell_groups.py verify --run RUN_DIRECTORY +python3 tools/perf/cell_groups.py stop --run RUN_DIRECTORY +``` + +The private configuration must bind a run UUID, explicit Docker context or process +host inventory, immutable Cellar/RustFS image digests or executable manifest, +source revision, dedicated data roots, credentials-file paths, port allocation, +node CPU/memory/swap/disk limits, failure domains and fencing-authority endpoint. +Validate all required fields; missing values never fall back to the current Docker +context or a default data directory. Credentials and unrestricted Docker inspect +output stay out of reports. Unsupported resources fail before provisioning. + +Use existing `tools/perf/raft_fleet.py` for managed-node provisioning patterns and +`tools/perf/process_recovery.py` for native process SIGKILL/erasure proof. Extend +their shared owners; do not implement a competing fake object-store or voter +fixture. Own resources by immutable IDs, run labels and exact paths. The destructive +profile must confirm ownership and join stopped processes before removing only +the recorded voter roots. Keep RustFS data and retained evidence outside voter +roots, build cleanup and global prune paths. `stop` retains data for inspection. + +## Workload accounting + +Build once before timing. Record source, compiler/features, binary digest, actual +CPU/memory limits, disk/filesystem, host activity, RustFS digest and TLS topology. +Run cohorts serially with fresh identical initial data and a frozen arrival trace. +Compare the old one-cell baseline against the replacement, and compare group +occupancies 1/8/32/64/128 within the replacement. Capture measured actual groups, +not only configured target occupancy. Each logical cell has a separate SQLite DB. + +The offered-load ledger records stable cell/client/sequence, input digest, request +start, deadline, response/timeout/cancel/refusal, returned result, cell version, +group witness and requested/achieved durability. Keep every offered request in +throughput/error denominators. Reconciliation is a separate phase and cannot +retroactively turn a timed-out request into a successful timed sample. + +Independently verify all acknowledged effects and receipt results on every serving +replica and the selected recovery cut. Check per-cell real-time strong-read order, +at-most-once mutation by command identity, allowed ambiguous outcomes and no +overlapping writable owners. Deliberately corrupt oracle inputs to prove it detects +lost effects, borrowed versions, double execution and stale strong reads. + +Measure CPU cores, process RSS, descriptors, group/cell counts, outgoing control +bytes below/above TLS with the exact boundary stated, log/image/manifest write +bytes, fsyncs, S3 PUT/GET bytes/requests and retained/archive lag. Report p50/p95/p99 +and sample counts by durability, hot/cold cell and fault phase. Warmup, setup, +election, cache restore, archive drain and joined shutdown are separate intervals. + +For ratios, require a positive meaningful matched baseline; a zero denominator +cannot pass a reduction gate. Resource peaks use actual OS/engine observations, +not configured limits or synthetic host metrics. A recovered follower is not proof +that every replica or every cell recovered. Selected-cut Quorum suffix loss is +reported separately from missing S3-protected history. + +## Required fault schedule + +Kill/restart at image sync, manifest sync, marker commit, enrollment, snapshot +install, archive publication, reclaim, seal authorization, parent seal, child seed, +root publication, activation, retirement, park and wake generation closure. +Inject corrupt/missing objects, duplicate/delayed messages, stale routes, unknown +CAS responses, leader/controller loss, a slow/minority follower, disk-full/sync +failure, network partitions and bounded S3 outage. Run foreground writes and +strong reads while splitting/merging and moving replicas. + +For zero-quorum tests, verify the archive/ledger boundary first, fence all possible +old voting lineages and demonstrate launch revocation. Delete all voter disks, +restore complete groups from real RustFS with no surviving peer images, verify +every cell and receipt, reject old tokens/old-host restart, acknowledge new writes, +and restart the fresh quorum on its new disks. Repeat before/after topology +publication and after reclamation. Managed inventory/transition journal failures +must fail closed. Unreconciled old-origin outbox work must remain refused. + +## Task evidence + +Store local raw evidence under an ignored repository-relative directory, for +example `.cellar/cell-groups-evidence/RUN/`; CI may download sanitized artifacts +there before validation. Do not commit credentials, cell data, opaque tokens or +the previously excluded MR-13/MR-14 progress JSON files. Add a relative record path +to a task's `evidence` only after its implementation and checks actually pass. + +A task record uses this shape (example values are placeholders, not evidence): + +```json +{ + "format": 1, + "task_id": "CG-01", + "source_revision": "FULL_40_HEX_COMMIT", + "executable_sha256": "64_HEX", + "qualified_executable_sha256": "DEPLOYED_CELLAR_BINARY_64_HEX_WHEN_PROFILES_ARE_PRESENT", + "environment": "Actual compiler, features, OS and resource observations", + "limitations": "Exact scope and untested boundaries", + "source_inputs": [{"path": "crates/...", "sha256": "64_HEX"}], + "commands": [{"argv": ["cargo", "+1.98.1", "test", "..."], "exit_code": 0, + "output": ".cellar/.../stdout.log", "sha256": "64_HEX"}], + "profiles": [{"path": ".cellar/.../profile.json", "sha256": "64_HEX"}] +} +``` + +`source_inputs` must include the task's pinned source anchor paths and its test +target; the validator verifies each hash against the named Git revision. Include +the remaining changed sources in the record as well. Test inventory and exact +one-test outputs must be hashed and retained. Every command in `plan.py show` +verification must appear with exit zero. Earlier failed runs stay in the run +directory and are described in the final report; do not aggregate passes across +different binaries into one task record. + +For CG-16, one record must contain all new tasks' verification commands at the same +release revision, plus the repository checks listed below and all required profile +reports. Dependencies' earlier passes alone cannot qualify a changed release. +The task record must not claim it verified different executable inputs. Later +plan-only status/evidence commits are permitted; the release command compares all +other repository inputs against the qualified source and rejects dirty files, +including examples, vendor code, build configuration and newly added input paths. + +Profile reports bind `profile_id`, `source_revision`, `executable_sha256`, the +SHA-256 of the entire frozen `profiles.json`, actual environment observations, +numeric `metrics`, and hashed artifacts for `ledger`, `measurements`, `topology`, +`fault_schedule` and `verification`. The profile must retain its controls, +repetitions and parameter matrix in those artifacts. `plan.py` checks every numeric +gate, required topology and artifact hash. A bare `passed: true` is insufficient. +Values must be finite and nonnegative; null, missing, NaN and boolean measurements +refuse. The report and every replacement cohort must match the task record's +`qualified_executable_sha256`. The primary `executable_sha256` identifies the native +test artifact; these separate fields avoid equating it with the deployed server. + +The measurements artifact is JSON with `metrics` equal to the report summary and +`cohorts` containing exactly the declared occupancy × image mode × repetition +matrix. Each cohort records: + +```text +variant: replacement | historical +occupancy, image_mode, repetition +source_revision, executable_sha256, ledger_sha256 +cells, nodes, voters, offered_requests_per_second, measured_seconds +metrics: numeric observations for that cohort +``` + +Historical controls are required only where `historical_control` is true, at +occupancy one and the pinned historical revision. Replacement controls and +candidates use the same qualified binary. Every cohort must meet its frozen +population, resources, arrival rate and minimum duration. Correctness/error gates +apply to every cohort, including historical and occupancy-one controls. Performance +gates apply to `gate_occupancies`. Summaries use the worst eligible cohort: maximum +for a max gate, minimum for a min gate, and equality in every cohort for an eq gate. +Never average a slow or failed cohort into a passing result. + +Ratio observations must equal the numerator divided by the matching control's +denominator, with the same image mode and repetition; `ratios` defines the exact +metric names and control occupancy. A missing/zero control refuses the ratio. +Hibernation's active and parked rates are separate phases of the same cohort. +The native harness derives the other measurements from actual ledgers/OS samples; +the plan validator does not replace the workload oracle or establish physical +infrastructure truth from a self-reported number. + +The topology artifact is JSON with `kind` and `nodes`, each containing `id`, +`physical_host` and `failure_domain`. Independent-domain runs require a different +physical host for each node and at least three failure domains. The verification +artifact records `source_revision`, `complete: true`, `ledger_sha256` and +`measurements_sha256`, binding its full history oracle to the actual retained +artifacts. Retain the detailed checks and physical isolation observations there, +not just these mandatory header fields. The fault schedule records requested and +observed events, timing, exact owned resource IDs and phase boundaries for every +cohort. Infrastructure evidence must still be reviewed before advertising a tier. + +## Repository release checks + +Run from the repository root with its own writable workspace target directory: + +```sh +cargo +1.98.1 format --check +cargo +1.98.1 check-layout +python3 tools/check_native_dependencies.py +python3 tools/docs/check.py +cargo +1.98.1 check --workspace --all-targets --all-features --locked +cargo +1.98.1 test --workspace --all-features --lib --tests --locked -- --test-threads=1 +cargo +1.98.1 test --workspace --all-features --doc --locked +cargo +1.98.1 clippy --workspace --all-targets --all-features --locked -- -D warnings +python3 -m unittest discover -s tools/perf -p 'test_*.py' +``` + +Feature-specific and external consumer checks in the SDK CI workflow remain +required. The new package runner supplements CI; it is not a substitute for it. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json new file mode 100644 index 000000000..1451ab7c0 --- /dev/null +++ b/plans/multi-raft/cell-groups/tasks.json @@ -0,0 +1,1986 @@ +{ + "format": 1, + "planned_at": "70b6022c8baba3d0a52de6bd9fa11866bdbd3b3f", + "toolchain": "1.98.1", + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I08", + "I09", + "I10", + "I11", + "I12", + "I13", + "I14" + ], + "tasks": [ + { + "id": "CG-01", + "title": "Separate logical cell history from physical group consensus", + "status": "TODO", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [], + "anchors": [ + "identity", + "codec", + "version", + "ci" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/application/consensus/", + "crates/cellar/durability/raft/version.rs", + "crates/cellar/lib.rs" + ], + "creates": [ + "crates/cellar-logic/cell_group.rs", + "crates/cellar-logic/cell_group_tests.rs", + "crates/cellar/tests/native_cell_groups_contracts.rs" + ], + "test_target": "native_cell_groups_contracts", + "steps": [ + { + "action": "Add dependency-free CellHistory, GroupHistory, CellKey, CellVersion and route/membership epoch types. Specify canonical hash bytes and infinity bounds exactly as design.md. Add private adapters while existing consumers still compile; do not introduce a public legacy/shared mode flag.", + "tests": [ + "identity_and_hash_boundaries" + ] + }, + { + "action": "Encode cell-addressed data and typed administrative entries under new marker-1 semantic domains. Reject unknown tags, duplicate identity fields, oversized declarations and wrong cell/schema/route before allocation or application.", + "tests": [ + "envelope_rejects_wrong_cell_and_domain" + ] + }, + { + "action": "Define version tokens and receipt provenance around stable cell incarnation and monotone mutation sequence. Preserve transfer compatibility within one cell history; refuse old-incarnation and cross-cell tokens. Wire validation through owning Result types.", + "tests": [ + "version_survives_transfer_not_disaster" + ] + }, + { + "action": "Add golden byte/negative tests and bounded decoding fixtures. Map the new module through existing lib.rs paths; keep raft-rs, I/O and serde out of cellar-logic.", + "tests": [ + "codec_limits_before_allocation" + ] + } + ], + "test_cases": [ + { + "name": "identity_and_hash_boundaries", + "oracle": "Empty/control IDs, hash ties, infinity endpoints and sequence overflow reject or order exactly as the canonical specification.", + "invariants": [ + "I01" + ] + }, + { + "name": "envelope_rejects_wrong_cell_and_domain", + "oracle": "Valid LTX under another cell/route/schema and old protocol domains cannot be decoded as authorized data.", + "invariants": [ + "I01", + "I04" + ] + }, + { + "name": "version_survives_transfer_not_disaster", + "oracle": "Same cell incarnation/sequence token remains comparable after a group move; another cell or disaster incarnation refuses.", + "invariants": [ + "I01", + "I03", + "I05" + ] + }, + { + "name": "codec_limits_before_allocation", + "oracle": "Truncated, duplicate and excessive metadata/payload declarations fail within configured memory bounds.", + "invariants": [ + "I01", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I04", + "I05", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "contracts" + }, + { + "id": "CG-02", + "title": "Persist complete group manifests with crash-safe cell image ownership", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01" + ], + "anchors": [ + "manifest", + "apply", + "extent", + "sql" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/extent/", + "crates/cellar-ltx/src/generation.rs" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/mod.rs", + "crates/cellar/durability/raft/cell_groups/manifest.rs", + "crates/cellar/durability/raft/cell_groups/manifest_tests.rs", + "crates/cellar/tests/native_cell_groups_storage.rs" + ], + "test_target": "native_cell_groups_storage", + "steps": [ + { + "action": "Implement authenticated paged copy-on-write cell manifests and a group root. Leaves bind cell history, image/version, last group mutation and effect summary. Bound page fanout, decoded bytes, tree depth and cell count; use explicit execution-domain storage owners.", + "tests": [ + "manifest_updates_only_changed_path" + ] + }, + { + "action": "Separate logical-cell image bindings from group ownership in both materialized and extent paths. Sync changed images and manifest paths before one engine marker publishes applied position/root. Preserve old roots and all physical pins through failure.", + "tests": [ + "manifest_marker_crash_cuts" + ] + }, + { + "action": "Recover only complete durable roots; reject a missing/corrupt leaf or referenced image. Treat staged unmarked objects as non-serving orphans and reclaim only after pin/marker reconciliation.", + "tests": [ + "missing_cell_image_refuses_group" + ] + }, + { + "action": "Inject failures before/after image sync, manifest sync, marker commit and publication. Verify independent SQLite files, receipt floors and bounded copy-on-write work for one changed cell.", + "tests": [ + "manifest_pin_reclamation", + "both_image_representations_share_contract" + ] + } + ], + "test_cases": [ + { + "name": "manifest_updates_only_changed_path", + "oracle": "An update rewrites bounded manifest paths and the target cell, with unchanged cell image commitments preserved.", + "invariants": [ + "I04", + "I12" + ] + }, + { + "name": "manifest_marker_crash_cuts", + "oracle": "Every crash cut recovers exactly the old or new durable root without partial cells or callback reexecution.", + "invariants": [ + "I02", + "I04" + ] + }, + { + "name": "missing_cell_image_refuses_group", + "oracle": "Removing/corrupting any referenced cell or tree page prevents serving the group.", + "invariants": [ + "I04" + ] + }, + { + "name": "manifest_pin_reclamation", + "oracle": "Reader/snapshot/apply/transition pins prevent physical deletion until their owners release them.", + "invariants": [ + "I07", + "I12" + ] + }, + { + "name": "both_image_representations_share_contract", + "oracle": "Materialized and extent images retain identical history, publication and receipt safety.", + "invariants": [ + "I01", + "I03", + "I04" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I07", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "storage" + }, + { + "id": "CG-03", + "title": "Prepare paged range routing and enrollment admission", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01", + "CG-02" + ], + "anchors": [ + "catalog", + "birth", + "host", + "fleet" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar/node/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/directory.rs", + "crates/cellar/durability/raft/cell_groups/enrollment.rs", + "crates/cellar/tests/native_cell_groups_catalog.rs" + ], + "test_target": "native_cell_groups_catalog", + "steps": [ + { + "action": "Implement immutable paged range-tree records and one conditional namespace-root mutation journal. Verify complete range coverage, no overlap and exact epochs. Initial explicit ranges can be lazily materialized using existing birth and permanent disk-claim rules.", + "tests": [ + "range_directory_has_single_owner" + ] + }, + { + "action": "Define the pure enrollment decision and codec around the current range, cell creation identity and schema. Concurrent proposals must converge on one cell history when committed. This task validates admission decisions only; CG-04 connects those decisions to the actual multi-cell group owner and independent SQLite initialization.", + "tests": [ + "enrollment_decision_checks_identity" + ] + }, + { + "action": "Reconcile lost birth/root/enrollment replies by canonical operation identity. Keep unconfirmed outcomes typed; a blank claimed voter cannot bootstrap under a new path. Refuse enrollment under sealed or retired ranges.", + "tests": [ + "birth_unknown_and_claimed_blank_refuse" + ] + }, + { + "action": "Add bounded tree traversal/cache tests, restart races and stale-route redirects with original deadlines/hop budgets. Ordinary known-group commands must not require an S3 catalog request.", + "tests": [ + "bounded_catalog_discovery", + "sealed_range_rejects_enrollment" + ] + } + ], + "test_cases": [ + { + "name": "range_directory_has_single_owner", + "oracle": "Duplicate/overlapping/gapped trees and stale root CAS cannot publish ownership.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "enrollment_decision_checks_identity", + "oracle": "Pure admission decisions reject wrong range/schema, preserve creation identity on retries and select one cell history; no native publication qualification is claimed here.", + "invariants": [ + "I01", + "I03" + ] + }, + { + "name": "birth_unknown_and_claimed_blank_refuse", + "oracle": "Unknown creation reconciles its winner and missing permanently claimed disk state refuses.", + "invariants": [ + "I01", + "I08" + ] + }, + { + "name": "bounded_catalog_discovery", + "oracle": "Lookup reads a bounded tree path without namespace-wide listing; cache bounds hold under churn.", + "invariants": [ + "I12" + ] + }, + { + "name": "sealed_range_rejects_enrollment", + "oracle": "No delayed enrollment can create a cell beyond a committed seal boundary.", + "invariants": [ + "I09" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I08", + "I09", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "catalog" + }, + { + "id": "CG-04", + "title": "Execute many independent SQLite cells through one real Raft group", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-01", + "CG-02", + "CG-03" + ], + "anchors": [ + "host", + "sql", + "runtime", + "identity", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar-logic/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/owner.rs", + "crates/cellar/tests/native_cell_groups_execution.rs" + ], + "test_target": "native_cell_groups_execution", + "steps": [ + { + "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands.", + "tests": [ + "three_cells_one_rawnode_per_voter", + "concurrent_enrollment_one_cell" + ] + }, + { + "action": "Admit one unresolved proposal per cell with fair cell queues and group/node byte quotas. Allow independent cells to prepare concurrently, then publish committed entries through one contiguous group prefix with per-cell base/version checks.", + "tests": [ + "shared_group_fair_admission" + ] + }, + { + "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes.", + "tests": [ + "cell_receipt_not_group_progress" + ] + }, + { + "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture.", + "tests": [ + "strong_read_at_group_barrier", + "apply_prefix_never_skips_slow_cell" + ] + } + ], + "test_cases": [ + { + "name": "three_cells_one_rawnode_per_voter", + "oracle": "Three separate SQLite histories are served by exactly three RawNodes total across three voters; interleaved values agree on all replicas.", + "invariants": [ + "I01", + "I02", + "I04" + ] + }, + { + "name": "shared_group_fair_admission", + "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks.", + "invariants": [ + "I12" + ] + }, + { + "name": "cell_receipt_not_group_progress", + "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent.", + "invariants": [ + "I02", + "I03" + ] + }, + { + "name": "strong_read_at_group_barrier", + "oracle": "ReadIndex through B serves unchanged A coherently; stale leader/minority/incorrect incarnation cannot satisfy a strong read.", + "invariants": [ + "I01", + "I05" + ] + }, + { + "name": "apply_prefix_never_skips_slow_cell", + "oracle": "Out-of-order staging cannot publish beyond a missing earlier cell entry after crash or election.", + "invariants": [ + "I04" + ] + }, + { + "name": "concurrent_enrollment_one_cell", + "oracle": "Actual simultaneous enrollment through the real group owner creates one SQLite history per cell and one initialization effect on all voters, including a lost response and leader restart.", + "invariants": [ + "I01", + "I03", + "I04" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "execution" + }, + { + "id": "CG-05", + "title": "Snapshot and rebuild complete groups through membership changes", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04" + ], + "anchors": [ + "snapshot", + "membership", + "manifest", + "extent" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/checkpoint.rs", + "crates/cellar/tests/native_cell_groups_snapshot.rs" + ], + "test_target": "native_cell_groups_snapshot", + "steps": [ + { + "action": "Capture one group root at cut K with every enrolled cell, membership, range and transition state. Pin immutable dependencies while cells continue preparing later writes; stream bounded chunks rather than aggregate all images in memory.", + "tests": [ + "snapshot_contains_every_cell_at_cut" + ] + }, + { + "action": "Verify and sync every receiver dependency before atomic group install. Incomplete, mixed-cut or wrong-history snapshots fail closed. Match materialized and extent representations, including an enrolled empty cell.", + "tests": [ + "partial_snapshot_never_serves" + ] + }, + { + "action": "Adapt learner admission, joint configuration, leadership handoff and disk replacement to whole groups. Preserve permanent disk lineage and learner catch-up proofs; snapshots do not authorize new voters.", + "tests": [ + "whole_group_learner_and_joint_change" + ] + }, + { + "action": "Run real peer catch-up, process restart and lost-disk scenarios while a different cell writes. Reclaim transferred scratch only after physical owner release.", + "tests": [ + "snapshot_empty_cell_and_extent_restart", + "snapshot_memory_and_pin_bounds" + ] + } + ], + "test_cases": [ + { + "name": "snapshot_contains_every_cell_at_cut", + "oracle": "A concurrent writer cannot produce a snapshot mixing cells from different group cuts.", + "invariants": [ + "I04", + "I07" + ] + }, + { + "name": "partial_snapshot_never_serves", + "oracle": "Interrupted/corrupt/foreign dependency install leaves the prior root or refuses admission.", + "invariants": [ + "I01", + "I04" + ] + }, + { + "name": "whole_group_learner_and_joint_change", + "oracle": "Fresh learner installs all cells before promotion; one failed voter remains tolerable and minority writes refuse.", + "invariants": [ + "I02", + "I08" + ] + }, + { + "name": "snapshot_empty_cell_and_extent_restart", + "oracle": "Empty enrolled cells and extent/materialized populated cells survive checkpoint and retained restart.", + "invariants": [ + "I03", + "I04" + ] + }, + { + "name": "snapshot_memory_and_pin_bounds", + "oracle": "Large group transfer stays within declared scratch/buffer slots and preserves live reader pins.", + "invariants": [ + "I07", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I07", + "I08", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "snapshot" + }, + { + "id": "CG-06", + "title": "Archive and reclaim the complete group recovery history", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04", + "CG-05" + ], + "anchors": [ + "archive", + "snapshot", + "disaster", + "manifest" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/archive.rs", + "crates/cellar/tests/native_cell_groups_archive.rs" + ], + "test_target": "native_cell_groups_archive", + "steps": [ + { + "action": "Replace single-cell archive envelopes with complete group checkpoint and cell-addressed tail closures. Verify membership, administrative entries and every cell base chain before publishing group coverage; reuse unchanged immutable payloads.", + "tests": [ + "group_archive_requires_all_dependencies" + ] + }, + { + "action": "Keep Quorum acknowledgement independent from asynchronous S3 publication; S3-mode settlement requires complete confirmed group coverage. Bound unarchived bytes and expose group lag without crediting uploaded-but-unverified objects.", + "tests": [ + "quorum_continues_bounded_s3_outage" + ] + }, + { + "action": "Tie local truncation and physical shared-engine purge to covering snapshots, archive policy and all live group/reader/transition pins. Preserve lagging-peer snapshot availability and prevent deleting another group range.", + "tests": [ + "group_reclaim_restart_keeps_history" + ] + }, + { + "action": "Implement unmanaged fresh-incarnation whole-group S3 seeding through the existing fencing authority. Preserve selected-cut receipts, reject old tokens and prove no callback replay; later CG-13 adds managed and in-transition recovery.", + "tests": [ + "fenced_group_seed_new_incarnations", + "archive_tails_cover_enrollment_and_controls" + ] + } + ], + "test_cases": [ + { + "name": "group_archive_requires_all_dependencies", + "oracle": "A delayed/corrupt image for B blocks coverage and S3 acknowledgement of a later A entry.", + "invariants": [ + "I06" + ] + }, + { + "name": "quorum_continues_bounded_s3_outage", + "oracle": "Quorum can progress without S3 until explicit retention admission limits; S3 outputs cannot.", + "invariants": [ + "I02", + "I06", + "I12" + ] + }, + { + "name": "group_reclaim_restart_keeps_history", + "oracle": "Actual log files can be reclaimed without losing any acknowledged cell effect/receipt on retained restart.", + "invariants": [ + "I03", + "I07" + ] + }, + { + "name": "fenced_group_seed_new_incarnations", + "oracle": "Complete archived group restores under fresh identities with exact results, old-token rejection and fresh-quorum admission.", + "invariants": [ + "I06", + "I11" + ] + }, + { + "name": "archive_tails_cover_enrollment_and_controls", + "oracle": "Enrollment, no-ops and membership/administrative cuts replay without holes or synthetic cells.", + "invariants": [ + "I04", + "I06" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I06", + "I07", + "I11", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "archive" + }, + { + "id": "CG-07", + "title": "Cut public SDK routing and managed fleet over to shared groups", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-03", + "CG-04", + "CG-05", + "CG-06", + "CG-08" + ], + "anchors": [ + "public", + "host", + "fleet", + "placement", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/application/consensus/", + "crates/cellar/node/", + "crates/cellar/cli/", + "crates/cellar-axum/", + "crates/cellar-admin/", + "examples/", + "crates/cellar/sdk.rs" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_public.rs" + ], + "test_target": "native_cell_groups_public", + "steps": [ + { + "action": "Switch the existing public host/startup/registration APIs to group placement plus cell enrollment. Preserve cell-addressed application calls and explicit query consistency. Remove production creation of one RawNode for each ordinary cell.", + "tests": [ + "sdk_many_cells_share_group" + ] + }, + { + "action": "Adapt managed namespace import, capacity accounting, repair, drain, maintenance and topology inventory to group replicas plus separate cell/image occupancy. Reuse existing CAS journal, permissions and disruption debt; do not bypass managed admission.", + "tests": [ + "managed_group_lifecycle" + ] + }, + { + "action": "Update native/Axum dispatch, CLI configuration, admin/operator views and examples. Validate tenant/cell authorization independently from group route; give stale hints bounded redirects and typed ambiguous mutation results.", + "tests": [ + "ingress_cannot_cross_cell_authority" + ] + }, + { + "action": "Audit all GroupScope.cell consumers, codecs, receipt/token paths and example configs. Delete superseded single-image/one-cell host assumptions after all consumers use shared groups; occupancy-one must use the exact same code.", + "tests": [ + "occupancy_one_uses_shared_path", + "feature_off_fleet_contract_unchanged" + ] + } + ], + "test_cases": [ + { + "name": "sdk_many_cells_share_group", + "oracle": "Public startup and cell handles serve independent SQLite cells under one configured group on all replicas.", + "invariants": [ + "I01", + "I02", + "I03" + ] + }, + { + "name": "managed_group_lifecycle", + "oracle": "Repair, drain, stop/return/resume and capacity charges move complete groups while preserving cell count and physical debt.", + "invariants": [ + "I08", + "I12" + ] + }, + { + "name": "ingress_cannot_cross_cell_authority", + "oracle": "Native/Axum/admin routes reject forged tenant/cell/group bindings; retries preserve caller command identity.", + "invariants": [ + "I01", + "I03" + ] + }, + { + "name": "occupancy_one_uses_shared_path", + "oracle": "Single-cell and multi-cell configurations exercise one host/manifest/snapshot/archive implementation with no legacy fallback.", + "invariants": [ + "I04", + "I06" + ] + }, + { + "name": "feature_off_fleet_contract_unchanged", + "oracle": "Existing non-Raft feature modes retain their ownership and durability behavior through the full regression suite.", + "invariants": [ + "I14" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I06", + "I08", + "I12", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "public" + }, + { + "id": "CG-08", + "title": "Preserve per-cell effects and bounded idle execution caches", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-04", + "CG-05", + "CG-06" + ], + "anchors": [ + "effects", + "sql", + "host", + "manifest" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-ltx/src/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/effect_index.rs", + "crates/cellar/tests/native_cell_groups_effects.rs" + ], + "test_target": "native_cell_groups_effects", + "steps": [ + { + "action": "Move alarm/outbox/inbox identity to CellHistory and retain existing receiver dedupe/client floors. Record effect summaries in the cell manifest and maintain a bounded next-due index owned by node services.", + "tests": [ + "effect_identity_stable_across_group_binding" + ] + }, + { + "action": "Close idle execution/SQLite caches only after unresolved proposals, dispatches and unsafe pins settle. Reopen against the exact committed cell root without resetting ownership or acknowledging based on cached state.", + "tests": [ + "idle_cache_reopens_exact_cell_base" + ] + }, + { + "action": "Make fair effect discovery independent of opening every cell database. Serialize control decisions with cell commands and preserve unknown dispatched outcomes through leadership change and future transfer seals.", + "tests": [ + "due_effect_without_client_request" + ] + }, + { + "action": "Integrate all cache/effect workers with shutdown, resource observation and physical permit lifetimes. Cover due alarms with no client traffic and mixed hot/cold group cells.", + "tests": [ + "pending_command_or_dispatch_pins_cache", + "cache_and_effect_shutdown_joins" + ] + } + ], + "test_cases": [ + { + "name": "effect_identity_stable_across_group_binding", + "oracle": "Changing placement metadata never allocates a new delivery identity or loses receiver duplicate suppression.", + "invariants": [ + "I03", + "I13" + ] + }, + { + "name": "idle_cache_reopens_exact_cell_base", + "oracle": "Eviction saves disposable resources and the next command starts from the verified committed image without an election requirement.", + "invariants": [ + "I04", + "I12" + ] + }, + { + "name": "due_effect_without_client_request", + "oracle": "Durable due summaries schedule alarms/outbox work without scanning or hydrating every cell.", + "invariants": [ + "I12", + "I13" + ] + }, + { + "name": "pending_command_or_dispatch_pins_cache", + "oracle": "Uncertain proposal/delivery cannot be evicted and forgotten after cancellation or leader loss.", + "invariants": [ + "I02", + "I03", + "I13" + ] + }, + { + "name": "cache_and_effect_shutdown_joins", + "oracle": "All retained cache/pin/timer workers finish or preserve durable debt before joined stop reports success.", + "invariants": [ + "I12" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I12", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "effects" + }, + { + "id": "CG-09", + "title": "Implement irreversible source sealing and destination activation", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-08" + ], + "anchors": [ + "fleet", + "membership", + "archive", + "version" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-logic/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/transition.rs", + "crates/cellar/durability/raft/cell_groups/transition_tests.rs", + "crates/cellar/tests/native_cell_groups_transitions.rs" + ], + "test_target": "native_cell_groups_transitions", + "steps": [ + { + "action": "Implement Planned/Prepared/SealAuthorized/Sealed/Installed/Published/Active/Retired journal states from protocols.md. Seal authorization and pre-seal cancellation CAS the same state; no source Seal request is allowed before authorization wins.", + "tests": [ + "seal_authorization_cancel_race" + ] + }, + { + "action": "Commit parent seals after accepted mutations, block new enrollment/writes/strong reads, and pin the exact cell/effect/receipt cuts. After SealAuthorized only roll forward, including unknown replies and partially sealed merges.", + "tests": [ + "source_seal_blocks_delayed_work" + ] + }, + { + "action": "Seed fresh non-serving child group histories with preserved CellHistory/versions/receipts/effect IDs. Require child quorum install receipts and an atomic range-tree publication certificate before committed child activation. Publish and verify a complete remote transfer closure before changing active ranges; pin its digest in the publication certificate so all-voter loss immediately after publication remains recoverable.", + "tests": [ + "child_requires_install_and_publication", + "publication_requires_remote_transfer_closure" + ] + }, + { + "action": "Persist stale-authority tombstones and transition resources across controller/leader/node restart. Release source data only after all child baseline archives and physical pins are safe; no timer may reopen the parent.", + "tests": [ + "transition_restart_retains_single_writer", + "transfer_preserves_tokens_receipts_effects", + "retirement_keeps_required_archive_pins" + ] + } + ], + "test_cases": [ + { + "name": "seal_authorization_cancel_race", + "oracle": "Exactly one root decision wins; cancelled intents cannot seal and authorized intents cannot roll back after unknown responses.", + "invariants": [ + "I09" + ] + }, + { + "name": "source_seal_blocks_delayed_work", + "oracle": "Accepted-before-seal work appears in the cut; delayed/new mutation/enrollment/strong read cannot pass the seal.", + "invariants": [ + "I01", + "I05", + "I09" + ] + }, + { + "name": "child_requires_install_and_publication", + "oracle": "Copied bytes or a catalog hint alone cannot activate a child; quorum seed plus exact publication and activation are required.", + "invariants": [ + "I02", + "I04", + "I09" + ] + }, + { + "name": "transition_restart_retains_single_writer", + "oracle": "Crash at every phase and lost CAS replies preserve a single writable owner and resume the same transition.", + "invariants": [ + "I09", + "I12" + ] + }, + { + "name": "transfer_preserves_tokens_receipts_effects", + "oracle": "Old cell-history versions and duplicate commands remain valid at the child while effect IDs and receiver dedupe survive.", + "invariants": [ + "I03", + "I05", + "I13" + ] + }, + { + "name": "retirement_keeps_required_archive_pins", + "oracle": "Source cleanup cannot orphan any child baseline, archived acknowledgement or unresolved physical owner.", + "invariants": [ + "I06", + "I07" + ] + }, + { + "name": "publication_requires_remote_transfer_closure", + "oracle": "Missing remote transfer dependencies block publication. Erase all local copies immediately after publication in the native fixture; independent archive replay of the pinned closure retains every prior S3-acknowledged cell effect. CG-13/15 qualify fenced live recovery through the public disaster API.", + "invariants": [ + "I06", + "I09", + "I11" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I09", + "I11", + "I12", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "transitions" + }, + { + "id": "CG-10", + "title": "Split hot groups and merge compatible adjacent ranges", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-09" + ], + "anchors": [ + "placement", + "fleet", + "host", + "catalog" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar-logic/", + "crates/cellar-admin/", + "crates/cellar/cli/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/partition.rs", + "crates/cellar/tests/native_cell_groups_split_merge.rs" + ], + "test_target": "native_cell_groups_split_merge", + "steps": [ + { + "action": "Build split decisions over whole ordered CellKeys with exact equal-union range proofs, including empty edges and hash collisions. Allocate fresh child histories; parent group retirement uses CG-09 instead of in-place identity reuse.", + "tests": [ + "split_equal_union_whole_cells" + ] + }, + { + "action": "Build merge decisions for adjacent compatible namespace/schema/durability/failure-domain ranges. Seal all parents, install one fresh child and publish one atomic root replacement; partial parent failure cannot admit the child.", + "tests": [ + "merge_requires_adjacent_compatible_ranges" + ] + }, + { + "action": "Pre-copy immutable state and apply final sealed deltas with bounded parallelism and resource reservations. Define explicit operator split/merge/status/resume APIs; waiting remains read-only and ambiguous requests retain their IDs.", + "tests": [ + "partial_merge_failure_no_double_writer" + ] + }, + { + "action": "Exercise repeated split/merge under foreground writes, stale client routes, controller loss and skew. A dedicated hot cell is a range with one cell using the common runtime.", + "tests": [ + "split_merge_under_writes", + "transition_precopy_resource_bounds" + ] + } + ], + "test_cases": [ + { + "name": "split_equal_union_whole_cells", + "oracle": "Children partition the parent exactly, never split a SQLite database, and tolerate hash ties without aliasing.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "merge_requires_adjacent_compatible_ranges", + "oracle": "Non-adjacent or policy/schema-incompatible sources refuse before sealing or allocating unsafe authority.", + "invariants": [ + "I01", + "I09" + ] + }, + { + "name": "partial_merge_failure_no_double_writer", + "oracle": "A sealed source stays fenced while another lacks quorum; no child admits prematurely.", + "invariants": [ + "I02", + "I09" + ] + }, + { + "name": "split_merge_under_writes", + "oracle": "Every acknowledged effect and receipt reconciles exactly through repeated transitions and cached stale routes.", + "invariants": [ + "I02", + "I03", + "I05", + "I09" + ] + }, + { + "name": "transition_precopy_resource_bounds", + "oracle": "Bounded precopy/delta streaming cannot starve live groups or release transfer debt before completion.", + "invariants": [ + "I07", + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I05", + "I07", + "I09", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "split_merge" + }, + { + "id": "CG-11", + "title": "Schedule group placement and adaptive partition size", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-10" + ], + "anchors": [ + "placement", + "fleet", + "membership", + "public" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/durability/raft/", + "crates/cellar-admin/", + "crates/cellar/cli/" + ], + "creates": [ + "crates/cellar-logic/cell_group_placement.rs", + "crates/cellar-logic/cell_group_placement_tests.rs", + "crates/cellar/tests/native_cell_groups_scheduler.rs" + ], + "test_target": "native_cell_groups_scheduler", + "steps": [ + { + "action": "Define dependency-free scheduler observations/actions using group bytes, cell count, write/apply rate, queue age, archive lag and failure-domain load. Keep observations advisory; actual owners revalidate membership, route and resource permits.", + "tests": [ + "scheduler_hysteresis_and_hot_isolation" + ] + }, + { + "action": "Add bounded split/merge hysteresis and cooldown with deterministic reason codes. Isolate an oversized/hot cell without claiming intra-cell parallelism; refuse unsafe merge and prevent repeated split/merge oscillation.", + "tests": [ + "scheduler_cannot_grant_authority" + ] + }, + { + "action": "Integrate scale-out/in, leader balance, learner movement, maintenance and repair with whole-group inventory. Serialize membership with transition/park/disaster ownership; reserve recovery capacity and keep exact physical charges.", + "tests": [ + "whole_group_scale_out_in" + ] + }, + { + "action": "Expose bounded metrics/status and qualify node loss under mixed load. Avoid broadcasting all cell/group metadata to every node and record topology-journal contention.", + "tests": [ + "repair_preserves_failure_domains_and_reserve", + "control_plane_fanout_bounded" + ] + } + ], + "test_cases": [ + { + "name": "scheduler_hysteresis_and_hot_isolation", + "oracle": "Stable decisions split hot/large ranges, merge eligible small neighbors and do not oscillate at thresholds.", + "invariants": [ + "I12" + ] + }, + { + "name": "scheduler_cannot_grant_authority", + "oracle": "Stale observed plans cannot bypass current route, membership, permit or quorum checks.", + "invariants": [ + "I01", + "I08", + "I09" + ] + }, + { + "name": "whole_group_scale_out_in", + "oracle": "Adding/removing nodes transfers group replicas without remapping or reinitializing individual cells.", + "invariants": [ + "I03", + "I08" + ] + }, + { + "name": "repair_preserves_failure_domains_and_reserve", + "oracle": "One-domain loss repairs groups within bounded slots; a second loss cannot fabricate quorum.", + "invariants": [ + "I02", + "I08", + "I12" + ] + }, + { + "name": "control_plane_fanout_bounded", + "oracle": "Paged inventory and targeted updates remain bounded under many enrolled cells and groups.", + "invariants": [ + "I12" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I08", + "I09", + "I12" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "scheduler" + }, + { + "id": "CG-12", + "title": "Coordinate group hibernation and authority-checked wakeup", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-08", + "CG-10", + "CG-11" + ], + "anchors": [ + "ticks", + "runtime", + "effects", + "membership" + ], + "scope": [ + "crates/cellar-logic/", + "crates/cellar/durability/raft/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/hibernate.rs", + "crates/cellar-logic/cell_group_hibernation.rs", + "crates/cellar/tests/native_cell_groups_hibernate.rs" + ], + "test_target": "native_cell_groups_hibernate", + "steps": [ + { + "action": "Implement PreparePark and generation-bound local durable park/closed records. Require all configured voters to acknowledge the committed applied cut, with explicit learner state and no unresolved proposal/read/topology work.", + "tests": [ + "park_requires_all_voters_and_no_pending_work" + ] + }, + { + "action": "Pause routine group ticks only after the exact park certificate is durable. Keep node transport/control/effect wake owners active and bounded; partial park delivery must be safe and stale certificates must never re-park a woken group.", + "tests": [ + "partial_park_and_stale_certificate" + ] + }, + { + "action": "Wake on API/peer/effect/topology activity or restart, durably close the park generation, resume retained Raft and obtain fresh current-term commit/read authority. Never reset votes, assume retained leadership or replace a missing quorum with S3.", + "tests": [ + "wake_requires_fresh_quorum_barrier" + ] + }, + { + "action": "Integrate alarm due indexes, membership serialization, cancellation and shutdown. Measure healthy cached wake, image-cache restore and leader-loss wake separately with enough samples.", + "tests": [ + "alarm_and_peer_wake_no_client", + "parked_restart_and_leader_loss" + ] + } + ], + "test_cases": [ + { + "name": "park_requires_all_voters_and_no_pending_work", + "oracle": "A missing voter or unresolved command/effect/topology operation prevents parking but not ordinary safe quorum operation.", + "invariants": [ + "I02", + "I10", + "I13" + ] + }, + { + "name": "partial_park_and_stale_certificate", + "oracle": "Lost/late messages, generation replay and new commands cannot cause unsafe re-parking or lost work.", + "invariants": [ + "I01", + "I10" + ] + }, + { + "name": "wake_requires_fresh_quorum_barrier", + "oracle": "Retained leader and minority wake cannot serve strong reads/writes without current quorum proof.", + "invariants": [ + "I02", + "I05", + "I10" + ] + }, + { + "name": "alarm_and_peer_wake_no_client", + "oracle": "Due effects and peer traffic wake parked groups without client traffic or per-cell threads.", + "invariants": [ + "I10", + "I12", + "I13" + ] + }, + { + "name": "parked_restart_and_leader_loss", + "oracle": "All-voter retained restart and parked leader failure preserve term/vote/history and eventually serve through ordinary elections.", + "invariants": [ + "I04", + "I08", + "I10" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I04", + "I05", + "I08", + "I10", + "I12", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "hibernate" + }, + { + "id": "CG-13", + "title": "Recover managed groups and incomplete transitions from S3", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-06", + "CG-09", + "CG-10", + "CG-11", + "CG-14" + ], + "anchors": [ + "disaster", + "fleet", + "recovery", + "archive" + ], + "scope": [ + "crates/cellar/durability/raft/", + "tools/cellar-recovery-tests/", + "crates/cellar/tests/support/raft_process_recovery/" + ], + "creates": [ + "crates/cellar/durability/raft/cell_groups/disaster.rs", + "crates/cellar/tests/native_cell_groups_disaster.rs" + ], + "test_target": "native_cell_groups_disaster", + "steps": [ + { + "action": "Extend recovery admission to the exact managed topology journal and transition component. Resolve publication outcome and enumerate all parent/child/replacement voting lineages that could still activate; preserve fleet debt/import exclusion.", + "tests": [ + "managed_disaster_blocks_old_lifecycle" + ] + }, + { + "action": "Verify complete selected recovery closures before fencing. For unpublished operations recover parent domain; for published operations require child sets or transfer closure plus child tails. A parent checkpoint alone cannot cover child S3 acknowledgements.", + "tests": [ + "transition_disaster_selects_complete_closure" + ] + }, + { + "action": "Use the explicit fencing-authority API to permanently revoke old launch authority and confirm isolation before publishing any new range. Create fresh cell/group/node/member/disk/process identities, seed receipts without callbacks, and require a fresh quorum.", + "tests": [ + "fence_all_possible_old_writers" + ] + }, + { + "action": "Test exact-request retry after unknown fencing/catalog results, old-host restart attempts, outbox ambiguity and retained-disk restart. Refuse partial fences, missing dependencies, unmanaged bypasses and unsupported rollback.", + "tests": [ + "all_disks_deleted_restore_all_cells", + "disaster_retry_and_old_token_refusal", + "disaster_outbox_ambiguity_refuses" + ] + } + ], + "test_cases": [ + { + "name": "managed_disaster_blocks_old_lifecycle", + "oracle": "Import, maintenance and failure recovery cannot concurrently activate old or replacement group histories.", + "invariants": [ + "I08", + "I09", + "I11" + ] + }, + { + "name": "transition_disaster_selects_complete_closure", + "oracle": "Published children require their acknowledged tails; unpublished parents recover an exact complete domain without duplication or gaps.", + "invariants": [ + "I06", + "I09", + "I11" + ] + }, + { + "name": "fence_all_possible_old_writers", + "oracle": "Partial inventory, incomplete launch revocation or a restartable old voter prevents new incarnation publication.", + "invariants": [ + "I11" + ] + }, + { + "name": "all_disks_deleted_restore_all_cells", + "oracle": "Real RustFS restore after erasing every owned voter disk preserves all selected-cut values/receipts and admits fresh writes only after quorum.", + "invariants": [ + "I02", + "I03", + "I06", + "I11" + ] + }, + { + "name": "disaster_retry_and_old_token_refusal", + "oracle": "Lost replies resume the same cut/identities; old tokens refuse and duplicate results return without callback replay.", + "invariants": [ + "I01", + "I03", + "I11" + ] + }, + { + "name": "disaster_outbox_ambiguity_refuses", + "oracle": "Old-origin uncertain external effects cannot be silently reissued under a fresh identity.", + "invariants": [ + "I11", + "I13" + ] + } + ], + "invariants": [ + "I01", + "I02", + "I03", + "I06", + "I08", + "I09", + "I11", + "I13" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "disaster" + }, + { + "id": "CG-14", + "title": "Build reproducible shared-group workload and RustFS harnesses", + "status": "TODO", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [ + "CG-07", + "CG-08" + ], + "anchors": [ + "harness", + "recovery", + "fixture", + "ci" + ], + "scope": [ + "tools/perf/", + "tools/cellar-recovery-tests/", + "crates/cellar/tests/support/" + ], + "creates": [ + "tools/perf/cell_groups.py", + "tools/perf/test_cell_groups.py", + "crates/cellar/tests/native_cell_groups_harness.rs" + ], + "test_target": "native_cell_groups_harness", + "steps": [ + { + "action": "Extend existing native workload and process-recovery fixtures with logical-cell/group population, mixed operations and per-cell independent ledgers. Observe actual RawNode/group/cell counts; do not substitute counter rows in one database.", + "tests": [ + "harness_counts_real_cells_and_groups" + ] + }, + { + "action": "Implement the planned harness CLI in qualification.md with explicit private fixture config, owned run directories, source/binary/image/profile hashes and resource preflight. Never change global Docker context or touch unowned resources.", + "tests": [ + "harness_owned_resources_only" + ] + }, + { + "action": "Separate build/setup/warmup/measurement and run equal-cell cohorts serially. Record all offered/accepted/refused/ambiguous operations, latencies, control bytes, CPU/RSS/FD, image/log writes, S3 requests/lag and cleanup/join outcomes.", + "tests": [ + "harness_ledger_preserves_all_outcomes" + ] + }, + { + "action": "Implement strict result validation for profiles.json and replayable fault schedules. Retain failed attempts; refuse unsupported profiles, zero selected tests, missing artifacts, source drift and fake multi-node topology claims.", + "tests": [ + "harness_rejects_missing_or_drifted_evidence", + "harness_profile_gates_measured" + ] + } + ], + "test_cases": [ + { + "name": "harness_counts_real_cells_and_groups", + "oracle": "Metrics distinguish independent SQLite cells, RawNodes, groups and replicas and match physical inventory.", + "invariants": [ + "I14" + ] + }, + { + "name": "harness_owned_resources_only", + "oracle": "Destructive actions refuse wrong context, label, directory, container identity or run UUID.", + "invariants": [ + "I11", + "I12", + "I14" + ] + }, + { + "name": "harness_ledger_preserves_all_outcomes", + "oracle": "Refusals/timeouts/cancellations remain in offered-load and correctness accounting; no silent retry improves benchmark results.", + "invariants": [ + "I02", + "I03", + "I14" + ] + }, + { + "name": "harness_rejects_missing_or_drifted_evidence", + "oracle": "Changed sources/binaries/profiles, zero-test runs and absent logs cannot produce complete reports.", + "invariants": [ + "I14" + ] + }, + { + "name": "harness_profile_gates_measured", + "oracle": "Numeric metrics and source-bound artifacts, not a passed boolean alone, determine benchmark acceptance.", + "invariants": [ + "I12", + "I14" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I11", + "I12", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "harness" + }, + { + "id": "CG-15", + "title": "Qualify crashes, split merge, wake and zero-quorum recovery", + "status": "TODO", + "priority": "P1", + "effort": "L", + "risk": "HIGH", + "depends_on": [ + "CG-10", + "CG-11", + "CG-12", + "CG-13", + "CG-14" + ], + "anchors": [ + "fixture", + "recovery", + "harness", + "runtime" + ], + "scope": [ + "crates/cellar/durability/raft/", + "crates/cellar/tests/", + "tools/perf/", + "tools/cellar-recovery-tests/" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_faults.rs" + ], + "test_target": "native_cell_groups_faults", + "steps": [ + { + "action": "Run the full native history oracle over interleaved cells, both image modes and all new durable cuts. Include deterministic exhaustive transition/park event schedules and process SIGKILL at persistence boundaries; finite tests complement rather than prove the protocol.", + "tests": [ + "history_oracle_rejects_cross_cell_loss" + ] + }, + { + "action": "Run real RustFS campaigns with physical log reclamation, S3 outage, slow apply/minority, leader/controller restart and every split/merge cut. Verify per-cell values, versions, receipts, effect IDs and no two writable owners.", + "tests": [ + "all_new_durable_crash_cuts" + ] + }, + { + "action": "Delete all owned voter disks in standalone and in-transition scenarios after fencing; verify new incarnations, old-host restart rejection, every selected-cut cell and retained restart. Record quorum-only suffix loss separately from S3-protected effects.", + "tests": [ + "rustfs_reclaim_and_transition_recovery" + ] + }, + { + "action": "Run candidate density, mixed/skew, transition, hibernation, independent-domain and 24-hour profiles. Compare identical logical-cell populations and retain misses. Do not declare a physical tier from containers on one host.", + "tests": [ + "rustfs_zero_quorum_transition_matrix", + "load_and_wake_profiles_complete", + "soak_and_domain_loss_history" + ] + } + ], + "test_cases": [ + { + "name": "history_oracle_rejects_cross_cell_loss", + "oracle": "Mutated histories with a lost acknowledged effect, borrowed version or duplicate receipt execution are rejected by the oracle.", + "invariants": [ + "I02", + "I03", + "I05", + "I14" + ] + }, + { + "name": "all_new_durable_crash_cuts", + "oracle": "Native process kills cover manifest, enrollment, seed, publication, seal, activation, park and wake boundaries without partial admission.", + "invariants": [ + "I04", + "I09", + "I10" + ] + }, + { + "name": "rustfs_reclaim_and_transition_recovery", + "oracle": "Real object-store closure plus actual log-file reclamation survives group restart and transition interruption.", + "invariants": [ + "I06", + "I07", + "I09" + ] + }, + { + "name": "rustfs_zero_quorum_transition_matrix", + "oracle": "Every specified parent/child disaster case fences old histories and restores complete selected state under fresh quorum.", + "invariants": [ + "I06", + "I09", + "I11" + ] + }, + { + "name": "load_and_wake_profiles_complete", + "oracle": "All required profile metrics, errors, samples and topology attestations meet the frozen candidate gates.", + "invariants": [ + "I10", + "I12", + "I14" + ] + }, + { + "name": "soak_and_domain_loss_history", + "oracle": "24-hour and independent-domain runs preserve history, bounded resources and joined lifecycle under foreground load.", + "invariants": [ + "I08", + "I12", + "I13", + "I14" + ] + } + ], + "invariants": [ + "I02", + "I03", + "I04", + "I05", + "I06", + "I07", + "I08", + "I09", + "I10", + "I11", + "I12", + "I13", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "faults" + }, + { + "id": "CG-16", + "title": "Publish the unified group runtime and qualified release envelope", + "status": "TODO", + "priority": "P1", + "effort": "M", + "risk": "HIGH", + "depends_on": [ + "CG-15" + ], + "anchors": [ + "ci", + "host", + "catalog", + "version", + "fixture" + ], + "scope": [ + "crates/", + "examples/", + "docs/", + "tools/", + ".github/workflows/" + ], + "creates": [ + "crates/cellar/tests/native_cell_groups_release.rs" + ], + "test_target": "native_cell_groups_release", + "steps": [ + { + "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code.", + "tests": [ + "no_legacy_per_cell_consensus_path" + ] + }, + { + "action": "Update public docs, examples, CLI/admin error messages and operator runbooks for group placement, split/merge, park/wake and fenced S3 recovery. Document fresh-root cutover and rollback limits without deleting existing state.", + "tests": [ + "public_group_operations_end_to_end" + ] + }, + { + "action": "Run the complete workspace/feature-off/native/Axum regression matrix and all new task tests at the same immutable release source. Repeat deployment gates at that source; evidence from different revisions cannot be added into one release pass.", + "tests": [ + "release_all_modes_and_shutdown" + ] + }, + { + "action": "Publish source/artifact/profile hashes, exact supported topology/load envelope and every limitation. plan.py release must refuse until all tasks, invariant cases and mandatory profile reports pass with retained evidence.", + "tests": [ + "release_evidence_single_source" + ] + } + ], + "test_cases": [ + { + "name": "no_legacy_per_cell_consensus_path", + "oracle": "Public configuration cannot select a legacy one-cell host; physical group counts follow grouping, including occupancy one.", + "invariants": [ + "I01", + "I14" + ] + }, + { + "name": "public_group_operations_end_to_end", + "oracle": "SDK/native/Axum/admin/CLI share authorization and identity for enrollment, strong reads, split/merge, park/wake and status.", + "invariants": [ + "I01", + "I03", + "I05", + "I09", + "I10" + ] + }, + { + "name": "release_all_modes_and_shutdown", + "oracle": "Materialized/extents, managed/unmanaged, feature-off applications and owned shutdown pass the complete repository regression suite.", + "invariants": [ + "I04", + "I08", + "I12", + "I13", + "I14" + ] + }, + { + "name": "release_evidence_single_source", + "oracle": "All capabilities and numerical profiles are verified at the published source with no missing, stale or synthetic evidence.", + "invariants": [ + "I14" + ] + } + ], + "invariants": [ + "I01", + "I03", + "I04", + "I05", + "I08", + "I09", + "I10", + "I12", + "I13", + "I14" + ], + "evidence": [], + "stop_conditions": [ + "A current source excerpt no longer matches: review and record the successor baseline before edits.", + "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", + "The implementation requires an unrecorded wire/storage compatibility path or bypass of quorum/fencing." + ], + "out_of_scope": [ + "TiKV KV/MVCC APIs and cross-cell transactions", + "Online migration or deletion of existing user disks/archives", + "Feature-off Fleet/bucket durability semantics" + ], + "capability": "release" + } + ] +} diff --git a/plans/multi-raft/cell-groups/test_plan.py b/plans/multi-raft/cell-groups/test_plan.py new file mode 100644 index 000000000..02ec9bbfe --- /dev/null +++ b/plans/multi-raft/cell-groups/test_plan.py @@ -0,0 +1,377 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Negative tests for the delivery gate, not tests of the proposed Raft runtime.""" + +import copy +import hashlib +import importlib.util +import json +from pathlib import Path +import subprocess +import tempfile +import unittest +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location("cell_group_plan", Path(__file__).with_name("plan.py")) +plan = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(plan) + + +class PlanTests(unittest.TestCase): + def setUp(self): + self.manifest = plan.read_json(plan.PACKAGE / "tasks.json") + self.actual_manifest = copy.deepcopy(self.manifest) + for task in self.manifest["tasks"]: + task.update(status="TODO", evidence=[]) + self.baseline = plan.read_json(plan.PACKAGE / "baseline.json") + self.profiles = plan.read_json(plan.PACKAGE / "profiles.json") + self.temp = tempfile.TemporaryDirectory(prefix="cell-group-plan-test-") + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + self.revision = "a" * 40 + self.executable = "b" * 64 + self.profile_digest = "c" * 64 + + def contract(self): + return plan.validate_contract(self.manifest, self.baseline, self.profiles, plan.ROOT) + + def put(self, name, value): + path = self.root / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(value if isinstance(value, str) else json.dumps(value)) + return {"path": name, "sha256": plan.digest(path)} + + def evidence(self): + task = self.manifest["tasks"][0] + paths = {self.baseline["anchors"][a]["path"] for a in task["anchors"]} + paths.add(f'crates/cellar/tests/{task["test_target"]}.rs') + record = {"format": 1, "task_id": task["id"], "source_revision": self.revision, + "executable_sha256": self.executable, "environment": "Synthetic validator fixture only", + "limitations": "Not runtime evidence", "source_inputs": [], "commands": [], "profiles": []} + for path in sorted(paths): + record["source_inputs"].append({"path": path, "sha256": hashlib.sha256(path.encode()).hexdigest()}) + for index, command in enumerate(plan.commands_for(task, self.manifest)): + output = "Command exited successfully.\n" + if command[-1] == "--list": + output = "\n".join(c["name"] + ": test" for c in task["test_cases"]) + "\n" + elif "--exact" in command: + name = command[command.index("--exact") - 1] + output = f"running 1 test\ntest {name} ... ok\ntest result: ok. 1 passed; 0 failed; 0 ignored; 3 filtered out; finished in 0.01s\n" + item = self.put(f"commands/{index}.log", output) + record["commands"].append({"argv": command, "exit_code": 0, "output": item["path"], "sha256": item["sha256"]}) + return record + + def verify_evidence(self, record, changed=b""): + with patch.object(plan, "git_bytes", side_effect=lambda root, rev, path: path.encode()), \ + patch.object(plan.subprocess, "check_output", return_value=changed): + return plan.check_evidence(self.manifest["tasks"][0], record, self.manifest, self.baseline, + self.profiles, self.profile_digest, self.root) + + def profile_fixture(self, pid="mixed_quorum"): + profile = next(p for p in self.profiles["profiles"] if p["id"] == pid) + metrics = {name: next(iter(bounds.values())) for name, bounds in profile["checks"].items()} + for name in profile["ratios"]: + metrics[name] = 1.0 + ledger = self.put("ledger.jsonl", '{"validator_fixture_only": true}\n') + rows = [] + for variant, occupancies in [("replacement", profile["occupancies"]), ("historical", [1] if profile["historical_control"] else [])]: + for occupancy in occupancies: + for mode in profile["image_modes"]: + for rep in range(1, profile["repetitions"] + 1): + row = {"variant": variant, "occupancy": occupancy, "image_mode": mode, "repetition": rep, + "source_revision": self.revision if variant == "replacement" else profile["historical_revision"], + "executable_sha256": self.executable, "ledger_sha256": ledger["sha256"], + "measured_seconds": profile["duration_seconds"], "metrics": copy.deepcopy(metrics)} + row.update({k: profile[k] for k in ("cells", "nodes", "voters", "offered_requests_per_second")}) + for ratio in profile["ratios"].values(): + row["metrics"].setdefault(ratio["numerator"], 100) + row["metrics"].setdefault(ratio["denominator"], 100) + rows.append(row) + measured = {"metrics": metrics, "cohorts": rows} + measurements = self.put("measurements.json", measured) + topology = {"kind": profile["topology"], "nodes": [ + {"id": str(n), "physical_host": f"host{n}", "failure_domain": f"domain{n % 3}"} + for n in range(profile["nodes"])]} + verification = {"source_revision": self.revision, "complete": True, "ledger_sha256": ledger["sha256"], + "measurements_sha256": measurements["sha256"]} + report = {"format": 1, "profile_id": pid, "source_revision": self.revision, + "profiles_sha256": self.profile_digest, "executable_sha256": self.executable, + "environment": "Synthetic validator fixture only", "topology": profile["topology"], + "metrics": metrics, "artifacts": {"ledger": ledger, "measurements": measurements, + "topology": self.put("topology.json", topology), + "fault_schedule": self.put("faults.json", {"fixture": True}), + "verification": self.put("verification.json", verification)}} + return profile, report + + def edit_artifact(self, report, role, change): + item = report["artifacts"][role] + content = plan.read_json(self.root / item["path"]) + change(content) + report["artifacts"][role] = self.put(item["path"], content) + if role == "measurements": + self.edit_artifact(report, "verification", lambda obj: obj.update(measurements_sha256=report["artifacts"][role]["sha256"])) + + def verify_profile(self, profile, report): + plan.check_profile(report, profile, self.profile_digest, self.revision, self.root, self.executable) + + def test_current_contract_and_index(self): + stats = self.contract() + self.assertEqual(stats["tasks"], 16) + self.assertGreaterEqual(stats["required_cases"], 83) + self.assertEqual(stats["invariants"], 14) + self.assertEqual((plan.PACKAGE / "implementation-plan.md").read_text(), plan.index_text(self.actual_manifest)) + + def test_dependency_order_covers_effects_and_harness(self): + order = plan.ordered(self.manifest["tasks"]) + self.assertLess(order.index("CG-08"), order.index("CG-07")) + self.assertLess(order.index("CG-14"), order.index("CG-13")) + + def test_cycle_refuses(self): + self.manifest["tasks"][0]["depends_on"] = ["CG-16"] + with self.assertRaisesRegex(plan.PlanError, "cycle"): + self.contract() + + def test_unknown_dependency_refuses(self): + self.manifest["tasks"][0]["depends_on"] = ["CG-99"] + with self.assertRaisesRegex(plan.PlanError, "Unknown dependency"): + self.contract() + + def test_duplicate_task_refuses(self): + self.manifest["tasks"].append(copy.deepcopy(self.manifest["tasks"][0])) + with self.assertRaises(plan.PlanError): + self.contract() + + def test_done_without_evidence_refuses(self): + self.manifest["tasks"][0]["status"] = "DONE" + with self.assertRaisesRegex(plan.PlanError, "without evidence"): + self.contract() + + def test_done_before_dependency_refuses(self): + self.manifest["tasks"][1].update(status="DONE", evidence=["fake.json"]) + with self.assertRaisesRegex(plan.PlanError, "before dependency"): + self.contract() + + def test_missing_case_oracle_refuses(self): + self.manifest["tasks"][0]["test_cases"][0]["oracle"] = "" + with self.assertRaisesRegex(plan.PlanError, "empty oracle"): + self.contract() + + def test_uncovered_step_test_refuses(self): + self.manifest["tasks"][0]["steps"][0]["tests"] = ["imaginary_test"] + with self.assertRaisesRegex(plan.PlanError, "unknown step test"): + self.contract() + + def test_profile_removal_or_disable_refuses(self): + saved = copy.deepcopy(self.profiles) + self.profiles["profiles"].pop() + with self.assertRaisesRegex(plan.PlanError, "profile inventory"): + self.contract() + self.profiles = saved + self.profiles["profiles"][0]["required"] = False + with self.assertRaisesRegex(plan.PlanError, "disabled"): + self.contract() + + def test_path_escape_and_symlink_refuse(self): + with self.assertRaises(plan.PlanError): + plan.within(self.root, "../outside") + with tempfile.TemporaryDirectory() as outside: + (self.root / "link").symlink_to(outside, target_is_directory=True) + with self.assertRaises(plan.PlanError): + plan.within(self.root, "link/secret") + + def test_duplicate_json_and_nan_refuse(self): + for text in ('{"format":1,"format":2}', '{"metric":NaN}'): + self.put("bad.json", text) + with self.assertRaises(plan.PlanError): + plan.read_json(self.root / "bad.json") + + def test_actual_reviewed_source_preflight(self): + plan.preflight(self.baseline, plan.ROOT) + + def test_incorrect_excerpt_line_refuses(self): + self.baseline["anchors"]["codec"]["line"] += 1 + with self.assertRaisesRegex(plan.PlanError, "Excerpt line differs"): + plan.preflight(self.baseline, plan.ROOT) + + def test_source_drift_refuses(self): + anchor = copy.deepcopy(self.baseline["anchors"]["identity"]) + content = (plan.ROOT / anchor["path"]).read_bytes() + path = self.root / anchor["path"] + path.parent.mkdir(parents=True) + path.write_bytes(content + b"\n// changed\n") + baseline = {"source_revision": self.revision, "anchors": {"identity": anchor}} + with patch.object(plan, "git_bytes", return_value=content), self.assertRaisesRegex(plan.PlanError, "Source drift"): + plan.preflight(baseline, self.root) + + def test_evidence_accepts_complete_exact_test_records(self): + self.assertEqual(self.verify_evidence(self.evidence()), self.revision) + + def test_missing_command_evidence_refuses(self): + record = self.evidence() + record["commands"].pop() + with self.assertRaisesRegex(plan.PlanError, "Missing required command"): + self.verify_evidence(record) + + def test_zero_selected_or_ignored_test_refuses(self): + for text in ("test result: ok. 0 passed; 0 failed; 0 ignored;", "test result: ok. 0 passed; 0 failed; 1 ignored;"): + record = self.evidence() + run = next(x for x in record["commands"] if "--exact" in x["argv"]) + item = self.put(run["output"], text) + run["sha256"] = item["sha256"] + with self.assertRaisesRegex(plan.PlanError, "Case did not pass"): + self.verify_evidence(record) + + def test_false_inventory_refuses(self): + record = self.evidence() + run = next(x for x in record["commands"] if x["argv"][-1] == "--list") + item = self.put(run["output"], "some_other_test: test\n") + run["sha256"] = item["sha256"] + with self.assertRaisesRegex(plan.PlanError, "absent from inventory"): + self.verify_evidence(record) + + def test_tampered_log_or_failed_exit_refuses(self): + record = self.evidence() + (self.root / record["commands"][0]["output"]).write_text("tampered") + with self.assertRaisesRegex(plan.PlanError, "hash mismatch"): + self.verify_evidence(record) + record = self.evidence() + record["commands"][0]["exit_code"] = 1 + with self.assertRaisesRegex(plan.PlanError, "Failed/unconfirmed"): + self.verify_evidence(record) + + def test_missing_or_wrong_source_input_refuses(self): + record = self.evidence() + record["source_inputs"].pop() + with self.assertRaisesRegex(plan.PlanError, "omits source inputs"): + self.verify_evidence(record) + record = self.evidence() + record["source_inputs"][0]["sha256"] = "0" * 64 + with self.assertRaisesRegex(plan.PlanError, "Source hash mismatch"): + self.verify_evidence(record) + + def test_unrecorded_changed_runtime_source_refuses(self): + with self.assertRaisesRegex(plan.PlanError, "changed runtime inputs"): + self.verify_evidence(self.evidence(), b"crates/unrecorded.rs\0") + + def test_release_commands_repeat_every_exact_case(self): + commands = plan.commands_for(self.manifest["tasks"][-1], self.manifest) + selected = {x[x.index("--exact") - 1] for x in commands if "--exact" in x} + expected = {c["name"] for task in self.manifest["tasks"] for c in task["test_cases"]} + self.assertEqual(selected, expected) + self.assertIn(["python3", "tools/docs/check.py"], commands) + + def test_complete_profile_matrix_passes(self): + self.verify_profile(*self.profile_fixture()) + + def test_profile_source_binary_or_config_drift_refuses(self): + for field, value in (("source_revision", "d" * 40), ("executable_sha256", "d" * 64), ("profiles_sha256", "d" * 64)): + profile, report = self.profile_fixture() + report[field] = value + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_missing_or_duplicate_cohort_refuses(self): + for mutate in (lambda x: x["cohorts"].pop(), lambda x: x["cohorts"].append(copy.deepcopy(x["cohorts"][0]))): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", mutate) + with self.assertRaisesRegex(plan.PlanError, "cohort"): + self.verify_profile(profile, report) + + def test_short_or_wrong_population_cohort_refuses(self): + for field, value in (("measured_seconds", 1), ("cells", 1), ("nodes", 1), ("source_revision", "d" * 40), ("executable_sha256", "d" * 64)): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", lambda x: x["cohorts"][0].update({field: value})) + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_numeric_failure_negative_boolean_or_nan_refuses(self): + for value in (1, -1, True, float("nan")): + profile, report = self.profile_fixture() + report["metrics"]["lost_acknowledged_effects"] = value + with self.assertRaises(plan.PlanError): + self.verify_profile(profile, report) + + def test_better_aggregate_cannot_hide_slow_cohort(self): + profile, report = self.profile_fixture() + def slow(measured): + row = next(x for x in measured["cohorts"] if x["variant"] == "replacement" and x["occupancy"] == 64) + row["metrics"]["p99_ms"] = 400 + row["metrics"]["p99_ratio_to_occupancy1"] = 2 + self.edit_artifact(report, "measurements", slow) + with self.assertRaisesRegex(plan.PlanError, "worst cohort"): + self.verify_profile(profile, report) + + def test_ratio_must_match_control_and_nonzero_denominator(self): + profile, report = self.profile_fixture() + self.edit_artifact(report, "measurements", lambda x: x["cohorts"][0]["metrics"].update(p99_ms=0)) + with self.assertRaisesRegex(plan.PlanError, "denominator"): + self.verify_profile(profile, report) + + def test_safety_failure_in_control_or_historical_cohort_refuses(self): + for variant in ("replacement", "historical"): + profile, report = self.profile_fixture() + def lose(measured): + row = next(x for x in measured["cohorts"] if x["variant"] == variant and x["occupancy"] == 1) + row["metrics"]["lost_acknowledged_effects"] = 1 + self.edit_artifact(report, "measurements", lose) + with self.assertRaisesRegex(plan.PlanError, "worst cohort"): + self.verify_profile(profile, report) + + def test_independent_hosts_cannot_be_faked_by_container_count(self): + profile, report = self.profile_fixture("independent_domain_loss") + self.verify_profile(profile, report) + self.edit_artifact(report, "topology", lambda x: [n.update(physical_host="same-host") for n in x["nodes"]]) + with self.assertRaisesRegex(plan.PlanError, "independent hosts"): + self.verify_profile(profile, report) + + def test_verification_must_bind_actual_ledger(self): + profile, report = self.profile_fixture() + self.edit_artifact(report, "verification", lambda x: x.update(ledger_sha256="d" * 64)) + with self.assertRaisesRegex(plan.PlanError, "another ledger"): + self.verify_profile(profile, report) + + def test_verification_cannot_omit_artifact(self): + profile, report = self.profile_fixture() + del report["artifacts"]["fault_schedule"] + with self.assertRaisesRegex(plan.PlanError, "Missing profile artifact"): + self.verify_profile(profile, report) + + def test_brief_labels_unimplemented_target_and_exact_commands(self): + text = plan.brief(self.manifest["tasks"][0], self.manifest, self.baseline, self.root) + self.assertIn("PLANNED OUTPUT, currently absent", text) + self.assertIn("--exact --test-threads=1", text) + self.assertIn("A plan-only result is not DONE", text) + + def test_release_covers_all_executable_paths_and_plan_only_commits(self): + def git(*args): + return subprocess.check_output(["git", "-c", "core.hooksPath=/dev/null", "-c", "commit.gpgsign=false", + "-c", "user.name=Plan Validator Test", "-c", "user.email=plan-test@example.invalid", + *args], cwd=self.root, text=True, stderr=subprocess.DEVNULL) + git("init", "--quiet") + inputs = ["crates/lib.rs", "examples/main.rs", "vendor/lib.rs", ".cargo/config.toml", "build.rs", + "rust-toolchain.toml", "docs/api.md", "future-runtime-input.txt"] + for name in inputs + ["plans/status.json"]: + self.put(name, "original\n") + git("add", ".") + git("commit", "--quiet", "-m", "Synthetic validator fixture") + revision = git("rev-parse", "HEAD").strip() + plan.require_release_inputs(self.root, revision) + self.put("plans/status.json", "updated plan\n") + plan.require_release_inputs(self.root, revision) + git("add", "plans") + git("commit", "--quiet", "-m", "Plan-only status") + plan.require_release_inputs(self.root, revision) + for name in inputs: + self.put(name, "changed\n") + with self.assertRaisesRegex(plan.PlanError, "dirty"): + plan.require_release_inputs(self.root, revision) + self.put(name, "original\n") + self.put("examples/main.rs", "changed\n") + git("add", "examples") + git("commit", "--quiet", "-m", "Changed executable") + with self.assertRaisesRegex(plan.PlanError, "differ from qualified"): + plan.require_release_inputs(self.root, revision) + + +if __name__ == "__main__": + unittest.main() diff --git a/plans/multi-raft/density-decision.md b/plans/multi-raft/density-decision.md index 64f941481..8e2fc3435 100644 --- a/plans/multi-raft/density-decision.md +++ b/plans/multi-raft/density-decision.md @@ -7,6 +7,13 @@ are pending. This document does not claim that Cellar supports shared groups yet It supersedes one-cell-per-group as the target architecture, while the existing technical design and qualification records describe the implemented baseline. +The [executable delivery package](cell-groups/README.md) expands this decision into +the complete replacement, including dynamic split/merge and coordinated hibernation. +Its detailed protocols supersede the initial assignment sketch below: range-based +ownership and replicated enrollment replace per-cell reservation as the single +assignment authority, and stable cell history is separate from group placement. +Fixed assignment is an intermediate delivery milestone, not the final scope. + ## Decision Make a **cell group** the unit of Raft consensus, placement, membership, snapshot diff --git a/plans/multi-raft/scaling.md b/plans/multi-raft/scaling.md index 0d23b925f..f5a62c73e 100644 --- a/plans/multi-raft/scaling.md +++ b/plans/multi-raft/scaling.md @@ -110,7 +110,9 @@ do not extrapolate a small-counter benchmark to large databases. The [density decision](density-decision.md) selects shared cell groups and specifies their identity, apply, snapshot, archive and recovery boundaries before runtime -conversion. Separate SQLite databases and LTX chains remain per cell. Leadership, +conversion. The [executable delivery package](cell-groups/README.md) defines the +complete implementation, including safe dynamic split/merge and hibernation. +Separate SQLite databases and LTX chains remain per cell. Leadership, membership, placement and the contiguous apply/archive prefixes become group scoped. Dedicated hot cells use the same implementation at occupancy one. From 7484502a2209205d68be42d895abfce7ad15291f Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 08:20:15 -0700 Subject: [PATCH 003/568] feat: separate cell history and group version contracts --- crates/cellar-logic/cell_group.rs | 293 ++++++++++++++++++ crates/cellar-logic/cell_group_tests.rs | 155 +++++++++ crates/cellar-logic/lib.rs | 1 + .../application/consensus/group_contracts.rs | 60 ++++ crates/cellar/application/consensus/mod.rs | 11 + crates/cellar/durability/raft/version.rs | 234 ++++++++++++++ .../tests/native_cell_groups_contracts.rs | 205 ++++++++++++ plans/multi-raft/cell-groups/README.md | 5 +- plans/multi-raft/cell-groups/design.md | 3 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/progress.md | 54 ++++ plans/multi-raft/cell-groups/tasks.json | 2 +- 12 files changed, 1020 insertions(+), 5 deletions(-) create mode 100644 crates/cellar-logic/cell_group.rs create mode 100644 crates/cellar-logic/cell_group_tests.rs create mode 100644 crates/cellar/application/consensus/group_contracts.rs create mode 100644 crates/cellar/tests/native_cell_groups_contracts.rs create mode 100644 plans/multi-raft/cell-groups/progress.md diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs new file mode 100644 index 000000000..d5ee16bd1 --- /dev/null +++ b/crates/cellar-logic/cell_group.rs @@ -0,0 +1,293 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Logical cell identity is independent of the group that currently replicates it. +//! +//! These are pure contracts. Constructing a route, witness or version does not +//! prove ownership, persistence or quorum; adapters must supply those proofs. + +use crate::consensus::{ + ClusterId, GroupId, IncarnationId, LogPosition, NamespaceId, ProposalId, Refusal, +}; +use std::num::NonZeroU64; + +pub const MAX_CELL_ID_BYTES: usize = 1024; +pub const CELL_ROUTE_DOMAIN: &[u8] = b"cellar-cell-route-1\0"; + +/// Exact UTF-8 bytes, without Unicode or locale normalization. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellId(String); + +impl CellId { + pub fn new(value: impl AsRef) -> Result { + let value = value.as_ref(); + Self::validate(value)?; + Ok(Self(value.to_owned())) + } + + pub fn validate(value: &str) -> Result<(), Refusal> { + if value.is_empty() + || value.len() > MAX_CELL_ID_BYTES + || value.chars().any(char::is_control) + { + return Err(Refusal::InvalidIdentity); + } + Ok(()) + } + + pub fn as_str(&self) -> &str { + &self.0 + } + + /// Canonical SHA-256 input; the adapter performs hashing outside the core. + pub fn routing_preimage(&self, namespace: NamespaceId) -> Vec { + let mut bytes = Vec::with_capacity(CELL_ROUTE_DOMAIN.len() + 20 + self.0.len()); + bytes.extend_from_slice(CELL_ROUTE_DOMAIN); + bytes.extend_from_slice(&namespace.bytes()); + bytes.extend_from_slice(&(self.0.len() as u32).to_be_bytes()); + bytes.extend_from_slice(self.0.as_bytes()); + bytes + } +} + +/// Preserved across elections, placement changes and normal split/merge. +/// Disaster recovery must allocate a new cell incarnation. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellHistory { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub cell: CellId, + pub incarnation: IncarnationId, +} + +/// A physical Raft history; deliberately contains no cell identifier. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct GroupHistory { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub group: GroupId, + pub incarnation: IncarnationId, +} + +impl GroupHistory { + pub fn same_namespace(&self, cell: &CellHistory) -> bool { + self.cluster == cell.cluster && self.namespace == cell.namespace + } +} + +macro_rules! nonzero_counter { + ($($name:ident),+ $(,)?) => {$ ( + #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] + pub struct $name(NonZeroU64); + + impl $name { + pub fn new(value: u64) -> Result { + NonZeroU64::new(value).map(Self).ok_or(Refusal::InvalidIdentity) + } + + pub const fn get(self) -> u64 { + self.0.get() + } + + /// Overflow is a refusal, never a wraparound or saturated version. + pub fn checked_next(self) -> Result { + Self::new(self.get().checked_add(1).ok_or(Refusal::InvalidPosition)?) + } + } + )+}; +} + +nonzero_counter!(MutationSequence, RouteEpoch, MembershipRevision); + +/// Ordered hash plus the full ID. Digest collisions never alias distinct cells. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct CellKey { + digest: [u8; 32], + cell: CellId, +} + +impl CellKey { + /// The adapter must verify this digest against the namespace and canonical + /// routing preimage before using a key received over the wire. + pub fn from_digest(digest: [u8; 32], cell: CellId) -> Self { + Self { digest, cell } + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn cell(&self) -> &CellId { + &self.cell + } +} + +/// The declaration order is the canonical ordering, including infinity bounds. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub enum CellBound { + NegativeInfinity, + Key(CellKey), + PositiveInfinity, +} + +/// A nonempty half-open interval of whole cell routing keys. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellRange { + lower: CellBound, + upper: CellBound, +} + +impl CellRange { + pub fn new(lower: CellBound, upper: CellBound) -> Result { + if lower >= upper { + return Err(Refusal::InvalidIdentity); + } + Ok(Self { lower, upper }) + } + + pub fn all() -> Self { + Self { + lower: CellBound::NegativeInfinity, + upper: CellBound::PositiveInfinity, + } + } + + pub fn lower(&self) -> &CellBound { + &self.lower + } + + pub fn upper(&self) -> &CellBound { + &self.upper + } + + pub fn contains(&self, key: &CellKey) -> bool { + let lower = match &self.lower { + CellBound::NegativeInfinity => true, + CellBound::Key(bound) => key >= bound, + CellBound::PositiveInfinity => false, + }; + let upper = match &self.upper { + CellBound::PositiveInfinity => true, + CellBound::Key(bound) => key < bound, + CellBound::NegativeInfinity => false, + }; + lower && upper + } + + pub fn split(&self, at: CellKey) -> Result<(Self, Self), Refusal> { + let at = CellBound::Key(at); + Ok(( + Self::new(self.lower.clone(), at.clone())?, + Self::new(at, self.upper.clone())?, + )) + } + + /// Merge only adjacent intervals in order; overlap and gaps both refuse. + pub fn merge(&self, right: &Self) -> Result { + if self.upper != right.lower { + return Err(Refusal::NonContiguous); + } + Self::new(self.lower.clone(), right.upper.clone()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupRoute { + pub history: GroupHistory, + pub range: CellRange, + pub epoch: RouteEpoch, + pub birth_digest: [u8; 32], + pub schema_digest: [u8; 32], +} + +impl GroupRoute { + /// The supplied key must already have its namespace hash verified. + pub fn check_cell(&self, cell: &CellHistory, key: &CellKey) -> Result<(), Refusal> { + if !self.history.same_namespace(cell) + || key.cell() != &cell.cell + || !self.range.contains(key) + { + return Err(Refusal::WrongHistory); + } + Ok(()) + } +} + +/// An LTX position with a complete, domain-specific image commitment. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellImage { + pub epoch: u64, + pub txid: u64, + pub checksum: u64, + pub commitment: CellImageCommitment, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellImageCommitment { + Materialized([u8; 32]), + Extents { root: [u8; 32], manifest_bytes: u32 }, +} + +impl CellImage { + pub fn validate(self) -> Result<(), Refusal> { + if self.epoch == 0 + || matches!( + self.commitment, + CellImageCommitment::Extents { + manifest_bytes: 0, + .. + } + ) + { + return Err(Refusal::InvalidPosition); + } + Ok(()) + } +} + +/// Authenticated cell progress, independent of the current group's log index. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellVersion { + pub history: CellHistory, + pub sequence: MutationSequence, + pub image: CellImage, +} + +impl CellVersion { + /// Call only after authenticating the token and validating its schema and + /// installed lineage. A witness from an old group is not current authority. + pub fn covers(&self, required: &Self) -> Result { + self.image.validate()?; + required.image.validate()?; + if self.history != required.history + || (self.sequence == required.sequence && self.image != required.image) + { + return Err(Refusal::WrongHistory); + } + Ok(self.sequence >= required.sequence) + } +} + +/// Locates the original decision, including after transfer to a new group. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CommitWitness { + pub group: GroupHistory, + pub position: LogPosition, + pub proposal: ProposalId, +} + +impl CommitWitness { + pub fn validate(&self, cell: &CellHistory) -> Result<(), Refusal> { + self.position.validate()?; + if !self.group.same_namespace(cell) { + return Err(Refusal::WrongHistory); + } + if self.position.index.get() == 0 { + return Err(Refusal::InvalidPosition); + } + Ok(()) + } +} + +#[cfg(test)] +#[path = "cell_group_tests.rs"] +mod tests; diff --git a/crates/cellar-logic/cell_group_tests.rs b/crates/cellar-logic/cell_group_tests.rs new file mode 100644 index 000000000..4d68783b5 --- /dev/null +++ b/crates/cellar-logic/cell_group_tests.rs @@ -0,0 +1,155 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +fn key(digest: u8, id: &str) -> CellKey { + CellKey::from_digest([digest; 32], CellId::new(id).unwrap()) +} + +#[test] +fn routing_rejects_foreign_namespace_identity_and_range() { + let cell = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let route = GroupRoute { + history: GroupHistory { + cluster: cell.cluster, + namespace: cell.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + range: CellRange::all().split(key(3, "boundary")).unwrap().0, + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [7; 32], + schema_digest: [8; 32], + }; + assert_eq!(route.check_cell(&cell, &key(2, "alpha")), Ok(())); + assert_eq!( + route.check_cell(&cell, &key(2, "beta")), + Err(Refusal::WrongHistory) + ); + assert_eq!( + route.check_cell(&cell, &key(4, "alpha")), + Err(Refusal::WrongHistory) + ); + let mut foreign = cell.clone(); + foreign.namespace = NamespaceId::new([9; 16]).unwrap(); + assert_eq!( + route.check_cell(&foreign, &key(2, "alpha")), + Err(Refusal::WrongHistory) + ); + foreign.namespace = cell.namespace; + foreign.cluster = ClusterId::new([9; 16]).unwrap(); + assert_eq!( + route.check_cell(&foreign, &key(2, "alpha")), + Err(Refusal::WrongHistory) + ); +} + +#[test] +fn ranges_partition_digest_collisions_without_aliases() { + let first = key(2, "alpha"); + let second = key(2, "beta"); + let third = key(3, "alpha"); + assert!(first < second && second < third); + let all = CellRange::all(); + let (left, right) = all.split(second.clone()).unwrap(); + for value in [key(0, "a"), first, second, third, key(255, "z")] { + assert_ne!(left.contains(&value), right.contains(&value)); + } + assert_eq!(left.merge(&right), Ok(all)); + assert_eq!(right.merge(&left), Err(Refusal::NonContiguous)); + assert!(left.split(key(2, "beta")).is_err()); + assert!(right.split(key(2, "beta")).is_err()); + assert!(left.split(key(3, "outside")).is_err()); + for (lower, upper) in [ + (CellBound::PositiveInfinity, CellBound::PositiveInfinity), + (CellBound::PositiveInfinity, CellBound::NegativeInfinity), + (CellBound::NegativeInfinity, CellBound::NegativeInfinity), + ] { + assert!(CellRange::new(lower, upper).is_err()); + } +} + +#[test] +fn identifiers_and_counters_have_no_silent_normalization_or_wraparound() { + for value in ["", "\n", "a\0b", "\u{85}", "a\u{7f}"] { + assert_eq!(CellId::new(value), Err(Refusal::InvalidIdentity)); + } + assert!(CellId::new("é".repeat(512)).is_ok()); + assert!(CellId::new("é".repeat(513)).is_err()); + assert_ne!(CellId::new("é").unwrap(), CellId::new("e\u{301}").unwrap()); + assert!(MutationSequence::new(0).is_err()); + assert_eq!( + MutationSequence::new(1) + .unwrap() + .checked_next() + .unwrap() + .get(), + 2 + ); + assert_eq!( + MutationSequence::new(u64::MAX).unwrap().checked_next(), + Err(Refusal::InvalidPosition) + ); + assert!(RouteEpoch::new(0).is_err()); + assert!(RouteEpoch::new(u64::MAX).unwrap().checked_next().is_err()); + assert!(MembershipRevision::new(0).is_err()); + assert!(MembershipRevision::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); +} + +#[test] +fn cell_versions_require_exact_history_and_same_sequence_image() { + let version = CellVersion { + history: CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }, + sequence: MutationSequence::new(2).unwrap(), + image: CellImage { + epoch: 1, + txid: 2, + checksum: 123, + commitment: CellImageCommitment::Materialized([7; 32]), + }, + }; + assert_eq!(version.covers(&version), Ok(true)); + let mut later = version.clone(); + later.sequence = later.sequence.checked_next().unwrap(); + later.image.txid += 1; + assert_eq!(later.covers(&version), Ok(true)); + assert_eq!(version.covers(&later), Ok(false)); + for field in 0..4 { + let mut other = later.clone(); + match field { + 0 => other.history.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => other.history.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => other.history.cell = CellId::new("beta").unwrap(), + _ => other.history.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert_eq!(other.covers(&version), Err(Refusal::WrongHistory)); + } + for field in 0..4 { + let mut conflicting = version.clone(); + match field { + 0 => conflicting.image.epoch += 1, + 1 => conflicting.image.txid += 1, + 2 => conflicting.image.checksum += 1, + _ => { + conflicting.image.commitment = CellImageCommitment::Extents { + root: [7; 32], + manifest_bytes: 112, + } + } + } + assert_eq!(conflicting.covers(&version), Err(Refusal::WrongHistory)); + } +} diff --git a/crates/cellar-logic/lib.rs b/crates/cellar-logic/lib.rs index 6248188bf..5c9e018c4 100644 --- a/crates/cellar-logic/lib.rs +++ b/crates/cellar-logic/lib.rs @@ -14,6 +14,7 @@ use std::collections::{BTreeMap, BTreeSet, VecDeque}; pub mod alarm; pub mod cache; pub mod cell; +pub mod cell_group; pub mod consensus; pub mod cron; pub mod drain; diff --git a/crates/cellar/application/consensus/group_contracts.rs b/crates/cellar/application/consensus/group_contracts.rs new file mode 100644 index 000000000..c89de0e08 --- /dev/null +++ b/crates/cellar/application/consensus/group_contracts.rs @@ -0,0 +1,60 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Host-side hashing for the dependency-free cell-group contracts. + +use super::protocol::{ImageCommitment, LtxImage, ProtocolError}; +use cellar_logic::{ + cell_group::{CellId, CellImage, CellImageCommitment, CellKey}, + consensus::NamespaceId, +}; +use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; +use sha2::{Digest, Sha256}; + +pub fn cell_routing_key(namespace: NamespaceId, cell: &CellId) -> CellKey { + CellKey::from_digest( + Sha256::digest(cell.routing_preimage(namespace)).into(), + cell.clone(), + ) +} + +impl TryFrom for LtxImage { + type Error = ProtocolError; + + fn try_from(image: CellImage) -> Result { + image.validate().map_err(|_| ProtocolError::Chain)?; + let commitment = match image.commitment { + CellImageCommitment::Materialized(digest) => ImageCommitment::Sha256(digest), + CellImageCommitment::Extents { + root, + manifest_bytes, + } => ImageCommitment::Extents( + ExtentRootReference::new(ImageRoot::from_bytes(root), manifest_bytes) + .map_err(|_| ProtocolError::Identity)?, + ), + }; + Ok(Self { + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + commitment, + }) + } +} + +impl From for CellImage { + fn from(image: LtxImage) -> Self { + let commitment = match image.commitment { + ImageCommitment::Sha256(digest) => CellImageCommitment::Materialized(digest), + ImageCommitment::Extents(reference) => CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + }, + }; + Self { + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + commitment, + } + } +} diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 9a56d25de..c7e48f9b7 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -10,6 +10,7 @@ mod archive_status; pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; pub(crate) mod client; +mod group_contracts; mod performance; pub use performance::{ RaftDurationBucket, RaftOperationPerformance, RaftPerformanceCounter, RaftPerformanceSnapshot, @@ -206,4 +207,14 @@ pub mod verification { pub use crate::raft::transport_options::{ NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, TransportOptions, }; + /// Contracts under integration into the replacement group runtime. + pub mod cell_groups { + pub use super::super::group_contracts::cell_routing_key; + pub use crate::raft::version::{CellVersionObservation, CellVersionToken}; + pub use cellar_logic::cell_group::{ + CellBound, CellHistory, CellId, CellImage, CellImageCommitment, CellKey, CellRange, + CellVersion, CommitWitness, GroupHistory, GroupRoute, MembershipRevision, + MutationSequence, RouteEpoch, + }; + } } diff --git a/crates/cellar/durability/raft/version.rs b/crates/cellar/durability/raft/version.rs index 22e680fc6..49312c9d3 100644 --- a/crates/cellar/durability/raft/version.rs +++ b/crates/cellar/durability/raft/version.rs @@ -4,6 +4,10 @@ //! and contain no leader identity: a normal election preserves their meaning. use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; +use cellar_logic::cell_group::{ + CellHistory, CellId, CellImage, CellImageCommitment, CellVersion, CommitWitness, GroupHistory, + MutationSequence, +}; use cellar_logic::consensus::*; use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; use sha2::{Digest, Sha256}; @@ -15,6 +19,55 @@ const MAX_BYTES: usize = 1408; const MAX_ENCODED: usize = 1878; const MAX_KEYS: usize = 64; const MAX_LIFETIME_MS: u64 = 7 * 24 * 60 * 60 * 1000; +const CELL_MAGIC: &[u8; 8] = b"CLRCVS01"; +const CELL_DOMAIN: &[u8] = b"cellar-cell-group-version-1"; + +// Framing/key/time, cell identity, sequence/image, schema, witness and signature. +const CELL_FIXED_BYTES: usize = 8 + 32 + 16 + 48 + 2 + 8 + 24 + 37 + 32 + 64 + 16 + 16 + 64; +const CELL_MAX_BYTES: usize = CELL_FIXED_BYTES + cellar_logic::cell_group::MAX_CELL_ID_BYTES; +const CELL_MAX_ENCODED: usize = CELL_MAX_BYTES.div_ceil(3) * 4; + +/// Cell-scoped token for the group runtime. Old group-scoped tokens cannot be +/// interpreted in this domain. The original witness is provenance, not routing. +#[derive(Clone, PartialEq, Eq)] +pub struct CellVersionToken(String); + +impl CellVersionToken { + pub fn from_encoded(encoded: &str) -> Result { + if encoded.len() > CELL_MAX_ENCODED { + return Err(VersionError::Limit); + } + let bytes = URL_SAFE_NO_PAD + .decode(encoded) + .map_err(|_| VersionError::Invalid)?; + if bytes.len() < CELL_FIXED_BYTES + 1 + || bytes.len() > CELL_MAX_BYTES + || !bytes.starts_with(CELL_MAGIC) + || URL_SAFE_NO_PAD.encode(&bytes) != encoded + { + return Err(VersionError::Invalid); + } + Ok(Self(encoded.into())) + } + + pub fn as_str(&self) -> &str { + &self.0 + } +} + +impl fmt::Debug for CellVersionToken { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("CellVersionToken") + .field("bytes", &self.0.len()) + .finish() + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellVersionObservation { + pub version: CellVersion, + pub witness: CommitWitness, +} #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub enum VersionError { @@ -170,6 +223,187 @@ impl fmt::Debug for VersionKeys { } impl VersionKeys { + pub fn issue_cell( + &self, + observation: &CellVersionObservation, + schema: [u8; 32], + now_ms: u64, + ) -> Result { + let version = &observation.version; + let witness = &observation.witness; + witness + .validate(&version.history) + .map_err(|_| VersionError::Invalid)?; + crate::application::consensus::LtxImage::try_from(version.image) + .map_err(|_| VersionError::Invalid)?; + self.validate_retention(&[], now_ms)?; + let mut bytes = Vec::with_capacity(CELL_FIXED_BYTES + version.history.cell.as_str().len()); + bytes.extend_from_slice(CELL_MAGIC); + bytes.extend_from_slice(&self.0.active.verifier.id()); + bytes.extend_from_slice(&now_ms.to_be_bytes()); + bytes.extend_from_slice(&(now_ms + self.0.lifetime_ms).to_be_bytes()); + for id in [ + version.history.cluster.bytes(), + version.history.namespace.bytes(), + version.history.incarnation.bytes(), + ] { + bytes.extend_from_slice(&id); + } + let cell = version.history.cell.as_str().as_bytes(); + bytes.extend_from_slice(&(cell.len() as u16).to_be_bytes()); + bytes.extend_from_slice(cell); + bytes.extend_from_slice(&version.sequence.get().to_be_bytes()); + for value in [ + version.image.epoch, + version.image.txid, + version.image.checksum, + ] { + bytes.extend_from_slice(&value.to_be_bytes()); + } + let (tag, digest, length) = match version.image.commitment { + CellImageCommitment::Materialized(digest) => (1, digest, 0), + CellImageCommitment::Extents { + root, + manifest_bytes, + } => (2, root, manifest_bytes), + }; + bytes.push(tag); + bytes.extend_from_slice(&digest); + bytes.extend_from_slice(&length.to_be_bytes()); + bytes.extend_from_slice(&schema); + for id in [ + witness.group.cluster.bytes(), + witness.group.namespace.bytes(), + witness.group.group.bytes(), + witness.group.incarnation.bytes(), + ] { + bytes.extend_from_slice(&id); + } + bytes.extend_from_slice(&witness.position.index.get().to_be_bytes()); + bytes.extend_from_slice(&witness.position.term.get().to_be_bytes()); + bytes.extend_from_slice(&witness.proposal.bytes()); + let signature = self.0.active.secret.sign(&[CELL_DOMAIN, &bytes].concat()); + bytes.extend_from_slice(&signature.to_bytes()); + CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(bytes)) + } + + /// Authenticate against logical history. The caller must separately check + /// installed transfer provenance and current quorum/read authority. + pub fn verify_cell( + &self, + token: &CellVersionToken, + history: &CellHistory, + schema: [u8; 32], + now_ms: u64, + ) -> Result { + let bytes = URL_SAFE_NO_PAD + .decode(token.as_str()) + .map_err(|_| VersionError::Invalid)?; + let (payload, signature) = + bytes.split_at(bytes.len().checked_sub(64).ok_or(VersionError::Invalid)?); + let mut input = Input(payload); + if input.take(8)? != CELL_MAGIC { + return Err(VersionError::Invalid); + } + let key = self + .0 + .verification + .get(&input.array()?) + .ok_or(VersionError::Invalid)?; + VerifyingKey::from_bytes(&key.public) + .map_err(|_| VersionError::Invalid)? + .verify_strict( + &[CELL_DOMAIN, payload].concat(), + &Signature::from_slice(signature).map_err(|_| VersionError::Invalid)?, + ) + .map_err(|_| VersionError::Invalid)?; + let issued = input.u64()?; + let expires = input.u64()?; + if issued > now_ms + || now_ms >= expires + || expires > key.verify_until_ms + || expires <= issued + || expires - issued > MAX_LIFETIME_MS + { + return Err(VersionError::Expired); + } + let cluster = ClusterId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let namespace = NamespaceId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let incarnation = IncarnationId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let n = u16::from_be_bytes(input.array()?) as usize; + if n > cellar_logic::cell_group::MAX_CELL_ID_BYTES { + return Err(VersionError::Limit); + } + let cell = + CellId::new(std::str::from_utf8(input.take(n)?).map_err(|_| VersionError::Invalid)?) + .map_err(|_| VersionError::Invalid)?; + let decoded = CellHistory { + cluster, + namespace, + cell, + incarnation, + }; + if &decoded != history { + return Err(VersionError::WrongHistory); + } + let sequence = MutationSequence::new(input.u64()?).map_err(|_| VersionError::Invalid)?; + let epoch = input.u64()?; + let txid = input.u64()?; + let checksum = input.u64()?; + let tag = input.array::<1>()?[0]; + let root = input.array()?; + let manifest_bytes = u32::from_be_bytes(input.array()?); + let commitment = match (tag, manifest_bytes) { + (1, 0) => CellImageCommitment::Materialized(root), + (2, _) => CellImageCommitment::Extents { + root, + manifest_bytes, + }, + _ => return Err(VersionError::Invalid), + }; + let image = CellImage { + epoch, + txid, + checksum, + commitment, + }; + crate::application::consensus::LtxImage::try_from(image) + .map_err(|_| VersionError::Invalid)?; + if input.array::<32>()? != schema { + return Err(VersionError::WrongSchema); + } + let group = GroupHistory { + cluster: ClusterId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + namespace: NamespaceId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + group: GroupId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + incarnation: IncarnationId::new(input.array()?).map_err(|_| VersionError::Invalid)?, + }; + let position = LogPosition { + index: LogIndex::new(input.u64()?), + term: Term::new(input.u64()?), + }; + let proposal = ProposalId::new(input.array()?).map_err(|_| VersionError::Invalid)?; + let witness = CommitWitness { + group, + position, + proposal, + }; + witness + .validate(&decoded) + .map_err(|_| VersionError::Invalid)?; + if !input.0.is_empty() { + return Err(VersionError::Invalid); + } + Ok(CellVersionObservation { + version: CellVersion { + history: decoded, + sequence, + image, + }, + witness, + }) + } + pub fn new( active: VersionSigningKey, retained: Vec, diff --git a/crates/cellar/tests/native_cell_groups_contracts.rs b/crates/cellar/tests/native_cell_groups_contracts.rs new file mode 100644 index 000000000..71188d5e1 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_contracts.rs @@ -0,0 +1,205 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-group protocol boundary tests. These do not qualify a running group. + +use cellar::application::consensus::verification::cell_groups::*; +use cellar::application::consensus::NamespaceId; +use cellar::application::consensus::{ + ClusterId, GroupId, IncarnationId, LogIndex, LogPosition, ProposalId, Term, VersionError, + VersionKeys, VersionSigningKey, VersionToken, +}; +use std::time::Duration; + +fn observation() -> CellVersionObservation { + let history = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("alpha").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let group = GroupHistory { + cluster: history.cluster, + namespace: history.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }; + CellVersionObservation { + version: CellVersion { + history, + sequence: MutationSequence::new(7).unwrap(), + image: CellImage { + epoch: 1, + txid: 10, + checksum: 17, + commitment: CellImageCommitment::Materialized([8; 32]), + }, + }, + witness: CommitWitness { + group, + position: LogPosition { + index: LogIndex::new(100), + term: Term::new(9), + }, + proposal: ProposalId::new([6; 16]).unwrap(), + }, + } +} + +fn keys() -> VersionKeys { + VersionKeys::new( + VersionSigningKey::new([7; 32], 100_000).unwrap(), + vec![], + Duration::from_secs(10), + ) + .unwrap() +} + +#[test] +fn version_survives_transfer_not_disaster() { + let keys = keys(); + let original = observation(); + let schema = [42; 32]; + let token = keys.issue_cell(&original, schema, 100).unwrap(); + assert!(VersionToken::from_encoded(token.as_str()).is_err()); + let decoded = keys + .verify_cell(&token, &original.version.history, schema, 101) + .unwrap(); + assert_eq!(decoded, original); + let mut moved = original.clone(); + moved.witness.group.group = GroupId::new([10; 16]).unwrap(); + moved.witness.group.incarnation = IncarnationId::new([11; 16]).unwrap(); + moved.witness.position = LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }; + moved.version.sequence = moved.version.sequence.checked_next().unwrap(); + moved.version.image.txid += 1; + let successor = keys.issue_cell(&moved, schema, 102).unwrap(); + let successor = keys + .verify_cell(&successor, &original.version.history, schema, 103) + .unwrap(); + assert!(successor.version.covers(&decoded.version).unwrap()); + assert!(!decoded.version.covers(&successor.version).unwrap()); + for field in 0..4 { + let mut other = original.version.history.clone(); + match field { + 0 => other.cluster = ClusterId::new([12; 16]).unwrap(), + 1 => other.namespace = NamespaceId::new([12; 16]).unwrap(), + 2 => other.cell = CellId::new("beta").unwrap(), + _ => other.incarnation = IncarnationId::new([12; 16]).unwrap(), + } + assert_eq!( + keys.verify_cell(&token, &other, schema, 103), + Err(VersionError::WrongHistory) + ); + } + assert_eq!( + keys.verify_cell(&token, &original.version.history, [43; 32], 103), + Err(VersionError::WrongSchema) + ); + assert_eq!( + keys.verify_cell(&token, &original.version.history, schema, 10_100), + Err(VersionError::Expired) + ); + assert_eq!( + keys.verify_cell(&token, &original.version.history, schema, 99), + Err(VersionError::Expired) + ); +} + +#[test] +fn cell_token_bounds_tampering_and_commitment_domains() { + use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; + let keys = keys(); + let mut original = observation(); + original.version.history.cell = CellId::new("a").unwrap(); + let shortest = keys.issue_cell(&original, [42; 32], 100).unwrap(); + assert_eq!( + URL_SAFE_NO_PAD.decode(shortest.as_str()).unwrap().len(), + 368 + ); + assert_eq!( + keys.verify_cell(&shortest, &original.version.history, [42; 32], 101) + .unwrap(), + original + ); + original.version.history.cell = CellId::new("é".repeat(512)).unwrap(); + original.version.image.commitment = CellImageCommitment::Extents { + root: [9; 32], + manifest_bytes: 112, + }; + let token = keys.issue_cell(&original, [42; 32], 100).unwrap(); + assert_eq!( + keys.verify_cell(&token, &original.version.history, [42; 32], 101) + .unwrap(), + original + ); + let bytes = URL_SAFE_NO_PAD.decode(token.as_str()).unwrap(); + for offset in [0, 8, 40, 120, bytes.len() - 65, bytes.len() - 1] { + let mut changed = bytes.clone(); + changed[offset] ^= 1; + if let Ok(tampered) = CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(changed)) { + assert!(keys + .verify_cell(&tampered, &original.version.history, [42; 32], 101) + .is_err()); + } + } + for end in 0..bytes.len() { + if let Ok(truncated) = + CellVersionToken::from_encoded(&URL_SAFE_NO_PAD.encode(&bytes[..end])) + { + assert!(keys + .verify_cell(&truncated, &original.version.history, [42; 32], 101) + .is_err()); + } + } + assert_eq!( + CellVersionToken::from_encoded(&"a".repeat(2049)), + Err(VersionError::Limit) + ); + original.version.image.commitment = CellImageCommitment::Extents { + root: [9; 32], + manifest_bytes: 113, + }; + assert_eq!( + keys.issue_cell(&original, [42; 32], 100), + Err(VersionError::Invalid) + ); + original.version.image.commitment = CellImageCommitment::Materialized([9; 32]); + original.witness.group.namespace = NamespaceId::new([15; 16]).unwrap(); + assert_eq!( + keys.issue_cell(&original, [42; 32], 100), + Err(VersionError::Invalid) + ); +} + +#[test] +fn identity_and_hash_boundaries() { + let namespace = NamespaceId::new([2; 16]).unwrap(); + let alpha = CellId::new("alpha").unwrap(); + let key = cell_routing_key(namespace, &alpha); + // Independent SHA-256 golden vector for the specified preimage, including NUL. + assert_eq!( + hex::encode(key.digest()), + "9125bede939b7512fcd2b4567a7c60ee066a3baf5847a265088e48449e408f1b" + ); + assert_ne!( + key, + cell_routing_key(NamespaceId::new([3; 16]).unwrap(), &alpha) + ); + let tied = CellKey::from_digest(key.digest(), CellId::new("beta").unwrap()); + assert!(key < tied); + let (left, right) = CellRange::all().split(tied.clone()).unwrap(); + assert!(left.contains(&key) && !right.contains(&key)); + assert!(!left.contains(&tied) && right.contains(&tied)); + assert_eq!(left.merge(&right).unwrap(), CellRange::all()); + assert!(right.merge(&left).is_err()); + assert!(CellRange::new(CellBound::PositiveInfinity, CellBound::NegativeInfinity).is_err()); + assert!(CellId::new("").is_err()); + assert!(CellId::new("a\nb").is_err()); + assert!(CellId::new("a".repeat(1025)).is_err()); + assert!(MutationSequence::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); +} diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index fcda07fa2..03342e43d 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -2,8 +2,9 @@ # Cell group delivery package -Status: executable design and implementation plan, requested on 2026-10-29. -The production runtime is unchanged. All CG implementation tasks are TODO. +Status: implementation started on 2026-10-29 following the executable plan. +CG-01 is in progress; the production host has not yet cut over to shared groups. +See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. This package implements the direction selected in the diff --git a/plans/multi-raft/cell-groups/design.md b/plans/multi-raft/cell-groups/design.md index fade0eb12..4162a7b9c 100644 --- a/plans/multi-raft/cell-groups/design.md +++ b/plans/multi-raft/cell-groups/design.md @@ -2,7 +2,8 @@ # Cell group design -Status: target contract for CG-01 through CG-16; not implemented. +Status: target contract for CG-01 through CG-16; implementation in progress. +See [implementation progress](progress.md); no runtime cutover is qualified yet. Cellar baseline: `1be27d234d4674d4d014785c60cecbfe394bdab9`. TiKV investigation: `b3995dab28308f09cbb7c125bf32f789dbc203fe` in the supplied checkout. The [decision](../density-decision.md) records the inspected source. diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index a79427dde..f95448fbf 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -12,7 +12,7 @@ never executed by the plan runner. Read the [design](design.md), | Task | Deliverable | Dependencies | Status | Required cases | | --- | --- | --- | --- | ---: | -| CG-01 | Separate logical cell history from physical group consensus | None | TODO | 4 | +| CG-01 | Separate logical cell history from physical group consensus | None | IN_PROGRESS | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | TODO | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md new file mode 100644 index 000000000..cd49c9c0e --- /dev/null +++ b/plans/multi-raft/cell-groups/progress.md @@ -0,0 +1,54 @@ + + +# Cell-group implementation progress + +The active objective is the entire design and a hard cutover, including all CG-01 +through CG-16 qualification gates. Intermediate contracts do not qualify a shared +runtime. No production legacy/shared mode flag is being added. + +## CG-01 — in progress + +Implemented contracts: + +- Dependency-free validated cell IDs, logical cell histories and physical group + histories; nonzero mutation, route and membership counters refuse overflow. +- Canonical namespace routing preimage and adapter SHA-256, collision-safe ordered + keys, half-open ranges, adjacent merge and whole-key split boundaries. +- Cell versions compare mutation sequence within one logical history, rejecting + equal-sequence conflicting image commitments. Commit witnesses remain separate. +- Domain-separated signed cell-version tokens authenticate schema, cell history, + sequence, LTX/image commitment and the original physical commit witness. Transfer + can preserve the token's history; disaster recovery's new incarnation refuses it. +- Materialized/extent commitment conversion validates the actual extent reference + layout at the host boundary. Core contracts add no dependencies or I/O. + +The new contracts are exposed through the existing unstable verification surface +while host integration is pending. Existing public runtime consumers still use the +old contracts and will switch together at CG-07, before legacy removal at CG-16. + +Remaining CG-01 work includes the cell-addressed data and typed administrative +entry codec, bounded decoder tests, complete task checks and source-bound evidence. +CG-02 and later tasks remain TODO. No task is DONE and no benchmark, RustFS drill, +hibernation latency, recovery guarantee or production cutover is claimed here. + +Checks completed for this intermediate change: + +- `native_cell_groups_contracts`: 3 cases pass, including the smallest and largest + cell IDs in the signed format, tampering, truncation and transfer/DR identity. +- `native_raft_contract`: all 6 existing cases pass. +- `native_raft_reads`: exact `query_token_auth_scope` and + `version_key_deadline_and_payload_limits` regressions pass. +- `cellar-logic --lib`: all 18 cases pass, including 4 new group-contract cases. +- Clippy for the affected libraries and new native test passes with `-D warnings`; + module layout, native dependency isolation and repository documentation pass. + +The first native run exposed an incorrect minimum token length for short IDs. +The bound now derives from the fixed wire fields, and a one-byte ID regression +passes. Local logs, including that failed run, remain under the ignored +`.cellar/cell-groups-evidence/CG-01-in-progress/` directory. They are intermediate +checks, not the complete source-bound CG-01 evidence record. + +The original source preflight was inspected before edits. `version.rs` is now an +intentional CG-01 change; update its reviewed baseline at the completed source +revision before dependent implementation starts. Do not replace a mismatched hash +without reading the changed contract. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 1451ab7c0..deaccc452 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -22,7 +22,7 @@ { "id": "CG-01", "title": "Separate logical cell history from physical group consensus", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "M", "risk": "HIGH", From 4a844aa02be113390f7eb7c66c8e6b9252237dd9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 08:21:47 -0700 Subject: [PATCH 004/568] docs: record verified cell-group contract checkpoint --- plans/multi-raft/cell-groups/baseline.json | 8 ++++---- plans/multi-raft/cell-groups/progress.md | 11 +++++++---- 2 files changed, 11 insertions(+), 8 deletions(-) diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 2495c47c4..22802c4a3 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "70b6022c8baba3d0a52de6bd9fa11866bdbd3b3f", + "source_revision": "7484502a2209205d68be42d895abfce7ad15291f", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: inspected current owners, callers, native fixtures and CI; planning-only branch at 70b6022c. Reconcile changed source explicitly before implementation.", + "review_note": "2026-10-29: reviewed CG-01 intermediate contracts at 7484502a2209205d68be42d895abfce7ad15291f. version.rs adds the separate cell-token domain and logical-history verification; original group-token consumers remain pending cutover. Other pinned owner inputs are unchanged. CG-01 is IN_PROGRESS, not qualified.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -73,8 +73,8 @@ }, "version": { "path": "crates/cellar/durability/raft/version.rs", - "sha256": "950d023f70e7b3ab1242e9b9d2a02475da4097f8aed0e9bf888e21c35a214c37", - "line": 20, + "sha256": "1b5294e02c9a5299e752fa5656ee153876d1b76fa5ffaa06c7da2d53defe53fe", + "line": 73, "excerpt": "pub enum VersionError {\n Invalid,\n Expired,\n WrongHistory,\n WrongSchema,\n KeyRetention,\n Limit,\n}\n" }, "snapshot": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index cd49c9c0e..25224f45e 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -41,6 +41,8 @@ Checks completed for this intermediate change: - `cellar-logic --lib`: all 18 cases pass, including 4 new group-contract cases. - Clippy for the affected libraries and new native test passes with `-D warnings`; module layout, native dependency isolation and repository documentation pass. +- Repository formatting check, all 36 plan-validator tests, package validation + and the reviewed source preflight pass at this checkpoint. The first native run exposed an incorrect minimum token length for short IDs. The bound now derives from the fixed wire fields, and a one-byte ID regression @@ -48,7 +50,8 @@ passes. Local logs, including that failed run, remain under the ignored `.cellar/cell-groups-evidence/CG-01-in-progress/` directory. They are intermediate checks, not the complete source-bound CG-01 evidence record. -The original source preflight was inspected before edits. `version.rs` is now an -intentional CG-01 change; update its reviewed baseline at the completed source -revision before dependent implementation starts. Do not replace a mismatched hash -without reading the changed contract. +The original source preflight was inspected before edits. The reviewed source +baseline advances to `7484502a` for this intermediate checkpoint; `version.rs` adds +the separate token domain and all other pinned owners remain unchanged. This +baseline update is not task completion. Review further changed contracts before +refreshing their hashes or beginning dependent implementation. From ee7f4a8d2dec220904f5a6860041f12c14db1cf5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 08:47:09 -0700 Subject: [PATCH 005/568] feat: encode bounded cell-group data and administrative entries --- crates/cellar-logic/cell_group.rs | 24 +- .../consensus/group_protocol/README.md | 81 +++ .../consensus/group_protocol/codec.rs | 625 ++++++++++++++++++ .../consensus/group_protocol/mod.rs | 9 + .../consensus/group_protocol/types.rs | 369 +++++++++++ crates/cellar/application/consensus/mod.rs | 4 +- .../tests/native_cell_groups_contracts.rs | 541 +++++++++++++++ plans/multi-raft/cell-groups/progress.md | 11 +- plans/multi-raft/cell-groups/tasks.json | 6 +- 9 files changed, 1665 insertions(+), 5 deletions(-) create mode 100644 crates/cellar/application/consensus/group_protocol/README.md create mode 100644 crates/cellar/application/consensus/group_protocol/codec.rs create mode 100644 crates/cellar/application/consensus/group_protocol/mod.rs create mode 100644 crates/cellar/application/consensus/group_protocol/types.rs diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs index d5ee16bd1..ba694ab6f 100644 --- a/crates/cellar-logic/cell_group.rs +++ b/crates/cellar-logic/cell_group.rs @@ -96,7 +96,29 @@ macro_rules! nonzero_counter { )+}; } -nonzero_counter!(MutationSequence, RouteEpoch, MembershipRevision); +nonzero_counter!( + MutationSequence, + RouteEpoch, + MembershipRevision, + ParkGeneration +); + +/// Non-reusable identity of a namespace topology operation. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] +pub struct TransitionId([u8; 16]); + +impl TransitionId { + pub fn new(bytes: [u8; 16]) -> Result { + if bytes == [0; 16] { + return Err(Refusal::InvalidIdentity); + } + Ok(Self(bytes)) + } + + pub const fn bytes(self) -> [u8; 16] { + self.0 + } +} /// Ordered hash plus the full ID. Digest collisions never alias distinct cells. #[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] diff --git a/crates/cellar/application/consensus/group_protocol/README.md b/crates/cellar/application/consensus/group_protocol/README.md new file mode 100644 index 000000000..25b939b8d --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/README.md @@ -0,0 +1,81 @@ + + +# Cell-group entry encoding + +This module encodes the replacement group runtime's cell-addressed log entries. +The existing host is not switched over by the codec. Storage and runtime owners +must verify quorum, physical images, membership and transition proofs separately. + +## Frame and bounds + +Every integer is unsigned big-endian. The marker stays at 1; the new magic and +semantic domain refuse old per-cell entries in both directions. + +| Field | Bytes | +| --- | ---: | +| `CLRCGE01` | 8 | +| Marker, exactly 1 | 4 | +| Header length | 4 | +| Payload length | 8 | +| Header | Declared, bounded | +| Concatenated LTX files | Declared, bounded | +| SHA-256 of all preceding bytes | 32 | + +The header starts with SHA-256 of +`cellar-cell-group-entry-1/cell-history-1/range-route-1/administration-1`, followed +by group history (four 16-byte identities), route epoch (8), birth digest (32), +schema digest (32), proposal ID (16) and command tag (1). The receiving owner +supplies its expected route; mismatched history, epoch, birth or schema refuses. + +Cell addresses contain the cell incarnation (16), ID byte length (2), and exact +UTF-8 ID. Cluster and namespace occur once in the group binding and are inherited +by the cell address. A key is recomputed from its namespace and validated ID. +An encoded range boundary also carries its digest; it must match recomputation. +Ranges have explicit infinity tags and retain the full ID as a collision tie-break. + +An image is epoch/txid/checksum (three 8-byte integers), commitment tag (1), digest +(32), and manifest length (4). Materialized commitments require length zero. +Extent commitments use the extent store's actual root-reference validation. + +File tables begin with a 4-byte count. Each record is five 8-byte integers: +encoded length, minimum txid, maximum txid, page count and decoded bytes. Payload +spans are consecutive and exhaustive; overlapping offsets cannot be represented. +Counts, header/payload limits and accumulated page/decoded-byte limits are checked +before allocation or physical staging. The decoder borrows payload slices from +the original entry buffer. The LTX applier must independently enforce actual +page/decompression limits; envelope declarations are not proof of physical size. + +Fixed-position fields are mandatory and occur once. Unknown tags/domains, extra +fields, trailing bytes, zero identities, overflow, truncated declarations and +noncanonical image commitments refuse. The frame digest detects corruption; it +is not authentication or a quorum certificate. + +## Commands + +| Tag | Command | Required owner behavior beyond decoding | +| --- | --- | --- | +| 1 | Mutation | Check exact current cell history/base image and next sequence before staging; verify every LTX file and publish only a contiguous group prefix. | +| 2 | Enroll | Sequence starts at one. A nonempty image carries its full LTX chain from txid one in the log payload. Verify and sync the initial image before publication; empty enrollment must verify the canonical empty root. | +| 3 | Membership | Bind previous/next membership revision and physical member identities. Use the actual Raft ConfChangeV2 path and membership authorization; never treat it as an ordinary data entry that implicitly changes voters. | +| 4 | Checkpoint | Verify the complete manifest and its referenced image closure at the exact committed position and membership revision. | +| 5 | Seal | Verify the immutable seal authorization and complete parent/child topology union before irrevocably sealing the parent. Child routes must be distinct, compatible and contiguous. | +| 6 | Install | Verify all sealed source cuts and the destination manifest; sync the whole seed closure on each installing replica. | +| 7 | Activate | Verify publication, seed and current-term quorum before admission. | +| 8 | PreparePark | Bind park generation, term and membership; collect durable acknowledgements and retain wake ownership. | +| 9 | Retire | Verify publication and persist the parent tombstone; reclamation still needs archive and pin proofs. | + +A decoded mutation exposes `check_base` against the coherent manifest leaf. This +check refuses another cell/incarnation, a skipped or repeated mutation sequence, +or a different base commitment. It must run before image preparation, and remains +owned by the serial apply path so another completion cannot change the base. + +Enrollment carries bytes rather than only a root reference: majority persistence +of a root alone cannot protect data that exists solely on the proposing host. +Oversized initialization must refuse the configured envelope budget; it cannot +fall back to an unproven local or S3 reference. + +The native contract fixture uses SQLite-produced LTX and verifies replay of an +enrollment from the entry payload alone. It also checks a separately calculated +retirement byte vector, hostile declarations, duplicate identities, every +administrative command, old-domain rejection, and borrowed payload ownership. +These are codec tests, not qualification of a running shared Raft group. diff --git a/crates/cellar/application/consensus/group_protocol/codec.rs b/crates/cellar/application/consensus/group_protocol/codec.rs new file mode 100644 index 000000000..9d026fcc8 --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/codec.rs @@ -0,0 +1,625 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::protocol::{EnvelopeLimits, ImageCommitment, LtxImage, ProtocolError}; +use super::types::*; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; +use sha2::{Digest, Sha256}; + +const MAGIC: &[u8; 8] = b"CLRCGE01"; +const MARKER: u32 = 1; +const PREFIX: usize = 24; +const DIGEST: usize = 32; +const DOMAIN: &[u8] = b"cellar-cell-group-entry-1/cell-history-1/range-route-1/administration-1"; + +impl<'a> CellGroupEntry<'a> { + pub fn encode( + &self, + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result, ProtocolError> { + let payload_len = self.validate(route, limits)?; + let mut header = Output { + bytes: Vec::new(), + limit: limits.envelope.max_header_bytes, + }; + header.put(&Sha256::digest(DOMAIN))?; + header.group(self.binding.group)?; + header.u64(self.binding.route_epoch.get())?; + header.put(&self.binding.birth_digest)?; + header.put(&self.binding.schema_digest)?; + header.put(&self.binding.proposal.bytes())?; + match &self.command { + GroupCommand::Mutation(update) => { + header.tag(1)?; + header.cell(&update.cell)?; + header.u64(update.sequence.get())?; + header.image(update.base)?; + header.image(update.result)?; + header.files(&update.files)?; + } + GroupCommand::Enroll { cell, image, files } => { + header.tag(2)?; + header.cell(cell)?; + header.image(*image)?; + header.files(files)?; + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } => { + header.tag(3)?; + header.u64(previous.get())?; + header.u64(next.get())?; + header.tag(match transition { + GroupMembershipTransition::Auto => 1, + GroupMembershipTransition::JointImplicit => 2, + GroupMembershipTransition::JointExplicit => 3, + GroupMembershipTransition::LeaveJoint => 4, + })?; + header.u32(changes.len() as u32)?; + for change in changes { + header.u64(change.member.get())?; + header.put(&change.node.bytes())?; + header.put(&change.disk.bytes())?; + header.tag(match change.role { + GroupMemberRole::Voter => 1, + GroupMemberRole::Learner => 2, + GroupMemberRole::Remove => 3, + })?; + } + } + GroupCommand::Checkpoint { + position, + manifest, + membership, + } => { + header.tag(4)?; + header.position(*position)?; + header.put(manifest)?; + header.u64(membership.get())?; + } + GroupCommand::Seal { + transition, + authorization, + children, + } => { + header.tag(5)?; + header.put(&transition.bytes())?; + header.put(authorization)?; + header.u32(children.len() as u32)?; + for child in children { + header.route(child)?; + } + } + GroupCommand::Install { + transition, + sources, + manifest, + } => { + header.tag(6)?; + header.put(&transition.bytes())?; + header.put(manifest)?; + header.u32(sources.len() as u32)?; + for source in sources { + header.group(source.group)?; + header.position(source.position)?; + header.put(&source.manifest)?; + } + } + GroupCommand::Activate { + transition, + publication, + manifest, + } => { + header.tag(7)?; + header.put(&transition.bytes())?; + header.put(publication)?; + header.put(manifest)?; + } + GroupCommand::PreparePark { + generation, + term, + membership, + } => { + header.tag(8)?; + header.u64(generation.get())?; + header.u64(term.get())?; + header.u64(membership.get())?; + } + GroupCommand::Retire { + transition, + publication, + } => { + header.tag(9)?; + header.put(&transition.bytes())?; + header.put(publication)?; + } + } + let size = PREFIX + .checked_add(header.bytes.len()) + .and_then(|n| n.checked_add(payload_len)) + .and_then(|n| n.checked_add(DIGEST)) + .ok_or(ProtocolError::Limit)?; + let mut encoded = Vec::with_capacity(size); + encoded.extend_from_slice(MAGIC); + encoded.extend_from_slice(&MARKER.to_be_bytes()); + encoded.extend_from_slice(&(header.bytes.len() as u32).to_be_bytes()); + encoded.extend_from_slice(&(payload_len as u64).to_be_bytes()); + encoded.extend_from_slice(&header.bytes); + let files = match &self.command { + GroupCommand::Mutation(update) => update.files.as_slice(), + GroupCommand::Enroll { files, .. } => files.as_slice(), + _ => &[], + }; + for file in files { + encoded.extend_from_slice(file.bytes); + } + let digest = Sha256::digest(&encoded); + encoded.extend_from_slice(&digest); + Ok(encoded) + } + + /// No payload bytes are copied. Bounds, domain and the expected group route + /// are checked before decoding variable-length cell/file metadata. The + /// caller still checks the coherent cell base and all physical LTX content. + pub fn decode( + bytes: &'a [u8], + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result { + limits.validate()?; + let mut prefix = Input(bytes); + if prefix.take(8)? != MAGIC { + return Err(ProtocolError::Protocol); + } + if prefix.u32()? != MARKER { + return Err(ProtocolError::Protocol); + } + let header_len = prefix.u32()? as usize; + let payload_len = prefix.u64()?; + if header_len > limits.envelope.max_header_bytes + || payload_len > limits.envelope.max_payload_bytes as u64 + { + return Err(ProtocolError::Limit); + } + let payload_len = usize::try_from(payload_len).map_err(|_| ProtocolError::Limit)?; + let payload_start = PREFIX.checked_add(header_len).ok_or(ProtocolError::Limit)?; + let end = payload_start + .checked_add(payload_len) + .ok_or(ProtocolError::Limit)?; + if end.checked_add(DIGEST) != Some(bytes.len()) { + return Err(ProtocolError::Framing); + } + if Sha256::digest(&bytes[..end])[..] != bytes[end..] { + return Err(ProtocolError::Digest); + } + let mut input = Input(&bytes[PREFIX..payload_start]); + let mut payload = Input(&bytes[payload_start..end]); + if input.take(DIGEST)? != &Sha256::digest(DOMAIN)[..] { + return Err(ProtocolError::Protocol); + } + let binding = GroupEntryBinding { + group: input.group()?, + route_epoch: input.counter(RouteEpoch::new)?, + birth_digest: input.array()?, + schema_digest: input.array()?, + proposal: input.id(ProposalId::new)?, + }; + binding.check(route)?; + let command = match input.tag()? { + 1 => { + let cell = input.cell(binding.group)?; + check_cell(route, &cell)?; + let sequence = input.counter(MutationSequence::new)?; + let base = input.image()?; + let result = input.image()?; + let files = input.files(&mut payload, limits.envelope)?; + GroupCommand::Mutation(GroupMutation { + cell, + sequence, + base, + result, + files, + }) + } + 2 => { + let cell = input.cell(binding.group)?; + check_cell(route, &cell)?; + GroupCommand::Enroll { + cell, + image: input.image()?, + files: input.files(&mut payload, limits.envelope)?, + } + } + 3 => { + let previous = input.counter(MembershipRevision::new)?; + let next = input.counter(MembershipRevision::new)?; + let transition = match input.tag()? { + 1 => GroupMembershipTransition::Auto, + 2 => GroupMembershipTransition::JointImplicit, + 3 => GroupMembershipTransition::JointExplicit, + 4 => GroupMembershipTransition::LeaveJoint, + _ => return Err(ProtocolError::Protocol), + }; + let count = input.count(limits.max_members, 41)?; + let mut changes = Vec::with_capacity(count); + for _ in 0..count { + let member = input.counter(MemberId::new)?; + let node = input.id(NodeId::new)?; + let disk = input.id(DiskLineage::new)?; + let role = match input.tag()? { + 1 => GroupMemberRole::Voter, + 2 => GroupMemberRole::Learner, + 3 => GroupMemberRole::Remove, + _ => return Err(ProtocolError::Protocol), + }; + changes.push(GroupMemberChange { + member, + node, + disk, + role, + }); + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } + } + 4 => GroupCommand::Checkpoint { + position: input.position()?, + manifest: input.array()?, + membership: input.counter(MembershipRevision::new)?, + }, + 5 => { + let transition = input.id(TransitionId::new)?; + let authorization = input.array()?; + let count = input.count(limits.max_transition_groups, 138)?; + let mut children = Vec::with_capacity(count); + for _ in 0..count { + children.push(input.route()?); + } + GroupCommand::Seal { + transition, + authorization, + children, + } + } + 6 => { + let transition = input.id(TransitionId::new)?; + let manifest = input.array()?; + let count = input.count(limits.max_transition_groups, 112)?; + let mut sources = Vec::with_capacity(count); + for _ in 0..count { + sources.push(GroupSeal { + group: input.group()?, + position: input.position()?, + manifest: input.array()?, + }); + } + GroupCommand::Install { + transition, + sources, + manifest, + } + } + 7 => GroupCommand::Activate { + transition: input.id(TransitionId::new)?, + publication: input.array()?, + manifest: input.array()?, + }, + 8 => GroupCommand::PreparePark { + generation: input.counter(ParkGeneration::new)?, + term: Term::new(input.u64()?), + membership: input.counter(MembershipRevision::new)?, + }, + 9 => GroupCommand::Retire { + transition: input.id(TransitionId::new)?, + publication: input.array()?, + }, + _ => return Err(ProtocolError::Protocol), + }; + if !input.0.is_empty() || !payload.0.is_empty() { + return Err(ProtocolError::Framing); + } + let entry = Self { binding, command }; + entry.validate(route, limits)?; + Ok(entry) + } +} + +struct Output { + bytes: Vec, + limit: usize, +} + +impl Output { + fn put(&mut self, value: &[u8]) -> Result<(), ProtocolError> { + if self + .bytes + .len() + .checked_add(value.len()) + .is_none_or(|n| n > self.limit) + { + return Err(ProtocolError::Limit); + } + self.bytes.extend_from_slice(value); + Ok(()) + } + + fn tag(&mut self, value: u8) -> Result<(), ProtocolError> { + self.put(&[value]) + } + + fn u32(&mut self, value: u32) -> Result<(), ProtocolError> { + self.put(&value.to_be_bytes()) + } + + fn u64(&mut self, value: u64) -> Result<(), ProtocolError> { + self.put(&value.to_be_bytes()) + } + + fn group(&mut self, value: GroupHistory) -> Result<(), ProtocolError> { + for id in [ + value.cluster.bytes(), + value.namespace.bytes(), + value.group.bytes(), + value.incarnation.bytes(), + ] { + self.put(&id)?; + } + Ok(()) + } + + fn text(&mut self, value: &CellId) -> Result<(), ProtocolError> { + self.put(&(value.as_str().len() as u16).to_be_bytes())?; + self.put(value.as_str().as_bytes()) + } + + fn cell(&mut self, value: &CellHistory) -> Result<(), ProtocolError> { + self.put(&value.incarnation.bytes())?; + self.text(&value.cell) + } + + fn image(&mut self, value: LtxImage) -> Result<(), ProtocolError> { + for n in [value.epoch, value.txid, value.checksum] { + self.u64(n)?; + } + match value.commitment { + ImageCommitment::Sha256(digest) => { + self.tag(1)?; + self.put(&digest)?; + self.u32(0) + } + ImageCommitment::Extents(root) => { + self.tag(2)?; + self.put(&root.root().bytes())?; + self.u32(root.encoded_length()) + } + } + } + + fn files(&mut self, files: &[GroupLtx<'_>]) -> Result<(), ProtocolError> { + self.u32(files.len() as u32)?; + for file in files { + for value in [ + file.bytes.len() as u64, + file.min_txid, + file.max_txid, + file.page_count, + file.decoded_bytes, + ] { + self.u64(value)?; + } + } + Ok(()) + } + + fn position(&mut self, value: LogPosition) -> Result<(), ProtocolError> { + self.u64(value.index.get())?; + self.u64(value.term.get()) + } + + fn bound(&mut self, value: &CellBound) -> Result<(), ProtocolError> { + match value { + CellBound::NegativeInfinity => self.tag(0), + CellBound::Key(key) => { + self.tag(1)?; + self.put(&key.digest())?; + self.text(key.cell()) + } + CellBound::PositiveInfinity => self.tag(2), + } + } + + fn route(&mut self, value: &GroupRoute) -> Result<(), ProtocolError> { + self.group(value.history)?; + self.u64(value.epoch.get())?; + self.put(&value.birth_digest)?; + self.put(&value.schema_digest)?; + self.bound(value.range.lower())?; + self.bound(value.range.upper()) + } +} + +struct Input<'a>(&'a [u8]); + +impl<'a> Input<'a> { + fn take(&mut self, n: usize) -> Result<&'a [u8], ProtocolError> { + let (value, rest) = self.0.split_at_checked(n).ok_or(ProtocolError::Framing)?; + self.0 = rest; + Ok(value) + } + + fn array(&mut self) -> Result<[u8; N], ProtocolError> { + self.take(N)?.try_into().map_err(|_| ProtocolError::Framing) + } + + fn tag(&mut self) -> Result { + Ok(self.array::<1>()?[0]) + } + + fn u32(&mut self) -> Result { + Ok(u32::from_be_bytes(self.array()?)) + } + + fn u64(&mut self) -> Result { + Ok(u64::from_be_bytes(self.array()?)) + } + + fn id( + &mut self, + make: impl FnOnce([u8; 16]) -> Result, + ) -> Result { + make(self.array()?).map_err(|_| ProtocolError::Identity) + } + + fn counter( + &mut self, + make: impl FnOnce(u64) -> Result, + ) -> Result { + make(self.u64()?).map_err(|_| ProtocolError::Identity) + } + + fn count(&mut self, maximum: usize, minimum_bytes: usize) -> Result { + let count = self.u32()? as usize; + if count > maximum { + return Err(ProtocolError::Limit); + } + if count + .checked_mul(minimum_bytes) + .is_none_or(|n| n > self.0.len()) + { + return Err(ProtocolError::Framing); + } + Ok(count) + } + + fn group(&mut self) -> Result { + Ok(GroupHistory { + cluster: self.id(ClusterId::new)?, + namespace: self.id(NamespaceId::new)?, + group: self.id(GroupId::new)?, + incarnation: self.id(IncarnationId::new)?, + }) + } + + fn text(&mut self) -> Result { + let n = u16::from_be_bytes(self.array()?) as usize; + if n > MAX_CELL_ID_BYTES { + return Err(ProtocolError::Limit); + } + CellId::new(std::str::from_utf8(self.take(n)?).map_err(|_| ProtocolError::Identity)?) + .map_err(|_| ProtocolError::Identity) + } + + fn cell(&mut self, group: GroupHistory) -> Result { + let incarnation = self.id(IncarnationId::new)?; + Ok(CellHistory { + cluster: group.cluster, + namespace: group.namespace, + incarnation, + cell: self.text()?, + }) + } + + fn image(&mut self) -> Result { + let epoch = self.u64()?; + let txid = self.u64()?; + let checksum = self.u64()?; + let tag = self.tag()?; + let digest = self.array()?; + let length = self.u32()?; + let commitment = match (tag, length) { + (1, 0) => ImageCommitment::Sha256(digest), + (2, _) => ImageCommitment::Extents( + ExtentRootReference::new(ImageRoot::from_bytes(digest), length) + .map_err(|_| ProtocolError::Identity)?, + ), + _ => return Err(ProtocolError::Protocol), + }; + let image = LtxImage { + epoch, + txid, + checksum, + commitment, + }; + LtxImage::try_from(CellImage::from(image))?; + Ok(image) + } + + fn files( + &mut self, + payload: &mut Input<'a>, + limits: EnvelopeLimits, + ) -> Result>, ProtocolError> { + let count = self.count(limits.max_files, 40)?; + let mut files = Vec::with_capacity(count); + let (mut decoded, mut pages) = (0_u64, 0_u64); + for _ in 0..count { + let length = self.u64()?; + if length == 0 || length > limits.max_payload_bytes as u64 { + return Err(ProtocolError::Limit); + } + let min_txid = self.u64()?; + let max_txid = self.u64()?; + let page_count = self.u64()?; + let decoded_bytes = self.u64()?; + pages = pages.checked_add(page_count).ok_or(ProtocolError::Limit)?; + decoded = decoded + .checked_add(decoded_bytes) + .ok_or(ProtocolError::Limit)?; + if pages > limits.max_pages || decoded > limits.max_decoded_bytes { + return Err(ProtocolError::Limit); + } + let length = usize::try_from(length).map_err(|_| ProtocolError::Limit)?; + files.push(GroupLtx { + min_txid, + max_txid, + page_count, + decoded_bytes, + bytes: payload.take(length)?, + }); + } + Ok(files) + } + + fn position(&mut self) -> Result { + Ok(LogPosition { + index: LogIndex::new(self.u64()?), + term: Term::new(self.u64()?), + }) + } + + fn bound(&mut self) -> Result { + match self.tag()? { + 0 => Ok(CellBound::NegativeInfinity), + 1 => { + let digest = self.array()?; + Ok(CellBound::Key(CellKey::from_digest(digest, self.text()?))) + } + 2 => Ok(CellBound::PositiveInfinity), + _ => Err(ProtocolError::Protocol), + } + } + + fn route(&mut self) -> Result { + let history = self.group()?; + let epoch = self.counter(RouteEpoch::new)?; + let birth_digest = self.array()?; + let schema_digest = self.array()?; + let range = + CellRange::new(self.bound()?, self.bound()?).map_err(|_| ProtocolError::Identity)?; + Ok(GroupRoute { + history, + epoch, + birth_digest, + schema_digest, + range, + }) + } +} diff --git a/crates/cellar/application/consensus/group_protocol/mod.rs b/crates/cellar/application/consensus/group_protocol/mod.rs new file mode 100644 index 000000000..6f5171421 --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/mod.rs @@ -0,0 +1,9 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-addressed entries for one physical group. Decoding verifies the wire +//! binding and borrows payload bytes; it does not grant quorum or image proofs. + +mod codec; +mod types; + +pub use types::*; diff --git a/crates/cellar/application/consensus/group_protocol/types.rs b/crates/cellar/application/consensus/group_protocol/types.rs new file mode 100644 index 000000000..f19df02fb --- /dev/null +++ b/crates/cellar/application/consensus/group_protocol/types.rs @@ -0,0 +1,369 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::protocol::{ImageCommitment, LtxImage, ProtocolError}; +use super::super::{group_contracts::cell_routing_key, protocol::EnvelopeLimits}; +use cellar_logic::{ + cell_group::*, + consensus::{DiskLineage, LogPosition, MemberId, NodeId, ProposalId, Term}, +}; +use std::collections::BTreeSet; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupEntryLimits { + pub envelope: EnvelopeLimits, + pub max_members: usize, + pub max_transition_groups: usize, +} + +impl Default for GroupEntryLimits { + fn default() -> Self { + Self { + envelope: EnvelopeLimits::default(), + max_members: 64, + max_transition_groups: 64, + } + } +} + +impl GroupEntryLimits { + pub fn validate(self) -> Result<(), ProtocolError> { + self.envelope.validate()?; + if !(1..=64).contains(&self.max_members) || !(1..=64).contains(&self.max_transition_groups) + { + return Err(ProtocolError::Limit); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupEntryBinding { + pub group: GroupHistory, + pub route_epoch: RouteEpoch, + pub birth_digest: [u8; 32], + pub schema_digest: [u8; 32], + pub proposal: ProposalId, +} + +impl GroupEntryBinding { + pub fn check(&self, route: &GroupRoute) -> Result<(), ProtocolError> { + if self.group != route.history + || self.route_epoch != route.epoch + || self.birth_digest != route.birth_digest + || self.schema_digest != route.schema_digest + { + return Err(ProtocolError::Identity); + } + check_range_keys(route)?; + Ok(()) + } +} + +/// The entry's original encoded buffer owns these bytes until staging finishes. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupLtx<'a> { + pub min_txid: u64, + pub max_txid: u64, + pub page_count: u64, + pub decoded_bytes: u64, + pub bytes: &'a [u8], +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupMutation<'a> { + pub cell: CellHistory, + pub sequence: MutationSequence, + pub base: LtxImage, + pub result: LtxImage, + pub files: Vec>, +} + +impl GroupMutation<'_> { + /// Call against the coherent manifest leaf before allocating image work. + /// Valid framing or another cell's progress cannot establish this base. + pub fn check_base(&self, current: &CellVersion) -> Result<(), ProtocolError> { + if self.cell != current.history + || current.sequence.checked_next().ok() != Some(self.sequence) + || CellImage::from(self.base) != current.image + { + return Err(ProtocolError::Chain); + } + Ok(()) + } + + pub(super) fn validate(&self, limits: EnvelopeLimits) -> Result { + LtxImage::try_from(CellImage::from(self.base))?; + LtxImage::try_from(CellImage::from(self.result))?; + if self.sequence.get() <= 1 + || self.base.epoch != self.result.epoch + || self.result.txid <= self.base.txid + || !matches!( + (self.base.commitment, self.result.commitment), + (ImageCommitment::Sha256(_), ImageCommitment::Sha256(_)) + | (ImageCommitment::Extents(_), ImageCommitment::Extents(_)) + ) + { + return Err(ProtocolError::Chain); + } + validate_files(&self.files, limits, self.base.txid, self.result.txid) + } +} + +pub(super) fn validate_files( + files: &[GroupLtx<'_>], + limits: EnvelopeLimits, + base: u64, + result: u64, +) -> Result { + if files.len() > limits.max_files || (files.is_empty() && (base != 0 || result != 0)) { + return Err(ProtocolError::Limit); + } + let (mut txid, mut payload, mut decoded, mut pages) = (base, 0_usize, 0_u64, 0_u64); + for file in files { + if txid.checked_add(1) != Some(file.min_txid) + || file.max_txid < file.min_txid + || file.bytes.is_empty() + { + return Err(ProtocolError::Chain); + } + txid = file.max_txid; + payload = payload + .checked_add(file.bytes.len()) + .ok_or(ProtocolError::Limit)?; + decoded = decoded + .checked_add(file.decoded_bytes) + .ok_or(ProtocolError::Limit)?; + pages = pages + .checked_add(file.page_count) + .ok_or(ProtocolError::Limit)?; + if payload > limits.max_payload_bytes + || decoded > limits.max_decoded_bytes + || pages > limits.max_pages + { + return Err(ProtocolError::Limit); + } + } + if txid != result { + return Err(ProtocolError::Chain); + } + Ok(payload) +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GroupMemberRole { + Voter, + Learner, + Remove, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupMemberChange { + pub member: MemberId, + pub node: NodeId, + pub disk: DiskLineage, + pub role: GroupMemberRole, +} + +/// Matches the raft-rs configuration transition semantics. The owning runtime +/// must encode this command as the actual ConfChangeV2 entry, not a data no-op. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum GroupMembershipTransition { + Auto, + JointImplicit, + JointExplicit, + LeaveJoint, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GroupSeal { + pub group: GroupHistory, + pub position: LogPosition, + pub manifest: [u8; 32], +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum GroupCommand<'a> { + Mutation(GroupMutation<'a>), + /// Enrollment starts at sequence one. A nonempty image carries a complete + /// LTX chain starting at txid one, so majority log persistence protects its + /// bytes. Publication still requires independent image verification/sync. + Enroll { + cell: CellHistory, + image: LtxImage, + files: Vec>, + }, + Membership { + previous: MembershipRevision, + next: MembershipRevision, + transition: GroupMembershipTransition, + changes: Vec, + }, + Checkpoint { + position: LogPosition, + manifest: [u8; 32], + membership: MembershipRevision, + }, + Seal { + transition: TransitionId, + authorization: [u8; 32], + children: Vec, + }, + Install { + transition: TransitionId, + sources: Vec, + manifest: [u8; 32], + }, + Activate { + transition: TransitionId, + publication: [u8; 32], + manifest: [u8; 32], + }, + PreparePark { + generation: ParkGeneration, + term: Term, + membership: MembershipRevision, + }, + Retire { + transition: TransitionId, + publication: [u8; 32], + }, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellGroupEntry<'a> { + pub binding: GroupEntryBinding, + pub command: GroupCommand<'a>, +} + +impl CellGroupEntry<'_> { + pub(super) fn validate( + &self, + route: &GroupRoute, + limits: GroupEntryLimits, + ) -> Result { + limits.validate()?; + self.binding.check(route)?; + match &self.command { + GroupCommand::Mutation(update) => { + check_cell(route, &update.cell)?; + return update.validate(limits.envelope); + } + GroupCommand::Enroll { cell, image, files } => { + check_cell(route, cell)?; + LtxImage::try_from(CellImage::from(*image))?; + if image.txid == 0 && image.checksum != 0 { + return Err(ProtocolError::Chain); + } + return validate_files(files, limits.envelope, 0, image.txid); + } + GroupCommand::Membership { + previous, + next, + transition, + changes, + } => { + if previous.checked_next().ok() != Some(*next) { + return Err(ProtocolError::Chain); + } + if changes.len() > limits.max_members { + return Err(ProtocolError::Limit); + } + if (*transition == GroupMembershipTransition::LeaveJoint) != changes.is_empty() { + return Err(ProtocolError::Header); + } + let mut members = BTreeSet::new(); + for change in changes { + if !members.insert(change.member) { + return Err(ProtocolError::Identity); + } + } + } + GroupCommand::Checkpoint { position, .. } => check_position(*position)?, + GroupCommand::Seal { children, .. } => { + if children.is_empty() || children.len() > limits.max_transition_groups { + return Err(ProtocolError::Limit); + } + let mut groups = BTreeSet::new(); + let mut last: Option<&GroupRoute> = None; + for child in children { + check_related(self.binding.group, child.history)?; + if child.schema_digest != route.schema_digest { + return Err(ProtocolError::Identity); + } + if !groups.insert(child.history.group) { + return Err(ProtocolError::Identity); + } + if let Some(previous) = last { + if previous.range.upper() != child.range.lower() { + return Err(ProtocolError::Chain); + } + } + check_range_keys(child)?; + last = Some(child); + } + // A merge child may include neighboring parent ranges. The + // transition owner verifies the complete parent/child union. + if children[0].range.lower() > route.range.lower() + || children[children.len() - 1].range.upper() < route.range.upper() + { + return Err(ProtocolError::Chain); + } + } + GroupCommand::Install { sources, .. } => { + if sources.is_empty() || sources.len() > limits.max_transition_groups { + return Err(ProtocolError::Limit); + } + let mut groups = BTreeSet::new(); + for source in sources { + check_related(self.binding.group, source.group)?; + if !groups.insert(source.group.group) { + return Err(ProtocolError::Identity); + } + check_position(source.position)?; + } + } + GroupCommand::PreparePark { term, .. } if term.get() == 0 => { + return Err(ProtocolError::Identity) + } + GroupCommand::Activate { .. } + | GroupCommand::PreparePark { .. } + | GroupCommand::Retire { .. } => {} + } + Ok(0) + } +} + +pub(super) fn check_cell(route: &GroupRoute, cell: &CellHistory) -> Result<(), ProtocolError> { + route + .check_cell(cell, &cell_routing_key(cell.namespace, &cell.cell)) + .map_err(|_| ProtocolError::Identity) +} + +pub(super) fn check_range_keys(route: &GroupRoute) -> Result<(), ProtocolError> { + for bound in [route.range.lower(), route.range.upper()] { + if let CellBound::Key(key) = bound { + if &cell_routing_key(route.history.namespace, key.cell()) != key { + return Err(ProtocolError::Identity); + } + } + } + Ok(()) +} + +fn check_related(owner: GroupHistory, other: GroupHistory) -> Result<(), ProtocolError> { + if owner.cluster != other.cluster + || owner.namespace != other.namespace + || owner.group == other.group + { + return Err(ProtocolError::Identity); + } + Ok(()) +} + +fn check_position(position: LogPosition) -> Result<(), ProtocolError> { + position.validate().map_err(|_| ProtocolError::Chain)?; + if position.index.get() == 0 { + return Err(ProtocolError::Chain); + } + Ok(()) +} diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index c7e48f9b7..96522ea8e 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -11,6 +11,7 @@ pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; pub(crate) mod client; mod group_contracts; +pub(crate) mod group_protocol; mod performance; pub use performance::{ RaftDurationBucket, RaftOperationPerformance, RaftPerformanceCounter, RaftPerformanceSnapshot, @@ -210,11 +211,12 @@ pub mod verification { /// Contracts under integration into the replacement group runtime. pub mod cell_groups { pub use super::super::group_contracts::cell_routing_key; + pub use super::super::group_protocol::*; pub use crate::raft::version::{CellVersionObservation, CellVersionToken}; pub use cellar_logic::cell_group::{ CellBound, CellHistory, CellId, CellImage, CellImageCommitment, CellKey, CellRange, CellVersion, CommitWitness, GroupHistory, GroupRoute, MembershipRevision, - MutationSequence, RouteEpoch, + MutationSequence, ParkGeneration, RouteEpoch, TransitionId, }; } } diff --git a/crates/cellar/tests/native_cell_groups_contracts.rs b/crates/cellar/tests/native_cell_groups_contracts.rs index 71188d5e1..61e6c4bf6 100644 --- a/crates/cellar/tests/native_cell_groups_contracts.rs +++ b/crates/cellar/tests/native_cell_groups_contracts.rs @@ -203,3 +203,544 @@ fn identity_and_hash_boundaries() { .checked_next() .is_err()); } + +#[allow(dead_code)] +#[path = "support/raft_ltx_fixture.rs"] +mod image_fixture; + +fn group_route() -> GroupRoute { + GroupRoute { + history: observation().witness.group, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [22; 32], + schema_digest: [42; 32], + } +} + +fn binding(route: &GroupRoute) -> GroupEntryBinding { + GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([6; 16]).unwrap(), + } +} + +fn ltx_cut() -> image_fixture::Cut { + let before = image_fixture::image( + None, + "CREATE TABLE value(n INTEGER); INSERT INTO value VALUES (1);", + ) + .unwrap(); + let after = image_fixture::image(Some(&before), "UPDATE value SET n=2;").unwrap(); + image_fixture::Cut::new(&before, &after, 1, true, true) +} + +fn image_value(image: cellar_ltx::apply::ImageState) -> cellar::application::consensus::LtxImage { + cellar::application::consensus::LtxImage { + epoch: image.epoch, + txid: image.txid.0, + checksum: image.checksum, + commitment: image.digest.into(), + } +} + +fn mutation_entry(cut: &image_fixture::Cut) -> CellGroupEntry<'_> { + CellGroupEntry { + binding: binding(&group_route()), + command: GroupCommand::Mutation(GroupMutation { + cell: observation().version.history, + sequence: MutationSequence::new(2).unwrap(), + base: image_value(cut.base), + result: image_value(cut.result), + files: vec![GroupLtx { + min_txid: cut.base.txid.0 + 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }), + } +} + +fn rehash(bytes: &mut [u8]) { + use sha2::{Digest, Sha256}; + let end = bytes.len() - 32; + let hash = Sha256::digest(&bytes[..end]); + bytes[end..].copy_from_slice(&hash); +} + +#[test] +fn envelope_rejects_wrong_cell_and_domain() { + use cellar::application::consensus::{ + CellUpdate, EnvelopeLimits, GroupScope, LtxUpdate, ProtocolError, + }; + let cut = ltx_cut(); + assert!(cellar_ltx::ltx::decode_file(&cut.bytes).is_ok()); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let entry = mutation_entry(&cut); + let bytes = entry.encode(&route, limits).unwrap(); + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + assert_eq!(decoded, entry); + let GroupCommand::Mutation(update) = &decoded.command else { + panic!("mutation changed kind") + }; + let current = CellVersion { + history: observation().version.history, + sequence: MutationSequence::new(1).unwrap(), + image: image_value(cut.base).into(), + }; + assert!(update.check_base(¤t).is_ok()); + for field in 0..6 { + let mut other = current.clone(); + match field { + 0 => other.history.cell = CellId::new("beta").unwrap(), + 1 => other.history.incarnation = IncarnationId::new([90; 16]).unwrap(), + 2 => other.history.namespace = NamespaceId::new([90; 16]).unwrap(), + 3 => other.sequence = MutationSequence::new(2).unwrap(), + 4 => other.image.checksum ^= 1, + _ => other.image.commitment = CellImageCommitment::Materialized([90; 32]), + } + assert_eq!(update.check_base(&other), Err(ProtocolError::Chain)); + } + for field in 0..7 { + let mut other = route.clone(); + match field { + 0 => other.history.cluster = ClusterId::new([90; 16]).unwrap(), + 1 => other.history.namespace = NamespaceId::new([90; 16]).unwrap(), + 2 => other.history.group = GroupId::new([90; 16]).unwrap(), + 3 => other.history.incarnation = IncarnationId::new([90; 16]).unwrap(), + 4 => other.epoch = RouteEpoch::new(2).unwrap(), + 5 => other.schema_digest = [90; 32], + _ => other.birth_digest = [90; 32], + } + assert_eq!( + CellGroupEntry::decode(&bytes, &other, limits), + Err(ProtocolError::Identity) + ); + } + let mut outside = route.clone(); + outside.range = CellRange::all() + .split(cell_routing_key( + route.history.namespace, + ¤t.history.cell, + )) + .unwrap() + .0; + assert_eq!( + CellGroupEntry::decode(&bytes, &outside, limits), + Err(ProtocolError::Identity) + ); + let old = CellUpdate { + scope: GroupScope { + cluster: route.history.cluster, + namespace: route.history.namespace, + group: route.history.group, + incarnation: route.history.incarnation, + cell: current.history.cell.as_str().into(), + }, + application_digest: route.schema_digest, + proposal: entry.binding.proposal, + base: image_value(cut.base), + result: image_value(cut.result), + files: vec![LtxUpdate { + min_txid: 2, + max_txid: 2, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: cut.bytes.clone(), + }], + } + .encode(EnvelopeLimits::default()) + .unwrap(); + assert_eq!( + CellGroupEntry::decode(&old, &route, limits), + Err(ProtocolError::Protocol) + ); + assert!(CellUpdate::decode(&bytes, EnvelopeLimits::default()).is_err()); + let mut unknown = bytes.clone(); + unknown[24] ^= 1; // New-format framing with an unrecognized semantic domain. + rehash(&mut unknown); + assert_eq!( + CellGroupEntry::decode(&unknown, &route, limits), + Err(ProtocolError::Protocol) + ); +} + +#[test] +fn codec_limits_before_allocation() { + use cellar::application::consensus::ProtocolError; + let cut = ltx_cut(); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let bytes = mutation_entry(&cut).encode(&route, limits).unwrap(); + let header_len = u32::from_be_bytes(bytes[12..16].try_into().unwrap()) as usize; + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + let GroupCommand::Mutation(update) = decoded.command else { + panic!("not a mutation") + }; + assert_eq!( + update.files[0].bytes.as_ptr(), + bytes[24 + header_len..].as_ptr() + ); + for end in 0..bytes.len() { + assert!( + CellGroupEntry::decode(&bytes[..end], &route, limits).is_err(), + "truncation {end}" + ); + } + for field in 0..5 { + let mut bounded = limits; + match field { + 0 => bounded.envelope.max_header_bytes = 1, + 1 => bounded.envelope.max_payload_bytes = 1, + 2 => bounded.envelope.max_decoded_bytes = 1, + 3 => bounded.envelope.max_pages = 1, + _ => bounded.envelope.max_files = 0, + } + assert_eq!( + CellGroupEntry::decode(&bytes, &route, bounded), + Err(ProtocolError::Limit) + ); + } + let file_count = 24 + 185 + 16 + 2 + "alpha".len() + 8 + 2 * 61; + let malformed_fields = [ + (12, u32::MAX.to_be_bytes().to_vec()), + (16, u64::MAX.to_be_bytes().to_vec()), + (file_count, u32::MAX.to_be_bytes().to_vec()), + (file_count + 4, u64::MAX.to_be_bytes().to_vec()), + (file_count + 4 + 24, u64::MAX.to_be_bytes().to_vec()), + (file_count + 4 + 32, u64::MAX.to_be_bytes().to_vec()), + ]; + for (offset, value) in malformed_fields { + let mut oversized = bytes.clone(); + oversized[offset..offset + value.len()].copy_from_slice(&value); + rehash(&mut oversized); + assert_eq!( + CellGroupEntry::decode(&oversized, &route, limits), + Err(ProtocolError::Limit), + "field {offset}" + ); + } + let mut unknown = bytes.clone(); + unknown[24 + 184] = 255; + rehash(&mut unknown); + assert_eq!( + CellGroupEntry::decode(&unknown, &route, limits), + Err(ProtocolError::Protocol) + ); + let mut marker = bytes.clone(); + marker[8..12].copy_from_slice(&2_u32.to_be_bytes()); + rehash(&mut marker); + assert_eq!( + CellGroupEntry::decode(&marker, &route, limits), + Err(ProtocolError::Protocol) + ); + // Fixed-position fields occur exactly once. An appended duplicate group + // identity cannot be reinterpreted as an override or optional metadata. + let mut duplicate = bytes.clone(); + duplicate.splice( + 24 + header_len..24 + header_len, + bytes[56..120].iter().copied(), + ); + duplicate[12..16].copy_from_slice(&((header_len + 64) as u32).to_be_bytes()); + rehash(&mut duplicate); + assert_eq!( + CellGroupEntry::decode(&duplicate, &route, limits), + Err(ProtocolError::Framing) + ); + let mut sequence = bytes.clone(); + let at = 24 + 185 + 16 + 2 + "alpha".len(); + sequence[at..at + 8].copy_from_slice(&1_u64.to_be_bytes()); + rehash(&mut sequence); + assert_eq!( + CellGroupEntry::decode(&sequence, &route, limits), + Err(ProtocolError::Chain) + ); + let mut corrupt = bytes.clone(); + corrupt[24 + header_len] ^= 1; + assert_eq!( + CellGroupEntry::decode(&corrupt, &route, limits), + Err(ProtocolError::Digest) + ); +} + +#[test] +fn group_administration_roundtrips_and_golden() { + use cellar::application::consensus::{DiskLineage, MemberId, NodeId, ProtocolError}; + let route = group_route(); + let limits = GroupEntryLimits::default(); + let transition = TransitionId::new([77; 16]).unwrap(); + let initial = image_fixture::image(None, "CREATE TABLE seed(n INTEGER);").unwrap(); + let cut = image_fixture::Cut::new(&[], &initial, 0, true, true); + let (left, right) = CellRange::all() + .split(cell_routing_key( + route.history.namespace, + &CellId::new("alpha").unwrap(), + )) + .unwrap(); + let children: Vec<_> = [left, right] + .into_iter() + .enumerate() + .map(|(i, range)| { + let mut child = route.clone(); + child.history.group = GroupId::new([40 + i as u8; 16]).unwrap(); + child.history.incarnation = IncarnationId::new([50 + i as u8; 16]).unwrap(); + child.birth_digest = [60 + i as u8; 32]; + child.range = range; + child + }) + .collect(); + let seal = GroupCommand::Seal { + transition, + authorization: [66; 32], + children: children.clone(), + }; + let membership = GroupCommand::Membership { + previous: MembershipRevision::new(1).unwrap(), + next: MembershipRevision::new(2).unwrap(), + transition: GroupMembershipTransition::JointExplicit, + changes: vec![GroupMemberChange { + member: MemberId::new(1).unwrap(), + node: NodeId::new([7; 16]).unwrap(), + disk: DiskLineage::new([8; 16]).unwrap(), + role: GroupMemberRole::Voter, + }], + }; + let commands = vec![ + GroupCommand::Enroll { + cell: observation().version.history, + image: image_value(cut.result), + files: vec![GroupLtx { + min_txid: 1, + max_txid: 1, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }, + membership.clone(), + GroupCommand::Membership { + previous: MembershipRevision::new(2).unwrap(), + next: MembershipRevision::new(3).unwrap(), + transition: GroupMembershipTransition::LeaveJoint, + changes: vec![], + }, + GroupCommand::Checkpoint { + position: observation().witness.position, + manifest: [25; 32], + membership: MembershipRevision::new(1).unwrap(), + }, + seal.clone(), + GroupCommand::Install { + transition, + sources: children + .iter() + .map(|child| GroupSeal { + group: child.history, + position: observation().witness.position, + manifest: [26; 32], + }) + .collect(), + manifest: [27; 32], + }, + GroupCommand::Activate { + transition, + publication: [88; 32], + manifest: [27; 32], + }, + GroupCommand::PreparePark { + generation: ParkGeneration::new(1).unwrap(), + term: Term::new(9), + membership: MembershipRevision::new(1).unwrap(), + }, + GroupCommand::Retire { + transition, + publication: [88; 32], + }, + ]; + for command in commands { + let entry = CellGroupEntry { + binding: binding(&route), + command, + }; + let bytes = entry.encode(&route, limits).unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, limits).unwrap(), + entry + ); + assert_eq!(&bytes[8..12], &1_u32.to_be_bytes()); + if matches!(entry.command, GroupCommand::Retire { .. }) { + assert_eq!(bytes.len(), 289); + assert_eq!( + hex::encode(&bytes[bytes.len() - 32..]), + "77aa30713c91abfafa483e9709b1680271a04873894bbd60bd9374114ed24000" + ); + } + } + let mut repeated = membership; + for transition in [ + GroupMembershipTransition::Auto, + GroupMembershipTransition::JointImplicit, + GroupMembershipTransition::JointExplicit, + ] { + for role in [ + GroupMemberRole::Voter, + GroupMemberRole::Learner, + GroupMemberRole::Remove, + ] { + let mut command = repeated.clone(); + if let GroupCommand::Membership { + transition: mode, + changes, + .. + } = &mut command + { + *mode = transition; + changes[0].role = role; + } + let entry = CellGroupEntry { + binding: binding(&route), + command, + }; + let bytes = entry.encode(&route, limits).unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, limits).unwrap(), + entry + ); + } + } + let mut two_members = repeated.clone(); + if let GroupCommand::Membership { changes, .. } = &mut two_members { + changes.push(GroupMemberChange { + member: MemberId::new(2).unwrap(), + node: NodeId::new([9; 16]).unwrap(), + disk: DiskLineage::new([10; 16]).unwrap(), + role: GroupMemberRole::Learner, + }); + } + let mut duplicated_member = CellGroupEntry { + binding: binding(&route), + command: two_members, + } + .encode(&route, limits) + .unwrap(); + let second_member = 24 + 185 + 8 + 8 + 1 + 4 + 41; + duplicated_member[second_member..second_member + 8].copy_from_slice(&1_u64.to_be_bytes()); + rehash(&mut duplicated_member); + assert_eq!( + CellGroupEntry::decode(&duplicated_member, &route, limits), + Err(ProtocolError::Identity) + ); + if let GroupCommand::Membership { changes, .. } = &mut repeated { + changes.push(changes[0]); + } + assert_eq!( + CellGroupEntry { + binding: binding(&route), + command: repeated + } + .encode(&route, limits), + Err(ProtocolError::Identity) + ); + let mut duplicate_child = seal.clone(); + if let GroupCommand::Seal { children, .. } = &mut duplicate_child { + children[1].history.group = children[0].history.group; + } + assert!(CellGroupEntry { + binding: binding(&route), + command: duplicate_child + } + .encode(&route, limits) + .is_err()); + let mut wrong_hash = seal.clone(); + if let GroupCommand::Seal { children, .. } = &mut wrong_hash { + children[0].range = CellRange::new( + CellBound::NegativeInfinity, + CellBound::Key(CellKey::from_digest([0; 32], CellId::new("alpha").unwrap())), + ) + .unwrap(); + } + assert!(CellGroupEntry { + binding: binding(&route), + command: wrong_hash + } + .encode(&route, limits) + .is_err()); + let mut bounded = limits; + bounded.max_transition_groups = 1; + let bytes = CellGroupEntry { + binding: binding(&route), + command: seal, + } + .encode(&route, limits) + .unwrap(); + assert_eq!( + CellGroupEntry::decode(&bytes, &route, bounded), + Err(ProtocolError::Limit) + ); + assert!(ParkGeneration::new(0).is_err()); + assert!(ParkGeneration::new(u64::MAX) + .unwrap() + .checked_next() + .is_err()); + assert!(TransitionId::new([0; 16]).is_err()); +} + +#[test] +fn group_enrollment_contains_replayable_initial_image() { + use cellar::application::consensus::{ImageCommitment, ProtocolError}; + use sha2::{Digest, Sha256}; + let initial = image_fixture::image( + None, + "CREATE TABLE seed(n INTEGER); INSERT INTO seed VALUES (17);", + ) + .unwrap(); + let seed = image_fixture::Cut::new(&[], &initial, 0, true, true); + let route = group_route(); + let limits = GroupEntryLimits::default(); + let entry = CellGroupEntry { + binding: binding(&route), + command: GroupCommand::Enroll { + cell: observation().version.history, + image: image_value(seed.result), + files: vec![GroupLtx { + min_txid: 1, + max_txid: 1, + page_count: seed.pages.len() as u64, + decoded_bytes: seed.pages.len() as u64 * 512, + bytes: &seed.bytes, + }], + }, + }; + let bytes = entry.encode(&route, limits).unwrap(); + let decoded = CellGroupEntry::decode(&bytes, &route, limits).unwrap(); + let GroupCommand::Enroll { image, files, .. } = decoded.command else { + panic!("enrollment changed kind") + }; + // A remote replica can reconstruct this initial SQLite image from the log + // bytes alone, without the proposing host's disk or an object-store read. + let recovered = cellar_ltx::ltx::decode_database_image(files[0].bytes).unwrap(); + assert_eq!(recovered, initial); + assert_eq!( + image.commitment, + ImageCommitment::Sha256(Sha256::digest(&recovered).into()) + ); + let mut no_bytes = entry.clone(); + if let GroupCommand::Enroll { files, .. } = &mut no_bytes.command { + files.clear(); + } + assert_eq!(no_bytes.encode(&route, limits), Err(ProtocolError::Limit)); + let mut missing_prefix = entry; + if let GroupCommand::Enroll { files, .. } = &mut missing_prefix.command { + files[0].min_txid = 2; + } + assert_eq!( + missing_prefix.encode(&route, limits), + Err(ProtocolError::Chain) + ); +} diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 25224f45e..dbbf38aa4 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -26,8 +26,15 @@ The new contracts are exposed through the existing unstable verification surface while host integration is pending. Existing public runtime consumers still use the old contracts and will switch together at CG-07, before legacy removal at CG-16. -Remaining CG-01 work includes the cell-addressed data and typed administrative -entry codec, bounded decoder tests, complete task checks and source-bound evidence. +The [entry codec](../../../crates/cellar/application/consensus/group_protocol/README.md) +now carries cell-addressed mutations and typed enrollment, membership, checkpoint, +seal/install/activate, park and retire commands in a new marker-1 domain. It bounds +metadata before allocation and borrows LTX payloads. Enrollment carries its initial +LTX snapshot in the log instead of relying on an unproven image reference. +All 7 native contract cases pass, including both remaining required decoder cases, +an independent byte vector and SQLite reconstruction from the enrollment payload. + +Remaining CG-01 work is the complete task checks and source-bound evidence. CG-02 and later tasks remain TODO. No task is DONE and no benchmark, RustFS drill, hibernation latency, recovery guarantee or production cutover is claimed here. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index deaccc452..e2bbfe7f2 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -42,7 +42,11 @@ "creates": [ "crates/cellar-logic/cell_group.rs", "crates/cellar-logic/cell_group_tests.rs", - "crates/cellar/tests/native_cell_groups_contracts.rs" + "crates/cellar/tests/native_cell_groups_contracts.rs", + "crates/cellar/application/consensus/group_contracts.rs", + "crates/cellar/application/consensus/group_protocol/mod.rs", + "crates/cellar/application/consensus/group_protocol/types.rs", + "crates/cellar/application/consensus/group_protocol/codec.rs" ], "test_target": "native_cell_groups_contracts", "steps": [ From bad761e045bc0f10053e9ed043a62b0e1633526a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 09:06:01 -0700 Subject: [PATCH 006/568] docs: finalize cell-group execution handoff and contract evidence --- plans/multi-raft/cell-groups/README.md | 11 ++- plans/multi-raft/cell-groups/baseline.json | 4 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/progress.md | 99 ++++++++++--------- plans/multi-raft/cell-groups/tasks.json | 6 +- 5 files changed, 66 insertions(+), 56 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 03342e43d..830a9ca7d 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -2,8 +2,9 @@ # Cell group delivery package -Status: implementation started on 2026-10-29 following the executable plan. -CG-01 is in progress; the production host has not yet cut over to shared groups. +Status: executable design and implementation plan delivered on 2026-10-29. +Existing CG-01 contracts have passed their local evidence gate; CG-02 is next. +The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -32,6 +33,12 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package only with verified evidence and completed dependencies. Final qualification repeats the full capability matrix at one release revision. +This checkout retains CG-01's evidence under the ignored +`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/` directory. A fresh checkout must +receive that retained directory at the same relative path before validating the +DONE checkpoint. Missing evidence fails validation; it is never treated as a pass. +The design, task briefs and validator unit tests are committed in this package. + ```sh python3 plans/multi-raft/cell-groups/plan.py validate python3 plans/multi-raft/cell-groups/plan.py preflight diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 22802c4a3..b49267f23 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "7484502a2209205d68be42d895abfce7ad15291f", + "source_revision": "ee7f4a8d2dec220904f5a6860041f12c14db1cf5", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed CG-01 intermediate contracts at 7484502a2209205d68be42d895abfce7ad15291f. version.rs adds the separate cell-token domain and logical-history verification; original group-token consumers remain pending cutover. Other pinned owner inputs are unchanged. CG-01 is IN_PROGRESS, not qualified.", + "review_note": "2026-10-29: reviewed and qualified CG-01 at ee7f4a8d2dec220904f5a6860041f12c14db1cf5. Cell/group histories, cell tokens and bounded typed entry codec are complete at their contract boundary. Enrollment carries replayable initial LTX. Existing host/storage owners remain unchanged and await CG-02 through CG-16.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index f95448fbf..e4bacbeaa 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -12,7 +12,7 @@ never executed by the plan runner. Read the [design](design.md), | Task | Deliverable | Dependencies | Status | Required cases | | --- | --- | --- | --- | ---: | -| CG-01 | Separate logical cell history from physical group consensus | None | IN_PROGRESS | 4 | +| CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | TODO | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index dbbf38aa4..833c8bd5e 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -2,63 +2,64 @@ # Cell-group implementation progress -The active objective is the entire design and a hard cutover, including all CG-01 -through CG-16 qualification gates. Intermediate contracts do not qualify a shared +The delivered design and implementation plan covers CG-01 through CG-16 and a +hard cutover. The current handoff is the executable plan; further runtime work +follows its dependency order. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. -## CG-01 — in progress +## CG-01 — complete at the contract boundary -Implemented contracts: +Qualified source: `ee7f4a8d2dec220904f5a6860041f12c14db1cf5`. -- Dependency-free validated cell IDs, logical cell histories and physical group - histories; nonzero mutation, route and membership counters refuse overflow. -- Canonical namespace routing preimage and adapter SHA-256, collision-safe ordered - keys, half-open ranges, adjacent merge and whole-key split boundaries. -- Cell versions compare mutation sequence within one logical history, rejecting - equal-sequence conflicting image commitments. Commit witnesses remain separate. -- Domain-separated signed cell-version tokens authenticate schema, cell history, - sequence, LTX/image commitment and the original physical commit witness. Transfer - can preserve the token's history; disaster recovery's new incarnation refuses it. -- Materialized/extent commitment conversion validates the actual extent reference - layout at the host boundary. Core contracts add no dependencies or I/O. +- Dependency-free cell/group histories, validated cell IDs, monotone counters, + canonical routing hashes and collision-safe half-open ranges. +- Logical cell versions and signed tokens independent of physical log position. + Normal transfer preserves history; disaster recovery changes incarnation. +- Materialized and extent commitment adapters with geometry validation. +- A bounded, domain-separated entry codec for cell-addressed mutations and typed + enrollment, membership, checkpoint, seal/install/activate, park and retire. + Enrollment carries replayable initial LTX bytes; decoding borrows payloads. -The new contracts are exposed through the existing unstable verification surface -while host integration is pending. Existing public runtime consumers still use the -old contracts and will switch together at CG-07, before legacy removal at CG-16. +The [entry codec contract](../../../crates/cellar/application/consensus/group_protocol/README.md) +documents wire layout and checks that remain the runtime owner's responsibility. +The contracts are exposed through the unstable verification surface. Production +consumers still use the old contracts until CG-07; legacy removal follows in CG-16. -The [entry codec](../../../crates/cellar/application/consensus/group_protocol/README.md) -now carries cell-addressed mutations and typed enrollment, membership, checkpoint, -seal/install/activate, park and retire commands in a new marker-1 domain. It bounds -metadata before allocation and borrows LTX payloads. Enrollment carries its initial -LTX snapshot in the log instead of relying on an unproven image reference. -All 7 native contract cases pass, including both remaining required decoder cases, -an independent byte vector and SQLite reconstruction from the enrollment payload. +The retained source-bound record is +`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/record.json`. It binds the tested +Git revision, source inputs, native test executable and all 13 command logs by +hash. The complete gate passed: -Remaining CG-01 work is the complete task checks and source-bound evidence. -CG-02 and later tasks remain TODO. No task is DONE and no benchmark, RustFS drill, -hibernation latency, recovery guarantee or production cutover is claimed here. +- All-target checks and Clippy for `cellar` and `cellar-logic`, with locked builds + and Clippy warnings denied. +- All four required CG-01 cases selected individually by exact name, plus test + inventory verification to reject an empty selection. +- All 7 native cell-group contract cases, 6 existing native Raft contract cases + and 18 core library cases: 31 distinct Rust tests. +- Repository formatting, module layout, native dependency isolation and docs. -Checks completed for this intermediate change: +Tests include SQLite reconstruction from enrollment bytes, an independent wire +vector, truncated/oversized metadata, cross-cell/history refusal, token tampering +and transfer versus disaster identity. These are contract tests, not proof of a +running multi-cell group, a RustFS drill or a performance claim. -- `native_cell_groups_contracts`: 3 cases pass, including the smallest and largest - cell IDs in the signed format, tampering, truncation and transfer/DR identity. -- `native_raft_contract`: all 6 existing cases pass. -- `native_raft_reads`: exact `query_token_auth_scope` and - `version_key_deadline_and_payload_limits` regressions pass. -- `cellar-logic --lib`: all 18 cases pass, including 4 new group-contract cases. -- Clippy for the affected libraries and new native test passes with `-D warnings`; - module layout, native dependency isolation and repository documentation pass. -- Repository formatting check, all 36 plan-validator tests, package validation - and the reviewed source preflight pass at this checkpoint. +The earlier short-ID token-bound failure and intermediate logs remain under +ignored local evidence. The corrected regression passes. Code review also caught +and removed a root-only enrollment gap before the final gate: nonempty enrollment +must include its initial replayable LTX chain. -The first native run exposed an incorrect minimum token length for short IDs. -The bound now derives from the fixed wire fields, and a one-byte ID regression -passes. Local logs, including that failed run, remain under the ignored -`.cellar/cell-groups-evidence/CG-01-in-progress/` directory. They are intermediate -checks, not the complete source-bound CG-01 evidence record. +## Remaining implementation -The original source preflight was inspected before edits. The reviewed source -baseline advances to `7484502a` for this intermediate checkpoint; `version.rs` adds -the separate token domain and all other pinned owners remain unchanged. This -baseline update is not task completion. Review further changed contracts before -refreshing their hashes or beginning dependent implementation. +CG-02 through CG-16 remain TODO. CG-02 is dependency-eligible and must implement +complete paged group manifests, real engine publication, recovery refusal for +missing dependencies and image/pin reclamation tests in both image modes. + +The reviewed source baseline advances to the qualified CG-01 revision. Original +host/storage source anchors remain unchanged; newly produced contract files are +bound by the CG-01 evidence record. Review changed inputs before refreshing the +baseline for later tasks. A new executor must receive the ignored evidence +folder before validating this DONE checkpoint. + +No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured +density or final hard cutover is qualified. The release gate remains closed until +all task and deployment evidence passes at one final revision. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index e2bbfe7f2..c1b2e65ce 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -22,7 +22,7 @@ { "id": "CG-01", "title": "Separate logical cell history from physical group consensus", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "M", "risk": "HIGH", @@ -116,7 +116,9 @@ "I05", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/record.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From f68a9bea175da786101d9fc72959bd3efdf3bc69 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 09:26:08 -0700 Subject: [PATCH 007/568] feat: add authenticated copy-on-write cell manifests --- crates/cellar/application/consensus/mod.rs | 2 + .../durability/raft/cell_groups/README.md | 44 ++ .../durability/raft/cell_groups/manifest.rs | 402 ++++++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 13 + .../durability/raft/cell_groups/page_store.rs | 174 ++++++++ .../durability/raft/cell_groups/page_wire.rs | 186 ++++++++ crates/cellar/durability/raft/mod.rs | 1 + .../tests/native_cell_groups_storage.rs | 276 ++++++++++++ crates/cellar/tests/support/raft_ltx_fault.rs | 4 + plans/multi-raft/cell-groups/README.md | 2 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/progress.md | 36 +- plans/multi-raft/cell-groups/tasks.json | 11 +- 13 files changed, 1145 insertions(+), 8 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/README.md create mode 100644 crates/cellar/durability/raft/cell_groups/manifest.rs create mode 100644 crates/cellar/durability/raft/cell_groups/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/page_store.rs create mode 100644 crates/cellar/durability/raft/cell_groups/page_wire.rs create mode 100644 crates/cellar/tests/native_cell_groups_storage.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 96522ea8e..23134b7c0 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -11,6 +11,7 @@ pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; pub(crate) mod client; mod group_contracts; +pub(crate) use group_contracts::cell_routing_key; pub(crate) mod group_protocol; mod performance; pub use performance::{ @@ -212,6 +213,7 @@ pub mod verification { pub mod cell_groups { pub use super::super::group_contracts::cell_routing_key; pub use super::super::group_protocol::*; + pub use crate::raft::cell_groups::*; pub use crate::raft::version::{CellVersionObservation, CellVersionToken}; pub use cellar_logic::cell_group::{ CellBound, CellHistory, CellId, CellImage, CellImageCommitment, CellKey, CellRange, diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md new file mode 100644 index 000000000..1bc0ec817 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -0,0 +1,44 @@ + + +# Cell-group durable state + +This component is under CG-02 implementation. It currently supplies immutable +manifest pages and their filesystem owner. It is not yet connected to a Raft +engine apply marker and does not admit requests or establish durability proofs. + +A manifest is a persistent AVL tree ordered by the namespace routing digest and +full cell ID. Each bounded page holds one cell leaf and at most two child +references. References bind SHA-256, height and subtree count. Insertions and +updates copy bounded paths; rotations preserve balance. Unchanged cell images +are reused. The format uses `CLRCGM01`, the marker-1 page hash domain, canonical +JSON and a 16 KiB decode bound. Tree depth and cell count are explicit limits. +These limits are admission ceilings, not qualified fleet capacities. + +Leaves bind logical cell history, mutation sequence, image commitment and +geometry, last group mutation position, schema and effect summary. Local image +filenames are excluded: replicas may store the same image under different local +names and still derive the same logical manifest. A physical image adapter must +resolve and pin each image by the selected logical leaf. Extent and materialized +commitments retain separate domains. + +The page store uses an injected filesystem, a lifetime exclusive lock, bounded +inventory and byte/page quotas. A new page crosses file and directory sync before +its reference is returned. Any ambiguous write/sync quarantines the owner until +reopen. Reopened filenames are only inventory; reads verify length and digest, +and reuse verifies and syncs bytes again. Pages are never deleted in this initial +slice, so old roots remain readable. Authoritative root retention, pin-aware +reclamation and recovery of partial orphan writes remain CG-02 work. + +`verify` traverses the entire authenticated closure, checks ordering and the +selected applied cut, and delegates actual image verification/pinning to its +caller. The callback must release any partial pins if verification fails. A +successful page traversal alone is not an image seal, Raft commit or serving +capability. Updates compare the expected cell version and require increasing +sequence, LTX position and group mutation position. + +Next integration must sync complete cell images and manifest paths before one +Raft Engine transaction selects the complete group root and contiguous applied +position. Recovery must refuse any missing page or image. Reader, apply, +snapshot and transfer pins must protect dependencies before deletion. Native +engine crash cuts and both physical image modes remain required before CG-02 +can be marked DONE. diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs new file mode 100644 index 000000000..2eada7eb9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -0,0 +1,402 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable authenticated AVL pages. Every cell occupies one bounded page; +//! changing a cell copies O(log cells) pages, preserving all other image roots. + +use super::page_store::ManifestPageStore; +use super::page_wire::{decode, encode, Page, PageRef}; +use crate::application::consensus::{cell_routing_key, LtxImage}; +use cellar_logic::{cell_group::*, consensus::LogPosition}; +use std::cmp::Ordering; + +#[derive(Debug)] +pub enum ManifestError { + Invalid(&'static str), + Limit, + Conflict, + Quarantined, + Io(std::io::Error), +} + +impl std::fmt::Display for ManifestError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group manifest: {self:?}") + } +} + +impl std::error::Error for ManifestError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Io(error) => Some(error), + _ => None, + } + } +} + +impl From for ManifestError { + fn from(error: std::io::Error) -> Self { + Self::Io(error) + } +} + +/// Geometry is logical data. Local generation names never enter a replicated +/// manifest commitment; each replica resolves and pins its own image storage. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellImageGeometry { + pub length: u64, + pub page_size: u32, +} + +#[derive(Clone, Copy, Debug)] +pub struct ManifestLimits { + pub max_cells: u32, + pub max_depth: u8, +} + +impl Default for ManifestLimits { + fn default() -> Self { + Self { + max_cells: 1_000_000, + max_depth: 32, + } + } +} + +impl ManifestLimits { + pub(super) fn validate(self) -> Result<(), ManifestError> { + if self.max_cells == 0 + || self.max_cells > 1_000_000 + || self.max_depth == 0 + || self.max_depth > 32 + { + return Err(ManifestError::Limit); + } + Ok(()) + } +} + +/// Authenticated scheduling hints. Callbacks and delivery identity still come +/// from the selected cell image; a summary grants no execution authority. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] +pub struct EffectSummary { + pub next_alarm_unix_millis: Option, + pub next_outbox_unix_millis: Option, + pub pending_outbox: u64, + pub pending_inbox: u64, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellManifestLeaf { + pub version: CellVersion, + pub geometry: CellImageGeometry, + pub last_mutation: LogPosition, + pub effects: EffectSummary, +} + +impl CellManifestLeaf { + pub(super) fn validate(&self, route: &GroupRoute) -> Result { + let key = cell_routing_key(route.history.namespace, &self.version.history.cell); + route + .check_cell(&self.version.history, &key) + .map_err(|_| ManifestError::Invalid("cell route"))?; + self.last_mutation + .validate() + .map_err(|_| ManifestError::Invalid("mutation position"))?; + if self.last_mutation.index.get() == 0 { + return Err(ManifestError::Invalid("zero mutation index")); + } + let image = LtxImage::try_from(self.version.image) + .map_err(|_| ManifestError::Invalid("image commitment"))?; + if !cellar_ltx::ltx::is_valid_page_size(self.geometry.page_size) + || self.geometry.length > cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES + || !self + .geometry + .length + .is_multiple_of(u64::from(self.geometry.page_size)) + || (image.txid == 0) != (self.geometry.length == 0) + || (image.txid == 0 && image.checksum != 0) + { + return Err(ManifestError::Invalid("image geometry")); + } + if let CellImageCommitment::Extents { manifest_bytes, .. } = self.version.image.commitment { + let expected = 112 + + 32 * self + .geometry + .length + .div_ceil(cellar_ltx::internal::EXTENT_BYTES); + if self.geometry.length == 0 || u64::from(manifest_bytes) != expected { + return Err(ManifestError::Invalid("extent geometry")); + } + } + Ok(key) + } +} + +/// An untrusted tree descriptor until checked against an authoritative group +/// root and traversed by `verify`. It is never an applied marker or image seal. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestRoot(pub(super) Option); + +impl ManifestRoot { + pub fn cells(self) -> u32 { + self.0.map_or(0, |p| p.cells) + } + + pub fn depth(self) -> u8 { + self.0.map_or(0, |p| p.height) + } + + pub fn digest(self) -> Option<[u8; 32]> { + self.0.map(|p| p.digest) + } +} + +/// Synchronous operations belong on the owning blocking execution domain. +/// Retain every root used by apply/read/snapshot/transfer before reclamation. +/// This layer performs no page deletion and cannot publish serving authority. +pub struct ManifestTree { + route: GroupRoute, + limits: ManifestLimits, +} + +impl ManifestTree { + pub fn new(route: GroupRoute, limits: ManifestLimits) -> Result { + limits.validate()?; + for bound in [route.range.lower(), route.range.upper()] { + if let CellBound::Key(key) = bound { + if &cell_routing_key(route.history.namespace, key.cell()) != key { + return Err(ManifestError::Invalid("range hash")); + } + } + } + Ok(Self { route, limits }) + } + + pub fn get( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + cell: &CellId, + ) -> Result, ManifestError> { + let key = cell_routing_key(self.route.history.namespace, cell); + let mut cursor = root.0; + let mut remaining = self.limits.max_depth; + while let Some(reference) = cursor { + if remaining == 0 { + return Err(ManifestError::Limit); + } + remaining -= 1; + let page = self.read(store, reference)?; + match key.cmp(&page.leaf.validate(&self.route)?) { + Ordering::Less => cursor = page.left, + Ordering::Greater => cursor = page.right, + Ordering::Equal => return Ok(Some(page.leaf)), + } + } + Ok(None) + } + + /// Compare-and-replace one logical cell. `None` requires absence; transfer + /// installation may insert a retained sequence. Enrollment authorization, + /// image sealing and the group's contiguous apply proof belong to the caller. + pub fn update( + &self, + store: &mut ManifestPageStore, + root: ManifestRoot, + leaf: CellManifestLeaf, + expected: Option<&CellVersion>, + ) -> Result { + let key = leaf.validate(&self.route)?; + let current = self.get(store, root, &leaf.version.history.cell)?; + match (¤t, expected) { + (None, None) => { + if root.cells() >= self.limits.max_cells { + return Err(ManifestError::Limit); + } + } + (Some(old), Some(expected)) if &old.version == expected => { + if old.version.history != leaf.version.history + || old.version.sequence.checked_next().ok() != Some(leaf.version.sequence) + || leaf.version.image.epoch != old.version.image.epoch + || leaf.version.image.txid <= old.version.image.txid + || leaf.last_mutation.index <= old.last_mutation.index + || leaf.last_mutation.term < old.last_mutation.term + { + return Err(ManifestError::Conflict); + } + } + _ => return Err(ManifestError::Conflict), + } + self.insert(store, root.0, leaf, &key, 0) + .map(|p| ManifestRoot(Some(p))) + } + + /// Walk the entire authenticated closure before recovery admission. The + /// callback must independently verify/pin each actual image and receipt state. + /// No success or partial callback traversal is itself a quorum proof. + pub fn verify( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + applied: LogPosition, + mut image: impl FnMut(&CellManifestLeaf) -> Result<(), ManifestError>, + ) -> Result<(), ManifestError> { + applied + .validate() + .map_err(|_| ManifestError::Invalid("applied position"))?; + let mut visited = 0; + self.walk( + store, + root.0, + (None, None), + applied, + &mut visited, + &mut image, + )?; + if visited != root.cells() { + return Err(ManifestError::Invalid("cell count")); + } + Ok(()) + } + + fn walk( + &self, + store: &ManifestPageStore, + reference: Option, + bounds: (Option<&CellKey>, Option<&CellKey>), + applied: LogPosition, + visited: &mut u32, + image: &mut impl FnMut(&CellManifestLeaf) -> Result<(), ManifestError>, + ) -> Result<(), ManifestError> { + let (lower, upper) = bounds; + let Some(reference) = reference else { + return Ok(()); + }; + *visited = visited.checked_add(1).ok_or(ManifestError::Limit)?; + if *visited > self.limits.max_cells { + return Err(ManifestError::Limit); + } + let page = self.read(store, reference)?; + let key = page.leaf.validate(&self.route)?; + if lower.is_some_and(|l| key <= *l) + || upper.is_some_and(|u| key >= *u) + || page.leaf.last_mutation.index > applied.index + || page.leaf.last_mutation.term > applied.term + || (page.leaf.last_mutation.index == applied.index + && page.leaf.last_mutation.term != applied.term) + { + return Err(ManifestError::Invalid("tree order or apply cut")); + } + self.walk( + store, + page.left, + (lower, Some(&key)), + applied, + visited, + image, + )?; + image(&page.leaf)?; + self.walk( + store, + page.right, + (Some(&key), upper), + applied, + visited, + image, + ) + } + + fn read(&self, store: &ManifestPageStore, reference: PageRef) -> Result { + let bytes = store.read(reference.digest)?; + let page = decode(&bytes, &self.route, self.limits)?; + if page.reference(&bytes, self.limits)? != reference { + return Err(ManifestError::Invalid("page reference")); + } + Ok(page) + } + + fn save(&self, store: &mut ManifestPageStore, page: Page) -> Result { + let bytes = encode(&page, &self.route, self.limits)?; + let reference = page.reference(&bytes, self.limits)?; + store.put(reference.digest, &bytes)?; + Ok(reference) + } + + fn insert( + &self, + store: &mut ManifestPageStore, + root: Option, + leaf: CellManifestLeaf, + key: &CellKey, + depth: u8, + ) -> Result { + if depth >= self.limits.max_depth { + return Err(ManifestError::Limit); + } + let Some(reference) = root else { + return self.save( + store, + Page { + leaf, + left: None, + right: None, + }, + ); + }; + let mut page = self.read(store, reference)?; + match key.cmp(&page.leaf.validate(&self.route)?) { + Ordering::Less => { + page.left = Some(self.insert(store, page.left, leaf, key, depth + 1)?) + } + Ordering::Greater => { + page.right = Some(self.insert(store, page.right, leaf, key, depth + 1)?) + } + Ordering::Equal => page.leaf = leaf, + } + self.balance(store, page) + } + + fn balance( + &self, + store: &mut ManifestPageStore, + mut page: Page, + ) -> Result { + let height = |r: Option| r.map_or(0, |p| i16::from(p.height)); + let balance = height(page.left) - height(page.right); + if balance > 1 { + let mut left = + self.read(store, page.left.ok_or(ManifestError::Invalid("rotation"))?)?; + if height(left.left) < height(left.right) { + let mut middle = + self.read(store, left.right.ok_or(ManifestError::Invalid("rotation"))?)?; + left.right = middle.left; + middle.left = Some(self.save(store, left)?); + page.left = middle.right; + middle.right = Some(self.save(store, page)?); + return self.save(store, middle); + } + page.left = left.right; + left.right = Some(self.save(store, page)?); + return self.save(store, left); + } + if balance < -1 { + let mut right = + self.read(store, page.right.ok_or(ManifestError::Invalid("rotation"))?)?; + if height(right.right) < height(right.left) { + let mut middle = + self.read(store, right.left.ok_or(ManifestError::Invalid("rotation"))?)?; + right.left = middle.right; + middle.right = Some(self.save(store, right)?); + page.right = middle.left; + middle.left = Some(self.save(store, page)?); + return self.save(store, middle); + } + page.right = right.left; + right.left = Some(self.save(store, page)?); + return self.save(store, right); + } + self.save(store, page) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs new file mode 100644 index 000000000..195bb8b2e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -0,0 +1,13 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete group-state storage. A synced page is data, not Raft authority. + +mod manifest; +mod page_store; +mod page_wire; + +pub use manifest::{ + CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, + ManifestRoot, ManifestTree, +}; +pub use page_store::{ManifestPageStore, ManifestStoreUsage}; diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs new file mode 100644 index 000000000..605309944 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -0,0 +1,174 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exclusively owned, bounded content-addressed page files. No filesystem work +//! escapes the supplied host. Call on a blocking execution-domain worker. + +use super::{ + manifest::ManifestError, + page_wire::{digest, MAX_PAGE_BYTES}, +}; +use cellar_ltx::{FileSystem, HostFileLock}; +use std::{ + collections::BTreeMap, + path::{Path, PathBuf}, + sync::Arc, +}; + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestStoreUsage { + pub pages: usize, + pub bytes: u64, + pub written_pages: u64, +} + +pub struct ManifestPageStore { + directory: PathBuf, + filesystem: Arc, + inventory: BTreeMap<[u8; 32], u64>, + usage: ManifestStoreUsage, + max_pages: usize, + max_bytes: u64, + quarantined: bool, + _lock: HostFileLock, +} + +impl ManifestPageStore { + pub fn open( + directory: &Path, + filesystem: Arc, + max_pages: usize, + max_bytes: u64, + ) -> Result { + if max_pages == 0 + || max_pages > 4_000_000 + || max_bytes == 0 + || max_bytes > 64 * 1024 * 1024 * 1024 + { + return Err(ManifestError::Limit); + } + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), directory)?; + let lock = filesystem.lock_file(&directory.join("LOCK"))?; + let mut store = Self { + directory: directory.to_owned(), + filesystem, + inventory: BTreeMap::new(), + usage: ManifestStoreUsage::default(), + max_pages, + max_bytes, + quarantined: false, + _lock: lock, + }; + for entry in store.filesystem.read_dir_iter(directory)? { + let entry = entry?; + let name = entry + .file_name + .to_str() + .ok_or(ManifestError::Invalid("page name"))?; + if name == "LOCK" { + continue; + } + if store.inventory.len() >= max_pages { + return Err(ManifestError::Limit); + } + let hex = name + .strip_suffix(".page") + .filter(|s| { + s.len() == 64 + && s.bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) + }) + .ok_or(ManifestError::Invalid("page name"))?; + let mut key = [0; 32]; + hex::decode_to_slice(hex, &mut key).map_err(|_| ManifestError::Invalid("page name"))?; + // Use our own joined basename, never a provider's arbitrary path. + let metadata = store.filesystem.symlink_metadata(&directory.join(name))?; + if !metadata.is_file || metadata.len > MAX_PAGE_BYTES as u64 { + return Err(ManifestError::Invalid("page file")); + } + store.usage.bytes = store + .usage + .bytes + .checked_add(metadata.len) + .ok_or(ManifestError::Limit)?; + if store.usage.bytes > max_bytes { + return Err(ManifestError::Limit); + } + if store.inventory.insert(key, metadata.len).is_some() { + return Err(ManifestError::Invalid("duplicate page")); + } + } + store.usage.pages = store.inventory.len(); + Ok(store) + } + + pub fn usage(&self) -> ManifestStoreUsage { + self.usage + } + + pub(super) fn read(&self, key: [u8; 32]) -> Result, ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + let expected = self + .inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))?; + let path = self.directory.join(format!("{}.page", hex::encode(key))); + let mut file = self.filesystem.open(&path)?; + let length = file.file_len()?; + if length != *expected || length > MAX_PAGE_BYTES as u64 { + return Err(ManifestError::Invalid("page length")); + } + let bytes = file.read_exact_at(0, length as usize)?; + if digest(&bytes) != key { + return Err(ManifestError::Invalid("page digest")); + } + Ok(bytes) + } + + pub(super) fn put(&mut self, key: [u8; 32], bytes: &[u8]) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if bytes.len() > MAX_PAGE_BYTES || digest(bytes) != key { + return Err(ManifestError::Invalid("page bytes")); + } + let path = self.directory.join(format!("{}.page", hex::encode(key))); + if self.inventory.contains_key(&key) { + if self.read(key)? != bytes { + return Err(ManifestError::Invalid("page collision")); + } + // Restart inventory is not proof of a prior completed fsync. + self.quarantined = true; + self.filesystem.open(&path)?.sync_all()?; + self.filesystem.sync_all(&self.directory)?; + self.quarantined = false; + return Ok(()); + } + let next_bytes = self + .usage + .bytes + .checked_add(bytes.len() as u64) + .ok_or(ManifestError::Limit)?; + if self.inventory.len() >= self.max_pages || next_bytes > self.max_bytes { + return Err(ManifestError::Limit); + } + // Any uncertain write/sync leaves the owner quarantined. No refunds or + // deletion before a new owner reconstructs authoritative root references. + self.quarantined = true; + let mut file = self.filesystem.create_new(&path)?; + file.write_all(bytes)?; + file.sync_all()?; + self.filesystem.sync_all(&self.directory)?; + self.inventory.insert(key, bytes.len() as u64); + self.usage.pages += 1; + self.usage.bytes = next_bytes; + self.usage.written_pages = self + .usage + .written_pages + .checked_add(1) + .ok_or(ManifestError::Limit)?; + self.quarantined = false; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs new file mode 100644 index 000000000..0dc08e9b3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -0,0 +1,186 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::manifest::{ + CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, +}; +use crate::application::consensus::{ImageCommitment, LtxImage}; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +const MAGIC: &[u8; 8] = b"CLRCGM01"; +pub(super) const MAX_PAGE_BYTES: usize = 16 * 1024; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct PageRef { + pub digest: [u8; 32], + pub height: u8, + pub cells: u32, +} + +pub(super) struct Page { + pub leaf: CellManifestLeaf, + pub left: Option, + pub right: Option, +} + +impl Page { + pub fn reference( + &self, + bytes: &[u8], + limits: ManifestLimits, + ) -> Result { + let height = |p: Option| p.map_or(0, |p| p.height); + let count = |p: Option| p.map_or(0, |p| p.cells); + for child in [self.left, self.right].into_iter().flatten() { + if child.height == 0 + || child.height > limits.max_depth + || child.cells == 0 + || child.cells > limits.max_cells + { + return Err(ManifestError::Invalid("child bounds")); + } + } + if height(self.left).abs_diff(height(self.right)) > 1 { + return Err(ManifestError::Invalid("unbalanced page")); + } + let height = height(self.left) + .max(height(self.right)) + .checked_add(1) + .ok_or(ManifestError::Limit)?; + let cells = count(self.left) + .checked_add(count(self.right)) + .and_then(|n| n.checked_add(1)) + .ok_or(ManifestError::Limit)?; + if height > limits.max_depth || cells > limits.max_cells { + return Err(ManifestError::Limit); + } + Ok(PageRef { + digest: digest(bytes), + height, + cells, + }) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WirePage { + schema: [u8; 32], + cluster: [u8; 16], + namespace: [u8; 16], + cell: String, + incarnation: [u8; 16], + sequence: u64, + commitment: ImageCommitment, + epoch: u64, + txid: u64, + checksum: u64, + geometry: CellImageGeometry, + mutation_index: u64, + mutation_term: u64, + effects: EffectSummary, + left: Option, + right: Option, +} + +pub(super) fn digest(bytes: &[u8]) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-group-manifest-page-1\0"); + hash.update(bytes); + hash.finalize().into() +} + +pub(super) fn encode( + page: &Page, + route: &GroupRoute, + limits: ManifestLimits, +) -> Result, ManifestError> { + page.leaf.validate(route)?; + let leaf = &page.leaf; + let history = &leaf.version.history; + let wire = WirePage { + schema: route.schema_digest, + cluster: history.cluster.bytes(), + namespace: history.namespace.bytes(), + cell: history.cell.as_str().to_owned(), + incarnation: history.incarnation.bytes(), + sequence: leaf.version.sequence.get(), + commitment: LtxImage::try_from(leaf.version.image) + .map_err(|_| ManifestError::Invalid("image commitment"))? + .commitment, + epoch: leaf.version.image.epoch, + txid: leaf.version.image.txid, + checksum: leaf.version.image.checksum, + geometry: leaf.geometry, + mutation_index: leaf.last_mutation.index.get(), + mutation_term: leaf.last_mutation.term.get(), + effects: leaf.effects, + left: page.left, + right: page.right, + }; + let mut bytes = MAGIC.to_vec(); + bytes.extend(serde_json::to_vec(&wire).map_err(|_| ManifestError::Invalid("page encoding"))?); + if bytes.len() > MAX_PAGE_BYTES { + return Err(ManifestError::Limit); + } + page.reference(&bytes, limits)?; + Ok(bytes) +} + +pub(super) fn decode( + bytes: &[u8], + route: &GroupRoute, + limits: ManifestLimits, +) -> Result { + // Reject length before JSON allocation. The schema has no unbounded vectors + // or recursive fields; serde's recursion limit remains enabled. + if bytes.len() > MAX_PAGE_BYTES { + return Err(ManifestError::Limit); + } + if !bytes.starts_with(MAGIC) { + return Err(ManifestError::Invalid("page domain")); + } + let wire: WirePage = serde_json::from_slice(&bytes[MAGIC.len()..]) + .map_err(|_| ManifestError::Invalid("page decoding"))?; + if wire.schema != route.schema_digest { + return Err(ManifestError::Invalid("page schema")); + } + let history = CellHistory { + cluster: ClusterId::new(wire.cluster).map_err(|_| ManifestError::Invalid("cluster"))?, + namespace: NamespaceId::new(wire.namespace) + .map_err(|_| ManifestError::Invalid("namespace"))?, + cell: CellId::new(wire.cell).map_err(|_| ManifestError::Invalid("cell ID"))?, + incarnation: IncarnationId::new(wire.incarnation) + .map_err(|_| ManifestError::Invalid("incarnation"))?, + }; + let image = LtxImage { + epoch: wire.epoch, + txid: wire.txid, + checksum: wire.checksum, + commitment: wire.commitment, + }; + let page = Page { + leaf: CellManifestLeaf { + version: CellVersion { + history, + sequence: MutationSequence::new(wire.sequence) + .map_err(|_| ManifestError::Invalid("sequence"))?, + image: image.into(), + }, + geometry: wire.geometry, + last_mutation: LogPosition { + index: LogIndex::new(wire.mutation_index), + term: Term::new(wire.mutation_term), + }, + effects: wire.effects, + }, + left: wire.left, + right: wire.right, + }; + if encode(&page, route, limits)? != bytes { + return Err(ManifestError::Invalid("noncanonical page")); + } + Ok(page) +} diff --git a/crates/cellar/durability/raft/mod.rs b/crates/cellar/durability/raft/mod.rs index e81f7618c..a8ce22a04 100644 --- a/crates/cellar/durability/raft/mod.rs +++ b/crates/cellar/durability/raft/mod.rs @@ -8,6 +8,7 @@ pub(crate) mod budget; pub(crate) mod cache; pub(crate) mod catalog; mod catalog_wire; +pub(crate) mod cell_groups; pub(crate) mod completion; pub(crate) mod coordination_index; pub(crate) mod disaster; diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs new file mode 100644 index 000000000..d728c84dc --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -0,0 +1,276 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Manifest-page contracts on independently sealed SQLite images. Engine marker +//! publication and complete recovery qualification are added at the next boundary. +#![allow(clippy::disallowed_methods)] + +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; + +#[path = "support/raft_ltx_fault.rs"] +#[allow(dead_code)] +mod faults; + +use anyhow::Result; +use cellar::application::consensus::{ + verification::{cell_groups::*, CommittedImage}, + *, +}; +use cellar_ltx::{ + generation::{GenerationLimits, GenerationPin, GenerationStore}, + DirectFileSystem, +}; +use std::{collections::BTreeMap, sync::Arc}; + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn position(index: u64) -> LogPosition { + LogPosition { + index: LogIndex::new(index), + term: Term::new(1), + } +} + +fn leaf(id: u8, pin: &GenerationPin, sequence: u64, index: u64) -> CellManifestLeaf { + let state = pin.state(); + let image = CommittedImage { + name: format!("{}.sqlite", hex::encode(pin.id().bytes())), + commitment: state.digest.into(), + length: state.length, + ltx_epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + page_size: state.page_size, + }; + CellManifestLeaf { + version: CellVersion { + history: CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new(format!("cell-{id}")).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + }, + sequence: MutationSequence::new(sequence).unwrap(), + image: LtxImage { + epoch: image.ltx_epoch, + txid: image.txid, + checksum: image.checksum, + commitment: image.commitment, + } + .into(), + }, + geometry: CellImageGeometry { + length: image.length, + page_size: image.page_size, + }, + last_mutation: position(index), + effects: EffectSummary::default(), + } +} + +#[test] +fn manifest_updates_only_changed_path() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits { + max_generations: 128, + ..Default::default() + }, + )?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 4096, + 64 * 1024 * 1024, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut root = ManifestRoot::default(); + let mut cells = BTreeMap::new(); + for id in 1..=64 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + assert_eq!(images::read(&pin)?, bytes); + let value = leaf(id, &pin, 1, u64::from(id)); + root = tree.update(&mut pages, root, value.clone(), None)?; + cells.insert(value.version.history.cell.clone(), (value, pin)); + } + assert_eq!(root.cells(), 64); + assert!(root.depth() <= 9, "AVL height must remain logarithmic"); + let old_root = root; + let target = CellId::new("cell-32").unwrap(); + let (old, pin) = cells.get(&target).unwrap(); + let before = images::read(pin)?; + let after = images::image(Some(&before), "UPDATE value SET n = 99;")?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let changed = generations.prepare(images::id(100), Some(pin), &cut.update())?; + let value = leaf(32, &changed, 2, 65); + let written = pages.usage().written_pages; + root = tree.update(&mut pages, root, value.clone(), Some(&old.version))?; + assert!(pages.usage().written_pages - written <= u64::from(old_root.depth())); + assert_eq!(tree.get(&pages, old_root, &target)?.unwrap(), *old); + assert_eq!(tree.get(&pages, root, &target)?.unwrap(), value); + for (cell, (expected, _)) in &cells { + if cell != &target { + assert_eq!(tree.get(&pages, root, cell)?.unwrap(), *expected); + } + } + tree.verify(&pages, root, position(65), |actual| { + let pin = if actual.version.history.cell == target { + &changed + } else { + &cells[&actual.version.history.cell].1 + }; + if actual.version.image.commitment != CellImageCommitment::Materialized(pin.state().digest) + { + return Err(ManifestError::Invalid("fixture image")); + } + // The retained GenerationPin is an independently verified/synced image. + Ok(()) + })?; + drop(pages); + let reopened = ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024)?; + assert_eq!(tree.get(&reopened, root, &target)?.unwrap(), value); + tree.verify(&reopened, root, position(65), |_| Ok(()))?; + Ok(()) +} + +#[test] +fn manifest_rejects_conflicts_bounds_and_missing_pages() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let value = leaf(1, &pin, 1, 1); + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024)?; + let tree = ManifestTree::new( + route(), + ManifestLimits { + max_cells: 1, + max_depth: 1, + }, + )?; + let root = tree.update(&mut pages, ManifestRoot::default(), value.clone(), None)?; + let before = pages.usage(); + assert!(tree.update(&mut pages, root, value.clone(), None).is_err()); + assert!(tree + .update(&mut pages, root, value.clone(), Some(&value.version)) + .is_err()); + assert!(tree + .update(&mut pages, root, leaf(2, &pin, 1, 2), None) + .is_err()); + assert_eq!(pages.usage(), before); + assert!(tree.verify(&pages, root, position(0), |_| Ok(())).is_err()); + let mut other = route(); + other.schema_digest = [9; 32]; + assert!(ManifestTree::new(other, ManifestLimits::default())? + .get(&pages, root, &value.version.history.cell) + .is_err()); + let path = temp + .path() + .join("pages") + .join(format!("{}.page", hex::encode(root.digest().unwrap()))); + let original = std::fs::read(&path)?; + std::fs::write(&path, vec![0; 16 * 1024 + 1])?; + assert!(tree.get(&pages, root, &value.version.history.cell).is_err()); + std::fs::write(&path, original)?; + std::fs::remove_file(&path)?; + assert!(tree.verify(&pages, root, position(1), |_| Ok(())).is_err()); + Ok(()) +} + +#[test] +fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<()> { + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (3);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let first = generations.prepare(images::id(1), None, &cut.update())?; + let second = generations.prepare(images::id(2), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut a = ManifestPageStore::open(&temp.path().join("a"), fs.clone(), 64, 1024 * 1024)?; + let mut b = ManifestPageStore::open(&temp.path().join("b"), fs, 64, 1024 * 1024)?; + let a_root = tree.update(&mut a, ManifestRoot::default(), leaf(1, &first, 1, 1), None)?; + let b_root = tree.update( + &mut b, + ManifestRoot::default(), + leaf(1, &second, 1, 1), + None, + )?; + assert_ne!(first.id(), second.id()); + assert_eq!( + a_root, b_root, + "local generation names must not enter logical roots" + ); + let directory = temp.path().join("faulted"); + let faults = faults::Faults::new(directory.clone(), None); + let filesystem = Arc::new(faults::FaultFileSystem { + faults: faults.clone(), + }); + let mut pages = ManifestPageStore::open(&directory, filesystem.clone(), 64, 1024 * 1024)?; + faults.mode.store( + faults::FAIL_DIRECTORY_SYNC, + std::sync::atomic::Ordering::Release, + ); + assert!(tree + .update( + &mut pages, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None + ) + .is_err()); + assert!(matches!( + tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None + ), + Err(ManifestError::Quarantined) + )); + drop(pages); + let mut recovered = ManifestPageStore::open(&directory, filesystem, 64, 1024 * 1024)?; + // Retried bytes must be independently checked and synced, even if the + // interrupted attempt left a complete content-addressed filename. + let recovered_root = tree.update( + &mut recovered, + ManifestRoot::default(), + leaf(1, &first, 1, 1), + None, + )?; + assert_eq!(a_root, recovered_root); + assert!(!faults.synced_paths.lock().unwrap().is_empty()); + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_ltx_fault.rs b/crates/cellar/tests/support/raft_ltx_fault.rs index 2e896b525..b04b80dde 100644 --- a/crates/cellar/tests/support/raft_ltx_fault.rs +++ b/crates/cellar/tests/support/raft_ltx_fault.rs @@ -256,6 +256,10 @@ impl FileSystem for FaultFileSystem { DirectFileSystem.metadata(path) } + fn symlink_metadata(&self, path: &Path) -> io::Result { + DirectFileSystem.symlink_metadata(path) + } + fn create_read_write(&self, path: &Path) -> io::Result { let image_output = path.extension().is_some_and(|p| p == "stage") || path.extension().is_some_and(|p| p == "sqlite") diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 830a9ca7d..960b7529e 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -3,7 +3,7 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -Existing CG-01 contracts have passed their local evidence gate; CG-02 is next. +CG-01 contracts have passed their local evidence gate; CG-02 is in progress. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index e4bacbeaa..2f0f0c56e 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -13,7 +13,7 @@ never executed by the plan runner. Read the [design](design.md), | Task | Deliverable | Dependencies | Status | Required cases | | --- | --- | --- | --- | ---: | | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | -| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | TODO | 5 | +| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | IN_PROGRESS | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 833c8bd5e..2bd146169 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,8 +3,7 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The current handoff is the executable plan; further runtime work -follows its dependency order. Intermediate contracts do not qualify a shared +hard cutover. Implementation is active and follows the full dependency order. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. ## CG-01 — complete at the contract boundary @@ -50,7 +49,7 @@ must include its initial replayable LTX chain. ## Remaining implementation -CG-02 through CG-16 remain TODO. CG-02 is dependency-eligible and must implement +CG-02 is IN_PROGRESS; CG-03 through CG-16 remain TODO. CG-02 must complete complete paged group manifests, real engine publication, recovery refusal for missing dependencies and image/pin reclamation tests in both image modes. @@ -63,3 +62,34 @@ folder before validating this DONE checkpoint. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until all task and deployment evidence passes at one final revision. + + +## CG-02 — manifest pages in progress + +The first storage slice implements a bounded immutable AVL manifest and an +exclusively owned filesystem page store. Leaves bind logical versions, geometry, +last mutation and effect summaries; local image filenames do not affect roots. +Updates compare the old cell version and copy only the changed tree path. +Page publication syncs files and directories and quarantines uncertain writes. + +The new native storage target uses real SQLite/LTX and sealed generation pins. +All three current cases pass: path-copy bounds across 64 independently sealed +SQLite cells, unchanged image commitments and old-root reads; restart and +malformed/missing page refusal; conflicts/capacity, replica-independent roots and +injected page-directory sync failure. These are page-store checks; the complete CG-02 +gate has not run. The first build failed on an unavailable error derive and a +follow-up build caught argument mismatches. The first running suite exposed a +missing link-aware metadata method in the fault fixture; it now delegates to the +real filesystem instead of weakening the page-store inventory check. + +The engine root marker, image adapters for both representations, physical root +pins/reclamation and actual engine crash-cut tests remain outstanding. No +manifest can yet grant shared-group serving authority. The [component contract](../../../crates/cellar/durability/raft/cell_groups/README.md) +records this boundary. Checkpoint logs are retained under +`.cellar/cell-groups-evidence/CG-02-manifest-pages/`. The CG-02 scope includes the consensus module's unstable +verification exports so the native tests exercise the actual component. + +Clippy passes for the library and new native target with warnings denied. The +36 plan-validator tests, source preflight, native dependency isolation and docs +checks also pass. These checks do not substitute for the still-missing CG-02 +engine publication, pin reclamation and both-image-mode qualification cases. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index c1b2e65ce..0d2347dcf 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -134,7 +134,7 @@ { "id": "CG-02", "title": "Persist complete group manifests with crash-safe cell image ownership", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -150,13 +150,18 @@ "scope": [ "crates/cellar/durability/raft/", "crates/cellar-ltx/src/extent/", - "crates/cellar-ltx/src/generation.rs" + "crates/cellar-ltx/src/generation.rs", + "crates/cellar/application/consensus/mod.rs", + "crates/cellar/tests/support/raft_ltx_fault.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/mod.rs", "crates/cellar/durability/raft/cell_groups/manifest.rs", "crates/cellar/durability/raft/cell_groups/manifest_tests.rs", - "crates/cellar/tests/native_cell_groups_storage.rs" + "crates/cellar/tests/native_cell_groups_storage.rs", + "crates/cellar/durability/raft/cell_groups/page_store.rs", + "crates/cellar/durability/raft/cell_groups/page_wire.rs", + "crates/cellar/durability/raft/cell_groups/README.md" ], "test_target": "native_cell_groups_storage", "steps": [ From 4387fedd2256d4eed3e38473ac42a8b46f8e68b8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 09:51:56 -0700 Subject: [PATCH 008/568] feat: publish complete cell-group roots through Raft Engine --- crates/cellar/durability/raft/cache.rs | 95 +++++- .../durability/raft/cell_groups/README.md | 34 +- .../durability/raft/cell_groups/image_pin.rs | 79 +++++ .../durability/raft/cell_groups/manifest.rs | 8 +- .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../durability/raft/cell_groups/page_store.rs | 20 ++ .../durability/raft/cell_groups/root.rs | 259 +++++++++++++++ crates/cellar/durability/raft/records.rs | 32 +- .../cellar/durability/raft/snapshot_seed.rs | 2 + .../tests/native_cell_groups_storage.rs | 303 ++++++++++++++++++ plans/multi-raft/cell-groups/progress.md | 27 ++ plans/multi-raft/cell-groups/tasks.json | 4 +- 12 files changed, 854 insertions(+), 14 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/image_pin.rs create mode 100644 crates/cellar/durability/raft/cell_groups/root.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 711490abe..fee93281d 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -2,6 +2,7 @@ //! Durable storage views. Raft reads these bounded caches without disk I/O. +use super::cell_groups::{CellImagePin, GroupManifest, PreparedGroupManifest}; use super::member_state::GroupMembership; use super::storage::LogError; use crate::application::consensus::ImageCommitment; @@ -104,6 +105,7 @@ pub struct GroupWrite { pub conf_state: Option, pub membership: Option, pub applied: Option, + pub group_manifest: Option, pub snapshot: Option, } @@ -116,6 +118,7 @@ impl GroupWrite { conf_state: None, membership: None, applied: None, + group_manifest: None, snapshot: None, } } @@ -151,6 +154,13 @@ impl GroupWrite { .checked_add(membership.reserved_bytes()) .ok_or(LogError::Overloaded)?; } + if let Some(manifest) = &self.group_manifest { + bytes = bytes + .checked_add( + 16 * 1024 + manifest.pinned_images() * std::mem::size_of::(), + ) + .ok_or(LogError::Overloaded)?; + } Ok(bytes) } } @@ -189,7 +199,18 @@ impl GroupLog { } pub fn applied(&self) -> Result { - Ok(self.read()?.applied.clone()) + let cache = self.read()?; + if cache.group_manifest.is_some() { + return Err(LogError::Invalid( + "cell-group root requires complete-manifest recovery".into(), + )); + } + Ok(cache.applied.clone()) + } + + /// Engine-selected metadata only. Reverify every page/image before admission. + pub fn group_manifest(&self) -> Result, LogError> { + Ok(self.read()?.group_manifest.clone()) } /// Last synced bindings for this group history. This observation alone is @@ -362,6 +383,8 @@ pub(crate) struct CachedGroup { pub snapshot: Snapshot, pub entries: BTreeMap, pub applied: DurableApply, + pub group_manifest: Option, + pub manifest_pins: Option>, } impl CachedGroup { @@ -400,7 +423,9 @@ impl CachedGroup { } pub fn bytes(&self) -> usize { - 1024 + self.membership.reserved_bytes() + 1024 + self.group_manifest.as_ref().map_or(0, |m| { + 16 * 1024 + m.root().cells() as usize * std::mem::size_of::() + }) + self.membership.reserved_bytes() + self.snapshot.compute_size() as usize + self.conf_state.compute_size() as usize + self @@ -449,11 +474,35 @@ impl CachedGroup { if let Some(image) = &self.applied.image { image.validate()?; } + if let Some(manifest) = &self.group_manifest { + manifest + .check() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + if self.applied.image.is_some() + || manifest.position() != self.applied.position + || manifest.membership_revision() != self.membership.revision().get() + || self + .manifest_pins + .as_ref() + .is_some_and(|pins| pins.len() != manifest.root().cells() as usize) + { + return Err(LogError::Corrupt("incoherent cell-group root".into())); + } + } else if self.manifest_pins.is_some() { + return Err(LogError::Corrupt("image pins without group root".into())); + } Ok(()) } pub fn changed(&self, write: &GroupWrite, max_entries: usize) -> Result { let mut next = self.clone(); + if (self.group_manifest.is_some() || write.group_manifest.is_some()) + && (write.applied.is_some() || write.snapshot.is_some()) + { + return Err(LogError::Invalid( + "mixed image/group apply or unsupported group snapshot".into(), + )); + } if let Some(change) = &write.snapshot { let snapshot = match change { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, @@ -552,6 +601,46 @@ impl CachedGroup { } next.applied = applied.clone(); } + if let Some(prepared) = &write.group_manifest { + let marker = prepared.marker(); + marker + .check() + .map_err(|e| LogError::Invalid(e.to_string()))?; + if !Arc::ptr_eq(&prepared.engine_owner, &write.group.owner) + || !marker.matches_group(&write.group.identity.scope, write.group.birth_digest) + || marker.position().index < self.applied.position.index + || (marker.position().index == self.applied.position.index + && self.group_manifest.as_ref() != Some(marker)) + || (marker.position().index > self.applied.position.index + && self.applied.position.index.get().checked_add(1) + != Some(marker.position().index.get())) + || self.applied.image.is_some() + { + return Err(LogError::Invalid( + "invalid cell-group apply binding or prefix".into(), + )); + } + if let Some(previous) = &self.group_manifest { + if marker.route() != previous.route() { + return Err(LogError::Invalid( + "group route changed outside topology protocol".into(), + )); + } + } else if self.applied.position.index.get() != 1 + || self.snapshot.get_metadata().index != 1 + || !self.snapshot.data.is_empty() + { + return Err(LogError::Invalid( + "cell-group state requires a fresh group".into(), + )); + } + next.applied = DurableApply { + position: marker.position(), + image: None, + }; + next.group_manifest = Some(marker.clone()); + next.manifest_pins = Some(prepared.images.clone()); + } if let Some(conf) = &write.conf_state { if next.applied.position.index <= self.applied.position.index && conf != &self.conf_state @@ -652,6 +741,8 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { conf_state, snapshot, entries: BTreeMap::new(), + group_manifest: None, + manifest_pins: None, applied: DurableApply { position: LogPosition { index: LogIndex::new(1), diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 1bc0ec817..03d0520ae 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -2,9 +2,9 @@ # Cell-group durable state -This component is under CG-02 implementation. It currently supplies immutable -manifest pages and their filesystem owner. It is not yet connected to a Raft -engine apply marker and does not admit requests or establish durability proofs. +This component is under CG-02 implementation. It supplies immutable manifest pages, their filesystem owner, complete-root +preparation and atomic Raft Engine root publication. Production host admission +is not yet wired to these contracts; a stored root is not quorum authority. A manifest is a persistent AVL tree ordered by the namespace routing digest and full cell ID. Each bounded page holds one cell leaf and at most two child @@ -36,9 +36,31 @@ successful page traversal alone is not an image seal, Raft commit or serving capability. Updates compare the expected cell version and require increasing sequence, LTX position and group mutation position. -Next integration must sync complete cell images and manifest paths before one -Raft Engine transaction selects the complete group root and contiguous applied -position. Recovery must refuse any missing page or image. Reader, apply, +`prepare_complete` is the full-closure path for bootstrap, recovery and +checkpoint preparation. Every leaf must match a real materialized generation +pin, sealed extent image, or canonical empty image. Extent roots bind the stable +logical cell-history digest and schema. The method verifies and syncs all pages, +retains physical image pins and binds the preparation to the exact engine owner. +This full scan is not the ordinary-write path; incremental preparation remains +required before host cutover. + +The engine persists group root, applied position, membership and log/hard state +in one synced batch. It requires contiguous apply, matching group/birth/range, +matching membership revision and the current engine owner. Cached readers see +the old root until sync completes. Old single-image accessors and snapshot writes +refuse a group root; future group snapshot installation uses its own complete +closure protocol. Recovery first observes the selected metadata and then repeats +full closure verification before any serving admission. Missing images or pages +cannot become empty cells. + +The local application record now has explicit image/group semantic domains at +marker 1. Old bare-image record shapes refuse; there is no old-layout decoder or +automatic migration. The staged image variant supports existing code while the +remaining production consumers are converted, and is removed at hard cutover. +Existing disks and archives must be retained, not reset by this component. + +Further integration must add incremental preparation and authoritative physical +root retention/reclamation. Reader, apply, snapshot and transfer pins must protect dependencies before deletion. Native engine crash cuts and both physical image modes remain required before CG-02 can be marked DONE. diff --git a/crates/cellar/durability/raft/cell_groups/image_pin.rs b/crates/cellar/durability/raft/cell_groups/image_pin.rs new file mode 100644 index 000000000..c7c94cd4d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/image_pin.rs @@ -0,0 +1,79 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical image proofs used while preparing complete group roots. + +use super::{CellManifestLeaf, ManifestError}; +use cellar_logic::cell_group::{CellHistory, CellImageCommitment}; +use cellar_ltx::{generation::GenerationPin, internal::SealedImageRoot}; +use sha2::{Digest, Sha256}; +use std::sync::Arc; + +#[derive(Clone)] +pub enum CellImagePin { + Materialized(GenerationPin), + Extents(Arc), + /// Only the canonical zero-length image; never a missing-image fallback. + Empty, +} + +/// Stable across group transfer, distinct from a physical group birth digest. +pub fn cell_image_history_digest(history: &CellHistory) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-image-history-1\0"); + hash.update(history.cluster.bytes()); + hash.update(history.namespace.bytes()); + hash.update(history.incarnation.bytes()); + hash.update((history.cell.as_str().len() as u32).to_be_bytes()); + hash.update(history.cell.as_str().as_bytes()); + hash.finalize().into() +} + +impl CellImagePin { + pub(super) fn check( + &self, + leaf: &CellManifestLeaf, + schema: [u8; 32], + ) -> Result<(), ManifestError> { + let image = leaf.version.image; + let valid = match self { + Self::Materialized(pin) => { + let state = pin.state(); + image.commitment == CellImageCommitment::Materialized(state.digest) + && image.epoch == state.epoch + && image.txid == state.txid.0 + && image.checksum == state.checksum + && leaf.geometry.length == state.length + && leaf.geometry.page_size == state.page_size + } + Self::Extents(pin) => { + let manifest = pin.manifest(); + let binding = manifest.binding(); + let reference = pin.reference(); + binding.history == cell_image_history_digest(&leaf.version.history) + && binding.schema == schema + && binding.epoch == image.epoch + && binding.txid == image.txid + && binding.checksum == image.checksum + && manifest.geometry().length() == leaf.geometry.length + && manifest.geometry().page_size() == leaf.geometry.page_size + && image.commitment + == CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + } + } + Self::Empty => { + image.txid == 0 + && image.checksum == 0 + && leaf.geometry.length == 0 + && image.commitment + == CellImageCommitment::Materialized(Sha256::digest([]).into()) + } + }; + if valid { + Ok(()) + } else { + Err(ManifestError::Invalid("cell image seal")) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 2eada7eb9..1c2cc20fd 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -157,7 +157,7 @@ impl ManifestRoot { /// Retain every root used by apply/read/snapshot/transfer before reclamation. /// This layer performs no page deletion and cannot publish serving authority. pub struct ManifestTree { - route: GroupRoute, + pub(super) route: GroupRoute, limits: ManifestLimits, } @@ -308,7 +308,11 @@ impl ManifestTree { ) } - fn read(&self, store: &ManifestPageStore, reference: PageRef) -> Result { + pub(super) fn read( + &self, + store: &ManifestPageStore, + reference: PageRef, + ) -> Result { let bytes = store.read(reference.digest)?; let page = decode(&bytes, &self.route, self.limits)?; if page.reference(&bytes, self.limits)? != reference { diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 195bb8b2e..4d993a137 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -2,12 +2,17 @@ //! Complete group-state storage. A synced page is data, not Raft authority. +mod image_pin; mod manifest; mod page_store; mod page_wire; +mod root; pub use manifest::{ CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, ManifestRoot, ManifestTree, }; pub use page_store::{ManifestPageStore, ManifestStoreUsage}; + +pub use image_pin::{cell_image_history_digest, CellImagePin}; +pub use root::{GroupManifest, PreparedGroupManifest}; diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs index 605309944..ac827cfb6 100644 --- a/crates/cellar/durability/raft/cell_groups/page_store.rs +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -126,6 +126,26 @@ impl ManifestPageStore { Ok(bytes) } + pub(super) fn sync_page(&mut self, key: [u8; 32]) -> Result<(), ManifestError> { + self.read(key)?; + self.quarantined = true; + self.filesystem + .open(&self.directory.join(format!("{}.page", hex::encode(key))))? + .sync_all()?; + self.quarantined = false; + Ok(()) + } + + pub(super) fn sync_directory(&mut self) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + self.quarantined = true; + self.filesystem.sync_all(&self.directory)?; + self.quarantined = false; + Ok(()) + } + pub(super) fn put(&mut self, key: [u8; 32], bytes: &[u8]) -> Result<(), ManifestError> { if self.quarantined { return Err(ManifestError::Quarantined); diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs new file mode 100644 index 000000000..a6cfabca6 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -0,0 +1,259 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete-root preparation and engine metadata. Neither a marker observation +//! nor physical image seals establish current-term quorum/serving authority. + +use super::{ + CellImagePin, CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestRoot, + ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use std::sync::Arc; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "WireRoot", into = "WireRoot")] +pub struct GroupManifest { + pub(super) route: GroupRoute, + pub(super) position: LogPosition, + pub(super) membership_revision: u64, + pub(super) root: ManifestRoot, +} + +impl GroupManifest { + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn position(&self) -> LogPosition { + self.position + } + + pub fn membership_revision(&self) -> u64 { + self.membership_revision + } + + pub fn root(&self) -> ManifestRoot { + self.root + } + + pub(crate) fn check(&self) -> Result<(), ManifestError> { + self.position + .validate() + .map_err(|_| ManifestError::Invalid("root position"))?; + if self.position.index.get() == 0 + || self.membership_revision == 0 + || self.membership_revision > self.position.index.get() + { + return Err(ManifestError::Invalid("root membership")); + } + ManifestTree::new(self.route.clone(), ManifestLimits::default())?; + if let Some(root) = self.root.0 { + if root.cells == 0 || root.cells > 1_000_000 || root.height == 0 || root.height > 32 { + return Err(ManifestError::Invalid("root bounds")); + } + } + Ok(()) + } + + pub(crate) fn matches_group(&self, scope: &GroupScope, birth: [u8; 32]) -> bool { + self.route.history + == GroupHistory { + cluster: scope.cluster, + namespace: scope.namespace, + group: scope.group, + incarnation: scope.incarnation, + } + && self.route.birth_digest == birth + } +} + +/// A complete, synced closure. Image pins remain alive while the engine request +/// is queued or syncing. Durable-root retention must also survive uncertain +/// outcomes; physical stores must reconcile engine roots before any reclamation. +#[derive(Clone)] +pub struct PreparedGroupManifest { + pub(crate) marker: GroupManifest, + // The pins are deliberately inaccessible to mutation/publication callers. + pub(crate) images: Arc<[CellImagePin]>, + pub(crate) engine_owner: Arc<()>, +} + +impl PreparedGroupManifest { + pub fn marker(&self) -> &GroupManifest { + &self.marker + } + + pub fn pinned_images(&self) -> usize { + self.images.len() + } +} + +impl ManifestTree { + /// Full closure verification for bootstrap, recovery and checkpoint install. + /// Ordinary apply needs the incremental preparation path before host cutover. + /// The callback returns actual sealed images, never merely accepted markers. + pub fn prepare_complete( + &self, + group: &crate::raft::cache::GroupLog, + store: &mut ManifestPageStore, + root: ManifestRoot, + position: LogPosition, + membership_revision: u64, + mut image: impl FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + let marker = GroupManifest { + route: self.route.clone(), + position, + membership_revision, + root, + }; + marker.check()?; + group + .cache_revision() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + if !marker.matches_group(&group.identity().scope, group.birth_digest) { + return Err(ManifestError::Invalid("root/engine identity")); + } + let mut pins = Vec::new(); + self.verify(store, root, position, |leaf| { + let pin = image(leaf)?; + pin.check(leaf, self.route.schema_digest)?; + pins.push(pin); + Ok(()) + })?; + self.sync_pages(store, root.0)?; + store.sync_directory()?; + Ok(PreparedGroupManifest { + marker, + images: pins.into(), + engine_owner: group.owner.clone(), + }) + } + + fn sync_pages( + &self, + store: &mut ManifestPageStore, + reference: Option, + ) -> Result<(), ManifestError> { + let Some(reference) = reference else { + return Ok(()); + }; + let page = self.read(store, reference)?; + self.sync_pages(store, page.left)?; + self.sync_pages(store, page.right)?; + store.sync_page(reference.digest) + } +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireRoot { + domain: String, + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + lower: Bound, + upper: Bound, + route_epoch: u64, + birth: [u8; 32], + schema: [u8; 32], + index: u64, + term: u64, + membership_revision: u64, + #[serde(deserialize_with = "required_root")] + root: Option, +} + +fn required_root<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(tag = "bound", content = "cell", deny_unknown_fields)] +enum Bound { + NegativeInfinity, + Key(String), + PositiveInfinity, +} + +impl Bound { + fn encode(value: &CellBound) -> Self { + match value { + CellBound::NegativeInfinity => Self::NegativeInfinity, + CellBound::Key(k) => Self::Key(k.cell().as_str().into()), + CellBound::PositiveInfinity => Self::PositiveInfinity, + } + } + + fn decode(self, namespace: NamespaceId) -> Result { + Ok(match self { + Self::NegativeInfinity => CellBound::NegativeInfinity, + Self::PositiveInfinity => CellBound::PositiveInfinity, + Self::Key(s) => CellBound::Key(cell_routing_key( + namespace, + &CellId::new(s).map_err(|_| ManifestError::Invalid("range ID"))?, + )), + }) + } +} + +impl From for WireRoot { + fn from(m: GroupManifest) -> Self { + Self { + domain: "cellar-group-root-1".into(), + cluster: m.route.history.cluster.bytes(), + namespace: m.route.history.namespace.bytes(), + group: m.route.history.group.bytes(), + incarnation: m.route.history.incarnation.bytes(), + lower: Bound::encode(m.route.range.lower()), + upper: Bound::encode(m.route.range.upper()), + route_epoch: m.route.epoch.get(), + birth: m.route.birth_digest, + schema: m.route.schema_digest, + index: m.position.index.get(), + term: m.position.term.get(), + membership_revision: m.membership_revision, + root: m.root.0, + } + } +} + +impl TryFrom for GroupManifest { + type Error = ManifestError; + + fn try_from(w: WireRoot) -> Result { + if w.domain != "cellar-group-root-1" { + return Err(ManifestError::Invalid("root domain")); + } + let invalid = |_| ManifestError::Invalid("root identity"); + let namespace = NamespaceId::new(w.namespace).map_err(invalid)?; + let result = Self { + route: GroupRoute { + history: GroupHistory { + cluster: ClusterId::new(w.cluster).map_err(invalid)?, + namespace, + group: GroupId::new(w.group).map_err(invalid)?, + incarnation: IncarnationId::new(w.incarnation).map_err(invalid)?, + }, + range: CellRange::new(w.lower.decode(namespace)?, w.upper.decode(namespace)?) + .map_err(invalid)?, + epoch: RouteEpoch::new(w.route_epoch).map_err(invalid)?, + birth_digest: w.birth, + schema_digest: w.schema, + }, + position: LogPosition { + index: LogIndex::new(w.index), + term: Term::new(w.term), + }, + membership_revision: w.membership_revision, + root: ManifestRoot(w.root), + }; + result.check()?; + Ok(result) + } +} diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index d7155e21d..ebc45575e 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -95,7 +95,18 @@ pub(crate) struct GroupRecord { pub last_index: u64, pub applied_index: u64, pub applied_term: u64, - pub image: Option, + pub application: ApplicationRecord, +} + +// Explicit domains refuse the old bare-image record shape. This staged image +// variant is removed with legacy runtime consumers at the public hard cutover. +#[derive(Serialize, Deserialize)] +#[serde(tag = "domain", content = "state", deny_unknown_fields)] +pub(crate) enum ApplicationRecord { + #[serde(rename = "cellar-image-apply-1")] + Image(Option), + #[serde(rename = "cellar-group-apply-1")] + Group(Box), } impl GroupRecord { @@ -109,7 +120,10 @@ impl GroupRecord { last_index: cache.last_index(), applied_index: cache.applied.position.index.get(), applied_term: cache.applied.position.term.get(), - image: cache.applied.image.clone(), + application: match &cache.group_manifest { + Some(marker) => ApplicationRecord::Group(Box::new(marker.clone())), + None => ApplicationRecord::Image(cache.applied.image.clone()), + }, }) } @@ -117,18 +131,30 @@ impl GroupRecord { if self.mapping.format != 1 { return Err(LogError::Corrupt("unsupported group marker".into())); } + let (image, group_manifest) = match &self.application { + ApplicationRecord::Image(image) => (image.clone(), None), + ApplicationRecord::Group(marker) => { + let scope = self.mapping.scope.clone().decode().map_err(corrupt)?; + if !marker.matches_group(&scope, self.mapping.birth_digest) { + return Err(LogError::WrongIdentity); + } + (None, Some(marker.as_ref().clone())) + } + }; Ok(CachedGroup { hard_state: HardState::parse_from_bytes(&self.hard_state).map_err(corrupt)?, conf_state: ConfState::parse_from_bytes(&self.conf_state).map_err(corrupt)?, membership: self.membership.clone(), snapshot: Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?, entries: Default::default(), + group_manifest, + manifest_pins: None, applied: DurableApply { position: LogPosition { index: LogIndex::new(self.applied_index), term: Term::new(self.applied_term), }, - image: self.image.clone(), + image, }, }) } diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index 07b20f941..262ce44d0 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -293,6 +293,8 @@ impl PreparedSeed { membership: self.descriptor.membership.clone(), snapshot, entries: std::collections::BTreeMap::new(), + group_manifest: None, + manifest_pins: None, applied: crate::raft::cache::DurableApply { position: LogPosition { index: LogIndex::new(self.descriptor.index), diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index d728c84dc..33e4b8973 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -12,6 +12,10 @@ mod images; #[allow(dead_code)] mod faults; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod engine_fault; + use anyhow::Result; use cellar::application::consensus::{ verification::{cell_groups::*, CommittedImage}, @@ -274,3 +278,302 @@ fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<( assert!(!faults.synced_paths.lock().unwrap().is_empty()); Ok(()) } + +fn engine_bootstrap() -> cellar::application::consensus::verification::GroupBootstrap { + cellar::application::consensus::verification::GroupBootstrap { + scope: GroupScope { + cluster: route().history.cluster, + namespace: route().history.namespace, + group: route().history.group, + incarnation: route().history.incarnation, + cell: "storage-fixture-group".into(), + }, + member: MemberId::new(1).unwrap(), + birth_digest: route().birth_digest, + voters: vec![MemberId::new(1).unwrap()], + learners: vec![], + } +} + +fn engine_options( + path: &std::path::Path, + mode: cellar::application::consensus::verification::LogOpenMode, + process: u8, +) -> cellar::application::consensus::verification::ReplicaLogOptions { + cellar::application::consensus::verification::ReplicaLogOptions::new( + path.to_owned(), + NodeId::new([8; 16]).unwrap(), + DiskLineage::new([9; 16]).unwrap(), + ProcessGeneration::new([process; 16]).unwrap(), + mode, + ) +} + +fn root_write( + group: &cellar::application::consensus::verification::GroupLog, + prepared: PreparedGroupManifest, +) -> Result { + use raft::Storage as _; + let index = prepared.marker().position().index.get(); + let mut write = cellar::application::consensus::verification::GroupWrite::new(group); + // Storage fixture payload: application interpretation is qualified by the + // group apply owner separately; the real engine still checks term/prefix. + write.entries = vec![raft::prelude::Entry { + index, + term: 1, + data: b"manifest-cut".to_vec().into(), + ..Default::default() + }]; + let mut hard = group.initial_state()?.hard_state; + hard.commit = index; + write.hard_state = Some(hard); + write.group_manifest = Some(prepared); + Ok(write) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_manifest_engine_sync_publication() -> Result<()> { + use cellar::application::consensus::verification::{ + DurableApply, GroupWrite, LogOpenMode, ReplicaLog, + }; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = engine_fault::Gate::new(None); + let filesystem = Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }); + let node = temp.path().join("node"); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + filesystem.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let fs = Arc::new(DirectFileSystem); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES (11);", + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs.clone(), 256, 4 * 1024 * 1024)?; + let root = tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + assert!(tree + .prepare_complete(&group, &mut pages, root, position(2), 1, |_| Ok( + CellImagePin::Empty + )) + .is_err()); + let prepared = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + let expected = prepared.marker().clone(); + let stale_preparation = prepared.clone(); + let release = gate.arm(engine_fault::HOLD_SYNC); + let write = root_write(&group, prepared)?; + let selected = log.clone(); + let mut pending = + domain.spawn(async move { selected.persist(OperationId::new(1), vec![write]).await })?; + tokio::time::timeout(std::time::Duration::from_secs(5), gate.entered.notified()).await?; + assert!(group.group_manifest()?.is_none()); + assert_eq!(group.applied()?.position, position(1)); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + pending.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert!( + group.applied().is_err(), + "legacy empty-image consumers must refuse" + ); + let mut legacy = GroupWrite::new(&group); + legacy.applied = Some(DurableApply { + position: position(3), + image: None, + }); + assert!(log + .persist(OperationId::new(2), vec![legacy]) + .await + .is_err()); + // A complete closure does not permit skipping an unapplied log index. + let gap = tree.prepare_complete(&group, &mut pages, root, position(4), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + assert!(log + .persist(OperationId::new(3), vec![root_write(&group, gap)?]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + log.close().await?; + drop(group); + drop(log); + drop(pin); + drop(generations); + drop(pages); + let resumed = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + filesystem, + )) + .await??; + let group = resumed.group(&engine_bootstrap().scope)?; + let observed = group.group_manifest()?.unwrap(); + assert_eq!(observed, expected); + let mut stale = GroupWrite::new(&group); + stale.group_manifest = Some(stale_preparation); + assert!( + resumed + .persist(OperationId::new(4), vec![stale]) + .await + .is_err(), + "prepared closure belongs to the old engine instance" + ); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024)?; + let restored = tree.prepare_complete( + &group, + &mut pages, + observed.root(), + observed.position(), + observed.membership_revision(), + |_| { + generations + .adopt(images::id(1), cut.result) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("recovery image")) + }, + )?; + assert_eq!(restored.marker(), &expected); + assert_eq!(restored.pinned_images(), 1); + resumed.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_manifest_recovery_requires_every_image() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let node = temp.path().join("node"); + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut states = Vec::new(); + { + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs.clone(), 256, 4 * 1024 * 1024)?; + let mut pins = Vec::new(); + let mut root = ManifestRoot::default(); + for id in 1..=2 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + states.push(cut.result); + pins.push(pin.clone()); + root = tree.update(&mut pages, root, leaf(id, &pin, 1, u64::from(id) + 1), None)?; + let prepared = tree.prepare_complete( + &group, + &mut pages, + root, + position(u64::from(id) + 1), + 1, + |cell| { + let selected = if cell.version.history.cell.as_str() == "cell-1" { + 0 + } else { + 1 + }; + Ok(CellImagePin::Materialized(pins[selected].clone())) + }, + )?; + log.persist( + OperationId::new(u64::from(id)), + vec![root_write(&group, prepared)?], + ) + .await?; + } + log.close().await?; + } + // Only this owned fixture's image is removed. The actual engine marker and + // every manifest page survive; recovering the other cell cannot cover it. + std::fs::remove_file( + temp.path() + .join("images") + .join(format!("{}.sqlite", hex::encode(images::id(1).bytes()))), + )?; + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + let marker = group.group_manifest()?.unwrap(); + assert_eq!(marker.root().cells(), 2); + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024)?; + assert!(tree + .prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |cell| { + let id = if cell.version.history.cell.as_str() == "cell-1" { + 1 + } else { + 2 + }; + generations + .adopt(images::id(id), states[usize::from(id - 1)]) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("missing selected cell image")) + } + ) + .is_err()); + assert!(group.applied().is_err()); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 2bd146169..8b38d88ca 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -93,3 +93,30 @@ Clippy passes for the library and new native target with warnings denied. The 36 plan-validator tests, source preflight, native dependency isolation and docs checks also pass. These checks do not substitute for the still-missing CG-02 engine publication, pin reclamation and both-image-mode qualification cases. + + +### CG-02 engine publication checkpoint + +Complete-root preparation now checks each leaf against an actual image pin, +syncs the manifest closure and holds image pins through engine persistence. The +root binds group history, range/epoch, birth/schema, membership revision and the +exact applied position. Raft Engine atomically selects that root with log/hard +state and publishes its cache only after sync. Legacy image accessors refuse a +shared-group root, and normal apply cannot skip an index or change its route. + +The local application record uses explicit marker-1 image/group domains; old +bare-image record shapes refuse. No disk migration or deletion is performed. +The existing image variant remains a staging contract until the public cutover. + +New native tests exercise a stalled real-engine sync, restart, missing-cell-image +refusal and rejection of preparation from a previous engine instance. Five +manifest cases and the 19-case existing storage suite pass, including the +engine-instance check. All-target Clippy passes after boxing the larger metadata +variant; the failed lint output is retained. Final checks are recorded under +`.cellar/cell-groups-evidence/CG-02-engine-root/`. +The standalone storage child-process fixture is intentionally ignored and is +launched by its parent persistence tests. + +Incremental apply preparation, durable root/pin reclamation, root-specific process +kill cuts and full extent-mode recovery tests remain CG-02 work. The production +shared-group host, topology and final release gates remain pending. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 0d2347dcf..1d84837f0 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -161,7 +161,9 @@ "crates/cellar/tests/native_cell_groups_storage.rs", "crates/cellar/durability/raft/cell_groups/page_store.rs", "crates/cellar/durability/raft/cell_groups/page_wire.rs", - "crates/cellar/durability/raft/cell_groups/README.md" + "crates/cellar/durability/raft/cell_groups/README.md", + "crates/cellar/durability/raft/cell_groups/root.rs", + "crates/cellar/durability/raft/cell_groups/image_pin.rs" ], "test_target": "native_cell_groups_storage", "steps": [ From f4ee4a33add59f404c73b9a44741cd7857f8b8ce Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 09:55:26 -0700 Subject: [PATCH 009/568] fix: keep task evidence bound to the frozen implementation base --- plans/multi-raft/cell-groups/baseline.json | 10 +++++----- plans/multi-raft/cell-groups/plan.py | 5 ++++- plans/multi-raft/cell-groups/progress.md | 10 ++++++++++ plans/multi-raft/cell-groups/qualification.md | 4 +++- plans/multi-raft/cell-groups/test_plan.py | 14 ++++++++++++++ 5 files changed, 36 insertions(+), 7 deletions(-) diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index b49267f23..43a377132 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ee7f4a8d2dec220904f5a6860041f12c14db1cf5", + "source_revision": "4387fedd2256d4eed3e38473ac42a8b46f8e68b8", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed and qualified CG-01 at ee7f4a8d2dec220904f5a6860041f12c14db1cf5. Cell/group histories, cell tokens and bounded typed entry codec are complete at their contract boundary. Enrollment carries replayable initial LTX. Existing host/storage owners remain unchanged and await CG-02 through CG-16.", + "review_note": "2026-10-29: reviewed CG-02 engine-root checkpoint at 4387fedd2256d4eed3e38473ac42a8b46f8e68b8. The cache now atomically binds a complete group manifest to its applied prefix and membership, retains image pins during publication, and refuses legacy image access for group roots. The source anchor shows both staged state contracts. Group-record domains refuse the previous bare-image shape. Incremental preparation, reclamation, extent recovery and production host cutover remain pending; this is not CG-02 completion.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "c10a9c6eb84c3b7ba3866de68ca331456a926ad219bc276b6d767a4d7b27f770", - "line": 82, - "excerpt": "pub struct DurableApply {\n pub position: LogPosition,\n /// None denotes the canonical empty state before the schema command.\n pub image: Option,\n}\n\n#[derive(Clone, Debug)]\npub enum SnapshotChange {\n /// A verified local snapshot; retain entries beyond its included index." + "sha256": "1873c7f987f20a3de81d561c07d9117320a99ec7cacef18435ff4e776b79c801", + "line": 379, + "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option>,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", diff --git a/plans/multi-raft/cell-groups/plan.py b/plans/multi-raft/cell-groups/plan.py index 5d276da85..77586c0a2 100644 --- a/plans/multi-raft/cell-groups/plan.py +++ b/plans/multi-raft/cell-groups/plan.py @@ -342,8 +342,11 @@ def check_evidence(task, record, manifest, baseline, profiles, profile_digest, r content = git_bytes(root, revision, path) require(hashlib.sha256(content).hexdigest() == item.get("sha256"), f"Source hash mismatch: {path}") require(required_paths <= paths, f"Evidence omits source inputs: {sorted(required_paths - paths)}") + # Evidence belongs to its tested revision. A later reviewed source baseline + # must not make historical evidence depend on future edits, or stop requiring + # earlier implementation inputs. Compare every record to the frozen plan base. changed = subprocess.check_output(["git", "diff", "--name-only", "--diff-filter=ACMRT", "-z", - baseline["source_revision"], revision, "--", *RUNTIME_PATHS], cwd=root).decode().split("\0") + manifest["planned_at"], revision, "--", *RUNTIME_PATHS], cwd=root).decode().split("\0") require(set(changed) - {""} <= paths, "Evidence omits changed runtime inputs") outputs = {} for run in record.get("commands", []): diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 8b38d88ca..e4d43f760 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -120,3 +120,13 @@ launched by its parent persistence tests. Incremental apply preparation, durable root/pin reclamation, root-specific process kill cuts and full extent-mode recovery tests remain CG-02 work. The production shared-group host, topology and final release gates remain pending. + + +The engine-root checkpoint passes all-target Clippy, formatting and module layout, +24 directly selected native tests, dependency isolation and docs checks. The +reviewed source baseline is `4387fedd`. All 37 plan-validator tests pass. Updating +the baseline exposed a verifier bug: historical evidence was compared backward +against the moving review baseline. Evidence now covers every runtime change +since the frozen plan starting revision. A regression test ensures later baseline +reviews neither invalidate historical inputs nor waive earlier changed files. +This correction does not make CG-02 DONE or relax the final release-source gate. diff --git a/plans/multi-raft/cell-groups/qualification.md b/plans/multi-raft/cell-groups/qualification.md index 642279729..4113ec83a 100644 --- a/plans/multi-raft/cell-groups/qualification.md +++ b/plans/multi-raft/cell-groups/qualification.md @@ -141,7 +141,9 @@ A task record uses this shape (example values are placeholders, not evidence): `source_inputs` must include the task's pinned source anchor paths and its test target; the validator verifies each hash against the named Git revision. Include -the remaining changed sources in the record as well. Test inventory and exact +all runtime sources changed since the frozen `tasks.json` planned-at revision in +the record as well. Later source-baseline reviews do not rewrite this evidence +boundary or waive earlier implementation inputs. Test inventory and exact one-test outputs must be hashed and retained. Every command in `plan.py show` verification must appear with exit zero. Earlier failed runs stay in the run directory and are described in the final report; do not aggregate passes across diff --git a/plans/multi-raft/cell-groups/test_plan.py b/plans/multi-raft/cell-groups/test_plan.py index 02ec9bbfe..b61af8a47 100644 --- a/plans/multi-raft/cell-groups/test_plan.py +++ b/plans/multi-raft/cell-groups/test_plan.py @@ -121,6 +121,20 @@ def test_current_contract_and_index(self): self.assertEqual(stats["invariants"], 14) self.assertEqual((plan.PACKAGE / "implementation-plan.md").read_text(), plan.index_text(self.actual_manifest)) + def test_evidence_coverage_keeps_frozen_base_after_baseline_advances(self): + record = self.evidence() + self.baseline["source_revision"] = "f" * 40 + with patch.object(plan, "git_bytes", side_effect=lambda root, rev, path: path.encode()), \ + patch.object(plan.subprocess, "check_output", return_value=b"") as diff: + plan.check_evidence(self.manifest["tasks"][0], record, self.manifest, + self.baseline, self.profiles, self.profile_digest, self.root) + argv = diff.call_args.args[0] + self.assertEqual(argv[5:7], [self.manifest["planned_at"], self.revision]) + # Earlier implementation inputs stay required even when a later review + # baseline already includes them. Advancing the baseline cannot waive them. + with self.assertRaisesRegex(plan.PlanError, "omits changed runtime inputs"): + self.verify_evidence(record, b"crates/cellar/earlier-input.rs\0") + def test_dependency_order_covers_effects_and_harness(self): order = plan.ordered(self.manifest["tasks"]) self.assertLess(order.index("CG-08"), order.index("CG-07")) From b71f62b5841bab473481661bb5d7d6be9f366e29 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 11:15:59 -0700 Subject: [PATCH 010/568] feat: prepare cell-group manifests with persistent image pins --- crates/cellar/durability/raft/cache.rs | 30 +- .../durability/raft/cell_groups/README.md | 25 +- .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/page_store.rs | 2 + .../durability/raft/cell_groups/pins.rs | 153 ++++++++++ .../durability/raft/cell_groups/root.rs | 81 +++-- .../tests/native_cell_groups_storage.rs | 281 +++++++++++++++++- 7 files changed, 539 insertions(+), 35 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/pins.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index fee93281d..9610c6d82 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -2,7 +2,7 @@ //! Durable storage views. Raft reads these bounded caches without disk I/O. -use super::cell_groups::{CellImagePin, GroupManifest, PreparedGroupManifest}; +use super::cell_groups::{GroupManifest, ManifestPins, PreparedGroupManifest}; use super::member_state::GroupMembership; use super::storage::LogError; use crate::application::consensus::ImageCommitment; @@ -156,9 +156,7 @@ impl GroupWrite { } if let Some(manifest) = &self.group_manifest { bytes = bytes - .checked_add( - 16 * 1024 + manifest.pinned_images() * std::mem::size_of::(), - ) + .checked_add(16 * 1024 + manifest.reserved_bytes()) .ok_or(LogError::Overloaded)?; } Ok(bytes) @@ -384,7 +382,7 @@ pub(crate) struct CachedGroup { pub entries: BTreeMap, pub applied: DurableApply, pub group_manifest: Option, - pub manifest_pins: Option>, + pub manifest_pins: Option, } impl CachedGroup { @@ -424,7 +422,11 @@ impl CachedGroup { pub fn bytes(&self) -> usize { 1024 + self.group_manifest.as_ref().map_or(0, |m| { - 16 * 1024 + m.root().cells() as usize * std::mem::size_of::() + 16 * 1024 + + self.manifest_pins.as_ref().map_or( + ManifestPins::maximum_bytes(m.root().cells()), + ManifestPins::reserved_bytes, + ) }) + self.membership.reserved_bytes() + self.snapshot.compute_size() as usize + self.conf_state.compute_size() as usize @@ -484,7 +486,7 @@ impl CachedGroup { || self .manifest_pins .as_ref() - .is_some_and(|pins| pins.len() != manifest.root().cells() as usize) + .is_some_and(|pins| pins.root() != manifest.root()) { return Err(LogError::Corrupt("incoherent cell-group root".into())); } @@ -603,6 +605,20 @@ impl CachedGroup { } if let Some(prepared) = &write.group_manifest { let marker = prepared.marker(); + if let Some(base) = &prepared.base { + let fresh = self.group_manifest.is_none() + && base.root().cells() == 0 + && base.position() == self.applied.position + && self.applied.position.index.get() == 1; + if self.group_manifest.as_ref() != Some(marker) + && self.group_manifest.as_ref() != Some(base) + && !fresh + { + return Err(LogError::Invalid( + "cell-group preparation base changed".into(), + )); + } + } marker .check() .map_err(|e| LogError::Invalid(e.to_string()))?; diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 03d0520ae..ff8fa500d 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -41,8 +41,25 @@ checkpoint preparation. Every leaf must match a real materialized generation pin, sealed extent image, or canonical empty image. Extent roots bind the stable logical cell-history digest and schema. The method verifies and syncs all pages, retains physical image pins and binds the preparation to the exact engine owner. -This full scan is not the ordinary-write path; incremental preparation remains -required before host cutover. +This full scan establishes the capability used by ordinary writes. + +`prepare_update` compares the prior cell version, requires the next group index, +verifies the changed image seal and synchronizes the changed manifest path. A +persistent in-memory AVL pin tree mirrors the manifest. Unchanged subtrees are +shared across prepared/published roots; changed paths and rotations reuse the +unchanged cell image pins. Preparation performs O(log cells) page work and node +allocation, with O(log² cells) bounded in-memory comparisons. Cloning a root is +constant-time. `pin_cell` selects a coherent leaf/image pair from that root; read +authority still belongs to the caller. Memory admission conservatively charges +each owner for its entire tree, including cell IDs and pin-node allocations. + +Incremental capabilities bind the exact previous manifest and page-store owner. +The engine refuses a staged successor of a different branch at the same index; +an identical already-selected result remains retryable. Reopening the page store +requires full preparation again. A live old root or individual image reader +prevents generation reclamation after a newer root publishes. Stores still need +authoritative retention reconciliation before retiring inventory, especially +after an uncertain engine outcome; a disposable request pin alone is insufficient. The engine persists group root, applied position, membership and log/hard state in one synced batch. It requires contiguous apply, matching group/birth/range, @@ -59,8 +76,8 @@ automatic migration. The staged image variant supports existing code while the remaining production consumers are converted, and is removed at hard cutover. Existing disks and archives must be retained, not reset by this component. -Further integration must add incremental preparation and authoritative physical -root retention/reclamation. Reader, apply, +Further integration must add authoritative physical root retention/reclamation, +including page reclamation and uncertain publication reconciliation. Reader, apply, snapshot and transfer pins must protect dependencies before deletion. Native engine crash cuts and both physical image modes remain required before CG-02 can be marked DONE. diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 4d993a137..9ab3f982b 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -6,6 +6,7 @@ mod image_pin; mod manifest; mod page_store; mod page_wire; +mod pins; mod root; pub use manifest::{ @@ -15,4 +16,5 @@ pub use manifest::{ pub use page_store::{ManifestPageStore, ManifestStoreUsage}; pub use image_pin::{cell_image_history_digest, CellImagePin}; +pub(crate) use pins::ManifestPins; pub use root::{GroupManifest, PreparedGroupManifest}; diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs index ac827cfb6..ddee93bb5 100644 --- a/crates/cellar/durability/raft/cell_groups/page_store.rs +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -22,6 +22,7 @@ pub struct ManifestStoreUsage { } pub struct ManifestPageStore { + pub(super) owner: Arc<()>, directory: PathBuf, filesystem: Arc, inventory: BTreeMap<[u8; 32], u64>, @@ -49,6 +50,7 @@ impl ManifestPageStore { cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), directory)?; let lock = filesystem.lock_file(&directory.join("LOCK"))?; let mut store = Self { + owner: Arc::new(()), directory: directory.to_owned(), filesystem, inventory: BTreeMap::new(), diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs new file mode 100644 index 000000000..197d90df2 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -0,0 +1,153 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent physical ownership follows the authenticated AVL topology. Cloning +//! a root is constant-time; changing a cell copies only its path and rotations. + +use super::{ + page_wire::PageRef, CellImagePin, CellManifestLeaf, ManifestError, ManifestPageStore, + ManifestRoot, ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellKey, MAX_CELL_ID_BYTES}; +use std::{cmp::Ordering, sync::Arc}; + +#[derive(Clone, Default)] +pub(crate) struct ManifestPins(Option>); + +struct PinnedPage { + reference: PageRef, + key: CellKey, + leaf: CellManifestLeaf, + image: CellImagePin, + left: ManifestPins, + right: ManifestPins, + bytes: usize, +} + +impl ManifestPins { + pub(crate) fn len(&self) -> usize { + self.0.as_ref().map_or(0, |p| p.reference.cells as usize) + } + + pub(crate) fn root(&self) -> ManifestRoot { + ManifestRoot(self.0.as_ref().map(|p| p.reference)) + } + + pub(crate) fn maximum_bytes(cells: u32) -> usize { + cells as usize + * (std::mem::size_of::() + + 2 * std::mem::size_of::() + + 2 * MAX_CELL_ID_BYTES) + } + + /// Conservatively charge shared subtrees to every owner. Include both cell + /// ID allocations and Arc control blocks, not just the image handle size. + pub(crate) fn reserved_bytes(&self) -> usize { + self.0.as_ref().map_or(0, |p| p.bytes) + } + + pub(super) fn get(&self, key: &CellKey) -> Option<(&CellManifestLeaf, CellImagePin)> { + self.find(key).map(|p| (&p.leaf, p.image.clone())) + } + + fn find(&self, key: &CellKey) -> Option<&Arc> { + let mut cursor = &self.0; + while let Some(page) = cursor { + match key.cmp(&page.key) { + Ordering::Less => cursor = &page.left.0, + Ordering::Greater => cursor = &page.right.0, + Ordering::Equal => return Some(page), + } + } + None + } + + fn node( + reference: PageRef, + key: CellKey, + leaf: CellManifestLeaf, + image: CellImagePin, + left: Self, + right: Self, + ) -> Self { + let bytes = std::mem::size_of::() + + 2 * std::mem::size_of::() + + key.cell().as_str().len() + + leaf.version.history.cell.as_str().len() + + left.reserved_bytes() + + right.reserved_bytes(); + Self(Some(Arc::new(PinnedPage { + reference, + key, + leaf, + image, + left, + right, + bytes, + }))) + } + + /// The caller first verifies the complete root's ordering and applied cut. + /// Every read rechecks content addressing; the second walk cannot silently + /// substitute different leaves after that verification. + pub(super) fn complete( + tree: &ManifestTree, + store: &mut ManifestPageStore, + root: ManifestRoot, + image: &mut impl FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + let Some(reference) = root.0 else { + return Ok(Self::default()); + }; + let page = tree.read(store, reference)?; + let left = Self::complete(tree, store, ManifestRoot(page.left), image)?; + let pin = image(&page.leaf)?; + pin.check(&page.leaf, tree.route.schema_digest)?; + let right = Self::complete(tree, store, ManifestRoot(page.right), image)?; + store.sync_page(reference.digest)?; + let key = cell_routing_key( + tree.route.history.namespace, + &page.leaf.version.history.cell, + ); + Ok(Self::node(reference, key, page.leaf, pin, left, right)) + } + + /// Only accepts the root produced by the tree's compare-and-replace method. + /// Search the old tree by the new page's key so AVL rotations also find and + /// reuse unchanged subtrees. Disk reads/node allocations stay O(log cells); + /// the bounded in-memory searches take O(log² cells) comparisons. + pub(super) fn updated( + &self, + tree: &ManifestTree, + store: &ManifestPageStore, + root: ManifestRoot, + changed: &CellManifestLeaf, + image: &CellImagePin, + ) -> Result { + let Some(reference) = root.0 else { + return Ok(Self::default()); + }; + let page = tree.read(store, reference)?; + let key = cell_routing_key( + tree.route.history.namespace, + &page.leaf.version.history.cell, + ); + let previous = self.find(&key); + if let Some(old) = previous { + if old.reference == reference { + return Ok(Self(Some(old.clone()))); + } + } + let pin = if page.leaf == *changed { + image.clone() + } else { + let old = previous + .filter(|old| old.leaf == page.leaf) + .ok_or(ManifestError::Invalid("unexpected changed leaf"))?; + old.image.clone() + }; + let left = self.updated(tree, store, ManifestRoot(page.left), changed, image)?; + let right = self.updated(tree, store, ManifestRoot(page.right), changed, image)?; + Ok(Self::node(reference, key, page.leaf, pin, left, right)) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index a6cfabca6..ca19f094d 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -4,8 +4,8 @@ //! nor physical image seals establish current-term quorum/serving authority. use super::{ - CellImagePin, CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestRoot, - ManifestTree, + CellImagePin, CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestPins, + ManifestRoot, ManifestTree, }; use crate::application::consensus::cell_routing_key; use cellar_logic::{cell_group::*, consensus::*}; @@ -76,8 +76,10 @@ impl GroupManifest { pub struct PreparedGroupManifest { pub(crate) marker: GroupManifest, // The pins are deliberately inaccessible to mutation/publication callers. - pub(crate) images: Arc<[CellImagePin]>, + pub(crate) images: ManifestPins, pub(crate) engine_owner: Arc<()>, + pub(crate) base: Option, + page_owner: Arc<()>, } impl PreparedGroupManifest { @@ -88,11 +90,22 @@ impl PreparedGroupManifest { pub fn pinned_images(&self) -> usize { self.images.len() } + + /// Pin the exact image/version selected by this coherent root. This is a + /// physical read view; the caller still needs the requested read authority. + pub fn pin_cell(&self, cell: &CellId) -> Option<(&CellManifestLeaf, CellImagePin)> { + self.images + .get(&cell_routing_key(self.marker.route.history.namespace, cell)) + } + + pub(crate) fn reserved_bytes(&self) -> usize { + self.images.reserved_bytes() + } } impl ManifestTree { /// Full closure verification for bootstrap, recovery and checkpoint install. - /// Ordinary apply needs the incremental preparation path before host cutover. + /// Ordinary cell mutations use `prepare_update` instead. /// The callback returns actual sealed images, never merely accepted markers. pub fn prepare_complete( &self, @@ -116,34 +129,58 @@ impl ManifestTree { if !marker.matches_group(&group.identity().scope, group.birth_digest) { return Err(ManifestError::Invalid("root/engine identity")); } - let mut pins = Vec::new(); - self.verify(store, root, position, |leaf| { - let pin = image(leaf)?; - pin.check(leaf, self.route.schema_digest)?; - pins.push(pin); - Ok(()) - })?; - self.sync_pages(store, root.0)?; + self.verify(store, root, position, |_| Ok(()))?; + let pins = ManifestPins::complete(self, store, root, &mut image)?; store.sync_directory()?; Ok(PreparedGroupManifest { marker, - images: pins.into(), + images: pins, engine_owner: group.owner.clone(), + base: None, + page_owner: store.owner.clone(), }) } - fn sync_pages( + /// Prepare one cell mutation/enrollment at the next contiguous group index. + /// The previous complete capability supplies unchanged physical image pins; + /// only the changed image and O(log cells) page path need preparation. It + /// may be chained for staging, but engine publication still checks order. + pub fn prepare_update( &self, store: &mut ManifestPageStore, - reference: Option, - ) -> Result<(), ManifestError> { - let Some(reference) = reference else { - return Ok(()); + previous: &PreparedGroupManifest, + leaf: CellManifestLeaf, + expected: Option<&CellVersion>, + image: CellImagePin, + ) -> Result { + if previous.marker.route != self.route || !Arc::ptr_eq(&previous.page_owner, &store.owner) { + return Err(ManifestError::Invalid("root/page owner")); + } + if previous.marker.position.index.get().checked_add(1) + != Some(leaf.last_mutation.index.get()) + || leaf.last_mutation.term < previous.marker.position.term + { + return Err(ManifestError::Invalid("noncontiguous mutation")); + } + image.check(&leaf, self.route.schema_digest)?; + let root = self.update(store, previous.marker.root, leaf.clone(), expected)?; + // update synchronizes every newly referenced page before returning. All + // reused subtrees belong to a complete preparation on this store owner. + let pins = previous.images.updated(self, store, root, &leaf, &image)?; + let marker = GroupManifest { + route: self.route.clone(), + position: leaf.last_mutation, + membership_revision: previous.marker.membership_revision, + root, }; - let page = self.read(store, reference)?; - self.sync_pages(store, page.left)?; - self.sync_pages(store, page.right)?; - store.sync_page(reference.digest) + marker.check()?; + Ok(PreparedGroupManifest { + marker, + images: pins, + engine_owner: previous.engine_owner.clone(), + base: Some(previous.marker.clone()), + page_owner: store.owner.clone(), + }) } } diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index 33e4b8973..b210f2637 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Manifest-page contracts on independently sealed SQLite images. Engine marker -//! publication and complete recovery qualification are added at the next boundary. +//! Manifest and engine publication contracts on independently sealed SQLite images. +//! Full crash-cut and both-image-mode qualification remain separate gates. #![allow(clippy::disallowed_methods)] #[path = "support/raft_ltx_fixture.rs"] @@ -577,3 +577,280 @@ async fn group_manifest_recovery_requires_every_image() -> Result<()> { domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits { + max_generations: 128, + ..Default::default() + }, + )?; + let page_path = temp.path().join("pages"); + let trace = faults::Faults::new(page_path.clone(), None); + let page_fs = Arc::new(faults::FaultFileSystem { + faults: trace.clone(), + }); + let mut pages = ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pins = BTreeMap::new(); + let mut root = ManifestRoot::default(); + // All cells are part of this fixture's first complete checkpoint at index 2. + for id in 1..=64 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + let value = leaf(id, &pin, 1, 2); + root = tree.update(&mut pages, root, value.clone(), None)?; + pins.insert(value.version.history.cell, pin); + } + let original = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |cell| { + Ok(CellImagePin::Materialized( + pins[&cell.version.history.cell].clone(), + )) + })?; + log.persist( + OperationId::new(1), + vec![root_write(&group, original.clone())?], + ) + .await?; + drop(pins); + let target = CellId::new("cell-32")?; + let (old_leaf, CellImagePin::Materialized(reader)) = original.pin_cell(&target).unwrap() else { + panic!("materialized fixture"); + }; + let before = images::read(&reader)?; + let after = images::image(Some(&before), "UPDATE value SET n = 999;")?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let changed = generations.prepare(images::id(100), Some(&reader), &cut.update())?; + let next_leaf = leaf(32, &changed, 2, 3); + let written = pages.usage().written_pages; + let reads = trace.reads.load(std::sync::atomic::Ordering::Acquire); + trace.synced_paths.lock().unwrap().clear(); + assert!(tree + .prepare_update( + &mut pages, + &original, + next_leaf.clone(), + Some(&old_leaf.version), + CellImagePin::Empty + ) + .is_err()); + assert_eq!(pages.usage().written_pages, written); + let next = tree.prepare_update( + &mut pages, + &original, + next_leaf.clone(), + Some(&old_leaf.version), + CellImagePin::Materialized(changed.clone()), + )?; + assert_eq!(next.pinned_images(), 64); + assert!(pages.usage().written_pages - written <= u64::from(root.depth())); + let read_count = trace.reads.load(std::sync::atomic::Ordering::Acquire) - reads; + assert!(read_count <= 4 * usize::from(root.depth()) + 2); + assert!( + trace.synced_paths.lock().unwrap().len() <= usize::from(root.depth()) * 2, + "mutation must not sync the complete closure" + ); + assert_eq!(original.pin_cell(&target).unwrap().0, old_leaf); + assert_eq!(next.pin_cell(&target).unwrap().0, &next_leaf); + tree.verify(&pages, next.marker().root(), position(3), |cell| { + assert_eq!(next.pin_cell(&cell.version.history.cell).unwrap().0, cell); + if cell.version.history.cell != target { + assert_eq!( + original.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + } + Ok(()) + })?; + // Stage a different branch at 3 and its successor at 4. Matching the next + // index alone must never allow it to replace the selected branch at 3. + let mut branch = original.clone(); + for id in 65..=66 { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES ({id});"), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(id), None, &cut.update())?; + branch = tree.prepare_update( + &mut pages, + &branch, + leaf(id, &pin, 1, u64::from(id) - 62), + None, + CellImagePin::Materialized(pin), + )?; + } + log.persist(OperationId::new(2), vec![root_write(&group, next.clone())?]) + .await?; + assert!(log + .persist(OperationId::new(3), vec![root_write(&group, branch)?]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(next.marker())); + let mut retry = cellar::application::consensus::verification::GroupWrite::new(&group); + retry.group_manifest = Some(next.clone()); + log.persist(OperationId::new(4), vec![retry]).await?; + // Explicitly retire fixture inventory after authoritative roots are pinned. + // Only the two abandoned branch images may disappear at this point. + generations.retain_only(&[])?; + assert_eq!(generations.reclaim(128)?.files, 2); + drop(original); + assert_eq!( + generations.reclaim(128)?.files, + 0, + "old reader still pins cell-32" + ); + assert_eq!(images::read(&reader)?, before); + drop(reader); + assert_eq!( + generations.reclaim(128)?.files, + 1, + "only the superseded cell image is now reclaimable" + ); + drop(changed); + assert_eq!(generations.reclaim(128)?.files, 0); + // A previous owner cannot certify a reopened page store, even on the same + // directory. Full closure recovery must establish a fresh capability. + drop(pages); + let mut pages = ManifestPageStore::open(&page_path, page_fs, 4096, 64 * 1024 * 1024)?; + let mut candidate = next_leaf.clone(); + candidate.last_mutation = position(4); + assert!(matches!( + tree.prepare_update( + &mut pages, + &next, + candidate, + Some(&next_leaf.version), + next.pin_cell(&target).unwrap().1 + ), + Err(ManifestError::Invalid("root/page owner")) + )); + drop(next); + assert_eq!( + generations.reclaim(128)?.files, + 0, + "engine owns the selected root's images" + ); + log.close().await?; + drop(group); + drop(log); + assert_eq!(generations.reclaim(128)?.files, 64); + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn incremental_enrollment_preserves_rotated_subtrees() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + // Identical initial bytes may be physically shared; logical cell identity + // and independently ordered enrollment still remain distinct. + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut current = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| panic!("empty closure"), + )?; + let empty = current.clone(); + let mut historical = None; + for id in 1..=96 { + let before = pages.usage().written_pages; + let depth = current.marker().root().depth(); + current = tree.prepare_update( + &mut pages, + ¤t, + leaf(id, &pin, 1, u64::from(id) + 1), + None, + CellImagePin::Materialized(pin.clone()), + )?; + assert_eq!(current.pinned_images(), usize::from(id)); + assert!(pages.usage().written_pages - before <= 3 * u64::from(depth) + 3); + tree.verify( + &pages, + current.marker().root(), + current.marker().position(), + |cell| { + assert_eq!( + current.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + Ok(()) + }, + )?; + if id == 32 { + historical = Some(current.clone()); + } + } + assert_eq!(empty.pinned_images(), 0); + let historical = historical.unwrap(); + assert_eq!(historical.pinned_images(), 32); + assert!(historical.pin_cell(&CellId::new("cell-33")?).is_none()); + tree.verify( + &pages, + historical.marker().root(), + historical.marker().position(), + |cell| { + assert_eq!( + historical.pin_cell(&cell.version.history.cell).unwrap().0, + cell + ); + Ok(()) + }, + )?; + let full = tree.prepare_complete( + &group, + &mut pages, + current.marker().root(), + current.marker().position(), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + assert_eq!(full.marker(), current.marker()); + assert_eq!(full.pinned_images(), current.pinned_images()); + log.close().await?; + domain.shutdown().await; + Ok(()) +} From 1cd2c1edc8b5eca5470068200da0bd5b2e19791f Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 11:19:10 -0700 Subject: [PATCH 011/568] docs: record incremental cell-group preparation checkpoint --- plans/multi-raft/cell-groups/baseline.json | 10 +++--- plans/multi-raft/cell-groups/progress.md | 37 ++++++++++++++++++++++ plans/multi-raft/cell-groups/tasks.json | 1 + 3 files changed, 43 insertions(+), 5 deletions(-) diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 43a377132..f9eb54721 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "4387fedd2256d4eed3e38473ac42a8b46f8e68b8", + "source_revision": "b71f62b5841bab473481661bb5d7d6be9f366e29", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed CG-02 engine-root checkpoint at 4387fedd2256d4eed3e38473ac42a8b46f8e68b8. The cache now atomically binds a complete group manifest to its applied prefix and membership, retains image pins during publication, and refuses legacy image access for group roots. The source anchor shows both staged state contracts. Group-record domains refuse the previous bare-image shape. Incremental preparation, reclamation, extent recovery and production host cutover remain pending; this is not CG-02 completion.", + "review_note": "2026-10-29: reviewed incremental preparation at b71f62b5841bab473481661bb5d7d6be9f366e29. The cache now owns persistent manifest pin trees, validates their exact root reference, charges node/ID allocations, and checks the prior root for staged mutations while allowing identical selected retries. Page-store instance binding prevents reuse after reopen. Page reclamation, uncertain-publication retention, full extent recovery and production host cutover remain pending; CG-02 is not complete.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "1873c7f987f20a3de81d561c07d9117320a99ec7cacef18435ff4e776b79c801", - "line": 379, - "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option>,\n}\n" + "sha256": "6e3a022cd16636f7587dd28be3edf86bd6ee0f13778d771ff8af0f6e4cf39edc", + "line": 377, + "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index e4d43f760..eb935926a 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -130,3 +130,40 @@ against the moving review baseline. Evidence now covers every runtime change since the frozen plan starting revision. A regression test ensures later baseline reviews neither invalidate historical inputs nor waive earlier changed files. This correction does not make CG-02 DONE or relax the final release-source gate. + +### CG-02 incremental preparation checkpoint + +Ordinary cell mutations now prepare against a complete prior root without scanning +or copying all cell pins. The physical pin tree mirrors the authenticated AVL +manifest, shares unchanged subtrees and copies changed paths/rotations. Each +mutation checks the changed image, expected cell version and next group position. +Page reads and allocations are logarithmic; lookup comparisons are bounded by +O(log² cells). Root clones are constant-time and memory admission charges the +complete owned tree, including node and cell-ID allocations. + +The engine checks the exact base root, so a staged successor from a competing +branch cannot replace the selected branch merely because its index is next. +Identical already-selected roots remain retryable. Capabilities also bind the +page-store instance; reopening requires complete preparation. A coherent root can +return a selected cell leaf plus an independent physical image pin. + +Native cases cover one update among 64 real SQLite cells, bounded page reads and +writes, competing-branch refusal, identical retry, page-owner replacement, and +96 incremental enrollments across AVL rotations. Reclamation checks keep the old +image while its reader survives, reclaim it after release, and retain all 64 +selected images through the engine's root owner after caller handles are dropped. +The first incremental run exposed an overly strict retry base check; the corrected +run is retained beside that failure under ignored local evidence. + +This is not the complete pin/reclamation gate. Page reclamation, authoritative +retention across uncertain engine outcomes, root-specific process-kill cuts and +full extent-mode recovery remain CG-02 work. The production shared-group host and +the full hard-cutover goal remain pending; CG-02 stays IN_PROGRESS. + +Runtime checkpoint `b71f62b5` passes all 7 native manifest cases and 19 existing +Raft storage cases, all-target Clippy with warnings denied, formatting, layout, +native dependency isolation and docs checks. All 37 plan-validator tests and the +reviewed source preflight pass. The initial source-drift refusal correctly required +review of the changed cache anchor before advancing the baseline. Source and +executable hashes, failed/corrected test output and final check logs are retained +under `.cellar/cell-groups-evidence/CG-02-incremental-pins/`. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 1d84837f0..98db0de14 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -163,6 +163,7 @@ "crates/cellar/durability/raft/cell_groups/page_wire.rs", "crates/cellar/durability/raft/cell_groups/README.md", "crates/cellar/durability/raft/cell_groups/root.rs", + "crates/cellar/durability/raft/cell_groups/pins.rs", "crates/cellar/durability/raft/cell_groups/image_pin.rs" ], "test_target": "native_cell_groups_storage", From f41f470f40f510746ba5d8f3178b35624f3cecb2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 11:44:42 -0700 Subject: [PATCH 012/568] feat: retain cell-group roots and reclaim unpinned pages --- .../durability/raft/cell_groups/README.md | 41 +- .../cellar/durability/raft/cell_groups/mod.rs | 3 +- .../durability/raft/cell_groups/page_store.rs | 147 +++++- .../durability/raft/cell_groups/pins.rs | 23 +- .../durability/raft/cell_groups/retention.rs | 238 ++++++++++ .../durability/raft/cell_groups/root.rs | 23 +- crates/cellar/durability/raft/storage.rs | 22 + .../tests/native_cell_groups_storage.rs | 445 +++++++++++++++++- .../tests/support/raft_storage_fault.rs | 6 + 9 files changed, 912 insertions(+), 36 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/retention.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index ff8fa500d..7082f2581 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -25,9 +25,10 @@ The page store uses an injected filesystem, a lifetime exclusive lock, bounded inventory and byte/page quotas. A new page crosses file and directory sync before its reference is returned. Any ambiguous write/sync quarantines the owner until reopen. Reopened filenames are only inventory; reads verify length and digest, -and reuse verifies and syncs bytes again. Pages are never deleted in this initial -slice, so old roots remain readable. Authoritative root retention, pin-aware -reclamation and recovery of partial orphan writes remain CG-02 work. +and reuse verifies and syncs bytes again. Every prepared pin-tree node holds a +page lease and the directory lock. Dropping a store handle cannot let a replacement +owner collect pages still selected by old readers. The store explicitly bounds +retained root slots (2–4096), reserving current and pending slots for each group. `verify` traverses the entire authenticated closure, checks ordering and the selected applied cut, and delegates actual image verification/pinning to its @@ -61,6 +62,32 @@ prevents generation reclamation after a newer root publishes. Stores still need authoritative retention reconciliation before retiring inventory, especially after an uncertain engine outcome; a disposable request pin alone is insufficient. +After validating an engine batch, storage reserves physical root pins before +starting the write. Unarmed reservations can be cancelled without persistence. +Once armed, dropping the reservation before cache publication finishes quarantines +the retention owner and keeps both old and candidate closures. Successful cache +publication selects the candidate and releases the old engine reference. Identical +retries use the same retention protocol. No retention mutex spans engine or file I/O. + +`prepare_complete` automatically registers a recovered root only when it matches +the engine-selected marker. An existing group cannot publish through an unrecovered +retention owner. Before collection, the owning service must prepare **every** +engine-selected root assigned to this directory and call `reconcile_retained` with +the complete additional snapshot/archive/transition root set. This is a trusted +inventory handoff, not discovery from page filenames or proof supplied by clients. +The future shared host must obtain that inventory from its authoritative catalog. +Replacing the additional set never releases the engine's current roots. Pending +publication, an uncertain outcome or engine closure prevents reconciliation and +collection until restart. Reopening requires recovery and reconciliation again. + +`reclaim` scans at most the explicit scan limit (1–4096) and resumes from its last +inventory key. Only pages without physical leases can be selected. Deletions and +the directory barrier finish before page/byte quotas are refunded; failure keeps +the prior charge and quarantines the owner. Reader, apply, snapshot and transition +pins independently protect their closures. Image-store collection still follows +its own complete durable-reference reconciliation contract; the retained manifest +pins prevent it from deleting selected or uncertain images in the live process. + The engine persists group root, applied position, membership and log/hard state in one synced batch. It requires contiguous apply, matching group/birth/range, matching membership revision and the current engine owner. Cached readers see @@ -76,8 +103,6 @@ automatic migration. The staged image variant supports existing code while the remaining production consumers are converted, and is removed at hard cutover. Existing disks and archives must be retained, not reset by this component. -Further integration must add authoritative physical root retention/reclamation, -including page reclamation and uncertain publication reconciliation. Reader, apply, -snapshot and transfer pins must protect dependencies before deletion. Native -engine crash cuts and both physical image modes remain required before CG-02 -can be marked DONE. +Further integration must wire complete catalog/checkpoint/transition inventory +handoff into the shared host. Native engine process-kill cuts and full qualification +of both physical image modes remain required before CG-02 can be marked DONE. diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 9ab3f982b..668a966da 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -7,13 +7,14 @@ mod manifest; mod page_store; mod page_wire; mod pins; +mod retention; mod root; pub use manifest::{ CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, ManifestRoot, ManifestTree, }; -pub use page_store::{ManifestPageStore, ManifestStoreUsage}; +pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; pub use image_pin::{cell_image_history_digest, CellImagePin}; pub(crate) use pins::ManifestPins; diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs index ddee93bb5..51dc6cc54 100644 --- a/crates/cellar/durability/raft/cell_groups/page_store.rs +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -6,6 +6,8 @@ use super::{ manifest::ManifestError, page_wire::{digest, MAX_PAGE_BYTES}, + retention::RootRetentions, + PreparedGroupManifest, }; use cellar_ltx::{FileSystem, HostFileLock}; use std::{ @@ -22,35 +24,62 @@ pub struct ManifestStoreUsage { } pub struct ManifestPageStore { - pub(super) owner: Arc<()>, + pub(super) owner: Arc, + pub(super) retention: Arc, directory: PathBuf, filesystem: Arc, - inventory: BTreeMap<[u8; 32], u64>, + inventory: BTreeMap<[u8; 32], PageEntry>, usage: ManifestStoreUsage, max_pages: usize, max_bytes: u64, quarantined: bool, + collection_cursor: Option<[u8; 32]>, +} + +pub(super) struct PageOwner { _lock: HostFileLock, } +struct PageEntry { + length: u64, + lease: Arc<()>, +} + +/// A page lease also keeps its directory locked. A replacement store cannot +/// collect pages still selected by readers from a previous owner. +pub(super) struct ManifestPagePin { + _owner: Arc, + _lease: Arc<()>, +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct ManifestReclaimReport { + pub scanned: usize, + pub pages: usize, + pub bytes: u64, +} + impl ManifestPageStore { pub fn open( directory: &Path, filesystem: Arc, max_pages: usize, max_bytes: u64, + max_retained_roots: usize, ) -> Result { if max_pages == 0 || max_pages > 4_000_000 || max_bytes == 0 || max_bytes > 64 * 1024 * 1024 * 1024 + || !(2..=4096).contains(&max_retained_roots) { return Err(ManifestError::Limit); } cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), directory)?; let lock = filesystem.lock_file(&directory.join("LOCK"))?; let mut store = Self { - owner: Arc::new(()), + owner: Arc::new(PageOwner { _lock: lock }), + retention: Arc::new(RootRetentions::new(max_retained_roots)), directory: directory.to_owned(), filesystem, inventory: BTreeMap::new(), @@ -58,7 +87,7 @@ impl ManifestPageStore { max_pages, max_bytes, quarantined: false, - _lock: lock, + collection_cursor: None, }; for entry in store.filesystem.read_dir_iter(directory)? { let entry = entry?; @@ -95,7 +124,17 @@ impl ManifestPageStore { if store.usage.bytes > max_bytes { return Err(ManifestError::Limit); } - if store.inventory.insert(key, metadata.len).is_some() { + if store + .inventory + .insert( + key, + PageEntry { + length: metadata.len, + lease: Arc::new(()), + }, + ) + .is_some() + { return Err(ManifestError::Invalid("duplicate page")); } } @@ -107,6 +146,94 @@ impl ManifestPageStore { self.usage } + /// Reconcile this directory's complete authoritative root inventory before + /// collection. First prepare every engine-selected root through + /// `prepare_complete`; the engine retains those roots automatically. Supply + /// all additional snapshot/archive/transition roots here, even without live + /// readers. Calling again replaces only that additional retained set. + /// Missing roots must refuse recovery, never be omitted to permit collection. + pub fn reconcile_retained( + &mut self, + roots: &[PreparedGroupManifest], + ) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if roots.len() > 4096 { + return Err(ManifestError::Limit); + } + if roots + .iter() + .any(|root| !Arc::ptr_eq(&root.page_owner, &self.owner)) + { + return Err(ManifestError::Invalid("retained root/page owner")); + } + self.retention + .reconcile(roots.iter().map(|root| root.images.clone()).collect()) + } + + /// Scan a bounded inventory window and remove unpinned orphan pages. Root + /// preparation requires this store's exclusive mutable borrow, so no lease + /// can appear for a selected page during deletion. Quotas are refunded only + /// after the directory barrier; any uncertain failure quarantines the owner. + pub fn reclaim(&mut self, scan_limit: usize) -> Result { + use std::ops::Bound::{Excluded, Unbounded}; + if self.quarantined { + return Err(ManifestError::Quarantined); + } + if !(1..=4096).contains(&scan_limit) { + return Err(ManifestError::Limit); + } + self.retention.can_reclaim()?; + let start = self.collection_cursor.map_or(Unbounded, Excluded); + let mut report = ManifestReclaimReport::default(); + let mut last = None; + let mut selected = Vec::new(); + for (key, entry) in self.inventory.range((start, Unbounded)).take(scan_limit) { + report.scanned += 1; + last = Some(*key); + if Arc::strong_count(&entry.lease) == 1 { + selected.push(*key); + report.bytes += entry.length; + } + } + if !selected.is_empty() { + self.quarantined = true; + for key in &selected { + self.filesystem + .remove_file(&self.directory.join(format!("{}.page", hex::encode(key))))?; + } + self.filesystem.sync_all(&self.directory)?; + for key in &selected { + self.inventory.remove(key); + } + report.pages = selected.len(); + self.usage.pages -= report.pages; + self.usage.bytes -= report.bytes; + self.quarantined = false; + } + self.collection_cursor = if report.scanned < scan_limit { + None + } else { + last + }; + Ok(report) + } + + pub(super) fn pin(&self, key: [u8; 32]) -> Result { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + let entry = self + .inventory + .get(&key) + .ok_or(ManifestError::Invalid("missing page"))?; + Ok(ManifestPagePin { + _owner: self.owner.clone(), + _lease: entry.lease.clone(), + }) + } + pub(super) fn read(&self, key: [u8; 32]) -> Result, ManifestError> { if self.quarantined { return Err(ManifestError::Quarantined); @@ -118,7 +245,7 @@ impl ManifestPageStore { let path = self.directory.join(format!("{}.page", hex::encode(key))); let mut file = self.filesystem.open(&path)?; let length = file.file_len()?; - if length != *expected || length > MAX_PAGE_BYTES as u64 { + if length != expected.length || length > MAX_PAGE_BYTES as u64 { return Err(ManifestError::Invalid("page length")); } let bytes = file.read_exact_at(0, length as usize)?; @@ -182,7 +309,13 @@ impl ManifestPageStore { file.write_all(bytes)?; file.sync_all()?; self.filesystem.sync_all(&self.directory)?; - self.inventory.insert(key, bytes.len() as u64); + self.inventory.insert( + key, + PageEntry { + length: bytes.len() as u64, + lease: Arc::new(()), + }, + ); self.usage.pages += 1; self.usage.bytes = next_bytes; self.usage.written_pages = self diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs index 197d90df2..64c80d042 100644 --- a/crates/cellar/durability/raft/cell_groups/pins.rs +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -15,6 +15,7 @@ use std::{cmp::Ordering, sync::Arc}; pub(crate) struct ManifestPins(Option>); struct PinnedPage { + _page: super::page_store::ManifestPagePin, reference: PageRef, key: CellKey, leaf: CellManifestLeaf, @@ -69,6 +70,7 @@ impl ManifestPins { image: CellImagePin, left: Self, right: Self, + page: super::page_store::ManifestPagePin, ) -> Self { let bytes = std::mem::size_of::() + 2 * std::mem::size_of::() @@ -77,6 +79,7 @@ impl ManifestPins { + left.reserved_bytes() + right.reserved_bytes(); Self(Some(Arc::new(PinnedPage { + _page: page, reference, key, leaf, @@ -109,7 +112,15 @@ impl ManifestPins { tree.route.history.namespace, &page.leaf.version.history.cell, ); - Ok(Self::node(reference, key, page.leaf, pin, left, right)) + Ok(Self::node( + reference, + key, + page.leaf, + pin, + left, + right, + store.pin(reference.digest)?, + )) } /// Only accepts the root produced by the tree's compare-and-replace method. @@ -148,6 +159,14 @@ impl ManifestPins { }; let left = self.updated(tree, store, ManifestRoot(page.left), changed, image)?; let right = self.updated(tree, store, ManifestRoot(page.right), changed, image)?; - Ok(Self::node(reference, key, page.leaf, pin, left, right)) + Ok(Self::node( + reference, + key, + page.leaf, + pin, + left, + right, + store.pin(reference.digest)?, + )) } } diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs new file mode 100644 index 000000000..b2f696b9f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -0,0 +1,238 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical retention around engine publication. An uncertain write keeps both +//! closures and disables collection until a new owner recovers durable markers. + +use super::{GroupManifest, ManifestError, ManifestPins, PreparedGroupManifest}; +use crate::raft::cache::GroupLog; +use std::{ + collections::BTreeMap, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, MutexGuard, + }, +}; + +type GroupKey = [[u8; 16]; 4]; + +struct Root { + marker: GroupManifest, + _pins: ManifestPins, +} + +struct Group { + engine: Arc<()>, + closed: Arc, + current: Option, + pending: Option, +} + +#[derive(Default)] +struct State { + groups: BTreeMap, + retained: Vec, + reconciled: bool, + quarantined: bool, +} + +pub(super) struct RootRetentions { + state: Mutex, + max_roots: usize, +} + +fn key(marker: &GroupManifest) -> GroupKey { + let h = marker.route().history; + [ + h.cluster.bytes(), + h.namespace.bytes(), + h.group.bytes(), + h.incarnation.bytes(), + ] +} + +impl RootRetentions { + pub(super) fn new(max_roots: usize) -> Self { + Self { + state: Mutex::new(State::default()), + max_roots, + } + } + + fn state(&self) -> Result, ManifestError> { + let state = self.state.lock().map_err(|_| ManifestError::Quarantined)?; + if state.quarantined { + return Err(ManifestError::Quarantined); + } + Ok(state) + } + + fn admit_group(&self, state: &State, key: GroupKey) -> Result<(), ManifestError> { + // Reserve current + pending slots before a group can submit a write. + if !state.groups.contains_key(&key) + && (state.groups.len() + 1) * 2 + state.retained.len() > self.max_roots + { + return Err(ManifestError::Limit); + } + Ok(()) + } + + pub(super) fn recover( + &self, + group: &GroupLog, + marker: &GroupManifest, + pins: &ManifestPins, + ) -> Result<(), ManifestError> { + if group.closed.load(Ordering::Acquire) { + return Err(ManifestError::Quarantined); + } + let mut state = self.state()?; + let key = key(marker); + self.admit_group(&state, key)?; + let selected = state.groups.entry(key).or_insert_with(|| Group { + engine: group.owner.clone(), + closed: group.closed.clone(), + current: None, + pending: None, + }); + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.pending.is_some() + || selected + .current + .as_ref() + .is_some_and(|r| r.marker != *marker) + { + return Err(ManifestError::Conflict); + } + if selected.current.is_none() { + selected.current = Some(Root { + marker: marker.clone(), + _pins: pins.clone(), + }); + } + Ok(()) + } + + pub(super) fn begin( + self: &Arc, + group: &GroupLog, + prepared: &PreparedGroupManifest, + previous: Option<&GroupManifest>, + ) -> Result { + if group.closed.load(Ordering::Acquire) { + return Err(ManifestError::Quarantined); + } + let mut state = self.state()?; + let key = key(prepared.marker()); + self.admit_group(&state, key)?; + if !state.groups.contains_key(&key) && previous.is_some() { + return Err(ManifestError::Invalid( + "selected root has not been recovered", + )); + } + let selected = state.groups.entry(key).or_insert_with(|| Group { + engine: group.owner.clone(), + closed: group.closed.clone(), + current: None, + pending: None, + }); + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.pending.is_some() + || selected.current.as_ref().map(|r| &r.marker) != previous + { + return Err(ManifestError::Conflict); + } + selected.pending = Some(Root { + marker: prepared.marker().clone(), + _pins: prepared.images.clone(), + }); + Ok(PublicationPin { + owner: self.clone(), + key, + armed: false, + finished: false, + }) + } + + pub(super) fn reconcile(&self, retained: Vec) -> Result<(), ManifestError> { + let mut state = self.state()?; + if state + .groups + .values() + .any(|g| g.closed.load(Ordering::Acquire)) + { + return Err(ManifestError::Quarantined); + } + if state.groups.values().any(|g| g.pending.is_some()) { + return Err(ManifestError::Conflict); + } + if state.groups.len() * 2 + retained.len() > self.max_roots { + return Err(ManifestError::Limit); + } + state.retained = retained; + state.reconciled = true; + Ok(()) + } + + pub(super) fn can_reclaim(&self) -> Result<(), ManifestError> { + let state = self.state()?; + if state + .groups + .values() + .any(|g| g.closed.load(Ordering::Acquire)) + { + return Err(ManifestError::Quarantined); + } + if !state.reconciled { + return Err(ManifestError::Invalid("root inventory not reconciled")); + } + if state.groups.values().any(|g| g.pending.is_some()) { + return Err(ManifestError::Conflict); + } + Ok(()) + } +} + +pub(crate) struct PublicationPin { + owner: Arc, + key: GroupKey, + armed: bool, + finished: bool, +} + +impl PublicationPin { + pub(crate) fn arm(&mut self) { + self.armed = true; + } + + pub(crate) fn commit(mut self) -> Result<(), ManifestError> { + { + let mut state = self.owner.state()?; + let group = state + .groups + .get_mut(&self.key) + .ok_or(ManifestError::Conflict)?; + group.current = Some(group.pending.take().ok_or(ManifestError::Conflict)?); + } + self.finished = true; + Ok(()) + } +} + +impl Drop for PublicationPin { + fn drop(&mut self) { + if self.finished { + return; + } + let mut state = self.owner.state.lock().unwrap_or_else(|e| e.into_inner()); + if self.armed { + // Do not release either old or new image/page pins after an unknown + // engine outcome, including unwinding between sync and publication. + state.quarantined = true; + } else if let Some(group) = state.groups.get_mut(&self.key) { + group.pending = None; + if group.current.is_none() { + state.groups.remove(&self.key); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index ca19f094d..a6121f44b 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -79,7 +79,8 @@ pub struct PreparedGroupManifest { pub(crate) images: ManifestPins, pub(crate) engine_owner: Arc<()>, pub(crate) base: Option, - page_owner: Arc<()>, + pub(super) page_owner: Arc, + retention: Arc, } impl PreparedGroupManifest { @@ -101,6 +102,14 @@ impl PreparedGroupManifest { pub(crate) fn reserved_bytes(&self) -> usize { self.images.reserved_bytes() } + + pub(crate) fn begin_publication( + &self, + group: &crate::raft::cache::GroupLog, + previous: Option<&GroupManifest>, + ) -> Result { + self.retention.begin(group, self, previous) + } } impl ManifestTree { @@ -132,12 +141,23 @@ impl ManifestTree { self.verify(store, root, position, |_| Ok(()))?; let pins = ManifestPins::complete(self, store, root, &mut image)?; store.sync_directory()?; + // Recover physical ownership only from the engine's selected marker. + // A prospective checkpoint or mutation remains a preparation capability. + if group + .group_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .as_ref() + == Some(&marker) + { + store.retention.recover(group, &marker, &pins)?; + } Ok(PreparedGroupManifest { marker, images: pins, engine_owner: group.owner.clone(), base: None, page_owner: store.owner.clone(), + retention: store.retention.clone(), }) } @@ -180,6 +200,7 @@ impl ManifestTree { engine_owner: previous.engine_owner.clone(), base: Some(previous.marker.clone()), page_owner: store.owner.clone(), + retention: store.retention.clone(), }) } } diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index de2316034..63c459983 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1172,6 +1172,23 @@ impl Worker { .map_err(engine_error)?; candidates.push(next); } + // Retain both closures before any engine I/O. Failed preflight drops + // unarmed reservations; once armed, an unknown outcome keeps both roots + // and quarantines collection until recovery reconstructs the selection. + let mut publications = Vec::new(); + for write in &writes { + if let Some(prepared) = &write.group_manifest { + let previous = write.group.group_manifest()?; + publications.push( + prepared + .begin_publication(&write.group, previous.as_ref()) + .map_err(|e| LogError::Invalid(e.to_string()))?, + ); + } + } + for publication in &mut publications { + publication.arm(); + } let sequence = self.fence(&mut batch)?; // No cache write lock is held during engine I/O. Readers may continue // to observe the previous durable prefix while a sync is stalled. @@ -1191,6 +1208,11 @@ impl Worker { .cache_revision .store(sequence, Ordering::Release); } + for publication in publications { + publication + .commit() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + } self.cache_bytes = next_bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); self.performance diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index b210f2637..a7e0de1bd 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -103,6 +103,7 @@ fn manifest_updates_only_changed_path() -> Result<()> { fs.clone(), 4096, 64 * 1024 * 1024, + 64, )?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; let mut root = ManifestRoot::default(); @@ -153,7 +154,8 @@ fn manifest_updates_only_changed_path() -> Result<()> { Ok(()) })?; drop(pages); - let reopened = ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024)?; + let reopened = + ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024, 64)?; assert_eq!(tree.get(&reopened, root, &target)?.unwrap(), value); tree.verify(&reopened, root, position(65), |_| Ok(()))?; Ok(()) @@ -172,7 +174,7 @@ fn manifest_rejects_conflicts_bounds_and_missing_pages() -> Result<()> { let cut = images::Cut::new(&[], &bytes, 0, true, false); let pin = generations.prepare(images::id(1), None, &cut.update())?; let value = leaf(1, &pin, 1, 1); - let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024)?; + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024, 64)?; let tree = ManifestTree::new( route(), ManifestLimits { @@ -223,8 +225,8 @@ fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<( let first = generations.prepare(images::id(1), None, &cut.update())?; let second = generations.prepare(images::id(2), None, &cut.update())?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; - let mut a = ManifestPageStore::open(&temp.path().join("a"), fs.clone(), 64, 1024 * 1024)?; - let mut b = ManifestPageStore::open(&temp.path().join("b"), fs, 64, 1024 * 1024)?; + let mut a = ManifestPageStore::open(&temp.path().join("a"), fs.clone(), 64, 1024 * 1024, 64)?; + let mut b = ManifestPageStore::open(&temp.path().join("b"), fs, 64, 1024 * 1024, 64)?; let a_root = tree.update(&mut a, ManifestRoot::default(), leaf(1, &first, 1, 1), None)?; let b_root = tree.update( &mut b, @@ -242,7 +244,7 @@ fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<( let filesystem = Arc::new(faults::FaultFileSystem { faults: faults.clone(), }); - let mut pages = ManifestPageStore::open(&directory, filesystem.clone(), 64, 1024 * 1024)?; + let mut pages = ManifestPageStore::open(&directory, filesystem.clone(), 64, 1024 * 1024, 64)?; faults.mode.store( faults::FAIL_DIRECTORY_SYNC, std::sync::atomic::Ordering::Release, @@ -265,7 +267,7 @@ fn manifest_ignores_local_image_names_and_quarantines_sync_failure() -> Result<( Err(ManifestError::Quarantined) )); drop(pages); - let mut recovered = ManifestPageStore::open(&directory, filesystem, 64, 1024 * 1024)?; + let mut recovered = ManifestPageStore::open(&directory, filesystem, 64, 1024 * 1024, 64)?; // Retried bytes must be independently checked and synced, even if the // interrupted attempt left a complete content-addressed filename. let recovered_root = tree.update( @@ -363,8 +365,13 @@ async fn group_manifest_engine_sync_publication() -> Result<()> { let cut = images::Cut::new(&[], &bytes, 0, true, false); let pin = generations.prepare(images::id(1), None, &cut.update())?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; - let mut pages = - ManifestPageStore::open(&temp.path().join("pages"), fs.clone(), 256, 4 * 1024 * 1024)?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 64, + )?; let root = tree.update( &mut pages, ManifestRoot::default(), @@ -448,7 +455,8 @@ async fn group_manifest_engine_sync_publication() -> Result<()> { fs.clone(), GenerationLimits::default(), )?; - let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024)?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 64)?; let restored = tree.prepare_complete( &group, &mut pages, @@ -493,8 +501,13 @@ async fn group_manifest_recovery_requires_every_image() -> Result<()> { fs.clone(), GenerationLimits::default(), )?; - let mut pages = - ManifestPageStore::open(&temp.path().join("pages"), fs.clone(), 256, 4 * 1024 * 1024)?; + let mut pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 64, + )?; let mut pins = Vec::new(); let mut root = ManifestRoot::default(); for id in 1..=2 { @@ -551,7 +564,8 @@ async fn group_manifest_recovery_requires_every_image() -> Result<()> { fs.clone(), GenerationLimits::default(), )?; - let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024)?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 64)?; assert!(tree .prepare_complete( &group, @@ -606,7 +620,8 @@ async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> let page_fs = Arc::new(faults::FaultFileSystem { faults: trace.clone(), }); - let mut pages = ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024)?; + let mut pages = + ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024, 64)?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; let mut pins = BTreeMap::new(); let mut root = ManifestRoot::default(); @@ -729,10 +744,19 @@ async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> ); drop(changed); assert_eq!(generations.reclaim(128)?.files, 0); - // A previous owner cannot certify a reopened page store, even on the same - // directory. Full closure recovery must establish a fresh capability. + // Live page pins keep the directory lock even after the store handle drops. + // A different owner also cannot accept this preparation capability. drop(pages); - let mut pages = ManifestPageStore::open(&page_path, page_fs, 4096, 64 * 1024 * 1024)?; + assert!( + ManifestPageStore::open(&page_path, page_fs.clone(), 4096, 64 * 1024 * 1024, 64).is_err() + ); + let mut pages = ManifestPageStore::open( + &temp.path().join("other-pages"), + page_fs.clone(), + 4096, + 64 * 1024 * 1024, + 64, + )?; let mut candidate = next_leaf.clone(); candidate.last_mutation = position(4); assert!(matches!( @@ -755,6 +779,7 @@ async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> drop(group); drop(log); assert_eq!(generations.reclaim(128)?.files, 64); + let _reopened = ManifestPageStore::open(&page_path, page_fs, 4096, 64 * 1024 * 1024, 64)?; domain.shutdown().await; Ok(()) } @@ -784,7 +809,7 @@ async fn incremental_enrollment_preserves_rotated_subtrees() -> Result<()> { let cut = images::Cut::new(&[], &bytes, 0, true, false); let pin = generations.prepare(images::id(1), None, &cut.update())?; let mut pages = - ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024)?; + ManifestPageStore::open(&temp.path().join("pages"), fs, 4096, 64 * 1024 * 1024, 64)?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; let mut current = tree.prepare_complete( &group, @@ -854,3 +879,389 @@ async fn incremental_enrollment_preserves_rotated_subtrees() -> Result<()> { domain.shutdown().await; Ok(()) } + +fn collect_pages(pages: &mut ManifestPageStore) -> Result { + let mut deleted = 0; + for _ in 0..257 { + let report = pages.reclaim(16)?; + assert!(report.scanned <= 16); + deleted += report.pages; + if report.scanned < 16 { + return Ok(deleted); + } + } + anyhow::bail!("bounded fixture inventory did not finish scanning") +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_pin_reclamation() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let gate = engine_fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let root = tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + let old = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + log.persist(OperationId::new(1), vec![root_write(&group, old.clone())?]) + .await?; + assert!( + pages.reclaim(16).is_err(), + "inventory must first be reconciled" + ); + let reader = old.clone(); + let transition = old.clone(); + // The stored snapshot reference survives release of the caller's handle. + assert!(matches!( + pages.reconcile_retained(&vec![old.clone(); 15]), + Err(ManifestError::Limit) + )); + assert!( + pages.reclaim(16).is_err(), + "failed reconciliation must not open collection" + ); + pages.reconcile_retained(std::slice::from_ref(&old))?; + let orphan = tree.update(&mut pages, root, leaf(3, &pin, 1, 3), None)?; + let next = tree.prepare_update( + &mut pages, + &old, + leaf(2, &pin, 1, 3), + None, + CellImagePin::Materialized(pin.clone()), + )?; + let release = gate.arm(engine_fault::HOLD_SYNC); + let writer = log.clone(); + let write = root_write(&group, next.clone())?; + let pending = + domain.spawn(async move { writer.persist(OperationId::new(2), vec![write]).await })?; + tokio::time::timeout(std::time::Duration::from_secs(5), gate.entered.notified()).await?; + assert!(matches!(pages.reclaim(16), Err(ManifestError::Conflict))); + assert!( + pages.reconcile_retained(&[]).is_err(), + "cannot retire roots during publication" + ); + drop(release); + pending.await??; + assert!( + collect_pages(&mut pages)? > 0, + "unmarked paths are reclaimable" + ); + assert!(tree + .verify(&pages, orphan, position(3), |_| Ok(())) + .is_err()); + tree.verify(&pages, root, position(2), |_| Ok(()))?; + drop(old); + drop(reader); + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + tree.verify(&pages, root, position(2), |_| Ok(()))?; + pages.reconcile_retained(std::slice::from_ref(&transition))?; + drop(transition); + collect_pages(&mut pages)?; + tree.verify(&pages, root, position(2), |_| Ok(()))?; + pages.reconcile_retained(&[])?; + assert!(collect_pages(&mut pages)? > 0); + assert!(tree.verify(&pages, root, position(2), |_| Ok(())).is_err()); + tree.verify(&pages, next.marker().root(), position(3), |_| Ok(()))?; + assert_eq!(pages.usage().pages, 2); + drop(next); + assert_eq!( + collect_pages(&mut pages)?, + 0, + "engine-selected root survives caller release" + ); + log.close().await?; + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn uncertain_manifest_publication_retains_both_roots() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let gate = engine_fault::Gate::new(None); + let node = temp.path().join("node"); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Create, 1), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let image_path = temp.path().join("images"); + let page_path = temp.path().join("pages"); + let generations = + GenerationStore::open(image_path.clone(), fs.clone(), GenerationLimits::default())?; + let before = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &before, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let mut pages = ManifestPageStore::open(&page_path, fs.clone(), 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let old_leaf = leaf(1, &pin, 1, 2); + let root = tree.update(&mut pages, ManifestRoot::default(), old_leaf.clone(), None)?; + let current = tree.prepare_complete(&group, &mut pages, root, position(2), 1, |_| { + Ok(CellImagePin::Materialized(pin.clone())) + })?; + log.persist( + OperationId::new(1), + vec![root_write(&group, current.clone())?], + ) + .await?; + pages.reconcile_retained(&[])?; + let after = images::image(Some(&before), "UPDATE value SET n = 2;")?; + let changed = images::Cut::new(&before, &after, 1, true, false); + let next_pin = generations.prepare(images::id(2), Some(&pin), &changed.update())?; + let next = tree.prepare_update( + &mut pages, + ¤t, + leaf(1, &next_pin, 2, 3), + Some(&old_leaf.version), + CellImagePin::Materialized(next_pin.clone()), + )?; + let expected = next.marker().clone(); + drop(pin); + drop(next_pin); + drop(current); + generations.retain_only(&[])?; + let release = gate.arm(engine_fault::FAIL_AFTER_SYNC); + assert!(log + .persist(OperationId::new(2), vec![root_write(&group, next)?]) + .await + .is_err()); + drop(release); + assert!(log.close().await.is_err()); + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + assert!(pages.reconcile_retained(&[]).is_err()); + assert_eq!( + generations.reclaim(16)?.files, + 0, + "both images survive loss of request and engine authority" + ); + drop(group); + drop(log); + // The page owner, rather than the failed engine cache, still retains both. + assert_eq!(generations.reclaim(16)?.files, 0); + drop(pages); + drop(generations); + let log = domain + .run(ReplicaLog::open( + engine_options(&node, LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + assert_eq!( + group.group_manifest()?.as_ref(), + Some(&expected), + "injected failure was after actual fsync" + ); + let generations = GenerationStore::open(image_path, fs.clone(), GenerationLimits::default())?; + let mut pages = ManifestPageStore::open(&page_path, fs, 256, 4 * 1024 * 1024, 16)?; + assert!(pages.reclaim(16).is_err()); + let restored = tree.prepare_complete( + &group, + &mut pages, + expected.root(), + expected.position(), + expected.membership_revision(), + |_| { + generations + .adopt(images::id(2), changed.result) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("recovered image")) + }, + )?; + pages.reconcile_retained(&[])?; + assert_eq!(collect_pages(&mut pages)?, 1); + generations.retain_only(&[])?; + assert_eq!(generations.reclaim(16)?.files, 1); + let (_, CellImagePin::Materialized(restored_image)) = + restored.pin_cell(&CellId::new("cell-1")?).unwrap() + else { + panic!("materialized fixture"); + }; + assert_eq!(images::read(&restored_image)?, after); + drop(restored_image); + drop(restored); + assert_eq!( + generations.reclaim(16)?.files, + 0, + "recovery registers selected root retention without a new write" + ); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[test] +fn manifest_collection_barrier_failure_keeps_quota() -> Result<()> { + for mode in [faults::FAIL_REMOVE, faults::FAIL_DIRECTORY_SYNC] { + let temp = tempfile::tempdir()?; + let path = temp.path().join("pages"); + let trace = faults::Faults::new(path.clone(), None); + let fs = Arc::new(faults::FaultFileSystem { + faults: trace.clone(), + }); + let generations = GenerationStore::open( + temp.path().join("images"), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open(&path, fs.clone(), 64, 1024 * 1024, 8)?; + // No engine root was published: every page in this fixture is an orphan. + tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + pages.reconcile_retained(&[])?; + let before = pages.usage(); + trace.mode.store(mode, std::sync::atomic::Ordering::Release); + assert!(pages.reclaim(16).is_err()); + assert_eq!( + pages.usage(), + before, + "uncertain deletion must not refund quota" + ); + assert!(matches!(pages.reclaim(16), Err(ManifestError::Quarantined))); + drop(pages); + let mut pages = ManifestPageStore::open(&path, fs, 64, 1024 * 1024, 8)?; + assert!(pages.reclaim(16).is_err()); + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + assert_eq!(pages.usage().pages, 0); + assert_eq!(pages.usage().bytes, 0); + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_batch_retention_refusal_cancels_unwritten_reservations() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar::sdk::ExecutionDomain; + use cellar_logic::consensus::OperationId; + use raft::Storage as _; + let temp = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let first_group = log.create_group(engine_bootstrap()).await?; + let mut second_birth = engine_bootstrap(); + second_birth.scope.group = GroupId::new([33; 16])?; + let second_group = log.create_group(second_birth).await?; + let mut second_route = route(); + second_route.history.group = second_group.identity().scope.group; + let first_tree = ManifestTree::new(route(), ManifestLimits::default())?; + let second_tree = ManifestTree::new(second_route, ManifestLimits::default())?; + let generations = GenerationStore::open( + temp.path().join("images"), + fs.clone(), + GenerationLimits::default(), + )?; + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + // Capacity admits one group's current/pending pair, never two groups. + let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 64, 1024 * 1024, 2)?; + let first_root = first_tree.update( + &mut pages, + ManifestRoot::default(), + leaf(1, &pin, 1, 2), + None, + )?; + let second_root = second_tree.update( + &mut pages, + ManifestRoot::default(), + leaf(2, &pin, 1, 2), + None, + )?; + let first = first_tree.prepare_complete( + &first_group, + &mut pages, + first_root, + position(2), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + let second = second_tree.prepare_complete( + &second_group, + &mut pages, + second_root, + position(2), + 1, + |_| Ok(CellImagePin::Materialized(pin.clone())), + )?; + assert!(log + .persist( + OperationId::new(1), + vec![ + root_write(&first_group, first.clone())?, + root_write(&second_group, second.clone())? + ] + ) + .await + .is_err()); + for group in [&first_group, &second_group] { + assert!(group.group_manifest()?.is_none()); + assert_eq!( + group.last_index()?, + 1, + "no part of the rejected batch reached storage" + ); + } + pages.reconcile_retained(&[])?; + assert_eq!( + collect_pages(&mut pages)?, + 0, + "prepared apply pins protect both staged roots" + ); + log.persist(OperationId::new(2), vec![root_write(&first_group, first)?]) + .await?; + drop(second); + assert_eq!(collect_pages(&mut pages)?, 1); + assert_eq!(pages.usage().pages, 1); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index cf192e8fb..33059eefd 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -27,6 +27,7 @@ pub const HOLD_LOCK_DROP: u8 = 10; pub const HOLD_IMAGE_READ: u8 = 11; pub const HOLD_SNAPSHOT_TEMP: u8 = 12; pub const HOLD_EXTENT_READ: u8 = 13; +pub const FAIL_AFTER_SYNC: u8 = 14; pub struct Gate { pub mode: AtomicU8, @@ -187,6 +188,7 @@ impl HostFileIo for FaultFile { | FAIL_SYNC | CRASH_BEFORE_SYNC | CRASH_AFTER_SYNC + | FAIL_AFTER_SYNC | FAIL_SYNC_HOLD_LOCK_DROP ) && self .gate @@ -201,6 +203,10 @@ impl HostFileIo for FaultFile { match mode { HOLD_SYNC | CRASH_BEFORE_SYNC => self.gate.pause()?, FAIL_SYNC => return Err(io::Error::other("injected Raft sync failure")), + FAIL_AFTER_SYNC => { + self.inner.sync_all()?; + return Err(io::Error::other("injected failure after durable Raft sync")); + } FAIL_SYNC_HOLD_LOCK_DROP => { self.gate.mode.store(HOLD_LOCK_DROP, Ordering::Release); return Err(io::Error::other( From 65ab08947a006207c013284a60d7b0eda6b57823 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 11:47:33 -0700 Subject: [PATCH 013/568] docs: record cell-group reclamation checkpoint --- plans/multi-raft/cell-groups/baseline.json | 4 +-- plans/multi-raft/cell-groups/progress.md | 38 ++++++++++++++++++++++ plans/multi-raft/cell-groups/tasks.json | 4 ++- 3 files changed, 43 insertions(+), 3 deletions(-) diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index f9eb54721..172dc1ae8 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "b71f62b5841bab473481661bb5d7d6be9f366e29", + "source_revision": "f41f470f40f510746ba5d8f3178b35624f3cecb2", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed incremental preparation at b71f62b5841bab473481661bb5d7d6be9f366e29. The cache now owns persistent manifest pin trees, validates their exact root reference, charges node/ID allocations, and checks the prior root for staged mutations while allowing identical selected retries. Page-store instance binding prevents reuse after reopen. Page reclamation, uncertain-publication retention, full extent recovery and production host cutover remain pending; CG-02 is not complete.", + "review_note": "2026-10-29: reviewed reclamation and uncertain-publication checkpoint at f41f470f40f510746ba5d8f3178b35624f3cecb2. Page leases retain the directory lock. Storage reserves old/candidate closures before engine I/O, commits retention after cache publication and preserves both closures on an armed failure. Recovery binds retention to the selected engine marker; bounded collection requires complete owner-supplied root reconciliation and a successful deletion barrier. Existing source excerpts remain unchanged. Complete shared-host inventory integration, root process-kill cuts and full extent-mode qualification remain pending; CG-02 is not complete.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index eb935926a..bf607446a 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -167,3 +167,41 @@ reviewed source preflight pass. The initial source-drift refusal correctly requi review of the changed cache anchor before advancing the baseline. Source and executable hashes, failed/corrected test output and final check logs are retained under `.cellar/cell-groups-evidence/CG-02-incremental-pins/`. + +### CG-02 reclamation and uncertain publication checkpoint + +Prepared pin-tree nodes now lease their physical manifest pages and keep the +directory lock alive. Bounded collection scans an explicit inventory window, +removes only unleased pages, and refunds page/byte quotas only after the directory +sync. Remove or sync failure quarantines the owner without speculative refunds. +Every reopened store requires root inventory reconciliation before collection. + +The engine now reserves old/candidate closures before writing and selects the new +retention reference only after sync and cache publication. Dropping an armed +reservation on failure keeps both page/image closures and blocks collection until +recovery; pre-write refusals cancel unarmed reservations. The page owner retains +uncertain images even after request and engine handles disappear. Complete +recovery automatically retains a root only when it matches the selected engine +marker. Additional snapshot/archive/transition roots have an explicit bounded +retained set, independent of disposable caller handles. + +The owning shared host must still supply the complete catalog/checkpoint/transition +inventory to `reconcile_retained`. This is a trusted owner contract; page filenames +cannot discover authority, and these storage fixtures do not qualify that future +host integration. CG-02 also still requires the root-specific process-kill matrix +and full extent-mode recovery qualification. It remains IN_PROGRESS. + +Runtime checkpoint `f41f470f` passes 11 native manifest cases and 19 existing Raft +storage cases. New checks cover independent snapshot/transition retention, +collection during stalled apply, quotas after removal/barrier failures, and +two-group batch refusal before any write. A real failure injected after fsync +retains both images after request/engine handles drop; restart selects the new +durable root before reclaiming the superseded image/page. + +All-target Clippy with warnings denied, formatting, layout, native dependency +isolation, docs, source preflight and all 37 plan-validator tests pass. The first +test build required owned fixture paths; the first lint run required replacing +an unnecessary clone used to construct a one-element slice. Neither fix weakened +an assertion. Source hashes, copied test executables and failed/corrected logs +are retained under `.cellar/cell-groups-evidence/CG-02-reclamation/`. This is a +storage checkpoint, not the complete CG-02 evidence gate or a RustFS qualification. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 98db0de14..c4430c620 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -152,7 +152,8 @@ "crates/cellar-ltx/src/extent/", "crates/cellar-ltx/src/generation.rs", "crates/cellar/application/consensus/mod.rs", - "crates/cellar/tests/support/raft_ltx_fault.rs" + "crates/cellar/tests/support/raft_ltx_fault.rs", + "crates/cellar/tests/support/raft_storage_fault.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/mod.rs", @@ -164,6 +165,7 @@ "crates/cellar/durability/raft/cell_groups/README.md", "crates/cellar/durability/raft/cell_groups/root.rs", "crates/cellar/durability/raft/cell_groups/pins.rs", + "crates/cellar/durability/raft/cell_groups/retention.rs", "crates/cellar/durability/raft/cell_groups/image_pin.rs" ], "test_target": "native_cell_groups_storage", From ec9d277e50416f786fe3c1cd851b4147f9cd4b43 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 12:05:25 -0700 Subject: [PATCH 014/568] test: qualify cell-group recovery across image representations --- .../durability/raft/cell_groups/README.md | 10 +- .../tests/native_cell_groups_storage.rs | 24 + .../tests/support/cell_group_recovery.rs | 782 ++++++++++++++++++ crates/cellar/tests/support/raft_ltx_fault.rs | 8 + plans/multi-raft/cell-groups/README.md | 23 +- plans/multi-raft/cell-groups/tasks.json | 2 +- 6 files changed, 844 insertions(+), 5 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_recovery.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 7082f2581..60ff1157d 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -104,5 +104,11 @@ remaining production consumers are converted, and is removed at hard cutover. Existing disks and archives must be retained, not reset by this component. Further integration must wire complete catalog/checkpoint/transition inventory -handoff into the shared host. Native engine process-kill cuts and full qualification -of both physical image modes remain required before CG-02 can be marked DONE. +handoff into the shared host. Native recovery fixtures exercise both physical +image modes with independent SQLite value and receipt rows. Process-kill cuts +cover image/page writes and sync, engine sync, publication and reclamation; +restart verifies every selected cell before collecting orphaned objects. Missing +or corrupt pages, images, extent roots and non-header extents refuse complete-root +preparation. These are storage-boundary checks, not quorum, SDK deduplication or +RustFS qualification. CG-02 completion requires the full source-bound evidence +gate recorded in the delivery plan. diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index a7e0de1bd..173b0bacf 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -16,6 +16,9 @@ mod faults; #[allow(dead_code)] mod engine_fault; +#[path = "support/cell_group_recovery.rs"] +mod recovery; + use anyhow::Result; use cellar::application::consensus::{ verification::{cell_groups::*, CommittedImage}, @@ -27,6 +30,27 @@ use cellar_ltx::{ }; use std::{collections::BTreeMap, sync::Arc}; +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn both_image_representations_share_contract() -> Result<()> { + recovery::both_images().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn missing_cell_image_refuses_group() -> Result<()> { + recovery::missing_images().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn manifest_marker_crash_cuts() -> Result<()> { + recovery::crash_matrix().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[ignore = "process-kill fixture, launched only by manifest_marker_crash_cuts"] +async fn manifest_process_fixture() -> Result<()> { + recovery::process_fixture().await +} + fn route() -> GroupRoute { GroupRoute { history: GroupHistory { diff --git a/crates/cellar/tests/support/cell_group_recovery.rs b/crates/cellar/tests/support/cell_group_recovery.rs new file mode 100644 index 000000000..5dd50546f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_recovery.rs @@ -0,0 +1,782 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recovery qualification on actual SQLite/LTX, Raft Engine and both physical +//! image stores. These fixtures provide storage cuts, not a running Raft quorum. +#![allow(clippy::disallowed_methods)] + +use super::*; +use anyhow::{bail, ensure, Context}; +use cellar::application::consensus::verification::{GroupLog, LogOpenMode, ReplicaLog}; +use cellar::sdk::ExecutionDomain; +use cellar_logic::consensus::OperationId; +use cellar_ltx::{ + apply::{ApplyLimits, ImageState}, + generation::ImageId, + internal::{ + ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRoot, + ImageRootUpdate, NodeExtentStore, EXTENT_BYTES, + }, + FileSystem, TXID, +}; +use std::{ + path::{Path, PathBuf}, + sync::atomic::Ordering, + time::Duration, +}; + +#[derive(Clone, Copy, Debug)] +enum Mode { + Materialized, + Extents, +} + +impl Mode { + fn name(self) -> &'static str { + match self { + Self::Materialized => "materialized", + Self::Extents => "extents", + } + } + + fn parse(value: &str) -> Result { + match value { + "materialized" => Ok(Self::Materialized), + "extents" => Ok(Self::Extents), + _ => bail!("unknown image mode"), + } + } +} + +enum Images { + Materialized(GenerationStore), + Extents(NodeExtentStore), +} + +fn history(cell: u8) -> CellHistory { + CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new(format!("cell-{cell}")).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + } +} + +fn image_id(leaf: &CellManifestLeaf) -> Result { + let cell: u8 = leaf + .version + .history + .cell + .as_str() + .strip_prefix("cell-") + .context("fixture cell")? + .parse()?; + let sequence = u8::try_from(leaf.version.sequence.get())?; + let mut bytes = [0; 16]; + bytes[0] = cell; + bytes[1] = sequence; + bytes[2] = 123; + Ok(ImageId::new(bytes)?) +} + +fn extent_reference(manifest: &ExtentManifest) -> Result { + Ok(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len() as u32, + )?) +} + +fn leaf_reference(leaf: &CellManifestLeaf) -> Result { + let CellImageCommitment::Extents { + root, + manifest_bytes, + } = leaf.version.image.commitment + else { + bail!("extent leaf required"); + }; + Ok(ExtentRootReference::new( + ImageRoot::from_bytes(root), + manifest_bytes, + )?) +} + +impl Images { + fn open(path: &Path, mode: Mode, fs: Arc) -> Result { + Ok(match mode { + Mode::Materialized => Self::Materialized(GenerationStore::open( + path.to_owned(), + fs, + GenerationLimits::default(), + )?), + Mode::Extents => Self::Extents(NodeExtentStore::open( + path.to_owned(), + fs, + Default::default(), + )?), + }) + } + + fn prepare( + &self, + cell: u8, + base: Option<&CellImagePin>, + cut: &images::Cut, + after: &[u8], + index: u64, + ) -> Result<(CellManifestLeaf, CellImagePin)> { + let state = cut.result; + let mut leaf = CellManifestLeaf { + version: CellVersion { + history: history(cell), + sequence: MutationSequence::new(state.txid.0)?, + image: CellImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: CellImageCommitment::Materialized(state.digest), + }, + }, + geometry: CellImageGeometry { + length: state.length, + page_size: state.page_size, + }, + last_mutation: position(index), + effects: EffectSummary::default(), + }; + let pin = match self { + Self::Materialized(store) => { + let base = match base { + None => None, + Some(CellImagePin::Materialized(pin)) => Some(pin), + _ => bail!("wrong physical base"), + }; + CellImagePin::Materialized(store.prepare(image_id(&leaf)?, base, &cut.update())?) + } + Self::Extents(store) => { + let binding = ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + schema: route().schema_digest, + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }; + let geometry = Geometry::new(state.page_size, state.length)?; + let ids = after + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?; + let manifest = ExtentManifest::new(binding, geometry, &ids)?; + let reference = extent_reference(&manifest)?; + let base = match base { + None => None, + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + _ => bail!("wrong physical base"), + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0)?, + &[], + )?; + let update = ImageRootUpdate { + base: base + .map(|p| p.reference()) + .unwrap_or(extent_reference(&empty)?), + result: reference, + binding, + geometry, + segments: cut.update().segments, + }; + let prepared = store.prepare_image_root( + base, + &update, + &ApplyLimits::default(), + &mut vec![0; EXTENT_BYTES as usize], + )?; + leaf.version.image.commitment = CellImageCommitment::Extents { + root: reference.root().bytes(), + manifest_bytes: reference.encoded_length(), + }; + CellImagePin::Extents(Arc::new(prepared.image)) + } + }; + Ok((leaf, pin)) + } + + fn recover(&self, leaf: &CellManifestLeaf) -> Result { + Ok(match self { + Self::Materialized(store) => { + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + bail!("materialized leaf required"); + }; + CellImagePin::Materialized(store.adopt( + image_id(leaf)?, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + digest, + }, + )?) + } + Self::Extents(store) => { + let view = store.open_root(leaf_reference(leaf)?)?; + let manifest = view.manifest().clone(); + // Inventory/root metadata is never a seal after restart. Verify + // every dependency, including extents after the SQLite header. + for i in 0..manifest.geometry().extent_count() { + store.verify_extent(manifest.extent(i).context("extent closure")?)?; + } + CellImagePin::Extents(Arc::new( + store.seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize])?, + )) + } + }) + } + + fn collect(&self, leaves: &[CellManifestLeaf]) -> Result<()> { + match self { + Self::Materialized(store) => { + store.retain_only(&leaves.iter().map(image_id).collect::>>()?)?; + store.reclaim(4096)?; + } + Self::Extents(store) => { + store.retain_only( + &leaves + .iter() + .map(leaf_reference) + .collect::>>()?, + )?; + store.reclaim(4096)?; + } + } + Ok(()) + } +} + +fn read_image(pin: &CellImagePin) -> Result> { + match pin { + CellImagePin::Materialized(pin) => images::read(pin), + CellImagePin::Extents(pin) => { + let manifest = pin.manifest(); + let mut bytes = Vec::new(); + for i in 0..manifest.geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(manifest.extent(i).context("extent")?)?, + ); + } + ensure!(bytes.len() as u64 == manifest.geometry().length()); + Ok(bytes) + } + CellImagePin::Empty => Ok(Vec::new()), + } +} + +fn initial(cell: u8, large: bool) -> Result> { + let padding = if large { + "CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob(4202496));" + } else { + "" + }; + images::image( + None, + &format!( + "CREATE TABLE value(n INTEGER); INSERT INTO value VALUES ({}); + CREATE TABLE receipts(client TEXT PRIMARY KEY, floor INTEGER, result BLOB); + INSERT INTO receipts VALUES ('client-{cell}',1,X'{}'); {padding}", + cell * 10 + 1, + hex::encode(format!("r{cell}-1")) + ), + ) +} + +fn mutate(before: &[u8]) -> Result> { + images::image( + Some(before), + "UPDATE value SET n = n + 1; UPDATE receipts SET floor = 2, result = X'72312d32';", + ) +} + +fn assert_receipt(pin: &CellImagePin, cell: u8, sequence: u64) -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join("oracle.sqlite"); + std::fs::write(&path, read_image(pin)?)?; + let db = + rusqlite::Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + let value: u64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let row: (String, u64, Vec) = + db.query_row("SELECT client, floor, result FROM receipts", [], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?)) + })?; + assert_eq!(value, u64::from(cell) * 10 + sequence); + assert_eq!( + row, + ( + format!("client-{cell}"), + sequence, + format!("r{cell}-{sequence}").into_bytes() + ) + ); + assert_eq!( + db.query_row("SELECT COUNT(*) FROM receipts", [], |r| r.get::<_, u64>(0))?, + 1 + ); + Ok(()) +} + +fn prepare_root( + tree: &ManifestTree, + group: &GroupLog, + pages: &mut ManifestPageStore, + marker: &GroupManifest, + images: &Images, +) -> Result { + Ok(tree.prepare_complete( + group, + pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + images + .recover(leaf) + .map_err(|_| ManifestError::Invalid("incomplete physical image")) + }, + )?) +} + +async fn seed(path: &Path, mode: Mode, large: bool) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log.create_group(engine_bootstrap()).await?; + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let mut prepared = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| unreachable!(), + )?; + for cell in 1..=2 { + let bytes = initial(cell, large && cell == 1)?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let (leaf, pin) = images.prepare(cell, None, &cut, &bytes, u64::from(cell) + 1)?; + prepared = tree.prepare_update(&mut pages, &prepared, leaf, None, pin)?; + log.persist( + OperationId::new(u64::from(cell)), + vec![root_write(&group, prepared.clone())?], + ) + .await?; + } + let marker = prepared.marker().clone(); + log.close().await?; + domain.shutdown().await; + Ok(marker) +} + +pub async fn both_images() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + let temp = tempfile::tempdir()?; + seed(temp.path(), mode, true).await?; + { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.context("selected root")?, + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + let sibling = current.pin_cell(&history(2).cell).unwrap().0.clone(); + let before = read_image(&old_pin)?; + let after = mutate(&before)?; + let cut = images::Cut::new(&before, &after, 1, true, false); + let (changed, next_pin) = images.prepare(1, Some(&old_pin), &cut, &after, 4)?; + if let (CellImagePin::Extents(old), CellImagePin::Extents(next)) = (&old_pin, &next_pin) + { + assert!(old.manifest().geometry().extent_count() > 1); + assert_eq!( + old.manifest().extent(1), + next.manifest().extent(1), + "unchanged tail extent is shared" + ); + } + let next = + tree.prepare_update(&mut pages, ¤t, changed, Some(&old.version), next_pin)?; + log.persist(OperationId::new(3), vec![root_write(&group, next.clone())?]) + .await?; + assert_eq!(next.pin_cell(&history(2).cell).unwrap().0, &sibling); + assert_receipt(&old_pin, 1, 1)?; + assert_receipt(&next.pin_cell(&history(1).cell).unwrap().1, 1, 2)?; + assert_receipt(&next.pin_cell(&history(2).cell).unwrap().1, 2, 1)?; + log.close().await?; + domain.shutdown().await; + } + recover_and_check(temp.path(), mode, Some(4)).await?; + } + Ok(()) +} + +async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 3), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + let marker = group.group_manifest()?.context("selected root")?; + let index = marker.position().index.get(); + if let Some(expected) = expected { + assert_eq!(index, expected); + } + assert!([3, 4].contains(&index)); + assert_eq!(marker.root().cells(), 2); + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let restored = prepare_root(&tree, &group, &mut pages, &marker, &images)?; + let mut leaves = Vec::new(); + for cell in 1..=2 { + let (leaf, pin) = restored.pin_cell(&history(cell).cell).unwrap(); + let sequence = if cell == 1 && index == 4 { 2 } else { 1 }; + assert_eq!(leaf.version.history, history(cell)); + assert_eq!(leaf.version.sequence.get(), sequence); + assert_receipt(&pin, cell, sequence)?; + leaves.push(leaf.clone()); + } + pages.reconcile_retained(&[])?; + collect_pages(&mut pages)?; + images.collect(&leaves)?; + // Collection must not turn either selected image into an empty/missing cell. + let verified = prepare_root(&tree, &group, &mut pages, &marker, &images)?; + assert_eq!(verified.marker(), &marker); + assert_eq!(pages.usage().pages, 2); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +pub async fn missing_images() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + let kinds: &[&str] = match mode { + Mode::Materialized => &[ + "page-missing", + "page-corrupt", + "image-missing", + "image-corrupt", + ], + Mode::Extents => &[ + "page-missing", + "page-corrupt", + "root-missing", + "root-corrupt", + "tail-missing", + "tail-corrupt", + ], + }; + for kind in kinds { + let temp = tempfile::tempdir()?; + let marker = seed(temp.path(), mode, true).await?; + let victim = if kind.starts_with("page-") { + temp.path().join("pages").join(format!( + "{}.page", + hex::encode(marker.root().digest().unwrap()) + )) + } else { + let fs = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &temp.path().join("pages"), + fs.clone(), + 256, + 4 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let cell = if kind.starts_with("tail-") { 1 } else { 2 }; + let leaf = tree + .get(&pages, marker.root(), &history(cell).cell)? + .unwrap(); + match mode { + Mode::Materialized => temp + .path() + .join("images") + .join(format!("{}.sqlite", hex::encode(image_id(&leaf)?.bytes()))), + Mode::Extents => { + let reference = leaf_reference(&leaf)?; + if kind.starts_with("root-") { + temp.path() + .join("images") + .join(format!("r-{}", hex::encode(reference.root().bytes()))) + } else { + let store = NodeExtentStore::open( + temp.path().join("images"), + fs, + Default::default(), + )?; + let view = store.open_root(reference)?; + assert!(view.manifest().geometry().extent_count() > 1); + let extent = view.manifest().extent(1).unwrap(); + temp.path().join("images").join(format!( + "e-{:08x}-{}", + extent.length(), + hex::encode(extent.bytes()) + )) + } + } + } + }; + if kind.ends_with("missing") { + std::fs::remove_file(&victim)?; + } else { + let mut bytes = std::fs::read(&victim)?; + let last = bytes.len() - 1; + bytes[last] ^= 0x80; + std::fs::write(&victim, bytes)?; + } + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Resume, 2), + fs.clone(), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&marker)); + let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; + let mut pages = + ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + assert!( + prepare_root(&tree, &group, &mut pages, &marker, &images).is_err(), + "{mode:?}/{kind}" + ); + assert!(group.applied().is_err()); + assert!( + pages.reclaim(16).is_err(), + "failed recovery cannot authorize reclamation" + ); + log.close().await?; + domain.shutdown().await; + } + } + Ok(()) +} + +struct Child(std::process::Child); + +impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +pub async fn crash_matrix() -> Result<()> { + for mode in [Mode::Materialized, Mode::Extents] { + for cut in [ + "before-image", + "image-write", + "image-before-sync", + "image-file-sync", + "image-directory-sync", + "after-image", + "manifest-write", + "manifest-before-sync", + "manifest-file-sync", + "manifest-directory-sync", + "after-manifest", + "engine-before-sync", + "engine-after-sync", + "after-publication", + "reclaim-after-remove", + ] { + let temp = tempfile::tempdir()?; + seed(temp.path(), mode, false).await?; + let signal = temp.path().join("signal"); + let output = temp.path().join("child.log"); + let file = std::fs::File::create(&output)?; + let mut child = Child( + std::process::Command::new(std::env::current_exe()?) + .args([ + "--ignored", + "--exact", + "manifest_process_fixture", + "--nocapture", + "--test-threads=1", + ]) + .env("CELLAR_GROUP_TEST_ROOT", temp.path()) + .env("CELLAR_GROUP_TEST_MODE", mode.name()) + .env("CELLAR_GROUP_TEST_CUT", cut) + .env("CELLAR_GROUP_TEST_SIGNAL", &signal) + .stdout(file.try_clone()?) + .stderr(file) + .spawn()?, + ); + tokio::time::timeout(Duration::from_secs(30), async { + while !signal.exists() { + if let Some(status) = child.0.try_wait()? { + bail!( + "{mode:?}/{cut} exited {status}: {}", + std::fs::read_to_string(&output)? + ); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + child.0.kill()?; + let status = child.0.wait()?; + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + assert_eq!(status.signal(), Some(9)); + } + assert!(!status.success()); + let expected = match cut { + "engine-before-sync" => None, + "engine-after-sync" | "after-publication" | "reclaim-after-remove" => Some(4), + _ => Some(3), + }; + recover_and_check(temp.path(), mode, expected) + .await + .with_context(|| format!("{mode:?}/{cut}"))?; + assert_eq!( + std::fs::read_to_string(temp.path().join("executions"))?, + "1", + "recovery must not rerun the callback" + ); + } + } + Ok(()) +} + +pub async fn process_fixture() -> Result<()> { + let path = PathBuf::from(std::env::var("CELLAR_GROUP_TEST_ROOT")?); + let mode = Mode::parse(&std::env::var("CELLAR_GROUP_TEST_MODE")?)?; + let cut_name = std::env::var("CELLAR_GROUP_TEST_CUT")?; + let signal = PathBuf::from(std::env::var("CELLAR_GROUP_TEST_SIGNAL")?); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = engine_fault::Gate::new(Some(signal.clone())); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 2), + Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.group(&engine_bootstrap().scope)?; + let image_fault = faults::Faults::new(path.join("images"), Some(signal.clone())); + let page_fault = faults::Faults::new(path.join("pages"), Some(signal)); + image_fault + .all_directory_files + .store(true, Ordering::Release); + page_fault + .all_directory_files + .store(true, Ordering::Release); + let images = Images::open( + &path.join("images"), + mode, + Arc::new(faults::FaultFileSystem { + faults: image_fault.clone(), + }), + )?; + let mut pages = ManifestPageStore::open( + &path.join("pages"), + Arc::new(faults::FaultFileSystem { + faults: page_fault.clone(), + }), + 256, + 4 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.unwrap(), + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + let before = read_image(&old_pin)?; + let after = mutate(&before)?; + // This independent callback ledger is durable before any crash gate arms. + std::fs::write(path.join("executions"), b"1")?; + std::fs::File::open(path.join("executions"))?.sync_all()?; + std::fs::File::open(&path)?.sync_all()?; + if cut_name == "before-image" { + image_fault.pause()?; + } + let select = |prefix: &str| match cut_name.strip_prefix(prefix) { + Some("write") => faults::AFTER_IMAGE_WRITE, + Some("before-sync") => faults::BEFORE_IMAGE_SYNC, + Some("file-sync") => faults::AFTER_IMAGE_SYNC, + Some("directory-sync") => faults::AFTER_DIRECTORY_SYNC, + _ => 0, + }; + image_fault.mode.store(select("image-"), Ordering::Release); + let cut = images::Cut::new(&before, &after, 1, true, false); + let (changed, pin) = images.prepare(1, Some(&old_pin), &cut, &after, 4)?; + if cut_name == "after-image" { + image_fault.pause()?; + } + page_fault + .mode + .store(select("manifest-"), Ordering::Release); + let next = tree.prepare_update(&mut pages, ¤t, changed, Some(&old.version), pin)?; + if cut_name == "after-manifest" { + page_fault.pause()?; + } + let _release = gate.arm(match cut_name.as_str() { + "engine-before-sync" => engine_fault::CRASH_BEFORE_SYNC, + "engine-after-sync" => engine_fault::CRASH_AFTER_SYNC, + _ => engine_fault::OFF, + }); + log.persist(OperationId::new(3), vec![root_write(&group, next)?]) + .await?; + if cut_name == "after-publication" { + page_fault.pause()?; + } + drop(current); + drop(old_pin); + pages.reconcile_retained(&[])?; + if cut_name == "reclaim-after-remove" { + page_fault + .mode + .store(faults::AFTER_REMOVE, Ordering::Release); + collect_pages(&mut pages)?; + } + bail!("requested cut was never reached: {cut_name}") +} diff --git a/crates/cellar/tests/support/raft_ltx_fault.rs b/crates/cellar/tests/support/raft_ltx_fault.rs index b04b80dde..c81aea2c4 100644 --- a/crates/cellar/tests/support/raft_ltx_fault.rs +++ b/crates/cellar/tests/support/raft_ltx_fault.rs @@ -27,6 +27,7 @@ pub struct Faults { pub directory: PathBuf, pub mode: AtomicU8, pub fail_reads: AtomicBool, + pub all_directory_files: AtomicBool, pub collision: AtomicBool, pub reads: AtomicUsize, pub image_creates: AtomicUsize, @@ -50,6 +51,7 @@ impl Faults { directory, mode: AtomicU8::new(0), fail_reads: AtomicBool::new(false), + all_directory_files: AtomicBool::new(false), collision: AtomicBool::new(false), reads: AtomicUsize::new(0), image_creates: AtomicUsize::new(0), @@ -136,6 +138,12 @@ impl Drop for File { impl FaultFileSystem { fn wrap(&self, path: &Path, inner: HostFile, image_output: bool) -> HostFile { + let image_output = image_output + || (self.faults.all_directory_files.load(Ordering::Acquire) + && path.parent() == Some(self.faults.directory.as_path()) + && path + .file_name() + .is_some_and(|name| name != "LOCK" && name != "images.lock")); HostFile::from_io(File { inner, path: path.into(), diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 960b7529e..50b6609f1 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -8,6 +8,10 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. +The design package is delivered; implementation now proceeds toward the complete +shared-group runtime and hard cutover. Implementation checkpoints are recorded +separately; delivering this package does not mark the remaining tasks complete. + This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned multi-Raft: many cells in a group, many groups on a node, group replication across @@ -24,7 +28,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-01` emits a complete executor brief, source excerpts, + eligibility; `show CG-02` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -43,7 +47,7 @@ The design, task briefs and validator unit tests are committed in this package. python3 plans/multi-raft/cell-groups/plan.py validate python3 plans/multi-raft/cell-groups/plan.py preflight python3 plans/multi-raft/cell-groups/plan.py ready -python3 plans/multi-raft/cell-groups/plan.py show CG-01 +python3 plans/multi-raft/cell-groups/plan.py show CG-02 python3 -m unittest discover -s plans/multi-raft/cell-groups -p 'test_*.py' ``` @@ -60,6 +64,21 @@ containers apart from read-only Git source inspection. Commands displayed in executor briefs are explicitly run by the implementation agent. Destructive qualification belongs only to the harness's recorded, owned disposable resources. +### Current executor handoff + +Resume CG-02, which is already IN_PROGRESS, before starting CG-03. At this +checkpoint `ready` lists no new TODO task because CG-03 depends on CG-02's +completion; an empty list does not mean the project is finished. Read +[progress](progress.md), inspect the working diff and retain existing work before +executing the CG-02 brief. Its complete source-bound evidence gate must pass before +changing its status to DONE. Continue in the generated dependency order afterward. + +The implementation executor should deliver code, updated component contracts and +the required retained evidence for each task on the active PR. The final handoff +requires CG-16 and the release gate, including the real RustFS and independent +failure-domain profiles. Successful plan validation alone satisfies this design +deliverable, not those implementation or deployment gates. + ## Deliverables and source of truth | Artifact | Responsibility | diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index c4430c620..f2680c4ef 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -158,7 +158,7 @@ "creates": [ "crates/cellar/durability/raft/cell_groups/mod.rs", "crates/cellar/durability/raft/cell_groups/manifest.rs", - "crates/cellar/durability/raft/cell_groups/manifest_tests.rs", + "crates/cellar/tests/support/cell_group_recovery.rs", "crates/cellar/tests/native_cell_groups_storage.rs", "crates/cellar/durability/raft/cell_groups/page_store.rs", "crates/cellar/durability/raft/cell_groups/page_wire.rs", From 8dacfaded11204a46a3223f00d47d6faf4b1a6a0 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 12:15:13 -0700 Subject: [PATCH 015/568] docs: record completed cell-group storage qualification --- plans/multi-raft/cell-groups/README.md | 28 ++++++------ plans/multi-raft/cell-groups/baseline.json | 4 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/progress.md | 43 +++++++++++++++---- plans/multi-raft/cell-groups/tasks.json | 6 ++- 5 files changed, 57 insertions(+), 26 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 50b6609f1..f980d4c44 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -3,7 +3,8 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 contracts have passed their local evidence gate; CG-02 is in progress. +CG-01 contracts and CG-02 storage have passed their local evidence gates. +CG-03 routing and enrollment is next. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -28,7 +29,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-02` emits a complete executor brief, source excerpts, + eligibility; `show CG-03` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -37,17 +38,18 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package only with verified evidence and completed dependencies. Final qualification repeats the full capability matrix at one release revision. -This checkout retains CG-01's evidence under the ignored -`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/` directory. A fresh checkout must -receive that retained directory at the same relative path before validating the -DONE checkpoint. Missing evidence fails validation; it is never treated as a pass. +This checkout retains evidence under the ignored +`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/` and +`.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` directories. A fresh checkout must +receive both retained directories at the same relative paths before validating +the DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. The design, task briefs and validator unit tests are committed in this package. ```sh python3 plans/multi-raft/cell-groups/plan.py validate python3 plans/multi-raft/cell-groups/plan.py preflight python3 plans/multi-raft/cell-groups/plan.py ready -python3 plans/multi-raft/cell-groups/plan.py show CG-02 +python3 plans/multi-raft/cell-groups/plan.py show CG-03 python3 -m unittest discover -s plans/multi-raft/cell-groups -p 'test_*.py' ``` @@ -66,12 +68,12 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -Resume CG-02, which is already IN_PROGRESS, before starting CG-03. At this -checkpoint `ready` lists no new TODO task because CG-03 depends on CG-02's -completion; an empty list does not mean the project is finished. Read -[progress](progress.md), inspect the working diff and retain existing work before -executing the CG-02 brief. Its complete source-bound evidence gate must pass before -changing its status to DONE. Continue in the generated dependency order afterward. +Start CG-03 after reading [progress](progress.md) and inspecting the working diff. +CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the +remaining tasks still require implementation. `ready` lists eligible TODO tasks; +an IN_PROGRESS task is resumed from its existing brief and working state. Each +task's complete source-bound evidence gate must pass before changing its status +to DONE. Continue in the generated dependency order afterward. The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 172dc1ae8..7247d1a67 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "f41f470f40f510746ba5d8f3178b35624f3cecb2", + "source_revision": "ec9d277e50416f786fe3c1cd851b4147f9cd4b43", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed reclamation and uncertain-publication checkpoint at f41f470f40f510746ba5d8f3178b35624f3cecb2. Page leases retain the directory lock. Storage reserves old/candidate closures before engine I/O, commits retention after cache publication and preserves both closures on an armed failure. Recovery binds retention to the selected engine marker; bounded collection requires complete owner-supplied root reconciliation and a successful deletion barrier. Existing source excerpts remain unchanged. Complete shared-host inventory integration, root process-kill cuts and full extent-mode qualification remain pending; CG-02 is not complete.", + "review_note": "2026-10-29: reviewed and qualified CG-02 at ec9d277e50416f786fe3c1cd851b4147f9cd4b43. All five required cases and complete command gate pass. Both physical image modes recover complete selected roots through 30 process-kill cuts; missing/corrupt image, extent and manifest dependencies refuse. Reclamation retains selected and uncertain closures. Existing anchored source hashes/excerpts are unchanged. Production shared-host authoritative inventory, quorum, snapshots/archive, topology, hibernation and real RustFS recovery remain CG-03 through CG-16 work.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index 2f0f0c56e..984d32ce4 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -13,7 +13,7 @@ never executed by the plan runner. Read the [design](design.md), | Task | Deliverable | Dependencies | Status | Required cases | | --- | --- | --- | --- | ---: | | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | -| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | IN_PROGRESS | 5 | +| CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index bf607446a..a251d8e0c 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -49,15 +49,13 @@ must include its initial replayable LTX chain. ## Remaining implementation -CG-02 is IN_PROGRESS; CG-03 through CG-16 remain TODO. CG-02 must complete -complete paged group manifests, real engine publication, recovery refusal for -missing dependencies and image/pin reclamation tests in both image modes. +CG-01 and CG-02 have passed their complete task gates. CG-03 through CG-16 remain +TODO. The latest storage qualification is recorded below; the earlier sections +retain the intermediate checkpoints and their limitations. -The reviewed source baseline advances to the qualified CG-01 revision. Original -host/storage source anchors remain unchanged; newly produced contract files are -bound by the CG-01 evidence record. Review changed inputs before refreshing the -baseline for later tasks. A new executor must receive the ignored evidence -folder before validating this DONE checkpoint. +The reviewed source baseline is `ec9d277e50416f786fe3c1cd851b4147f9cd4b43`. +The original anchored source hashes and excerpts remain unchanged. A new executor +must receive both ignored DONE-task evidence folders before validation. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until @@ -205,3 +203,32 @@ an unnecessary clone used to construct a one-element slice. Neither fix weakened an assertion. Source hashes, copied test executables and failed/corrected logs are retained under `.cellar/cell-groups-evidence/CG-02-reclamation/`. This is a storage checkpoint, not the complete CG-02 evidence gate or a RustFS qualification. + + +### CG-02 — complete storage qualification + +Qualified source: `ec9d277e50416f786fe3c1cd851b4147f9cd4b43`. +The retained record is `.cellar/cell-groups-evidence/CG-02-ec9d277e5041/record.json`. +It binds 45 source inputs, 16 successful command logs and three copied native +executables. All five required cases were present in the inventory and each +passed when selected individually. The complete gate includes all-target checks +and Clippy, formatting/layout, dependency isolation, docs and 37 validator tests. + +The 14-case manifest suite covers both materialized and extent-backed SQLite +images. Thirty actual child-process kills span image and manifest writes/sync, +engine sync, publication and reclamation. Recovery selects a complete old or new +root, checks both cells' independent values and receipt rows, and verifies that +selected data remains readable after orphan reclamation. Missing or corrupt +manifest pages, sibling images, extent roots and non-header extents refuse +complete recovery. Extent updates also preserve an unchanged shared tail extent. + +All 19 existing Raft storage cases and the generation-store image sync regression +pass. The latter exercises five additional child-process cuts with the shared +fault adapter's original behavior. Standalone child fixtures remain intentionally +ignored by direct suite execution and are launched by their parent tests. + +This completes CG-02 at its storage boundary. Process kills do not simulate power +loss. Receipt rows are physical-state checks, not SDK deduplication qualification. +The shared host must still integrate authoritative root inventories, admission, +quorum, snapshots and archives. No RustFS drill, split/merge, hibernation latency, +density claim or final hard cutover is qualified by these tests. CG-03 is next. diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index f2680c4ef..f90b531f5 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -134,7 +134,7 @@ { "id": "CG-02", "title": "Persist complete group manifests with crash-safe cell image ownership", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -246,7 +246,9 @@ "I07", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-02-ec9d277e5041/record.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From b1d27582fa83c8316d1b902774fbbf720d0b7509 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 12:29:29 -0700 Subject: [PATCH 016/568] feat: validate cell-group enrollment identity and lifecycle --- .../durability/raft/cell_groups/README.md | 28 +- .../durability/raft/cell_groups/enrollment.rs | 202 +++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 4 + .../tests/native_cell_groups_catalog.rs | 274 ++++++++++++++++++ plans/multi-raft/cell-groups/README.md | 4 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/progress.md | 34 ++- plans/multi-raft/cell-groups/protocols.md | 10 + plans/multi-raft/cell-groups/tasks.json | 4 +- 9 files changed, 552 insertions(+), 10 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/enrollment.rs create mode 100644 crates/cellar/tests/native_cell_groups_catalog.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 60ff1157d..2ff58ccf0 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -2,7 +2,7 @@ # Cell-group durable state -This component is under CG-02 implementation. It supplies immutable manifest pages, their filesystem owner, complete-root +CG-02's storage boundary is qualified. The component supplies immutable manifest pages, their filesystem owner, complete-root preparation and atomic Raft Engine root publication. Production host admission is not yet wired to these contracts; a stored root is not quorum authority. @@ -110,5 +110,27 @@ cover image/page writes and sync, engine sync, publication and reclamation; restart verifies every selected cell before collecting orphaned objects. Missing or corrupt pages, images, extent roots and non-header extents refuse complete-root preparation. These are storage-boundary checks, not quorum, SDK deduplication or -RustFS qualification. CG-02 completion requires the full source-bound evidence -gate recorded in the delivery plan. +RustFS qualification. The full CG-02 source-bound evidence gate is recorded in the +delivery plan. + +## Enrollment admission (CG-03 in progress) + +`decide_enrollment` checks a bounded cell-addressed enrollment entry against the +current group route and committed lifecycle state. Sealed and retired ranges +refuse, including delayed entries prepared before the seal. Active admission +returns either a prospective creation record or the identical committed record. +A competing operation, cell incarnation, schema or initial image cannot replace +the winner. The same decision must run again at ordered log application; a +preparation-time check is not permission to bypass a later seal. + +`EnrollmentRecord` binds logical cell history, original creation operation, +schema and initial image. It deliberately excludes physical group placement, so +normal transfer preserves the record while the request adopts the current route. +Its bounded canonical marker-1 codec verifies data shape, not authority. CG-04 +must publish this record atomically with enrollment, retain it through mutations, +and include it in complete snapshots/archives. An exact retry must not initialize +again or roll the current image back to the initial one. Current-term quorum and +read authority remain required from the host even for a recognized retry. + +Paged range-directory publication, birth reconciliation and host integration are +still pending. Pure admission tests do not qualify those boundaries. diff --git a/crates/cellar/durability/raft/cell_groups/enrollment.rs b/crates/cellar/durability/raft/cell_groups/enrollment.rs new file mode 100644 index 000000000..49316e44c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment.rs @@ -0,0 +1,202 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Deterministic enrollment admission. Decisions and decoded records are data, +//! never proof of leadership, a committed enrollment or a published image. + +use crate::application::consensus::{ + group_protocol::{CellGroupEntry, GroupCommand, GroupEntryLimits}, + ImageCommitment, LtxImage, ProtocolError, +}; +use cellar_logic::{cell_group::*, consensus::*}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +const RECORD_DOMAIN: &str = "cellar-cell-enrollment-1"; +const MAX_RECORD_BYTES: usize = 4096; + +/// The group owner's committed lifecycle state. Recheck it at log application; +/// an earlier decision made while active cannot authorize an entry after a seal. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EnrollmentPhase { + Active, + Sealed, + Retired, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum EnrollmentError { + Protocol(ProtocolError), + NotEnrollment, + Sealed, + Retired, + Conflict, + InvalidRecord, + Limit, +} + +impl std::fmt::Display for EnrollmentError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell enrollment: {self:?}") + } +} + +impl std::error::Error for EnrollmentError {} + +/// Immutable creation identity retained alongside the cell's committed state. +/// It survives subsequent cell mutations and normal group transfers. The host +/// must publish it atomically with the initial cell image and carry it through +/// snapshots/archives; the latest image cannot stand in for this original image. +/// Decoding only checks shape and canonical encoding, not commitment authority. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EnrollmentRecord { + cell: CellHistory, + creation: ProposalId, + schema: [u8; 32], + initial: CellImage, +} + +impl EnrollmentRecord { + pub fn cell(&self) -> &CellHistory { + &self.cell + } + + pub fn creation(&self) -> ProposalId { + self.creation + } + + pub fn schema(&self) -> [u8; 32] { + self.schema + } + + pub fn initial(&self) -> CellImage { + self.initial + } + + pub fn encode(&self) -> Result, EnrollmentError> { + let image = LtxImage::try_from(self.initial).map_err(EnrollmentError::Protocol)?; + if image.txid == 0 + && (image.checksum != 0 + || image.commitment != ImageCommitment::Sha256(Sha256::digest([]).into())) + { + return Err(EnrollmentError::InvalidRecord); + } + let wire = WireRecord { + domain: RECORD_DOMAIN.into(), + cluster: self.cell.cluster.bytes(), + namespace: self.cell.namespace.bytes(), + cell: self.cell.cell.as_str().into(), + incarnation: self.cell.incarnation.bytes(), + creation: self.creation.bytes(), + schema: self.schema, + initial_epoch: image.epoch, + initial_txid: image.txid, + initial_checksum: image.checksum, + initial_commitment: image.commitment, + }; + let bytes = serde_json::to_vec(&wire).map_err(|_| EnrollmentError::InvalidRecord)?; + if bytes.len() > MAX_RECORD_BYTES { + return Err(EnrollmentError::Limit); + } + Ok(bytes) + } + + pub fn decode(bytes: &[u8]) -> Result { + if bytes.len() > MAX_RECORD_BYTES { + return Err(EnrollmentError::Limit); + } + let wire: WireRecord = + serde_json::from_slice(bytes).map_err(|_| EnrollmentError::InvalidRecord)?; + if wire.domain != RECORD_DOMAIN { + return Err(EnrollmentError::InvalidRecord); + } + let record = Self { + cell: CellHistory { + cluster: ClusterId::new(wire.cluster) + .map_err(|_| EnrollmentError::InvalidRecord)?, + namespace: NamespaceId::new(wire.namespace) + .map_err(|_| EnrollmentError::InvalidRecord)?, + cell: CellId::new(wire.cell).map_err(|_| EnrollmentError::InvalidRecord)?, + incarnation: IncarnationId::new(wire.incarnation) + .map_err(|_| EnrollmentError::InvalidRecord)?, + }, + creation: ProposalId::new(wire.creation).map_err(|_| EnrollmentError::InvalidRecord)?, + schema: wire.schema, + initial: LtxImage { + epoch: wire.initial_epoch, + txid: wire.initial_txid, + checksum: wire.initial_checksum, + commitment: wire.initial_commitment, + } + .into(), + }; + if record.encode()? != bytes { + return Err(EnrollmentError::InvalidRecord); + } + Ok(record) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum EnrollmentDecision { + /// No committed creation exists. The host still needs current-term authority, + /// physical image verification and replicated atomic publication. + Propose(EnrollmentRecord), + /// Exact retry of the supplied committed creation. Do not rerun initialization + /// or replace the cell's current image with its original enrollment image. + Existing(EnrollmentRecord), +} + +/// Validate the bounded cell-addressed wire request against the current route +/// and the group's committed lifecycle and creation record. Payload is borrowed +/// by the codec; admission never allocates another copy of the LTX image bytes. +/// The caller supplies only committed records, not pending proposals. Concurrent +/// proposals are rechecked in log order, so the first publication fixes identity. +pub fn decide_enrollment( + route: &GroupRoute, + phase: EnrollmentPhase, + request: &[u8], + committed: Option<&EnrollmentRecord>, + limits: GroupEntryLimits, +) -> Result { + match phase { + EnrollmentPhase::Sealed => return Err(EnrollmentError::Sealed), + EnrollmentPhase::Retired => return Err(EnrollmentError::Retired), + EnrollmentPhase::Active => {} + } + let entry = + CellGroupEntry::decode(request, route, limits).map_err(EnrollmentError::Protocol)?; + let GroupCommand::Enroll { cell, image, .. } = entry.command else { + return Err(EnrollmentError::NotEnrollment); + }; + let requested = EnrollmentRecord { + cell, + creation: entry.binding.proposal, + schema: entry.binding.schema_digest, + initial: image.into(), + }; + // Bound the durable creation record before an owner reserves image work. + requested.encode()?; + match committed { + Some(existing) if existing == &requested => { + Ok(EnrollmentDecision::Existing(existing.clone())) + } + Some(_) => Err(EnrollmentError::Conflict), + None => Ok(EnrollmentDecision::Propose(requested)), + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireRecord { + domain: String, + cluster: [u8; 16], + namespace: [u8; 16], + cell: String, + incarnation: [u8; 16], + creation: [u8; 16], + schema: [u8; 32], + initial_epoch: u64, + initial_txid: u64, + initial_checksum: u64, + initial_commitment: ImageCommitment, +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 668a966da..49fa5d3e2 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -2,6 +2,7 @@ //! Complete group-state storage. A synced page is data, not Raft authority. +mod enrollment; mod image_pin; mod manifest; mod page_store; @@ -16,6 +17,9 @@ pub use manifest::{ }; pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; +pub use enrollment::{ + decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, +}; pub use image_pin::{cell_image_history_digest, CellImagePin}; pub(crate) use pins::ManifestPins; pub use root::{GroupManifest, PreparedGroupManifest}; diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs new file mode 100644 index 000000000..1b4ad5b9f --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -0,0 +1,274 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog admission contracts. Pure decisions do not qualify a live group or +//! catalog root publication; those have separate native integration gates. + +use cellar::application::consensus::{ + verification::cell_groups::*, ClusterId, GroupId, ImageCommitment, IncarnationId, LtxImage, + NamespaceId, ProposalId, +}; +use sha2::{Digest, Sha256}; + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn entry(route: &GroupRoute) -> CellGroupEntry<'static> { + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([7; 16]).unwrap(), + }, + command: GroupCommand::Enroll { + cell: CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell: CellId::new("counter").unwrap(), + incarnation: IncarnationId::new([8; 16]).unwrap(), + }, + image: LtxImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: ImageCommitment::Sha256(Sha256::digest([]).into()), + }, + files: vec![], + }, + } +} + +fn encoded(entry: &CellGroupEntry<'_>, route: &GroupRoute) -> Vec { + entry.encode(route, GroupEntryLimits::default()).unwrap() +} + +fn admitted(route: &GroupRoute, request: &[u8]) -> EnrollmentRecord { + match decide_enrollment( + route, + EnrollmentPhase::Active, + request, + None, + GroupEntryLimits::default(), + ) + .unwrap() + { + EnrollmentDecision::Propose(record) => record, + EnrollmentDecision::Existing(_) => panic!("new cell cannot be an existing creation"), + } +} + +#[test] +fn enrollment_decision_checks_identity() { + let route = route(); + let original = entry(&route); + let request = encoded(&original, &route); + let selected = admitted(&route, &request); + assert_eq!(selected.creation(), original.binding.proposal); + assert_eq!(selected.cell().cell.as_str(), "counter"); + assert_eq!(selected.schema(), route.schema_digest); + assert_eq!(selected.initial().txid, 0); + + // A lost reply is reconciled using the durable original creation record, + // even after subsequent mutations have replaced the initial cell image. + let saved = selected.encode().unwrap(); + let restarted = EnrollmentRecord::decode(&saved).unwrap(); + let retry = |request: &[u8], current: &GroupRoute| { + decide_enrollment( + current, + EnrollmentPhase::Active, + request, + Some(&restarted), + GroupEntryLimits::default(), + ) + }; + assert_eq!( + retry(&request, &route), + Ok(EnrollmentDecision::Existing(selected.clone())) + ); + + // These requests may both be prepared while no committed record exists. + // Rechecking against the selected record cannot replace its incarnation or + // repurpose the same operation ID for a different initial image. + for field in 0..4 { + let mut competing = original.clone(); + let GroupCommand::Enroll { cell, image, .. } = &mut competing.command else { + unreachable!() + }; + match field { + 0 => competing.binding.proposal = ProposalId::new([9; 16]).unwrap(), + 1 => cell.incarnation = IncarnationId::new([10; 16]).unwrap(), + 2 => cell.cell = CellId::new("other").unwrap(), + _ => image.epoch += 1, + } + let competitor = encoded(&competing, &route); + assert_ne!(admitted(&route, &competitor), selected); + assert_eq!(retry(&competitor, &route), Err(EnrollmentError::Conflict)); + } + + for field in 0..7 { + let mut current = route.clone(); + match field { + 0 => current.history.cluster = ClusterId::new([11; 16]).unwrap(), + 1 => current.history.namespace = NamespaceId::new([11; 16]).unwrap(), + 2 => current.history.group = GroupId::new([11; 16]).unwrap(), + 3 => current.history.incarnation = IncarnationId::new([11; 16]).unwrap(), + 4 => current.epoch = current.epoch.checked_next().unwrap(), + 5 => current.schema_digest = [11; 32], + _ => current.birth_digest = [11; 32], + } + assert!(matches!( + retry(&request, ¤t), + Err(EnrollmentError::Protocol(_)) + )); + } + + // A target at a half-open upper bound belongs to the neighboring range. + let mut outside = route.clone(); + outside.range = CellRange::new( + CellBound::NegativeInfinity, + CellBound::Key(cell_routing_key( + route.history.namespace, + &selected.cell().cell, + )), + ) + .unwrap(); + assert!(matches!( + retry(&request, &outside), + Err(EnrollmentError::Protocol(_)) + )); + + // Normal transfer rebinds the request to the current physical route while + // the immutable logical creation record remains unchanged. + let mut destination = route.clone(); + destination.history.group = GroupId::new([12; 16]).unwrap(); + destination.history.incarnation = IncarnationId::new([13; 16]).unwrap(); + destination.epoch = route.epoch.checked_next().unwrap(); + destination.birth_digest = [14; 32]; + let mut moved = original.clone(); + moved.binding = entry(&destination).binding; + assert_eq!( + retry(&encoded(&moved, &destination), &destination), + Ok(EnrollmentDecision::Existing(selected)) + ); + assert!(retry(&request, &destination).is_err()); +} + +#[test] +fn sealed_range_rejects_enrollment() { + let route = route(); + let mut entry = entry(&route); + let prepared_before_seal = encoded(&entry, &route); + let selected = admitted(&route, &prepared_before_seal); + for (phase, error) in [ + (EnrollmentPhase::Sealed, EnrollmentError::Sealed), + (EnrollmentPhase::Retired, EnrollmentError::Retired), + ] { + for existing in [None, Some(&selected)] { + assert_eq!( + decide_enrollment( + &route, + phase, + &prepared_before_seal, + existing, + GroupEntryLimits::default(), + ), + Err(error) + ); + } + } + entry.command = GroupCommand::Retire { + transition: TransitionId::new([17; 16]).unwrap(), + publication: [18; 32], + }; + assert_eq!( + decide_enrollment( + &route, + EnrollmentPhase::Active, + &encoded(&entry, &route), + None, + GroupEntryLimits::default(), + ), + Err(EnrollmentError::NotEnrollment) + ); +} + +#[test] +fn enrollment_record_is_bounded_canonical_data() { + let route = route(); + let mut entry = entry(&route); + let GroupCommand::Enroll { cell, .. } = &mut entry.command else { + unreachable!() + }; + cell.cell = CellId::new("\\".repeat(1024)).unwrap(); + let record = admitted(&route, &encoded(&entry, &route)); + let bytes = record.encode().unwrap(); + assert_eq!(EnrollmentRecord::decode(&bytes).unwrap(), record); + assert_eq!( + EnrollmentRecord::decode(&vec![0; 4097]), + Err(EnrollmentError::Limit) + ); + for end in 0..bytes.len() { + assert!(EnrollmentRecord::decode(&bytes[..end]).is_err()); + } + let mut spaced = bytes.clone(); + spaced.push(b' '); + assert!(EnrollmentRecord::decode(&spaced).is_err()); + let original: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + let replace_field = |field: &str, value: serde_json::Value| { + // Preserve canonical field order so refusal must come from the changed + // value, not from Value's map ordering when serializing the whole object. + String::from_utf8(bytes.clone()) + .unwrap() + .replacen( + &format!("\"{field}\":{}", original[field]), + &format!("\"{field}\":{value}"), + 1, + ) + .into_bytes() + }; + let another = replace_field("creation", serde_json::json!(vec![9; 16])); + assert_eq!( + EnrollmentRecord::decode(&another).unwrap().creation(), + ProposalId::new([9; 16]).unwrap() + ); + for (field, value) in [ + ("domain", serde_json::json!("cellar-cell-enrollment-2")), + ("cluster", serde_json::json!(vec![0; 16])), + ("creation", serde_json::json!(vec![0; 16])), + ("initial_epoch", serde_json::json!(0)), + ("initial_checksum", serde_json::json!(1)), + ] { + assert!( + EnrollmentRecord::decode(&replace_field(field, value)).is_err(), + "{field}" + ); + } + let mut unknown = bytes[..bytes.len() - 1].to_vec(); + unknown.extend_from_slice(b",\"extra\":true}"); + assert!(EnrollmentRecord::decode(&unknown).is_err()); + let mut damaged = encoded(&entry, &route); + *damaged.last_mut().unwrap() ^= 1; + assert!(matches!( + decide_enrollment( + &route, + EnrollmentPhase::Active, + &damaged, + None, + GroupEntryLimits::default() + ), + Err(EnrollmentError::Protocol(_)) + )); +} diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index f980d4c44..f24970c00 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -4,7 +4,7 @@ Status: executable design and implementation plan delivered on 2026-10-29. CG-01 contracts and CG-02 storage have passed their local evidence gates. -CG-03 routing and enrollment is next. +CG-03 routing and enrollment is in progress. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -68,7 +68,7 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -Start CG-03 after reading [progress](progress.md) and inspecting the working diff. +Resume CG-03 after reading [progress](progress.md) and inspecting the working diff. CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; an IN_PROGRESS task is resumed from its existing brief and working state. Each diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index 984d32ce4..0e26e48c1 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -14,7 +14,7 @@ never executed by the plan runner. Read the [design](design.md), | --- | --- | --- | --- | ---: | | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | -| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | TODO | 5 | +| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | IN_PROGRESS | 5 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index a251d8e0c..a7585a778 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -49,8 +49,8 @@ must include its initial replayable LTX chain. ## Remaining implementation -CG-01 and CG-02 have passed their complete task gates. CG-03 through CG-16 remain -TODO. The latest storage qualification is recorded below; the earlier sections +CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; +CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. The reviewed source baseline is `ec9d277e50416f786fe3c1cd851b4147f9cd4b43`. @@ -232,3 +232,33 @@ loss. Receipt rows are physical-state checks, not SDK deduplication qualificatio The shared host must still integrate authoritative root inventories, admission, quorum, snapshots and archives. No RustFS drill, split/merge, hibernation latency, density claim or final hard cutover is qualified by these tests. CG-03 is next. + + +## CG-03 — enrollment admission in progress + +The first CG-03 slice adds deterministic admission using the existing bounded +cell-addressed entry codec. Requests verify group history, route epoch, birth, +schema and the cell's half-open range. Sealed or retired lifecycle state refuses +delayed enrollment. Concurrent preparations must be checked again in committed +log order; only the selected original creation record can resolve a retry. + +The bounded canonical `EnrollmentRecord` binds logical cell history, original +creation operation, schema and initial image commitment. It survives normal +transfer without binding to the old physical group. The host must retain it +atomically with enrollment, preserve it through mutations and include it in +snapshots/archives. Returning an existing record grants no quorum/read authority +and must never restore the initial image over newer state. + +Three native admission cases pass: creation conflict/retry and route identity, +sealed/retired refusal, and bounded canonical record decoding. Negative record +checks preserve field order so rejection cannot be attributed only to JSON map +reordering. Empty records require the canonical empty image and zero checksum. +The initial build required explicit image wire fields; an intermediate running +suite exposed the absent empty-image checksum check. Failed and corrected logs +are retained with the checkpoint rather than suppressed. + +This is a pure admission boundary. The paged range tree and conditional namespace +root journal, lost birth/root response reconciliation, permanent blank-disk +refusal integration and bounded route discovery remain CG-03 work. CG-04 must +connect admission and durable creation records to the real shared group host. +CG-03 remains IN_PROGRESS; the production hard cutover is not complete. diff --git a/plans/multi-raft/cell-groups/protocols.md b/plans/multi-raft/cell-groups/protocols.md index a4930dbe7..0aa2c11a1 100644 --- a/plans/multi-raft/cell-groups/protocols.md +++ b/plans/multi-raft/cell-groups/protocols.md @@ -33,6 +33,16 @@ under that route epoch. The group checks the key is in its active range and no other live incarnation has enrolled that ID. The command is idempotent by its stable creation identity. Only committed/published enrollment becomes visible. +Persist the original creation operation, logical cell history, schema and initial +image commitment atomically with enrollment. Retain this creation record through +later mutations and normal transfers; comparing a retry to the latest image would +confuse a successful earlier creation with a conflict. The initial commitment is +provenance, not a requirement to retain the superseded initial image forever. +Snapshots and archives carry the record with the selected current cell state. +Admission decisions must be checked again in committed log order against the +current lifecycle state. Recognizing an exact retry grants neither current-term +authority nor permission to restore the initial image over newer mutations. + | Crash/collision | Recovery | | --- | --- | | Birth CAS result unknown | Read the canonical birth/mutation journal; never allocate competing voter history. | diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index f90b531f5..d08f9b922 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -264,7 +264,7 @@ { "id": "CG-03", "title": "Prepare paged range routing and enrollment admission", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -407,7 +407,7 @@ "test_target": "native_cell_groups_execution", "steps": [ { - "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands.", + "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands. Atomically retain the immutable enrollment creation record with the selected cell state; preserve it through mutations and transfer so retries never reinitialize or restore the initial image over newer state.", "tests": [ "three_cells_one_rawnode_per_voter", "concurrent_enrollment_one_cell" From 325214aca40990763ffaa4113033d7187aa25cf8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 12:31:31 -0700 Subject: [PATCH 017/568] docs: record cell-group enrollment checkpoint --- plans/multi-raft/cell-groups/progress.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index a7585a778..cddfa617b 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -262,3 +262,13 @@ root journal, lost birth/root response reconciliation, permanent blank-disk refusal integration and bounded route discovery remain CG-03 work. CG-04 must connect admission and durable creation records to the real shared group host. CG-03 remains IN_PROGRESS; the production hard cutover is not complete. + + +Enrollment checkpoint `b1d27582` passes its three native cases, all-target Clippy +with warnings denied, formatting/layout, native dependency isolation, docs, +source preflight and all 37 plan-validator tests. Evidence under +`.cellar/cell-groups-evidence/CG-03-enrollment/checkpoint.json` binds 40 source +inputs, nine successful checks and a copied native executable. The failed initial +build/test outputs and a dependency-check environment failure remain retained; +the latter passed after restoring the Cargo path. This is a partial checkpoint, +not a CG-03 DONE record. From dcc52840dfd84b6955a80ab5315b1392491838a7 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:01:41 -0700 Subject: [PATCH 018/568] docs: hand off executable cell-group implementation plan --- plans/multi-raft/cell-groups/README.md | 10 ++++++--- plans/multi-raft/cell-groups/progress.md | 27 ++++++++++++++++++++++-- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index f24970c00..9b1da228b 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The design package is delivered; implementation now proceeds toward the complete -shared-group runtime and hard cutover. Implementation checkpoints are recorded -separately; delivering this package does not mark the remaining tasks complete. +The requested deliverable is the executable design and implementation plan. +Implementation checkpoints are recorded separately; delivering this package does +not mark the remaining tasks complete or qualify the shared-group runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,6 +68,10 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff +The user's clarified scope for this handoff is the plan. Future implementation +follows the task briefs below; completion of this handoff does not instruct the +current executor to continue through the runtime cutover. + Resume CG-03 after reading [progress](progress.md) and inspecting the working diff. CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index cddfa617b..206912a4f 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,8 +3,31 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. Implementation is active and follows the full dependency order. Intermediate contracts do not qualify a shared -runtime. No production legacy/shared mode flag is being added. +hard cutover. The user's clarified deliverable is the executable plan; the +implementation checkpoints below describe existing work for a future executor. +Intermediate contracts do not qualify a shared runtime. No production +legacy/shared mode flag is being added. + +## Plan handoff — 2026-10-29 + +The package validator passes: 16 tasks, 83 required cases, 14 invariants and nine +profiles. Task briefs include source excerpts, file scope, dependencies, exact +commands and evidence requirements. The TiKV investigation pin still matches the +supplied checkout. The branch includes freshly fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. + +Uncommitted CG-03 directory work remains in this checkout: immutable range pages, +root publication and mutation journals, with native catalog tests. The observed +native run passes seven tests. This is not a complete CG-03 qualification and is +not included in a DONE record. Preserve the working diff for its next executor. + +The source preflight correctly refuses the draft `catalog.rs` module exports +against the recorded baseline. Consequently the current-checkout validator suite +passes 36 of 37 tests; its actual-source preflight test refuses that drift. Do not +blindly refresh the baseline or mark CG-03 complete. Review and qualify the draft, +commit its source, then record a successor baseline with updated excerpt lines. +Birth/disk-claim integration and bounded redirects remain outstanding. The final +release gate remains closed. ## CG-01 — complete at the contract boundary From a0c11969763edbe86ee08fc7d1650129058c8ee0 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:10:27 -0700 Subject: [PATCH 019/568] feat: publish bounded cell-group range directory with exact retry journal --- crates/cellar/durability/raft/catalog.rs | 6 + .../durability/raft/cell_groups/README.md | 46 +- .../raft/cell_groups/directory/mod.rs | 596 ++++++++++++++++++ .../raft/cell_groups/directory/store.rs | 128 ++++ .../raft/cell_groups/directory/tree.rs | 209 ++++++ .../raft/cell_groups/directory/wire.rs | 231 +++++++ .../cellar/durability/raft/cell_groups/mod.rs | 3 + .../tests/native_cell_groups_catalog.rs | 31 + .../tests/support/cell_group_directory.rs | 488 ++++++++++++++ .../tests/support/raft_catalog_fault.rs | 16 + plans/multi-raft/cell-groups/tasks.json | 11 +- 11 files changed, 1760 insertions(+), 5 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/store.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/tree.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/wire.rs create mode 100644 crates/cellar/tests/support/cell_group_directory.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 363d1c35a..117f05027 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -22,6 +22,12 @@ use std::{collections::BTreeSet, fmt, sync::Arc}; mod births; pub use births::BirthDisposition; +#[path = "cell_groups/directory/mod.rs"] +mod range_directory; +pub use range_directory::{ + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, +}; + #[path = "catalog_inventory/mod.rs"] mod inventory; pub(crate) use inventory::{BirthImportCut, ManagedBirthInventory}; diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 2ff58ccf0..392fcfdd2 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -132,5 +132,47 @@ and include it in complete snapshots/archives. An exact retry must not initializ again or roll the current image back to the initial one. Current-term quorum and read authority remain required from the host even for a recognized retry. -Paged range-directory publication, birth reconciliation and host integration are -still pending. Pure admission tests do not qualify those boundaries. +## Range directory (CG-03 in progress) + +`RangeDirectory` stores a namespace-bound, immutable AVL range tree. Each page +binds its route, child hashes, subtree bounds, height, count and coverage. A lookup +descends one path and verifies each page; the cache has independent page and byte +limits. A pinned `DirectoryView` can remain readable after a topology change and +requires no head refresh for lookup. A fully cached path performs no S3 requests. +Views are hints; the destination must check its own current route and quorum +authority. Observation does not traverse a complete recovery inventory. + +Initialization publishes at most 128 contiguous ranges covering the whole routing +domain. A replacement removes up to 64 exact parent routes and inserts up to 64 +fresh groups covering the identical contiguous union. Children advance the maximum +parent epoch by one. Paths are copied incrementally; incomplete intermediate trees +cannot become a selected root. Group IDs remain reserved after retirement. + +`DirectoryMutation` contains canonical bounded bytes binding the namespace, +operation, expected head and exact routes. Persist these bytes before submission; +after an unknown result, decode and retry the identical request. One conditional +head update selects the tree, immutable journal and operation/group indexes. +Read-back confirms the selected operation even after another publisher advances +the head. Reusing an operation for different bytes refuses. The returned receipt +describes its historical decision, not the latest route or serving authority. + +Genesis writes the head before a permanent pin. An interrupted initializer can +finish the same genesis, including after another initializer pins and advances it. +A missing head with a retained pin refuses reinitialization. Head observation +verifies the pin, selected journal and operation-index binding. Missing or corrupt +objects refuse instead of returning an empty directory. No namespace-wide listing +is used, and historical pages/journals are retained; remote metadata GC is not +implemented here. + +Run async operations within the owning execution domain. Each operation has an +explicit timeout; a timed-out publication is unconfirmed and retains its original +identity for reconciliation. Cache locks never span I/O. Object and traversal +limits bound individual operations; defaults are not a qualified capacity claim. + +Directory publication is metadata only. The future topology owner must obtain +the seal, installation, archive and activation proofs before exposing replacement +ownership. Group birth/permanent disk claims, managed admission and bounded client +redirects remain to be integrated. Native directory tests exercise real conditional +object-store operations in memory, including races, dropped replies, corruption, +restart and bounded split/merge churn. They do not qualify live Raft ownership, +real RustFS or the full CG-03 gate. diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs new file mode 100644 index 000000000..65a49632c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -0,0 +1,596 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Range discovery and conditional topology metadata. Neither a route nor a +//! directory publication grants Raft, transfer, birth or serving authority. + +use super::{CatalogError, NamespaceCatalog}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{catalog_wire as record, coordination_index::Index}; +use cellar_logic::{ + cell_group::*, + consensus::{GroupId, IncarnationId, NamespaceId, ProposalId}, +}; +use object_store::{path::Path, PutMode, PutOptions, UpdateVersion}; +use std::{ + collections::BTreeSet, + sync::{Arc, Mutex}, + time::Duration, +}; + +mod store; +mod tree; +mod wire; + +#[derive(Clone, Copy, Debug)] +pub struct DirectoryLimits { + pub max_groups: u32, + pub max_mutations: u64, + pub max_cached_pages: usize, + pub max_cache_bytes: usize, + pub operation_timeout_ms: u64, +} + +impl Default for DirectoryLimits { + fn default() -> Self { + Self { + max_groups: 1_000_000, + max_mutations: 1_000_000, + max_cached_pages: 64, + max_cache_bytes: 4 * 1024 * 1024, + operation_timeout_ms: 5000, + } + } +} + +#[derive(Clone)] +pub struct RangeDirectory { + namespace: NamespaceCatalog, + limits: DirectoryLimits, + cache: Arc>, +} + +/// A pinned catalog observation, not authority or an automatic freshness claim. +#[derive(Clone)] +pub struct DirectoryView { + head: wire::Head, + digest: [u8; 32], + owner: Arc<()>, +} + +impl DirectoryView { + pub fn revision(&self) -> u64 { + self.head.revision + } + + pub fn groups(&self) -> u32 { + self.head.tree.groups + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } +} + +/// Persist these canonical bytes before issuing a topology mutation. Resuming +/// decodes the same expected root and operation, rather than rebasing a retry. +pub struct DirectoryMutation { + request: wire::Request, + bytes: Vec, + digest: [u8; 32], + operation: ProposalId, +} + +impl DirectoryMutation { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn operation(&self) -> ProposalId { + self.operation + } +} + +/// Exact historical catalog decision. A later directory head may supersede its +/// ranges; destination admission still needs the replicated transition protocol. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct DirectoryPublication { + operation: ProposalId, + revision: u64, + tree: [u8; 32], + journal: [u8; 32], +} + +impl DirectoryPublication { + pub fn operation(&self) -> ProposalId { + self.operation + } + + pub fn revision(&self) -> u64 { + self.revision + } + + pub fn tree_digest(&self) -> [u8; 32] { + self.tree + } + + pub fn journal_digest(&self) -> [u8; 32] { + self.journal + } +} + +impl RangeDirectory { + pub fn new(namespace: NamespaceCatalog, limits: DirectoryLimits) -> Result { + namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; + if !(1..=1_000_000).contains(&limits.max_groups) + || !(1..=1_000_000).contains(&limits.max_mutations) + || limits.max_cached_pages > 1024 + || limits.max_cache_bytes > 64 * 1024 * 1024 + || (limits.max_cached_pages == 0) != (limits.max_cache_bytes == 0) + || !(1..=60_000).contains(&limits.operation_timeout_ms) + { + return Err(CatalogError::Limit); + } + Ok(Self { + namespace, + limits, + cache: Arc::new(Mutex::new(store::Cache::default())), + }) + } + + fn check_view(&self, view: &DirectoryView) -> Result<(), CatalogError> { + if !Arc::ptr_eq(&view.owner, &self.namespace.inner.owner) + || view.head.namespace != self.namespace.root_digest + { + return Err(CatalogError::Conflict("foreign directory view")); + } + Ok(()) + } + + fn request( + &self, + operation: ProposalId, + expected: Option<[u8; 32]>, + parents: &[GroupRoute], + children: &[GroupRoute], + ) -> Result { + if parents.len() > 64 || children.len() > if expected.is_none() { 128 } else { 64 } { + return Err(CatalogError::Limit); + } + let request = wire::Request { + namespace: self.namespace.root_digest, + operation: operation.bytes(), + expected, + parents: parents + .iter() + .map(|r| wire::Route::encode(r, self)) + .collect::>()?, + children: children + .iter() + .map(|r| wire::Route::encode(r, self)) + .collect::>()?, + }; + self.decode_mutation(&record::encode(wire::REQUEST, &request, wire::LIMIT)?) + } + + pub fn prepare_initialization( + &self, + operation: ProposalId, + ranges: &[GroupRoute], + ) -> Result { + self.request(operation, None, &[], ranges) + } + + /// Prepare metadata for one exact contiguous replacement. The topology + /// owner must separately obtain seal/install/archive/activation proofs before + /// using this publication; this staging API does not perform a transfer. + pub fn prepare_replacement( + &self, + view: &DirectoryView, + operation: ProposalId, + parents: &[GroupRoute], + children: &[GroupRoute], + ) -> Result { + self.check_view(view)?; + self.request(operation, Some(view.digest), parents, children) + } + + pub fn decode_mutation(&self, bytes: &[u8]) -> Result { + let request: wire::Request = record::decode(wire::REQUEST, bytes, wire::LIMIT)?; + self.check_request(&request)?; + let operation = ProposalId::new(request.operation) + .map_err(|_| CatalogError::Invalid("directory operation"))?; + Ok(DirectoryMutation { + request, + operation, + digest: record::digest(bytes), + bytes: bytes.to_vec(), + }) + } + + fn union(&self, routes: &[wire::Route]) -> Result { + let first = routes + .first() + .ok_or(CatalogError::Invalid("empty range set"))? + .decode(self)?; + let mut range = first.range; + let mut ids = BTreeSet::from([first.history.group]); + for route in &routes[1..] { + let route = route.decode(self)?; + if !ids.insert(route.history.group) { + return Err(CatalogError::Conflict("duplicate range group")); + } + range = range + .merge(&route.range) + .map_err(|_| CatalogError::Conflict("overlap or gap in range set"))?; + } + Ok(range) + } + + fn check_request(&self, request: &wire::Request) -> Result<(), CatalogError> { + if request.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("directory namespace")); + } + if request.children.is_empty() + || request.children.len() > if request.expected.is_none() { 128 } else { 64 } + || request.parents.len() > 64 + { + return Err(CatalogError::Limit); + } + let children = self.union(&request.children)?; + let epoch = if request.expected.is_none() { + if !request.parents.is_empty() || children != CellRange::all() { + return Err(CatalogError::Invalid("initial range coverage")); + } + 1 + } else { + if self.union(&request.parents)? != children { + return Err(CatalogError::Conflict("changed range union")); + } + request + .parents + .iter() + .map(|r| r.epoch) + .max() + .ok_or(CatalogError::Invalid("parent ranges"))? + .checked_add(1) + .ok_or(CatalogError::Limit)? + }; + if request.children.iter().any(|r| r.epoch != epoch) { + return Err(CatalogError::Conflict("range epoch successor")); + } + Ok(()) + } + + fn check_head(&self, head: &wire::Head) -> Result<(), CatalogError> { + if head.namespace != self.namespace.root_digest + || head.revision == 0 + || head.revision > self.limits.max_mutations + || head.genesis == [0; 32] + || head.journal == [0; 32] + || head.operations.is_none() + || head.groups.is_none() + { + return Err(CatalogError::Corrupt("directory head")); + } + head.tree.require_complete(self) + } + + async fn head(&self) -> Result, CatalogError> { + self.namespace + .inner + .read_versioned_bounded(&self.key("head"), wire::LIMIT) + .await? + .map(|(bytes, version)| { + let head: wire::Head = record::decode(wire::HEAD, &bytes, wire::LIMIT)?; + self.check_head(&head)?; + if !version.e_tag.as_ref().is_some_and(|s| !s.is_empty()) + && !version.version.as_ref().is_some_and(|s| !s.is_empty()) + { + return Err(CatalogError::Corrupt("directory CAS token")); + } + Ok(( + DirectoryView { + head, + digest: record::digest(&bytes), + owner: self.namespace.inner.owner.clone(), + }, + version, + )) + }) + .transpose() + } + + async fn current(&self) -> Result<(DirectoryView, UpdateVersion), CatalogError> { + let pin: wire::Pin = record::decode( + wire::PIN, + &self + .read("pin") + .await? + .ok_or(CatalogError::Corrupt("missing directory pin"))?, + wire::LIMIT, + )?; + let (view, version) = self + .head() + .await? + .ok_or(CatalogError::Corrupt("missing directory head"))?; + if pin.namespace != self.namespace.root_digest || pin.genesis != view.head.genesis { + return Err(CatalogError::Corrupt("directory genesis pin")); + } + let journal = self.journal(view.head.journal).await?; + if journal.revision != view.head.revision + || journal.tree != view.head.tree + || (journal.revision == 1 && view.head.genesis != view.head.journal) + || Index::new(self) + .lookup(view.head.operations, &hex::encode(journal.operation)) + .await? + != Some(view.head.journal.to_vec()) + { + return Err(CatalogError::Corrupt("directory head/journal disagreement")); + } + Ok((view, version)) + } + + async fn bounded( + &self, + work: impl std::future::Future>, + ) -> Result { + crate::asyncrt::timeout( + Duration::from_millis(self.limits.operation_timeout_ms), + work, + ) + .await + .map_err(|_| CatalogError::Unconfirmed)? + } + + pub async fn observe(&self) -> Result { + self.bounded(async { Ok(self.current().await?.0) }).await + } + + /// Looks up only an immutable tree path. No head refresh, namespace listing + /// or object-store call is required when that path is already cached. + pub async fn lookup( + &self, + view: &DirectoryView, + cell: &CellId, + ) -> Result { + self.check_view(view)?; + self.bounded(self.lookup_tree(&view.head.tree, cell)).await + } + + async fn confirmed( + &self, + head: &wire::Head, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + let key = hex::encode(mutation.request.operation); + let Some(value) = Index::new(self).lookup(head.operations, &key).await? else { + return Ok(None); + }; + let id: [u8; 32] = value + .try_into() + .map_err(|_| CatalogError::Corrupt("directory journal pointer"))?; + let journal = self.journal(id).await?; + if journal.operation != mutation.request.operation || journal.revision > head.revision { + return Err(CatalogError::Corrupt("directory journal identity")); + } + if journal.request != mutation.digest { + return Err(CatalogError::Conflict("directory operation reused")); + } + Ok(Some(DirectoryPublication { + operation: mutation.operation, + revision: journal.revision, + tree: journal.tree.digest, + journal: id, + })) + } + + async fn journal(&self, id: [u8; 32]) -> Result { + let bytes = self + .read(&format!("journal/{}", hex::encode(id))) + .await? + .ok_or(CatalogError::Corrupt("missing directory journal"))?; + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("directory journal hash")); + } + let journal: wire::Journal = record::decode(wire::JOURNAL, &bytes, wire::LIMIT)?; + journal.tree.require_complete(self)?; + if journal.namespace != self.namespace.root_digest + || journal.revision == 0 + || journal.revision > self.limits.max_mutations + || ProposalId::new(journal.operation).is_err() + || journal.previous.is_none() != (journal.revision == 1) + { + return Err(CatalogError::Corrupt("directory journal identity")); + } + Ok(journal) + } + + /// `None` is an observation, never permission to replace the operation ID + /// of a potentially submitted mutation. Re-submit the identical bytes. + pub async fn publication( + &self, + mutation: &DirectoryMutation, + ) -> Result, CatalogError> { + self.check_request(&mutation.request)?; + self.bounded(async { + self.confirmed(&self.current().await?.0.head, mutation) + .await + }) + .await + } + + pub async fn publish( + &self, + mutation: &DirectoryMutation, + ) -> Result { + self.check_request(&mutation.request)?; + self.bounded(self.publish_inner(mutation)).await + } + + async fn publish_inner( + &self, + mutation: &DirectoryMutation, + ) -> Result { + let request = &mutation.request; + let current = if request.expected.is_some() { + Some(self.current().await?) + } else { + let pin = self.read("pin").await?; + let head = self.head().await?; + if pin.is_some() { + let (view, _) = self.current().await?; + return self + .confirmed(&view.head, mutation) + .await? + .ok_or(CatalogError::Conflict("directory already initialized")); + } + if head.as_ref().is_some_and(|(v, _)| v.head.revision != 1) { + return Err(CatalogError::Corrupt("missing active directory pin")); + } + head + }; + if let Some((view, _)) = ¤t { + if let Some(publication) = self.confirmed(&view.head, mutation).await? { + if request.expected.is_some() { + return Ok(publication); + } + // An interrupted genesis resumes only its identical first head. + } + if request + .expected + .is_some_and(|expected| expected != view.digest) + { + return Err(CatalogError::Conflict("stale directory root")); + } + } + let base = if request.expected.is_some() { + current.as_ref().map(|(v, _)| &v.head) + } else { + None + }; + let revision = base + .map_or(Some(1), |h| h.revision.checked_add(1)) + .ok_or(CatalogError::Limit)?; + if revision > self.limits.max_mutations { + return Err(CatalogError::Limit); + } + let mut tree = base.map(|h| h.tree.clone()); + for parent in &request.parents { + tree = self.remove(tree, parent.clone(), 0).await?; + } + let mut groups = base.and_then(|h| h.groups); + for child in &request.children { + let key = hex::encode(child.group); + if Index::new(self).lookup(groups, &key).await?.is_some() { + return Err(CatalogError::Conflict("group identity already assigned")); + } + groups = Index::new(self) + .update(groups, &key, Some(child.incarnation.to_vec()), 0) + .await?; + tree = Some(self.insert(tree, child.clone(), 0).await?); + } + let tree = tree.ok_or(CatalogError::Invalid("empty directory"))?; + tree.require_complete(self)?; + let journal = wire::Journal { + namespace: self.namespace.root_digest, + operation: request.operation, + request: mutation.digest, + revision, + previous: base.map(|h| h.journal), + tree: tree.clone(), + }; + let bytes = record::encode(wire::JOURNAL, &journal, wire::LIMIT)?; + let id = record::digest(&bytes); + self.immutable("journal", id, &bytes).await?; + let operations = Index::new(self) + .update( + base.and_then(|h| h.operations), + &hex::encode(request.operation), + Some(id.to_vec()), + 0, + ) + .await?; + let head = wire::Head { + namespace: self.namespace.root_digest, + genesis: base.map_or(id, |h| h.genesis), + revision, + tree, + operations, + groups, + journal: id, + }; + self.check_head(&head)?; + let bytes = record::encode(wire::HEAD, &head, wire::LIMIT)?; + if request.expected.is_none() { + // Unlike an immutable page, this object may advance after Create + // succeeds and before its read-back. Confirm via its pinned journal. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Create), + ) + .await; + let (observed, _) = self.head().await?.ok_or(CatalogError::Unconfirmed)?; + if observed.head != head { + if observed.head.genesis != head.genesis { + return Err(CatalogError::Conflict("directory genesis winner")); + } + let (selected, _) = self.current().await?; + return self + .confirmed(&selected.head, mutation) + .await? + .ok_or(CatalogError::Corrupt("missing genesis operation")); + } + let pin = record::encode( + wire::PIN, + &wire::Pin { + namespace: head.namespace, + genesis: head.genesis, + }, + wire::LIMIT, + )?; + if self + .namespace + .inner + .create_and_observe_bounded(&self.key("pin"), &pin, wire::LIMIT) + .await? + != pin + { + return Err(CatalogError::Corrupt("directory pin collision")); + } + } else { + let (_, version) = current.ok_or(CatalogError::Conflict("missing directory base"))?; + // A transport error can follow a committed CAS. Reconcile by the + // root-selected operation index, including after subsequent writes. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + } + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + match self.confirmed(&after.head, mutation).await { + Ok(Some(publication)) => Ok(publication), + Ok(None) if request.expected.is_some_and(|base| base != after.digest) => { + Err(CatalogError::Conflict("directory CAS lost")) + } + _ => Err(CatalogError::Unconfirmed), + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/store.rs b/crates/cellar/durability/raft/cell_groups/directory/store.rs new file mode 100644 index 000000000..c0140f6a2 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/store.rs @@ -0,0 +1,128 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::coordination_index::PageStore; +use bytes::Bytes; +use futures_util::future::BoxFuture; + +#[derive(Default)] +pub(super) struct Cache { + pages: std::collections::BTreeMap<[u8; 32], Bytes>, + order: std::collections::VecDeque<[u8; 32]>, + bytes: usize, +} + +impl RangeDirectory { + pub(super) fn key(&self, suffix: &str) -> Path { + Path::from(format!( + "{}cell-groups/{suffix}", + self.namespace.namespace_key() + )) + } + + pub(super) async fn read(&self, suffix: &str) -> Result>, CatalogError> { + Ok(self + .namespace + .inner + .read_versioned_bounded(&self.key(suffix), wire::LIMIT) + .await? + .map(|r| r.0)) + } + + pub(super) async fn immutable( + &self, + category: &str, + id: [u8; 32], + bytes: &[u8], + ) -> Result<(), CatalogError> { + if record::digest(bytes) != id { + return Err(CatalogError::Invalid("immutable range object digest")); + } + let key = self.key(&format!("{category}/{}", hex::encode(id))); + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, bytes, wire::LIMIT) + .await?; + if observed != bytes { + return Err(CatalogError::Corrupt("immutable range object collision")); + } + Ok(()) + } + + pub(super) async fn range_bytes(&self, id: [u8; 32]) -> Result { + if let Some(bytes) = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))? + .pages + .get(&id) + .cloned() + { + return Ok(bytes); + } + let bytes = Bytes::from( + self.read(&format!("ranges/{}", hex::encode(id))) + .await? + .ok_or(CatalogError::Corrupt("missing range page"))?, + ); + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("range page digest")); + } + let mut cache = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))?; + if self.limits.max_cached_pages > 0 + && bytes.len() <= self.limits.max_cache_bytes + && !cache.pages.contains_key(&id) + { + while cache.pages.len() >= self.limits.max_cached_pages + || cache.bytes + bytes.len() > self.limits.max_cache_bytes + { + let oldest = cache + .order + .pop_front() + .ok_or(CatalogError::Corrupt("range cache accounting"))?; + if let Some(removed) = cache.pages.remove(&oldest) { + cache.bytes -= removed.len(); + } + } + cache.bytes += bytes.len(); + cache.order.push_back(id); + cache.pages.insert(id, bytes.clone()); + } + Ok(bytes) + } + + /// Cached immutable bytes only. Returned routes remain hints and every + /// destination must verify the current route and its own quorum authority. + pub fn cache_usage(&self) -> Result<(usize, usize), CatalogError> { + let cache = self + .cache + .lock() + .map_err(|_| CatalogError::Corrupt("range cache"))?; + Ok((cache.pages.len(), cache.bytes)) + } +} + +impl PageStore for RangeDirectory { + type Error = CatalogError; + + fn page_limit(&self) -> usize { + wire::LIMIT + } + + fn read_page(&self, id: [u8; 32]) -> BoxFuture<'_, Result> { + Box::pin(async move { + self.read(&format!("index/{}", hex::encode(id))) + .await? + .map(Bytes::from) + .ok_or(CatalogError::Corrupt("missing range index page")) + }) + } + + fn create_page(&self, id: [u8; 32], bytes: Vec) -> BoxFuture<'_, Result<(), CatalogError>> { + Box::pin(async move { self.immutable("index", id, &bytes).await }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/tree.rs b/crates/cellar/durability/raft/cell_groups/directory/tree.rs new file mode 100644 index 000000000..b3ca5e197 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/tree.rs @@ -0,0 +1,209 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent ordered AVL pages. Intermediate edits may have gaps; only a +//! complete all-domain tree can enter a published namespace root. + +use super::{wire::*, *}; +use futures_util::future::BoxFuture; + +impl RangeDirectory { + pub(super) async fn page(&self, reference: &Reference) -> Result { + reference.range(self)?; + let bytes = self.range_bytes(reference.digest).await?; + let page: Page = record::decode(PAGE, &bytes, LIMIT)?; + if page.reference(&bytes, self)? != *reference { + return Err(CatalogError::Corrupt("range page reference")); + } + Ok(page) + } + + async fn save(&self, page: Page) -> Result { + let bytes = record::encode(PAGE, &page, LIMIT)?; + let reference = page.reference(&bytes, self)?; + self.immutable("ranges", reference.digest, &bytes).await?; + Ok(reference) + } + + pub(super) async fn lookup_tree( + &self, + root: &Reference, + cell: &CellId, + ) -> Result { + root.require_complete(self)?; + let key = cell_routing_key(self.namespace.config.namespace, cell); + let mut next = Some(root.clone()); + for _ in 0..32 { + let page = self + .page( + &next + .take() + .ok_or(CatalogError::Corrupt("range coverage hole"))?, + ) + .await?; + let route = page.route.decode(self)?; + if route.range.contains(&key) { + return Ok(route); + } + next = if CellBound::Key(key.clone()) < *route.range.lower() { + page.left + } else { + page.right + }; + } + Err(CatalogError::Limit) + } + + pub(super) fn insert<'a>( + &'a self, + root: Option, + route: Route, + depth: u8, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + if depth >= 32 { + return Err(CatalogError::Limit); + } + let Some(reference) = root else { + return self + .save(Page { + namespace: self.namespace.root_digest, + route, + left: None, + right: None, + }) + .await; + }; + let mut page = self.page(&reference).await?; + match route + .decode(self)? + .range + .lower() + .cmp(page.route.decode(self)?.range.lower()) + { + std::cmp::Ordering::Less => { + page.left = Some(self.insert(page.left.take(), route, depth + 1).await?) + } + std::cmp::Ordering::Greater => { + page.right = Some(self.insert(page.right.take(), route, depth + 1).await?) + } + std::cmp::Ordering::Equal => { + return Err(CatalogError::Conflict("duplicate range lower bound")) + } + } + self.balance(page).await + }) + } + + pub(super) fn remove<'a>( + &'a self, + root: Option, + expected: Route, + depth: u8, + ) -> BoxFuture<'a, Result, CatalogError>> { + Box::pin(async move { + if depth >= 32 { + return Err(CatalogError::Limit); + } + let reference = root.ok_or(CatalogError::Conflict("missing parent range"))?; + let mut page = self.page(&reference).await?; + match expected + .decode(self)? + .range + .lower() + .cmp(page.route.decode(self)?.range.lower()) + { + std::cmp::Ordering::Less => { + page.left = self.remove(page.left.take(), expected, depth + 1).await? + } + std::cmp::Ordering::Greater => { + page.right = self.remove(page.right.take(), expected, depth + 1).await? + } + std::cmp::Ordering::Equal => { + if page.route != expected { + return Err(CatalogError::Conflict("stale parent range")); + } + match (&page.left, &page.right) { + (None, _) => return Ok(page.right), + (_, None) => return Ok(page.left), + (Some(_), Some(right)) => { + let mut successor = self.page(right).await?; + for _ in depth..32 { + match &successor.left { + None => break, + Some(left) => successor = self.page(left).await?, + } + } + if successor.left.is_some() { + return Err(CatalogError::Limit); + } + page.route = successor.route; + page.right = self + .remove(page.right.take(), page.route.clone(), depth + 1) + .await?; + } + } + } + } + Ok(Some(self.balance(page).await?)) + }) + } + + async fn balance(&self, mut page: Page) -> Result { + let height = |r: &Option| r.as_ref().map_or(0, |r| i16::from(r.height)); + let balance = height(&page.left) - height(&page.right); + if balance > 1 { + let mut left = self + .page( + page.left + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + if height(&left.left) < height(&left.right) { + let mut middle = self + .page( + left.right + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + left.right = middle.left.take(); + middle.left = Some(self.save(left).await?); + page.left = middle.right.take(); + middle.right = Some(self.save(page).await?); + return self.save(middle).await; + } + page.left = left.right.take(); + left.right = Some(self.save(page).await?); + return self.save(left).await; + } + if balance < -1 { + let mut right = self + .page( + page.right + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + if height(&right.right) < height(&right.left) { + let mut middle = self + .page( + right + .left + .as_ref() + .ok_or(CatalogError::Corrupt("range rotation"))?, + ) + .await?; + right.left = middle.right.take(); + middle.right = Some(self.save(right).await?); + page.right = middle.left.take(); + middle.left = Some(self.save(page).await?); + return self.save(middle).await; + } + page.right = right.left.take(); + right.left = Some(self.save(page).await?); + return self.save(right).await; + } + self.save(page).await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs new file mode 100644 index 000000000..3383932b9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -0,0 +1,231 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use serde::{Deserialize, Serialize}; + +pub(super) const PAGE: u8 = 81; +pub(super) const HEAD: u8 = 82; +pub(super) const JOURNAL: u8 = 83; +pub(super) const PIN: u8 = 84; +pub(super) const REQUEST: u8 = 85; +pub(super) const LIMIT: usize = 64 * 1024; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "bound", content = "cell", deny_unknown_fields)] +pub(super) enum Bound { + NegativeInfinity, + Key(String), + PositiveInfinity, +} + +impl Bound { + pub fn from_bound(value: &CellBound) -> Self { + match value { + CellBound::NegativeInfinity => Self::NegativeInfinity, + CellBound::Key(k) => Self::Key(k.cell().as_str().into()), + CellBound::PositiveInfinity => Self::PositiveInfinity, + } + } + + pub fn decode(&self, ns: NamespaceId) -> Result { + Ok(match self { + Self::NegativeInfinity => CellBound::NegativeInfinity, + Self::PositiveInfinity => CellBound::PositiveInfinity, + Self::Key(cell) => CellBound::Key(cell_routing_key( + ns, + &CellId::new(cell).map_err(|_| CatalogError::Corrupt("range cell ID"))?, + )), + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Route { + pub group: [u8; 16], + pub incarnation: [u8; 16], + pub lower: Bound, + pub upper: Bound, + pub epoch: u64, + pub birth: [u8; 32], +} + +impl Route { + pub fn encode(value: &GroupRoute, owner: &RangeDirectory) -> Result { + let route = Self { + group: value.history.group.bytes(), + incarnation: value.history.incarnation.bytes(), + lower: Bound::from_bound(value.range.lower()), + upper: Bound::from_bound(value.range.upper()), + epoch: value.epoch.get(), + birth: value.birth_digest, + }; + if route.decode(owner)? != *value { + return Err(CatalogError::Invalid("range route binding")); + } + Ok(route) + } + + pub fn decode(&self, owner: &RangeDirectory) -> Result { + let config = owner.namespace.config(); + if self.birth == [0; 32] { + return Err(CatalogError::Corrupt("range birth digest")); + } + Ok(GroupRoute { + history: GroupHistory { + cluster: config.cluster, + namespace: config.namespace, + group: GroupId::new(self.group) + .map_err(|_| CatalogError::Corrupt("range group"))?, + incarnation: IncarnationId::new(self.incarnation) + .map_err(|_| CatalogError::Corrupt("range incarnation"))?, + }, + range: CellRange::new( + self.lower.decode(config.namespace)?, + self.upper.decode(config.namespace)?, + ) + .map_err(|_| CatalogError::Corrupt("range bounds"))?, + epoch: RouteEpoch::new(self.epoch).map_err(|_| CatalogError::Corrupt("range epoch"))?, + birth_digest: self.birth, + schema_digest: config.schema_digest, + }) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Reference { + pub digest: [u8; 32], + pub height: u8, + pub groups: u32, + pub lower: Bound, + pub upper: Bound, + pub complete: bool, +} + +impl Reference { + pub fn range(&self, owner: &RangeDirectory) -> Result { + if self.height == 0 + || self.height > 32 + || self.groups == 0 + || self.groups > owner.limits.max_groups + { + return Err(CatalogError::Limit); + } + CellRange::new( + self.lower.decode(owner.namespace.config.namespace)?, + self.upper.decode(owner.namespace.config.namespace)?, + ) + .map_err(|_| CatalogError::Corrupt("page bounds")) + } + + pub fn require_complete(&self, owner: &RangeDirectory) -> Result<(), CatalogError> { + if self.range(owner)? != CellRange::all() || !self.complete { + return Err(CatalogError::Corrupt("incomplete range directory")); + } + Ok(()) + } +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Page { + pub namespace: [u8; 32], + pub route: Route, + pub left: Option, + pub right: Option, +} + +impl Page { + pub fn reference( + &self, + bytes: &[u8], + owner: &RangeDirectory, + ) -> Result { + if self.namespace != owner.namespace.root_digest { + return Err(CatalogError::Corrupt("range page namespace")); + } + let route = self.route.decode(owner)?; + let height = |r: &Option| r.as_ref().map_or(0, |r| r.height); + if height(&self.left).abs_diff(height(&self.right)) > 1 { + return Err(CatalogError::Corrupt("unbalanced range page")); + } + let mut groups = 1_u32; + let mut lower = route.range.lower().clone(); + let mut upper = route.range.upper().clone(); + let mut complete = true; + if let Some(left) = &self.left { + let range = left.range(owner)?; + if range.upper() > route.range.lower() { + return Err(CatalogError::Corrupt("overlapping left ranges")); + } + complete &= left.complete && range.upper() == route.range.lower(); + lower = range.lower().clone(); + groups = groups.checked_add(left.groups).ok_or(CatalogError::Limit)?; + } + if let Some(right) = &self.right { + let range = right.range(owner)?; + if range.lower() < route.range.upper() { + return Err(CatalogError::Corrupt("overlapping right ranges")); + } + complete &= right.complete && range.lower() == route.range.upper(); + upper = range.upper().clone(); + groups = groups + .checked_add(right.groups) + .ok_or(CatalogError::Limit)?; + } + let result = Reference { + digest: record::digest(bytes), + height: height(&self.left) + .max(height(&self.right)) + .checked_add(1) + .ok_or(CatalogError::Limit)?, + groups, + lower: Bound::from_bound(&lower), + upper: Bound::from_bound(&upper), + complete, + }; + result.range(owner)?; + Ok(result) + } +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Head { + pub namespace: [u8; 32], + pub genesis: [u8; 32], + pub revision: u64, + pub tree: Reference, + pub operations: Option<[u8; 32]>, + pub groups: Option<[u8; 32]>, + pub journal: [u8; 32], +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub(super) struct Pin { + pub namespace: [u8; 32], + pub genesis: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Request { + pub namespace: [u8; 32], + pub operation: [u8; 16], + pub expected: Option<[u8; 32]>, + pub parents: Vec, + pub children: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Journal { + pub namespace: [u8; 32], + pub operation: [u8; 16], + pub request: [u8; 32], + pub revision: u64, + pub previous: Option<[u8; 32]>, + pub tree: Reference, +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 49fa5d3e2..2098cf876 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -17,6 +17,9 @@ pub use manifest::{ }; pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; +pub use crate::raft::catalog::{ + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, +}; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, }; diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index 1b4ad5b9f..82f8602ef 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -9,6 +9,37 @@ use cellar::application::consensus::{ }; use sha2::{Digest, Sha256}; +#[path = "support/cell_group_directory.rs"] +mod directory; +#[path = "support/raft_catalog_fault.rs"] +#[allow(dead_code)] +mod fault; + +#[tokio::test] +async fn range_directory_has_single_owner() -> anyhow::Result<()> { + directory::single_owner().await +} + +#[tokio::test] +async fn bounded_catalog_discovery() -> anyhow::Result<()> { + directory::bounded_discovery().await +} + +#[tokio::test] +async fn directory_reconciles_unknown_and_refuses_missing_state() -> anyhow::Result<()> { + directory::recovery().await +} + +#[tokio::test] +async fn directory_genesis_retry_after_later_mutation() -> anyhow::Result<()> { + directory::genesis_race().await +} + +#[tokio::test] +async fn directory_churn_preserves_exact_partition_and_old_views() -> anyhow::Result<()> { + directory::churn().await +} + fn route() -> GroupRoute { GroupRoute { history: GroupHistory { diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs new file mode 100644 index 000000000..793510546 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -0,0 +1,488 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real conditional object-store operations with dropped responses. No fake +//! publication successes, voter claims or Raft authority are synthesized here. + +use super::*; +use anyhow::{Context, Result}; +use cellar::application::consensus::verification::{ + Catalog, CatalogError, CatalogOptions, NamespaceConfig, +}; +use cellar::sdk::ExecutionDomain; +use object_store::{path::Path, ObjectStore}; +use std::{ + sync::{atomic::Ordering, Arc}, + time::Duration, +}; + +struct Fixture { + domain: ExecutionDomain, + store: Arc, + directory: RangeDirectory, + config: NamespaceConfig, + limits: DirectoryLimits, +} + +impl Fixture { + async fn new(limits: DirectoryLimits) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let store = Arc::new(fault::Store::default()); + store.reject_listing.store(true, Ordering::Release); + let route = route(); + let config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain.run(catalog.open_namespace(config.clone())).await??; + let directory = RangeDirectory::new(namespace, limits)?; + Ok(Self { + domain, + store, + directory, + config, + limits, + }) + } + + async fn run(&self, work: F) -> F::Output { + self.domain + .run(work) + .await + .expect("fixture execution domain closed before joined shutdown") + } + + async fn reopen(&self) -> Result { + let catalog = Catalog::new(self.store.clone(), CatalogOptions::default())?; + Ok(RangeDirectory::new( + self.run(catalog.existing_namespace(self.config.clone())) + .await?, + self.limits, + )?) + } + + fn key(&self, name: &str) -> Path { + Path::from(format!( + "namespaces/{}/cell-groups/{name}", + hex::encode(self.config.namespace.bytes()) + )) + } +} + +fn operation(n: u64) -> ProposalId { + let mut bytes = [0; 16]; + bytes[..8].copy_from_slice(&n.to_be_bytes()); + ProposalId::new(bytes).unwrap() +} + +fn assigned(id: u64, range: CellRange, epoch: u64) -> GroupRoute { + let mut r = route(); + r.history.group = GroupId::new(operation(id).bytes()).unwrap(); + r.history.incarnation = IncarnationId::new(operation(id + 10000).bytes()).unwrap(); + r.birth_digest[..8].copy_from_slice(&id.to_be_bytes()); + r.range = range; + r.epoch = RouteEpoch::new(epoch).unwrap(); + r +} + +fn ranges(count: usize) -> Vec { + let ns = route().history.namespace; + let mut keys: Vec<_> = (1..count) + .map(|i| cell_routing_key(ns, &CellId::new(format!("boundary-{i}")).unwrap())) + .collect(); + keys.sort(); + let mut lower = CellBound::NegativeInfinity; + let mut result = vec![]; + for (i, upper) in keys + .into_iter() + .map(CellBound::Key) + .chain([CellBound::PositiveInfinity]) + .enumerate() + { + result.push(assigned( + i as u64 + 1, + CellRange::new(lower, upper.clone()).unwrap(), + 1, + )); + lower = upper; + } + result +} + +fn split(parent: &GroupRoute, id: u64) -> Vec { + let key = (0..100_000) + .map(|i| { + cell_routing_key( + parent.history.namespace, + &CellId::new(format!("split-{id}-{i}")).unwrap(), + ) + }) + .find(|key| { + parent.range.contains(key) && CellBound::Key(key.clone()) != *parent.range.lower() + }) + .unwrap(); + let (left, right) = parent.range.split(key).unwrap(); + vec![ + assigned(id, left, parent.epoch.get() + 1), + assigned(id + 1, right, parent.epoch.get() + 1), + ] +} + +pub async fn single_owner() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let initial = ranges(2); + let init = d.prepare_initialization(operation(1), &initial)?; + for bad in [ + vec![], + vec![initial[0].clone()], + vec![initial[1].clone(), initial[0].clone()], + vec![initial[0].clone(), initial[0].clone()], + ] { + assert!(d.prepare_initialization(operation(2), &bad).is_err()); + } + let mut duplicate = initial.clone(); + duplicate[1].history.group = duplicate[0].history.group; + assert!(d.prepare_initialization(operation(2), &duplicate).is_err()); + let mut wrong_epoch = initial.clone(); + wrong_epoch[0].epoch = RouteEpoch::new(2)?; + assert!(d + .prepare_initialization(operation(2), &wrong_epoch) + .is_err()); + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + let left = split(&initial[0], 100); + let right = split(&initial[1], 200); + let a = d.prepare_replacement(&view, operation(2), &initial[..1], &left)?; + let b = d.prepare_replacement(&view, operation(3), &initial[1..], &right)?; + let mut stale_parent = initial[0].clone(); + stale_parent.birth_digest = [22; 32]; + let stale = d.prepare_replacement(&view, operation(4), &[stale_parent], &left)?; + assert!(f.run(d.publish(&stale)).await.is_err()); + f.store.range_before.armed.store(true, Ordering::Release); + let entered = f.store.range_before.entered.notified(); + let (loser, winner) = f + .run(cellar::asyncrt::timeout(Duration::from_secs(15), async { + tokio::join!(d.publish(&a), async { + entered.await; + let result = d.publish(&b).await; + f.store.range_before.resume.notify_one(); + result + }) + })) + .await?; + assert!(matches!(loser, Err(CatalogError::Conflict(_)))); + assert_eq!(winner?.revision(), 2); + let after = f.run(d.observe()).await?; + assert_eq!(after.groups(), 3); + assert_eq!(view.groups(), 2); + assert!(matches!( + f.run(d.publish(&a)).await, + Err(CatalogError::Conflict(_)) + )); + // Previously assigned group IDs stay reserved even after range replacement. + let mut reused = assigned(201, initial[0].range.clone(), 2); + reused.history.incarnation = IncarnationId::new([42; 16])?; + let reused = d.prepare_replacement(&after, operation(5), &initial[..1], &[reused])?; + assert!(matches!( + f.run(d.publish(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + for i in 0..100 { + let cell = CellId::new(format!("check-{i}"))?; + let key = cell_routing_key(f.config.namespace, &cell); + let old = initial.iter().find(|r| r.range.contains(&key)).unwrap(); + let expected = if old == &initial[0] { + old + } else { + right.iter().find(|r| r.range.contains(&key)).unwrap() + }; + assert_eq!(&f.run(d.lookup(&after, &cell)).await?, expected); + assert_eq!(&f.run(d.lookup(&view, &cell)).await?, old); + } + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn bounded_discovery() -> Result<()> { + let limits = DirectoryLimits { + max_cached_pages: 4, + max_cache_bytes: 4 * 64 * 1024, + operation_timeout_ms: 30_000, + ..Default::default() + }; + let f = Fixture::new(limits).await?; + let d = &f.directory; + let initial = ranges(128); + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + for i in 0..300 { + let cell = CellId::new(format!("lookup-{i}"))?; + let (found, reads) = fault::count_reads(f.run(d.lookup(&view, &cell))).await; + assert!( + reads.values().sum::() <= 8, + "lookup must read one AVL path: {reads:?}" + ); + assert!(reads.keys().all(|k| k.contains("/cell-groups/ranges/"))); + let key = cell_routing_key(f.config.namespace, &cell); + assert_eq!( + &found?, + initial.iter().find(|r| r.range.contains(&key)).unwrap() + ); + let (pages, bytes) = d.cache_usage()?; + assert!(pages <= 4 && bytes <= limits.max_cache_bytes); + } + let parent = &initial[63]; + let children = split(parent, 1000); + let mutation = + d.prepare_replacement(&view, operation(2), std::slice::from_ref(parent), &children)?; + let before = f.store.range_page_puts.load(Ordering::Acquire); + f.run(d.publish(&mutation)).await?; + let pages = f.store.range_page_puts.load(Ordering::Acquire) - before; + assert!( + pages <= 64, + "one split rewrote {pages} pages for 128 groups" + ); + let after = f.run(d.observe()).await?; + assert_eq!(after.groups(), 129); + let merged = assigned(2000, parent.range.clone(), 3); + let merge = d.prepare_replacement( + &after, + operation(3), + &children, + std::slice::from_ref(&merged), + )?; + f.run(d.publish(&merge)).await?; + let final_view = f.run(d.observe()).await?; + assert_eq!(final_view.groups(), 128); + let catalog = Catalog::new(f.store.clone(), CatalogOptions::default())?; + let namespace = f.run(catalog.existing_namespace(f.config.clone())).await?; + let warm = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let warm_view = f.run(warm.observe()).await?; + let cell = CellId::new("warm-cell")?; + f.run(warm.lookup(&warm_view, &cell)).await?; + let (result, reads) = fault::count_reads(f.run(warm.lookup(&warm_view, &cell))).await; + result?; + assert!( + reads.is_empty(), + "cached route lookup must not refresh S3 metadata" + ); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn recovery() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let initial = ranges(1); + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + let children = split(&initial[0], 100); + let first = d.prepare_replacement(&view, operation(2), &initial, &children)?; + f.store.fail_head_after.store(true, Ordering::Release); + f.store + .fail_head_confirmation + .store(true, Ordering::Release); + assert!(matches!( + f.run(d.publish(&first)).await, + Err(CatalogError::Unconfirmed) + )); + let restarted = f.reopen().await?; + let retry = restarted.decode_mutation(first.encode())?; + let selected = f.run(restarted.observe()).await?; + assert_eq!(selected.revision(), 2); + let second_children = split(&children[0], 200); + let second = + restarted.prepare_replacement(&selected, operation(3), &children[..1], &second_children)?; + let latest_publication = f.run(restarted.publish(&second)).await?; + let journal_key = f.key(&format!( + "journal/{}", + hex::encode(latest_publication.journal_digest()) + )); + assert!( + fault::missing_read(journal_key.to_string(), f.run(restarted.observe())) + .await + .is_err(), + "a head without its selected journal cannot be observed as valid" + ); + let receipt = f.run(restarted.publish(&retry)).await?; + assert_eq!( + receipt.revision(), + 2, + "retry finds its own publication after a later CAS" + ); + assert_eq!(f.run(restarted.publication(&retry)).await?, Some(receipt)); + let latest = f.run(restarted.observe()).await?; + let reused = restarted.prepare_replacement( + &latest, + operation(2), + &children[1..], + &split(&children[1], 300), + )?; + assert!(matches!( + f.run(restarted.publish(&reused)).await, + Err(CatalogError::Conflict(_)) + )); + assert!( + restarted.lookup(&view, &CellId::new("x")?).await.is_err(), + "old process view cannot attach to a new owner" + ); + // Cold discovery must verify the selected immutable page, not just its name. + let cold = f.reopen().await?; + let cold_view = f.run(cold.observe()).await?; + let cell = CellId::new("corruption-probe")?; + let (result, reads) = fault::count_reads(f.run(cold.lookup(&cold_view, &cell))).await; + result?; + let path = reads + .keys() + .find(|k| k.contains("/ranges/")) + .context("range read")? + .clone(); + let original = f + .store + .inner + .get(&Path::from(path.clone())) + .await? + .bytes() + .await?; + let mut corrupt = original.to_vec(); + *corrupt.last_mut().unwrap() ^= 1; + f.store + .inner + .put(&Path::from(path.clone()), corrupt.into()) + .await?; + let fresh = f.reopen().await?; + let fresh_view = f.run(fresh.observe()).await?; + assert!(f.run(fresh.lookup(&fresh_view, &cell)).await.is_err()); + f.store + .inner + .put(&Path::from(path.clone()), original.into()) + .await?; + assert!( + fault::missing_read(path, f.run(fresh.lookup(&fresh_view, &cell))) + .await + .is_err() + ); + f.store.inner.delete(&f.key("head")).await?; + assert!(f.run(restarted.observe()).await.is_err()); + assert!( + f.run(restarted.publish(&init)).await.is_err(), + "missing occupied metadata cannot reinitialize" + ); + assert!(matches!( + f.store.inner.get(&f.key("head")).await, + Err(object_store::Error::NotFound { .. }) + )); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn genesis_race() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let initial = ranges(1); + let init = d.prepare_initialization(operation(1), &initial)?; + f.store + .range_genesis_after + .armed + .store(true, Ordering::Release); + let entered = f.store.range_genesis_after.entered.notified(); + let (first, second) = f + .run(cellar::asyncrt::timeout(Duration::from_secs(15), async { + tokio::join!(d.publish(&init), async { + entered.await; + let result = async { + let receipt = d.publish(&init).await?; + let view = d.observe().await?; + let replacement = d.prepare_replacement( + &view, + operation(2), + &initial, + &split(&initial[0], 100), + )?; + d.publish(&replacement).await?; + Ok::<_, CatalogError>(receipt) + } + .await; + f.store.range_genesis_after.resume.notify_one(); + result + }) + })) + .await?; + assert_eq!(first?, second?); + assert_eq!(f.run(d.observe()).await?.revision(), 2); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn churn() -> Result<()> { + let f = Fixture::new(DirectoryLimits { + max_cached_pages: 3, + max_cache_bytes: 3 * 64 * 1024, + operation_timeout_ms: 30_000, + ..Default::default() + }) + .await?; + let d = &f.directory; + let initial = ranges(16); + let init = d.prepare_initialization(operation(1), &initial)?; + f.run(d.publish(&init)).await?; + let original_view = f.run(d.observe()).await?; + // The oracle is an independently maintained flat partition. Alternate + // splitting and merging different interior ranges so deletions exercise + // successor replacement and rotations, rather than undoing the last split. + let mut expected = initial.clone(); + for step in 0..80_usize { + let view = f.run(d.observe()).await?; + let position = (step * 7 + 3) % (expected.len() - 1); + let id = 1000 + step as u64 * 10; + let (parents, children) = if step % 2 == 0 { + let parent = expected[position].clone(); + let children = split(&parent, id); + (vec![parent], children) + } else { + let parents = expected[position..position + 2].to_vec(); + let range = parents[0].range.merge(&parents[1].range)?; + let epoch = parents.iter().map(|r| r.epoch.get()).max().unwrap() + 1; + (parents, vec![assigned(id, range, epoch)]) + }; + let mutation = + d.prepare_replacement(&view, operation(step as u64 + 2), &parents, &children)?; + let receipt = f.run(d.publish(&mutation)).await?; + expected.splice(position..position + parents.len(), children); + let after = f.run(d.observe()).await?; + assert_eq!(after.revision(), step as u64 + 2); + assert_eq!(after.groups() as usize, expected.len()); + assert_eq!(f.run(d.publication(&mutation)).await?, Some(receipt)); + // Probe every boundary itself (half-open ownership) plus independent + // cell keys that are not selected from the tree under test. + let probes = expected + .iter() + .filter_map(|r| match r.range.lower() { + CellBound::Key(key) => Some(key.cell().clone()), + _ => None, + }) + .chain((0..32).map(|i| CellId::new(format!("churn-{i}")).unwrap())); + for cell in probes { + let key = cell_routing_key(f.config.namespace, &cell); + let owners: Vec<_> = expected.iter().filter(|r| r.range.contains(&key)).collect(); + assert_eq!(owners.len(), 1); + assert_eq!(&f.run(d.lookup(&after, &cell)).await?, owners[0]); + let old = initial.iter().find(|r| r.range.contains(&key)).unwrap(); + assert_eq!(&f.run(d.lookup(&original_view, &cell)).await?, old); + } + assert!(d.cache_usage()?.0 <= 3); + } + let restarted = f.reopen().await?; + assert_eq!(f.run(restarted.observe()).await?.revision(), 81); + assert_eq!(f.run(restarted.publish(&init)).await?.revision(), 1); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_catalog_fault.rs b/crates/cellar/tests/support/raft_catalog_fault.rs index 2db8a6c74..dfe94dc83 100644 --- a/crates/cellar/tests/support/raft_catalog_fault.rs +++ b/crates/cellar/tests/support/raft_catalog_fault.rs @@ -59,6 +59,9 @@ impl RegistryGate { #[derive(Debug, Default)] pub struct Store { + pub range_before: RegistryGate, + pub range_genesis_after: RegistryGate, + pub range_page_puts: AtomicUsize, pub fail_closure_after: AtomicBool, pub fail_fleet_after: AtomicBool, pub fail_fleet_confirmation: AtomicBool, @@ -244,6 +247,14 @@ impl ObjectStore for Store { payload: PutPayload, options: PutOptions, ) -> object_store::Result { + if path.as_ref().contains("/cell-groups/ranges/") { + self.range_page_puts.fetch_add(1, Ordering::AcqRel); + } + if path.as_ref().ends_with("/cell-groups/head") + && matches!(options.mode, PutMode::Update(_)) + { + self.range_before.pause().await; + } if path.as_ref().contains("/root-extents/") { self.root_extent_puts.fetch_add(1, Ordering::AcqRel); } @@ -374,7 +385,12 @@ impl ObjectStore for Store { } let fleet_cas = path.as_ref().ends_with("/fleet/head") && matches!(options.mode, PutMode::Update(_)); + let range_genesis = + path.as_ref().ends_with("/cell-groups/head") && matches!(options.mode, PutMode::Create); let result = self.inner.put_opts(path, payload, options).await?; + if range_genesis { + self.range_genesis_after.pause().await; + } if path.as_ref().contains("/closures/") && self.fail_closure_after.swap(false, Ordering::AcqRel) { diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index d08f9b922..d1e4fc04b 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -281,12 +281,17 @@ "scope": [ "crates/cellar/durability/raft/", "crates/cellar/application/consensus/", - "crates/cellar/node/" + "crates/cellar/node/", + "crates/cellar/tests/support/raft_catalog_fault.rs" ], "creates": [ - "crates/cellar/durability/raft/cell_groups/directory.rs", "crates/cellar/durability/raft/cell_groups/enrollment.rs", - "crates/cellar/tests/native_cell_groups_catalog.rs" + "crates/cellar/tests/native_cell_groups_catalog.rs", + "crates/cellar/durability/raft/cell_groups/directory/mod.rs", + "crates/cellar/durability/raft/cell_groups/directory/wire.rs", + "crates/cellar/durability/raft/cell_groups/directory/tree.rs", + "crates/cellar/durability/raft/cell_groups/directory/store.rs", + "crates/cellar/tests/support/cell_group_directory.rs" ], "test_target": "native_cell_groups_catalog", "steps": [ From 268445ef59afd23dccdf8e2850464fb59f7c5d92 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:12:41 -0700 Subject: [PATCH 020/568] docs: record cell-group directory qualification checkpoint --- plans/multi-raft/cell-groups/README.md | 12 ++--- plans/multi-raft/cell-groups/baseline.json | 12 ++--- plans/multi-raft/cell-groups/progress.md | 52 ++++++++++++++++++---- 3 files changed, 56 insertions(+), 20 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 9b1da228b..679540f9b 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The requested deliverable is the executable design and implementation plan. -Implementation checkpoints are recorded separately; delivering this package does -not mark the remaining tasks complete or qualify the shared-group runtime. +The design package is delivered. The active goal is now its end-to-end +implementation and hard cutover to the shared-group runtime. Checkpoints are +recorded separately; they do not mark the remaining tasks complete. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,9 +68,9 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The user's clarified scope for this handoff is the plan. Future implementation -follows the task briefs below; completion of this handoff does not instruct the -current executor to continue through the runtime cutover. +The resumed implementation goal includes every task and the final runtime cutover. +The earlier plan-only handoff is historical; package validation does not satisfy +the active implementation objective. Resume CG-03 after reading [progress](progress.md) and inspecting the working diff. CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 7247d1a67..595d43a69 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ec9d277e50416f786fe3c1cd851b4147f9cd4b43", + "source_revision": "a0c11969763edbe86ee08fc7d1650129058c8ee0", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed and qualified CG-02 at ec9d277e50416f786fe3c1cd851b4147f9cd4b43. All five required cases and complete command gate pass. Both physical image modes recover complete selected roots through 30 process-kill cuts; missing/corrupt image, extent and manifest dependencies refuse. Reclamation retains selected and uncertain closures. Existing anchored source hashes/excerpts are unchanged. Production shared-host authoritative inventory, quorum, snapshots/archive, topology, hibernation and real RustFS recovery remain CG-03 through CG-16 work.", + "review_note": "2026-10-29: reviewed CG-03 range-directory checkpoint at a0c11969763edbe86ee08fc7d1650129058c8ee0. The catalog.rs diff adds only the six-line range-directory module/export block; namespace and birth excerpts are unchanged and move to lines 110 and 179. Existing birth selection, bootstrap claims and managed admission are unchanged. New directory code verifies bounded immutable AVL ranges, one conditional head with journal/index bindings, historical retry reconciliation and permanent genesis pinning. Eight cell-group catalog cases and thirteen existing fleet catalog cases pass, including 80 split/merge metadata transitions. This is a partial CG-03 checkpoint: no group birth/disk-claim integration, client redirect path, production shared host or live topology authority is qualified. Full hard-cutover implementation remains active.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "65053cb9c70931f8234041d99e43ccd748c8f64b7479ed299888a57030850e1a", - "line": 104, + "sha256": "3250b6a2a4d611850dd59c51c3aa4dfaa173ec956bbe4e5d6014e5afbf03b842", + "line": 110, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "65053cb9c70931f8234041d99e43ccd748c8f64b7479ed299888a57030850e1a", - "line": 173, + "sha256": "3250b6a2a4d611850dd59c51c3aa4dfaa173ec956bbe4e5d6014e5afbf03b842", + "line": 179, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 206912a4f..f6e565251 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,12 +3,14 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The user's clarified deliverable is the executable plan; the -implementation checkpoints below describe existing work for a future executor. -Intermediate contracts do not qualify a shared runtime. No production -legacy/shared mode flag is being added. +hard cutover. The active goal resumes end-to-end implementation after the plan +handoff recorded below. Intermediate contracts do not qualify a shared runtime. +No production legacy/shared mode flag is being added. -## Plan handoff — 2026-10-29 +## Historical plan handoff — 2026-10-29 + +This section records the state at `dcc52840`; the directory checkpoint below +supersedes its working-diff and source-preflight limitations. The package validator passes: 16 tasks, 83 required cases, 14 invariants and nine profiles. Task briefs include source excerpts, file scope, dependencies, exact @@ -76,9 +78,10 @@ CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `ec9d277e50416f786fe3c1cd851b4147f9cd4b43`. -The original anchored source hashes and excerpts remain unchanged. A new executor -must receive both ignored DONE-task evidence folders before validation. +The reviewed source baseline is `a0c11969763edbe86ee08fc7d1650129058c8ee0`. +The catalog anchor adds the directory module/export block; its unchanged namespace +and birth excerpts move by six lines. Other source anchors remain unchanged. A new +executor must receive both ignored DONE-task evidence folders before validation. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until @@ -295,3 +298,36 @@ inputs, nine successful checks and a copied native executable. The failed initia build/test outputs and a dependency-check environment failure remain retained; the latter passed after restoring the Cargo path. This is a partial checkpoint, not a CG-03 DONE record. + +### CG-03 range-directory checkpoint + +Source `a0c11969763edbe86ee08fc7d1650129058c8ee0` adds bounded immutable AVL range +pages, path-copy replacement, a conditional head selecting the exact mutation +journal and operation/group indexes, and permanent genesis pinning. Historical +retry confirmation survives later root updates. Duplicate IDs, stale parents, +changed coverage, wrong epochs, reused operations and missing/corrupt selected +dependencies refuse. Retired group IDs remain reserved. A cached discovery path +does not refresh S3 metadata or list the namespace. + +All eight native cell-group catalog tests pass. These include concurrent CAS, +lost response/restart, initialization racing a later mutation, 128-range bounded +lookup/path-copy checks, and 80 alternating split/merge metadata transitions. An +independent flat partition checks every resulting boundary plus unrelated cell +keys; original views remain readable throughout churn. The 13 existing native +fleet-catalog regressions also pass with the shared fault adapter. The adapter's +new barriers are inactive unless explicitly armed for directory operations. + +The directory has no serving authority. Birth/disk claims, managed birth +authorization, deadline/hop-bounded client redirects and the real shared host +remain outstanding. No live split/merge, Raft quorum or RustFS qualification is +claimed. CG-03 remains IN_PROGRESS and CG-04 through CG-16 remain TODO. The full +hard-cutover goal is active; the earlier plan-only handoff does not satisfy it. + +The partial record is retained under +`.cellar/cell-groups-evidence/CG-03-directory/checkpoint.json`. It includes source +and executable hashes, command output and the corrected initial fixture compile +failure. This checkpoint is separate from the still-incomplete CG-03 task gate. +The record binds 63 source inputs, nine successful command logs and two copied +native executables. All-target Clippy with warnings denied, formatting/layout, +native dependency isolation, docs, source preflight and all 37 plan-validator +tests pass at this checkpoint. From 9db4ba8fc91e6d67de244c1a684001393bcc9e56 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:31:40 -0700 Subject: [PATCH 021/568] feat: bind lazy group births to reserved physical histories --- crates/cellar/durability/raft/catalog.rs | 3 +- .../durability/raft/cell_groups/README.md | 34 +- .../raft/cell_groups/directory/birth.rs | 304 ++++++++++++++++++ .../raft/cell_groups/directory/mod.rs | 8 + .../raft/cell_groups/directory/wire.rs | 16 + .../cellar/durability/raft/cell_groups/mod.rs | 3 +- .../tests/native_cell_groups_catalog.rs | 11 + .../tests/support/cell_group_directory.rs | 291 +++++++++++++++-- plans/multi-raft/cell-groups/tasks.json | 3 +- 9 files changed, 650 insertions(+), 23 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/birth.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 117f05027..ec3074cf3 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -25,7 +25,8 @@ pub use births::BirthDisposition; #[path = "cell_groups/directory/mod.rs"] mod range_directory; pub use range_directory::{ - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, RangeDirectory, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 392fcfdd2..cf24974ea 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -143,7 +143,8 @@ Views are hints; the destination must check its own current route and quorum authority. Observation does not traverse a complete recovery inventory. Initialization publishes at most 128 contiguous ranges covering the whole routing -domain. A replacement removes up to 64 exact parent routes and inserts up to 64 +domain. Every new route must match its permanently reserved group birth before +publication. A replacement removes up to 64 exact parent routes and inserts up to 64 fresh groups covering the identical contiguous union. Children advance the maximum parent epoch by one. Paths are copied incrementally; incomplete intermediate trees cannot become a selected root. Group IDs remain reserved after retirement. @@ -176,3 +177,34 @@ redirects remain to be integrated. Native directory tests exercise real conditio object-store operations in memory, including races, dropped replies, corruption, restart and bounded split/merge churn. They do not qualify live Raft ownership, real RustFS or the full CG-03 gate. + +## Cell-independent group birth (CG-03 in progress) + +`CellGroupBirthSpec` binds physical group history, range, route epoch, original +creation operation and initial members. It has no cell name. The namespace binds +schema, storage representation and durability policy. Preparation sorts members +canonically and enforces the namespace voter policy and distinct member/node IDs. +The bounded marker-1 birth encoding has its own semantic record kind; its digest +is the route's birth binding. Persist the returned request bytes before submission. + +Use `prepare_group_birth`, `reserve_group_birth`, directory publication, then +`materialize_group_birth`. Reservation conditionally creates the existing global +group-ID key with the complete canonical birth. Another namespace, incarnation, +placement or creation operation cannot reuse that ID. Existing records from the +old catalog also prevent reuse; they are not converted or reinterpreted. Losing +a directory CAS retains the reservation permanently. Directory publication checks +every child's exact reservation, but does not eagerly create its birth object. + +Materialization verifies the currently selected route and its global reservation, +conditionally writes the exact birth, then checks selection again. Missing +confirmation remains unconfirmed; reopen and decode the original request before +retrying. A retired or replaced range cannot materialize again. A birth observation +is bound to its catalog owner; `require_current_group_birth` checks current route, +reservation and selected immutable bytes, and refuses missing or changed state. + +These methods expose no voter-start capability and write no permanent member disk +claim. The physical log still needs cell-independent genesis and recovery identity +before the new birth can be integrated with actual disk-claim admission. Managed +namespaces refuse these unmanaged APIs until their rooted fleet authorization is +connected. This staged boundary does not permit blank-disk recovery, replace +current quorum membership, or complete CG-03 qualification. diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs new file mode 100644 index 000000000..da3f74f6a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -0,0 +1,304 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-independent immutable group births. A selected birth is metadata; only +//! durable genesis plus permanent member claims can authorize voter startup. + +use super::super::InitialMember; +use super::*; + +pub struct CellGroupBirthSpec { + pub history: GroupHistory, + pub range: CellRange, + pub epoch: RouteEpoch, + pub creation: ProposalId, + pub voters: Vec, + pub learners: Vec, +} + +/// Persist the canonical request before reserving its global group ID. Retrying +/// must keep these bytes, including placement, creation identity and incarnation. +#[derive(Clone)] +pub struct CellGroupBirthRequest { + bytes: Vec, + route: GroupRoute, + creation: ProposalId, + voters: Vec, + learners: Vec, +} + +impl CellGroupBirthRequest { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn creation(&self) -> ProposalId { + self.creation + } + + pub fn voters(&self) -> &[InitialMember] { + &self.voters + } + + pub fn learners(&self) -> &[InitialMember] { + &self.learners + } +} + +/// Read-back of an immutable birth selected by an observed directory root. +/// This grants neither membership nor permission to initialize a blank disk. +pub struct CellGroupBirth { + request: CellGroupBirthRequest, + owner: Arc<()>, +} + +impl CellGroupBirth { + pub fn route(&self) -> &GroupRoute { + self.request.route() + } + + pub fn creation(&self) -> ProposalId { + self.request.creation() + } + + pub fn voters(&self) -> &[InitialMember] { + self.request.voters() + } + + pub fn learners(&self) -> &[InitialMember] { + self.request.learners() + } +} + +impl RangeDirectory { + fn unmanaged_birth(&self) -> Result<(), CatalogError> { + if self.namespace.config.managed_fleet.is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } + + pub fn prepare_group_birth( + &self, + mut spec: CellGroupBirthSpec, + ) -> Result { + self.unmanaged_birth()?; + if spec.history.cluster != self.namespace.config.cluster + || spec.history.namespace != self.namespace.config.namespace + { + return Err(CatalogError::Conflict("group birth namespace")); + } + if spec.voters.len() > 32 || spec.learners.len() > 32 { + return Err(CatalogError::Limit); + } + spec.voters.sort_by_key(|m| m.member); + spec.learners.sort_by_key(|m| m.member); + self.namespace.check_members(&spec.voters, &spec.learners)?; + let value = wire::Birth { + namespace: self.namespace.root_digest, + group: spec.history.group.bytes(), + incarnation: spec.history.incarnation.bytes(), + lower: wire::Bound::from_bound(spec.range.lower()), + upper: wire::Bound::from_bound(spec.range.upper()), + epoch: spec.epoch.get(), + creation: spec.creation.bytes(), + voters: spec + .voters + .into_iter() + .map(record::Member::encode) + .collect(), + learners: spec + .learners + .into_iter() + .map(record::Member::encode) + .collect(), + }; + let request = + self.decode_group_birth(&record::encode(wire::GROUP_BIRTH, &value, wire::LIMIT)?)?; + if request.route.range != spec.range { + return Err(CatalogError::Invalid("noncanonical group birth range")); + } + Ok(request) + } + + pub fn decode_group_birth(&self, bytes: &[u8]) -> Result { + self.unmanaged_birth()?; + let value: wire::Birth = record::decode(wire::GROUP_BIRTH, bytes, wire::LIMIT)?; + if value.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("group birth namespace")); + } + let voters = value + .voters + .iter() + .map(record::Member::decode) + .collect::, _>>()?; + let learners = value + .learners + .iter() + .map(record::Member::decode) + .collect::, _>>()?; + self.namespace.check_members(&voters, &learners)?; + let route = wire::Route { + group: value.group, + incarnation: value.incarnation, + lower: value.lower, + upper: value.upper, + epoch: value.epoch, + birth: record::digest(bytes), + } + .decode(self)?; + let creation = ProposalId::new(value.creation) + .map_err(|_| CatalogError::Corrupt("group birth creation"))?; + Ok(CellGroupBirthRequest { + bytes: bytes.to_vec(), + route, + creation, + voters, + learners, + }) + } + + fn reservation_key(&self, route: &GroupRoute) -> Path { + // Same permanent global key as the prior catalog: an occupied identity + // cannot be reused across namespaces, incarnations or semantic domains. + Path::from(format!( + "{}groups/{}", + self.namespace.inner.prefix, + hex::encode(route.history.group.bytes()) + )) + } + + fn group_birth_key(&self, route: &GroupRoute) -> Path { + self.key(&format!( + "births/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + /// Permanently reserve before directory publication. An orphan reservation + /// is retained; a failed topology CAS does not release its identity. + pub async fn reserve_group_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + self.decode_group_birth(request.encode())?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.reservation_key(request.route()), + request.encode(), + wire::LIMIT, + ) + .await?; + if observed != request.encode() { + return Err(CatalogError::Conflict("group birth identity reserved")); + } + Ok(()) + }) + .await + } + + pub(super) async fn verify_birth_reservation( + &self, + route: &GroupRoute, + ) -> Result { + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.reservation_key(route), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing group birth reservation"))? + .0; + let request = self.decode_group_birth(&bytes)?; + if request.route() != route { + return Err(CatalogError::Conflict("group birth reservation binding")); + } + Ok(request) + } + + async fn selected_birth_route(&self, route: &GroupRoute) -> Result<(), CatalogError> { + let (view, _) = self.current().await?; + let mut next = Some(view.head.tree); + for _ in 0..32 { + let reference = next.take().ok_or(CatalogError::Fenced)?; + let page = self.page(&reference).await?; + let found = page.route.decode(self)?; + match route.range.lower().cmp(found.range.lower()) { + std::cmp::Ordering::Less => next = page.left, + std::cmp::Ordering::Greater => next = page.right, + std::cmp::Ordering::Equal => { + return if found == *route { + Ok(()) + } else { + Err(CatalogError::Fenced) + } + } + } + } + Err(CatalogError::Limit) + } + + /// Materialize the exact reserved birth only after directory selection. + /// This is lazy metadata creation, not local genesis or a bootstrap permit. + pub async fn materialize_group_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.decode_group_birth(request.encode())?; + self.bounded(async { + self.selected_birth_route(request.route()).await?; + self.verify_birth_reservation(request.route()).await?; + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + request.encode(), + wire::LIMIT, + ) + .await?; + if observed != request.encode() { + return Err(CatalogError::Conflict("selected group birth")); + } + self.selected_birth_route(request.route()).await?; + Ok(CellGroupBirth { + request: request.clone(), + owner: self.namespace.inner.owner.clone(), + }) + }) + .await + } + + /// Recheck a birth observation's current directory and immutable metadata. + /// Future voter admission must additionally verify disk claims and quorum. + pub async fn require_current_group_birth( + &self, + birth: &CellGroupBirth, + ) -> Result<(), CatalogError> { + if !Arc::ptr_eq(&birth.owner, &self.namespace.inner.owner) { + return Err(CatalogError::Conflict("foreign group birth observation")); + } + self.bounded(async { + self.selected_birth_route(birth.route()).await?; + self.verify_birth_reservation(birth.route()).await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(birth.route()), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing selected group birth"))? + .0; + if bytes != birth.request.encode() { + return Err(CatalogError::Corrupt("selected group birth changed")); + } + Ok(()) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index 65a49632c..d62e55f8b 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -17,10 +17,13 @@ use std::{ time::Duration, }; +mod birth; mod store; mod tree; mod wire; +pub use birth::{CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec}; + #[derive(Clone, Copy, Debug)] pub struct DirectoryLimits { pub max_groups: u32, @@ -479,6 +482,11 @@ impl RangeDirectory { return Err(CatalogError::Limit); } let mut tree = base.map(|h| h.tree.clone()); + // A route can be lazy, but its exact cell-independent birth and global + // group identity must be reserved before it can enter the directory. + for child in &request.children { + self.verify_birth_reservation(&child.decode(self)?).await?; + } for parent in &request.parents { tree = self.remove(tree, parent.clone(), 0).await?; } diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 3383932b9..73465f69d 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -8,6 +8,8 @@ pub(super) const HEAD: u8 = 82; pub(super) const JOURNAL: u8 = 83; pub(super) const PIN: u8 = 84; pub(super) const REQUEST: u8 = 85; +// Fleet maintenance already uses 86 in the shared catalog envelope. +pub(super) const GROUP_BIRTH: u8 = 186; pub(super) const LIMIT: usize = 64 * 1024; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -229,3 +231,17 @@ pub(super) struct Journal { pub previous: Option<[u8; 32]>, pub tree: Reference, } + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Birth { + pub namespace: [u8; 32], + pub group: [u8; 16], + pub incarnation: [u8; 16], + pub lower: Bound, + pub upper: Bound, + pub epoch: u64, + pub creation: [u8; 16], + pub voters: Vec, + pub learners: Vec, +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 2098cf876..3cee6218d 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -18,7 +18,8 @@ pub use manifest::{ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; pub use crate::raft::catalog::{ - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, RangeDirectory, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index 82f8602ef..d8ffa482c 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -40,6 +40,17 @@ async fn directory_churn_preserves_exact_partition_and_old_views() -> anyhow::Re directory::churn().await } +#[tokio::test] +async fn group_birth_is_lazy_and_reconciles_unknown_by_exact_identity() -> anyhow::Result<()> { + directory::birth_recovery().await +} + +#[tokio::test] +async fn group_birth_reservations_bind_global_identity_and_canonical_placement( +) -> anyhow::Result<()> { + directory::birth_reservations().await +} + fn route() -> GroupRoute { GroupRoute { history: GroupHistory { diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 793510546..d2d9b592f 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -6,9 +6,10 @@ use super::*; use anyhow::{Context, Result}; use cellar::application::consensus::verification::{ - Catalog, CatalogError, CatalogOptions, NamespaceConfig, + Catalog, CatalogError, CatalogOptions, InitialMember, NamespaceConfig, }; use cellar::sdk::ExecutionDomain; +use cellar_logic::consensus::{MemberId, NodeId}; use object_store::{path::Path, ObjectStore}; use std::{ sync::{atomic::Ordering, Arc}, @@ -63,6 +64,16 @@ impl Fixture { )?) } + async fn reserve(&self, routes: &mut [GroupRoute]) -> Result<(), CatalogError> { + for route in routes { + let request = self.directory.prepare_group_birth(birth_spec(route))?; + self.run(self.directory.reserve_group_birth(&request)) + .await?; + *route = request.route().clone(); + } + Ok(()) + } + fn key(&self, name: &str) -> Path { Path::from(format!( "namespaces/{}/cell-groups/{name}", @@ -71,6 +82,22 @@ impl Fixture { } } +fn birth_spec(route: &GroupRoute) -> CellGroupBirthSpec { + CellGroupBirthSpec { + history: route.history, + range: route.range.clone(), + epoch: route.epoch, + creation: ProposalId::new(route.history.group.bytes()).unwrap(), + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + } +} + fn operation(n: u64) -> ProposalId { let mut bytes = [0; 16]; bytes[..8].copy_from_slice(&n.to_be_bytes()); @@ -133,7 +160,8 @@ fn split(parent: &GroupRoute, id: u64) -> Vec { pub async fn single_owner() -> Result<()> { let f = Fixture::new(DirectoryLimits::default()).await?; let d = &f.directory; - let initial = ranges(2); + let mut initial = ranges(2); + f.reserve(&mut initial).await?; let init = d.prepare_initialization(operation(1), &initial)?; for bad in [ vec![], @@ -153,8 +181,10 @@ pub async fn single_owner() -> Result<()> { .is_err()); f.run(d.publish(&init)).await?; let view = f.run(d.observe()).await?; - let left = split(&initial[0], 100); - let right = split(&initial[1], 200); + let mut left = split(&initial[0], 100); + let mut right = split(&initial[1], 200); + f.reserve(&mut left).await?; + f.reserve(&mut right).await?; let a = d.prepare_replacement(&view, operation(2), &initial[..1], &left)?; let b = d.prepare_replacement(&view, operation(3), &initial[1..], &right)?; let mut stale_parent = initial[0].clone(); @@ -216,7 +246,8 @@ pub async fn bounded_discovery() -> Result<()> { }; let f = Fixture::new(limits).await?; let d = &f.directory; - let initial = ranges(128); + let mut initial = ranges(128); + f.reserve(&mut initial).await?; let init = d.prepare_initialization(operation(1), &initial)?; f.run(d.publish(&init)).await?; let view = f.run(d.observe()).await?; @@ -237,7 +268,8 @@ pub async fn bounded_discovery() -> Result<()> { assert!(pages <= 4 && bytes <= limits.max_cache_bytes); } let parent = &initial[63]; - let children = split(parent, 1000); + let mut children = split(parent, 1000); + f.reserve(&mut children).await?; let mutation = d.prepare_replacement(&view, operation(2), std::slice::from_ref(parent), &children)?; let before = f.store.range_page_puts.load(Ordering::Acquire); @@ -249,7 +281,8 @@ pub async fn bounded_discovery() -> Result<()> { ); let after = f.run(d.observe()).await?; assert_eq!(after.groups(), 129); - let merged = assigned(2000, parent.range.clone(), 3); + let mut merged = assigned(2000, parent.range.clone(), 3); + f.reserve(std::slice::from_mut(&mut merged)).await?; let merge = d.prepare_replacement( &after, operation(3), @@ -279,11 +312,13 @@ pub async fn bounded_discovery() -> Result<()> { pub async fn recovery() -> Result<()> { let f = Fixture::new(DirectoryLimits::default()).await?; let d = &f.directory; - let initial = ranges(1); + let mut initial = ranges(1); + f.reserve(&mut initial).await?; let init = d.prepare_initialization(operation(1), &initial)?; f.run(d.publish(&init)).await?; let view = f.run(d.observe()).await?; - let children = split(&initial[0], 100); + let mut children = split(&initial[0], 100); + f.reserve(&mut children).await?; let first = d.prepare_replacement(&view, operation(2), &initial, &children)?; f.store.fail_head_after.store(true, Ordering::Release); f.store @@ -297,7 +332,8 @@ pub async fn recovery() -> Result<()> { let retry = restarted.decode_mutation(first.encode())?; let selected = f.run(restarted.observe()).await?; assert_eq!(selected.revision(), 2); - let second_children = split(&children[0], 200); + let mut second_children = split(&children[0], 200); + f.reserve(&mut second_children).await?; let second = restarted.prepare_replacement(&selected, operation(3), &children[..1], &second_children)?; let latest_publication = f.run(restarted.publish(&second)).await?; @@ -386,7 +422,8 @@ pub async fn recovery() -> Result<()> { pub async fn genesis_race() -> Result<()> { let f = Fixture::new(DirectoryLimits::default()).await?; let d = &f.directory; - let initial = ranges(1); + let mut initial = ranges(1); + f.reserve(&mut initial).await?; let init = d.prepare_initialization(operation(1), &initial)?; f.store .range_genesis_after @@ -400,12 +437,10 @@ pub async fn genesis_race() -> Result<()> { let result = async { let receipt = d.publish(&init).await?; let view = d.observe().await?; - let replacement = d.prepare_replacement( - &view, - operation(2), - &initial, - &split(&initial[0], 100), - )?; + let mut children = split(&initial[0], 100); + f.reserve(&mut children).await?; + let replacement = + d.prepare_replacement(&view, operation(2), &initial, &children)?; d.publish(&replacement).await?; Ok::<_, CatalogError>(receipt) } @@ -430,7 +465,8 @@ pub async fn churn() -> Result<()> { }) .await?; let d = &f.directory; - let initial = ranges(16); + let mut initial = ranges(16); + f.reserve(&mut initial).await?; let init = d.prepare_initialization(operation(1), &initial)?; f.run(d.publish(&init)).await?; let original_view = f.run(d.observe()).await?; @@ -442,7 +478,7 @@ pub async fn churn() -> Result<()> { let view = f.run(d.observe()).await?; let position = (step * 7 + 3) % (expected.len() - 1); let id = 1000 + step as u64 * 10; - let (parents, children) = if step % 2 == 0 { + let (parents, mut children) = if step % 2 == 0 { let parent = expected[position].clone(); let children = split(&parent, id); (vec![parent], children) @@ -452,6 +488,7 @@ pub async fn churn() -> Result<()> { let epoch = parents.iter().map(|r| r.epoch.get()).max().unwrap() + 1; (parents, vec![assigned(id, range, epoch)]) }; + f.reserve(&mut children).await?; let mutation = d.prepare_replacement(&view, operation(step as u64 + 2), &parents, &children)?; let receipt = f.run(d.publish(&mutation)).await?; @@ -486,3 +523,219 @@ pub async fn churn() -> Result<()> { f.domain.shutdown().await; Ok(()) } + +pub async fn birth_recovery() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let request = d.prepare_group_birth(birth_spec(&ranges(1)[0]))?; + let route = request.route(); + let birth_key = f.key(&format!( + "births/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )); + let init = d.prepare_initialization(operation(99), std::slice::from_ref(route))?; + assert!(matches!( + f.run(d.publish(&init)).await, + Err(CatalogError::Corrupt(_)) + )); + assert!(matches!( + f.store.inner.get(&f.key("head")).await, + Err(object_store::Error::NotFound { .. }) + )); + f.run(d.reserve_group_birth(&request)).await?; + assert!( + f.run(d.materialize_group_birth(&request)).await.is_err(), + "an unselected reservation is not a birth" + ); + f.run(d.publish(&init)).await?; + assert!( + matches!( + f.store.inner.get(&birth_key).await, + Err(object_store::Error::NotFound { .. }) + ), + "range publication must not eagerly materialize groups" + ); + let unknown = fault::missing_read( + birth_key.to_string(), + f.run(d.materialize_group_birth(&request)), + ) + .await; + assert!(matches!(unknown, Err(CatalogError::Unconfirmed))); + assert_eq!( + f.store.inner.get(&birth_key).await?.bytes().await?.as_ref(), + request.encode(), + "unknown confirmation follows actual immutable creation" + ); + let restarted = f.reopen().await?; + let retry = restarted.decode_group_birth(request.encode())?; + assert_eq!(retry.route(), request.route()); + let birth = f.run(restarted.materialize_group_birth(&retry)).await?; + assert_eq!(birth.route(), route); + assert_eq!(birth.creation(), request.creation()); + assert_eq!(birth.voters(), request.voters()); + assert!(birth.learners().is_empty()); + f.run(restarted.require_current_group_birth(&birth)).await?; + assert!(matches!( + f.run(d.require_current_group_birth(&birth)).await, + Err(CatalogError::Conflict(_)) + )); + assert!(matches!( + fault::missing_read( + birth_key.to_string(), + f.run(restarted.require_current_group_birth(&birth)) + ) + .await, + Err(CatalogError::Corrupt(_)) + )); + let mut corrupt = request.encode().to_vec(); + *corrupt.last_mut().unwrap() ^= 1; + f.store.inner.put(&birth_key, corrupt.into()).await?; + assert!(matches!( + f.run(restarted.require_current_group_birth(&birth)).await, + Err(CatalogError::Corrupt(_)) + )); + f.store + .inner + .put(&birth_key, request.encode().to_vec().into()) + .await?; + let view = f.run(restarted.observe()).await?; + let mut children = split(route, 700); + f.reserve(&mut children).await?; + let replace = restarted.prepare_replacement( + &view, + operation(100), + std::slice::from_ref(route), + &children, + )?; + f.run(restarted.publish(&replace)).await?; + assert!(matches!( + f.run(restarted.require_current_group_birth(&birth)).await, + Err(CatalogError::Fenced) + )); + assert!(matches!( + f.run(restarted.materialize_group_birth(&retry)).await, + Err(CatalogError::Fenced) + )); + assert_eq!( + f.store.inner.get(&birth_key).await?.bytes().await?.as_ref(), + request.encode() + ); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn birth_reservations() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let d = &f.directory; + let route = ranges(1).remove(0); + let original = d.prepare_group_birth(birth_spec(&route))?; + assert_eq!(&original.encode()[..9], b"CLRCAT01\xba"); + let mut wrong_domain = original.encode().to_vec(); + wrong_domain[8] = 86; + assert!(d.decode_group_birth(&wrong_domain).is_err()); + let mut unordered = birth_spec(&route); + unordered.voters.reverse(); + assert_eq!( + d.prepare_group_birth(unordered)?.encode(), + original.encode() + ); + let key = Path::from(format!( + "groups/{}", + hex::encode(route.history.group.bytes()) + )); + assert!(matches!( + fault::missing_read(key.to_string(), f.run(d.reserve_group_birth(&original))).await, + Err(CatalogError::Unconfirmed) + )); + assert_eq!( + f.store.inner.get(&key).await?.bytes().await?.as_ref(), + original.encode() + ); + f.run(d.reserve_group_birth(&original)).await?; + for which in 0..4 { + let mut spec = birth_spec(&route); + match which { + 0 => spec.creation = operation(555), + 1 => spec.history.incarnation = IncarnationId::new([55; 16])?, + 2 => spec.voters[0].node = NodeId::new([55; 16])?, + _ => spec.epoch = RouteEpoch::new(2)?, + } + let changed = d.prepare_group_birth(spec)?; + assert_ne!(changed.encode(), original.encode()); + assert!(matches!( + f.run(d.reserve_group_birth(&changed)).await, + Err(CatalogError::Conflict(_)) + )); + } + let mut duplicate = birth_spec(&route); + duplicate.voters[1].node = duplicate.voters[0].node; + assert!(d.prepare_group_birth(duplicate).is_err()); + let mut wrong_count = birth_spec(&route); + wrong_count.voters.pop(); + assert!(d.prepare_group_birth(wrong_count).is_err()); + let mut oversized = birth_spec(&route); + oversized.learners = vec![oversized.voters[0]; 33]; + assert!(matches!( + d.prepare_group_birth(oversized), + Err(CatalogError::Limit) + )); + for end in 0..original.encode().len() { + assert!(d.decode_group_birth(&original.encode()[..end]).is_err()); + } + assert!(d.decode_group_birth(&vec![0; 64 * 1024 + 1]).is_err()); + let mut config = f.config.clone(); + config.namespace = NamespaceId::new([42; 16])?; + let catalog = Catalog::new(f.store.clone(), CatalogOptions::default())?; + let foreign = RangeDirectory::new( + f.run(catalog.open_namespace(config.clone())).await?, + DirectoryLimits::default(), + )?; + assert!(matches!( + foreign.decode_group_birth(original.encode()), + Err(CatalogError::Conflict(_)) + )); + let mut spec = birth_spec(&route); + spec.history.namespace = config.namespace; + let competing = foreign.prepare_group_birth(spec)?; + assert!(matches!( + f.run(foreign.reserve_group_birth(&competing)).await, + Err(CatalogError::Conflict(_)) + )); + f.run(d.reserve_group_birth(&original)).await?; + // The original reservation survives every conflicting attempt unchanged. + assert_eq!( + f.store.inner.get(&key).await?.bytes().await?.as_ref(), + original.encode() + ); + // Configuring a managed policy does not authorize the unmanaged APIs. No + // fleet permit is fabricated, and refusal must precede any store access. + config.namespace = NamespaceId::new([43; 16])?; + config.managed_fleet = Some(serde_json::from_value(serde_json::json!({ + "fleet": vec![44; 16], "catalog": vec![45; 32], + }))?); + let managed = RangeDirectory::new( + f.run(catalog.open_namespace(config.clone())).await?, + DirectoryLimits::default(), + )?; + let mut spec = birth_spec(&route); + spec.history.namespace = config.namespace; + assert!(matches!( + managed.prepare_group_birth(spec), + Err(CatalogError::ManagedAdmissionRequired) + )); + let (result, reads) = fault::count_reads(f.run(managed.reserve_group_birth(&original))).await; + assert!(matches!( + result, + Err(CatalogError::ManagedAdmissionRequired) + )); + assert!(reads.is_empty()); + assert!(matches!( + f.run(managed.materialize_group_birth(&original)).await, + Err(CatalogError::ManagedAdmissionRequired) + )); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index d1e4fc04b..7e79725db 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -291,7 +291,8 @@ "crates/cellar/durability/raft/cell_groups/directory/wire.rs", "crates/cellar/durability/raft/cell_groups/directory/tree.rs", "crates/cellar/durability/raft/cell_groups/directory/store.rs", - "crates/cellar/tests/support/cell_group_directory.rs" + "crates/cellar/tests/support/cell_group_directory.rs", + "crates/cellar/durability/raft/cell_groups/directory/birth.rs" ], "test_target": "native_cell_groups_catalog", "steps": [ From 40c3c0b143f72496f0345be890413ed855c241ae Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:39:14 -0700 Subject: [PATCH 022/568] style: apply declaration spacing to group birth tag --- crates/cellar/durability/raft/cell_groups/directory/wire.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 73465f69d..303a05bad 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -8,6 +8,7 @@ pub(super) const HEAD: u8 = 82; pub(super) const JOURNAL: u8 = 83; pub(super) const PIN: u8 = 84; pub(super) const REQUEST: u8 = 85; + // Fleet maintenance already uses 86 in the shared catalog envelope. pub(super) const GROUP_BIRTH: u8 = 186; pub(super) const LIMIT: usize = 64 * 1024; From 951c1c6c6371ef52c5783d540734ab2f04aa023e Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 13:46:33 -0700 Subject: [PATCH 023/568] docs: record cell-group birth metadata checkpoint --- .../durability/raft/cell_groups/README.md | 5 +- plans/multi-raft/cell-groups/baseline.json | 12 ++--- plans/multi-raft/cell-groups/progress.md | 53 +++++++++++++++++-- 3 files changed, 59 insertions(+), 11 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index cf24974ea..2312f21ba 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -181,8 +181,9 @@ real RustFS or the full CG-03 gate. ## Cell-independent group birth (CG-03 in progress) `CellGroupBirthSpec` binds physical group history, range, route epoch, original -creation operation and initial members. It has no cell name. The namespace binds -schema, storage representation and durability policy. Preparation sorts members +creation operation and initial members. It has no single-cell ownership field; +range boundaries still contain full cell keys. The namespace binds schema, +storage representation and durability policy. Preparation sorts members canonically and enforces the namespace voter policy and distinct member/node IDs. The bounded marker-1 birth encoding has its own semantic record kind; its digest is the route's birth binding. Persist the returned request bytes before submission. diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 595d43a69..12c33c159 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "a0c11969763edbe86ee08fc7d1650129058c8ee0", + "source_revision": "40c3c0b143f72496f0345be890413ed855c241ae", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed CG-03 range-directory checkpoint at a0c11969763edbe86ee08fc7d1650129058c8ee0. The catalog.rs diff adds only the six-line range-directory module/export block; namespace and birth excerpts are unchanged and move to lines 110 and 179. Existing birth selection, bootstrap claims and managed admission are unchanged. New directory code verifies bounded immutable AVL ranges, one conditional head with journal/index bindings, historical retry reconciliation and permanent genesis pinning. Eight cell-group catalog cases and thirteen existing fleet catalog cases pass, including 80 split/merge metadata transitions. This is a partial CG-03 checkpoint: no group birth/disk-claim integration, client redirect path, production shared host or live topology authority is qualified. Full hard-cutover implementation remains active.", + "review_note": "2026-10-29: reviewed cell-independent birth source at 9db4ba8fc91e6d67de244c1a684001393bcc9e56. The only catalog.rs change exports the new birth types; existing namespace and birth excerpts are unchanged and move by one line. New birth requests bind group history, range/epoch, creation identity, canonical members and namespace policy with distinct catalog kind 186. Global identity reservations must match before directory publication; materialization verifies current selection and exact immutable bytes. These are metadata observations, never voter-start permits. Existing physical bootstrap, node pins and disk claims are unchanged and still require cell-independent engine integration. CG-03 remains IN_PROGRESS; full hard cutover is active. Formatting successor 40c3c0b143f72496f0345be890413ed855c241ae adds only the required declaration separator; anchored source hashes and excerpts are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3250b6a2a4d611850dd59c51c3aa4dfaa173ec956bbe4e5d6014e5afbf03b842", - "line": 110, + "sha256": "d925a1f3902fd9288d51ab8b252680f062a829e36ad537667d922be34f7cf8fc", + "line": 111, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3250b6a2a4d611850dd59c51c3aa4dfaa173ec956bbe4e5d6014e5afbf03b842", - "line": 179, + "sha256": "d925a1f3902fd9288d51ab8b252680f062a829e36ad537667d922be34f7cf8fc", + "line": 180, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index f6e565251..1fe6bef1f 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -78,9 +78,9 @@ CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `a0c11969763edbe86ee08fc7d1650129058c8ee0`. -The catalog anchor adds the directory module/export block; its unchanged namespace -and birth excerpts move by six lines. Other source anchors remain unchanged. A new +The reviewed source baseline is `40c3c0b143f72496f0345be890413ed855c241ae`. +The catalog anchor adds directory and birth exports; its unchanged namespace +and birth excerpts move by seven lines. Other source anchors remain unchanged. A new executor must receive both ignored DONE-task evidence folders before validation. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured @@ -331,3 +331,50 @@ The record binds 63 source inputs, nine successful command logs and two copied native executables. All-target Clippy with warnings denied, formatting/layout, native dependency isolation, docs, source preflight and all 37 plan-validator tests pass at this checkpoint. + +### CG-03 cell-independent birth checkpoint + +Validation source: `40c3c0b143f72496f0345be890413ed855c241ae` (declaration-spacing successor). + +Source `9db4ba8fc91e6d67de244c1a684001393bcc9e56` binds a group birth to physical +history, range/epoch, creation operation, namespace policy and canonical initial +members. It has no single-cell ownership field; range boundaries still carry full +cell routing keys. The bounded catalog encoding uses +the distinct kind 186; the provisional overlapping tag was corrected before this +checkpoint. No older birth layout is decoded or migrated by the new API. + +Global group-ID reservation uses the existing permanent key and the entire +canonical birth. Different namespaces, incarnations, creation IDs or placements +cannot reuse that key. The range directory now verifies every new route against +its exact reserved birth before publication. Losing a topology CAS retains the +reservation. Selected ranges can remain unmaterialized until their first use. + +Lazy materialization checks current range selection and the reservation before +conditional creation, then checks selection again. Unknown outcomes reconcile +with the original encoded request. A verified birth observation remains metadata: +it refuses wrong catalog owners, missing or changed immutable data and retired +routes, and grants no permission to initialize a voter. Managed namespaces refuse +the unmanaged APIs before I/O; their rooted fleet authorization remains to be +integrated. + +All ten native catalog tests and all 16 existing node-directory regressions pass, +including the distinct wire-tag assertion. The new cases cover canonical placement, +lazy creation, missing confirmations after actual writes, exact retry after reopen, +global cross-namespace conflicts, retired-range refusal and managed-policy refusal. +Existing bounded routing, races and the 80-transition churn oracle still pass. + +Physical `GroupBootstrap`, engine mappings and replica identities still carry the +old cell-scoped identity. The next step is to make that durable log path explicitly +group-scoped, then connect the verified birth to real synced genesis, node storage +pins and permanent member claims. A synthetic cell name is not an acceptable +adapter. The required `birth_unknown_and_claimed_blank_refuse` integration case +remains absent and CG-03 remains IN_PROGRESS. Redirect budgets, the shared host and +all later task/release gates remain outstanding. + +The partial evidence record at +`.cellar/cell-groups-evidence/CG-03-birth/checkpoint.json` binds 67 source inputs, +nine successful command logs and two copied native executables. All-target Clippy +with warnings denied, formatting/layout, native dependency isolation, docs, source +preflight and all 37 plan-validator tests pass. The failed declaration-spacing +check is retained with its source revision and corrected result. This is not a +CG-03 DONE record or proof of shared-group disk-loss recovery. From 462aa2eb46c7902228b504694e4fb1d5e569a98a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 14:02:52 -0700 Subject: [PATCH 024/568] docs: finalize executable cell-group plan handoff --- plans/multi-raft/cell-groups/README.md | 25 ++++++++++++++++------ plans/multi-raft/cell-groups/progress.md | 27 ++++++++++++++++++++++-- plans/multi-raft/cell-groups/tasks.json | 7 +++--- 3 files changed, 47 insertions(+), 12 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 679540f9b..05eb94cb5 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,10 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The design package is delivered. The active goal is now its end-to-end -implementation and hard cutover to the shared-group runtime. Checkpoints are -recorded separately; they do not mark the remaining tasks complete. +The current requested deliverable is the executable design and implementation +plan. The target remains an end-to-end hard cutover to the shared-group runtime; +executing all remaining tasks is a separate implementation phase. Existing code +checkpoints are preserved and do not mark the remaining tasks complete. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,17 +69,27 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The resumed implementation goal includes every task and the final runtime cutover. -The earlier plan-only handoff is historical; package validation does not satisfy -the active implementation objective. +The latest user clarification selects an executable design and implementation +plan. This handoff supersedes earlier instructions in the progress history to +continue implementation immediately. No further runtime changes are part of this +plan handoff; all remaining implementation and qualification gates stay explicit. -Resume CG-03 after reading [progress](progress.md) and inspecting the working diff. +When implementation is resumed, start with CG-03 after reading +[progress](progress.md) and inspecting the working diff. CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. +The next concrete boundary is physical group bootstrap: remove the cell binding +from replica/log identity, durable engine mappings, node inventory and permanent +member claims. Connect the verified birth to synced genesis and test restart, +unknown outcomes and refusal after a claimed voter disk is lost. A birth metadata +observation alone cannot authorize voter startup. A fabricated cell name is not +a substitute for a group identity. Then complete bounded redirects before CG-04 +connects enrollment to the shared runtime. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 1fe6bef1f..3733a600a 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,10 +3,33 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The active goal resumes end-to-end implementation after the plan -handoff recorded below. Intermediate contracts do not qualify a shared runtime. +hard cutover. The latest user clarification requests the executable plan as the +current deliverable. Earlier implementation checkpoints below remain historical +evidence; they do not authorize continuing runtime changes in this handoff. +Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## Current plan handoff — 2026-10-29 + +The package describes the complete replacement architecture, ordered implementation +briefs and source-bound qualification gates. CG-01 and CG-02 retain their existing +DONE evidence; CG-03 remains IN_PROGRESS and CG-04 through CG-16 remain TODO. +The next executor starts with physical group identity, real genesis/member-claim +integration and bounded redirects, then follows the task graph. This handoff adds +no runtime implementation or new runtime qualification claim. + +The branch includes fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. The supplied TiKV checkout remains at +`b3995dab28308f09cbb7c125bf32f789dbc203fe`. Existing local DONE evidence must +accompany a fresh checkout to validate those checkpoints. Final release remains +closed; a validated plan does not prove shared-group performance or recovery. + +Handoff checks pass: package validation (16 tasks, 83 required cases, 14 invariants, +nine profiles), reviewed source preflight, all 37 validator unit tests, repository +documentation checks and `git diff --check`. The CG-03 executor brief renders +successfully. `release` exits nonzero as expected because implementation evidence +is incomplete. Runtime suites were not rerun for these plan-only edits. + ## Historical plan handoff — 2026-10-29 This section records the state at `dcc52840`; the directory checkpoint below diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 7e79725db..e091b02d9 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -282,7 +282,8 @@ "crates/cellar/durability/raft/", "crates/cellar/application/consensus/", "crates/cellar/node/", - "crates/cellar/tests/support/raft_catalog_fault.rs" + "crates/cellar/tests/support/raft_catalog_fault.rs", + "crates/cellar-logic/" ], "creates": [ "crates/cellar/durability/raft/cell_groups/enrollment.rs", @@ -309,7 +310,7 @@ ] }, { - "action": "Reconcile lost birth/root/enrollment replies by canonical operation identity. Keep unconfirmed outcomes typed; a blank claimed voter cannot bootstrap under a new path. Refuse enrollment under sealed or retired ranges.", + "action": "Make physical replica/log identity, engine registry and durable mappings, node storage inventory and permanent member claims group-scoped, without a synthetic cell name. Connect the exact verified birth to synced real-engine genesis before permitting voter startup. Reject old storage domains rather than adding a legacy decoder. Reconcile lost birth/root/enrollment replies by canonical operation identity. Keep unconfirmed outcomes typed; a blank claimed voter cannot bootstrap under a new path. Refuse enrollment under sealed or retired ranges. Metadata-only birth tests do not satisfy this integration boundary.", "tests": [ "birth_unknown_and_claimed_blank_refuse" ] @@ -341,7 +342,7 @@ }, { "name": "birth_unknown_and_claimed_blank_refuse", - "oracle": "Unknown creation reconciles its winner and missing permanently claimed disk state refuses.", + "oracle": "Real-engine genesis binds the selected group birth and permanent member claim without a cell owner; restart preserves that exact identity. Unknown creation reconciles its winner, identity substitution and old storage domains refuse, and deleting an owned claimed voter fixture cannot authorize fresh genesis under another path. No metadata observation alone grants voter-start authority.", "invariants": [ "I01", "I08" From d72ea068a527d2027e806ca78ab390256e8634fa Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 14:16:48 -0700 Subject: [PATCH 025/568] refactor: key engine catalog by physical group history --- crates/cellar-logic/cell_group.rs | 13 ++ crates/cellar-logic/cell_group_tests.rs | 25 +++ .../durability/raft/cell_groups/README.md | 14 ++ .../durability/raft/physical_inventory.rs | 17 +- crates/cellar/durability/raft/records.rs | 71 +++++++- crates/cellar/durability/raft/storage.rs | 57 ++++--- .../durability/raft/storage_genesis_tests.rs | 158 ++++++++++++++++++ 7 files changed, 327 insertions(+), 28 deletions(-) diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs index ba694ab6f..d27803e2d 100644 --- a/crates/cellar-logic/cell_group.rs +++ b/crates/cellar-logic/cell_group.rs @@ -74,6 +74,19 @@ impl GroupHistory { } } +/// Projects the physical history from the temporary single-cell adapter. +/// This does not validate or authorize the adapter's cell binding. +impl From<&crate::consensus::GroupScope> for GroupHistory { + fn from(scope: &crate::consensus::GroupScope) -> Self { + Self { + cluster: scope.cluster, + namespace: scope.namespace, + group: scope.group, + incarnation: scope.incarnation, + } + } +} + macro_rules! nonzero_counter { ($($name:ident),+ $(,)?) => {$ ( #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] diff --git a/crates/cellar-logic/cell_group_tests.rs b/crates/cellar-logic/cell_group_tests.rs index 4d68783b5..7851b0e1d 100644 --- a/crates/cellar-logic/cell_group_tests.rs +++ b/crates/cellar-logic/cell_group_tests.rs @@ -153,3 +153,28 @@ fn cell_versions_require_exact_history_and_same_sequence_image() { assert_eq!(conflicting.covers(&version), Err(Refusal::WrongHistory)); } } + +#[test] +fn physical_history_projection_preserves_every_group_identity_field() { + let original = crate::consensus::GroupScope { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + cell: "alpha".into(), + }; + let history = GroupHistory::from(&original); + let mut other = original.clone(); + other.cell = "beta".into(); + assert_eq!(GroupHistory::from(&other), history); + for field in 0..4 { + let mut other = original.clone(); + match field { + 0 => other.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => other.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => other.group = GroupId::new([9; 16]).unwrap(), + _ => other.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert_ne!(GroupHistory::from(&other), history); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 2312f21ba..5e67affe6 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -209,3 +209,17 @@ before the new birth can be integrated with actual disk-claim admission. Managed namespaces refuse these unmanaged APIs until their rooted fleet authorization is connected. This staged boundary does not permit blank-disk recovery, replace current quorum membership, or complete CG-03 qualification. + +## Physical log identity cutover + +The engine registry uses `GroupHistory` (cluster, namespace, group and incarnation), +independent of application cell names. Marker-1 mappings use an explicit +`cellar-group-mapping-1` domain and a separate physical history. Old bare `scope` +mapping records refuse during recovery; there is no compatibility reader. + +The staged single-cell host still stores its real cell binding as application +adapter metadata. Its lookup, birth retries, pending slot accounting and physical +inventory must reject another cell binding for the same physical history. This +metadata is not part of the registry key and is removed with the remaining +single-cell runtime consumers. This foundation does not expose shared-group voter +startup or establish the permanent member claim required by the new birth API. diff --git a/crates/cellar/durability/raft/physical_inventory.rs b/crates/cellar/durability/raft/physical_inventory.rs index f8ab10d68..4aab1859b 100644 --- a/crates/cellar/durability/raft/physical_inventory.rs +++ b/crates/cellar/durability/raft/physical_inventory.rs @@ -6,6 +6,7 @@ use super::cache::GroupLog; use super::host::RaftHostObservation; use super::storage::LogError; use super::transport::TransportError; +use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{GroupScope, MemberId}; use sha2::{Digest, Sha256}; use std::collections::BTreeMap; @@ -148,15 +149,25 @@ pub(crate) fn scope_digest(scope: &GroupScope) -> Result<[u8; 32], LogError> { } pub(crate) fn capture( - groups: &BTreeMap, + groups: &BTreeMap, pending: Option<&GroupScope>, ) -> Result { if groups.len() > MAX_GROUPS { return Err(LogError::Overloaded); } + if let Some(scope) = pending { + scope.validate().map_err(|_| LogError::WrongIdentity)?; + if groups + .get(&GroupHistory::from(scope)) + .is_some_and(|group| &group.identity().scope != scope) + { + return Err(LogError::WrongIdentity); + } + } let mut histories = Vec::with_capacity(groups.len()); - for (scope, group) in groups { - if scope != &group.identity().scope || group.birth_digest == [0; 32] { + for (history, group) in groups { + let scope = &group.identity().scope; + if history != &GroupHistory::from(scope) || group.birth_digest == [0; 32] { return Err(LogError::WrongIdentity); } let cache = group.read()?; diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index ebc45575e..502432ada 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -4,9 +4,12 @@ use super::cache::{CachedGroup, CommittedImage, DurableApply}; use super::storage::{GroupBootstrap, LogError}; -use crate::application::consensus::protocol::WireScope; use ::raft::prelude::{ConfState, HardState, Snapshot}; -use cellar_logic::consensus::{DiskLineage, EngineInstanceId, LogIndex, LogPosition, NodeId, Term}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{ + ClusterId, DiskLineage, EngineInstanceId, GroupId, GroupScope, IncarnationId, LogIndex, + LogPosition, NamespaceId, NodeId, Term, +}; use protobuf::Message; use serde::{Deserialize, Serialize}; @@ -57,7 +60,11 @@ pub(crate) struct RootRecord { pub(crate) struct MappingRecord { pub format: u32, pub key: u64, - pub scope: WireScope, + domain: MappingDomain, + history: PhysicalHistory, + // Temporary application adapter metadata, never the physical registry key. + // Removed with the remaining single-cell host consumers at public cutover. + application_cell: String, pub member: u64, pub birth_digest: [u8; 32], pub initial_conf: Vec, @@ -68,15 +75,30 @@ impl MappingRecord { Ok(Self { format: 1, key, - scope: WireScope::from(&bootstrap.scope), + domain: MappingDomain::Group, + history: PhysicalHistory::from(GroupHistory::from(&bootstrap.scope)), + application_cell: bootstrap.scope.cell.clone(), member: bootstrap.member.get(), birth_digest: bootstrap.birth_digest, initial_conf: conf.write_to_bytes().map_err(corrupt)?, }) } + pub fn application_scope(&self) -> Result { + let history = self.history.decode()?; + let scope = GroupScope { + cluster: history.cluster, + namespace: history.namespace, + group: history.group, + incarnation: history.incarnation, + cell: self.application_cell.clone(), + }; + scope.validate().map_err(corrupt)?; + Ok(scope) + } + pub fn catalog_key(&self) -> Result, LogError> { - let scope = self.scope.clone().decode().map_err(corrupt)?; + let scope = self.history.decode()?; let mut key = b"g:".to_vec(); key.extend_from_slice(&scope.group.bytes()); key.extend_from_slice(&scope.incarnation.bytes()); @@ -84,6 +106,43 @@ impl MappingRecord { } } +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +enum MappingDomain { + #[serde(rename = "cellar-group-mapping-1")] + Group, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct PhysicalHistory { + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], +} + +impl From for PhysicalHistory { + fn from(history: GroupHistory) -> Self { + Self { + cluster: history.cluster.bytes(), + namespace: history.namespace.bytes(), + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + } + } +} + +impl PhysicalHistory { + fn decode(&self) -> Result { + Ok(GroupHistory { + cluster: ClusterId::new(self.cluster).map_err(corrupt)?, + namespace: NamespaceId::new(self.namespace).map_err(corrupt)?, + group: GroupId::new(self.group).map_err(corrupt)?, + incarnation: IncarnationId::new(self.incarnation).map_err(corrupt)?, + }) + } +} + #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct GroupRecord { @@ -134,7 +193,7 @@ impl GroupRecord { let (image, group_manifest) = match &self.application { ApplicationRecord::Image(image) => (image.clone(), None), ApplicationRecord::Group(marker) => { - let scope = self.mapping.scope.clone().decode().map_err(corrupt)?; + let scope = self.mapping.application_scope()?; if !marker.matches_group(&scope, self.mapping.birth_digest) { return Err(LogError::WrongIdentity); } diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 63c459983..e26530017 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -13,6 +13,7 @@ use super::records::{ self, GroupRecord, MappingRecord, NodeRecord, RootRecord, GROUP_KEY, ROOT_KEY, }; use ::raft::prelude::{ConfState, Entry}; +use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{GroupScope, MemberId, OperationId, ReplicaIdentity}; use cellar_ltx::{FileSystem, HostFileLock}; use protobuf::Message; @@ -119,7 +120,7 @@ pub struct SyncReceipt { pub groups: Vec, } -type Catalog = Arc>>; +type Catalog = Arc>>; type Reply = oneshot::Sender>; #[derive(Clone)] @@ -442,11 +443,22 @@ impl ReplicaLog { ) -> Result { self.admit()?; let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + let additional = match pending { + Some(scope) => { + scope.validate().map_err(|_| LogError::WrongIdentity)?; + match groups.get(&GroupHistory::from(scope)) { + Some(group) if &group.identity.scope != scope => { + return Err(LogError::WrongIdentity); + } + Some(_) => false, + None => true, + } + } + None => false, + }; groups .len() - .checked_add(usize::from( - pending.is_some_and(|scope| !groups.contains_key(scope)), - )) + .checked_add(usize::from(additional)) .ok_or(LogError::Overloaded) } @@ -476,13 +488,15 @@ impl ReplicaLog { pub fn group(&self, scope: &GroupScope) -> Result { self.admit()?; - self.inner - .groups - .read() - .map_err(|_| LogError::Closed)? - .get(scope) - .cloned() - .ok_or(LogError::NotFound) + scope.validate().map_err(|_| LogError::WrongIdentity)?; + let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + let group = groups + .get(&GroupHistory::from(scope)) + .ok_or(LogError::NotFound)?; + if &group.identity.scope != scope { + return Err(LogError::WrongIdentity); + } + Ok(group.clone()) } pub async fn create_group(&self, bootstrap: GroupBootstrap) -> Result { @@ -877,7 +891,7 @@ impl Worker { .groups .write() .map_err(|_| LogError::Closed)? - .insert(handle.identity.scope.clone(), handle) + .insert(GroupHistory::from(&handle.identity.scope), handle) .is_some() { return Err(LogError::Corrupt("duplicate logical group mapping".into())); @@ -910,7 +924,7 @@ impl Worker { { return Err(LogError::Corrupt("member binding birth mismatch".into())); } - let scope = mapping.scope.clone().decode().map_err(records::corrupt)?; + let scope = mapping.application_scope()?; scope.validate().map_err(records::corrupt)?; cache.membership.validate_scope(&scope)?; Ok(GroupLog { @@ -990,13 +1004,14 @@ impl Worker { .groups .read() .map_err(|_| LogError::Closed)? - .get(&bootstrap.scope) + .get(&GroupHistory::from(&bootstrap.scope)) { let mapping = self .mappings .get(&group.key) .ok_or(LogError::LostStableState)?; - if mapping.member == bootstrap.member.get() + if group.identity.scope == bootstrap.scope + && mapping.member == bootstrap.member.get() && mapping.birth_digest == bootstrap.birth_digest && mapping.initial_conf == conf.write_to_bytes().map_err(records::corrupt)? { @@ -1039,13 +1054,17 @@ impl Worker { if !seed.authorized(&self.owner, &identity) { return Err(LogError::WrongIdentity); } - if self + if let Some(group) = self .groups .read() .map_err(|_| LogError::Closed)? - .contains_key(&identity.scope) + .get(&GroupHistory::from(&identity.scope)) { - return Err(LogError::AlreadyExists); + return Err(if group.identity.scope == identity.scope { + LogError::AlreadyExists + } else { + LogError::WrongIdentity + }); } let bootstrap = seed.bootstrap()?; let conf = seed.initial().clone(); @@ -1113,7 +1132,7 @@ impl Worker { self.groups .write() .map_err(|_| LogError::Closed)? - .insert(bootstrap.scope, handle.clone()); + .insert(GroupHistory::from(&bootstrap.scope), handle.clone()); self.mappings.insert(key, mapping); self.cache_bytes += bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); diff --git a/crates/cellar/durability/raft/storage_genesis_tests.rs b/crates/cellar/durability/raft/storage_genesis_tests.rs index 911345f36..27c2edc3a 100644 --- a/crates/cellar/durability/raft/storage_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_genesis_tests.rs @@ -1,3 +1,5 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; use crate::asyncrt::ExecutionDomain; use crate::raft::member_state::FleetGroupBinding; @@ -133,3 +135,159 @@ async fn managed_genesis_publication_waits_for_actual_engine_sync() -> anyhow::R domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn physical_catalog_rejects_cell_aliases_across_restart() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut options = options(root.path()); + let initial = bootstrap(5); + let mut alias = initial.clone(); + alias.scope.cell = "Counter:alias".into(); + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let key = log.create_group(initial.clone()).await?.numeric_key(); + for pass in 0..2 { + let log = if pass == 0 { + log.clone() + } else { + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([9; 16])?; + domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?? + }; + assert_eq!(log.group(&initial.scope)?.numeric_key(), key); + assert_eq!(log.create_group(initial.clone()).await?.numeric_key(), key); + assert!(matches!( + log.group(&alias.scope), + Err(LogError::WrongIdentity) + )); + assert!(matches!( + log.create_group(alias.clone()).await, + Err(LogError::WrongIdentity) + )); + assert_eq!(log.group_count_with_pending(Some(&initial.scope))?, 1); + assert!(matches!( + log.group_count_with_pending(Some(&alias.scope)), + Err(LogError::WrongIdentity) + )); + assert!(matches!( + log.physical_inventory(Some(&alias.scope)), + Err(LogError::WrongIdentity) + )); + let inventory = log.physical_inventory(Some(&initial.scope))?; + assert_eq!(inventory.histories().len(), 1); + assert_eq!( + inventory.history(&initial.scope)?.unwrap().birth_digest(), + initial.birth_digest + ); + assert_eq!(log.group_count_with_pending(Some(&bootstrap(7).scope))?, 2); + assert_eq!(log.groups()?.len(), 1); + log.close().await?; + } + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> anyhow::Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + for corruption in ["old-scope", "unknown-domain", "zero-history", "empty-cell"] { + let root = tempfile::tempdir()?; + let mut options = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let key = log.create_group(initial.clone()).await?.numeric_key(); + let performance = log.performance(); + log.close().await?; + let dir = options.directory.join("log").to_string_lossy().into_owned(); + domain + .blocking(move || -> anyhow::Result<()> { + let engine = Engine::open_with_file_system( + raft_engine::Config { + dir, + ..Default::default() + }, + Arc::new(EngineFileSystem( + Arc::new(cellar_ltx::DirectFileSystem), + performance, + )), + )?; + let mut record: serde_json::Value = + serde_json::from_slice(&engine.get(key, GROUP_KEY).unwrap())?; + let mapping = &mut record["mapping"]; + assert_eq!(mapping["format"], 1); + assert_eq!(mapping["domain"], "cellar-group-mapping-1"); + assert_eq!(mapping["application_cell"], initial.scope.cell); + assert_eq!(mapping["history"].as_object().unwrap().len(), 4); + assert!(mapping["history"].get("cell").is_none()); + let mut catalog_key = b"g:".to_vec(); + catalog_key.extend_from_slice(&initial.scope.group.bytes()); + catalog_key.extend_from_slice(&initial.scope.incarnation.bytes()); + assert_eq!( + serde_json::from_slice::( + &engine.get(0, &catalog_key).unwrap() + )?, + *mapping + ); + match corruption { + "old-scope" => { + let fields = mapping.as_object_mut().unwrap(); + fields.remove("domain"); + fields.remove("history"); + fields.remove("application_cell"); + fields.insert( + "scope".into(), + serde_json::to_value( + crate::application::consensus::protocol::WireScope::from( + &initial.scope, + ), + )?, + ); + } + "unknown-domain" => { + mapping["domain"] = serde_json::json!("cellar-image-mapping-1") + } + "zero-history" => { + mapping["history"]["group"] = serde_json::json!(vec![0u8; 16]) + } + "empty-cell" => mapping["application_cell"] = serde_json::json!(""), + _ => unreachable!(), + } + let mut batch = LogBatch::default(); + batch.put(0, catalog_key, serde_json::to_vec(mapping)?)?; + batch.put(key, GROUP_KEY.to_vec(), serde_json::to_vec(&record)?)?; + engine.write(&mut batch, true)?; + Ok(()) + })? + .await??; + options.mode = LogOpenMode::Resume; + assert!( + matches!( + domain + .run(ReplicaLog::open( + options, + Arc::new(cellar_ltx::DirectFileSystem) + )) + .await?, + Err(LogError::Corrupt(_)) + ), + "accepted {corruption}" + ); + } + domain.shutdown().await; + Ok(()) +} From aead5cc04a9788ab2f2ee5f3fba3d6468fde5b75 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 14:32:53 -0700 Subject: [PATCH 026/568] docs: record physical group registry qualification --- plans/multi-raft/cell-groups/README.md | 16 +++--- plans/multi-raft/cell-groups/baseline.json | 4 +- plans/multi-raft/cell-groups/progress.md | 63 ++++++++++++++++++---- 3 files changed, 63 insertions(+), 20 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 05eb94cb5..ea601e4ac 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,10 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The current requested deliverable is the executable design and implementation -plan. The target remains an end-to-end hard cutover to the shared-group runtime; -executing all remaining tasks is a separate implementation phase. Existing code -checkpoints are preserved and do not mark the remaining tasks complete. +The executable design package is delivered. The resumed active goal is its +end-to-end implementation and hard cutover to the shared-group runtime. Existing +checkpoints do not mark the remaining tasks complete. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -69,12 +68,11 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The latest user clarification selects an executable design and implementation -plan. This handoff supersedes earlier instructions in the progress history to -continue implementation immediately. No further runtime changes are part of this -plan handoff; all remaining implementation and qualification gates stay explicit. +The user has resumed end-to-end implementation. The previous plan-only handoff +is historical; all remaining runtime and qualification tasks are in scope. +Package validation alone does not satisfy the active objective. -When implementation is resumed, start with CG-03 after reading +Continue CG-03 after reading [progress](progress.md) and inspecting the working diff. CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 12c33c159..f63b24f64 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "40c3c0b143f72496f0345be890413ed855c241ae", + "source_revision": "d72ea068a527d2027e806ca78ab390256e8634fa", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed cell-independent birth source at 9db4ba8fc91e6d67de244c1a684001393bcc9e56. The only catalog.rs change exports the new birth types; existing namespace and birth excerpts are unchanged and move by one line. New birth requests bind group history, range/epoch, creation identity, canonical members and namespace policy with distinct catalog kind 186. Global identity reservations must match before directory publication; materialization verifies current selection and exact immutable bytes. These are metadata observations, never voter-start permits. Existing physical bootstrap, node pins and disk claims are unchanged and still require cell-independent engine integration. CG-03 remains IN_PROGRESS; full hard cutover is active. Formatting successor 40c3c0b143f72496f0345be890413ed855c241ae adds only the required declaration separator; anchored source hashes and excerpts are unchanged.", + "review_note": "2026-10-29: reviewed the physical engine-registry cutover at d72ea068a527d2027e806ca78ab390256e8634fa. The registry now keys by GroupHistory; durable mappings store an explicit group-mapping domain and physical history separately from temporary single-cell adapter metadata. Old mapping shapes refuse. Existing adapter calls reject cell aliases at lookup, creation, pending occupancy and inventory boundaries. Anchored source hashes and excerpts are unchanged from the previous reviewed birth checkpoint. GroupBootstrap, replica handles and permanent claims still need the cell-independent cutover; this source does not expose shared-group voter startup. CG-03 remains IN_PROGRESS and the full hard-cutover goal is active.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 3733a600a..94bfa4d93 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,13 +3,11 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The latest user clarification requests the executable plan as the -current deliverable. Earlier implementation checkpoints below remain historical -evidence; they do not authorize continuing runtime changes in this handoff. -Intermediate contracts do not qualify a shared runtime. +hard cutover. The user has resumed the full implementation goal following the +plan handoff below. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. -## Current plan handoff — 2026-10-29 +## Plan handoff at 462aa2eb — 2026-10-29 The package describes the complete replacement architecture, ordered implementation briefs and source-bound qualification gates. CG-01 and CG-02 retain their existing @@ -101,10 +99,10 @@ CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `40c3c0b143f72496f0345be890413ed855c241ae`. -The catalog anchor adds directory and birth exports; its unchanged namespace -and birth excerpts move by seven lines. Other source anchors remain unchanged. A new -executor must receive both ignored DONE-task evidence folders before validation. +The reviewed source baseline is `d72ea068a527d2027e806ca78ab390256e8634fa`. +The physical-registry change leaves all existing source anchor hashes and excerpts +unchanged. A new executor must receive both ignored DONE-task evidence folders +before validating those historical checkpoints. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until @@ -401,3 +399,50 @@ with warnings denied, formatting/layout, native dependency isolation, docs, sour preflight and all 37 plan-validator tests pass. The failed declaration-spacing check is retained with its source revision and corrected result. This is not a CG-03 DONE record or proof of shared-group disk-loss recovery. + + +### CG-03 physical engine registry checkpoint + +Source: `d72ea068a527d2027e806ca78ab390256e8634fa`. + +The engine catalog now keys histories by `GroupHistory`, excluding the application +cell name. Synced mapping records use the explicit `cellar-group-mapping-1` domain +and four physical identity fields. The old bare `scope` mapping shape refuses on +recovery; no compatibility decoder or disk migration is introduced. + +The staged single-cell host retains its real cell binding separately as application +metadata. Lookup, create retries, seeded attachment, pending-slot accounting and +physical inventory reject a different cell binding for an existing physical +history. These checks prevent an adapter caller from turning another cell name +into a second handle or phantom slot. The inventory wire still uses its existing +cell-scoped digest until its consumers undergo the later cutover. + +This is a storage prerequisite, not the full physical bootstrap conversion. +`GroupBootstrap`, replica handles, node pins and permanent member claims still +need the cell-independent path before the new birth API may start a voter. +The required claimed-blank-disk integration case remains absent. CG-03 stays +IN_PROGRESS; CG-04 through CG-16 and the final release gate remain pending. + + +Validation at this source passes 19 pure-core cases, 27 selected library cases, +all four genesis cases, 14 native manifest cases, 19 native storage cases, +16 node-directory cases, 10 cell-group catalog cases, 33 host cases and +54 snapshot cases. The new engine tests inspect the stored physical mapping, +restart the actual engine, reject cell aliases at lookup, creation, pending-slot +and inventory boundaries, and rewrite both catalog and group records to prove old-domain, unknown-domain, +zero-history and invalid-cell refusal. Existing managed genesis sync-barrier and +bootstrap claim tests still pass. The snapshot suite includes seeded restart, +lost-claim confirmation and selected-payload refusal; its explicitly ignored +RustFS case was not run. Parent-launched process fixtures remain intentionally +ignored by direct suite execution. + +All-target Clippy with warnings denied, formatting/layout, native dependency +isolation, docs, source preflight and all 37 plan-validator tests pass. The partial +record is retained under +`.cellar/cell-groups-evidence/CG-03-physical-identity/checkpoint.json` with source, +executable and command-output hashes: 72 source inputs, 13 successful command logs, +eight executables and 194 distinct Rust cases. An interrupted build using the machine's +older default compiler and a command naming a nonexistent test target are retained +as non-qualifying attempts. Final checks use Rust 1.98.1 and actual test targets. +These results do not qualify the shared host, new group member claims, RustFS +recovery, hibernation, performance or the final hard cutover. From 98aff95abafdecd2c8bd942d266509c3b6fd1db9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:07:19 -0700 Subject: [PATCH 027/568] docs: hand off executable cell-group implementation plan --- plans/multi-raft/cell-groups/README.md | 28 +++++++++--------- plans/multi-raft/cell-groups/progress.md | 37 ++++++++++++++++++++++-- 2 files changed, 50 insertions(+), 15 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index ea601e4ac..6d470b5b2 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The executable design package is delivered. The resumed active goal is its -end-to-end implementation and hard cutover to the shared-group runtime. Existing -checkpoints do not mark the remaining tasks complete. +The current requested deliverable is the executable design and implementation +plan. The complete shared-group runtime and hard cutover remain the implementation +target; delivering this package does not mark those tasks complete. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,9 +68,9 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The user has resumed end-to-end implementation. The previous plan-only handoff -is historical; all remaining runtime and qualification tasks are in scope. -Package validation alone does not satisfy the active objective. +The latest user clarification selects an executable design and implementation +plan. This handoff preserves existing implementation work and its evidence; it +does not qualify or publish the unfinished runtime changes in this checkout. Continue CG-03 after reading [progress](progress.md) and inspecting the working diff. @@ -80,13 +80,15 @@ an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. -The next concrete boundary is physical group bootstrap: remove the cell binding -from replica/log identity, durable engine mappings, node inventory and permanent -member claims. Connect the verified birth to synced genesis and test restart, -unknown outcomes and refusal after a claimed voter disk is lost. A birth metadata -observation alone cannot authorize voter startup. A fabricated cell name is not -a substitute for a group identity. Then complete bounded redirects before CG-04 -connects enrollment to the shared runtime. +The next concrete boundary is physical group bootstrap. Uncommitted changes +already introduce physical replica handles, genesis and member claims, with +associated tests. Inspect and qualify that work before extending it. The reviewed +source baseline remains `d72ea068a527d2027e806ca78ab390256e8634fa`; current source +preflight refuses drift in the unfinished implementation. Do not refresh hashes +to bypass this refusal. Complete node inventory, startup authority and bounded +redirects before CG-04 connects enrollment to the shared runtime. A birth metadata +observation alone cannot authorize voter startup; a fabricated cell name cannot +substitute for a group identity. The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 94bfa4d93..253a21679 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,10 +3,43 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The user has resumed the full implementation goal following the -plan handoff below. Intermediate contracts do not qualify a shared runtime. +hard cutover. The latest user clarification selects the plan as this handoff +deliverable. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## Current plan handoff — 2026-10-29 + +The package supplies the target architecture, protocol state machines, dependency +ordered executor briefs, exact test commands and source-bound acceptance gates. +CG-01 and CG-02 retain their historical DONE evidence, CG-03 is IN_PROGRESS, and +CG-04 through CG-16 remain TODO. The final release gate remains closed. + +The branch includes freshly fetched `origin/main` at +`1be27d234d4674d4d014785c60cecbfe394bdab9`. The supplied TiKV checkout matches the +investigation pin `b3995dab28308f09cbb7c125bf32f789dbc203fe`. + +Uncommitted physical bootstrap, replica-handle and permanent member-claim changes +remain in this checkout, including two new Rust files and native test changes. +They are preserved for the next implementation executor and are not qualified by +this documentation handoff. Review their full diff and test results before +committing them or advancing the source baseline. Do not restart this work from +the earlier metadata-only checkpoint. + +Package structural validation passes: 16 tasks, 83 required cases, 14 invariants +and nine profiles. Working-tree source preflight correctly refuses drift in +`crates/cellar-logic/consensus.rs`; 34 of 37 validator tests pass, with the three +source-sensitive checks failing against that unfinished source. These failures +are not waived. The unchanged committed source is checked separately from the +draft runtime. + +Handoff verification: a temporary read-only export of committed source +`aead5cc04a9788ab2f2ee5f3fba3d6468fde5b75` passes the source preflight function and +all 37 plan-validator unit tests. This export excludes the draft runtime and +does not requalify historical DONE evidence. Working-tree structural validation, +CG-03 brief generation, repository documentation checks and `git diff --check` +pass. `release` exits nonzero for incomplete implementation evidence, as required. +The requested MR-13/MR-14 progress JSON files have no diff from `origin/main`. + ## Plan handoff at 462aa2eb — 2026-10-29 The package describes the complete replacement architecture, ordered implementation From 5d0d8e5ede3b4efedc211c40122a1a26b43968b4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:16:20 -0700 Subject: [PATCH 028/568] feat: bootstrap physical cell groups with permanent member claims --- crates/cellar-logic/cell_group.rs | 6 + crates/cellar-logic/consensus.rs | 6 +- crates/cellar/application/consensus/mod.rs | 4 +- crates/cellar/durability/raft/cache.rs | 74 ++++- crates/cellar/durability/raft/catalog.rs | 4 +- .../durability/raft/cell_groups/README.md | 47 ++- .../raft/cell_groups/directory/birth.rs | 3 +- .../raft/cell_groups/directory/bootstrap.rs | 159 ++++++++++ .../raft/cell_groups/directory/mod.rs | 2 + .../raft/cell_groups/directory/wire.rs | 1 + .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../durability/raft/cell_groups/retention.rs | 8 +- .../durability/raft/cell_groups/root.rs | 41 ++- crates/cellar/durability/raft/member_state.rs | 13 +- .../durability/raft/physical_inventory.rs | 105 +++++-- .../raft/physical_inventory_tests.rs | 35 +++ crates/cellar/durability/raft/records.rs | 93 +++++- crates/cellar/durability/raft/storage.rs | 297 +++++++++++++++--- .../raft/storage_cell_genesis_tests.rs | 223 +++++++++++++ .../durability/raft/storage_genesis_tests.rs | 23 +- .../tests/native_cell_groups_catalog.rs | 10 + .../tests/native_cell_groups_storage.rs | 6 +- .../tests/support/cell_group_directory.rs | 194 ++++++++++++ .../tests/support/cell_group_recovery.rs | 27 +- plans/multi-raft/cell-groups/tasks.json | 4 +- 25 files changed, 1235 insertions(+), 154 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs create mode 100644 crates/cellar/durability/raft/storage_cell_genesis_tests.rs diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs index d27803e2d..048c24214 100644 --- a/crates/cellar-logic/cell_group.rs +++ b/crates/cellar-logic/cell_group.rs @@ -87,6 +87,12 @@ impl From<&crate::consensus::GroupScope> for GroupHistory { } } +impl From<&GroupHistory> for GroupHistory { + fn from(history: &GroupHistory) -> Self { + *history + } +} + macro_rules! nonzero_counter { ($($name:ident),+ $(,)?) => {$ ( #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] diff --git a/crates/cellar-logic/consensus.rs b/crates/cellar-logic/consensus.rs index 6883648a6..ff12466c9 100644 --- a/crates/cellar-logic/consensus.rs +++ b/crates/cellar-logic/consensus.rs @@ -109,15 +109,15 @@ pub struct NodeIdentity { /// A process operating a particular durable member disk. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct ReplicaIdentity { - pub scope: GroupScope, +pub struct ReplicaIdentity { + pub scope: S, pub node: NodeId, pub member: MemberId, pub disk: DiskLineage, pub process: ProcessGeneration, } -impl ReplicaIdentity { +impl ReplicaIdentity { pub fn node_identity(&self) -> NodeIdentity { NodeIdentity { node: self.node, diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 23134b7c0..06137294e 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -202,7 +202,9 @@ pub mod verification { PreparedSnapshot, RootSnapshotSet, RootSnapshotSource, SnapshotCheckpoint, SnapshotDownload, SnapshotError, SnapshotLimits, SnapshotPackage, SnapshotReference, }; - pub use crate::raft::storage::{GroupBootstrap, LogError, ReplicaLog, SyncReceipt}; + pub use crate::raft::storage::{ + CellGroupBootstrap, GroupBootstrap, LogError, ReplicaLog, SyncReceipt, + }; pub use crate::raft::transport::{ Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, }; diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 9610c6d82..0c614e1d2 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -8,7 +8,8 @@ use super::storage::LogError; use crate::application::consensus::ImageCommitment; use ::raft::prelude::{ConfState, Entry, EntryType, HardState, Snapshot}; use ::raft::{GetEntriesContext, RaftState, Storage, StorageError}; -use cellar_logic::consensus::{LogIndex, LogPosition, ReplicaIdentity, Term}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{GroupScope, LogIndex, LogPosition, ReplicaIdentity, Term}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; use std::collections::BTreeMap; @@ -98,8 +99,8 @@ pub enum SnapshotChange { } #[derive(Clone)] -pub struct GroupWrite { - pub group: GroupLog, +pub struct GroupWrite { + pub group: GroupLog, pub entries: Vec, pub hard_state: Option, pub conf_state: Option, @@ -109,8 +110,8 @@ pub struct GroupWrite { pub snapshot: Option, } -impl GroupWrite { - pub fn new(group: &GroupLog) -> Self { +impl GroupWrite { + pub fn new(group: &GroupLog) -> Self { Self { group: group.clone(), entries: Vec::new(), @@ -163,21 +164,24 @@ impl GroupWrite { } } +/// Engine-owned storage handle. `GroupHistory` is the cell-group scope; +/// `GroupScope` remains only for the staged single-cell application adapter. +/// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] -pub struct GroupLog { +pub struct GroupLog { pub(crate) scratch_directory: std::path::PathBuf, pub(crate) snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, pub(crate) filesystem: Arc, pub(crate) key: u64, pub(crate) birth_digest: [u8; 32], - pub(crate) identity: ReplicaIdentity, + pub(crate) identity: ReplicaIdentity, pub(crate) owner: Arc<()>, pub(crate) closed: Arc, pub(crate) cache: Arc>, pub(crate) cache_revision: Arc, } -impl std::fmt::Debug for GroupLog { +impl std::fmt::Debug for GroupLog { fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { formatter .debug_struct("GroupLog") @@ -187,8 +191,8 @@ impl std::fmt::Debug for GroupLog { } } -impl GroupLog { - pub fn identity(&self) -> &ReplicaIdentity { +impl GroupLog { + pub fn identity(&self) -> &ReplicaIdentity { &self.identity } @@ -241,6 +245,41 @@ impl GroupLog { } } +/// One engine catalog, with explicit staged application domains. A caller must +/// select the right handle type; enumeration never silently skips another kind. +#[derive(Clone)] +pub(crate) enum StoredGroup { + Cell(GroupLog), + Cells(GroupLog), +} + +impl StoredGroup { + pub(crate) fn as_cell(&self) -> Result<&GroupLog, LogError> { + match self { + Self::Cell(group) => Ok(group), + Self::Cells(_) => Err(LogError::WrongIdentity), + } + } + + pub(crate) fn as_cells(&self) -> Result<&GroupLog, LogError> { + match self { + Self::Cells(group) => Ok(group), + Self::Cell(_) => Err(LogError::WrongIdentity), + } + } + + pub(crate) fn history(&self) -> GroupHistory { + match self { + Self::Cell(group) => GroupHistory::from(&group.identity.scope), + Self::Cells(group) => group.identity.scope, + } + } + + pub(crate) fn is_cells(&self) -> bool { + matches!(self, Self::Cells(_)) + } +} + fn raft_error(error: LogError) -> ::raft::Error { ::raft::Error::Store(StorageError::Other(Box::new(error))) } @@ -262,7 +301,7 @@ impl ConsensusStorage { } } -impl GroupLog { +impl GroupLog { pub(crate) fn consensus_storage(&self) -> ConsensusStorage { ConsensusStorage { cache: self.cache.clone(), @@ -276,7 +315,7 @@ impl GroupLog { } } -impl Storage for GroupLog { +impl Storage for GroupLog { fn initial_state(&self) -> ::raft::Result { self.checked_storage()?.initial_state() } @@ -496,7 +535,12 @@ impl CachedGroup { Ok(()) } - pub fn changed(&self, write: &GroupWrite, max_entries: usize) -> Result { + pub fn changed( + &self, + write: &GroupWrite, + max_entries: usize, + history: GroupHistory, + ) -> Result { let mut next = self.clone(); if (self.group_manifest.is_some() || write.group_manifest.is_some()) && (write.applied.is_some() || write.snapshot.is_some()) @@ -623,7 +667,7 @@ impl CachedGroup { .check() .map_err(|e| LogError::Invalid(e.to_string()))?; if !Arc::ptr_eq(&prepared.engine_owner, &write.group.owner) - || !marker.matches_group(&write.group.identity.scope, write.group.birth_digest) + || !marker.matches_group(&history, write.group.birth_digest) || marker.position().index < self.applied.position.index || (marker.position().index == self.applied.position.index && self.group_manifest.as_ref() != Some(marker)) @@ -698,8 +742,6 @@ impl CachedGroup { "applied configuration lacks ConfState".into(), )); } - next.membership - .validate_scope(&write.group.identity().scope)?; next.validate(max_entries) .map_err(|error| LogError::Invalid(error.to_string()))?; Ok(next) diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index ec3074cf3..de9d6083f 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -25,8 +25,8 @@ pub use births::BirthDisposition; #[path = "cell_groups/directory/mod.rs"] mod range_directory; pub use range_directory::{ - CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, DirectoryLimits, DirectoryMutation, - DirectoryPublication, DirectoryView, RangeDirectory, + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 5e67affe6..194b1d680 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -203,9 +203,9 @@ retrying. A retired or replaced range cannot materialize again. A birth observat is bound to its catalog owner; `require_current_group_birth` checks current route, reservation and selected immutable bytes, and refuses missing or changed state. -These methods expose no voter-start capability and write no permanent member disk -claim. The physical log still needs cell-independent genesis and recovery identity -before the new birth can be integrated with actual disk-claim admission. Managed +These metadata methods expose no voter-start capability and write no permanent +member disk claim. The bootstrap path below connects the verified birth to real +engine genesis and permanent claims. Managed namespaces refuse these unmanaged APIs until their rooted fleet authorization is connected. This staged boundary does not permit blank-disk recovery, replace current quorum membership, or complete CG-03 qualification. @@ -221,5 +221,42 @@ The staged single-cell host still stores its real cell binding as application adapter metadata. Its lookup, birth retries, pending slot accounting and physical inventory must reject another cell binding for the same physical history. This metadata is not part of the registry key and is removed with the remaining -single-cell runtime consumers. This foundation does not expose shared-group voter -startup or establish the permanent member claim required by the new birth API. +single-cell runtime consumers. Cell-group genesis and claims use the separate +physical scope described below. + +Cell-independent engine handles now use `GroupLog` and +`ReplicaIdentity`. `create_cell_group` atomically syncs the physical +mapping, initial membership, Raft genesis and an empty route-bound group manifest. +`persist_cells` uses the same owned worker, quotas, batch sync and publication +path as the staged adapter. The engine rejects mixed application domains; old +single-cell enumeration and inventory cannot silently omit cell-group histories. +The mapping's required application tag also refuses the earlier untagged adapter +record shape. No old disk layout is decoded or migrated. + +`RangeDirectory::bootstrap_group_member` admits an initial unmanaged member only +after checking its current birth, permanent node-storage pin, actual synced +genesis and an exact immutable member claim. Claims bind namespace, history, +birth, member, node, disk and engine instance. Lost confirmation returns no permit; +retry uses those same identities. A claim with missing local group state, or a +replacement engine behind the same node identity, refuses. Retired routes and +foreign nodes/catalog owners refuse. Managed namespaces remain closed until their +rooted fleet authorization is integrated. + +The resulting permit is catalog startup evidence, not complete image recovery, +quorum, or application-serving authority. A future shared runtime must verify the +selected manifest closure, revalidate the permit and obtain current-term quorum +before accepting application work. The production host is not yet converted. +Native storage recovery fixtures exercise both image representations through real +cell-independent genesis without a placeholder cell. + +`ReplicaLog::cell_group_inventory` reads every physical group under the engine +catalog lock and short cache reads. It includes retained histories regardless of +current membership, plus an optional initializing history. Pending identities +already in the catalog count once. The existing bounded node-inventory wire carries +these rows without allocating cell names. A domain-separated fixed-width digest +binds cluster, namespace, group and incarnation; `group_history` performs exact +lookup. Legacy cell-scoped lookup cannot match that digest. The node observation +must still match the complete occupied count before encoding or accepting a +response. Inventory is an occupancy observation, not quorum or release proof. +The generic physical capture includes shared histories, while the group-native +accessor refuses a legacy application engine. diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs index da3f74f6a..4a295d3d4 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/birth.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -50,6 +50,7 @@ impl CellGroupBirthRequest { /// Read-back of an immutable birth selected by an observed directory root. /// This grants neither membership nor permission to initialize a blank disk. +#[derive(Clone)] pub struct CellGroupBirth { request: CellGroupBirthRequest, owner: Arc<()>, @@ -74,7 +75,7 @@ impl CellGroupBirth { } impl RangeDirectory { - fn unmanaged_birth(&self) -> Result<(), CatalogError> { + pub(super) fn unmanaged_birth(&self) -> Result<(), CatalogError> { if self.namespace.config.managed_fleet.is_some() { return Err(CatalogError::ManagedAdmissionRequired); } diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs new file mode 100644 index 000000000..2247c3549 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -0,0 +1,159 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Initial member admission after real genesis and a permanent physical claim. + +use super::*; +use crate::raft::{ + cache::GroupLog, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::MemberId; +use serde::{Deserialize, Serialize}; + +/// Initial member startup permission, not quorum or application-serving authority. +/// The shared runtime must verify the complete selected image/manifest closure +/// and revalidate this permit before attachment. It must obtain current-term +/// quorum authority separately before accepting application work. +pub struct CellGroupBootstrapPermit { + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, +} + +impl CellGroupBootstrapPermit { + pub fn group(&self) -> &GroupLog { + &self.group + } + + pub fn route(&self) -> &GroupRoute { + self.birth.route() + } + + pub async fn revalidate(&self) -> Result<(), CatalogError> { + self.directory + .bootstrap_group_member(&self.birth, &self.log) + .await + .map(|_| ()) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Claim { + namespace: [u8; 32], + group: [u8; 16], + incarnation: [u8; 16], + birth: [u8; 32], + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +fn check_member(group: &GroupLog) -> Result<(), CatalogError> { + let cache = group.read().map_err(CatalogError::Log)?; + if !crate::raft::member_state::members(&cache.conf_state) + .contains(&group.identity().member.get()) + || !cache.membership.permits_group(group.identity()) + { + return Err(CatalogError::Conflict("group member is no longer admitted")); + } + Ok(()) +} + +impl RangeDirectory { + fn group_claim_key(&self, history: GroupHistory, member: MemberId) -> Path { + self.key(&format!( + "claims/{}/{}/{:016x}", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()), + member.get() + )) + } + + /// Initialize or resume an initial member using its exact immutable birth. + /// Unknown outcomes retain the same history, disk and engine identities. + /// Managed admission remains closed until rooted fleet authorization exists. + pub async fn bootstrap_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.require_current_group_birth(birth).await?; + let selected = birth + .voters() + .iter() + .chain(birth.learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + // This permanent node pin refuses a recreated engine even before a + // member claim has been written. Keep the existing pin protocol. + let _node = self.namespace.require_node_storage(log).await?; + let history = birth.route().history; + let expected = record::encode( + wire::GROUP_CLAIM, + &Claim { + namespace: self.namespace.root_digest, + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + birth: birth.route().birth_digest, + member: selected.member.get(), + node: selected.node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }, + wire::LIMIT, + )?; + let key = self.group_claim_key(history, selected.member); + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.as_ref().is_some_and(|(bytes, _)| bytes != &expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + let local = match log.cell_group(&history) { + Ok(group) => Some(group), + Err(LogError::NotFound) => None, + Err(error) => return Err(CatalogError::Log(error)), + }; + if remote.is_some() && local.is_none() { + return Err(CatalogError::LostStableState); + } + // The worker creates the complete empty root with genesis and only + // returns after actual engine sync. Existing mappings must match + // the original route, membership and birth, even after later apply. + let group = log + .create_cell_group(CellGroupBootstrap { + route: birth.route().clone(), + member: selected.member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }) + .await + .map_err(CatalogError::Log)?; + check_member(&group)?; + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, &expected, wire::LIMIT) + .await?; + if observed != expected { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + check_member(&group)?; + Ok(CellGroupBootstrapPermit { + group, + birth: birth.clone(), + directory: self.clone(), + log: log.clone(), + }) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index d62e55f8b..413c23440 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -18,11 +18,13 @@ use std::{ }; mod birth; +mod bootstrap; mod store; mod tree; mod wire; pub use birth::{CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec}; +pub use bootstrap::CellGroupBootstrapPermit; #[derive(Clone, Copy, Debug)] pub struct DirectoryLimits { diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 303a05bad..8c0ce8a0c 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -11,6 +11,7 @@ pub(super) const REQUEST: u8 = 85; // Fleet maintenance already uses 86 in the shared catalog envelope. pub(super) const GROUP_BIRTH: u8 = 186; +pub(super) const GROUP_CLAIM: u8 = 187; pub(super) const LIMIT: usize = 64 * 1024; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 3cee6218d..b9ead733b 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -18,8 +18,8 @@ pub use manifest::{ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; pub use crate::raft::catalog::{ - CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, DirectoryLimits, DirectoryMutation, - DirectoryPublication, DirectoryView, RangeDirectory, + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs index b2f696b9f..210913d56 100644 --- a/crates/cellar/durability/raft/cell_groups/retention.rs +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -76,9 +76,9 @@ impl RootRetentions { Ok(()) } - pub(super) fn recover( + pub(super) fn recover( &self, - group: &GroupLog, + group: &GroupLog, marker: &GroupManifest, pins: &ManifestPins, ) -> Result<(), ManifestError> { @@ -112,9 +112,9 @@ impl RootRetentions { Ok(()) } - pub(super) fn begin( + pub(super) fn begin( self: &Arc, - group: &GroupLog, + group: &GroupLog, prepared: &PreparedGroupManifest, previous: Option<&GroupManifest>, ) -> Result { diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index a6121f44b..f91451824 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -57,15 +57,25 @@ impl GroupManifest { Ok(()) } - pub(crate) fn matches_group(&self, scope: &GroupScope, birth: [u8; 32]) -> bool { - self.route.history - == GroupHistory { - cluster: scope.cluster, - namespace: scope.namespace, - group: scope.group, - incarnation: scope.incarnation, - } - && self.route.birth_digest == birth + pub(crate) fn matches_group(&self, scope: &S, birth: [u8; 32]) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { + self.route.history == GroupHistory::from(scope) && self.route.birth_digest == birth + } + + pub(crate) fn genesis(route: GroupRoute) -> Result { + let marker = Self { + route, + position: LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }, + membership_revision: 1, + root: ManifestRoot::default(), + }; + marker.check()?; + Ok(marker) } } @@ -103,9 +113,9 @@ impl PreparedGroupManifest { self.images.reserved_bytes() } - pub(crate) fn begin_publication( + pub(crate) fn begin_publication( &self, - group: &crate::raft::cache::GroupLog, + group: &crate::raft::cache::GroupLog, previous: Option<&GroupManifest>, ) -> Result { self.retention.begin(group, self, previous) @@ -116,15 +126,18 @@ impl ManifestTree { /// Full closure verification for bootstrap, recovery and checkpoint install. /// Ordinary cell mutations use `prepare_update` instead. /// The callback returns actual sealed images, never merely accepted markers. - pub fn prepare_complete( + pub fn prepare_complete( &self, - group: &crate::raft::cache::GroupLog, + group: &crate::raft::cache::GroupLog, store: &mut ManifestPageStore, root: ManifestRoot, position: LogPosition, membership_revision: u64, mut image: impl FnMut(&CellManifestLeaf) -> Result, - ) -> Result { + ) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { let marker = GroupManifest { route: self.route.clone(), position, diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index f8c58eee3..8de797d91 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -129,9 +129,20 @@ impl GroupMembership { if self.validate_scope(&replica.scope).is_err() { return false; } + self.permits_binding(replica) + } + + pub(crate) fn permits_group( + &self, + replica: &cellar_logic::consensus::ReplicaIdentity, + ) -> bool { + self.fleet.is_none() && self.permits_binding(replica) + } + + fn permits_binding(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { self.bindings .iter() - .find(|b| b.member == replica.member.get()) + .find(|binding| binding.member == replica.member.get()) .map_or_else( || self.initial.contains(&replica.member.get()), |binding| { diff --git a/crates/cellar/durability/raft/physical_inventory.rs b/crates/cellar/durability/raft/physical_inventory.rs index 4aab1859b..737a4478d 100644 --- a/crates/cellar/durability/raft/physical_inventory.rs +++ b/crates/cellar/durability/raft/physical_inventory.rs @@ -2,7 +2,7 @@ //! Compact complete physical occupancy, including unattached/retired histories. //! Immutable identities need no Raft quorum or eager application registration. -use super::cache::GroupLog; +use super::cache::{GroupLog, StoredGroup}; use super::host::RaftHostObservation; use super::storage::LogError; use super::transport::TransportError; @@ -111,6 +111,16 @@ impl RaftPhysicalInventory { .map(|index| &self.histories[index])) } + /// Look up a cell-group history without inventing an application cell. + /// Its digest domain is distinct from the temporary single-cell adapter. + pub fn group_history(&self, history: &GroupHistory) -> Option<&RaftPhysicalHistory> { + let digest = group_digest(history); + self.histories + .binary_search_by_key(&digest, |h| h.scope) + .ok() + .map(|index| &self.histories[index]) + } + pub(crate) fn validate(&self, view: &RaftHostObservation) -> Result<(), TransportError> { super::node_inspection_wire::validate(view)?; if self.histories.len() > MAX_GROUPS @@ -148,8 +158,29 @@ pub(crate) fn scope_digest(scope: &GroupScope) -> Result<[u8; 32], LogError> { Ok(Sha256::digest(bytes).into()) } +fn group_digest(history: &GroupHistory) -> [u8; 32] { + let mut digest = Sha256::new(); + digest.update(b"cellar-cell-group-physical-history-1\0"); + digest.update(history.cluster.bytes()); + digest.update(history.namespace.bytes()); + digest.update(history.group.bytes()); + digest.update(history.incarnation.bytes()); + digest.finalize().into() +} + +pub(crate) fn capture_cells( + groups: &BTreeMap, + pending: Option<&GroupHistory>, +) -> Result { + // A group-native owner cannot attach to a legacy application engine. + if groups.values().any(|group| !group.is_cells()) { + return Err(LogError::WrongIdentity); + } + capture_histories(groups, pending.map(group_digest)) +} + pub(crate) fn capture( - groups: &BTreeMap, + groups: &BTreeMap, pending: Option<&GroupScope>, ) -> Result { if groups.len() > MAX_GROUPS { @@ -157,37 +188,30 @@ pub(crate) fn capture( } if let Some(scope) = pending { scope.validate().map_err(|_| LogError::WrongIdentity)?; - if groups - .get(&GroupHistory::from(scope)) - .is_some_and(|group| &group.identity().scope != scope) - { - return Err(LogError::WrongIdentity); + if let Some(group) = groups.get(&GroupHistory::from(scope)) { + if &group.as_cell()?.identity().scope != scope { + return Err(LogError::WrongIdentity); + } } } + capture_histories(groups, pending.map(scope_digest).transpose()?) +} + +fn capture_histories( + groups: &BTreeMap, + initializing: Option<[u8; 32]>, +) -> Result { + if groups.len() > MAX_GROUPS { + return Err(LogError::Overloaded); + } let mut histories = Vec::with_capacity(groups.len()); for (history, group) in groups { - let scope = &group.identity().scope; - if history != &GroupHistory::from(scope) || group.birth_digest == [0; 32] { + if history != &group.history() { return Err(LogError::WrongIdentity); } - let cache = group.read()?; - let completed_sequence = cache - .membership - .fleet_status() - .filter(|status| { - status.phase == Some(super::member_state::FleetPermitPhase::Complete) - && status - .permit - .as_ref() - .is_some_and(|permit| permit.sequence() == status.consumed_sequence) - }) - .map_or(0, |status| status.consumed_sequence); - histories.push(RaftPhysicalHistory { - scope: scope_digest(scope)?, - birth: group.birth_digest, - member: group.identity().member.get().to_be_bytes(), - completed_sequence: completed_sequence.to_be_bytes(), - in_configuration: contains_member(&cache.conf_state, group.identity().member), + histories.push(match group { + StoredGroup::Cell(group) => capture_row(group, scope_digest(&group.identity().scope)?)?, + StoredGroup::Cells(group) => capture_row(group, group_digest(history))?, }); } histories.sort_by_key(|h| h.scope); @@ -198,7 +222,32 @@ pub(crate) fn capture( } Ok(RaftPhysicalInventory { histories, - initializing: pending.map(scope_digest).transpose()?, + initializing, + }) +} + +fn capture_row(group: &GroupLog, scope: [u8; 32]) -> Result { + if group.birth_digest == [0; 32] { + return Err(LogError::WrongIdentity); + } + let cache = group.read()?; + let completed_sequence = cache + .membership + .fleet_status() + .filter(|status| { + status.phase == Some(super::member_state::FleetPermitPhase::Complete) + && status + .permit + .as_ref() + .is_some_and(|permit| permit.sequence() == status.consumed_sequence) + }) + .map_or(0, |status| status.consumed_sequence); + Ok(RaftPhysicalHistory { + scope, + birth: group.birth_digest, + member: group.identity().member.get().to_be_bytes(), + completed_sequence: completed_sequence.to_be_bytes(), + in_configuration: contains_member(&cache.conf_state, group.identity().member), }) } diff --git a/crates/cellar/durability/raft/physical_inventory_tests.rs b/crates/cellar/durability/raft/physical_inventory_tests.rs index d64c62fea..32139017d 100644 --- a/crates/cellar/durability/raft/physical_inventory_tests.rs +++ b/crates/cellar/durability/raft/physical_inventory_tests.rs @@ -183,3 +183,38 @@ fn physical_inventory_role_includes_every_joint_configuration_set() { member )); } + +#[test] +fn physical_inventory_group_history_is_canonical_and_domain_separated() { + let legacy = scope(); + let history = GroupHistory::from(&legacy); + let digest = group_digest(&history); + // Independent SHA-256 vector over the fixed domain and four 16-byte IDs. + assert_eq!( + hex::encode(digest), + "08040247cafda32726f8585b91312bb8d84c904fae55e05fa22009b1b4a276f6" + ); + assert_ne!(digest, scope_digest(&legacy).unwrap()); + let inventory = RaftPhysicalInventory { + histories: vec![RaftPhysicalHistory { + scope: digest, + birth: [8; 32], + member: 3u64.to_be_bytes(), + completed_sequence: 0u64.to_be_bytes(), + in_configuration: false, + }], + initializing: None, + }; + assert_eq!(inventory.group_history(&history).unwrap().member().get(), 3); + assert!(inventory.history(&legacy).unwrap().is_none()); + for field in 0..4 { + let mut changed = history; + match field { + 0 => changed.cluster = ClusterId::new([9; 16]).unwrap(), + 1 => changed.namespace = NamespaceId::new([9; 16]).unwrap(), + 2 => changed.group = GroupId::new([9; 16]).unwrap(), + _ => changed.incarnation = IncarnationId::new([9; 16]).unwrap(), + } + assert!(inventory.group_history(&changed).is_none()); + } +} diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index 502432ada..692d24b2b 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -3,7 +3,8 @@ //! Marker-1 local metadata, atomically stored with its Raft log changes. use super::cache::{CachedGroup, CommittedImage, DurableApply}; -use super::storage::{GroupBootstrap, LogError}; +use super::cell_groups::GroupManifest; +use super::storage::{CellGroupBootstrap, GroupBootstrap, LogError}; use ::raft::prelude::{ConfState, HardState, Snapshot}; use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{ @@ -62,9 +63,7 @@ pub(crate) struct MappingRecord { pub key: u64, domain: MappingDomain, history: PhysicalHistory, - // Temporary application adapter metadata, never the physical registry key. - // Removed with the remaining single-cell host consumers at public cutover. - application_cell: String, + application: MappingApplication, pub member: u64, pub birth_digest: [u8; 32], pub initial_conf: Vec, @@ -77,7 +76,7 @@ impl MappingRecord { key, domain: MappingDomain::Group, history: PhysicalHistory::from(GroupHistory::from(&bootstrap.scope)), - application_cell: bootstrap.scope.cell.clone(), + application: MappingApplication::SingleCell(bootstrap.scope.cell.clone()), member: bootstrap.member.get(), birth_digest: bootstrap.birth_digest, initial_conf: conf.write_to_bytes().map_err(corrupt)?, @@ -85,18 +84,80 @@ impl MappingRecord { } pub fn application_scope(&self) -> Result { - let history = self.history.decode()?; + let cell = match &self.application { + MappingApplication::SingleCell(cell) => cell.clone(), + MappingApplication::CellGroup(_) => return Err(LogError::WrongIdentity), + }; + let history = self.history()?; let scope = GroupScope { cluster: history.cluster, namespace: history.namespace, group: history.group, incarnation: history.incarnation, - cell: self.application_cell.clone(), + cell, }; scope.validate().map_err(corrupt)?; Ok(scope) } + pub fn new_cells( + key: u64, + bootstrap: &CellGroupBootstrap, + conf: &ConfState, + ) -> Result { + let marker = GroupManifest::genesis(bootstrap.route.clone()).map_err(corrupt)?; + Ok(Self { + format: 1, + key, + domain: MappingDomain::Group, + history: PhysicalHistory::from(bootstrap.route.history), + application: MappingApplication::CellGroup(Box::new(marker)), + member: bootstrap.member.get(), + birth_digest: bootstrap.route.birth_digest, + initial_conf: conf.write_to_bytes().map_err(corrupt)?, + }) + } + + pub fn history(&self) -> Result { + self.history.decode() + } + + pub fn is_cells(&self) -> bool { + matches!(&self.application, MappingApplication::CellGroup(_)) + } + + pub fn validate_cache(&self, cache: &CachedGroup) -> Result<(), LogError> { + if self.format != 1 { + return Err(corrupt("unsupported group marker")); + } + let history = self.history()?; + if let Some(marker) = &cache.group_manifest { + if !marker.matches_group(&history, self.birth_digest) { + return Err(LogError::WrongIdentity); + } + } + match &self.application { + MappingApplication::SingleCell(_) => { + cache.membership.validate_scope(&self.application_scope()?) + } + MappingApplication::CellGroup(genesis) => { + let expected = GroupManifest::genesis(genesis.route().clone()).map_err(corrupt)?; + if genesis.as_ref() != &expected + || !genesis.matches_group(&history, self.birth_digest) + || cache + .group_manifest + .as_ref() + .is_none_or(|root| root.route() != genesis.route()) + || cache.applied.image.is_some() + || cache.membership.fleet_status().is_some() + { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + } + } + pub fn catalog_key(&self) -> Result, LogError> { let scope = self.history.decode()?; let mut key = b"g:".to_vec(); @@ -106,6 +167,15 @@ impl MappingRecord { } } +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "domain", content = "state", deny_unknown_fields)] +enum MappingApplication { + #[serde(rename = "single-cell-1")] + SingleCell(String), + #[serde(rename = "cell-group-1")] + CellGroup(Box), +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] enum MappingDomain { #[serde(rename = "cellar-group-mapping-1")] @@ -170,6 +240,7 @@ pub(crate) enum ApplicationRecord { impl GroupRecord { pub fn from_cache(mapping: MappingRecord, cache: &CachedGroup) -> Result { + mapping.validate_cache(cache)?; Ok(Self { mapping, hard_state: cache.hard_state.write_to_bytes().map_err(corrupt)?, @@ -193,14 +264,14 @@ impl GroupRecord { let (image, group_manifest) = match &self.application { ApplicationRecord::Image(image) => (image.clone(), None), ApplicationRecord::Group(marker) => { - let scope = self.mapping.application_scope()?; + let scope = self.mapping.history()?; if !marker.matches_group(&scope, self.mapping.birth_digest) { return Err(LogError::WrongIdentity); } (None, Some(marker.as_ref().clone())) } }; - Ok(CachedGroup { + let cache = CachedGroup { hard_state: HardState::parse_from_bytes(&self.hard_state).map_err(corrupt)?, conf_state: ConfState::parse_from_bytes(&self.conf_state).map_err(corrupt)?, membership: self.membership.clone(), @@ -215,7 +286,9 @@ impl GroupRecord { }, image, }, - }) + }; + self.mapping.validate_cache(&cache)?; + Ok(cache) } } diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index e26530017..b4acec33a 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -2,7 +2,9 @@ //! One domain-owned engine worker. Every successful mutation has crossed fsync. -use super::cache::{genesis, validate_conf, CachedGroup, GroupLog, GroupWrite, SnapshotChange}; +use super::cache::{ + genesis, validate_conf, CachedGroup, GroupLog, GroupWrite, SnapshotChange, StoredGroup, +}; use super::filesystem::EngineFileSystem; use super::fleet_stop::{Guard, LifecycleRecord, LifecycleUpdate}; use super::options::{LogOpenMode, ReplicaLogOptions}; @@ -13,7 +15,7 @@ use super::records::{ self, GroupRecord, MappingRecord, NodeRecord, RootRecord, GROUP_KEY, ROOT_KEY, }; use ::raft::prelude::{ConfState, Entry}; -use cellar_logic::cell_group::GroupHistory; +use cellar_logic::cell_group::{GroupHistory, GroupRoute}; use cellar_logic::consensus::{GroupScope, MemberId, OperationId, ReplicaIdentity}; use cellar_ltx::{FileSystem, HostFileLock}; use protobuf::Message; @@ -25,6 +27,10 @@ use std::sync::{mpsc, Arc, Mutex, RwLock}; use std::time::Duration; use tokio::sync::{oneshot, Notify}; +#[cfg(test)] +#[path = "storage_cell_genesis_tests.rs"] +mod cell_genesis_tests; + #[cfg(test)] #[path = "storage_genesis_tests.rs"] mod genesis_tests; @@ -97,30 +103,55 @@ impl GroupBootstrap { self.scope .validate() .map_err(|error| LogError::Invalid(error.to_string()))?; - let mut conf = ConfState::default(); - conf.set_voters(self.voters.iter().map(|id| id.get()).collect()); - conf.set_learners(self.learners.iter().map(|id| id.get()).collect()); - validate_conf(&conf).map_err(|error| LogError::Invalid(error.to_string()))?; - super::member_state::GroupMembership::initial(&conf) - .validate(&conf, 1) + initial_configuration(self.member, &self.voters, &self.learners) + } +} + +/// Physical genesis for a whole cell range. Construction supplies no membership +/// claim or serving authority; those require the catalog bootstrap protocol. +#[derive(Clone, Debug)] +pub struct CellGroupBootstrap { + pub route: GroupRoute, + pub member: MemberId, + pub voters: Vec, + pub learners: Vec, +} + +impl CellGroupBootstrap { + fn configuration(&self) -> Result { + super::cell_groups::GroupManifest::genesis(self.route.clone()) .map_err(|error| LogError::Invalid(error.to_string()))?; - if !conf.voters.contains(&self.member.get()) && !conf.learners.contains(&self.member.get()) - { - return Err(LogError::WrongIdentity); - } - Ok(conf) + initial_configuration(self.member, &self.voters, &self.learners) } } +fn initial_configuration( + member: MemberId, + voters: &[MemberId], + learners: &[MemberId], +) -> Result { + let mut conf = ConfState::default(); + conf.set_voters(voters.iter().map(|id| id.get()).collect()); + conf.set_learners(learners.iter().map(|id| id.get()).collect()); + validate_conf(&conf).map_err(|error| LogError::Invalid(error.to_string()))?; + super::member_state::GroupMembership::initial(&conf) + .validate(&conf, 1) + .map_err(|error| LogError::Invalid(error.to_string()))?; + if !conf.voters.contains(&member.get()) && !conf.learners.contains(&member.get()) { + return Err(LogError::WrongIdentity); + } + Ok(conf) +} + /// Identity-bound observation of one successfully synced batch. #[derive(Clone, Debug)] -pub struct SyncReceipt { +pub struct SyncReceipt { pub operation: OperationId, pub sequence: u64, - pub groups: Vec, + pub groups: Vec>, } -type Catalog = Arc>>; +type Catalog = Arc>>; type Reply = oneshot::Sender>; #[derive(Clone)] @@ -213,8 +244,14 @@ enum Work { Option>, Reply, ), + CreateCells(CellGroupBootstrap, Reply>), Seeded(Box, Reply), Persist(OperationId, Vec, Reply), + PersistCells( + OperationId, + Vec>, + Reply>, + ), Purge(Reply<()>), } @@ -447,10 +484,12 @@ impl ReplicaLog { Some(scope) => { scope.validate().map_err(|_| LogError::WrongIdentity)?; match groups.get(&GroupHistory::from(scope)) { - Some(group) if &group.identity.scope != scope => { - return Err(LogError::WrongIdentity); + Some(group) => { + if &group.as_cell()?.identity.scope != scope { + return Err(LogError::WrongIdentity); + } + false } - Some(_) => false, None => true, } } @@ -464,14 +503,45 @@ impl ReplicaLog { pub fn groups(&self) -> Result, LogError> { self.admit()?; - Ok(self - .inner + self.inner + .groups + .read() + .map_err(|_| LogError::Closed)? + .values() + .map(|group| group.as_cell().cloned()) + .collect() + } + + pub fn cell_groups(&self) -> Result>, LogError> { + self.admit()?; + self.inner .groups .read() .map_err(|_| LogError::Closed)? .values() + .map(|group| group.as_cells().cloned()) + .collect() + } + + pub fn cell_group(&self, history: &GroupHistory) -> Result, LogError> { + self.admit()?; + self.inner + .groups + .read() + .map_err(|_| LogError::Closed)? + .get(history) + .ok_or(LogError::NotFound)? + .as_cells() .cloned() - .collect()) + } + + pub async fn create_cell_group( + &self, + bootstrap: CellGroupBootstrap, + ) -> Result, LogError> { + bootstrap.configuration()?; + self.request(16 * 1024, |reply| Work::CreateCells(bootstrap, reply)) + .await } /// Read occupied identities and locally durable completed membership under @@ -486,13 +556,26 @@ impl ReplicaLog { super::physical_inventory::capture(&groups, pending) } + /// Complete physical occupancy for a cell-group engine, including retained + /// histories and an optional initializing group. This is not serving or + /// membership authority. Legacy application engines refuse this accessor. + pub fn cell_group_inventory( + &self, + pending: Option<&GroupHistory>, + ) -> Result { + self.admit()?; + let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; + super::physical_inventory::capture_cells(&groups, pending) + } + pub fn group(&self, scope: &GroupScope) -> Result { self.admit()?; scope.validate().map_err(|_| LogError::WrongIdentity)?; let groups = self.inner.groups.read().map_err(|_| LogError::Closed)?; let group = groups .get(&GroupHistory::from(scope)) - .ok_or(LogError::NotFound)?; + .ok_or(LogError::NotFound)? + .as_cell()?; if &group.identity.scope != scope { return Err(LogError::WrongIdentity); } @@ -532,16 +615,36 @@ impl ReplicaLog { .await } + pub async fn persist_cells( + &self, + operation: OperationId, + writes: Vec>, + ) -> Result, LogError> { + let bytes = self.write_bytes(operation, &writes)?; + self.request(bytes, |reply| Work::PersistCells(operation, writes, reply)) + .await + } + pub async fn persist( &self, operation: OperationId, writes: Vec, ) -> Result { + let bytes = self.write_bytes(operation, &writes)?; + self.request(bytes, |reply| Work::Persist(operation, writes, reply)) + .await + } + + fn write_bytes( + &self, + operation: OperationId, + writes: &[GroupWrite], + ) -> Result { if operation.get() == 0 { return Err(LogError::Invalid("zero persistence operation".into())); } let mut bytes = 1024_usize; - for write in &writes { + for write in writes { bytes = bytes .checked_add(write.estimated_bytes()?) .ok_or(LogError::Overloaded)?; @@ -549,8 +652,7 @@ impl ReplicaLog { if bytes > self.inner.options.max_batch_bytes { return Err(LogError::Overloaded); } - self.request(bytes, |reply| Work::Persist(operation, writes, reply)) - .await + Ok(bytes) } /// Reclaim obsolete shared engine files, retaining every group's live log, @@ -836,7 +938,14 @@ impl Worker { return Err(error); } let mut known = BTreeSet::from([0]); + let mut application = None; for mapping in mappings { + if application + .replace(mapping.is_cells()) + .is_some_and(|kind| kind != mapping.is_cells()) + { + return Err(LogError::Corrupt("mixed engine application domains".into())); + } if mapping.key == 0 || mapping.key >= self.root.next_key || !known.insert(mapping.key) { return Err(LogError::Corrupt( "reused or unallocated numeric group key".into(), @@ -891,7 +1000,7 @@ impl Worker { .groups .write() .map_err(|_| LogError::Closed)? - .insert(GroupHistory::from(&handle.identity.scope), handle) + .insert(handle.history(), handle) .is_some() { return Err(LogError::Corrupt("duplicate logical group mapping".into())); @@ -909,7 +1018,7 @@ impl Worker { Ok(()) } - fn handle(&self, mapping: &MappingRecord, cache: CachedGroup) -> Result { + fn handle(&self, mapping: &MappingRecord, cache: CachedGroup) -> Result { let initial = ConfState::parse_from_bytes(&mapping.initial_conf).map_err(records::corrupt)?; validate_conf(&initial)?; @@ -924,9 +1033,28 @@ impl Worker { { return Err(LogError::Corrupt("member binding birth mismatch".into())); } - let scope = mapping.application_scope()?; - scope.validate().map_err(records::corrupt)?; - cache.membership.validate_scope(&scope)?; + mapping.validate_cache(&cache)?; + if mapping.is_cells() { + Ok(StoredGroup::Cells(self.scoped_handle( + mapping, + cache, + mapping.history()?, + )?)) + } else { + Ok(StoredGroup::Cell(self.scoped_handle( + mapping, + cache, + mapping.application_scope()?, + )?)) + } + } + + fn scoped_handle( + &self, + mapping: &MappingRecord, + cache: CachedGroup, + scope: S, + ) -> Result, LogError> { Ok(GroupLog { scratch_directory: self.options.directory.clone(), snapshot_scratch: self.snapshot_scratch.clone(), @@ -1006,6 +1134,7 @@ impl Worker { .map_err(|_| LogError::Closed)? .get(&GroupHistory::from(&bootstrap.scope)) { + let group = group.as_cell()?; let mapping = self .mappings .get(&group.key) @@ -1060,7 +1189,7 @@ impl Worker { .map_err(|_| LogError::Closed)? .get(&GroupHistory::from(&identity.scope)) { - return Err(if group.identity.scope == identity.scope { + return Err(if group.as_cell()?.identity.scope == identity.scope { LogError::AlreadyExists } else { LogError::WrongIdentity @@ -1079,22 +1208,66 @@ impl Worker { cache: CachedGroup, seed: Option<&super::snapshot::PreparedSeed>, ) -> Result { - if self.mappings.len() >= self.options.max_groups { - return Err(LogError::Overloaded); - } - // A group identifier cannot be reused for a different logical history. - if self + let mapping = MappingRecord::new(self.root.next_key, &bootstrap, &conf)?; + self.insert_mapping(mapping, cache, seed)? + .as_cell() + .cloned() + } + + fn create_cells( + &mut self, + bootstrap: CellGroupBootstrap, + ) -> Result, LogError> { + let conf = bootstrap.configuration()?; + if let Some(group) = self .groups .read() .map_err(|_| LogError::Closed)? - .keys() - .any(|scope| scope.group == bootstrap.scope.group) + .get(&bootstrap.route.history) { - return Err(LogError::WrongIdentity); + let group = group.as_cells()?; + let expected = MappingRecord::new_cells(group.key, &bootstrap, &conf)?; + if self.mappings.get(&group.key) != Some(&expected) { + return Err(LogError::WrongIdentity); + } + return Ok(group.clone()); + } + let mapping = MappingRecord::new_cells(self.root.next_key, &bootstrap, &conf)?; + let mut cache = genesis(conf); + cache.group_manifest = Some( + super::cell_groups::GroupManifest::genesis(bootstrap.route) + .map_err(|error| LogError::Invalid(error.to_string()))?, + ); + self.insert_mapping(mapping, cache, None)? + .as_cells() + .cloned() + } + + fn insert_mapping( + &mut self, + mapping: MappingRecord, + cache: CachedGroup, + seed: Option<&super::snapshot::PreparedSeed>, + ) -> Result { + if self.mappings.len() >= self.options.max_groups { + return Err(LogError::Overloaded); + } + let history = mapping.history()?; + { + let groups = self.groups.read().map_err(|_| LogError::Closed)?; + if groups.keys().any(|h| h.group == history.group) + || groups + .values() + .any(|group| group.is_cells() != mapping.is_cells()) + { + return Err(LogError::WrongIdentity); + } } let key = self.root.next_key; + if mapping.key != key { + return Err(LogError::WrongIdentity); + } let next_key = key.checked_add(1).ok_or(LogError::Overloaded)?; - let mapping = MappingRecord::new(key, &bootstrap, &conf)?; cache.validate(self.options.max_entries_per_group)?; let bytes = cache.bytes(); if self @@ -1132,18 +1305,18 @@ impl Worker { self.groups .write() .map_err(|_| LogError::Closed)? - .insert(GroupHistory::from(&bootstrap.scope), handle.clone()); + .insert(history, handle.clone()); self.mappings.insert(key, mapping); self.cache_bytes += bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); Ok(handle) } - fn persist( + fn persist( &mut self, operation: OperationId, - writes: Vec, - ) -> Result { + writes: Vec>, + ) -> Result, LogError> { let mut unique = BTreeSet::new(); let mut candidates = Vec::with_capacity(writes.len()); let mut next_bytes = self.cache_bytes; @@ -1152,8 +1325,16 @@ impl Worker { if !Arc::ptr_eq(&write.group.owner, &self.owner) || !unique.insert(write.group.key) { return Err(LogError::WrongIdentity); } + let mapping = self + .mappings + .get(&write.group.key) + .ok_or(LogError::WrongIdentity)?; let old = write.group.read()?; - let next = old.changed(write, self.options.max_entries_per_group)?; + let next = old.changed( + write, + self.options.max_entries_per_group, + mapping.history()?, + )?; next_bytes = next_bytes .checked_sub(old.bytes()) .and_then(|size| size.checked_add(next.bytes())) @@ -1162,10 +1343,6 @@ impl Worker { return Err(LogError::Overloaded); } drop(old); - let mapping = self - .mappings - .get(&write.group.key) - .ok_or(LogError::WrongIdentity)?; if let Some(change) = &write.snapshot { match change { SnapshotChange::Install { .. } => { @@ -1343,6 +1520,26 @@ impl Worker { let _ = reply.send(result); fatal } + Work::CreateCells(bootstrap, reply) => { + let result = self.create_cells(bootstrap); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } + Work::PersistCells(operation, writes, reply) => { + let timer = self.performance.timer(Measured::EnginePersist); + let result = self.persist(operation, writes); + timer.finish(result.is_ok()); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } Work::Seeded(seed, reply) => { let result = self.seeded(*seed); let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs new file mode 100644 index 000000000..d8cb23558 --- /dev/null +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -0,0 +1,223 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::asyncrt::ExecutionDomain; +use ::raft::Storage as _; +use cellar_logic::{ + cell_group::{CellRange, RouteEpoch}, + consensus::*, +}; + +fn bootstrap(id: u8) -> CellGroupBootstrap { + CellGroupBootstrap { + route: GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([id; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [9; 32], + schema_digest: [8; 32], + }, + member: MemberId::new(1).unwrap(), + voters: (1..=3).map(|n| MemberId::new(n).unwrap()).collect(), + learners: vec![], + } +} + +fn options(root: &std::path::Path) -> ReplicaLogOptions { + ReplicaLogOptions::new( + root.join("log"), + NodeId::new([1; 16]).unwrap(), + DiskLineage::new([2; 16]).unwrap(), + ProcessGeneration::new([3; 16]).unwrap(), + LogOpenMode::Create, + ) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cell_group_genesis_restores_physical_handles_and_exact_birth() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut opts = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + assert_eq!(marker.route(), &initial.route); + assert_eq!( + marker.position(), + LogPosition { + index: LogIndex::new(1), + term: Term::new(1) + } + ); + assert_eq!(marker.root().cells(), 0); + assert_eq!(group.identity().scope, initial.route.history); + assert!(group.applied().is_err()); + assert_eq!( + log.create_cell_group(initial.clone()).await?.numeric_key(), + group.numeric_key() + ); + for field in 0..4 { + let mut wrong = initial.clone(); + match field { + 0 => wrong.route.schema_digest = [7; 32], + 1 => wrong.route.epoch = RouteEpoch::new(2)?, + 2 => wrong.member = MemberId::new(2)?, + _ => wrong.voters.pop().map(|_| ()).unwrap(), + } + assert!(matches!( + log.create_cell_group(wrong).await, + Err(LogError::WrongIdentity) + )); + } + log.create_cell_group(bootstrap(7)).await?; + assert_eq!(log.cell_groups()?.len(), 2); + let inventory = log.cell_group_inventory(None)?; + assert_eq!(inventory.histories().len(), 2); + assert_eq!(inventory, log.physical_inventory(None)?); + let row = inventory.group_history(&initial.route.history).unwrap(); + assert_eq!(row.member(), initial.member); + assert_eq!(row.birth_digest(), initial.route.birth_digest); + assert!(row.in_configuration()); + assert_eq!(row.completed_sequence(), 0); + assert!(inventory + .group_history(&bootstrap(8).route.history) + .is_none()); + // Both a pending existing slot and a genuinely new slot must survive the + // bounded node wire representation, with the correct occupied count. + for (pending, count) in [(initial.route.history, 2), (bootstrap(8).route.history, 3)] { + let captured = log.cell_group_inventory(Some(&pending))?; + assert_eq!(captured.histories(), inventory.histories()); + let mut view = crate::raft::node_inspection_wire::tests::view(); + view.node.groups = count; + view.node.reserved_bytes = u64::from(count) * view.node.profile.bytes_per_group; + view.unattached_engine_histories = 2; + let encoded = crate::raft::physical_inventory::encode(&view, &captured)?; + let (_, decoded) = crate::raft::physical_inventory::decode(&encoded)?; + assert_eq!(*decoded, captured); + view.node.groups -= 1; + view.node.reserved_bytes -= view.node.profile.bytes_per_group; + assert!(crate::raft::physical_inventory::encode(&view, &captured).is_err()); + } + assert!(matches!(log.groups(), Err(LogError::WrongIdentity))); + let old = GroupBootstrap { + scope: GroupScope { + cluster: initial.route.history.cluster, + namespace: initial.route.history.namespace, + group: GroupId::new([10; 16])?, + incarnation: initial.route.history.incarnation, + cell: "actual-old-cell".into(), + }, + member: initial.member, + birth_digest: initial.route.birth_digest, + voters: initial.voters.clone(), + learners: vec![], + }; + assert!(matches!( + log.create_group(old).await, + Err(LogError::WrongIdentity) + )); + let mut write = GroupWrite::new(&group); + write.entries.push(Entry { + index: 2, + term: 2, + data: b"control-prefix".to_vec().into(), + ..Default::default() + }); + let mut hard = group.initial_state()?.hard_state; + hard.term = 2; + hard.commit = 2; + hard.vote = 1; + write.hard_state = Some(hard.clone()); + let receipt = log.persist_cells(OperationId::new(1), vec![write]).await?; + assert_eq!(receipt.groups, vec![group.identity().clone()]); + let instance = log.storage_instance()?; + log.close().await?; + opts.mode = LogOpenMode::Resume; + opts.process = ProcessGeneration::new([6; 16])?; + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!(log.storage_instance()?, instance); + let recovered = log.create_cell_group(initial.clone()).await?; + assert_eq!(recovered.identity().scope, initial.route.history); + assert_eq!(recovered.identity().process, opts.process); + assert_eq!(recovered.group_manifest()?, Some(marker)); + assert_eq!(recovered.initial_state()?.hard_state, hard); + assert_eq!( + recovered.entries(2, 3, None, ::raft::GetEntriesContext::empty(false))?[0] + .data + .as_ref(), + b"control-prefix" + ); + assert_eq!(log.cell_groups()?.len(), 2); + assert_eq!(log.cell_group_inventory(None)?, inventory); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn cell_group_genesis_stays_invisible_until_actual_sync() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let initial = bootstrap(5); + let release = gate.arm(fault::HOLD_SYNC); + let writer = log.clone(); + let proposal = initial.clone(); + let create = domain.spawn(async move { writer.create_cell_group(proposal).await })?; + let held = domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await?; + let hidden = matches!( + log.cell_group(&initial.route.history), + Err(LogError::NotFound) + ); + let count = log.group_count(); + let pending = log.cell_group_inventory(Some(&initial.route.history)); + drop(release); + let group = create.await??; + held?; + assert!(hidden); + assert_eq!(count?, 0); + let pending = pending?; + assert!(pending.histories().is_empty()); + assert_eq!( + pending.initializing(), + Some( + log.cell_group_inventory(None)? + .group_history(&initial.route.history) + .unwrap() + .scope_digest() + ) + ); + assert_eq!(group.group_manifest()?.unwrap().route(), &initial.route); + assert_eq!(log.group_count()?, 1); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/storage_genesis_tests.rs b/crates/cellar/durability/raft/storage_genesis_tests.rs index 27c2edc3a..1e49f6d66 100644 --- a/crates/cellar/durability/raft/storage_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_genesis_tests.rs @@ -183,6 +183,10 @@ async fn physical_catalog_rejects_cell_aliases_across_restart() -> anyhow::Resul log.physical_inventory(Some(&alias.scope)), Err(LogError::WrongIdentity) )); + assert!(matches!( + log.cell_group_inventory(None), + Err(LogError::WrongIdentity) + )); let inventory = log.physical_inventory(Some(&initial.scope))?; assert_eq!(inventory.histories().len(), 1); assert_eq!( @@ -200,7 +204,13 @@ async fn physical_catalog_rejects_cell_aliases_across_restart() -> anyhow::Resul #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> anyhow::Result<()> { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - for corruption in ["old-scope", "unknown-domain", "zero-history", "empty-cell"] { + for corruption in [ + "old-scope", + "old-adapter", + "unknown-domain", + "zero-history", + "empty-cell", + ] { let root = tempfile::tempdir()?; let mut options = options(root.path()); let initial = bootstrap(5); @@ -231,7 +241,8 @@ async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> any let mapping = &mut record["mapping"]; assert_eq!(mapping["format"], 1); assert_eq!(mapping["domain"], "cellar-group-mapping-1"); - assert_eq!(mapping["application_cell"], initial.scope.cell); + assert_eq!(mapping["application"]["domain"], "single-cell-1"); + assert_eq!(mapping["application"]["state"], initial.scope.cell); assert_eq!(mapping["history"].as_object().unwrap().len(), 4); assert!(mapping["history"].get("cell").is_none()); let mut catalog_key = b"g:".to_vec(); @@ -248,7 +259,7 @@ async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> any let fields = mapping.as_object_mut().unwrap(); fields.remove("domain"); fields.remove("history"); - fields.remove("application_cell"); + fields.remove("application"); fields.insert( "scope".into(), serde_json::to_value( @@ -258,13 +269,17 @@ async fn physical_mapping_domain_refuses_old_and_corrupt_engine_records() -> any )?, ); } + "old-adapter" => { + mapping.as_object_mut().unwrap().remove("application"); + mapping["application_cell"] = serde_json::json!(initial.scope.cell); + } "unknown-domain" => { mapping["domain"] = serde_json::json!("cellar-image-mapping-1") } "zero-history" => { mapping["history"]["group"] = serde_json::json!(vec![0u8; 16]) } - "empty-cell" => mapping["application_cell"] = serde_json::json!(""), + "empty-cell" => mapping["application"]["state"] = serde_json::json!(""), _ => unreachable!(), } let mut batch = LogBatch::default(); diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index d8ffa482c..c6f32242c 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -51,6 +51,16 @@ async fn group_birth_reservations_bind_global_identity_and_canonical_placement( directory::birth_reservations().await } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn birth_unknown_and_claimed_blank_refuse() -> anyhow::Result<()> { + directory::member_bootstrap().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn group_member_permit_refuses_retired_birth_and_foreign_node() -> anyhow::Result<()> { + directory::member_fencing().await +} + fn route() -> GroupRoute { GroupRoute { history: GroupHistory { diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index 173b0bacf..189f7b24b 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -335,10 +335,10 @@ fn engine_options( ) } -fn root_write( - group: &cellar::application::consensus::verification::GroupLog, +fn root_write( + group: &cellar::application::consensus::verification::GroupLog, prepared: PreparedGroupManifest, -) -> Result { +) -> Result> { use raft::Storage as _; let index = prepared.marker().position().index.get(); let mut write = cellar::application::consensus::verification::GroupWrite::new(group); diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index d2d9b592f..5f78028b2 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -739,3 +739,197 @@ pub async fn birth_reservations() -> Result<()> { f.domain.shutdown().await; Ok(()) } + +fn member_options( + path: &std::path::Path, + node: u8, +) -> cellar::application::consensus::verification::ReplicaLogOptions { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLogOptions}; + use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; + ReplicaLogOptions::new( + path.to_owned(), + NodeId::new([node; 16]).unwrap(), + DiskLineage::new([node + 10; 16]).unwrap(), + ProcessGeneration::new([1; 16]).unwrap(), + LogOpenMode::Create, + ) +} + +async fn selected_birth(f: &Fixture) -> Result { + let d = &f.directory; + let request = d.prepare_group_birth(birth_spec(&ranges(1)[0]))?; + f.run(d.reserve_group_birth(&request)).await?; + let initial = + d.prepare_initialization(operation(999), std::slice::from_ref(request.route()))?; + f.run(d.publish(&initial)).await?; + Ok(f.run(d.materialize_group_birth(&request)).await?) +} + +pub async fn member_bootstrap() -> Result<()> { + use cellar::application::consensus::verification::{LogOpenMode, ReplicaLog}; + use cellar_logic::consensus::ProcessGeneration; + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let temp = tempfile::tempdir()?; + let mut options = member_options(&temp.path().join("member"), 1); + let log = f + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let history = birth.route().history; + let claim_key = f.key(&format!( + "claims/{}/{}/0000000000000001", + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()) + )); + let result = fault::missing_read( + claim_key.to_string(), + f.run(f.directory.bootstrap_group_member(&birth, &log)), + ) + .await; + assert!(matches!(result, Err(CatalogError::Unconfirmed))); + let group = log.cell_group(&history)?; + assert_eq!(group.group_manifest()?.unwrap().route(), birth.route()); + assert_eq!(group.group_manifest()?.unwrap().root().cells(), 0); + let key = group.numeric_key(); + let claim = f.store.inner.get(&claim_key).await?.bytes().await?; + assert_eq!(&claim[..9], b"CLRCAT01\xbb"); + let original_engine = log.storage_instance()?; + log.close().await?; + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([9; 16])?; + let log = f + .run(ReplicaLog::open( + options.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let reopened = f.reopen().await?; + // An observation from a different catalog owner never grants admission. + assert!(matches!( + f.run(reopened.bootstrap_group_member(&birth, &log)).await, + Err(CatalogError::Conflict(_)) + )); + let request = reopened.prepare_group_birth(birth_spec(birth.route()))?; + let current = f.run(reopened.materialize_group_birth(&request)).await?; + let permit = f + .run(reopened.bootstrap_group_member(¤t, &log)) + .await?; + f.run(permit.revalidate()).await?; + assert_eq!(permit.group().numeric_key(), key); + assert_eq!(permit.group().identity().scope, history); + assert_eq!(permit.group().identity().process, options.process); + assert_eq!(log.storage_instance()?, original_engine); + assert_eq!(f.store.inner.get(&claim_key).await?.bytes().await?, claim); + // Corrupt claims are not overwritten or interpreted as absent. + f.store + .inner + .put(&claim_key, b"not-a-claim".to_vec().into()) + .await?; + assert!(matches!( + f.run(permit.revalidate()).await, + Err(CatalogError::Conflict(_)) + )); + f.store.inner.put(&claim_key, claim.clone().into()).await?; + drop(permit); + log.close().await?; + // Remove only this owned fixture's group state, preserving the actual + // engine instance and permanent node pin. This exercises the member claim + // refusal independently of the earlier node-storage pin check. + let dir = options.directory.join("log").to_string_lossy().into_owned(); + f.domain + .blocking(move || -> Result<()> { + let engine = raft_engine::Engine::open(raft_engine::Config { + dir, + ..Default::default() + })?; + let mut catalog_key = b"g:".to_vec(); + catalog_key.extend_from_slice(&history.group.bytes()); + catalog_key.extend_from_slice(&history.incarnation.bytes()); + let mut batch = raft_engine::LogBatch::default(); + batch.delete(0, catalog_key); + batch.add_command(key, raft_engine::Command::Clean); + engine.write(&mut batch, true)?; + Ok(()) + })? + .await??; + let log = f + .run(ReplicaLog::open( + options, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert_eq!(log.storage_instance()?, original_engine); + assert!(log.cell_groups()?.is_empty()); + assert!(matches!( + f.run(reopened.bootstrap_group_member(¤t, &log)).await, + Err(CatalogError::LostStableState) + )); + assert!(log.cell_groups()?.is_empty()); + log.close().await?; + let blank = f + .run(ReplicaLog::open( + member_options(&temp.path().join("new-path"), 1), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert!(matches!( + f.run(reopened.bootstrap_group_member(¤t, &blank)) + .await, + Err(CatalogError::LostStableState) + )); + assert!(blank.cell_groups()?.is_empty()); + blank.close().await?; + assert_eq!(f.store.inner.get(&claim_key).await?.bytes().await?, claim); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn member_fencing() -> Result<()> { + use cellar::application::consensus::verification::ReplicaLog; + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let temp = tempfile::tempdir()?; + let foreign = f + .run(ReplicaLog::open( + member_options(&temp.path().join("foreign"), 9), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + assert!(matches!( + f.run(f.directory.bootstrap_group_member(&birth, &foreign)) + .await, + Err(CatalogError::NotInitialMember) + )); + assert!(foreign.cell_groups()?.is_empty()); + foreign.close().await?; + let log = f + .run(ReplicaLog::open( + member_options(&temp.path().join("member"), 1), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await?; + let permit = f + .run(f.directory.bootstrap_group_member(&birth, &log)) + .await?; + let view = f.run(f.directory.observe()).await?; + let mut children = split(birth.route(), 700); + f.reserve(&mut children).await?; + let replace = f.directory.prepare_replacement( + &view, + operation(1000), + std::slice::from_ref(birth.route()), + &children, + )?; + f.run(f.directory.publish(&replace)).await?; + assert!(matches!( + f.run(permit.revalidate()).await, + Err(CatalogError::Fenced) + )); + assert_eq!(log.cell_groups()?.len(), 1); + log.close().await?; + f.domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_recovery.rs b/crates/cellar/tests/support/cell_group_recovery.rs index 5dd50546f..6ff45ee2b 100644 --- a/crates/cellar/tests/support/cell_group_recovery.rs +++ b/crates/cellar/tests/support/cell_group_recovery.rs @@ -24,6 +24,15 @@ use std::{ time::Duration, }; +fn engine_bootstrap() -> cellar::application::consensus::verification::CellGroupBootstrap { + cellar::application::consensus::verification::CellGroupBootstrap { + route: route(), + member: MemberId::new(1).unwrap(), + voters: vec![MemberId::new(1).unwrap()], + learners: vec![], + } +} + #[derive(Clone, Copy, Debug)] enum Mode { Materialized, @@ -332,7 +341,7 @@ fn assert_receipt(pin: &CellImagePin, cell: u8, sequence: u64) -> Result<()> { fn prepare_root( tree: &ManifestTree, - group: &GroupLog, + group: &GroupLog, pages: &mut ManifestPageStore, marker: &GroupManifest, images: &Images, @@ -360,7 +369,7 @@ async fn seed(path: &Path, mode: Mode, large: bool) -> Result { fs.clone(), )) .await??; - let group = log.create_group(engine_bootstrap()).await?; + let group = log.create_cell_group(engine_bootstrap()).await?; let images = Images::open(&path.join("images"), mode, fs.clone())?; let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; let tree = ManifestTree::new(route(), ManifestLimits::default())?; @@ -377,7 +386,7 @@ async fn seed(path: &Path, mode: Mode, large: bool) -> Result { let cut = images::Cut::new(&[], &bytes, 0, true, false); let (leaf, pin) = images.prepare(cell, None, &cut, &bytes, u64::from(cell) + 1)?; prepared = tree.prepare_update(&mut pages, &prepared, leaf, None, pin)?; - log.persist( + log.persist_cells( OperationId::new(u64::from(cell)), vec![root_write(&group, prepared.clone())?], ) @@ -402,7 +411,7 @@ pub async fn both_images() -> Result<()> { fs.clone(), )) .await??; - let group = log.group(&engine_bootstrap().scope)?; + let group = log.cell_group(&engine_bootstrap().route.history)?; let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; let mut pages = ManifestPageStore::open(&temp.path().join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; @@ -431,7 +440,7 @@ pub async fn both_images() -> Result<()> { } let next = tree.prepare_update(&mut pages, ¤t, changed, Some(&old.version), next_pin)?; - log.persist(OperationId::new(3), vec![root_write(&group, next.clone())?]) + log.persist_cells(OperationId::new(3), vec![root_write(&group, next.clone())?]) .await?; assert_eq!(next.pin_cell(&history(2).cell).unwrap().0, &sibling); assert_receipt(&old_pin, 1, 1)?; @@ -454,7 +463,7 @@ async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Re fs.clone(), )) .await??; - let group = log.group(&engine_bootstrap().scope)?; + let group = log.cell_group(&engine_bootstrap().route.history)?; let marker = group.group_manifest()?.context("selected root")?; let index = marker.position().index.get(); if let Some(expected) = expected { @@ -572,7 +581,7 @@ pub async fn missing_images() -> Result<()> { fs.clone(), )) .await??; - let group = log.group(&engine_bootstrap().scope)?; + let group = log.cell_group(&engine_bootstrap().route.history)?; assert_eq!(group.group_manifest()?.as_ref(), Some(&marker)); let images = Images::open(&temp.path().join("images"), mode, fs.clone())?; let mut pages = @@ -696,7 +705,7 @@ pub async fn process_fixture() -> Result<()> { Arc::new(engine_fault::FaultFileSystem { gate: gate.clone() }), )) .await??; - let group = log.group(&engine_bootstrap().scope)?; + let group = log.cell_group(&engine_bootstrap().route.history)?; let image_fault = faults::Faults::new(path.join("images"), Some(signal.clone())); let page_fault = faults::Faults::new(path.join("pages"), Some(signal)); image_fault @@ -764,7 +773,7 @@ pub async fn process_fixture() -> Result<()> { "engine-after-sync" => engine_fault::CRASH_AFTER_SYNC, _ => engine_fault::OFF, }); - log.persist(OperationId::new(3), vec![root_write(&group, next)?]) + log.persist_cells(OperationId::new(3), vec![root_write(&group, next)?]) .await?; if cut_name == "after-publication" { page_fault.pause()?; diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index e091b02d9..dbf5bf14b 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -293,7 +293,9 @@ "crates/cellar/durability/raft/cell_groups/directory/tree.rs", "crates/cellar/durability/raft/cell_groups/directory/store.rs", "crates/cellar/tests/support/cell_group_directory.rs", - "crates/cellar/durability/raft/cell_groups/directory/birth.rs" + "crates/cellar/durability/raft/cell_groups/directory/birth.rs", + "crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs", + "crates/cellar/durability/raft/storage_cell_genesis_tests.rs" ], "test_target": "native_cell_groups_catalog", "steps": [ From f36b711dddb1d9d427e1d258bdfd5e7b07750e7a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:26:49 -0700 Subject: [PATCH 029/568] docs: qualify physical cell-group bootstrap checkpoint --- plans/multi-raft/cell-groups/README.md | 28 +++++----- plans/multi-raft/cell-groups/baseline.json | 14 ++--- plans/multi-raft/cell-groups/progress.md | 65 ++++++++++++++++++++-- 3 files changed, 79 insertions(+), 28 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 6d470b5b2..6b8b262e6 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The current requested deliverable is the executable design and implementation -plan. The complete shared-group runtime and hard cutover remain the implementation -target; delivering this package does not mark those tasks complete. +The design package is delivered. The active implementation goal remains the +complete shared-group runtime and hard cutover. Delivering the plan or an +intermediate checkpoint does not mark that goal complete. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,9 +68,8 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The latest user clarification selects an executable design and implementation -plan. This handoff preserves existing implementation work and its evidence; it -does not qualify or publish the unfinished runtime changes in this checkout. +The end-to-end implementation goal is active. Continue from the existing code +and retained evidence; the historical plan handoffs do not replace that scope. Continue CG-03 after reading [progress](progress.md) and inspecting the working diff. @@ -80,15 +79,14 @@ an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. -The next concrete boundary is physical group bootstrap. Uncommitted changes -already introduce physical replica handles, genesis and member claims, with -associated tests. Inspect and qualify that work before extending it. The reviewed -source baseline remains `d72ea068a527d2027e806ca78ab390256e8634fa`; current source -preflight refuses drift in the unfinished implementation. Do not refresh hashes -to bypass this refusal. Complete node inventory, startup authority and bounded -redirects before CG-04 connects enrollment to the shared runtime. A birth metadata -observation alone cannot authorize voter startup; a fabricated cell name cannot -substitute for a group identity. +Physical group genesis, typed replica handles, permanent initial-member claims +and complete group inventory are implemented at `5d0d8e5e`; see the latest +[checkpoint](progress.md) for evidence and limits. Complete authenticated redirects +with the original invocation deadline and hop budget before closing CG-03. CG-04 +then connects enrollment and the catalog startup permit to the shared runtime, +verifies the complete selected manifest/image closure and acquires current-term +quorum authority. Managed fleet integration remains required by CG-07. A birth +metadata observation alone cannot authorize voter startup. The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index f63b24f64..d0847a2c0 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,13 +1,13 @@ { "format": 1, - "source_revision": "d72ea068a527d2027e806ca78ab390256e8634fa", + "source_revision": "5d0d8e5ede3b4efedc211c40122a1a26b43968b4", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed the physical engine-registry cutover at d72ea068a527d2027e806ca78ab390256e8634fa. The registry now keys by GroupHistory; durable mappings store an explicit group-mapping domain and physical history separately from temporary single-cell adapter metadata. Old mapping shapes refuse. Existing adapter calls reject cell aliases at lookup, creation, pending occupancy and inventory boundaries. Anchored source hashes and excerpts are unchanged from the previous reviewed birth checkpoint. GroupBootstrap, replica handles and permanent claims still need the cell-independent cutover; this source does not expose shared-group voter startup. CG-03 remains IN_PROGRESS and the full hard-cutover goal is active.", + "review_note": "2026-10-29: reviewed physical cell-group bootstrap at 5d0d8e5ede3b4efedc211c40122a1a26b43968b4. ReplicaIdentity and GroupLog now carry GroupHistory without a fabricated cell. The existing GroupScope excerpt remains for staged host consumers. CachedGroup fields are unchanged but their excerpt moves after typed storage handles; mapping/cache validation now binds membership and mandatory group roots on both persistence and recovery. Catalog/birth excerpts are unchanged; catalog.rs adds the typed bootstrap permit export. The new path syncs empty group genesis, checks permanent node and member claims, refuses missing claimed state and exposes bounded physical inventory. Shared manifest recovery fixtures use real physical handles. No managed fleet permit, shared runtime serving authority, authenticated redirect integration or final hard cutover is implied. CG-03 remains IN_PROGRESS.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { "path": "crates/cellar-logic/consensus.rs", - "sha256": "4003a743cae358bc29d1b4f5f527d7685e9c468c5cfaaa8d9fcded18b5194912", + "sha256": "624ccbc21e7c83a8039444fa243d9e9a9eca12da8e32bf14e46893f8a44a83a8", "line": 83, "excerpt": "pub struct GroupScope {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub cell: String,\n pub group: GroupId,\n pub incarnation: IncarnationId,\n}\n\nimpl GroupScope {" }, @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "6e3a022cd16636f7587dd28be3edf86bd6ee0f13778d771ff8af0f6e4cf39edc", - "line": 377, + "sha256": "f68c46b39524dc8f960c4760513e4ed470919a5bc93aa952bc7b4ca93b5f9ab6", + "line": 416, "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n}\n" }, "apply": { @@ -61,13 +61,13 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "d925a1f3902fd9288d51ab8b252680f062a829e36ad537667d922be34f7cf8fc", + "sha256": "3eba1f95021bdc27e94a34dab3f9a01b127b4f58ca5930abab6fba7b89226f65", "line": 111, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "d925a1f3902fd9288d51ab8b252680f062a829e36ad537667d922be34f7cf8fc", + "sha256": "3eba1f95021bdc27e94a34dab3f9a01b127b4f58ca5930abab6fba7b89226f65", "line": 180, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 253a21679..e2b9a7c8a 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,11 +3,63 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The latest user clarification selects the plan as this handoff -deliverable. Intermediate contracts do not qualify a shared runtime. +hard cutover. The full implementation goal is active following the plan handoff +below. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. -## Current plan handoff — 2026-10-29 + +## CG-03 physical bootstrap and inventory checkpoint + +Source: `5d0d8e5ede3b4efedc211c40122a1a26b43968b4`. CG-03 remains IN_PROGRESS. + +Physical handles now carry `GroupHistory` independently of application cells. +Real-engine genesis atomically syncs the mapping, initial membership and complete +empty group root. The same persistence worker owns both typed storage paths, +quotas, sync and cache publication. Required mapping application domains refuse +old untagged records; an engine cannot mix legacy and cell-group application +families. Existing single-cell consumers remain staged until the final cutover. + +The initial unmanaged-member bootstrap API verifies the current selected birth, +permanent node-storage pin and exact member claim after actual genesis sync. +Claims bind group history, birth, member, node, disk and engine instance. Unknown +confirmation grants no permit; same-identity restart reconciles it. Tests remove +one owned group's records while retaining the original engine identity, then +verify that the permanent claim refuses fresh genesis. A separate blank-engine +fixture exercises the node pin. Foreign nodes/catalog owners and retired routes +also refuse; managed admission still requires its later rooted fleet integration. + +Physical inventory now captures every shared history, including unattached or +retained members, and an optional initializing slot. A domain-separated digest +binds all four physical IDs without a cell name. The bounded existing wire checks +the complete occupied count; an already durable pending slot counts once. Legacy +cell-scoped lookup cannot match a group-native row. Real-engine tests verify +inventory across restart and while genesis sync is held, including wire refusal +when the node count omits a slot. + +All 216 distinct Rust cases pass across selected genesis, physical inventory, +membership, core, directory, catalog, manifest, host, snapshot and storage suites. +The shared manifest recovery fixtures now use actual cell-group genesis for both +image representations and process-kill cuts. The RustFS-specific snapshot case +remains explicitly ignored; child fixtures are invoked by their parent tests. +These results do not qualify a live shared-group host or independent failure +resilience. Source preflight, all 37 plan-validator tests, formatting, layout, +dependency isolation and repository docs checks pass. The dependency script's +initial missing-PATH failure is retained separately from its successful rerun. + +All-target Clippy for `cellar` and `cellar-logic` also passes with warnings denied. +The partial evidence record is retained at +`.cellar/cell-groups-evidence/CG-03-bootstrap/checkpoint.json`, binding 76 source +inputs, 14 successful command logs and eight executables to the source revision. +This is a checkpoint, not a CG-03 DONE record. + +A catalog bootstrap permit is not complete image recovery or serving authority. +The runtime still must verify the selected image/manifest closure, revalidate its +startup permit and obtain current-term quorum. Authenticated bounded redirects, +the shared runtime, managed fleet integration and CG-04 through CG-16 remain +required. No RustFS disaster drill, load result, performance improvement or full +CG-03/release qualification is claimed by this checkpoint. + +## Plan handoff at 98aff95a — 2026-10-29 The package supplies the target architecture, protocol state machines, dependency ordered executor briefs, exact test commands and source-bound acceptance gates. @@ -132,9 +184,10 @@ CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `d72ea068a527d2027e806ca78ab390256e8634fa`. -The physical-registry change leaves all existing source anchor hashes and excerpts -unchanged. A new executor must receive both ignored DONE-task evidence folders +The reviewed source baseline is `5d0d8e5ede3b4efedc211c40122a1a26b43968b4`. +The physical bootstrap change updates the identity, manifest, catalog and birth +source anchors. Their excerpts are unchanged; the manifest excerpt moves after +the typed storage handles. A new executor must receive both ignored DONE-task evidence folders before validating those historical checkpoints. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured From 7f37a60917bdf4d54022eb366050dba5d58ee4bf Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:41:42 -0700 Subject: [PATCH 030/568] feat: bound cell-group routing by invocation and trusted catalog --- crates/cellar/durability/raft/catalog.rs | 4 +- .../durability/raft/cell_groups/README.md | 34 +- .../raft/cell_groups/directory/mod.rs | 5 + .../raft/cell_groups/directory/routing.rs | 319 ++++++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../tests/native_cell_groups_catalog.rs | 15 + .../tests/support/cell_group_directory.rs | 282 ++++++++++++++++ .../tests/support/raft_catalog_fault.rs | 18 + plans/multi-raft/cell-groups/design.md | 10 +- .../cell-groups/implementation-plan.md | 2 +- plans/multi-raft/cell-groups/tasks.json | 36 +- 11 files changed, 718 insertions(+), 11 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/routing.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index de9d6083f..cbd6123f1 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -26,7 +26,9 @@ pub use births::BirthDisposition; mod range_directory; pub use range_directory::{ CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, + CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, + CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 194b1d680..8d0249db8 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -172,8 +172,8 @@ limits bound individual operations; defaults are not a qualified capacity claim. Directory publication is metadata only. The future topology owner must obtain the seal, installation, archive and activation proofs before exposing replacement -ownership. Group birth/permanent disk claims, managed admission and bounded client -redirects remain to be integrated. Native directory tests exercise real conditional +ownership. Birth/claim admission and invocation-bounded route confirmation are +described below. Managed admission and production client integration remain open. Native directory tests exercise real conditional object-store operations in memory, including races, dropped replies, corruption, restart and bounded split/merge churn. They do not qualify live Raft ownership, real RustFS or the full CG-03 gate. @@ -260,3 +260,33 @@ must still match the complete occupied count before encoding or accepting a response. Inventory is an occupancy observation, not quorum or release proof. The generic physical capture includes shared histories, while the group-native accessor refuses a legacy application engine. + + +## Invocation-bounded route confirmation + +`RangeDirectory::routing_session` captures an explicit execution domain, original +monotonic deadline and 1–16 total attempts before discovery. The immutable request +binds logical cell history, command/enrollment/query identity and the dispatcher's +canonical request fingerprint. Initial discovery, redirects and same-group retries +share this budget. A previously resolved route can begin an invocation without +object-store I/O, even during a catalog outage; it is a potentially stale hint. + +A relocation hint is checked against the namespace's configured trusted catalog. +The session observes a pinned directory root and requires an exact lookup match +for the requested cell. It refuses foreign namespaces/schemas, wrong ranges, +non-advancing epochs, unpublished histories and stale attempt tickets. This +confirms the mapping through catalog trust; it does not authenticate a network +connection. The production transport must still authenticate peers, and every +destination must verify the complete route, logical cell incarnation and current +serving authority. No redirect URL or caller-supplied catalog is followed. + +Attempts are consumed before awaiting reads. Cancellation cannot refund a budget +or make a delayed ticket current. Read timeouts use the original domain clock, +even when a waiter is polled elsewhere. Marking a command potentially submitted +makes subsequent routing failures carry its original ambiguous command ID; a +redirect never proves the earlier proposal uncommitted. Physical command producers +remain the dispatcher's responsibility and must outlive cancelled waiters. + +This component prepares routing for the shared host. Native tests exercise real +conditional directory changes and controlled reads, not synthetic serving proofs. +It does not dispatch commands or qualify the production routing integration. diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index 413c23440..aa918e6c3 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -19,12 +19,17 @@ use std::{ mod birth; mod bootstrap; +mod routing; mod store; mod tree; mod wire; pub use birth::{CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec}; pub use bootstrap::CellGroupBootstrapPermit; +pub use routing::{ + CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, + CellRoutingLimits, CellRoutingSession, ResolvedCellRoute, +}; #[derive(Clone, Copy, Debug)] pub struct DirectoryLimits { diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs new file mode 100644 index 000000000..f2018c719 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -0,0 +1,319 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Invocation-bounded catalog confirmation of untrusted relocation hints. +//! Attempts are routing observations, never admission or quorum capabilities. + +use super::*; +use crate::asyncrt::{DomainToken, ExecutionDomain}; +use cellar_logic::consensus::CommandId; +use std::fmt; + +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum CellRouteOperation { + Command(CommandId), + Enrollment(ProposalId), + Query(ProposalId), +} + +/// The dispatcher must compute the fingerprint over its complete canonical +/// request (contract, input, durability or consistency, and retry metadata). +/// The session retains this exact binding through every attempt; destination +/// admission still validates the actual request and logical cell incarnation. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CellRouteRequest { + pub cell: CellHistory, + pub operation: CellRouteOperation, + pub fingerprint: [u8; 32], +} + +#[derive(Clone, Copy, Debug)] +pub struct CellRoutingLimits { + pub timeout: Duration, + /// Includes the initial attempt, same-group retries and relocations. + pub max_attempts: u8, +} + +#[derive(Debug)] +pub enum CellRoutingFailure { + InvalidRequest, + Closed, + Deadline, + AttemptsExhausted, + StaleAttempt, + InvalidHint, + Catalog(CatalogError), +} + +#[derive(Debug)] +pub struct CellRoutingError { + pub reason: CellRoutingFailure, + /// Once dispatch may have submitted a command, neither a redirect nor a + /// failed catalog read proves it uncommitted. Retry this same command ID. + pub ambiguous_command: Option, +} + +impl fmt::Display for CellRoutingError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell-group routing: {:?}", self.reason) + } +} + +impl std::error::Error for CellRoutingError {} + +/// A cached observation obtained from the configured trusted catalog. It can +/// become stale; a serving destination must check the full route binding. +#[derive(Clone)] +pub struct ResolvedCellRoute { + cell: CellHistory, + route: GroupRoute, + revision: u64, + owner: Arc<()>, +} + +impl ResolvedCellRoute { + pub fn route(&self) -> &GroupRoute { + &self.route + } +} + +#[derive(Clone)] +pub struct CellRouteAttempt { + resolved: ResolvedCellRoute, + request: Arc, + deadline_ms: u64, + ordinal: u8, + owner: Arc<()>, +} + +impl CellRouteAttempt { + pub fn route(&self) -> &GroupRoute { + self.resolved.route() + } + + pub fn cached_route(&self) -> ResolvedCellRoute { + self.resolved.clone() + } + + pub fn request(&self) -> &CellRouteRequest { + &self.request + } + + /// Absolute monotonic deadline in the session's captured execution domain. + pub fn deadline_ms(&self) -> u64 { + self.deadline_ms + } + + pub fn ordinal(&self) -> u8 { + self.ordinal + } +} + +/// One invocation, with no automatic command submission or producer ownership. +/// Dispatchers retain physical producers independently of these routing waiters. +/// This owner is not cloneable: retries cannot copy or replenish its budget. +pub struct CellRoutingSession { + directory: RangeDirectory, + domain: DomainToken, + request: Arc, + deadline: u64, + maximum: u8, + attempts: u8, + active: bool, + submitted: bool, + cached: Option, + owner: Arc<()>, +} + +impl RangeDirectory { + /// Capture the deadline before initial discovery. A matching cached route + /// permits the first attempt without any object-store access. + pub fn routing_session( + &self, + domain: &ExecutionDomain, + request: CellRouteRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + let error = |reason| CellRoutingError { + reason, + ambiguous_command: None, + }; + let config = self.namespace.config(); + if request.cell.cluster != config.cluster + || request.cell.namespace != config.namespace + || request.fingerprint == [0; 32] + || limits.timeout.as_millis() == 0 + || limits.timeout > Duration::from_secs(60) + || !(1..=16).contains(&limits.max_attempts) + || cached.as_ref().is_some_and(|cached| { + cached.cell != request.cell + || !Arc::ptr_eq(&cached.owner, &self.namespace.inner.owner) + }) + { + return Err(error(CellRoutingFailure::InvalidRequest)); + } + let domain = domain + .services() + .domain_token() + .ok_or_else(|| error(CellRoutingFailure::Closed))?; + let now = domain + .mono_ms() + .map_err(|_| error(CellRoutingFailure::Closed))?; + let deadline = now + .checked_add(limits.timeout.as_millis() as u64) + .ok_or_else(|| error(CellRoutingFailure::InvalidRequest))?; + Ok(CellRoutingSession { + directory: self.clone(), + domain, + request: Arc::new(request), + deadline, + maximum: limits.max_attempts, + attempts: 0, + active: false, + submitted: false, + cached, + owner: Arc::new(()), + }) + } +} + +impl CellRoutingSession { + fn error(&self, reason: CellRoutingFailure) -> CellRoutingError { + CellRoutingError { + reason, + ambiguous_command: match self.request.operation { + CellRouteOperation::Command(id) if self.submitted => Some(id), + _ => None, + }, + } + } + + fn live(&self) -> Result<(), CellRoutingError> { + let now = self + .domain + .mono_ms() + .map_err(|_| self.error(CellRoutingFailure::Closed))?; + if now >= self.deadline { + return Err(self.error(CellRoutingFailure::Deadline)); + } + Ok(()) + } + + fn check_attempt(&self, attempt: &CellRouteAttempt) -> Result<(), CellRoutingError> { + if !self.active + || !Arc::ptr_eq(&self.owner, &attempt.owner) + || attempt.ordinal != self.attempts + { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + Ok(()) + } + + /// Mark before handing a mutation to a producer that may submit it. Only + /// explicit request reconciliation can later establish its actual outcome. + pub fn mark_submitted(&mut self, attempt: &CellRouteAttempt) -> Result<(), CellRoutingError> { + self.check_attempt(attempt)?; + self.live()?; + self.submitted = true; + Ok(()) + } + + pub async fn begin(&mut self) -> Result { + if self.attempts != 0 { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + self.next(None).await + } + + /// Same-group retry, including a different member, under the same budget. + pub async fn retry( + &mut self, + previous: &CellRouteAttempt, + ) -> Result { + self.check_attempt(previous)?; + self.next(None).await + } + + /// A relocation hint is untrusted until an exact lookup in the configured + /// catalog confirms its namespace, range, epoch, schema, birth and history. + /// No arbitrary URL or caller-supplied catalog is followed. + pub async fn redirect( + &mut self, + previous: &CellRouteAttempt, + hint: &GroupRoute, + ) -> Result { + self.check_attempt(previous)?; + self.next(Some(hint)).await + } + + async fn next( + &mut self, + hint: Option<&GroupRoute>, + ) -> Result { + // Consume the previous ticket and attempt before any await. Dropping a + // discovery future cannot refund attempts or authorize a late response. + self.active = false; + self.live()?; + if self.attempts >= self.maximum { + return Err(self.error(CellRoutingFailure::AttemptsExhausted)); + } + self.attempts += 1; + if let Some(hint) = hint { + let previous = self + .cached + .as_ref() + .ok_or_else(|| self.error(CellRoutingFailure::InvalidHint))?; + if wire::Route::encode(hint, &self.directory).is_err() + || hint.epoch <= previous.route.epoch + || !hint.range.contains(&cell_routing_key( + hint.history.namespace, + &self.request.cell.cell, + )) + { + return Err(self.error(CellRoutingFailure::InvalidHint)); + } + } + if self.cached.is_none() || hint.is_some() { + let minimum = self.cached.as_ref().map_or(0, |route| route.revision); + let work = async { + let view = self.directory.observe().await?; + let route = self + .directory + .lookup(&view, &self.request.cell.cell) + .await?; + Ok::<_, CatalogError>((view, route)) + }; + let result = self + .domain + .run(crate::asyncrt::timeout_at(self.deadline, work)) + .await + .map_err(|_| self.error(CellRoutingFailure::Closed))?; + self.live()?; + let (view, route) = result + .map_err(|_| self.error(CellRoutingFailure::Deadline))? + .map_err(|error| self.error(CellRoutingFailure::Catalog(error)))?; + if view.revision() < minimum || hint.is_some_and(|hint| &route != hint) { + return Err(self.error(CellRoutingFailure::InvalidHint)); + } + self.cached = Some(ResolvedCellRoute { + cell: self.request.cell.clone(), + route, + revision: view.revision(), + owner: self.directory.namespace.inner.owner.clone(), + }); + } + self.live()?; + let resolved = self + .cached + .clone() + .ok_or_else(|| self.error(CellRoutingFailure::InvalidHint))?; + self.active = true; + Ok(CellRouteAttempt { + resolved, + request: self.request.clone(), + deadline_ms: self.deadline, + ordinal: self.attempts, + owner: self.owner.clone(), + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index b9ead733b..215207666 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -19,7 +19,9 @@ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsag pub use crate::raft::catalog::{ CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, + CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, + CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index c6f32242c..c23654785 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -324,3 +324,18 @@ fn enrollment_record_is_bounded_canonical_data() { Err(EnrollmentError::Protocol(_)) )); } + +#[tokio::test] +async fn routing_redirect_requires_catalog_confirmation() -> anyhow::Result<()> { + directory::routing_authentication().await +} + +#[tokio::test] +async fn routing_cached_requests_preserve_identity_without_catalog_reads() -> anyhow::Result<()> { + directory::routing_cache_and_budget().await +} + +#[tokio::test] +async fn routing_attempt_deadline_survives_redirect_and_cancellation() -> anyhow::Result<()> { + directory::routing_deadline_and_cancellation().await +} diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 5f78028b2..fb5d4a46d 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -933,3 +933,285 @@ pub async fn member_fencing() -> Result<()> { f.domain.shutdown().await; Ok(()) } + +fn route_request() -> CellRouteRequest { + use cellar_logic::consensus::{ClientId, CommandId}; + CellRouteRequest { + cell: CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new("counter-routing").unwrap(), + incarnation: IncarnationId::new([81; 16]).unwrap(), + }, + operation: CellRouteOperation::Command( + CommandId::new(ClientId::new([82; 16]).unwrap(), 7).unwrap(), + ), + fingerprint: Sha256::digest(b"canonical command with input and durability").into(), + } +} + +fn routing_limits(max_attempts: u8) -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts, + } +} + +async fn publish_route_split( + f: &Fixture, + parent: &GroupRoute, + cell: &CellId, +) -> Result { + let view = f.run(f.directory.observe()).await?; + let mut children = split(parent, 700); + f.reserve(&mut children).await?; + let replace = f.directory.prepare_replacement( + &view, + operation(1000), + std::slice::from_ref(parent), + &children, + )?; + f.run(f.directory.publish(&replace)).await?; + let key = cell_routing_key(parent.history.namespace, cell); + Ok(children + .into_iter() + .find(|r| r.range.contains(&key)) + .unwrap()) +} + +pub async fn routing_authentication() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut session = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(4), None)?; + let first = session.begin().await?; + assert_eq!(first.route(), birth.route()); + session.mark_submitted(&first)?; + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + let second = session.redirect(&first, &next).await?; + assert_eq!(second.route(), &next); + assert_eq!(second.request(), &request); + assert_eq!(second.deadline_ms(), first.deadline_ms()); + assert_eq!(second.ordinal(), 2); + let stale = session.redirect(&first, &next).await.err().unwrap(); + assert!(matches!(stale.reason, CellRoutingFailure::StaleAttempt)); + assert!(stale.ambiguous_command.is_some()); + let third = session.retry(&second).await?; + assert_eq!(third.request(), first.request()); + assert_eq!(third.deadline_ms(), first.deadline_ms()); + + // Valid-looking but unpublished history must fail catalog confirmation. + let mut forged = next.clone(); + forged.history.group = GroupId::new([99; 16])?; + let mut refused = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(4), + Some(first.cached_route()), + )?; + let attempt = refused.begin().await?; + refused.mark_submitted(&attempt)?; + let (result, reads) = fault::count_reads(refused.redirect(&attempt, &forged)).await; + let failure = result.err().unwrap(); + assert!(matches!(failure.reason, CellRoutingFailure::InvalidHint)); + assert!(!reads.is_empty()); + let CellRouteOperation::Command(id) = request.operation else { + unreachable!() + }; + assert_eq!(failure.ambiguous_command, Some(id)); + assert!(matches!( + refused.retry(&attempt).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + + // Syntactically valid namespace/schema rebinding is rejected before I/O. + for field in 0..4 { + let mut wrong = next.clone(); + match field { + 0 => wrong.history.namespace = NamespaceId::new([91; 16])?, + 1 => wrong.history.cluster = ClusterId::new([92; 16])?, + 2 => wrong.schema_digest = [93; 32], + _ => wrong.epoch = birth.route().epoch, + } + let mut session = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(4), + Some(first.cached_route()), + )?; + let attempt = session.begin().await?; + let (result, reads) = fault::count_reads(session.redirect(&attempt, &wrong)).await; + assert!(matches!( + result.err().unwrap().reason, + CellRoutingFailure::InvalidHint + )); + assert!(reads.is_empty()); + } + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + +pub async fn routing_cache_and_budget() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut origin = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(2), None)?; + let first = origin.begin().await?; + let cached = first.cached_route(); + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + origin.mark_submitted(&first)?; + let second = origin.redirect(&first, &next).await?; + let error = origin.retry(&second).await.err().unwrap(); + assert!(matches!( + error.reason, + CellRoutingFailure::AttemptsExhausted + )); + assert!(error.ambiguous_command.is_some()); + + // Cached attempts can use a historical route during a catalog outage. + // This grants no serving authority: the destination must check its epoch. + f.store.fail_read.store(true, Ordering::Release); + let mut known = f.directory.routing_session( + &f.domain, + request.clone(), + routing_limits(2), + Some(cached.clone()), + )?; + let (attempt, reads) = fault::count_reads(known.begin()).await; + let attempt = attempt?; + assert!(reads.is_empty()); + assert!(f.store.fail_read.load(Ordering::Acquire)); + assert_eq!(attempt.route(), birth.route()); + assert_eq!(attempt.request(), &request); + let (retry, reads) = fault::count_reads(known.retry(&attempt)).await; + assert!(reads.is_empty()); + assert_eq!(retry?.request(), &request); + let foreign_ticket = known.mark_submitted(&first).unwrap_err(); + assert!(matches!( + foreign_ticket.reason, + CellRoutingFailure::StaleAttempt + )); + f.store.fail_read.store(false, Ordering::Release); + let foreign = f.reopen().await?; + assert!(matches!( + foreign + .routing_session( + &f.domain, + request.clone(), + routing_limits(2), + Some(cached.clone()) + ) + .err() + .unwrap() + .reason, + CellRoutingFailure::InvalidRequest + )); + for field in 0..3 { + let mut wrong = request.clone(); + match field { + 0 => wrong.cell.cell = CellId::new("another-cell")?, + 1 => wrong.cell.incarnation = IncarnationId::new([93; 16])?, + _ => wrong.fingerprint = [0; 32], + } + assert!(matches!( + f.directory + .routing_session(&f.domain, wrong, routing_limits(2), Some(cached.clone())) + .err() + .unwrap() + .reason, + CellRoutingFailure::InvalidRequest + )); + } + for maximum in [0, 17, 255] { + assert!(f + .directory + .routing_session(&f.domain, request.clone(), routing_limits(maximum), None) + .is_err()); + } + f.domain.shutdown().await; + assert!(matches!( + origin.retry(&second).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + Ok(()) +} + +#[allow(clippy::disallowed_methods)] +pub async fn routing_deadline_and_cancellation() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let birth = selected_birth(&f).await?; + let request = route_request(); + let mut initial = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(4), None)?; + let observed = initial.begin().await?; + let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; + let limits = CellRoutingLimits { + timeout: Duration::from_millis(100), + max_attempts: 4, + }; + let mut session = f.directory.routing_session( + &f.domain, + request.clone(), + limits, + Some(observed.cached_route()), + )?; + let attempt = session.begin().await?; + session.mark_submitted(&attempt)?; + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + let result = tokio::time::timeout( + Duration::from_secs(2), + fault::gate_read( + f.key("pin").to_string(), + gate.clone(), + session.redirect(&attempt, &next), + ), + ) + .await?; + assert!(!gate.armed.load(Ordering::Acquire)); + let error = result.err().unwrap(); + assert!(matches!(error.reason, CellRoutingFailure::Deadline)); + assert!(error.ambiguous_command.is_some()); + assert!(matches!( + session.retry(&attempt).await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + + // Dropping discovery cannot reset its attempt budget or accept its ticket. + let mut cancelled = + f.directory + .routing_session(&f.domain, request.clone(), routing_limits(3), None)?; + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + { + let call = fault::gate_read(f.key("pin").to_string(), gate.clone(), cancelled.begin()); + tokio::pin!(call); + tokio::select! { + result = &mut call => panic!("stalled discovery completed: {}", result.is_ok()), + _ = gate.entered.notified() => {}, + _ = tokio::time::sleep(Duration::from_secs(2)) => panic!("discovery did not enter the real store"), + } + } + assert!(matches!( + cancelled.begin().await.err().unwrap().reason, + CellRoutingFailure::StaleAttempt + )); + let mut closed = f.directory.routing_session( + &f.domain, + request, + routing_limits(2), + Some(observed.cached_route()), + )?; + f.domain.shutdown().await; + assert!(matches!( + closed.begin().await.err().unwrap().reason, + CellRoutingFailure::Closed + )); + Ok(()) +} diff --git a/crates/cellar/tests/support/raft_catalog_fault.rs b/crates/cellar/tests/support/raft_catalog_fault.rs index dfe94dc83..32c352860 100644 --- a/crates/cellar/tests/support/raft_catalog_fault.rs +++ b/crates/cellar/tests/support/raft_catalog_fault.rs @@ -11,6 +11,7 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; tokio::task_local! { static READ_COUNTS: std::cell::RefCell>; static MISSING_READ: String; + static GATED_READ: (String, std::sync::Arc); } // Keep a read fault local to one verifier; running nodes still see the store. @@ -19,6 +20,16 @@ pub async fn missing_read(path: String, work: impl std::future::Future( + path: String, + gate: std::sync::Arc, + work: impl std::future::Future, +) -> T { + GATED_READ.scope((path, gate), work).await +} + // Shared fault adapter: only catalog work-count regressions use this scope. #[allow(dead_code)] pub async fn count_reads( @@ -502,6 +513,13 @@ impl ObjectStore for Store { let _ = READ_COUNTS.try_with(|counts| { *counts.borrow_mut().entry(path.to_string()).or_default() += 1; }); + let gate = GATED_READ + .try_with(|(key, gate)| (key == path.as_ref()).then(|| gate.clone())) + .ok() + .flatten(); + if let Some(gate) = gate { + gate.pause().await; + } if MISSING_READ .try_with(|missing| missing == path.as_ref()) .unwrap_or(false) diff --git a/plans/multi-raft/cell-groups/design.md b/plans/multi-raft/cell-groups/design.md index 4162a7b9c..c8178ac3e 100644 --- a/plans/multi-raft/cell-groups/design.md +++ b/plans/multi-raft/cell-groups/design.md @@ -116,9 +116,13 @@ is replaced by unique range ownership plus replicated enrollment; do not build t assignment authorities. While a range is sealed, enrollment is refused/retried. Cached routes carry group history and routing epoch; every destination verifies -the complete binding. Redirects are authenticated hints, bounded by the original -deadline and hop count. An ambiguous submitted command is retried only with its -original cell-scoped command identity. An unavailable catalog may prevent births +the complete binding. Relocation hints are authenticated by exact confirmation +against the configured trusted range directory, bounded by the original invocation deadline and a finite +total attempt count. Known cached routes need no per-request catalog read. This +mapping confirmation does not replace transport peer authentication or destination +admission checks; a peer cannot introduce an arbitrary redirect URL or catalog. +An ambiguous submitted command is retried only with its original cell-scoped +command identity. An unavailable catalog may prevent births and topology changes; it does not turn a known group's write path into S3 quorum. ## Runtime and persistence diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index 0e26e48c1..d5d6d727d 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -14,7 +14,7 @@ never executed by the plan runner. Read the [design](design.md), | --- | --- | --- | --- | ---: | | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | -| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | IN_PROGRESS | 5 | +| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | IN_PROGRESS | 8 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index dbf5bf14b..83b10045e 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -295,7 +295,8 @@ "crates/cellar/tests/support/cell_group_directory.rs", "crates/cellar/durability/raft/cell_groups/directory/birth.rs", "crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs", - "crates/cellar/durability/raft/storage_cell_genesis_tests.rs" + "crates/cellar/durability/raft/storage_cell_genesis_tests.rs", + "crates/cellar/durability/raft/cell_groups/directory/routing.rs" ], "test_target": "native_cell_groups_catalog", "steps": [ @@ -321,7 +322,10 @@ "action": "Add bounded tree traversal/cache tests, restart races and stale-route redirects with original deadlines/hop budgets. Ordinary known-group commands must not require an S3 catalog request.", "tests": [ "bounded_catalog_discovery", - "sealed_range_rejects_enrollment" + "sealed_range_rejects_enrollment", + "routing_redirect_requires_catalog_confirmation", + "routing_cached_requests_preserve_identity_without_catalog_reads", + "routing_attempt_deadline_survives_redirect_and_cancellation" ] } ], @@ -363,6 +367,32 @@ "invariants": [ "I09" ] + }, + { + "name": "routing_redirect_requires_catalog_confirmation", + "oracle": "An untrusted relocation is followed only after an exact route lookup in the configured trusted catalog; stale tickets, foreign namespace/schema and non-advancing epochs refuse. The logical cell, original command identity, request fingerprint and invocation deadline survive a confirmed split redirect.", + "invariants": [ + "I01", + "I03", + "I09" + ] + }, + { + "name": "routing_cached_requests_preserve_identity_without_catalog_reads", + "oracle": "Cached route admission and same-group retries perform no object-store reads during a catalog outage. The original finite attempt budget covers retries and redirects; cached bindings cannot cross cells, incarnations or catalog owners. A cached route never grants serving authority.", + "invariants": [ + "I01", + "I03", + "I12" + ] + }, + { + "name": "routing_attempt_deadline_survives_redirect_and_cancellation", + "oracle": "A real held catalog read expires under the captured invocation deadline, preserving command ambiguity. Dropped discovery cannot refund an attempt, and a closed execution domain refuses even cached admission.", + "invariants": [ + "I03", + "I12" + ] } ], "invariants": [ @@ -416,7 +446,7 @@ "test_target": "native_cell_groups_execution", "steps": [ { - "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands. Atomically retain the immutable enrollment creation record with the selected cell state; preserve it through mutations and transfer so retries never reinitialize or restore the initial image over newer state.", + "action": "Split the combined host Cell owner into group consensus state and per-cell execution state. One GroupHistory owns exactly one RawNode per voter; cell count does not allocate additional RawNodes. Reuse shared persistence, transport and SQL pools. Integrate CG-03 enrollment decisions with replicated enrollment and independent SQLite initialization before exposing cell commands. Atomically retain the immutable enrollment creation record with the selected cell state; preserve it through mutations and transfer so retries never reinitialize or restore the initial image over newer state. Consume the prepared CellRoutingSession attempts when integrating dispatch: preserve the logical request fingerprint, original domain deadline and total attempt budget across member retries and group redirects. Catalog-confirmed mappings are hints; destination admission must validate the full binding and current authority. Command producers remain domain-owned after a routing waiter is cancelled.", "tests": [ "three_cells_one_rawnode_per_voter", "concurrent_enrollment_one_cell" From c60a7d23aa4c01bde405f04483d1792cb9375bc2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:52:02 -0700 Subject: [PATCH 031/568] test: prove redirect cannot replenish invocation deadline --- crates/cellar/tests/support/cell_group_directory.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index fb5d4a46d..800cdebef 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -1152,7 +1152,7 @@ pub async fn routing_deadline_and_cancellation() -> Result<()> { let observed = initial.begin().await?; let next = publish_route_split(&f, birth.route(), &request.cell.cell).await?; let limits = CellRoutingLimits { - timeout: Duration::from_millis(100), + timeout: Duration::from_secs(5), max_attempts: 4, }; let mut session = f.directory.routing_session( @@ -1163,6 +1163,10 @@ pub async fn routing_deadline_and_cancellation() -> Result<()> { )?; let attempt = session.begin().await?; session.mark_submitted(&attempt)?; + // Spend most of the original budget before redirecting. The two-second + // external guard must beat a fresh five-second timeout, but leaves a full + // second of scheduling tolerance for the original one-second remainder. + tokio::time::sleep(Duration::from_secs(4)).await; let gate = Arc::new(fault::RegistryGate::default()); gate.armed.store(true, Ordering::Release); let result = tokio::time::timeout( From 0df009a24d88408a8efc40e356f794ca35513a95 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 15:55:07 -0700 Subject: [PATCH 032/568] test: use a permitted cancellation selector --- .../cellar/tests/support/cell_group_directory.rs | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 800cdebef..9b91d54dd 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -1196,10 +1196,16 @@ pub async fn routing_deadline_and_cancellation() -> Result<()> { { let call = fault::gate_read(f.key("pin").to_string(), gate.clone(), cancelled.begin()); tokio::pin!(call); - tokio::select! { - result = &mut call => panic!("stalled discovery completed: {}", result.is_ok()), - _ = gate.entered.notified() => {}, - _ = tokio::time::sleep(Duration::from_secs(2)) => panic!("discovery did not enter the real store"), + match tokio::time::timeout( + Duration::from_secs(2), + futures_util::future::select(&mut call, Box::pin(gate.entered.notified())), + ) + .await? + { + futures_util::future::Either::Left((result, _)) => { + panic!("stalled discovery completed: {}", result.is_ok()) + } + futures_util::future::Either::Right(((), _pending)) => {} } } assert!(matches!( From b3b7eabdf102b146db40629b8a60bda1df398f2a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 16:03:51 -0700 Subject: [PATCH 033/568] docs: deliver current cell-group implementation plan handoff --- plans/multi-raft/cell-groups/README.md | 26 ++++++----- plans/multi-raft/cell-groups/baseline.json | 12 ++--- plans/multi-raft/cell-groups/progress.md | 51 +++++++++++++++++++--- 3 files changed, 65 insertions(+), 24 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 6b8b262e6..9ffcbbc73 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The design package is delivered. The active implementation goal remains the -complete shared-group runtime and hard cutover. Delivering the plan or an -intermediate checkpoint does not mark that goal complete. +The requested deliverable is the executable design and implementation plan. +The target for its implementation executor remains the complete shared-group +runtime and hard cutover. Plan delivery does not qualify that runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -68,8 +68,9 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The end-to-end implementation goal is active. Continue from the existing code -and retained evidence; the historical plan handoffs do not replace that scope. +The user selected a plan handoff for this delivery. The implementation executor +should continue from the existing code and retained evidence; no new runtime +work is required to read or review this package. Continue CG-03 after reading [progress](progress.md) and inspecting the working diff. @@ -79,12 +80,15 @@ an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. -Physical group genesis, typed replica handles, permanent initial-member claims -and complete group inventory are implemented at `5d0d8e5e`; see the latest -[checkpoint](progress.md) for evidence and limits. Complete authenticated redirects -with the original invocation deadline and hop budget before closing CG-03. CG-04 -then connects enrollment and the catalog startup permit to the shared runtime, -verifies the complete selected manifest/image closure and acquires current-term +Physical group genesis, typed replica handles, permanent initial-member claims, +complete group inventory and invocation-bounded catalog-confirmed redirects are +implemented through `0df009a2`; see the latest [checkpoint](progress.md) for +evidence and limits. Audit the retained CG-03 record against every task step +before marking it DONE. The record is at +`.cellar/cell-groups-evidence/CG-03-routing-final/record.json`; transfer that entire +directory along with the earlier DONE-task evidence to the implementation +executor. CG-04 then connects enrollment and the catalog startup permit to the +shared runtime, verifies the complete selected manifest/image closure and acquires current-term quorum authority. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index d0847a2c0..d94d7686c 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "5d0d8e5ede3b4efedc211c40122a1a26b43968b4", + "source_revision": "0df009a24d88408a8efc40e356f794ca35513a95", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group bootstrap at 5d0d8e5ede3b4efedc211c40122a1a26b43968b4. ReplicaIdentity and GroupLog now carry GroupHistory without a fabricated cell. The existing GroupScope excerpt remains for staged host consumers. CachedGroup fields are unchanged but their excerpt moves after typed storage handles; mapping/cache validation now binds membership and mandatory group roots on both persistence and recovery. Catalog/birth excerpts are unchanged; catalog.rs adds the typed bootstrap permit export. The new path syncs empty group genesis, checks permanent node and member claims, refuses missing claimed state and exposes bounded physical inventory. Shared manifest recovery fixtures use real physical handles. No managed fleet permit, shared runtime serving authority, authenticated redirect integration or final hard cutover is implied. CG-03 remains IN_PROGRESS.", + "review_note": "2026-10-29: reviewed invocation-bounded routing at 7f37a60917bdf4d54022eb366050dba5d58ee4bf. Catalog/birth source anchor excerpts are unchanged; catalog.rs adds exports for the new route session, request, attempt, cached observation and typed errors, shifting the excerpt lines. The directory verifies untrusted relocation hints by exact lookup in its configured trusted catalog under the original captured domain deadline. Immutable request identity and ambiguity survive redirects, budgets are consumed before awaits, and cached starts do not read the catalog. This prepares shared-host routing; it is not transport authentication, command execution or serving authority. Existing storage and host source anchors are unchanged. Successor c60a7d23aa4c01bde405f04483d1792cb9375bc2 strengthens the native deadline test by consuming four seconds of a five-second invocation before a stalled redirect; a two-second external guard rejects per-hop timeout replenishment. Anchored production source is unchanged. Successor 0df009a24d88408a8efc40e356f794ca35513a95 replaces the forbidden Tokio select macro in the external cancellation test with a bounded futures selector. No production source anchor changes.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3eba1f95021bdc27e94a34dab3f9a01b127b4f58ca5930abab6fba7b89226f65", - "line": 111, + "sha256": "ee51dcf3b884e9edf39364046054d8aac7769005a2637fef1dd7cb1cb7c3a4d8", + "line": 113, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3eba1f95021bdc27e94a34dab3f9a01b127b4f58ca5930abab6fba7b89226f65", - "line": 180, + "sha256": "ee51dcf3b884e9edf39364046054d8aac7769005a2637fef1dd7cb1cb7c3a4d8", + "line": 182, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index e2b9a7c8a..6f22fcca6 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -3,11 +3,47 @@ # Cell-group implementation progress The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The full implementation goal is active following the plan handoff -below. Intermediate contracts do not qualify a shared runtime. +hard cutover. The latest user selection scopes this delivery to the executable +design and implementation plan. Earlier implementation checkpoints are retained +below; their historical scope statements do not override that selection. +Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## Current executable-plan handoff + +The package contains 16 dependency-ordered tasks, 86 required cases, 14 invariants +and nine qualification profiles. Each task supplies source anchors, owned paths, +implementation steps, exact test commands and acceptance oracles. CG-01 and CG-02 +retain their historical DONE evidence. CG-03 remains IN_PROGRESS pending the +implementation executor's completion audit; CG-04 through CG-16 remain TODO. +This handoff does not advance runtime status merely because a plan validates. + +The reviewed source is `0df009a24d88408a8efc40e356f794ca35513a95`. Existing routing +sessions now retain the original logical request, command identity, deadline and +attempt budget across retries. Relocation requires exact confirmation against the +configured trusted catalog; cached requests need no catalog read. Cancelled +lookup cannot refund an attempt, and a possibly submitted command retains its +ambiguous outcome. Catalog confirmation does not authenticate a network peer or +grant serving authority. Actual shared dispatch remains CG-04/CG-07 work. + +The retained record at +`.cellar/cell-groups-evidence/CG-03-routing-final/record.json` binds 77 source +inputs, 20 successful command logs, three test executables and 44 distinct Rust +test cases to that revision. Its gate includes all-target check and Clippy, +formatting, layout, dependency isolation and docs checks. All 37 plan-validator +tests pass. The earlier Clippy rejection of a forbidden test macro is retained; +the final source replaces it with a permitted selector without suppressing the +lint. These are existing local qualification results, not a RustFS drill or a +shared runtime performance claim. + +For this handoff, package validation, reviewed-source preflight, validator tests, +executor-brief generation and repository docs checks are rerun. The release gate +must still refuse incomplete implementation. The branch includes freshly fetched +`origin/main` at `1be27d234d4674d4d014785c60cecbfe394bdab9`; the supplied TiKV checkout +matches the investigation pin. The requested MR-13/MR-14 progress JSON files +remain excluded from the PR. + ## CG-03 physical bootstrap and inventory checkpoint Source: `5d0d8e5ede3b4efedc211c40122a1a26b43968b4`. CG-03 remains IN_PROGRESS. @@ -184,11 +220,12 @@ CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `5d0d8e5ede3b4efedc211c40122a1a26b43968b4`. -The physical bootstrap change updates the identity, manifest, catalog and birth -source anchors. Their excerpts are unchanged; the manifest excerpt moves after -the typed storage handles. A new executor must receive both ignored DONE-task evidence folders -before validating those historical checkpoints. +The reviewed source baseline is `0df009a24d88408a8efc40e356f794ca35513a95`. +The routing change updates the catalog/birth anchor hashes and line positions; +their excerpts are unchanged. The baseline records the source review and test +corrections. A new executor must receive both ignored DONE-task evidence folders +before validating those historical checkpoints, plus the CG-03 routing directory +before auditing that task for completion. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until From 9f937f227928c1ffa7324c8727a62cd410967383 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 16:19:45 -0700 Subject: [PATCH 034/568] feat: retain immutable enrollment with complete cell-group state --- .../durability/raft/cell_groups/README.md | 50 +++-- .../durability/raft/cell_groups/enrollment.rs | 114 ++++++---- .../raft/cell_groups/enrollment_apply.rs | 120 ++++++++++ .../durability/raft/cell_groups/manifest.rs | 17 +- .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/page_store.rs | 7 + .../durability/raft/cell_groups/page_wire.rs | 7 + .../raft/cell_groups/page_wire_tests.rs | 132 +++++++++++ .../durability/raft/cell_groups/pins.rs | 5 +- .../durability/raft/cell_groups/root.rs | 25 +++ .../tests/native_cell_groups_storage.rs | 143 ++++++++++-- .../tests/support/cell_group_recovery.rs | 208 +++++++++++++++++- plans/multi-raft/cell-groups/README.md | 38 ++-- plans/multi-raft/cell-groups/design.md | 6 +- .../cell-groups/implementation-plan.md | 4 +- plans/multi-raft/cell-groups/progress.md | 54 +++-- plans/multi-raft/cell-groups/tasks.json | 12 +- 17 files changed, 815 insertions(+), 129 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/enrollment_apply.rs create mode 100644 crates/cellar/durability/raft/cell_groups/page_wire_tests.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 8d0249db8..290d0721d 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -15,7 +15,9 @@ JSON and a 16 KiB decode bound. Tree depth and cell count are explicit limits. These limits are admission ceilings, not qualified fleet capacities. Leaves bind logical cell history, mutation sequence, image commitment and -geometry, last group mutation position, schema and effect summary. Local image +geometry, last group mutation position, schema, effect summary and immutable +original enrollment identity. A page without the required enrollment field +refuses at marker 1; no old-layout decoder or inferred creation record is used. Local image filenames are excluded: replicas may store the same image under different local names and still derive the same logical manifest. A physical image adapter must resolve and pin each image by the selected logical leaf. Extent and materialized @@ -113,7 +115,7 @@ preparation. These are storage-boundary checks, not quorum, SDK deduplication or RustFS qualification. The full CG-02 source-bound evidence gate is recorded in the delivery plan. -## Enrollment admission (CG-03 in progress) +## Enrollment admission and ordered preparation `decide_enrollment` checks a bounded cell-addressed enrollment entry against the current group route and committed lifecycle state. Sealed and retired ranges @@ -126,13 +128,33 @@ preparation-time check is not permission to bypass a later seal. `EnrollmentRecord` binds logical cell history, original creation operation, schema and initial image. It deliberately excludes physical group placement, so normal transfer preserves the record while the request adopts the current route. -Its bounded canonical marker-1 codec verifies data shape, not authority. CG-04 -must publish this record atomically with enrollment, retain it through mutations, -and include it in complete snapshots/archives. An exact retry must not initialize -again or roll the current image back to the initial one. Current-term quorum and -read authority remain required from the host even for a recognized retry. - -## Range directory (CG-03 in progress) +Its bounded canonical marker-1 codec verifies data shape, not authority. The +record is now required in each authenticated manifest leaf and published with +the selected image. Ordinary mutation refuses any changed creation record. +The record matches the cell history/schema; sequence one must select its initial +image, and later sequences must advance that image's LTX position. A normal +transfer carries the same record. It records an initial image commitment without +pinning that superseded image forever. Complete snapshots and archives must +retain this metadata as part of their selected manifest closure. + +`prepare_enrollment` rechecks a bounded enrollment entry against the preceding +complete root and the caller's committed lifecycle phase. It verifies the next +contiguous group position before invoking the physical image producer. A new +creation prepares sequence one and its image seal together. An exact retry does +not invoke that producer or replace the latest image; `prepare_unchanged` advances +only the group position, sharing the same manifest and image pins without page +I/O. A quarantined page store refuses even this unchanged-root path. Competing creations refuse before physical preparation. The engine checks +the exact previous branch and committed position before publication. The future +shared owner must consume this boundary only for the actual ordered log entry; +preparation is not a quorum proof or command acknowledgement. + +Current-term quorum and read authority remain required even for a recognized +retry. Shared RawNode ownership, real concurrent enrollment and business callback +settlement remain CG-04 work. The storage fixtures exercise repeated enrollment +after mutation, restart and reclamation of the initial image in both physical +image formats; they do not qualify a live shared host. + +## Range directory `RangeDirectory` stores a namespace-bound, immutable AVL range tree. Each page binds its route, child hashes, subtree bounds, height, count and coverage. A lookup @@ -175,10 +197,10 @@ the seal, installation, archive and activation proofs before exposing replacemen ownership. Birth/claim admission and invocation-bounded route confirmation are described below. Managed admission and production client integration remain open. Native directory tests exercise real conditional object-store operations in memory, including races, dropped replies, corruption, -restart and bounded split/merge churn. They do not qualify live Raft ownership, -real RustFS or the full CG-03 gate. +restart and bounded split/merge churn. They do not qualify live Raft ownership or real RustFS. CG-03 combines these +checks with the separately retained physical-bootstrap and routing evidence. -## Cell-independent group birth (CG-03 in progress) +## Cell-independent group birth `CellGroupBirthSpec` binds physical group history, range, route epoch, original creation operation and initial members. It has no single-cell ownership field; @@ -207,8 +229,8 @@ These metadata methods expose no voter-start capability and write no permanent member disk claim. The bootstrap path below connects the verified birth to real engine genesis and permanent claims. Managed namespaces refuse these unmanaged APIs until their rooted fleet authorization is -connected. This staged boundary does not permit blank-disk recovery, replace -current quorum membership, or complete CG-03 qualification. +connected. This staged boundary does not permit blank-disk recovery or replace +current quorum membership. ## Physical log identity cutover diff --git a/crates/cellar/durability/raft/cell_groups/enrollment.rs b/crates/cellar/durability/raft/cell_groups/enrollment.rs index 49316e44c..e41cf1c13 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment.rs @@ -47,7 +47,8 @@ impl std::error::Error for EnrollmentError {} /// must publish it atomically with the initial cell image and carry it through /// snapshots/archives; the latest image cannot stand in for this original image. /// Decoding only checks shape and canonical encoding, not commitment authority. -#[derive(Clone, Debug, PartialEq, Eq)] +#[derive(Clone, Debug, PartialEq, Eq, Deserialize)] +#[serde(try_from = "WireRecord")] pub struct EnrollmentRecord { cell: CellHistory, creation: ProposalId, @@ -56,6 +57,24 @@ pub struct EnrollmentRecord { } impl EnrollmentRecord { + /// Construct bounded creation data. This grants no admission or commitment; + /// the owner must still validate the replicated enrollment against its route. + pub fn new( + cell: CellHistory, + creation: ProposalId, + schema: [u8; 32], + initial: CellImage, + ) -> Result { + let record = Self { + cell, + creation, + schema, + initial, + }; + record.encode()?; + Ok(record) + } + pub fn cell(&self) -> &CellHistory { &self.cell } @@ -80,20 +99,7 @@ impl EnrollmentRecord { { return Err(EnrollmentError::InvalidRecord); } - let wire = WireRecord { - domain: RECORD_DOMAIN.into(), - cluster: self.cell.cluster.bytes(), - namespace: self.cell.namespace.bytes(), - cell: self.cell.cell.as_str().into(), - incarnation: self.cell.incarnation.bytes(), - creation: self.creation.bytes(), - schema: self.schema, - initial_epoch: image.epoch, - initial_txid: image.txid, - initial_checksum: image.checksum, - initial_commitment: image.commitment, - }; - let bytes = serde_json::to_vec(&wire).map_err(|_| EnrollmentError::InvalidRecord)?; + let bytes = serde_json::to_vec(self).map_err(|_| EnrollmentError::InvalidRecord)?; if bytes.len() > MAX_RECORD_BYTES { return Err(EnrollmentError::Limit); } @@ -104,31 +110,8 @@ impl EnrollmentRecord { if bytes.len() > MAX_RECORD_BYTES { return Err(EnrollmentError::Limit); } - let wire: WireRecord = + let record: Self = serde_json::from_slice(bytes).map_err(|_| EnrollmentError::InvalidRecord)?; - if wire.domain != RECORD_DOMAIN { - return Err(EnrollmentError::InvalidRecord); - } - let record = Self { - cell: CellHistory { - cluster: ClusterId::new(wire.cluster) - .map_err(|_| EnrollmentError::InvalidRecord)?, - namespace: NamespaceId::new(wire.namespace) - .map_err(|_| EnrollmentError::InvalidRecord)?, - cell: CellId::new(wire.cell).map_err(|_| EnrollmentError::InvalidRecord)?, - incarnation: IncarnationId::new(wire.incarnation) - .map_err(|_| EnrollmentError::InvalidRecord)?, - }, - creation: ProposalId::new(wire.creation).map_err(|_| EnrollmentError::InvalidRecord)?, - schema: wire.schema, - initial: LtxImage { - epoch: wire.initial_epoch, - txid: wire.initial_txid, - checksum: wire.initial_checksum, - commitment: wire.initial_commitment, - } - .into(), - }; if record.encode()? != bytes { return Err(EnrollmentError::InvalidRecord); } @@ -200,3 +183,56 @@ struct WireRecord { initial_checksum: u64, initial_commitment: ImageCommitment, } + +impl TryFrom for EnrollmentRecord { + type Error = EnrollmentError; + + fn try_from(wire: WireRecord) -> Result { + if wire.domain != RECORD_DOMAIN { + return Err(EnrollmentError::InvalidRecord); + } + let record = EnrollmentRecord { + cell: CellHistory { + cluster: ClusterId::new(wire.cluster) + .map_err(|_| EnrollmentError::InvalidRecord)?, + namespace: NamespaceId::new(wire.namespace) + .map_err(|_| EnrollmentError::InvalidRecord)?, + cell: CellId::new(wire.cell).map_err(|_| EnrollmentError::InvalidRecord)?, + incarnation: IncarnationId::new(wire.incarnation) + .map_err(|_| EnrollmentError::InvalidRecord)?, + }, + creation: ProposalId::new(wire.creation).map_err(|_| EnrollmentError::InvalidRecord)?, + schema: wire.schema, + initial: LtxImage { + epoch: wire.initial_epoch, + txid: wire.initial_txid, + checksum: wire.initial_checksum, + commitment: wire.initial_commitment, + } + .into(), + }; + record.encode()?; + Ok(record) + } +} + +impl Serialize for EnrollmentRecord { + fn serialize(&self, serializer: S) -> Result { + let record = self; + let image = LtxImage::try_from(record.initial).map_err(serde::ser::Error::custom)?; + WireRecord { + domain: RECORD_DOMAIN.into(), + cluster: record.cell.cluster.bytes(), + namespace: record.cell.namespace.bytes(), + cell: record.cell.cell.as_str().into(), + incarnation: record.cell.incarnation.bytes(), + creation: record.creation.bytes(), + schema: record.schema, + initial_epoch: record.initial.epoch, + initial_txid: record.initial.txid, + initial_checksum: record.initial.checksum, + initial_commitment: image.commitment, + } + .serialize(serializer) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs new file mode 100644 index 000000000..e5bd68f8f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs @@ -0,0 +1,120 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Ordered enrollment preparation against one coherent group root. Publication +//! still belongs to the engine and requires the actual committed log position. + +use super::{ + decide_enrollment, CellImageGeometry, CellImagePin, CellManifestLeaf, EffectSummary, + EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, ManifestError, + ManifestPageStore, ManifestTree, PreparedGroupManifest, +}; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryLimits, +}; +use cellar_logic::{cell_group::*, consensus::LogPosition}; + +pub struct EnrollmentApply<'a> { + pub position: LogPosition, + pub phase: EnrollmentPhase, + pub entry: &'a [u8], + pub limits: GroupEntryLimits, +} + +/// Independently prepared and synced initial image, returned only for a new +/// enrollment. A repeated creation must never invoke its image producer again. +pub struct EnrollmentImage { + pub geometry: CellImageGeometry, + pub effects: EffectSummary, + pub pin: CellImagePin, +} + +#[derive(Debug)] +pub enum EnrollmentApplyError { + Enrollment(EnrollmentError), + Manifest(ManifestError), +} + +impl std::fmt::Display for EnrollmentApplyError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Enrollment(error) => error.fmt(f), + Self::Manifest(error) => error.fmt(f), + } + } +} + +impl std::error::Error for EnrollmentApplyError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Enrollment(error) => Some(error), + Self::Manifest(error) => Some(error), + } + } +} + +impl From for EnrollmentApplyError { + fn from(error: EnrollmentError) -> Self { + Self::Enrollment(error) + } +} + +impl From for EnrollmentApplyError { + fn from(error: ManifestError) -> Self { + Self::Manifest(error) + } +} + +impl ManifestTree { + /// Recheck enrollment against the complete preceding root at ordered apply. + /// Exactly matching retries advance the group prefix while retaining the + /// current image, original creation and cell sequence. Conflicts or invalid + /// prefixes refuse before invoking the physical image producer. + /// + /// Run this synchronous preparation on the owning blocking execution domain. + /// The returned decision is not an acknowledgement: publish its manifest + /// through `persist_cells` only for the corresponding committed entry. + pub fn prepare_enrollment( + &self, + store: &mut ManifestPageStore, + previous: &PreparedGroupManifest, + apply: EnrollmentApply<'_>, + image: impl FnOnce(&EnrollmentRecord) -> Result, + ) -> Result<(PreparedGroupManifest, EnrollmentDecision), EnrollmentApplyError> { + // Validate route, store ownership and contiguous position before image + // I/O. The unchanged candidate also supplies the idempotent apply path. + let unchanged = self.prepare_unchanged(store, previous, apply.position)?; + let entry = CellGroupEntry::decode(apply.entry, &self.route, apply.limits) + .map_err(EnrollmentError::Protocol)?; + let GroupCommand::Enroll { cell, .. } = entry.command else { + return Err(EnrollmentError::NotEnrollment.into()); + }; + let committed = previous.pin_cell(&cell.cell); + let decision = decide_enrollment( + &self.route, + apply.phase, + apply.entry, + committed.as_ref().map(|(leaf, _)| &leaf.enrollment), + apply.limits, + )?; + match &decision { + EnrollmentDecision::Existing(_) => Ok((unchanged, decision)), + EnrollmentDecision::Propose(record) => { + let prepared = image(record)?; + let leaf = CellManifestLeaf { + enrollment: record.clone(), + version: CellVersion { + history: record.cell().clone(), + sequence: MutationSequence::new(1) + .map_err(|_| ManifestError::Invalid("initial sequence"))?, + image: record.initial(), + }, + geometry: prepared.geometry, + last_mutation: apply.position, + effects: prepared.effects, + }; + let next = self.prepare_update(store, previous, leaf, None, prepared.pin)?; + Ok((next, decision)) + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 1c2cc20fd..26da49243 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -89,6 +89,9 @@ pub struct EffectSummary { #[derive(Clone, Debug, PartialEq, Eq)] pub struct CellManifestLeaf { + /// Original creation identity, retained atomically with every selected image. + /// Mutations and ordinary group transfers must preserve this record exactly. + pub enrollment: super::EnrollmentRecord, pub version: CellVersion, pub geometry: CellImageGeometry, pub last_mutation: LogPosition, @@ -107,6 +110,17 @@ impl CellManifestLeaf { if self.last_mutation.index.get() == 0 { return Err(ManifestError::Invalid("zero mutation index")); } + // The immutable record's constructor/decoder validates its bounded shape. + // Do not allocate its canonical encoding on every tree lookup. + let initial = self.enrollment.initial(); + if self.enrollment.cell() != &self.version.history + || self.enrollment.schema() != route.schema_digest + || initial.epoch != self.version.image.epoch + || (self.version.sequence.get() == 1 && initial != self.version.image) + || (self.version.sequence.get() > 1 && initial.txid >= self.version.image.txid) + { + return Err(ManifestError::Invalid("enrollment/image binding")); + } let image = LtxImage::try_from(self.version.image) .map_err(|_| ManifestError::Invalid("image commitment"))?; if !cellar_ltx::ltx::is_valid_page_size(self.geometry.page_size) @@ -217,7 +231,8 @@ impl ManifestTree { } } (Some(old), Some(expected)) if &old.version == expected => { - if old.version.history != leaf.version.history + if old.enrollment != leaf.enrollment + || old.version.history != leaf.version.history || old.version.sequence.checked_next().ok() != Some(leaf.version.sequence) || leaf.version.image.epoch != old.version.image.epoch || leaf.version.image.txid <= old.version.image.txid diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 215207666..aff100236 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -3,6 +3,7 @@ //! Complete group-state storage. A synced page is data, not Raft authority. mod enrollment; +mod enrollment_apply; mod image_pin; mod manifest; mod page_store; @@ -26,6 +27,7 @@ pub use crate::raft::catalog::{ pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, }; +pub use enrollment_apply::{EnrollmentApply, EnrollmentApplyError, EnrollmentImage}; pub use image_pin::{cell_image_history_digest, CellImagePin}; pub(crate) use pins::ManifestPins; pub use root::{GroupManifest, PreparedGroupManifest}; diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs index 51dc6cc54..3fd4f3c8d 100644 --- a/crates/cellar/durability/raft/cell_groups/page_store.rs +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -60,6 +60,13 @@ pub struct ManifestReclaimReport { } impl ManifestPageStore { + pub(super) fn require_live(&self) -> Result<(), ManifestError> { + if self.quarantined { + return Err(ManifestError::Quarantined); + } + Ok(()) + } + pub fn open( directory: &Path, filesystem: Arc, diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs index 0dc08e9b3..5605e1708 100644 --- a/crates/cellar/durability/raft/cell_groups/page_wire.rs +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -8,6 +8,10 @@ use cellar_logic::{cell_group::*, consensus::*}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; +#[cfg(test)] +#[path = "page_wire_tests.rs"] +mod tests; + const MAGIC: &[u8; 8] = b"CLRCGM01"; pub(super) const MAX_PAGE_BYTES: usize = 16 * 1024; @@ -67,6 +71,7 @@ impl Page { #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct WirePage { + enrollment: super::EnrollmentRecord, schema: [u8; 32], cluster: [u8; 16], namespace: [u8; 16], @@ -101,6 +106,7 @@ pub(super) fn encode( let leaf = &page.leaf; let history = &leaf.version.history; let wire = WirePage { + enrollment: leaf.enrollment.clone(), schema: route.schema_digest, cluster: history.cluster.bytes(), namespace: history.namespace.bytes(), @@ -163,6 +169,7 @@ pub(super) fn decode( }; let page = Page { leaf: CellManifestLeaf { + enrollment: wire.enrollment, version: CellVersion { history, sequence: MutationSequence::new(wire.sequence) diff --git a/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs new file mode 100644 index 000000000..a304ebd74 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs @@ -0,0 +1,132 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::EnrollmentRecord; + +fn fixture() -> (GroupRoute, Page) { + let history = CellHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + cell: CellId::new("cell").unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let route = GroupRoute { + history: GroupHistory { + cluster: history.cluster, + namespace: history.namespace, + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [7; 32], + }; + let initial = CellImage { + epoch: 1, + txid: 1, + checksum: 0, + commitment: CellImageCommitment::Materialized([8; 32]), + }; + let enrollment = EnrollmentRecord::new( + history.clone(), + ProposalId::new([9; 16]).unwrap(), + route.schema_digest, + initial, + ) + .unwrap(); + ( + route, + Page { + leaf: CellManifestLeaf { + enrollment, + version: CellVersion { + history, + sequence: MutationSequence::new(1).unwrap(), + image: initial, + }, + geometry: CellImageGeometry { + length: 512, + page_size: 512, + }, + last_mutation: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + effects: EffectSummary::default(), + }, + left: None, + right: None, + }, + ) +} + +#[test] +fn creation_record_is_required_in_marker_one_pages() { + let (route, page) = fixture(); + let limits = ManifestLimits::default(); + let encoded = encode(&page, &route, limits).unwrap(); + assert_eq!(decode(&encoded, &route, limits).unwrap().leaf, page.leaf); + let original = String::from_utf8(encoded[MAGIC.len()..].to_vec()).unwrap(); + let record = String::from_utf8(page.leaf.enrollment.encode().unwrap()).unwrap(); + let replace = |field: &str, value: serde_json::Value| { + let parsed: serde_json::Value = serde_json::from_str(&record).unwrap(); + record.replacen( + &format!("\"{field}\":{}", parsed[field]), + &format!("\"{field}\":{value}"), + 1, + ) + }; + // Keep every unaffected field in canonical order. A reordered JSON map + // must not be the reason these malformed identities refuse. + for changed in [ + None, + Some("null".to_owned()), + Some(replace("cell", "another-cell".into())), + Some(replace("schema", serde_json::json!(vec![0; 32]))), + Some(replace("initial_txid", 2.into())), + Some(replace("creation", serde_json::json!(vec![0; 16]))), + ] { + let changed = match changed { + Some(value) => original.replacen(&record, &value, 1), + None => original.replacen(&format!("\"enrollment\":{record},"), "", 1), + }; + assert_ne!(changed, original); + let mut bytes = MAGIC.to_vec(); + bytes.extend(changed.as_bytes()); + assert!(decode(&bytes, &route, limits).is_err()); + } +} + +#[test] +fn creation_binding_survives_later_versions_and_bounds_all_cell_names() { + let (route, mut page) = fixture(); + let original = page.leaf.enrollment.clone(); + page.leaf.version.sequence = MutationSequence::new(2).unwrap(); + assert!(encode(&page, &route, ManifestLimits::default()).is_err()); + page.leaf.version.image.txid = 2; + page.leaf.version.image.commitment = CellImageCommitment::Materialized([10; 32]); + page.leaf.last_mutation.index = LogIndex::new(3); + let bytes = encode(&page, &route, ManifestLimits::default()).unwrap(); + let restored = decode(&bytes, &route, ManifestLimits::default()).unwrap(); + assert_eq!(restored.leaf.enrollment, original); + assert_eq!(restored.leaf.version, page.leaf.version); + + let id = CellId::new("\\".repeat(MAX_CELL_ID_BYTES)).unwrap(); + page.leaf.version.history.cell = id; + page.leaf.enrollment = EnrollmentRecord::new( + page.leaf.version.history.clone(), + original.creation(), + original.schema(), + original.initial(), + ) + .unwrap(); + let bytes = encode(&page, &route, ManifestLimits::default()).unwrap(); + assert!(bytes.len() <= MAX_PAGE_BYTES); + assert_eq!( + decode(&bytes, &route, ManifestLimits::default()) + .unwrap() + .leaf, + page.leaf + ); +} diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs index 64c80d042..a3941c6ca 100644 --- a/crates/cellar/durability/raft/cell_groups/pins.rs +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -38,10 +38,10 @@ impl ManifestPins { cells as usize * (std::mem::size_of::() + 2 * std::mem::size_of::() - + 2 * MAX_CELL_ID_BYTES) + + 3 * MAX_CELL_ID_BYTES) } - /// Conservatively charge shared subtrees to every owner. Include both cell + /// Conservatively charge shared subtrees to every owner. Include all three cell /// ID allocations and Arc control blocks, not just the image handle size. pub(crate) fn reserved_bytes(&self) -> usize { self.0.as_ref().map_or(0, |p| p.bytes) @@ -76,6 +76,7 @@ impl ManifestPins { + 2 * std::mem::size_of::() + key.cell().as_str().len() + leaf.version.history.cell.as_str().len() + + leaf.enrollment.cell().cell.as_str().len() + left.reserved_bytes() + right.reserved_bytes(); Self(Some(Arc::new(PinnedPage { diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index f91451824..cd809267c 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -123,6 +123,31 @@ impl PreparedGroupManifest { } impl ManifestTree { + /// Advance one committed no-op or duplicate enrollment without changing any + /// cell version. The owner must supply the actual next committed position; + /// this preparation alone proves neither commitment nor serving authority. + pub fn prepare_unchanged( + &self, + store: &ManifestPageStore, + previous: &PreparedGroupManifest, + position: LogPosition, + ) -> Result { + store.require_live()?; + if previous.marker.route != self.route || !Arc::ptr_eq(&previous.page_owner, &store.owner) { + return Err(ManifestError::Invalid("root/page owner")); + } + if previous.marker.position.index.get().checked_add(1) != Some(position.index.get()) + || position.term < previous.marker.position.term + { + return Err(ManifestError::Invalid("noncontiguous no-op")); + } + let mut prepared = previous.clone(); + prepared.base = Some(previous.marker.clone()); + prepared.marker.position = position; + prepared.marker.check()?; + Ok(prepared) + } + /// Full closure verification for bootstrap, recovery and checkpoint install. /// Ordinary cell mutations use `prepare_update` instead. /// The callback returns actual sealed images, never merely accepted markers. diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index 189f7b24b..a66a00ead 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -84,23 +84,31 @@ fn leaf(id: u8, pin: &GenerationPin, sequence: u64, index: u64) -> CellManifestL checksum: state.checksum, page_size: state.page_size, }; - CellManifestLeaf { - version: CellVersion { - history: CellHistory { - cluster: route().history.cluster, - namespace: route().history.namespace, - cell: CellId::new(format!("cell-{id}")).unwrap(), - incarnation: IncarnationId::new([7; 16]).unwrap(), - }, - sequence: MutationSequence::new(sequence).unwrap(), - image: LtxImage { - epoch: image.ltx_epoch, - txid: image.txid, - checksum: image.checksum, - commitment: image.commitment, - } - .into(), + let version = CellVersion { + history: CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new(format!("cell-{id}")).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), }, + sequence: MutationSequence::new(sequence).unwrap(), + image: LtxImage { + epoch: image.ltx_epoch, + txid: image.txid, + checksum: image.checksum, + commitment: image.commitment, + } + .into(), + }; + CellManifestLeaf { + enrollment: EnrollmentRecord::new( + version.history.clone(), + ProposalId::new([id; 16]).unwrap(), + route().schema_digest, + version.image, + ) + .unwrap(), + version, geometry: CellImageGeometry { length: image.length, page_size: image.page_size, @@ -153,7 +161,8 @@ fn manifest_updates_only_changed_path() -> Result<()> { let after = images::image(Some(&before), "UPDATE value SET n = 99;")?; let cut = images::Cut::new(&before, &after, 1, true, false); let changed = generations.prepare(images::id(100), Some(pin), &cut.update())?; - let value = leaf(32, &changed, 2, 65); + let mut value = leaf(32, &changed, 2, 65); + value.enrollment = old.enrollment.clone(); let written = pages.usage().written_pages; root = tree.update(&mut pages, root, value.clone(), Some(&old.version))?; assert!(pages.usage().written_pages - written <= u64::from(old_root.depth())); @@ -208,6 +217,29 @@ fn manifest_rejects_conflicts_bounds_and_missing_pages() -> Result<()> { )?; let root = tree.update(&mut pages, ManifestRoot::default(), value.clone(), None)?; let before = pages.usage(); + for replace_initial in [false, true] { + let mut changed = value.clone(); + changed.version.sequence = MutationSequence::new(2)?; + changed.version.image.txid = 2; + changed.last_mutation = position(2); + let mut initial = value.enrollment.initial(); + let creation = if replace_initial { + initial.commitment = CellImageCommitment::Materialized([99; 32]); + value.enrollment.creation() + } else { + ProposalId::new([99; 16])? + }; + changed.enrollment = EnrollmentRecord::new( + value.version.history.clone(), + creation, + route().schema_digest, + initial, + )?; + assert!(matches!( + tree.update(&mut pages, root, changed, Some(&value.version)), + Err(ManifestError::Conflict) + )); + } assert!(tree.update(&mut pages, root, value.clone(), None).is_err()); assert!(tree .update(&mut pages, root, value.clone(), Some(&value.version)) @@ -680,7 +712,8 @@ async fn incremental_manifest_preserves_images_and_rejects_forks() -> Result<()> let after = images::image(Some(&before), "UPDATE value SET n = 999;")?; let cut = images::Cut::new(&before, &after, 1, true, false); let changed = generations.prepare(images::id(100), Some(&reader), &cut.update())?; - let next_leaf = leaf(32, &changed, 2, 3); + let mut next_leaf = leaf(32, &changed, 2, 3); + next_leaf.enrollment = old_leaf.enrollment.clone(); let written = pages.usage().written_pages; let reads = trace.reads.load(std::sync::atomic::Ordering::Acquire); trace.synced_paths.lock().unwrap().clear(); @@ -1069,7 +1102,10 @@ async fn uncertain_manifest_publication_retains_both_roots() -> Result<()> { let next = tree.prepare_update( &mut pages, ¤t, - leaf(1, &next_pin, 2, 3), + CellManifestLeaf { + enrollment: old_leaf.enrollment.clone(), + ..leaf(1, &next_pin, 2, 3) + }, Some(&old_leaf.version), CellImagePin::Materialized(next_pin.clone()), )?; @@ -1289,3 +1325,72 @@ async fn manifest_batch_retention_refusal_cancels_unwritten_reservations() -> Re domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn unchanged_group_root_refuses_quarantined_page_store() -> Result<()> { + use cellar::application::consensus::verification::{ + CellGroupBootstrap, LogOpenMode, ReplicaLog, + }; + use cellar::sdk::ExecutionDomain; + let temp = tempfile::tempdir()?; + let fs = Arc::new(DirectFileSystem); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + engine_options(&temp.path().join("node"), LogOpenMode::Create, 1), + fs.clone(), + )) + .await??; + let group = log + .create_cell_group(CellGroupBootstrap { + route: route(), + member: MemberId::new(1)?, + voters: vec![MemberId::new(1)?], + learners: vec![], + }) + .await?; + let page_path = temp.path().join("pages"); + let fault = faults::Faults::new(page_path.clone(), None); + let page_fs = Arc::new(faults::FaultFileSystem { + faults: fault.clone(), + }); + let mut pages = ManifestPageStore::open(&page_path, page_fs, 64, 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = tree.prepare_complete( + &group, + &mut pages, + ManifestRoot::default(), + position(1), + 1, + |_| panic!("genesis is empty"), + )?; + assert!(tree + .prepare_unchanged(&pages, ¤t, position(2)) + .is_ok()); + let generations = + GenerationStore::open(temp.path().join("images"), fs, GenerationLimits::default())?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let pin = generations.prepare(images::id(1), None, &cut.update())?; + fault.mode.store( + faults::AFTER_IMAGE_WRITE, + std::sync::atomic::Ordering::Release, + ); + assert!(tree + .prepare_update( + &mut pages, + ¤t, + leaf(1, &pin, 1, 2), + None, + CellImagePin::Materialized(pin) + ) + .is_err()); + assert!(matches!( + tree.prepare_unchanged(&pages, ¤t, position(2)), + Err(ManifestError::Quarantined) + )); + assert_eq!(group.group_manifest()?.unwrap().position(), position(1)); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_recovery.rs b/crates/cellar/tests/support/cell_group_recovery.rs index 6ff45ee2b..92009e632 100644 --- a/crates/cellar/tests/support/cell_group_recovery.rs +++ b/crates/cellar/tests/support/cell_group_recovery.rs @@ -127,13 +127,24 @@ impl Images { fn prepare( &self, cell: u8, - base: Option<&CellImagePin>, + base: Option<(&CellManifestLeaf, &CellImagePin)>, cut: &images::Cut, after: &[u8], index: u64, ) -> Result<(CellManifestLeaf, CellImagePin)> { let state = cut.result; let mut leaf = CellManifestLeaf { + enrollment: EnrollmentRecord::new( + history(cell), + ProposalId::new([cell; 16])?, + route().schema_digest, + CellImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: CellImageCommitment::Materialized(state.digest), + }, + )?, version: CellVersion { history: history(cell), sequence: MutationSequence::new(state.txid.0)?, @@ -151,6 +162,8 @@ impl Images { last_mutation: position(index), effects: EffectSummary::default(), }; + let previous = base.map(|(leaf, _)| leaf.enrollment.clone()); + let base = base.map(|(_, pin)| pin); let pin = match self { Self::Materialized(store) => { let base = match base { @@ -211,6 +224,15 @@ impl Images { CellImagePin::Extents(Arc::new(prepared.image)) } }; + leaf.enrollment = match previous { + Some(record) => record, + None => EnrollmentRecord::new( + history(cell), + ProposalId::new([cell; 16])?, + route().schema_digest, + leaf.version.image, + )?, + }; Ok((leaf, pin)) } @@ -385,12 +407,31 @@ async fn seed(path: &Path, mode: Mode, large: bool) -> Result { let bytes = initial(cell, large && cell == 1)?; let cut = images::Cut::new(&[], &bytes, 0, true, false); let (leaf, pin) = images.prepare(cell, None, &cut, &bytes, u64::from(cell) + 1)?; - prepared = tree.prepare_update(&mut pages, &prepared, leaf, None, pin)?; - log.persist_cells( - OperationId::new(u64::from(cell)), - vec![root_write(&group, prepared.clone())?], - ) - .await?; + let entry = enrollment_entry(&leaf.enrollment, &cut)?; + let (next, decision) = tree.prepare_enrollment( + &mut pages, + &prepared, + EnrollmentApply { + position: leaf.last_mutation, + phase: EnrollmentPhase::Active, + entry: &entry, + limits: GroupEntryLimits::default(), + }, + |record| { + assert_eq!(record, &leaf.enrollment); + Ok(EnrollmentImage { + geometry: leaf.geometry, + effects: leaf.effects, + pin, + }) + }, + )?; + assert!(matches!(decision, EnrollmentDecision::Propose(_))); + prepared = next; + let mut write = root_write(&group, prepared.clone())?; + write.entries[0].data = entry.into(); + log.persist_cells(OperationId::new(u64::from(cell)), vec![write]) + .await?; } let marker = prepared.marker().clone(); log.close().await?; @@ -428,7 +469,7 @@ pub async fn both_images() -> Result<()> { let before = read_image(&old_pin)?; let after = mutate(&before)?; let cut = images::Cut::new(&before, &after, 1, true, false); - let (changed, next_pin) = images.prepare(1, Some(&old_pin), &cut, &after, 4)?; + let (changed, next_pin) = images.prepare(1, Some((old, &old_pin)), &cut, &after, 4)?; if let (CellImagePin::Extents(old), CellImagePin::Extents(next)) = (&old_pin, &next_pin) { assert!(old.manifest().geometry().extent_count() > 1); @@ -450,6 +491,8 @@ pub async fn both_images() -> Result<()> { domain.shutdown().await; } recover_and_check(temp.path(), mode, Some(4)).await?; + replay_enrollment(temp.path(), mode).await?; + recover_and_check(temp.path(), mode, Some(5)).await?; } Ok(()) } @@ -469,7 +512,9 @@ async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Re if let Some(expected) = expected { assert_eq!(index, expected); } - assert!([3, 4].contains(&index)); + if expected != Some(5) { + assert!([3, 4].contains(&index)); + } assert_eq!(marker.root().cells(), 2); let images = Images::open(&path.join("images"), mode, fs.clone())?; let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; @@ -478,8 +523,11 @@ async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Re let mut leaves = Vec::new(); for cell in 1..=2 { let (leaf, pin) = restored.pin_cell(&history(cell).cell).unwrap(); - let sequence = if cell == 1 && index == 4 { 2 } else { 1 }; + let sequence = if cell == 1 && index >= 4 { 2 } else { 1 }; assert_eq!(leaf.version.history, history(cell)); + assert_eq!(leaf.enrollment.cell(), &history(cell)); + assert_eq!(leaf.enrollment.creation(), ProposalId::new([cell; 16])?); + assert_eq!(leaf.enrollment.initial().txid, 1); assert_eq!(leaf.version.sequence.get(), sequence); assert_receipt(&pin, cell, sequence)?; leaves.push(leaf.clone()); @@ -757,7 +805,7 @@ pub async fn process_fixture() -> Result<()> { }; image_fault.mode.store(select("image-"), Ordering::Release); let cut = images::Cut::new(&before, &after, 1, true, false); - let (changed, pin) = images.prepare(1, Some(&old_pin), &cut, &after, 4)?; + let (changed, pin) = images.prepare(1, Some((old, &old_pin)), &cut, &after, 4)?; if cut_name == "after-image" { image_fault.pause()?; } @@ -789,3 +837,141 @@ pub async fn process_fixture() -> Result<()> { } bail!("requested cut was never reached: {cut_name}") } + +fn enrollment_entry(record: &EnrollmentRecord, cut: &images::Cut) -> Result> { + let route = route(); + Ok(CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: record.creation(), + }, + command: GroupCommand::Enroll { + cell: record.cell().clone(), + image: LtxImage::try_from(record.initial())?, + files: vec![GroupLtx { + min_txid: 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }], + }, + } + .encode(&route, GroupEntryLimits::default())?) +} + +async fn replay_enrollment(path: &Path, mode: Mode) -> Result<()> { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let fs = Arc::new(DirectFileSystem); + let log = domain + .run(ReplicaLog::open( + engine_options(&path.join("node"), LogOpenMode::Resume, 4), + fs.clone(), + )) + .await??; + let group = log.cell_group(&route().history)?; + let images = Images::open(&path.join("images"), mode, fs.clone())?; + let mut pages = ManifestPageStore::open(&path.join("pages"), fs, 256, 4 * 1024 * 1024, 16)?; + let tree = ManifestTree::new(route(), ManifestLimits::default())?; + let current = prepare_root( + &tree, + &group, + &mut pages, + &group.group_manifest()?.context("selected root")?, + &images, + )?; + let (old, old_pin) = current.pin_cell(&history(1).cell).unwrap(); + assert_eq!(old.version.sequence.get(), 2); + assert_eq!(old.enrollment.initial().txid, 1); + assert_ne!(old.enrollment.initial(), old.version.image); + let initial_leaf = CellManifestLeaf { + version: CellVersion { + sequence: MutationSequence::new(1)?, + image: old.enrollment.initial(), + ..old.version.clone() + }, + ..old.clone() + }; + assert!( + images.recover(&initial_leaf).is_err(), + "the original image has been reclaimed; creation identity must survive independently" + ); + let bytes = initial(1, true)?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let entry = enrollment_entry(&old.enrollment, &cut)?; + let before = pages.usage().written_pages; + let apply = |phase, index| EnrollmentApply { + position: position(index), + phase, + entry: &entry, + limits: GroupEntryLimits::default(), + }; + for phase in [EnrollmentPhase::Sealed, EnrollmentPhase::Retired] { + assert!(tree + .prepare_enrollment(&mut pages, ¤t, apply(phase, 5), |_| panic!( + "sealed or retired retry cannot initialize" + )) + .is_err()); + } + for index in [4, 6] { + assert!(tree + .prepare_enrollment( + &mut pages, + ¤t, + apply(EnrollmentPhase::Active, index), + |_| panic!("invalid prefix cannot initialize") + ) + .is_err()); + } + let competing = EnrollmentRecord::new( + old.version.history.clone(), + ProposalId::new([99; 16])?, + route().schema_digest, + old.enrollment.initial(), + )?; + let competing_entry = enrollment_entry(&competing, &cut)?; + let error = tree.prepare_enrollment( + &mut pages, + ¤t, + EnrollmentApply { + entry: &competing_entry, + ..apply(EnrollmentPhase::Active, 5) + }, + |_| panic!("competing creation cannot initialize"), + ); + assert!(matches!( + error, + Err(EnrollmentApplyError::Enrollment(EnrollmentError::Conflict)) + )); + let (next, decision) = tree.prepare_enrollment( + &mut pages, + ¤t, + apply(EnrollmentPhase::Active, 5), + |_| panic!("duplicate must not rerun initialization"), + )?; + assert_eq!( + decision, + EnrollmentDecision::Existing(old.enrollment.clone()) + ); + assert_eq!(next.marker().root(), current.marker().root()); + assert_eq!(next.marker().position(), position(5)); + assert_eq!(pages.usage().written_pages, before); + for cell in 1..=2 { + assert_eq!( + next.pin_cell(&history(cell).cell).unwrap().0, + current.pin_cell(&history(cell).cell).unwrap().0 + ); + } + let mut write = root_write(&group, next.clone())?; + write.entries[0].data = entry.into(); + log.persist_cells(OperationId::new(7), vec![write]).await?; + assert_eq!(group.group_manifest()?.unwrap(), *next.marker()); + assert_receipt(&next.pin_cell(&history(1).cell).unwrap().1, 1, 2)?; + assert_receipt(&old_pin, 1, 2)?; + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 9ffcbbc73..7ae74eb43 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -3,15 +3,15 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 contracts and CG-02 storage have passed their local evidence gates. -CG-03 routing and enrollment is in progress. +CG-01 contracts, CG-02 storage and CG-03 routing/admission preparation have +passed their local evidence gates. CG-04 shared execution is in progress. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The requested deliverable is the executable design and implementation plan. -The target for its implementation executor remains the complete shared-group -runtime and hard cutover. Plan delivery does not qualify that runtime. +The plan handoff is complete. The active end-to-end implementation goal resumes +from that handoff and requires the complete shared-group runtime and hard cutover. +Plan delivery or an intermediate task gate does not qualify that runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -29,7 +29,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-03` emits a complete executor brief, source excerpts, + eligibility; `show CG-04` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -40,8 +40,9 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package This checkout retains evidence under the ignored `.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/` and -`.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` directories. A fresh checkout must -receive both retained directories at the same relative paths before validating +`.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` and +`.cellar/cell-groups-evidence/CG-03-routing-final/` directories. A fresh checkout must +receive all three retained directories at the same relative paths before validating the DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. The design, task briefs and validator unit tests are committed in this package. @@ -49,7 +50,7 @@ The design, task briefs and validator unit tests are committed in this package. python3 plans/multi-raft/cell-groups/plan.py validate python3 plans/multi-raft/cell-groups/plan.py preflight python3 plans/multi-raft/cell-groups/plan.py ready -python3 plans/multi-raft/cell-groups/plan.py show CG-03 +python3 plans/multi-raft/cell-groups/plan.py show CG-04 python3 -m unittest discover -s plans/multi-raft/cell-groups -p 'test_*.py' ``` @@ -68,13 +69,13 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The user selected a plan handoff for this delivery. The implementation executor -should continue from the existing code and retained evidence; no new runtime -work is required to read or review this package. +The active goal requires end-to-end runtime implementation. Continue from the +existing code and retained evidence; earlier plan-only handoffs do not satisfy +the resumed implementation objective. -Continue CG-03 after reading +Continue CG-04 after reading [progress](progress.md) and inspecting the working diff. -CG-01 and CG-02 are DONE only at their named contract/storage boundaries; the +CG-01 through CG-03 are DONE only at their named contract/storage/admission boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status @@ -83,12 +84,13 @@ to DONE. Continue in the generated dependency order afterward. Physical group genesis, typed replica handles, permanent initial-member claims, complete group inventory and invocation-bounded catalog-confirmed redirects are implemented through `0df009a2`; see the latest [checkpoint](progress.md) for -evidence and limits. Audit the retained CG-03 record against every task step -before marking it DONE. The record is at +evidence and limits. CG-03 has been audited at its preparation boundary. Its +record is at `.cellar/cell-groups-evidence/CG-03-routing-final/record.json`; transfer that entire directory along with the earlier DONE-task evidence to the implementation -executor. CG-04 then connects enrollment and the catalog startup permit to the -shared runtime, verifies the complete selected manifest/image closure and acquires current-term +executor. CG-04 is now implementing atomic creation retention and ordered +enrollment application. It must connect the catalog startup permit to the shared +runtime, verify the complete selected manifest/image closure and acquire current-term quorum authority. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/plans/multi-raft/cell-groups/design.md b/plans/multi-raft/cell-groups/design.md index c8178ac3e..06f85c061 100644 --- a/plans/multi-raft/cell-groups/design.md +++ b/plans/multi-raft/cell-groups/design.md @@ -148,8 +148,10 @@ cell's identity and base chain; batching is never a cross-cell transaction. The durable group root contains group history, applied position, routing/transition state, membership binding and an authenticated map of all cell image leaves. -Each leaf contains cell history/version, last group mutation position and effect -summary. Use a content-addressed paged tree with bounded copy-on-write paths, not +Each leaf contains cell history/version, immutable original enrollment identity, +last group mutation position and effect summary. Enrollment identity binds the +creation operation, original cell history, schema and initial image commitment; +it is retained even after that original image is reclaimed. Use a content-addressed paged tree with bounded copy-on-write paths, not an O(cell-count) full manifest rewrite for every update. An engine transaction publishes the durable root and contiguous apply position after referenced image and manifest pages are synced. Readers pin a published root and exact cell image. diff --git a/plans/multi-raft/cell-groups/implementation-plan.md b/plans/multi-raft/cell-groups/implementation-plan.md index d5d6d727d..44e777e8d 100644 --- a/plans/multi-raft/cell-groups/implementation-plan.md +++ b/plans/multi-raft/cell-groups/implementation-plan.md @@ -14,8 +14,8 @@ never executed by the plan runner. Read the [design](design.md), | --- | --- | --- | --- | ---: | | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | -| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | IN_PROGRESS | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | TODO | 6 | +| CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 6 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 6f22fcca6..1b470f570 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -2,15 +2,37 @@ # Cell-group implementation progress -The delivered design and implementation plan covers CG-01 through CG-16 and a -hard cutover. The latest user selection scopes this delivery to the executable -design and implementation plan. Earlier implementation checkpoints are retained -below; their historical scope statements do not override that selection. -Intermediate contracts do not qualify a shared runtime. -No production legacy/shared mode flag is being added. - - -## Current executable-plan handoff +The full end-to-end implementation goal is active following the plan handoff. +CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and +CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify a shared +runtime. No production legacy/shared mode flag is being added. + +## CG-03 completion audit and CG-04 continuation + +The audit checks the four CG-03 steps against retained evidence at +`0df009a24d88408a8efc40e356f794ca35513a95`: + +- Paged directory tests exercise exact coverage, bounded discovery, conditional + publication and recovery from lost replies using real conditional operations. +- Enrollment tests check pure creation identity and sealed/retired admission; + the task explicitly assigns real SQLite enrollment to CG-04. +- Native member bootstrap exercises synced physical genesis, permanent claims, + restart, unknown outcomes and refusal after claimed local records are removed. + Storage boundary tests separately verify physical identities and inventory. +- Route tests check trusted-catalog confirmation, cached operation during catalog + failure, original deadlines and attempt budgets, cancellation and command ambiguity. + +The full required command gate, source/log/artifact hashes and dependency evidence +validate. CG-03 is DONE only at that catalog/routing/admission-preparation boundary. +The record retains 77 source inputs, 20 successful commands, three executables and +44 distinct Rust cases. It does not qualify live shared dispatch, network peer +authentication, managed startup, current-term serving authority or RustFS. + +CG-04 now addresses original enrollment identity in the durable manifest, followed +by shared group ownership and execution. Its full six-case real-Raft execution +gate remains required; storage-only tests cannot substitute for that gate. + +## Plan handoff at b3b7eabd The package contains 16 dependency-ordered tasks, 86 required cases, 14 invariants and nine qualification profiles. Each task supplies source anchors, owned paths, @@ -216,16 +238,14 @@ must include its initial replayable LTX chain. ## Remaining implementation -CG-01 and CG-02 have passed their complete task gates. CG-03 is IN_PROGRESS; -CG-04 through CG-16 remain TODO. The latest storage qualification is recorded below; the earlier sections -retain the intermediate checkpoints and their limitations. +CG-01 through CG-03 have passed their scoped task gates. CG-04 is IN_PROGRESS; +CG-05 through CG-16 remain TODO. Historical entries below retain the intermediate +checkpoints and their limitations. The reviewed source baseline is `0df009a24d88408a8efc40e356f794ca35513a95`. -The routing change updates the catalog/birth anchor hashes and line positions; -their excerpts are unchanged. The baseline records the source review and test -corrections. A new executor must receive both ignored DONE-task evidence folders -before validating those historical checkpoints, plus the CG-03 routing directory -before auditing that task for completion. +A new executor must receive all three ignored DONE-task evidence directories +before validating those historical checkpoints. Current CG-04 source changes +remain subject to their own verification and recorded review. No shared runtime, split/merge, hibernation latency, real RustFS recovery, measured density or final hard cutover is qualified. The release gate remains closed until diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 83b10045e..10b94bd8d 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -264,7 +264,7 @@ { "id": "CG-03", "title": "Prepare paged range routing and enrollment admission", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -402,7 +402,9 @@ "I09", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-03-routing-final/record.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", @@ -418,7 +420,7 @@ { "id": "CG-04", "title": "Execute many independent SQLite cells through one real Raft group", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -441,7 +443,9 @@ ], "creates": [ "crates/cellar/durability/raft/cell_groups/owner.rs", - "crates/cellar/tests/native_cell_groups_execution.rs" + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/durability/raft/cell_groups/enrollment_apply.rs", + "crates/cellar/durability/raft/cell_groups/page_wire_tests.rs" ], "test_target": "native_cell_groups_execution", "steps": [ From c187c47773fd22d4658de1d912b6096f92a8e26d Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 16:26:21 -0700 Subject: [PATCH 035/568] test: inject the manifest directory barrier failure --- crates/cellar/tests/native_cell_groups_storage.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index a66a00ead..cdbe5ad95 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -1373,7 +1373,7 @@ async fn unchanged_group_root_refuses_quarantined_page_store() -> Result<()> { let cut = images::Cut::new(&[], &bytes, 0, true, false); let pin = generations.prepare(images::id(1), None, &cut.update())?; fault.mode.store( - faults::AFTER_IMAGE_WRITE, + faults::FAIL_DIRECTORY_SYNC, std::sync::atomic::Ordering::Release, ); assert!(tree @@ -1385,6 +1385,7 @@ async fn unchanged_group_root_refuses_quarantined_page_store() -> Result<()> { CellImagePin::Materialized(pin) ) .is_err()); + assert_eq!(fault.mode.load(std::sync::atomic::Ordering::Acquire), 0); assert!(matches!( tree.prepare_unchanged(&pages, ¤t, position(2)), Err(ManifestError::Quarantined) From fde3e6f96a388c3f17566667a81b14df49febc00 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 16:34:17 -0700 Subject: [PATCH 036/568] docs: record durable enrollment qualification and runtime handoff --- plans/multi-raft/cell-groups/README.md | 14 +++--- plans/multi-raft/cell-groups/baseline.json | 4 +- plans/multi-raft/cell-groups/progress.md | 53 +++++++++++++++++++++- 3 files changed, 61 insertions(+), 10 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 7ae74eb43..e1b555b13 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -39,7 +39,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package repeats the full capability matrix at one release revision. This checkout retains evidence under the ignored -`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/` and +`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/`, `.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` and `.cellar/cell-groups-evidence/CG-03-routing-final/` directories. A fresh checkout must receive all three retained directories at the same relative paths before validating @@ -75,8 +75,8 @@ the resumed implementation objective. Continue CG-04 after reading [progress](progress.md) and inspecting the working diff. -CG-01 through CG-03 are DONE only at their named contract/storage/admission boundaries; the -remaining tasks still require implementation. `ready` lists eligible TODO tasks; +CG-01 through CG-03 are DONE only at their named contract/storage/admission +boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. @@ -88,10 +88,10 @@ evidence and limits. CG-03 has been audited at its preparation boundary. Its record is at `.cellar/cell-groups-evidence/CG-03-routing-final/record.json`; transfer that entire directory along with the earlier DONE-task evidence to the implementation -executor. CG-04 is now implementing atomic creation retention and ordered -enrollment application. It must connect the catalog startup permit to the shared -runtime, verify the complete selected manifest/image closure and acquire current-term -quorum authority. Managed fleet integration remains required by CG-07. A birth +executor. CG-04 now retains creation identity with the selected image and +prepares ordered enrollment and exact retries. It must connect the catalog startup permit to the shared +runtime, verify the complete selected manifest/image closure and acquire +current-term quorum authority. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. The implementation executor should deliver code, updated component contracts and diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index d94d7686c..27e32da63 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "0df009a24d88408a8efc40e356f794ca35513a95", + "source_revision": "c187c47773fd22d4658de1d912b6096f92a8e26d", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed invocation-bounded routing at 7f37a60917bdf4d54022eb366050dba5d58ee4bf. Catalog/birth source anchor excerpts are unchanged; catalog.rs adds exports for the new route session, request, attempt, cached observation and typed errors, shifting the excerpt lines. The directory verifies untrusted relocation hints by exact lookup in its configured trusted catalog under the original captured domain deadline. Immutable request identity and ambiguity survive redirects, budgets are consumed before awaits, and cached starts do not read the catalog. This prepares shared-host routing; it is not transport authentication, command execution or serving authority. Existing storage and host source anchors are unchanged. Successor c60a7d23aa4c01bde405f04483d1792cb9375bc2 strengthens the native deadline test by consuming four seconds of a five-second invocation before a stalled redirect; a two-second external guard rejects per-hop timeout replenishment. Anchored production source is unchanged. Successor 0df009a24d88408a8efc40e356f794ca35513a95 replaces the forbidden Tokio select macro in the external cancellation test with a bounded futures selector. No production source anchor changes.", + "review_note": "2026-10-29: reviewed CG-04 enrollment persistence at 9f937f227928c1ffa7324c8727a62cd410967383. Manifest leaves now require the immutable creation record and bind it to history/schema and initial image; updates preserve it exactly. Ordered enrollment preparation uses the existing complete-root and engine publication path. Duplicate enrollment advances only the contiguous group prefix, retains the current image and cannot invoke physical initialization. Unchanged-root preparation refuses a quarantined page store. Marker-1 pages without enrollment data refuse; no legacy decoding is added. Memory admission charges the additional cell identity. Catalog, host, runtime and other anchored excerpts/hashes remain unchanged. CG-03 is DONE at its audited preparation boundary; CG-04 shared execution and the final hard cutover remain incomplete. Successor c187c47773fd22d4658de1d912b6096f92a8e26d corrects the new quarantine fixture to fail the manifest directory barrier and asserts that the fault was consumed. The original image-file-only fault did not affect page files; the failed run is retained. Production source and anchor excerpts are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 1b470f570..a4564cf20 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -7,6 +7,57 @@ CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## CG-04 durable enrollment checkpoint + +Source: `c187c47773fd22d4658de1d912b6096f92a8e26d` (successor to `9f937f22`). +CG-04 remains IN_PROGRESS. + +Each authenticated cell leaf now requires the original creation operation, +logical cell history, schema and initial image commitment. Publication includes +that record in the same complete root as the current image. Mutations refuse a +changed record; sequence one must match the initial image and later versions +advance its LTX position. Page decoding rejects missing/foreign creation records +at marker 1. No old-layout reader or inferred enrollment is introduced. The +physical pin-tree budget includes the record's additional cell identity. + +Ordered enrollment preparation checks the route, preceding complete root, +contiguous apply position and committed lifecycle phase before invoking a physical +image producer. Exact retries keep the latest image, receipts and cell version +and advance only the group prefix. This unchanged-root path reuses image/page +pins without page I/O and refuses quarantined stores. Engine publication still +requires the exact preceding branch and committed position. + +Native recovery exercises actual SQLite/LTX and Raft Engine in materialized and +extent modes. It mutates a cell, restarts, reclaims the original image, then +replays enrollment without invoking the image producer. The latest values and +receipts survive another restart, while the neighboring cell remains unchanged. +Sealed/retired ranges, competing creations and invalid apply positions refuse. +Existing process-kill cases also verify creation identity after recovery. + +The new page-wire tests preserve canonical JSON ordering when corrupting identity +fields, so format ordering cannot mask a missing identity check. A quarantine +fixture fails the actual manifest directory-sync barrier and checks fault +consumption. Its initial run selected an image-file-only fault and failed because +no page I/O was interrupted; that failed run is retained separately. + +All 32 selected Rust cases pass at this source: 15 catalog cases, 15 active +storage cases and two page-wire unit cases. The ignored process fixture is run +by its parent crash test. All-target Clippy for `cellar`/`cellar-logic` with warnings +denied, formatting (1,781 Rust files), layout, dependency isolation, repository +docs, source preflight and all 37 plan-validator tests pass. The retained partial +record is `.cellar/cell-groups-evidence/CG-04-enrollment/checkpoint.json`, binding +79 source inputs, nine successful command logs and three executables. +The failed fault-injection run remains under +`.cellar/cell-groups-evidence/CG-04-enrollment-failed-9f937f22/`. + +This is an ordered preparation/storage checkpoint. The existing `Replica` and +shared worker still consume the staged single-cell log/apply types. The next +implementation must connect physical `GroupHistory` and complete-root application +to that real consensus owner, then wire independent SQLite execution, fair +admission, receipts and reads. The six CG-04 real-Raft acceptance cases remain +required. No live shared owner, RustFS, load or hard-cutover qualification is +claimed by this checkpoint. + ## CG-03 completion audit and CG-04 continuation The audit checks the four CG-03 steps against retained evidence at @@ -242,7 +293,7 @@ CG-01 through CG-03 have passed their scoped task gates. CG-04 is IN_PROGRESS; CG-05 through CG-16 remain TODO. Historical entries below retain the intermediate checkpoints and their limitations. -The reviewed source baseline is `0df009a24d88408a8efc40e356f794ca35513a95`. +The reviewed source baseline is the CG-04 enrollment checkpoint at `c187c477`. A new executor must receive all three ignored DONE-task evidence directories before validating those historical checkpoints. Current CG-04 source changes remain subject to their own verification and recorded review. From 4ea67d9f90934e5a3aa119406449b01e86176b75 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 16:55:42 -0700 Subject: [PATCH 037/568] docs: deliver executable cell-group implementation handoff --- plans/multi-raft/cell-groups/README.md | 14 +-- plans/multi-raft/cell-groups/handoff.md | 111 +++++++++++++++++++++++ plans/multi-raft/cell-groups/progress.md | 34 ++++++- 3 files changed, 151 insertions(+), 8 deletions(-) create mode 100644 plans/multi-raft/cell-groups/handoff.md diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index e1b555b13..3af019c6e 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,11 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The plan handoff is complete. The active end-to-end implementation goal resumes -from that handoff and requires the complete shared-group runtime and hard cutover. -Plan delivery or an intermediate task gate does not qualify that runtime. +The current requested deliverable is the executable design and implementation +plan. See the [executor handoff](handoff.md) for the starting revision, existing +work, commands and completion boundary. The implementation described here still +requires the complete shared-group runtime and hard cutover; plan delivery or an +intermediate task gate does not qualify that runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -69,9 +71,9 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The active goal requires end-to-end runtime implementation. Continue from the -existing code and retained evidence; earlier plan-only handoffs do not satisfy -the resumed implementation objective. +The implementation executor should continue from the existing code and retained +evidence. This plan delivery leaves task status unchanged and does not certify +uncommitted runtime work. Review the working diff before starting CG-04. Continue CG-04 after reading [progress](progress.md) and inspecting the working diff. diff --git a/plans/multi-raft/cell-groups/handoff.md b/plans/multi-raft/cell-groups/handoff.md new file mode 100644 index 000000000..e849ed816 --- /dev/null +++ b/plans/multi-raft/cell-groups/handoff.md @@ -0,0 +1,111 @@ + + +# Cell-group implementation handoff + +This deliverable is an executable design and implementation plan. It does not +claim completion of the shared runtime, a production cutover or measured TiKV +performance parity. The target is the same partitioned multi-Raft pattern: many +whole cells per group, one Raft machine per group replica, and many groups sharing +node resources. A group spans voter nodes; it is not all cells on one node. + +## Starting point + +Use branch `codex/shared-cell-groups` and the existing +[draft PR 160](https://github.com/crabbuild/cellar/pull/160). +The committed handoff before this document is +`fde3e6f96a388c3f17566667a81b14df49febc00`; the reviewed source baseline is +`c187c47773fd22d4658de1d912b6096f92a8e26d`. +Fetched `origin/main` at `1be27d234d4674d4d014785c60cecbfe394bdab9` is already +included. Recheck origin before publishing later implementation changes. + +The [TiKV investigation](../density-decision.md) pins the supplied checkout at +`b3995dab28308f09cbb7c125bf32f789dbc203fe`. Follow its Region ownership and shared +worker model, adapting storage to independent SQLite/LTX cells. Do not copy KV +transactions, split a cell's SQLite history, or assume TiKV sizing thresholds. + +Read [design](design.md), [protocols](protocols.md), +[implementation order](implementation-plan.md) and +[qualification](qualification.md). `tasks.json` is the source for task ownership, +dependencies, steps, exact tests and acceptance oracles. `profiles.json` freezes +candidate workload and performance gates; those numbers are targets, not results. + +## First executor session + +Inspect `git status` and the existing diff before editing. This working checkout +contains an uncommitted physical owner prototype in +`crates/cellar/durability/raft/cell_groups/owner.rs` and its native execution +fixture, plus supporting edits in `cache.rs`, `storage.rs`, the group module and +directory bootstrap. It is not connected to the shared scheduler or public SDK. +Its test target has unresolved compilation failures documented in +[progress](progress.md). A clean checkout of the committed handoff does not +contain this prototype. Preserve or explicitly transfer it when assigning work; +do not mistake its presence for a passing task gate. + +Run from the repository root with Python 3.11+ and Rust 1.98.1: + +```sh +export PATH="$HOME/.cargo/bin:$PATH" +export CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/cellar-3617-cell-groups" +python3 plans/multi-raft/cell-groups/plan.py validate +python3 plans/multi-raft/cell-groups/plan.py preflight +python3 plans/multi-raft/cell-groups/plan.py show CG-04 +python3 -m unittest discover -s plans/multi-raft/cell-groups -p 'test_*.py' +``` + +Choose a different private target directory for a different checkout. The plan +runner validates and prints commands; it does not execute Rust tests or provision +infrastructure. `ready` currently returns no TODO task because CG-04 is already +IN_PROGRESS and its successors depend on it. Resume CG-04 rather than resetting +statuses. Global preflight intentionally rejects the unreviewed owner diff; +record a reviewed successor source only after reconciling that implementation. + +The three DONE records require their retained evidence directories listed in +[README](README.md). Transfer those artifacts with the source when validating on +another machine. Missing evidence must refuse; do not replace it with a pass flag +or reset completed tasks merely to bypass validation. + +## Delivery sequence + +| Milestone | Tasks | Required outcome | +| --- | --- | --- | +| Contracts and storage | CG-01–03, scoped gates already passed | Separate cell/group history, complete durable roots, bounded range routing and startup preparation | +| Shared runtime | CG-04–06 | Real multi-cell consensus and execution, complete-group snapshots, verified S3 archive and safe reclamation | +| Public cutover | CG-08 then CG-07 | Preserve effects and idle caches; route SDK and managed fleet through the shared runtime | +| Adaptive groups | CG-09–12 | Fenced transfers, split/merge, placement and coordinated hibernation/wake | +| Recovery and qualification | CG-14, CG-13, CG-15 | Reproducible load harness, fenced fresh-incarnation recovery, real RustFS and independent-domain tests | +| Final delivery | CG-16 | Remove the superseded one-cell architecture and pass all release gates at one revision | + +Use the generated dependency order for exact scheduling. A hot cell may occupy +one group through the same runtime. Do not introduce a permanent old/new mode or +an old-format migration reader; preserve marker 1 with explicit domain separation +and refuse unsupported persisted layouts. Never delete existing user data as +part of the hard cutover. + +CG-04 needs all six named real-Raft cases: `three_cells_one_rawnode_per_voter`, +`concurrent_enrollment_one_cell`, `shared_group_fair_admission`, +`cell_receipt_not_group_progress`, `strong_read_at_group_barrier` and +`apply_prefix_never_skips_slow_cell`. Fixing a prototype's compilation or passing +a low-level owner test does not establish fair SQL execution, SDK dispatch, +receipt settlement or strong-read serving. Its complete brief prints the owning +paths and exact command/evidence gate. + +## Release boundary + +Preserve local quorum as the Quorum acknowledgement boundary. S3 protection +requires a verified complete checkpoint and contiguous tail; uploaded log bytes +alone do not authorize reclaiming local recovery dependencies. A failed complete +group restore refuses admission. Disaster recovery requires the chosen fencing +authority API, Docker qualification, permanent old-host restart prevention and +fresh group/cell incarnations, including incomplete topology transitions. + +Measure occupancies 1/8/32/64/128, both image modes, resource use and request +latency under the frozen profiles. Report wake latency by its actual phases and +include timeouts and refusals. No fixed cost saving or immediate hibernation +response is promised before measurement. Local Docker/RustFS results and +independent failure-domain evidence are separate gates. + +Retain source, binary and output hashes, including failed runs. Update task status +only after its full gate passes. CG-16 reruns the complete capability matrix at +one release revision; `plan.py release` must continue to refuse until then. +Keep the requested MR-13/MR-14 progress JSON files excluded from the PR. Reuse the +active PR, report qualification limits and leave merging to a separate action. diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index a4564cf20..57d46c4b2 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -2,11 +2,41 @@ # Cell-group implementation progress -The full end-to-end implementation goal is active following the plan handoff. -CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and +The latest requested deliverable is the executable design and implementation +plan; see the [executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## Plan delivery and working-state audit — 2026-10-29 + +The package has 16 tasks, 86 required cases, 14 invariants and nine qualification +profiles. Its contract/evidence validation and CG-04 brief generation pass. +Freshly fetched `origin/main` is `1be27d234d4674d4d014785c60cecbfe394bdab9`, +already an ancestor of the delivery branch. The supplied TiKV checkout still +matches the investigation pin. Neither excluded MR-13/MR-14 progress JSON file +appears in the branch diff. + +There is additional uncommitted CG-04 owner work in this checkout. It is not +part of the reviewed baseline or completed evidence. The new execution test +target currently fails compilation: `GroupRoute` is ambiguous through glob +imports and the fixture calls the private `ImageCommitment::sha256` method. +The global preflight and its unit test refuse the changed `cache.rs`; the other +36 validator tests pass. Preserve this refusal until the implementation is +reviewed and a successor baseline is recorded. Do not reset those edits or +silently update hashes to make the handoff appear qualified. + +In a separate checkout of `fde3e6f9` with this plan update and the three retained +DONE evidence directories, source preflight and all 37 validator tests pass. +The release command exits nonzero with `implementation tasks/evidence incomplete`, +as required. Repository docs validation passes with 869 local references. These +checks qualify the committed plan handoff; they do not qualify the dirty runtime +prototype. An initial export-based validation attempt had incorrect Git isolation; +its temporary local metadata changes were restored before this independent rerun. + +The [handoff](handoff.md) separates this working state from the committed +starting point and lists the remaining CG-04 acceptance cases. No runtime task +is advanced by this plan delivery. + ## CG-04 durable enrollment checkpoint Source: `c187c47773fd22d4658de1d912b6096f92a8e26d` (successor to `9f937f22`). From 0951440e3748da20573a6de26b7a9e97298fb385 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 17:05:47 -0700 Subject: [PATCH 038/568] feat: add physical cell-group consensus ownership and ordered apply --- crates/cellar/durability/raft/README.md | 22 + crates/cellar/durability/raft/cache.rs | 3 + .../raft/cell_groups/directory/bootstrap.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 6 + .../durability/raft/cell_groups/owner.rs | 775 +++++++++++++++ crates/cellar/durability/raft/storage.rs | 1 + .../tests/native_cell_groups_execution.rs | 941 ++++++++++++++++++ 7 files changed, 1752 insertions(+) create mode 100644 crates/cellar/durability/raft/cell_groups/owner.rs create mode 100644 crates/cellar/tests/native_cell_groups_execution.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 8fdcdf527..2260d3ecf 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -14,6 +14,28 @@ under implementation. The [technical design](../../../../plans/multi-raft/technical-design.md) defines the complete contract. +The replacement [cell-group design](../../../../plans/multi-raft/cell-groups/design.md) +is under implementation. `verification::cell_groups::CellGroupOwner` now owns one +real RawNode for a physical group history and consumes a verified complete root +plus a revalidated initial-member bootstrap permit. It emits persistence, +transport and ordered application effects; it performs no SQLite I/O or +per-cell task creation. The existing node log worker syncs each write before +the owner releases dependent messages or advances the selected group root. +The exclusive ownership lease remains with queued engine work after cancellation, +so a replacement owner cannot start while that write is still in flight. + +Application completions bind the owner instance, exact entry, preceding root, +engine and manifest store. A slow cell holds the contiguous apply prefix while +other entries commit. Bounded backlog applies transport/proposal backpressure; +already collected Ready/LightReady batches retain room for their largest entry. +ReadIndex output is a barrier only: serving still requires current authority and +a coherent cell image after application through that index. The adapter currently +refuses snapshot/configuration/topology administration until those complete-group +protocols are connected. Shared scheduling, SQL admission, SDK dispatch and the +production hard cutover remain required; the existing host still uses its prior +single-cell owner. See the [implementation status](../../../../plans/multi-raft/cell-groups/progress.md) +for the exact qualification boundary. + Managed empty hosts expose `RaftCells::inspect_peer_node` under rooted fleet trust. It authenticates both physical processes and a requested minimum installed trust revision without constructing a cell. Current enrollment state/epoch is diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 0c614e1d2..b742a6e56 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -100,6 +100,7 @@ pub enum SnapshotChange { #[derive(Clone)] pub struct GroupWrite { + pub(crate) consensus_pin: Option>, pub group: GroupLog, pub entries: Vec, pub hard_state: Option, @@ -113,6 +114,7 @@ pub struct GroupWrite { impl GroupWrite { pub fn new(group: &GroupLog) -> Self { Self { + consensus_pin: None, group: group.clone(), entries: Vec::new(), hard_state: None, @@ -169,6 +171,7 @@ impl GroupWrite { /// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] pub struct GroupLog { + pub(crate) consensus_claimed: Arc, pub(crate) scratch_directory: std::path::PathBuf, pub(crate) snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, pub(crate) filesystem: Arc, diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs index 2247c3549..600d580c7 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -22,6 +22,10 @@ pub struct CellGroupBootstrapPermit { } impl CellGroupBootstrapPermit { + pub(crate) fn log(&self) -> &ReplicaLog { + &self.log + } + pub fn group(&self) -> &GroupLog { &self.group } diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index aff100236..5b4282fbe 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -6,6 +6,7 @@ mod enrollment; mod enrollment_apply; mod image_pin; mod manifest; +mod owner; mod page_store; mod page_wire; mod pins; @@ -31,3 +32,8 @@ pub use enrollment_apply::{EnrollmentApply, EnrollmentApplyError, EnrollmentImag pub use image_pin::{cell_image_history_digest, CellImagePin}; pub(crate) use pins::ManifestPins; pub use root::{GroupManifest, PreparedGroupManifest}; + +pub use owner::{ + CellGroupApply, CellGroupEffect, CellGroupOwner, CellGroupPersist, CellGroupPersisted, + CellGroupProposal, CellGroupStatus, CellGroupTicket, +}; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs new file mode 100644 index 000000000..92c21dd29 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -0,0 +1,775 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One physical group owns one RawNode. The node scheduler executes the emitted +//! effects; this owner performs no image I/O and starts no per-cell tasks. + +use super::{CellGroupBootstrapPermit, GroupManifest, PreparedGroupManifest}; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryLimits, +}; +use crate::raft::{ + cache::{ConsensusStorage, GroupLog, GroupWrite}, + completion::{entry_digest, entry_position, RuntimeError}, + options::RaftRuntimeOptions, + storage::ReplicaLog, +}; +use ::raft::{prelude::*, Config, LightReady, RawNode, Ready, StateRole, Storage}; +use cellar_logic::{ + cell_group::*, + consensus::{LogIndex, LogPosition, OperationId, Term}, +}; +use protobuf::Message as _; +use std::{ + collections::VecDeque, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }, +}; + +struct Lease(Arc); + +impl Drop for Lease { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +#[derive(Clone)] +pub struct CellGroupTicket { + lease: Arc, + sequence: u64, + position: LogPosition, + digest: [u8; 32], +} + +impl CellGroupTicket { + pub fn position(&self) -> LogPosition { + self.position + } + + pub fn entry_digest(&self) -> [u8; 32] { + self.digest + } + + fn matches(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.lease, &other.lease) + && self.sequence == other.sequence + && self.position == other.position + && self.digest == other.digest + } +} + +/// Physical work retains the owner lease even if its awaiting client is dropped. +/// The engine request also retains that lease until sync/publication finishes. +pub struct CellGroupPersist { + log: ReplicaLog, + ticket: CellGroupTicket, + write: GroupWrite, +} + +pub struct CellGroupPersisted { + ticket: CellGroupTicket, + sequence: u64, +} + +impl CellGroupPersist { + /// Execute through the existing node-wide persistence worker. A lost waiter + /// cannot advance the owner; its pending work must settle or restart recover. + pub async fn run(self) -> Result { + let operation = OperationId::new(self.ticket.sequence); + let receipt = self.log.persist_cells(operation, vec![self.write]).await?; + Ok(CellGroupPersisted { + ticket: self.ticket, + sequence: receipt.sequence, + }) + } +} + +pub struct CellGroupApply { + pub ticket: CellGroupTicket, + pub entry: Entry, + pub base: PreparedGroupManifest, +} + +/// A ReadIndex result is a barrier to wait for, not permission to serve a cell. +/// The dispatcher must recheck owner/term and pin an applied coherent cell view. +pub enum CellGroupEffect { + Persist(CellGroupPersist), + Message(Message), + Apply(CellGroupApply), + Applied { + ticket: CellGroupTicket, + manifest: GroupManifest, + }, + ReadIndex { + term: Term, + index: LogIndex, + context: Vec, + }, +} + +#[derive(Clone, Debug)] +pub struct CellGroupProposal { + pub group: GroupHistory, + pub position: LogPosition, + pub digest: [u8; 32], +} + +#[derive(Clone, Debug)] +pub struct CellGroupStatus { + pub term: Term, + pub leader: Option, + pub applied: LogPosition, + pub committed: LogIndex, + pub ready: bool, +} + +enum Phase { + Ready(Ready, Term), + Light(LightReady), + Apply(CellGroupTicket, PreparedGroupManifest, bool), +} + +pub struct CellGroupOwner { + lease: Arc, + log: ReplicaLog, + group: GroupLog, + node: RawNode, + current: PreparedGroupManifest, + options: RaftRuntimeOptions, + phase: Option, + waiting: Option, + effects: VecDeque, + apply: VecDeque, + apply_bytes: usize, + applying: Option, + next_ticket: u64, + last_sync: u64, + failed: Option, +} + +impl CellGroupOwner { + /// Attach only a claimed initial voter with a completely verified selected + /// root. The catalog permit is revalidated; serving additionally waits for + /// the current term's committed/applied Raft barrier. Managed members and + /// incoming snapshots remain closed until their complete protocols exist. + pub async fn start( + permit: CellGroupBootstrapPermit, + current: PreparedGroupManifest, + options: RaftRuntimeOptions, + seed: [u8; 32], + ) -> Result { + options.validate(permit.log().limits())?; + let group = permit.group().clone(); + group + .consensus_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached)?; + let lease = Arc::new(Lease(group.consensus_claimed.clone())); + permit + .revalidate() + .await + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + let cache = group.read()?; + if !Arc::ptr_eq(¤t.engine_owner, &group.owner) + || cache.group_manifest.as_ref() != Some(current.marker()) + || current.marker().route() != permit.route() + || !cache.membership.permits_group(group.identity()) + || cache.conf_state.voters.len() + cache.conf_state.learners.len() + > options.max_members_per_group + { + return Err(RuntimeError::Invalid( + "cell-group startup closure/member".into(), + )); + } + drop(cache); + let config = Config { + id: group.identity().member.get(), + applied: current.marker().position().index.get(), + election_tick: 20, + heartbeat_tick: 2, + check_quorum: true, + pre_vote: true, + read_only_option: ::raft::ReadOnlyOption::Safe, + max_size_per_msg: options.max_message_bytes as u64, + max_inflight_msgs: 8, + max_uncommitted_size: options.max_group_apply_bytes as u64, + // A valid entry has at least four encoded bytes (term/index). Its + // retained accounting adds 128 bytes. Budget for those small + // entries as well as payloads; raft-rs may still return one entry + // larger than this soft byte cap to guarantee progress. + max_committed_size_per_ready: (options.max_message_bytes / 66).max(1) as u64, + ..Default::default() + }; + let logger = slog::Logger::root(slog::Discard, slog::o!()); + let node = RawNode::new_with_election_seed( + &config, + group.consensus_storage(), + &logger, + Some(seed), + )?; + Ok(Self { + lease, + log: permit.log().clone(), + group, + node, + current, + options, + phase: None, + waiting: None, + effects: VecDeque::new(), + apply: VecDeque::new(), + apply_bytes: 0, + applying: None, + next_ticket: 0, + last_sync: 0, + failed: None, + }) + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.current + } + + pub fn status(&self) -> Result { + self.live()?; + let cache = self.group.read()?; + Ok(CellGroupStatus { + term: Term::new(self.node.raft.term), + leader: (self.node.raft.leader_id != 0).then_some(self.node.raft.leader_id), + applied: self.current.marker().position(), + committed: LogIndex::new(cache.hard_state.commit), + ready: self.node.raft.state == StateRole::Leader + && self.phase.is_none() + && self.apply_bytes < self.options.max_group_apply_bytes + && self.current.marker().position().term.get() == self.node.raft.term + && cache.hard_state.term == self.node.raft.term + && cache.group_manifest.as_ref() == Some(self.current.marker()), + }) + } + + pub fn campaign(&mut self) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() { + return Err(RuntimeError::NotReady); + } + self.node.campaign()?; + Ok(()) + } + + pub fn tick(&mut self) -> Result<(), RuntimeError> { + self.live()?; + // The scheduler coalesces a deferred tick until pending Ready work is + // drained. In particular, an arbitrarily slow fsync must not accumulate + // an unbounded vector of heartbeat/election messages inside RawNode. + if self.phase.is_some() || self.node.has_ready() { + return Err(RuntimeError::NotReady); + } + self.node.tick(); + Ok(()) + } + + pub fn step(&mut self, message: Message) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() { + return Err(RuntimeError::NotReady); + } + if self.apply_bytes >= self.options.max_group_apply_bytes { + return Err(RuntimeError::NotReady); + } + self.validate_message(&message)?; + self.node.step(message)?; + Ok(()) + } + + /// Protocol admission only. The shared cell executor must serialize each + /// cell's preparation, reserve physical work and retain ambiguous outcomes. + pub fn propose(&mut self, data: Vec) -> Result { + if !self.status()?.ready { + return Err(RuntimeError::NotReady); + } + if data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let entry = CellGroupEntry::decode( + &data, + self.current.marker().route(), + GroupEntryLimits::default(), + ) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + if !matches!( + entry.command, + GroupCommand::Enroll { .. } | GroupCommand::Mutation(_) + ) { + return Err(RuntimeError::Invalid( + "group administration is not attached".into(), + )); + } + let before = self.node.raft.raft_log.last_index(); + self.node.propose(Vec::new(), data)?; + let entry = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or(RuntimeError::NotReady)?; + if entry.index != before + 1 { + return Err(RuntimeError::NotReady); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(entry), + digest: entry_digest(entry)?, + }) + } + + pub fn read_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if !self.status()?.ready { + return Err(RuntimeError::NotReady); + } + if context.is_empty() || context.len() > 4096 { + return Err(RuntimeError::Invalid("read context".into())); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + + pub fn complete_persist(&mut self, completed: CellGroupPersisted) -> Result<(), RuntimeError> { + self.live()?; + if self + .waiting + .as_ref() + .is_none_or(|t| !t.matches(&completed.ticket)) + || completed.sequence <= self.last_sync + { + return Err(RuntimeError::StaleCompletion); + } + self.last_sync = completed.sequence; + self.waiting = None; + Ok(()) + } + + pub fn complete_apply( + &mut self, + ticket: CellGroupTicket, + next: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() { + return Err(RuntimeError::NotReady); + } + if self.applying.as_ref().is_none_or(|t| !t.matches(&ticket)) + || next.marker().position() != ticket.position + || next.base.as_ref() != Some(self.current.marker()) + || next.marker().route() != self.current.marker().route() + || !Arc::ptr_eq(&next.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&next.page_owner, &self.current.page_owner) + { + return Err(RuntimeError::StaleCompletion); + } + let entry = self.apply.front().ok_or(RuntimeError::StaleCompletion)?; + self.validate_apply(entry, &next)?; + self.phase = Some(Phase::Apply(ticket, next, false)); + Ok(()) + } + + /// Return one bounded effect. No Ready-dependent message or applied event is + /// released until its exact physical persistence completion is consumed. + pub fn poll(&mut self) -> Result, RuntimeError> { + self.live()?; + let result = self.poll_inner(); + if let Err(error) = &result { + self.failed = Some(error.clone()); + } + result + } + + fn live(&self) -> Result<(), RuntimeError> { + if let Some(error) = &self.failed { + return Err(error.clone()); + } + if self.group.closed.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + if self.node.raft.term >= u64::MAX - 2 + || self.node.raft.raft_log.last_index() >= u64::MAX - 2 + { + return Err(RuntimeError::Overloaded); + } + Ok(()) + } + + fn ticket( + &mut self, + position: LogPosition, + digest: [u8; 32], + ) -> Result { + self.next_ticket = self + .next_ticket + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + Ok(CellGroupTicket { + lease: self.lease.clone(), + sequence: self.next_ticket, + position, + digest, + }) + } + + fn persistence( + &mut self, + mut write: GroupWrite, + ) -> Result { + let ticket = self.ticket(LogPosition::default(), [0; 32])?; + write.consensus_pin = Some(self.lease.clone()); + self.waiting = Some(ticket.clone()); + Ok(CellGroupEffect::Persist(CellGroupPersist { + log: self.log.clone(), + ticket, + write, + })) + } + + fn enqueue(&mut self, entries: Vec) -> Result<(), RuntimeError> { + let mut previous = self + .apply + .back() + .map_or(self.current.marker().position().index.get(), |e| e.index); + let committed = self.group.initial_state()?.hard_state.commit; + for entry in entries { + if entry.index != previous + 1 || entry.index > committed { + return Err(RuntimeError::Protocol( + "noncontiguous/unpersisted group apply".into(), + )); + } + previous = entry.index; + self.apply_bytes = self + .apply_bytes + .checked_add(entry.compute_size() as usize + 128) + .ok_or(RuntimeError::Overloaded)?; + // Ready and its LightReady can each retain one oversize entry. + // Collection stops at the soft allowance; these two bounded + // in-flight batches must be drained, not treated as data failure. + if self.apply_bytes + > self.options.max_group_apply_bytes + 2 * self.options.max_message_bytes + 256 + { + return Err(RuntimeError::Overloaded); + } + self.apply.push_back(entry); + } + Ok(()) + } + + fn messages(&mut self, messages: Vec) -> Result<(), RuntimeError> { + if self.effects.len() + messages.len() > self.options.max_events { + return Err(RuntimeError::Overloaded); + } + let existing: usize = self + .effects + .iter() + .map(|event| match event { + CellGroupEffect::Message(message) => message.compute_size() as usize + 128, + CellGroupEffect::ReadIndex { context, .. } => context.len() + 128, + _ => 4096, + }) + .sum(); + let added: usize = messages + .iter() + .map(|m| m.compute_size() as usize + 128) + .sum(); + if existing.saturating_add(added) > self.options.max_event_bytes { + return Err(RuntimeError::Overloaded); + } + self.effects + .extend(messages.into_iter().map(CellGroupEffect::Message)); + Ok(()) + } + + fn poll_inner(&mut self) -> Result, RuntimeError> { + loop { + if let Some(event) = self.effects.pop_front() { + return Ok(Some(event)); + } + if self.waiting.is_some() { + return Ok(None); + } + match self.phase.take() { + Some(Phase::Ready(mut ready, term)) => { + self.messages(ready.take_messages())?; + self.messages(ready.take_persisted_messages())?; + let reads = ready.take_read_states(); + let queued: usize = self + .effects + .iter() + .map(|event| match event { + CellGroupEffect::Message(message) => { + message.compute_size() as usize + 128 + } + _ => 4096, + }) + .sum(); + let read_bytes: usize = reads.iter().map(|r| r.request_ctx.len() + 128).sum(); + if self.effects.len() + reads.len() > self.options.max_events + || queued.saturating_add(read_bytes) > self.options.max_event_bytes + { + return Err(RuntimeError::Overloaded); + } + for read in reads { + self.effects.push_back(CellGroupEffect::ReadIndex { + term, + index: LogIndex::new(read.index), + context: read.request_ctx.to_vec(), + }); + } + self.enqueue(ready.take_committed_entries())?; + let light = self.node.advance_append(ready); + let commit = light.commit_index(); + self.phase = Some(Phase::Light(light)); + if let Some(commit) = commit { + let mut write = GroupWrite::new(&self.group); + let mut hard = self.group.initial_state()?.hard_state; + hard.commit = commit; + write.hard_state = Some(hard); + return self.persistence(write).map(Some); + } + } + Some(Phase::Light(mut light)) => { + self.messages(light.take_messages())?; + self.enqueue(light.take_committed_entries())?; + } + Some(Phase::Apply(ticket, next, submitted)) => { + if !submitted { + let mut write = GroupWrite::new(&self.group); + write.group_manifest = Some(next.clone()); + self.phase = Some(Phase::Apply(ticket, next, true)); + return self.persistence(write).map(Some); + } + if self.group.group_manifest()?.as_ref() != Some(next.marker()) { + return Err(RuntimeError::StaleCompletion); + } + self.node.advance_apply_to(ticket.position.index.get()); + let entry = self + .apply + .pop_front() + .ok_or(RuntimeError::StaleCompletion)?; + self.apply_bytes -= entry.compute_size() as usize + 128; + self.applying = None; + self.current = next; + return Ok(Some(CellGroupEffect::Applied { + ticket, + manifest: self.current.marker().clone(), + })); + } + None => { + if self.node.has_ready() + && self.apply_bytes < self.options.max_group_apply_bytes + { + let ready = self.node.ready(); + if !ready.snapshot().is_empty() { + return Err(RuntimeError::Invalid( + "group snapshot is not attached".into(), + )); + } + let size: usize = ready + .entries() + .iter() + .chain(ready.committed_entries()) + .map(|e| e.compute_size() as usize + 128) + .sum::() + + ready + .messages() + .iter() + .chain(ready.persisted_messages()) + .map(|m| m.compute_size() as usize + 128) + .sum::() + + ready + .read_states() + .iter() + .map(|r| r.request_ctx.len() + 128) + .sum::(); + if size > self.options.max_ready_bytes { + return Err(RuntimeError::Overloaded); + } + let changed = !ready.entries().is_empty() || ready.hs().is_some(); + let mut write = GroupWrite::new(&self.group); + write.entries = ready.entries().clone(); + write.hard_state = ready.hs().cloned(); + self.phase = Some(Phase::Ready(ready, Term::new(self.node.raft.term))); + if changed { + return self.persistence(write).map(Some); + } + continue; + } + if self.applying.is_some() { + return Ok(None); + } + let Some(entry) = self.apply.front().cloned() else { + return Ok(None); + }; + let ticket = self.ticket(entry_position(&entry), entry_digest(&entry)?)?; + self.applying = Some(ticket.clone()); + return Ok(Some(CellGroupEffect::Apply(CellGroupApply { + ticket, + entry, + base: self.current.clone(), + }))); + } + } + } + } + + fn validate_apply( + &self, + entry: &Entry, + next: &PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + let invalid = || RuntimeError::Invalid("group apply does not match committed entry".into()); + if entry.get_entry_type() != EntryType::EntryNormal { + return Err(invalid()); + } + if entry.data.is_empty() { + return if next.marker().root() == self.current.marker().root() { + Ok(()) + } else { + Err(invalid()) + }; + } + let decoded = CellGroupEntry::decode( + &entry.data, + self.current.marker().route(), + GroupEntryLimits::default(), + ) + .map_err(|_| invalid())?; + match decoded.command { + GroupCommand::Enroll { ref cell, .. } => { + let old = self.current.pin_cell(&cell.cell); + let decision = super::decide_enrollment( + self.current.marker().route(), + super::EnrollmentPhase::Active, + &entry.data, + old.as_ref().map(|(leaf, _)| &leaf.enrollment), + GroupEntryLimits::default(), + ); + match decision { + Ok(super::EnrollmentDecision::Propose(record)) => { + let (leaf, _) = next.pin_cell(&cell.cell).ok_or_else(invalid)?; + if leaf.enrollment != record + || leaf.version.sequence.get() != 1 + || leaf.version.image != record.initial() + || leaf.last_mutation != entry_position(entry) + { + return Err(invalid()); + } + } + Ok(super::EnrollmentDecision::Existing(_)) + | Err(super::EnrollmentError::Conflict) => { + if next.marker().root() != self.current.marker().root() { + return Err(invalid()); + } + } + Err(_) => return Err(invalid()), + } + } + GroupCommand::Mutation(mutation) => { + let (old, _) = self + .current + .pin_cell(&mutation.cell.cell) + .ok_or_else(invalid)?; + mutation.check_base(&old.version).map_err(|_| invalid())?; + let (new, _) = next.pin_cell(&mutation.cell.cell).ok_or_else(invalid)?; + if new.version.history != mutation.cell + || new.version.sequence != mutation.sequence + || new.version.image != mutation.result.into() + || new.enrollment != old.enrollment + || new.last_mutation != entry_position(entry) + { + return Err(invalid()); + } + } + _ => return Err(invalid()), + } + Ok(()) + } + + fn validate_message(&self, message: &Message) -> Result<(), RuntimeError> { + let invalid = || RuntimeError::Invalid("cell-group peer message".into()); + let kind = message.get_msg_type(); + if message.to != self.group.identity().member.get() + || message.from == message.to + || message.from == 0 + || self.node.raft.prs().get(message.from).is_none() + || ::raft::raw_node::is_local_msg(kind) + || matches!( + kind, + MessageType::MsgPropose | MessageType::MsgTransferLeader | MessageType::MsgSnapshot + ) + || !message.get_snapshot().is_empty() + || message.entries.len() > 65_536 + || message.compute_size() as usize > self.options.max_message_bytes + || [ + message.term, + message.log_term, + message.index, + message.commit, + message.reject_hint, + message.commit_term, + message.request_snapshot, + ] + .iter() + .any(|&n| n >= u64::MAX - 2) + { + return Err(invalid()); + } + match kind { + MessageType::MsgAppend => { + let mut index = message.index; + let mut term = message.log_term; + if (index == 0) != (term == 0) { + return Err(invalid()); + } + for entry in &message.entries { + index = index.checked_add(1).ok_or_else(invalid)?; + if entry.index != index + || entry.term == 0 + || entry.term < term + || entry.term > message.term + || entry.get_entry_type() != EntryType::EntryNormal + || entry.data.len() > self.options.max_proposal_bytes + { + return Err(invalid()); + } + term = entry.term; + } + } + MessageType::MsgReadIndex | MessageType::MsgReadIndexResp => { + if message.entries.len() != 1 { + return Err(invalid()); + } + let entry = &message.entries[0]; + if entry.data.is_empty() + || entry.data.len() > 4096 + || entry.index != 0 + || entry.term != 0 + || !entry.context.is_empty() + || entry.get_entry_type() != EntryType::EntryNormal + { + return Err(invalid()); + } + if kind == MessageType::MsgReadIndex + && self.node.raft.read_only.pending_read_count() + >= self.options.max_pending_reads + { + return Err(RuntimeError::Overloaded); + } + } + _ if !message.entries.is_empty() => return Err(invalid()), + _ => (), + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index b4acec33a..3bfc929c5 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1056,6 +1056,7 @@ impl Worker { scope: S, ) -> Result, LogError> { Ok(GroupLog { + consensus_claimed: Arc::new(AtomicBool::new(false)), scratch_directory: self.options.directory.clone(), snapshot_scratch: self.snapshot_scratch.clone(), filesystem: self.filesystem.clone(), diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs new file mode 100644 index 000000000..e7bef5903 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -0,0 +1,941 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real raft-rs owners, three independent engines and cell-addressed SQLite/LTX. +//! This boundary does not qualify SDK dispatch, worker fairness or snapshots. +#![allow(clippy::disallowed_methods)] + +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; + +use anyhow::{Context, Result}; +use cellar::application::consensus::verification::cell_groups::GroupRoute; +use cellar::application::consensus::{ + verification::{cell_groups::*, *}, + *, +}; +use cellar::sdk::ExecutionDomain; +use cellar_ltx::{ + apply::{ImageState, ImageUpdate, LtxSegment}, + generation::{GenerationPin, GenerationStore, ImageId}, + DirectFileSystem, TXID, +}; +use raft::prelude::Message; +use sha2::{Digest, Sha256}; +use std::{collections::VecDeque, sync::Arc, time::Duration}; + +struct Voter { + log: ReplicaLog, + owner: Option, + pages: ManifestPageStore, + images: GenerationStore, + tree: ManifestTree, + gate: Arc, + paused: bool, + held: Option, +} + +struct Cluster { + _root: tempfile::TempDir, + domain: ExecutionDomain, + directory: RangeDirectory, + birth: CellGroupBirth, + voters: Vec, + network: VecDeque, + reads: Vec<(usize, Term, LogIndex, Vec)>, + options: RaftRuntimeOptions, +} + +impl Cluster { + async fn new() -> Result { + Self::with_options(RaftRuntimeOptions::default()).await + } + + async fn with_options(options: RaftRuntimeOptions) -> Result { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + history.cluster, + history.namespace, + cellar_logic::NamespaceMode::Raft, + [6; 32], + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([5; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let init = directory + .prepare_initialization(ProposalId::new([7; 16])?, &[request.route().clone()])?; + domain.run(directory.publish(&init)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + let mut voters = Vec::new(); + for n in 1..=3 { + let base = root.path().join(format!("node-{n}")); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([20; 16])?, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let marker = permit.group().group_manifest()?.context("genesis root")?; + let prepared = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| panic!("empty genesis"), + )?; + let owner = domain + .run(CellGroupOwner::start( + permit, + prepared, + options.clone(), + [n; 32], + )) + .await??; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + voters.push(Voter { + log, + owner: Some(owner), + pages, + images, + tree, + gate, + paused: false, + held: None, + }); + } + Ok(Self { + _root: root, + domain, + directory, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }) + } + + async fn permit(&self, voter: usize) -> Result { + Ok(self + .domain + .run( + self.directory + .bootstrap_group_member(&self.birth, &self.voters[voter].log), + ) + .await??) + } + + async fn drive(&mut self, isolated: bool) -> Result<()> { + for _ in 0..2000 { + let mut progress = false; + for (n, voter) in self.voters.iter_mut().enumerate() { + let event = voter.owner.as_mut().unwrap().poll()?; + let Some(event) = event else { continue }; + progress = true; + match event { + CellGroupEffect::Persist(work) => { + let done = work.run().await?; + voter.owner.as_mut().unwrap().complete_persist(done)?; + } + CellGroupEffect::Message(message) => { + if !isolated { + self.network.push_back(message); + } + } + CellGroupEffect::Apply(apply) => { + if voter.paused { + assert!(voter.held.replace(apply).is_none()); + } else { + let prepared = prepare(voter, &apply)?; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, prepared)?; + } + } + CellGroupEffect::Applied { ticket, manifest } => { + assert_eq!(ticket.position(), manifest.position()); + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(&manifest) + ); + } + CellGroupEffect::ReadIndex { + term, + index, + context, + } => self.reads.push((n, term, index, context)), + } + } + for _ in 0..self.network.len() { + let message = self.network.pop_front().unwrap(); + let target = message.to as usize - 1; + match self.voters[target] + .owner + .as_mut() + .unwrap() + .step(message.clone()) + { + Ok(()) => progress = true, + Err(RuntimeError::NotReady) => self.network.push_back(message), + Err(error) => return Err(error.into()), + } + } + // A bounded receiver may refuse messages until a held apply settles. + // Keep those messages for the next pump instead of busy-spinning. + if !progress { + return Ok(()); + } + } + anyhow::bail!("group did not quiesce") + } + + async fn elect(&mut self) -> Result<()> { + self.voters[0].owner.as_mut().unwrap().campaign()?; + self.drive(false).await?; + assert!(self.voters[0].owner.as_ref().unwrap().status()?.ready); + Ok(()) + } + + async fn restart(&mut self, n: usize) -> Result<()> { + let mut old = self.voters.remove(n); + drop(old.held.take()); + drop(old.owner.take()); + old.log.close().await?; + drop(old); + let number = n as u8 + 1; + let base = self._root.path().join(format!("node-{number}")); + let gate = fault::Gate::new(None); + let log = self + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let permit = self + .domain + .run(self.directory.bootstrap_group_member(&self.birth, &log)) + .await??; + let tree = ManifestTree::new(self.birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let marker = permit.group().group_manifest()?.context("selected root")?; + let prepared = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + return Err(ManifestError::Invalid("fixture materialized image")); + }; + let state = ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }; + let id = image_id(&leaf.version.history, leaf.version.sequence.get()) + .map_err(|_| ManifestError::Invalid("fixture image id"))?; + images + .adopt(id, state) + .map(CellImagePin::Materialized) + .map_err(|_| ManifestError::Invalid("fixture recovery image")) + }, + )?; + let owner = self + .domain + .run(CellGroupOwner::start( + permit, + prepared, + self.options.clone(), + [number; 32], + )) + .await??; + self.voters.insert( + n, + Voter { + log, + owner: Some(owner), + tree, + pages, + images, + gate, + paused: false, + held: None, + }, + ); + Ok(()) + } + + async fn close(self) -> Result<()> { + for mut voter in self.voters { + drop(voter.owner.take()); + voter.log.close().await?; + } + self.domain.shutdown().await; + Ok(()) + } +} + +fn image_id(history: &CellHistory, sequence: u64) -> Result { + let mut hash = Sha256::new(); + hash.update(history.cell.as_str()); + hash.update(sequence.to_be_bytes()); + Ok(ImageId::new(hash.finalize()[..16].try_into()?)?) +} + +fn apply_image( + store: &GenerationStore, + history: &CellHistory, + sequence: u64, + base: Option<&GenerationPin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + let header = cellar_ltx::ltx::Header::parse(files.last().context("LTX")?.bytes)?; + let ImageCommitment::Sha256(digest) = image.commitment else { + anyhow::bail!("materialized commitment required") + }; + let result = ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: header.page_size, + length: u64::from(header.page_size) * u64::from(header.commit), + }; + let update = ImageUpdate { + base: match base { + Some(pin) => pin.state(), + None => ImageState::empty(image.epoch)?, + }, + result, + segments: files + .iter() + .map(|f| LtxSegment { + bytes: f.bytes, + min_txid: TXID(f.min_txid), + max_txid: TXID(f.max_txid), + page_count: f.page_count as usize, + decoded_bytes: f.decoded_bytes, + }) + .collect(), + }; + Ok(store.prepare(image_id(history, sequence)?, base, &update)?) +} + +fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { + if apply.entry.data.is_empty() { + return Ok(voter.tree.prepare_unchanged( + &voter.pages, + &apply.base, + apply.ticket.position(), + )?); + } + let decoded = CellGroupEntry::decode( + &apply.entry.data, + apply.base.marker().route(), + GroupEntryLimits::default(), + )?; + match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let (next, _) = voter.tree.prepare_enrollment( + &mut voter.pages, + &apply.base, + EnrollmentApply { + position: apply.ticket.position(), + phase: EnrollmentPhase::Active, + entry: &apply.entry.data, + limits: GroupEntryLimits::default(), + }, + |_| { + let pin = apply_image(&voter.images, &cell, 1, None, image, &files) + .map_err(|_| ManifestError::Invalid("fixture initial LTX"))?; + Ok(EnrollmentImage { + geometry: CellImageGeometry { + length: pin.state().length, + page_size: pin.state().page_size, + }, + effects: EffectSummary::default(), + pin: CellImagePin::Materialized(pin), + }) + }, + )?; + Ok(next) + } + GroupCommand::Mutation(mutation) => { + let (old, CellImagePin::Materialized(base)) = + apply.base.pin_cell(&mutation.cell.cell).context("cell")? + else { + anyhow::bail!("materialized base") + }; + let pin = apply_image( + &voter.images, + &mutation.cell, + mutation.sequence.get(), + Some(&base), + mutation.result, + &mutation.files, + )?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: CellImageGeometry { + length: pin.state().length, + page_size: pin.state().page_size, + }, + last_mutation: apply.ticket.position(), + effects: old.effects, + }; + Ok(voter.tree.prepare_update( + &mut voter.pages, + &apply.base, + leaf, + Some(&old.version), + CellImagePin::Materialized(pin), + )?) + } + _ => anyhow::bail!("unexpected administration"), + } +} + +fn history(route: &GroupRoute, name: &str) -> Result { + Ok(CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell: CellId::new(name)?, + incarnation: IncarnationId::new([8; 16])?, + }) +} + +fn request( + route: &GroupRoute, + cell: CellHistory, + sequence: u64, + cut: &images::Cut, +) -> Result> { + let image = |state: ImageState| LtxImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: state.digest.into(), + }; + let files = vec![GroupLtx { + min_txid: cut.base.txid.0 + 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }]; + let command = if sequence == 1 { + GroupCommand::Enroll { + cell, + image: image(cut.result), + files, + } + } else { + GroupCommand::Mutation(GroupMutation { + cell, + sequence: MutationSequence::new(sequence)?, + base: image(cut.base), + result: image(cut.result), + files, + }) + }; + Ok(CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([sequence as u8; 16])?, + }, + command, + } + .encode(route, GroupEntryLimits::default())?) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn three_cells_one_rawnode_per_voter() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let mut expected = Vec::new(); + for (n, name) in ["alpha", "beta", "gamma"].into_iter().enumerate() { + let bytes = images::image( + None, + &format!( + "CREATE TABLE value(n); INSERT INTO value VALUES ({});", + n + 1 + ), + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let cell = history(&route, name)?; + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + expected.push((cell, bytes)); + } + for step in 0..9 { + let i = step % 3; + let (cell, before) = &mut expected[i]; + let sequence = step / 3 + 2; + let after = images::image(Some(before), "UPDATE value SET n = n + 10;")?; + let cut = images::Cut::new(before, &after, sequence as u64 - 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + sequence as u64, + &cut, + )?)?; + cluster.drive(false).await?; + *before = after; + } + assert_eq!(cluster.voters.len(), 3); + for n in 0..3 { + let owner = cluster.voters[n].owner.as_ref().unwrap(); + assert_eq!(owner.manifest().marker().root().cells(), 3); + for (cell, bytes) in &expected { + let (leaf, CellImagePin::Materialized(pin)) = + owner.manifest().pin_cell(&cell.cell).context("cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.history, *cell); + assert_eq!(leaf.version.sequence.get(), 4); + assert_eq!(&images::read(&pin)?, bytes); + } + let duplicate = cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(n).await?, + owner.manifest().clone(), + RaftRuntimeOptions::default(), + [50; 32], + )) + .await?; + assert!(matches!(duplicate, Err(RuntimeError::AlreadyAttached))); + assert_eq!( + owner.manifest().marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + } + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"barrier".to_vec())?; + cluster.drive(false).await?; + assert!(cluster.reads.iter().any(|(n, term, index, context)| *n == 0 + && context == b"barrier" + && *term + == cluster.voters[0] + .owner + .as_ref() + .unwrap() + .status() + .unwrap() + .term + && *index + <= cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index)); + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_requires_read_quorum() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"minority".to_vec())?; + cluster.drive(true).await?; + assert!(cluster.reads.is_empty()); + for _ in 0..50 { + cluster.voters[0].owner.as_mut().unwrap().tick()?; + cluster.drive(true).await?; + } + assert!(!cluster.voters[0].owner.as_ref().unwrap().status()?.ready); + assert!(cluster.voters[0] + .owner + .as_mut() + .unwrap() + .read_index(b"stale".to_vec()) + .is_err()); + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn apply_prefix_never_skips_slow_cell() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let mut expected = Vec::new(); + for name in ["slow", "later"] { + let cell = history(&route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + expected.push((cell, bytes)); + } + let before = cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + cluster.voters[1].paused = true; + cluster.voters[2].paused = true; + for (cell, bytes) in &mut expected { + let after = images::image(Some(bytes), "UPDATE value SET n = 2;")?; + let cut = images::Cut::new(bytes, &after, 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 2, + &cut, + )?)?; + cluster.drive(false).await?; + *bytes = after; + } + for voter in &cluster.voters[1..] { + let status = voter.owner.as_ref().unwrap().status()?; + assert_eq!(status.applied, before.position()); + assert_eq!(status.committed.get(), before.position().index.get() + 2); + assert_eq!(voter.owner.as_ref().unwrap().manifest().marker(), &before); + assert_eq!( + voter.held.as_ref().unwrap().ticket.position().index.get(), + before.position().index.get() + 1 + ); + } + let foreign = cluster.voters[2].held.as_ref().unwrap().ticket.clone(); + let voter = &mut cluster.voters[1]; + let apply = voter.held.take().context("held slow apply")?; + let next = prepare(voter, &apply)?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(foreign, next.clone()), + Err(RuntimeError::StaleCompletion) + )); + let unchanged = + voter + .tree + .prepare_unchanged(&voter.pages, &apply.base, apply.ticket.position())?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket.clone(), unchanged), + Err(RuntimeError::Invalid(_)) + )); + voter.paused = false; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, next)?; + cluster.drive(false).await?; + assert_eq!( + cluster.voters[1] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + + // Restart the other follower with both entries committed but neither applied. + // The restored marker stays behind until the real owner replays its log tail. + cluster.restart(2).await?; + assert_eq!( + cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + &before + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + let owner = voter.owner.as_ref().unwrap(); + assert_eq!( + owner.manifest().marker(), + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + ); + for (cell, bytes) in &expected { + let (leaf, CellImagePin::Materialized(pin)) = owner + .manifest() + .pin_cell(&cell.cell) + .context("restored cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.sequence.get(), 2); + assert_eq!(&images::read(&pin)?, bytes); + } + } + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_backpressures_slow_apply_without_losing_commits() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_proposal_bytes: 8192, + max_message_bytes: 12_288, + max_group_apply_bytes: 16_384, + max_apply_bytes: 32_768, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let cell = history(&route, "busy")?; + let mut bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(request(&route, cell.clone(), 1, &cut)?)?; + cluster.drive(false).await?; + let before = cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + cluster.voters[2].paused = true; + for sequence in 2..=40 { + let after = images::image(Some(&bytes), "UPDATE value SET n = n + 1;")?; + let cut = images::Cut::new(&bytes, &after, sequence - 1, true, false); + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + sequence, + &cut, + )?)?; + cluster.drive(false).await?; + bytes = after; + } + assert!( + !cluster.network.is_empty(), + "the slow voter must exert real transport backpressure" + ); + assert_eq!( + cluster.voters[2] + .owner + .as_ref() + .unwrap() + .manifest() + .marker(), + &before + ); + let voter = &mut cluster.voters[2]; + let apply = voter.held.take().context("held apply")?; + let next = prepare(voter, &apply)?; + voter.paused = false; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, next)?; + cluster.drive(false).await?; + assert!(cluster.network.is_empty()); + for voter in &cluster.voters { + let (leaf, CellImagePin::Materialized(pin)) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&cell.cell) + .context("caught-up cell")? + else { + anyhow::bail!("image") + }; + assert_eq!(leaf.version.sequence.get(), 40); + assert_eq!(images::read(&pin)?, bytes); + } + cluster.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_retains_lease_through_cancelled_sync() -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let permit = cluster.permit(0).await?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let entry = request(&route, history(&route, "held")?, 1, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + owner.propose(entry)?; + let CellGroupEffect::Persist(work) = owner.poll()?.context("append persistence")? else { + anyhow::bail!("append must persist before sending") + }; + assert!(owner.poll()?.is_none(), "no append messages before sync"); + for _ in 0..100 { + assert!(matches!(owner.tick(), Err(RuntimeError::NotReady))); + } + let gate = cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut write = Box::pin(work.run()); + match tokio::time::timeout( + Duration::from_secs(5), + futures_util::future::select(&mut write, Box::pin(gate.entered.notified())), + ) + .await? + { + futures_util::future::Either::Left(_) => anyhow::bail!("sync did not enter held gate"), + futures_util::future::Either::Right(_) => (), + } + drop(write); + drop(cluster.voters[0].owner.take()); + let duplicate = cluster + .domain + .run(CellGroupOwner::start( + permit, + root.clone(), + RaftRuntimeOptions::default(), + [61; 32], + )) + .await?; + assert!(matches!(duplicate, Err(RuntimeError::AlreadyAttached))); + drop(release); + cluster.voters[0].log.purge().await?; + let permit = cluster.permit(0).await?; + let owner = cluster + .domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [62; 32], + )) + .await??; + cluster.voters[0].owner = Some(owner); + cluster.close().await +} From 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 17:16:23 -0700 Subject: [PATCH 039/568] fix: bound cell-group effect layout and peer counters --- .../durability/raft/cell_groups/owner.rs | 34 ++++++++++-------- .../tests/native_cell_groups_execution.rs | 36 +++++++++++++++++-- 2 files changed, 52 insertions(+), 18 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 92c21dd29..664f3db8a 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -95,12 +95,12 @@ pub struct CellGroupApply { /// A ReadIndex result is a barrier to wait for, not permission to serve a cell. /// The dispatcher must recheck owner/term and pin an applied coherent cell view. pub enum CellGroupEffect { - Persist(CellGroupPersist), + Persist(Box), Message(Message), - Apply(CellGroupApply), + Apply(Box), Applied { ticket: CellGroupTicket, - manifest: GroupManifest, + manifest: Box, }, ReadIndex { term: Term, @@ -126,9 +126,9 @@ pub struct CellGroupStatus { } enum Phase { - Ready(Ready, Term), + Ready(Box, Term), Light(LightReady), - Apply(CellGroupTicket, PreparedGroupManifest, bool), + Apply(CellGroupTicket, Box, bool), } pub struct CellGroupOwner { @@ -374,7 +374,7 @@ impl CellGroupOwner { } let entry = self.apply.front().ok_or(RuntimeError::StaleCompletion)?; self.validate_apply(entry, &next)?; - self.phase = Some(Phase::Apply(ticket, next, false)); + self.phase = Some(Phase::Apply(ticket, Box::new(next), false)); Ok(()) } @@ -428,11 +428,11 @@ impl CellGroupOwner { let ticket = self.ticket(LogPosition::default(), [0; 32])?; write.consensus_pin = Some(self.lease.clone()); self.waiting = Some(ticket.clone()); - Ok(CellGroupEffect::Persist(CellGroupPersist { + Ok(CellGroupEffect::Persist(Box::new(CellGroupPersist { log: self.log.clone(), ticket, write, - })) + }))) } fn enqueue(&mut self, entries: Vec) -> Result<(), RuntimeError> { @@ -527,7 +527,7 @@ impl CellGroupOwner { }); } self.enqueue(ready.take_committed_entries())?; - let light = self.node.advance_append(ready); + let light = self.node.advance_append(*ready); let commit = light.commit_index(); self.phase = Some(Phase::Light(light)); if let Some(commit) = commit { @@ -545,7 +545,7 @@ impl CellGroupOwner { Some(Phase::Apply(ticket, next, submitted)) => { if !submitted { let mut write = GroupWrite::new(&self.group); - write.group_manifest = Some(next.clone()); + write.group_manifest = Some((*next).clone()); self.phase = Some(Phase::Apply(ticket, next, true)); return self.persistence(write).map(Some); } @@ -559,10 +559,10 @@ impl CellGroupOwner { .ok_or(RuntimeError::StaleCompletion)?; self.apply_bytes -= entry.compute_size() as usize + 128; self.applying = None; - self.current = next; + self.current = *next; return Ok(Some(CellGroupEffect::Applied { ticket, - manifest: self.current.marker().clone(), + manifest: Box::new(self.current.marker().clone()), })); } None => { @@ -599,7 +599,10 @@ impl CellGroupOwner { let mut write = GroupWrite::new(&self.group); write.entries = ready.entries().clone(); write.hard_state = ready.hs().cloned(); - self.phase = Some(Phase::Ready(ready, Term::new(self.node.raft.term))); + self.phase = Some(Phase::Ready( + Box::new(ready), + Term::new(self.node.raft.term), + )); if changed { return self.persistence(write).map(Some); } @@ -613,11 +616,11 @@ impl CellGroupOwner { }; let ticket = self.ticket(entry_position(&entry), entry_digest(&entry)?)?; self.applying = Some(ticket.clone()); - return Ok(Some(CellGroupEffect::Apply(CellGroupApply { + return Ok(Some(CellGroupEffect::Apply(Box::new(CellGroupApply { ticket, entry, base: self.current.clone(), - }))); + })))); } } } @@ -735,6 +738,7 @@ impl CellGroupOwner { for entry in &message.entries { index = index.checked_add(1).ok_or_else(invalid)?; if entry.index != index + || entry.index >= u64::MAX - 2 || entry.term == 0 || entry.term < term || entry.term > message.term diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index e7bef5903..3dd73b6ce 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -23,7 +23,7 @@ use cellar_ltx::{ generation::{GenerationPin, GenerationStore, ImageId}, DirectFileSystem, TXID, }; -use raft::prelude::Message; +use raft::prelude::{Entry, Message, MessageType}; use sha2::{Digest, Sha256}; use std::{collections::VecDeque, sync::Arc, time::Duration}; @@ -199,7 +199,7 @@ impl Cluster { } CellGroupEffect::Apply(apply) => { if voter.paused { - assert!(voter.held.replace(apply).is_none()); + assert!(voter.held.replace(*apply).is_none()); } else { let prepared = prepare(voter, &apply)?; voter @@ -213,7 +213,7 @@ impl Cluster { assert_eq!(ticket.position(), manifest.position()); assert_eq!( voter.log.cell_groups()?[0].group_manifest()?.as_ref(), - Some(&manifest) + Some(manifest.as_ref()) ); } CellGroupEffect::ReadIndex { @@ -639,6 +639,36 @@ async fn three_cells_one_rawnode_per_voter() -> Result<()> { cluster.close().await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_owner_rejects_exhausted_peer_entry_counters() -> Result<()> { + let mut cluster = Cluster::new().await?; + let before = cluster.voters[0].owner.as_ref().unwrap().status()?; + let message = Message { + msg_type: MessageType::MsgAppend, + from: 2, + to: 1, + term: 2, + log_term: 1, + index: u64::MAX - 3, + entries: vec![Entry { + term: 1, + index: u64::MAX - 2, + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!(matches!( + cluster.voters[0].owner.as_mut().unwrap().step(message), + Err(RuntimeError::Invalid(_)) + )); + let after = cluster.voters[0].owner.as_ref().unwrap().status()?; + assert_eq!(before.term, after.term); + assert_eq!(before.applied, after.applied); + cluster.elect().await?; + cluster.close().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn physical_owner_requires_read_quorum() -> Result<()> { let mut cluster = Cluster::new().await?; From 55317523ebcb34c82e9f80ad1030f9282d1997a6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 17:21:29 -0700 Subject: [PATCH 040/568] docs: retain physical cell-group owner qualification --- plans/multi-raft/cell-groups/README.md | 21 ++++---- plans/multi-raft/cell-groups/baseline.json | 8 +-- plans/multi-raft/cell-groups/handoff.md | 35 ++++++------ plans/multi-raft/cell-groups/progress.md | 62 +++++++++++++++++++++- 4 files changed, 95 insertions(+), 31 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 3af019c6e..1997172aa 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,11 +9,10 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The current requested deliverable is the executable design and implementation -plan. See the [executor handoff](handoff.md) for the starting revision, existing -work, commands and completion boundary. The implementation described here still -requires the complete shared-group runtime and hard cutover; plan delivery or an -intermediate task gate does not qualify that runtime. +The active goal resumes end-to-end implementation and the hard cutover from the +delivered plan. See the [executor handoff](handoff.md) for the starting revision, +existing work, commands and completion boundary. Plan delivery or an intermediate +task gate does not qualify the complete shared-group runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -72,8 +71,8 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff The implementation executor should continue from the existing code and retained -evidence. This plan delivery leaves task status unchanged and does not certify -uncommitted runtime work. Review the working diff before starting CG-04. +evidence. Review the working diff before resuming CG-04; intermediate owner and +storage checkpoints do not replace its full shared-execution gate. Continue CG-04 after reading [progress](progress.md) and inspecting the working diff. @@ -91,9 +90,11 @@ record is at `.cellar/cell-groups-evidence/CG-03-routing-final/record.json`; transfer that entire directory along with the earlier DONE-task evidence to the implementation executor. CG-04 now retains creation identity with the selected image and -prepares ordered enrollment and exact retries. It must connect the catalog startup permit to the shared -runtime, verify the complete selected manifest/image closure and acquire -current-term quorum authority. Managed fleet integration remains required by CG-07. A birth +prepares ordered enrollment and exact retries. Its physical group owner consumes +the catalog startup permit and verified selected image/manifest closure, and +obtains current-term quorum before protocol admission. Shared scheduler, SQL +execution, receipt settlement and serving integration remain required. Managed +fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. The implementation executor should deliver code, updated component contracts and diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index 27e32da63..a679dda2a 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "c187c47773fd22d4658de1d912b6096f92a8e26d", + "source_revision": "51e30eabde1ca76bf0b23963dd06d0e8130f8ee7", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed CG-04 enrollment persistence at 9f937f227928c1ffa7324c8727a62cd410967383. Manifest leaves now require the immutable creation record and bind it to history/schema and initial image; updates preserve it exactly. Ordered enrollment preparation uses the existing complete-root and engine publication path. Duplicate enrollment advances only the contiguous group prefix, retains the current image and cannot invoke physical initialization. Unchanged-root preparation refuses a quarantined page store. Marker-1 pages without enrollment data refuse; no legacy decoding is added. Memory admission charges the additional cell identity. Catalog, host, runtime and other anchored excerpts/hashes remain unchanged. CG-03 is DONE at its audited preparation boundary; CG-04 shared execution and the final hard cutover remain incomplete. Successor c187c47773fd22d4658de1d912b6096f92a8e26d corrects the new quarantine fixture to fail the manifest directory barrier and asserts that the fault was consumed. The original image-file-only fault did not affect page files; the failed run is retained. Production source and anchor excerpts are unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "f68c46b39524dc8f960c4760513e4ed470919a5bc93aa952bc7b4ca93b5f9ab6", - "line": 416, + "sha256": "ced64a028c84a88887a716b94ee3077382b6c4956ecf2fae39b1e2565c44e292", + "line": 419, "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n}\n" }, "apply": { diff --git a/plans/multi-raft/cell-groups/handoff.md b/plans/multi-raft/cell-groups/handoff.md index e849ed816..9cf066232 100644 --- a/plans/multi-raft/cell-groups/handoff.md +++ b/plans/multi-raft/cell-groups/handoff.md @@ -2,8 +2,9 @@ # Cell-group implementation handoff -This deliverable is an executable design and implementation plan. It does not -claim completion of the shared runtime, a production cutover or measured TiKV +The executable plan is delivered; the active goal now resumes its end-to-end +implementation and hard cutover. This handoff does not claim completion of the +shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many whole cells per group, one Raft machine per group replica, and many groups sharing node resources. A group spans voter nodes; it is not all cells on one node. @@ -12,9 +13,9 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [draft PR 160](https://github.com/crabbuild/cellar/pull/160). -The committed handoff before this document is -`fde3e6f96a388c3f17566667a81b14df49febc00`; the reviewed source baseline is -`c187c47773fd22d4658de1d912b6096f92a8e26d`. +The physical owner implementation and reviewed source baseline are at +`51e30eabde1ca76bf0b23963dd06d0e8130f8ee7`; see the latest checkpoint in +[progress](progress.md) for qualification and remaining work. Fetched `origin/main` at `1be27d234d4674d4d014785c60cecbfe394bdab9` is already included. Recheck origin before publishing later implementation changes. @@ -31,15 +32,13 @@ candidate workload and performance gates; those numbers are targets, not results ## First executor session -Inspect `git status` and the existing diff before editing. This working checkout -contains an uncommitted physical owner prototype in +Inspect `git status` and the existing diff before editing. The physical owner in `crates/cellar/durability/raft/cell_groups/owner.rs` and its native execution -fixture, plus supporting edits in `cache.rs`, `storage.rs`, the group module and -directory bootstrap. It is not connected to the shared scheduler or public SDK. -Its test target has unresolved compilation failures documented in -[progress](progress.md). A clean checkout of the committed handoff does not -contain this prototype. Preserve or explicitly transfer it when assigning work; -do not mistake its presence for a passing task gate. +fixture now use real three-voter Raft with complete group manifests and separate +SQLite images. The initial fixture compilation failures have been repaired. +The owner is not yet connected to the shared scheduler or public SDK. Preserve +its persistence, completion and physical lease boundaries during integration; +do not mistake the low-level owner cases for the full CG-04 gate. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -56,14 +55,20 @@ Choose a different private target directory for a different checkout. The plan runner validates and prints commands; it does not execute Rust tests or provision infrastructure. `ready` currently returns no TODO task because CG-04 is already IN_PROGRESS and its successors depend on it. Resume CG-04 rather than resetting -statuses. Global preflight intentionally rejects the unreviewed owner diff; -record a reviewed successor source only after reconciling that implementation. +statuses. Global preflight rejects any further unreviewed source drift; record +a reviewed successor source after reconciling each implementation change. The three DONE records require their retained evidence directories listed in [README](README.md). Transfer those artifacts with the source when validating on another machine. Missing evidence must refuse; do not replace it with a pass flag or reset completed tasks merely to bypass validation. +The current partial CG-04 evidence is retained at +`.cellar/cell-groups-evidence/CG-04-owner/checkpoint.json`. Transfer that directory +and the failed gate named by its record when handing off this checkpoint. It +records six physical-owner cases and the catalog/storage regressions, with +explicit limitations; it cannot stand in for the complete six-case CG-04 contract. + ## Delivery sequence | Milestone | Tasks | Required outcome | diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 57d46c4b2..7292dd8a4 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -2,11 +2,69 @@ # Cell-group implementation progress -The latest requested deliverable is the executable design and implementation -plan; see the [executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and +The full end-to-end implementation and hard-cutover goal is active again after +the plan delivery; see the [executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify a shared runtime. No production legacy/shared mode flag is being added. +## CG-04 physical consensus owner checkpoint + +Source: `51e30eabde1ca76bf0b23963dd06d0e8130f8ee7` (successor to `0951440e`). +CG-04 remains IN_PROGRESS. The physical owner now attaches one real RawNode to +each group replica using the catalog's initial-member permit and a completely +verified selected root. The permit is revalidated before attachment. Each +GroupLog shares an exclusive owner lease; persistence requests retain it until +physical engine work finishes, even if the owner and its awaiting future are +dropped. A duplicate attachment cannot race an outstanding write. + +Ready and LightReady persistence completes before dependent messages, committed +application or read barriers are released. Image preparation is an external +effect and cannot advance Raft apply. Completion must match the exact owner, +entry digest/position, preceding root, engine and page-store owner. The new root +is published through the existing engine transaction before advancing the +contiguous prefix. No-ops preserve all cell versions. ReadIndex output remains +a barrier, not SDK serving authority. + +The local three-voter harness replicates three independent SQLite histories and +interleaved LTX updates. It holds both followers' first apply while two entries +commit, rejects foreign and incorrect-image completions, then restarts one +follower and replays its committed tail. The restart joins the engine; it is not +a process-kill or power-loss test. Separate tests exercise minority read refusal, +cancellation while a real log sync is held, and 40 versions with a backpressured +follower that later reaches the exact final SQLite bytes. A peer counter-boundary +regression rejects input before changing the owner's term or applied state. + +Ready collection accounts for small-entry overhead and the two in-flight batches' +maximum entries. A saturated apply queue backpressures new protocol requests. +Ticks coalesce behind pending Ready work so a stalled sync does not accumulate +unbounded heartbeat/election messages. Large physical effect and phase payloads +are boxed, reducing the space paid by every lightweight event. + +All 36 selected Rust cases pass: six physical-owner cases, 15 catalog cases and +15 active storage cases. The ignored process fixture is run by its parent crash +test. All-target Clippy for `cellar`/`cellar-logic` with warnings denied, +formatting (1,783 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator tests pass. The retained partial record is +`.cellar/cell-groups-evidence/CG-04-owner/checkpoint.json`, binding 82 source inputs, +eight successful command logs and three executables. It is not a CG-04 DONE record. + +The initial owner fixture compilation and pre-vote persistence +expectation failed and are retained. Pre-vote has no durable vote; the corrected +sync test holds an actual data append after election. An initial Clippy run +rejected oversized enum variants; the implementation fixes their layout without +suppressing the lint. That source-bound failed gate remains at +`.cellar/cell-groups-evidence/CG-04-owner-failed-0951440e/gate-commands.json`; +earlier uncommitted fixture failure logs are retained under the final checkpoint's +`intermediate/` directory and are not treated as source-bound qualification. + +The existing shared scheduler and SQL worker still consume the prior single-cell +owner. Next, integrate this physical owner with those node resources, bounded +per-cell producers, fair admission, exact receipt settlement and coherent reads. +The complete CG-04 six-case gate remains required, including real enrollment +concurrency and callback ownership. Snapshot/configuration/topology requests +currently refuse; their complete-group protocols, managed startup, RustFS, +independent failure domains and final hard cutover remain unfinished. + ## Plan delivery and working-state audit — 2026-10-29 The package has 16 tasks, 86 required cases, 14 invariants and nine qualification From 4d860c958e146628db6d07e4a9065e6dbfc9e56a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 17:55:03 -0700 Subject: [PATCH 041/568] docs: clarify executable cell-group plan handoff --- plans/multi-raft/cell-groups/README.md | 9 ++++--- plans/multi-raft/cell-groups/handoff.md | 35 ++++++++++++++++++++++--- 2 files changed, 36 insertions(+), 8 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 1997172aa..0306250fa 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,10 +9,11 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The active goal resumes end-to-end implementation and the hard cutover from the -delivered plan. See the [executor handoff](handoff.md) for the starting revision, -existing work, commands and completion boundary. Plan delivery or an intermediate -task gate does not qualify the complete shared-group runtime. +The requested deliverable is this executable design and implementation plan. +See the [executor handoff](handoff.md) for the starting revision, uncommitted +runtime draft, latest-main integration requirement, commands and completion +boundary. Plan delivery or an intermediate task gate does not qualify the +complete shared-group runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned diff --git a/plans/multi-raft/cell-groups/handoff.md b/plans/multi-raft/cell-groups/handoff.md index 9cf066232..d10fa19e0 100644 --- a/plans/multi-raft/cell-groups/handoff.md +++ b/plans/multi-raft/cell-groups/handoff.md @@ -2,8 +2,9 @@ # Cell-group implementation handoff -The executable plan is delivered; the active goal now resumes its end-to-end -implementation and hard cutover. This handoff does not claim completion of the +This delivery is the executable design and implementation plan requested by the +user. End-to-end implementation and hard cutover remain the executor's work. +This handoff does not claim completion of the shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many whole cells per group, one Raft machine per group replica, and many groups sharing @@ -16,8 +17,14 @@ Use branch `codex/shared-cell-groups` and the existing The physical owner implementation and reviewed source baseline are at `51e30eabde1ca76bf0b23963dd06d0e8130f8ee7`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. -Fetched `origin/main` at `1be27d234d4674d4d014785c60cecbfe394bdab9` is already -included. Recheck origin before publishing later implementation changes. +The branch includes `1be27d234d4674d4d014785c60cecbfe394bdab9`. The latest fetch +found `origin/main` at `8a93ffd7` (documentation consolidation, PR 161), which is +not yet integrated. It moves the existing plan documentation into `docs/design/`. +Before resuming implementation, reconcile those moves with this branch's added +cell-group package, its relative links, generated index, validator paths and +source anchors. Preserve the branch's implementation and retained evidence; +do not replace its files with the main-branch tree. Run documentation and package +validation after integration and record the resulting reviewed revision. The [TiKV investigation](../density-decision.md) pins the supplied checkout at `b3995dab28308f09cbb7c125bf32f789dbc203fe`. Follow its Region ownership and shared @@ -30,6 +37,26 @@ Read [design](design.md), [protocols](protocols.md), dependencies, steps, exact tests and acceptance oracles. `profiles.json` freezes candidate workload and performance gates; those numbers are targets, not results. +## Working draft at plan delivery + +The checkout also contains an uncommitted CG-04 node-runtime draft in +`crates/cellar/durability/raft/cell_groups/runtime/`, owner scheduling changes, +shared shutdown completion in `crates/cellar/durability/raft/runtime.rs`, and +native tests in `crates/cellar/tests/support/cell_group_runtime.rs` and +`crates/cellar/tests/native_cell_groups_execution.rs`. These files are not part +of the source-bound physical-owner checkpoint. Preserve and review the whole +diff, including untracked files, before continuing it. A fresh checkout of the +PR does not contain this working draft. + +Package validation passes with 16 tasks, 86 required cases, 14 invariants and +nine qualification profiles. All 37 validator tests pass against a clean export +of committed revision `55317523ebcb34c82e9f80ad1030f9282d1997a6`. In the working +checkout, preflight and its corresponding unit test correctly refuse the changed +`runtime.rs`; the other 36 validator tests pass. Do not refresh its hash merely +to remove that refusal. Review and qualify the runtime draft, commit its source, +then update the baseline with that revision and reviewed excerpts. Neither these +plan checks nor the draft's intermediate test runs complete CG-04. + ## First executor session Inspect `git status` and the existing diff before editing. The physical owner in From 865f253da0a14d730b62f77b17cefc6d1497890b Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 17:55:49 -0700 Subject: [PATCH 042/568] feat: multiplex physical cell-group owners on a bounded node runtime --- crates/cellar/durability/raft/README.md | 18 +- .../cellar/durability/raft/cell_groups/mod.rs | 3 + .../durability/raft/cell_groups/owner.rs | 129 +++- .../raft/cell_groups/runtime/mod.rs | 380 +++++++++++ .../raft/cell_groups/runtime/worker.rs | 606 ++++++++++++++++++ crates/cellar/durability/raft/runtime.rs | 18 +- .../tests/native_cell_groups_execution.rs | 23 + .../tests/support/cell_group_runtime.rs | 451 +++++++++++++ 8 files changed, 1605 insertions(+), 23 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/runtime/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/runtime/worker.rs create mode 100644 crates/cellar/tests/support/cell_group_runtime.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 2260d3ecf..10a639215 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -31,9 +31,21 @@ already collected Ready/LightReady batches retain room for their largest entry. ReadIndex output is a barrier only: serving still requires current authority and a coherent cell image after application through that index. The adapter currently refuses snapshot/configuration/topology administration until those complete-group -protocols are connected. Shared scheduling, SQL admission, SDK dispatch and the -production hard cutover remain required; the existing host still uses its prior -single-cell owner. See the [implementation status](../../../../plans/multi-raft/cell-groups/progress.md) +protocols are connected. `CellGroupRuntime` now multiplexes physical group owners +on one execution-domain worker per node. It reuses the bounded mailbox, byte/item +admission, joined terminal result and existing persistence engine. A fair active +queue collects writes across histories into one engine batch; queued and in-flight +work stay charged. Peer-frame admission retains its node/group reservations until +execution or discard and grants no Raft acknowledgement. Completion requests have +reserved capacity independent from ordinary traffic. Physical settlement can +progress under full event admission, releasing the apply reservation before its +Applied event is admitted. Shutdown stops admission, joins the submitted batch, +then fences the engine before releasing exclusive runtime ownership. + +Per-cell SQL admission, SDK dispatch and the production hard cutover remain +required; the existing host still uses its prior single-cell owner. Transport +authentication and actual image preparation remain the caller's responsibility +at this verification boundary. See the [implementation status](../../../../plans/multi-raft/cell-groups/progress.md) for the exact qualification boundary. Managed empty hosts expose `RaftCells::inspect_peer_node` under rooted fleet diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 5b4282fbe..ca57a3aa4 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -12,6 +12,9 @@ mod page_wire; mod pins; mod retention; mod root; +mod runtime; + +pub use runtime::{CellGroupRuntime, CellGroupRuntimeEvent, CellGroupRuntimeEvents}; pub use manifest::{ CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 664f3db8a..ad71b5a4b 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -41,6 +41,7 @@ pub struct CellGroupTicket { sequence: u64, position: LogPosition, digest: [u8; 32], + pub(super) admission: Option>, } impl CellGroupTicket { @@ -64,13 +65,13 @@ impl CellGroupTicket { /// The engine request also retains that lease until sync/publication finishes. pub struct CellGroupPersist { log: ReplicaLog, - ticket: CellGroupTicket, - write: GroupWrite, + pub(super) ticket: CellGroupTicket, + pub(super) write: GroupWrite, } pub struct CellGroupPersisted { - ticket: CellGroupTicket, - sequence: u64, + pub(super) ticket: CellGroupTicket, + pub(super) sequence: u64, } impl CellGroupPersist { @@ -86,6 +87,7 @@ impl CellGroupPersist { } } +#[derive(Clone)] pub struct CellGroupApply { pub ticket: CellGroupTicket, pub entry: Entry, @@ -109,6 +111,18 @@ pub enum CellGroupEffect { }, } +impl CellGroupEffect { + pub(super) fn bytes(&self) -> usize { + 4096 + match self { + Self::Persist(work) => work.write.estimated_bytes().unwrap_or(usize::MAX - 4096), + Self::Message(message) => message.compute_size() as usize, + Self::Apply(work) => work.entry.compute_size() as usize + work.base.reserved_bytes(), + Self::ReadIndex { context, .. } => context.len(), + Self::Applied { .. } => 1024, + } + } +} + #[derive(Clone, Debug)] pub struct CellGroupProposal { pub group: GroupHistory, @@ -381,14 +395,87 @@ impl CellGroupOwner { /// Return one bounded effect. No Ready-dependent message or applied event is /// released until its exact physical persistence completion is consumed. pub fn poll(&mut self) -> Result, RuntimeError> { + self.poll_budget(true, true) + } + + pub(super) fn poll_budget( + &mut self, + collect: bool, + emit: bool, + ) -> Result, RuntimeError> { self.live()?; - let result = self.poll_inner(); + let result = self.poll_inner(collect, emit); if let Err(error) = &result { self.failed = Some(error.clone()); } result } + pub(super) fn idle(&self) -> bool { + self.phase.is_none() + && self.waiting.is_none() + && self.effects.is_empty() + && !self.node.has_ready() + } + + pub(super) fn waiting(&self) -> bool { + self.waiting.is_some() + } + + pub(super) fn control_pending(&self) -> bool { + self.phase.is_some() + } + + pub(super) fn has_work(&self) -> bool { + !self.effects.is_empty() + || self.phase.is_some() + || self.node.has_ready() + || (self.applying.is_none() && !self.apply.is_empty()) + } + + pub(super) fn usage(&self) -> (usize, usize, usize) { + let entries = |entries: &[Entry]| { + entries + .iter() + .map(|e| e.compute_size() as usize + 128) + .sum::() + }; + let messages = |messages: &[Message]| { + messages + .iter() + .map(|m| m.compute_size() as usize + 128) + .sum::() + }; + let ready = match &self.phase { + Some(Phase::Ready(ready, _)) => { + entries(ready.entries()) + + entries(ready.committed_entries()) + + messages(ready.messages()) + + messages(ready.persisted_messages()) + + ready + .read_states() + .iter() + .map(|r| r.request_ctx.len() + 128) + .sum::() + } + Some(Phase::Light(light)) => { + entries(light.committed_entries()) + messages(light.messages()) + } + Some(Phase::Apply(_, next, _)) => next.reserved_bytes() + 4096, + None => 0, + }; + ( + ready + + self + .effects + .iter() + .map(CellGroupEffect::bytes) + .sum::(), + self.apply_bytes, + self.node.raft.read_only.pending_read_count(), + ) + } + fn live(&self) -> Result<(), RuntimeError> { if let Some(error) = &self.failed { return Err(error.clone()); @@ -418,6 +505,7 @@ impl CellGroupOwner { sequence: self.next_ticket, position, digest, + admission: None, }) } @@ -490,13 +578,14 @@ impl CellGroupOwner { Ok(()) } - fn poll_inner(&mut self) -> Result, RuntimeError> { + fn poll_inner( + &mut self, + collect: bool, + emit: bool, + ) -> Result, RuntimeError> { loop { - if let Some(event) = self.effects.pop_front() { - return Ok(Some(event)); - } if self.waiting.is_some() { - return Ok(None); + return Ok(if emit { self.effects.pop_front() } else { None }); } match self.phase.take() { Some(Phase::Ready(mut ready, term)) => { @@ -542,7 +631,7 @@ impl CellGroupOwner { self.messages(light.take_messages())?; self.enqueue(light.take_committed_entries())?; } - Some(Phase::Apply(ticket, next, submitted)) => { + Some(Phase::Apply(mut ticket, next, submitted)) => { if !submitted { let mut write = GroupWrite::new(&self.group); write.group_manifest = Some((*next).clone()); @@ -560,13 +649,27 @@ impl CellGroupOwner { self.apply_bytes -= entry.compute_size() as usize + 128; self.applying = None; self.current = *next; + // Physical preparation and publication have settled. Its + // reservation must not deadlock admission of Applied. + ticket.admission = None; return Ok(Some(CellGroupEffect::Applied { ticket, manifest: Box::new(self.current.marker().clone()), })); } None => { - if self.node.has_ready() + // Complete already-persisted protocol phases even when + // output admission is full. Otherwise held Apply events + // can prevent the completion which releases their quota. + if emit { + if let Some(event) = self.effects.pop_front() { + return Ok(Some(event)); + } + } else { + return Ok(None); + } + if collect + && self.node.has_ready() && self.apply_bytes < self.options.max_group_apply_bytes { let ready = self.node.ready(); @@ -699,7 +802,7 @@ impl CellGroupOwner { Ok(()) } - fn validate_message(&self, message: &Message) -> Result<(), RuntimeError> { + pub(super) fn validate_message(&self, message: &Message) -> Result<(), RuntimeError> { let invalid = || RuntimeError::Invalid("cell-group peer message".into()); let kind = message.get_msg_type(); if message.to != self.group.identity().member.get() diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs new file mode 100644 index 000000000..af2dcfe40 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -0,0 +1,380 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One execution-domain worker multiplexes physical cell-group owners per node. + +mod worker; + +use super::*; +use crate::raft::{ + budget::{Budget, Permit}, + completion::RuntimeError, + mailbox, + options::RaftRuntimeOptions, + runtime::Terminal, + storage::ReplicaLog, +}; +use ::raft::prelude::Message; +use cellar_logic::cell_group::GroupHistory; +use protobuf::Message as _; +use sha2::{Digest, Sha256}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, +}; +use tokio::sync::{mpsc, oneshot, Notify}; + +const COMPLETION_SLOTS: usize = 8; + +pub struct CellGroupRuntimeEvent { + group: GroupHistory, + effect: Result, + _permit: Arc, +} + +impl CellGroupRuntimeEvent { + pub fn group(&self) -> GroupHistory { + self.group + } + + pub fn effect(&self) -> Result<&CellGroupEffect, &RuntimeError> { + self.effect.as_ref() + } + + /// The cloned ticket retains event admission through physical preparation + /// and publication, even if the receiving task drops this event. + pub fn retain_apply(&self) -> Option { + match &self.effect { + Ok(CellGroupEffect::Apply(work)) => Some((**work).clone()), + _ => None, + } + } +} + +pub struct CellGroupRuntimeEvents { + receiver: mpsc::Receiver, +} + +impl CellGroupRuntimeEvents { + pub async fn recv(&mut self) -> Option { + self.receiver.recv().await + } + + pub fn try_recv(&mut self) -> Result { + self.receiver.try_recv() + } +} + +#[derive(Clone)] +pub struct CellGroupRuntime { + inner: Arc, +} + +struct Handle { + log: ReplicaLog, + domain: crate::asyncrt::DomainToken, + options: RaftRuntimeOptions, + requests: mailbox::Sender, + budget: Arc, + completions: Arc, + closing: Arc, + terminal: Arc, + stop: Arc, +} + +enum Command { + Attach(Box), + Campaign, + Step(Message), + AdmitStep(Message), + Propose(Vec), + ReadIndex(Vec), + Apply(CellGroupTicket, Box), + Status, + Tick, +} + +enum Response { + Unit, + Proposal(CellGroupProposal), + Status(CellGroupStatus), +} + +struct Request { + group: Option, + command: Command, + reply: oneshot::Sender>, + bytes: usize, + _permit: Permit, + inbound: Option, +} + +impl CellGroupRuntime { + /// Start parked: no recovered group is attached without its verified root + /// and catalog/member permit. The engine admits only one node runtime. + pub async fn start( + log: ReplicaLog, + options: RaftRuntimeOptions, + ) -> Result<(Self, CellGroupRuntimeEvents), RuntimeError> { + options.validate(log.limits())?; + // Reject an engine containing the superseded application family. + log.cell_groups()?; + let lease = log.claim_runtime()?; + let domain = log.domain(); + let (sender, requests) = mailbox::channel(options.max_requests + COMPLETION_SLOTS); + let (events, receiver) = mpsc::channel(options.max_events); + let closing = Arc::new(AtomicBool::new(false)); + let terminal = Arc::new(Terminal::new()); + let stop = Arc::new(Notify::new()); + let (ready, wait) = oneshot::channel(); + let handle = Self { + inner: Arc::new(Handle { + log: log.clone(), + domain: domain.clone(), + budget: Budget::new(options.max_request_bytes, options.max_requests), + completions: Budget::new(options.max_ready_bytes, COMPLETION_SLOTS), + options: options.clone(), + requests: sender, + closing: closing.clone(), + terminal: terminal.clone(), + stop: stop.clone(), + }), + }; + let worker_domain = domain.clone(); + domain + .spawn_detached("cell-group-consensus", async move { + struct Closed(Arc, Arc); + + impl Drop for Closed { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + self.1.complete(Err(RuntimeError::Closed)); + } + } + let _closed = Closed(closing.clone(), terminal.clone()); + let mut worker = worker::Worker::new(log, options, worker_domain, events); + if ready.send(()).is_err() { + lease.release(); + return; + } + let result = worker.run(requests, &closing, &stop).await; + closing.store(true, Ordering::Release); + let fence = worker.finish().await; + let result = result.and(fence); + drop(worker); + if result.is_ok() { + lease.release(); + } + terminal.complete(result); + }) + .map_err(|_| RuntimeError::Closed)?; + wait.await.map_err(|_| RuntimeError::Closed)?; + handle.admit()?; + Ok((handle, CellGroupRuntimeEvents { receiver })) + } + + fn admit(&self) -> Result<(), RuntimeError> { + if self.inner.closing.load(Ordering::Acquire) || self.inner.domain.mono_ms().is_err() { + return Err(RuntimeError::Closed); + } + Ok(()) + } + + async fn call( + &self, + group: Option, + command: Command, + bytes: usize, + ) -> Result { + self.admit()?; + let completion = matches!(command, Command::Apply(..)); + let budget = if completion { + &self.inner.completions + } else { + &self.inner.budget + }; + if bytes > budget.capacity().0 { + return Err(RuntimeError::Overloaded); + } + let permit = loop { + let wake = budget.wake.notified(); + tokio::pin!(wake); + wake.as_mut().enable(); + self.admit()?; + if let Some(permit) = budget.acquire(bytes) { + break permit; + } + if !completion { + return Err(RuntimeError::Overloaded); + } + crate::asyncrt::select! { + _ = wake => (), + _ = self.inner.terminal.wait() => return Err(RuntimeError::Closed), + } + }; + let (reply, wait) = oneshot::channel(); + self.inner + .requests + .try_send(Request { + group, + command, + reply, + bytes, + _permit: permit, + inbound: None, + }) + .map_err(|error| match error { + mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + })?; + let result = wait.await.map_err(|_| RuntimeError::Closed)?; + self.admit()?; + result + } + + pub async fn attach( + &self, + permit: CellGroupBootstrapPermit, + current: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + self.admit()?; + let history = permit.route().history; + let local = self.inner.log.cell_group(&history)?; + if !Arc::ptr_eq(&local.owner, &permit.group().owner) { + return Err(RuntimeError::StaleCompletion); + } + let mut seed = Sha256::new(); + let election_seed = self + .inner + .domain + .run(async { + self.inner.options.election_seed.unwrap_or_else(|| { + use rand::RngCore as _; + let mut seed = [0; 32]; + crate::asyncrt::rng("cell-group-election").fill_bytes(&mut seed); + seed + }) + }) + .await + .map_err(|_| RuntimeError::Closed)?; + seed.update(election_seed); + seed.update(history.group.bytes()); + seed.update(history.incarnation.bytes()); + seed.update(local.identity().member.get().to_be_bytes()); + seed.update(local.identity().node.bytes()); + let bytes = current.reserved_bytes() + 4096; + let owner = self + .inner + .domain + .run(CellGroupOwner::start( + permit, + current, + self.inner.options.clone(), + seed.finalize().into(), + )) + .await + .map_err(|_| RuntimeError::Closed)??; + self.call(Some(history), Command::Attach(Box::new(owner)), bytes) + .await + .map(|_| ()) + } + + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { + self.call(Some(history), Command::Campaign, 4096) + .await + .map(|_| ()) + } + + pub async fn step(&self, history: GroupHistory, message: Message) -> Result<(), RuntimeError> { + let bytes = message.compute_size() as usize; + if bytes > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::Step(message), bytes + 4096) + .await + .map(|_| ()) + } + + pub async fn propose( + &self, + history: GroupHistory, + data: Vec, + ) -> Result { + let bytes = data.len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + match self + .call(Some(history), Command::Propose(data), bytes + 4096) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("proposal reply".into())), + } + } + + /// Queue an authenticated peer frame without waiting for protocol work. + /// Node/group admission remains owned until execution or discard. Success + /// is neither a Raft acknowledgement nor persistence/application proof. + pub async fn admit_step( + &self, + history: GroupHistory, + message: Message, + ) -> Result<(), RuntimeError> { + let bytes = message.compute_size() as usize; + if bytes > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::AdmitStep(message), bytes + 4096) + .await + .map(|_| ()) + } + + pub async fn read_index( + &self, + history: GroupHistory, + context: Vec, + ) -> Result<(), RuntimeError> { + let bytes = context.len(); + if bytes > 4096 { + return Err(RuntimeError::Overloaded); + } + self.call(Some(history), Command::ReadIndex(context), bytes + 4096) + .await + .map(|_| ()) + } + + pub async fn complete_apply( + &self, + history: GroupHistory, + work: CellGroupApply, + next: PreparedGroupManifest, + ) -> Result<(), RuntimeError> { + let bytes = next.reserved_bytes() + 4096; + self.call( + Some(history), + Command::Apply(work.ticket, Box::new(next)), + bytes, + ) + .await + .map(|_| ()) + } + + pub async fn status(&self, history: GroupHistory) -> Result { + match self.call(Some(history), Command::Status, 4096).await? { + Response::Status(status) => Ok(status), + _ => Err(RuntimeError::Protocol("status reply".into())), + } + } + + pub async fn tick(&self) -> Result<(), RuntimeError> { + self.call(None, Command::Tick, 4096).await.map(|_| ()) + } + + /// Close admission, join submitted persistence and release runtime ownership + /// only after a final engine fence. Cancelling this waiter does not cancel closure. + pub async fn close(&self) -> Result<(), RuntimeError> { + self.inner.closing.store(true, Ordering::Release); + self.inner.stop.notify_one(); + self.inner.terminal.wait().await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs new file mode 100644 index 000000000..a523a13c7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -0,0 +1,606 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fair group turns, node-wide persistence batches and joined shutdown. + +use super::*; +use crate::raft::storage::{LogError, SyncReceipt}; +use cellar_logic::consensus::OperationId; +use futures_util::future::BoxFuture; +use std::collections::{BTreeMap, VecDeque}; + +struct Group { + owner: Box, + queued: bool, + deferred: VecDeque, + completion: Option, + inbound: Arc, + tick: bool, + write: Option>, + write_bytes: usize, + pending: Option>, + settled: Option, + failed: Option, + usage: (usize, usize, usize), +} + +impl Group { + fn wake(&mut self, history: GroupHistory, order: &mut VecDeque) { + if !self.queued && (!self.owner.waiting() || self.failed.is_some()) { + self.queued = true; + order.push_back(history); + } + } + + fn usage(&self) -> (usize, usize, usize) { + let (ready, apply, reads) = self.owner.usage(); + let pending = self + .pending + .as_ref() + .map_or(0, |e| e.as_ref().map_or(4096, CellGroupEffect::bytes)); + ( + ready + .saturating_add(pending) + .saturating_add(self.write_bytes) + .saturating_add(self.settled.as_ref().map_or(0, CellGroupEffect::bytes)), + apply, + reads, + ) + } + + fn fail(&mut self, error: RuntimeError) { + self.failed = Some(error.clone()); + self.pending = Some(Err(error.clone())); + self.settled = None; + self.write = None; + self.write_bytes = 0; + self.tick = false; + for request in self.deferred.drain(..) { + reply(request, Err(error.clone())); + } + if let Some(request) = self.completion.take() { + reply(request, Err(error)); + } + } +} + +fn reply(request: Request, result: Result) { + let Request { + reply, + command, + _permit, + inbound, + .. + } = request; + drop(command); + drop(_permit); + drop(inbound); + let _ = reply.send(result); +} + +struct Job { + history: GroupHistory, + work: Box, + bytes: usize, +} + +type Batch = ( + Vec, + OperationId, + Result, LogError>, +); + +pub(super) struct Worker { + log: ReplicaLog, + options: RaftRuntimeOptions, + domain: crate::asyncrt::DomainToken, + groups: BTreeMap, + order: VecDeque, + writes: VecDeque, + events: mpsc::Sender, + outbox: VecDeque, + output: Arc, + inflight: Option>, + inflight_bytes: usize, + sequence: u64, + retry_at: u64, + ready: usize, + apply: usize, + reads: usize, +} + +impl Worker { + pub(super) fn new( + log: ReplicaLog, + options: RaftRuntimeOptions, + domain: crate::asyncrt::DomainToken, + events: mpsc::Sender, + ) -> Self { + Self { + log, + output: Budget::new(options.max_event_bytes, options.max_events), + options, + domain, + groups: BTreeMap::new(), + order: VecDeque::new(), + writes: VecDeque::new(), + events, + outbox: VecDeque::new(), + inflight: None, + inflight_bytes: 0, + sequence: 0, + retry_at: 0, + ready: 0, + apply: 0, + reads: 0, + } + } + + fn refresh(&mut self, history: GroupHistory) { + let group = self + .groups + .get_mut(&history) + .expect("scheduled group exists"); + let usage = group.usage(); + self.ready = self.ready - group.usage.0 + usage.0; + self.apply = self.apply - group.usage.1 + usage.1; + self.reads = self.reads - group.usage.2 + usage.2; + group.usage = usage; + } + + fn dispatch(&mut self, mut request: Request) { + if matches!(request.command, Command::Tick) { + self.tick(); + reply(request, Ok(Response::Unit)); + return; + } + let Some(history) = request.group else { + reply(request, Err(RuntimeError::NotFound)); + return; + }; + if matches!(request.command, Command::Attach(_)) { + if self.groups.contains_key(&history) { + reply(request, Err(RuntimeError::AlreadyAttached)); + return; + } + if self.groups.len() >= self.options.max_groups { + reply(request, Err(RuntimeError::Overloaded)); + return; + } + let Request { + command: Command::Attach(owner), + reply: answer, + _permit, + .. + } = request + else { + unreachable!() + }; + let mut group = Group { + owner, + queued: false, + deferred: VecDeque::new(), + completion: None, + inbound: Budget::new( + (self.options.max_request_bytes / 4).max(self.options.max_message_bytes + 4096), + (self.options.max_requests / 8).clamp(1, 32), + ), + tick: false, + write: None, + write_bytes: 0, + pending: None, + settled: None, + failed: None, + usage: (0, 0, 0), + }; + group.wake(history, &mut self.order); + self.groups.insert(history, group); + self.refresh(history); + drop(_permit); + let _ = answer.send(Ok(Response::Unit)); + return; + } + let Some(group) = self.groups.get_mut(&history) else { + reply(request, Err(RuntimeError::NotFound)); + return; + }; + if let Some(error) = &group.failed { + reply(request, Err(error.clone())); + return; + } + if matches!(request.command, Command::Status) { + reply(request, group.owner.status().map(Response::Status)); + return; + } + if let Command::AdmitStep(message) = &request.command { + if let Err(error) = group.owner.validate_message(message) { + reply(request, Err(error)); + return; + } + let Some(inbound) = group.inbound.acquire(request.bytes) else { + reply(request, Err(RuntimeError::Overloaded)); + return; + }; + let Request { + command: Command::AdmitStep(message), + reply: answer, + bytes, + _permit, + .. + } = request + else { + unreachable!() + }; + let (reply, _) = oneshot::channel(); + group.deferred.push_back(Request { + group: Some(history), + command: Command::Step(message), + reply, + bytes, + _permit, + inbound: Some(inbound), + }); + group.wake(history, &mut self.order); + let _ = answer.send(Ok(Response::Unit)); + return; + } + if matches!(request.command, Command::Apply(..)) { + if group.completion.is_some() { + reply(request, Err(RuntimeError::Overloaded)); + return; + } + group.completion = Some(request); + } else { + let Some(permit) = group.inbound.acquire(request.bytes) else { + reply(request, Err(RuntimeError::Overloaded)); + return; + }; + request.inbound = Some(permit); + group.deferred.push_back(request); + } + group.wake(history, &mut self.order); + } + + fn tick(&mut self) { + for (history, group) in &mut self.groups { + if group.failed.is_none() { + group.tick = true; + group.wake(*history, &mut self.order); + } + } + } + + fn progress(&mut self) -> Result { + let mut made = false; + for _ in 0..self.order.len() { + let Some(history) = self.order.pop_front() else { + break; + }; + let collect = self.ready.saturating_add(self.inflight_bytes) + < self.options.max_ready_bytes + && self.apply < self.options.max_apply_bytes; + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + group.queued = false; + if group.failed.is_none() { + if let Some(request) = group.completion.take() { + let Command::Apply(ticket, next) = &request.command else { + return Err(RuntimeError::Protocol("completion queue".into())); + }; + let result = group.owner.complete_apply(ticket.clone(), (**next).clone()); + if matches!(result, Err(RuntimeError::NotReady)) { + group.completion = Some(request); + } else { + reply(request, result.map(|()| Response::Unit)); + made = true; + } + } + if collect && group.owner.idle() && group.pending.is_none() { + if group.tick { + match group.owner.tick() { + Ok(()) => { + group.tick = false; + made = true; + } + Err(RuntimeError::NotReady) => (), + Err(error) => group.fail(error), + } + } else if let Some(request) = group.deferred.pop_front() { + let Request { + command, + reply: answer, + _permit, + inbound, + .. + } = request; + let new_read = matches!(command, Command::ReadIndex(_)) + || matches!(&command, Command::Step(m) if m.get_msg_type()==::raft::prelude::MessageType::MsgReadIndex); + let result = if new_read && self.reads >= self.options.max_pending_reads { + Err(RuntimeError::Overloaded) + } else { + match command { + Command::Campaign => { + group.owner.campaign().map(|()| Response::Unit) + } + Command::Step(message) => { + group.owner.step(message).map(|()| Response::Unit) + } + Command::Propose(data) => { + group.owner.propose(data).map(Response::Proposal) + } + Command::ReadIndex(context) => { + group.owner.read_index(context).map(|()| Response::Unit) + } + _ => Err(RuntimeError::Protocol("ordinary queue".into())), + } + }; + drop(_permit); + drop(inbound); + let _ = answer.send(result); + made = true; + } + } + if group.write.is_none() + && (group.pending.is_none() && group.settled.is_none() + || group.owner.control_pending()) + { + let emit = group.pending.is_none() && group.settled.is_none(); + let control = group.owner.control_pending(); + match group.owner.poll_budget(collect && emit, emit) { + Ok(Some(CellGroupEffect::Persist(work))) => { + group.write_bytes = work.write.estimated_bytes()?; + group.write = Some(work); + self.writes.push_back(history); + made = true; + } + Ok(Some(effect)) => { + if group.pending.is_none() { + group.pending = Some(Ok(effect)); + } else { + if !matches!(effect, CellGroupEffect::Applied { .. }) + || group.settled.is_some() + { + return Err(RuntimeError::Protocol( + "reserved physical completion output".into(), + )); + } + group.settled = Some(effect); + } + made = true; + } + Ok(None) => (), + Err(RuntimeError::Closed) => return Err(RuntimeError::Closed), + Err(error) => group.fail(error), + } + made |= control && !group.owner.control_pending(); + } + } + if group.pending.is_none() { + group.pending = group.settled.take().map(Ok); + } + if let Some(mut pending) = group.pending.take() { + let bytes = pending.as_ref().map_or(4096, CellGroupEffect::bytes); + if bytes > self.options.max_event_bytes { + group.fail(RuntimeError::Overloaded); + } else if let Some(permit) = self.output.acquire(bytes) { + let permit = Arc::new(permit); + if let Ok(CellGroupEffect::Apply(work)) = &mut pending { + work.ticket.admission = Some(permit.clone()); + } + self.outbox.push_back(CellGroupRuntimeEvent { + group: history, + effect: pending, + _permit: permit, + }); + made = true; + } else { + group.pending = Some(pending); + } + } + if group.pending.is_some() + || group.settled.is_some() + || (group.failed.is_none() + && (group.tick + || !group.deferred.is_empty() + || group.completion.is_some() + || group.owner.has_work())) + { + group.wake(history, &mut self.order); + } + self.refresh(history); + } + Ok(made) + } + + fn operation(&mut self) -> Result { + self.sequence = self + .sequence + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + Ok(OperationId::new(self.sequence)) + } + + fn start_batch(&mut self) -> Result { + if self.inflight.is_some() + || self.domain.mono_ms().map_err(|_| RuntimeError::Closed)? < self.retry_at + { + return Ok(false); + } + let mut jobs = Vec::new(); + let mut bytes = 1024; + for _ in 0..self.writes.len() { + let Some(history) = self.writes.pop_front() else { + break; + }; + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + let Some(work) = group.write.take() else { + continue; + }; + let size = std::mem::take(&mut group.write_bytes); + if size + 1024 > self.log.limits().max_batch_bytes { + group.fail(RuntimeError::Overloaded); + // The unsubmitted write is discarded; release its waiting ticket + // only by retiring this failed owner at node shutdown. + group.queued = true; + self.order.push_back(history); + } else if size + bytes <= self.log.limits().max_batch_bytes { + bytes += size; + jobs.push(Job { + history, + work, + bytes: size, + }); + } else { + group.write = Some(work); + group.write_bytes = size; + self.writes.push_front(history); + break; + } + self.refresh(history); + if jobs.len() >= self.options.max_batch_groups { + break; + } + } + if jobs.is_empty() { + return Ok(false); + } + let operation = self.operation()?; + let log = self.log.clone(); + self.inflight_bytes = bytes; + self.inflight = Some(Box::pin(async move { + let writes = jobs.iter().map(|job| job.work.write.clone()).collect(); + let result = log.persist_cells(operation, writes).await; + (jobs, operation, result) + })); + Ok(true) + } + + fn finish_batch(&mut self, batch: Batch) -> Result<(), RuntimeError> { + self.inflight = None; + self.inflight_bytes = 0; + let (jobs, operation, result) = batch; + match result { + Ok(receipt) => { + if receipt.operation != operation + || receipt.groups + != jobs + .iter() + .map(|j| j.work.write.group.identity().clone()) + .collect::>() + { + return Err(RuntimeError::StaleCompletion); + } + for job in jobs { + let group = self + .groups + .get_mut(&job.history) + .ok_or(RuntimeError::NotFound)?; + group.owner.complete_persist(CellGroupPersisted { + ticket: job.work.ticket, + sequence: receipt.sequence, + })?; + group.wake(job.history, &mut self.order); + self.refresh(job.history); + } + } + Err(LogError::Overloaded) => { + self.retry_at = self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .saturating_add(10); + for job in jobs.into_iter().rev() { + let group = self + .groups + .get_mut(&job.history) + .ok_or(RuntimeError::NotFound)?; + group.write = Some(job.work); + group.write_bytes = job.bytes; + self.writes.push_front(job.history); + self.refresh(job.history); + } + } + Err(error) => return Err(error.into()), + } + Ok(()) + } + + fn flush(&mut self) -> bool { + let mut made = false; + while let Some(event) = self.outbox.pop_front() { + match self.events.try_send(event) { + Ok(()) => made = true, + Err(error) => { + self.outbox.push_front(error.into_inner()); + break; + } + } + } + made + } + + pub(super) async fn run( + &mut self, + mut requests: mailbox::Receiver, + closing: &AtomicBool, + stop: &Notify, + ) -> Result<(), RuntimeError> { + let mut tick_at = self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .saturating_add(self.options.tick_interval_ms.unwrap_or(1000)); + loop { + if closing.load(Ordering::Acquire) { + return Ok(()); + } + let made = self.progress()? | self.start_batch()? | self.flush(); + let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + let deadline = if self.retry_at > now { + tick_at.min(self.retry_at) + } else { + tick_at + }; + let timer = self + .domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + let events = self.events.clone(); + let output = self.output.clone(); + let io = async { + match self.inflight.as_mut() { + Some(io) => io.await, + None => std::future::pending().await, + } + }; + crate::asyncrt::select! { + request = requests.recv() => { + let Some(request) = request else { return Ok(()); }; self.dispatch(request); + for _ in 1..64 { let Ok(request) = requests.try_recv() else { break; }; self.dispatch(request); } + } + _ = stop.notified() => return Ok(()), + result = io => self.finish_batch(result)?, + _ = timer => { let now=self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; if now>=tick_at { if self.options.tick_interval_ms.is_some() { self.tick(); } tick_at=now.saturating_add(self.options.tick_interval_ms.unwrap_or(1000)); } } + _ = events.closed() => return Ok(()), + capacity = events.reserve(), if !self.outbox.is_empty() => { if capacity.is_err() { return Ok(()); } } + _ = output.wake.notified() => (), + _ = std::future::ready(()), if made => (), + } + } + } + + pub(super) async fn finish(&mut self) -> Result<(), RuntimeError> { + if let Some(io) = self.inflight.take() { + let batch = io.await; + batch.2.as_ref().map_err(Clone::clone)?; + self.finish_batch(batch)?; + } + let operation = self.operation()?; + self.log.persist_cells(operation, Vec::new()).await?; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/runtime.rs b/crates/cellar/durability/raft/runtime.rs index 7d0312474..c11c8cd12 100644 --- a/crates/cellar/durability/raft/runtime.rs +++ b/crates/cellar/durability/raft/runtime.rs @@ -143,13 +143,20 @@ impl Drop for Requests { } } -struct Terminal { +pub(crate) struct Terminal { result: Mutex>>, changed: Notify, } impl Terminal { - fn complete(&self, result: Result<(), RuntimeError>) { + pub(crate) fn new() -> Self { + Self { + result: Mutex::new(None), + changed: Notify::new(), + } + } + + pub(crate) fn complete(&self, result: Result<(), RuntimeError>) { let mut terminal = self.result.lock().unwrap_or_else(|e| e.into_inner()); if terminal.is_none() { *terminal = Some(result); @@ -158,7 +165,7 @@ impl Terminal { self.changed.notify_waiters(); } - async fn wait(&self) -> Result<(), RuntimeError> { + pub(crate) async fn wait(&self) -> Result<(), RuntimeError> { loop { let changed = self.changed.notified(); tokio::pin!(changed); @@ -246,10 +253,7 @@ impl RaftRuntime { let (ready, wait) = oneshot::channel(); let closed = Arc::new(AtomicBool::new(false)); let closing = Arc::new(AtomicBool::new(false)); - let terminal = Arc::new(Terminal { - result: Mutex::new(None), - changed: Notify::new(), - }); + let terminal = Arc::new(Terminal::new()); let stop = Arc::new(Notify::new()); let requests = Requests { receiver: requests, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 3dd73b6ce..877e2ea8c 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -7,6 +7,29 @@ #[path = "support/raft_storage_fault.rs"] #[allow(dead_code)] mod fault; + +#[path = "support/cell_group_runtime.rs"] +mod runtime_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_replicates_independent_cells() -> Result<()> { + runtime_fixture::replicated_cells().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_cancelled_close_joins_physical_sync() -> Result<()> { + runtime_fixture::cancelled_close_joins_sync().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_batches_distinct_group_histories() -> Result<()> { + runtime_fixture::shared_batches().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_reserved_completion_releases_apply_admission() -> Result<()> { + runtime_fixture::bounded_apply_completion().await +} #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs new file mode 100644 index 000000000..90bb8b99b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -0,0 +1,451 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real node workers share the existing SQLite/image fixture and engine owners. + +use super::*; +use std::collections::BTreeMap; + +struct RuntimeCluster { + cluster: Cluster, + runtimes: Vec, + events: Vec, + applied: BTreeMap>, + extra: BTreeMap>, +} + +impl RuntimeCluster { + async fn new() -> Result { + Self::with_events(256).await + } + + async fn with_events(max_events: usize) -> Result { + let mut cluster = Cluster::new().await?; + let mut runtimes = Vec::new(); + let mut events = Vec::new(); + let mut applied = Vec::new(); + for n in 0..3 { + let owner = cluster.voters[n].owner.take().context("old direct owner")?; + let root = owner.manifest().clone(); + drop(owner); + applied.push(root.marker().position().index); + let (runtime, receiver) = CellGroupRuntime::start( + cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + max_events, + ..Default::default() + }, + ) + .await?; + runtime.attach(cluster.permit(n).await?, root).await?; + runtimes.push(runtime); + events.push(receiver); + } + let applied = BTreeMap::from([(cluster.birth.route().history, applied)]); + Ok(Self { + cluster, + runtimes, + events, + applied, + extra: BTreeMap::new(), + }) + } + + async fn pump( + &mut self, + hold: &[GroupHistory], + held: &mut BTreeMap, + ) -> Result<()> { + let primary = self.cluster.birth.route().history; + let mut messages = Vec::new(); + for n in 0..3 { + for _ in 0..64 { + let Ok(event) = self.events[n].try_recv() else { + break; + }; + let scope = event.group(); + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) => messages.push((scope, message.clone())), + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("retained apply")?; + if n == 0 && hold.contains(&scope) { + assert!(held.insert(scope, work).is_none()); + } else { + let voter = if scope == primary { + &mut self.cluster.voters[n] + } else { + &mut self.extra.get_mut(&scope).context("extra voter")?[n] + }; + let next = prepare(voter, &work)?; + self.runtimes[n].complete_apply(scope, work, next).await?; + } + } + CellGroupEffect::Applied { manifest, .. } => { + self.applied.get_mut(&scope).context("applied")?[n] = + manifest.position().index + } + CellGroupEffect::ReadIndex { .. } => (), + CellGroupEffect::Persist(_) => anyhow::bail!("node worker leaked persistence"), + } + } + } + for result in + futures_util::future::join_all(messages.into_iter().map(|(scope, message)| { + self.runtimes[message.to as usize - 1].admit_step(scope, message) + })) + .await + { + result?; + } + Ok(()) + } + + async fn settle(&mut self, histories: &[GroupHistory]) -> Result<()> { + tokio::time::timeout(Duration::from_secs(20), async { + loop { + self.pump(&[], &mut BTreeMap::new()).await?; + let mut done = true; + for &scope in histories { + let status = self.runtimes[0].status(scope).await?; + done &= status.ready + && self.applied[&scope] + .iter() + .all(|index| *index >= status.applied.index); + } + if done { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? + } + + async fn drive(&mut self, target: Option) -> Result<()> { + let history = self.cluster.birth.route().history; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + self.pump(&[], &mut BTreeMap::new()).await?; + let status = self.runtimes[0].status(history).await?; + let target = target.unwrap_or(status.applied.index); + if status.ready && self.applied[&history].iter().all(|index| *index >= target) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? + } + + async fn close(self) -> Result<()> { + for runtime in &self.runtimes { + runtime.close().await?; + } + drop(self.events); + drop(self.runtimes); + drop(self.extra); + self.cluster.close().await + } +} + +pub(super) async fn replicated_cells() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + for name in ["worker-a", "worker-b", "worker-c"] { + let cell = history(&route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose(route.history, request(&route, cell, 1, &cut)?) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + } + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + assert_eq!( + group + .group_manifest()? + .context("published group")? + .root() + .cells(), + 3 + ); + } + fixture.close().await +} + +pub(super) async fn cancelled_close_joins_sync() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + fixture.runtimes[0] + .propose( + route.history, + request(&route, history(&route, "closing")?, 1, &cut)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[0].clone(); + let closing = runtime.close(); + assert!(tokio::time::timeout(Duration::from_millis(30), closing) + .await + .is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + drop(release); + tokio::time::timeout(Duration::from_secs(5), runtime.close()).await??; + let (replacement, events) = CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + fixture.close().await +} + +impl RuntimeCluster { + async fn add_group(&mut self, number: u8) -> Result { + let scope = GroupHistory { + cluster: self.cluster.birth.route().history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let domain = &self.cluster.domain; + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + [6; 32], + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let initialization = directory.prepare_initialization( + ProposalId::new([number + 1; 16])?, + &[request.route().clone()], + )?; + domain.run(directory.publish(&initialization)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + let mut voters = Vec::new(); + let mut applied = Vec::new(); + for n in 0..3 { + let log = self.cluster.voters[n].log.clone(); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let base = self + .cluster + ._root + .path() + .join(format!("extra-{number}-node-{n}")); + let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let marker = permit.group().group_manifest()?.context("genesis")?; + applied.push(marker.position().index); + let root = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| panic!("empty genesis"), + )?; + self.runtimes[n].attach(permit, root).await?; + voters.push(Voter { + log, + owner: None, + pages, + images, + tree, + gate: self.cluster.voters[n].gate.clone(), + paused: false, + held: None, + }); + } + self.applied.insert(scope, applied); + self.extra.insert(scope, voters); + Ok(scope) + } +} + +pub(super) async fn shared_batches() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let second = fixture.add_group(31).await?; + let third = fixture.add_group(41).await?; + let histories = [fixture.cluster.birth.route().history, second, third]; + let before = fixture.cluster.voters[0].log.performance_snapshot(); + for result in futures_util::future::join_all( + histories + .iter() + .map(|&history| fixture.runtimes[0].campaign(history)), + ) + .await + { + result?; + } + fixture.settle(&histories).await?; + let after = fixture.cluster.voters[0].log.performance_snapshot(); + assert!(before.available && after.available && !before.saturated && !after.saturated); + let groups = |s: &RaftPerformanceSnapshot| { + s.counters + .iter() + .find(|c| c.name == "engine_persist_groups") + .unwrap() + .value + }; + let batches = |s: &RaftPerformanceSnapshot| { + s.operations + .iter() + .find(|o| o.name == "engine_persist") + .unwrap() + .succeeded + }; + let group_delta = groups(&after) - groups(&before); + let batch_delta = batches(&after) - batches(&before); + assert!(batch_delta > 0 && group_delta > batch_delta, "actual engine batches must include multiple groups: {group_delta} groups / {batch_delta} batches"); + fixture.close().await +} + +pub(super) async fn bounded_apply_completion() -> Result<()> { + let mut fixture = RuntimeCluster::with_events(2).await?; + let first = fixture.cluster.birth.route().history; + let second = fixture.add_group(51).await?; + let histories = [first, second]; + for result in futures_util::future::join_all( + histories + .iter() + .map(|&scope| fixture.runtimes[0].campaign(scope)), + ) + .await + { + result?; + } + fixture + .settle(&histories) + .await + .context("settling bounded-output groups")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (7);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut targets = BTreeMap::new(); + for scope in histories { + let route = if scope == first { + fixture.cluster.birth.route().clone() + } else { + fixture.extra[&scope][0] + .log + .cell_group(&scope)? + .group_manifest()? + .context("route")? + .route() + .clone() + }; + let accepted = fixture.runtimes[0] + .propose(scope, request(&route, history(&route, "quota")?, 1, &cut)?) + .await?; + targets.insert(scope, accepted.position.index); + } + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + while held.len() != 2 { + fixture.pump(&histories, &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| { + format!( + "bounded apply progress: held={}, applied={:?}", + held.len(), + fixture.applied + ) + })??; + // Both output slots are retained by physical work. Completing one must + // release its reservation before admitting its distinct Applied event. + let work = held.remove(&first).context("first physical job")?; + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(first, work, next) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&first][0] < targets[&first] { + fixture.pump(&[second], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| { + format!( + "bounded apply progress: held={}, applied={:?}", + held.len(), + fixture.applied + ) + })??; + assert_eq!(held.len(), 1, "another group's physical job remains held"); + let work = held.remove(&second).context("second physical job")?; + let next = prepare( + &mut fixture.extra.get_mut(&second).context("second voter")?[0], + &work, + )?; + fixture.runtimes[0] + .complete_apply(second, work, next) + .await?; + fixture + .settle(&histories) + .await + .context("settling bounded-output groups")?; + for scope in histories { + assert!(fixture.applied[&scope] + .iter() + .all(|index| *index >= targets[&scope])); + } + fixture.close().await +} From 05d665e56a62f2f69f6fcc3d146f82f0aa95e988 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:01:30 -0700 Subject: [PATCH 043/568] docs: retain shared-node runtime qualification --- plans/multi-raft/cell-groups/README.md | 8 ++-- plans/multi-raft/cell-groups/baseline.json | 12 ++--- plans/multi-raft/cell-groups/handoff.md | 51 ++++++++++------------ plans/multi-raft/cell-groups/progress.md | 49 +++++++++++++++++++-- plans/multi-raft/cell-groups/tasks.json | 5 ++- 5 files changed, 82 insertions(+), 43 deletions(-) diff --git a/plans/multi-raft/cell-groups/README.md b/plans/multi-raft/cell-groups/README.md index 0306250fa..8824e4e73 100644 --- a/plans/multi-raft/cell-groups/README.md +++ b/plans/multi-raft/cell-groups/README.md @@ -9,9 +9,9 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The requested deliverable is this executable design and implementation plan. -See the [executor handoff](handoff.md) for the starting revision, uncommitted -runtime draft, latest-main integration requirement, commands and completion +The active goal implements this design end to end and hard cuts over the runtime. +See the [executor handoff](handoff.md) for the starting revision, shared-node +runtime checkpoint, latest-main integration requirement, commands and completion boundary. Plan delivery or an intermediate task gate does not qualify the complete shared-group runtime. @@ -93,7 +93,7 @@ directory along with the earlier DONE-task evidence to the implementation executor. CG-04 now retains creation identity with the selected image and prepares ordered enrollment and exact retries. Its physical group owner consumes the catalog startup permit and verified selected image/manifest closure, and -obtains current-term quorum before protocol admission. Shared scheduler, SQL +obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. Per-cell SQL execution, receipt settlement and serving integration remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/plans/multi-raft/cell-groups/baseline.json b/plans/multi-raft/cell-groups/baseline.json index a679dda2a..a51cabfeb 100644 --- a/plans/multi-raft/cell-groups/baseline.json +++ b/plans/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "51e30eabde1ca76bf0b23963dd06d0e8130f8ee7", + "source_revision": "865f253da0a14d730b62f77b17cefc6d1497890b", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -49,14 +49,14 @@ }, "runtime": { "path": "crates/cellar/durability/raft/runtime.rs", - "sha256": "96b2b60c65333f0a16c8e2a538399ecdbc1c4456c14e504e904feb1bc41f655d", - "line": 727, + "sha256": "8f35279c42414980eca58899cdc87e17c98b45ae26bac9c68e1209a58452b890", + "line": 731, "excerpt": "struct Group {\n replica: Replica,\n scheduled: bool,\n write_scheduled: bool,\n deferred: VecDeque,\n inbound: Arc,\n hints: BTreeMap,\n tick: bool,\n failed: Option," }, "ticks": { "path": "crates/cellar/durability/raft/runtime.rs", - "sha256": "96b2b60c65333f0a16c8e2a538399ecdbc1c4456c14e504e904feb1bc41f655d", - "line": 1055, + "sha256": "8f35279c42414980eca58899cdc87e17c98b45ae26bac9c68e1209a58452b890", + "line": 1059, "excerpt": " fn tick(&mut self) {\n for (scope, group) in &mut self.groups {\n group.tick = true;\n group.wake(scope, &mut self.order);\n }\n }\n\n fn progress(&mut self) -> Result {\n let mut made = false;" }, "catalog": { diff --git a/plans/multi-raft/cell-groups/handoff.md b/plans/multi-raft/cell-groups/handoff.md index d10fa19e0..c33e400d7 100644 --- a/plans/multi-raft/cell-groups/handoff.md +++ b/plans/multi-raft/cell-groups/handoff.md @@ -2,8 +2,8 @@ # Cell-group implementation handoff -This delivery is the executable design and implementation plan requested by the -user. End-to-end implementation and hard cutover remain the executor's work. +The active goal is end-to-end implementation and hard cutover from the executable +design. Continue through the full task and release gates. This handoff does not claim completion of the shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many @@ -14,8 +14,8 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [draft PR 160](https://github.com/crabbuild/cellar/pull/160). -The physical owner implementation and reviewed source baseline are at -`51e30eabde1ca76bf0b23963dd06d0e8130f8ee7`; see the latest checkpoint in +The shared-node runtime implementation and reviewed source baseline are at +`865f253d`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch includes `1be27d234d4674d4d014785c60cecbfe394bdab9`. The latest fetch found `origin/main` at `8a93ffd7` (documentation consolidation, PR 161), which is @@ -37,25 +37,19 @@ Read [design](design.md), [protocols](protocols.md), dependencies, steps, exact tests and acceptance oracles. `profiles.json` freezes candidate workload and performance gates; those numbers are targets, not results. -## Working draft at plan delivery - -The checkout also contains an uncommitted CG-04 node-runtime draft in -`crates/cellar/durability/raft/cell_groups/runtime/`, owner scheduling changes, -shared shutdown completion in `crates/cellar/durability/raft/runtime.rs`, and -native tests in `crates/cellar/tests/support/cell_group_runtime.rs` and -`crates/cellar/tests/native_cell_groups_execution.rs`. These files are not part -of the source-bound physical-owner checkpoint. Preserve and review the whole -diff, including untracked files, before continuing it. A fresh checkout of the -PR does not contain this working draft. - -Package validation passes with 16 tasks, 86 required cases, 14 invariants and -nine qualification profiles. All 37 validator tests pass against a clean export -of committed revision `55317523ebcb34c82e9f80ad1030f9282d1997a6`. In the working -checkout, preflight and its corresponding unit test correctly refuse the changed -`runtime.rs`; the other 36 validator tests pass. Do not refresh its hash merely -to remove that refusal. Review and qualify the runtime draft, commit its source, -then update the baseline with that revision and reviewed excerpts. Neither these -plan checks nor the draft's intermediate test runs complete CG-04. +## Current implementation checkpoint + +The CG-04 node runtime is committed in +`crates/cellar/durability/raft/cell_groups/runtime/`. It multiplexes physical owners, +batches engine persistence across groups, retains event admission through apply, +and joins submitted physical work on close. Native cases use independent SQLite +images and actual node workers; see [progress](progress.md) for qualification. +The prior uncommitted runtime draft has been incorporated into this source. + +The next integration is per-cell SQL ownership and fair admission, concurrent +initialization, exact receipt settlement and coherent read serving. Preserve the +node worker's physical completion and cancellation boundaries. Do not treat the +shared scheduler's isolated tests as completion of the six-case CG-04 gate. ## First executor session @@ -63,8 +57,9 @@ Inspect `git status` and the existing diff before editing. The physical owner in `crates/cellar/durability/raft/cell_groups/owner.rs` and its native execution fixture now use real three-voter Raft with complete group manifests and separate SQLite images. The initial fixture compilation failures have been repaired. -The owner is not yet connected to the shared scheduler or public SDK. Preserve -its persistence, completion and physical lease boundaries during integration; +The owner now runs on the shared node scheduler; public SDK and per-cell SQL +integration remain incomplete. Preserve its persistence, completion and physical +lease boundaries during integration; do not mistake the low-level owner cases for the full CG-04 gate. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -91,9 +86,9 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The current partial CG-04 evidence is retained at -`.cellar/cell-groups-evidence/CG-04-owner/checkpoint.json`. Transfer that directory -and the failed gate named by its record when handing off this checkpoint. It -records six physical-owner cases and the catalog/storage regressions, with +`.cellar/cell-groups-evidence/CG-04-runtime/checkpoint.json`. Transfer that +directory alongside the earlier owner checkpoint and failed gate. The runtime +record covers 59 selected Rust cases and eight verification commands, with explicit limitations; it cannot stand in for the complete six-case CG-04 contract. ## Delivery sequence diff --git a/plans/multi-raft/cell-groups/progress.md b/plans/multi-raft/cell-groups/progress.md index 7292dd8a4..a0fa88acf 100644 --- a/plans/multi-raft/cell-groups/progress.md +++ b/plans/multi-raft/cell-groups/progress.md @@ -2,10 +2,51 @@ # Cell-group implementation progress -The full end-to-end implementation and hard-cutover goal is active again after -the plan delivery; see the [executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and -CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify a shared -runtime. No production legacy/shared mode flag is being added. +The end-to-end implementation and hard-cutover goal remains active; see the +[executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; +CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts +do not qualify the public shared runtime. No production legacy/shared mode flag +is being added. + +## CG-04 shared-node runtime + +Source: `865f253da0a14d730b62f77b17cefc6d1497890b`. + +One execution-domain task now multiplexes physical cell-group owners on each +node. It uses the existing engine, mailbox and admission primitives, with a +single in-flight persistence batch containing writes from multiple group +histories. Exact engine receipts complete the corresponding owner tickets. +Group turns and deferred transport/proposal work remain bounded; ticks coalesce. +An admitted transport frame is not a persistence or application acknowledgement. + +Apply events retain admission through image preparation and physical root +publication. Completion requests have reserved capacity. Already-started physical +phases can finish when ordinary event delivery is full; one reserved settlement +slot permits releasing an apply reservation without admitting more ordinary +output. A two-slot test exposed the original circular wait, and the fix changes +phase progress rather than increasing the limit. + +Shutdown closes admission and joins submitted engine work before releasing node +runtime ownership. Cancelling the caller's close waiter cannot release that +ownership. The existing runtime shares the terminal completion primitive with +the new worker; its behavior remains covered by its native regression suite. + +All 59 selected Rust cases pass: ten execution, 15 catalog, 15 active storage +and 19 active existing-runtime regressions. Two process helpers are ignored by +direct execution and exercised by their parent tests. All-target Clippy for +`cellar`/`cellar-logic` with warnings denied, formatting (1,786 Rust files), layout, +native dependency isolation, docs, source preflight and all 37 plan-validator +tests pass. The retained partial record is +`.cellar/cell-groups-evidence/CG-04-runtime/checkpoint.json`, binding 85 source +inputs, eight command logs and four native executables. The initial two-slot +failure and successful intermediate repeats are retained without claiming a +source-bound intermediate qualification. CG-04 remains IN_PROGRESS. + +This is the shared consensus/persistence scheduler. Per-cell SQL preparation and +fair admission, concurrent initialization, exact receipt settlement and SDK read +serving still need integration. The full CG-04 gate remains required. Complete +group snapshots, membership/topology, managed startup, RustFS qualification and +the final hard cutover remain unfinished. ## CG-04 physical consensus owner checkpoint diff --git a/plans/multi-raft/cell-groups/tasks.json b/plans/multi-raft/cell-groups/tasks.json index 10b94bd8d..5e8ec88a2 100644 --- a/plans/multi-raft/cell-groups/tasks.json +++ b/plans/multi-raft/cell-groups/tasks.json @@ -445,7 +445,10 @@ "crates/cellar/durability/raft/cell_groups/owner.rs", "crates/cellar/tests/native_cell_groups_execution.rs", "crates/cellar/durability/raft/cell_groups/enrollment_apply.rs", - "crates/cellar/durability/raft/cell_groups/page_wire_tests.rs" + "crates/cellar/durability/raft/cell_groups/page_wire_tests.rs", + "crates/cellar/durability/raft/cell_groups/runtime/mod.rs", + "crates/cellar/durability/raft/cell_groups/runtime/worker.rs", + "crates/cellar/tests/support/cell_group_runtime.rs" ], "test_target": "native_cell_groups_execution", "steps": [ From aec4029e77124848dc6b155c098f8eec9ccdacc4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:08:32 -0700 Subject: [PATCH 044/568] fix: reserve node capacity before cell-group owner startup --- crates/cellar/durability/raft/README.md | 3 +- .../raft/cell_groups/runtime/mod.rs | 31 +++++++++-- .../tests/native_cell_groups_execution.rs | 6 +++ .../tests/support/cell_group_runtime.rs | 51 +++++++++++++++++++ 4 files changed, 86 insertions(+), 5 deletions(-) diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 7ee34ae0d..45fd2abf6 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -33,7 +33,8 @@ a coherent cell image after application through that index. The adapter currentl refuses snapshot/configuration/topology administration until those complete-group protocols are connected. `CellGroupRuntime` now multiplexes physical group owners on one execution-domain worker per node. It reuses the bounded mailbox, byte/item -admission, joined terminal result and existing persistence engine. A fair active +admission, joined terminal result and existing persistence engine. Attachment +reserves ordinary node admission before catalog revalidation or RawNode construction. A fair active queue collects writes across histories into one engine batch; queued and in-flight work stay charged. Peer-frame admission retains its node/group reservations until execution or discard and grants no Raft acknowledgement. Completion requests have diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index af2dcfe40..88325e667 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -211,6 +211,17 @@ impl CellGroupRuntime { _ = self.inner.terminal.wait() => return Err(RuntimeError::Closed), } }; + self.send_reserved(group, command, bytes, permit).await + } + + async fn send_reserved( + &self, + group: Option, + command: Command, + bytes: usize, + permit: Permit, + ) -> Result { + self.admit()?; let (reply, wait) = oneshot::channel(); self.inner .requests @@ -237,6 +248,14 @@ impl CellGroupRuntime { current: PreparedGroupManifest, ) -> Result<(), RuntimeError> { self.admit()?; + // Reserve before catalog revalidation and RawNode construction, keeping + // concurrent attachment work inside the node's ordinary admission cap. + let bytes = current.reserved_bytes() + 4096; + let admission = self + .inner + .budget + .acquire(bytes) + .ok_or(RuntimeError::Overloaded)?; let history = permit.route().history; let local = self.inner.log.cell_group(&history)?; if !Arc::ptr_eq(&local.owner, &permit.group().owner) { @@ -261,7 +280,6 @@ impl CellGroupRuntime { seed.update(history.incarnation.bytes()); seed.update(local.identity().member.get().to_be_bytes()); seed.update(local.identity().node.bytes()); - let bytes = current.reserved_bytes() + 4096; let owner = self .inner .domain @@ -273,9 +291,14 @@ impl CellGroupRuntime { )) .await .map_err(|_| RuntimeError::Closed)??; - self.call(Some(history), Command::Attach(Box::new(owner)), bytes) - .await - .map(|_| ()) + self.send_reserved( + Some(history), + Command::Attach(Box::new(owner)), + bytes, + admission, + ) + .await + .map(|_| ()) } pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 877e2ea8c..1eab0b2dc 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -30,6 +30,12 @@ async fn node_runtime_batches_distinct_group_histories() -> Result<()> { async fn node_runtime_reserved_completion_releases_apply_admission() -> Result<()> { runtime_fixture::bounded_apply_completion().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_attachment_reserves_before_owner() -> Result<()> { + runtime_fixture::attachment_reserves_before_owner().await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 90bb8b99b..9b13ea099 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -449,3 +449,54 @@ pub(super) async fn bounded_apply_completion() -> Result<()> { } fixture.close().await } + +// A pending peer/protocol request retains ordinary admission through a physical +// sync. Attachment must refuse before attempting to claim an existing owner. +pub(super) async fn attachment_reserves_before_owner() -> Result<()> { + let mut cluster = Cluster::new().await?; + let scope = cluster.birth.route().history; + let owner = cluster.voters[0].owner.take().context("direct owner")?; + let root = owner.manifest().clone(); + drop(owner); + let (runtime, events) = CellGroupRuntime::start( + cluster.voters[0].log.clone(), + RaftRuntimeOptions { + max_requests: 1, + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + runtime + .attach(cluster.permit(0).await?, root.clone()) + .await?; + let duplicate = cluster.permit(0).await?; + let gate = cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut vote = Message::default(); + vote.set_msg_type(MessageType::MsgRequestVote); + vote.from = 2; + vote.to = 1; + vote.term = root.marker().position().term.get() + 1; + vote.index = root.marker().position().index.get(); + vote.log_term = root.marker().position().term.get(); + runtime.step(scope, vote.clone()).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let mut pending = Box::pin(runtime.step(scope, vote)); + assert!( + tokio::time::timeout(Duration::from_millis(30), pending.as_mut()) + .await + .is_err() + ); + assert!(matches!( + runtime.attach(duplicate, root).await, + Err(RuntimeError::Overloaded) + )); + drop(release); + tokio::time::timeout(Duration::from_secs(5), pending).await??; + runtime.close().await?; + drop(events); + drop(runtime); + cluster.close().await +} From 960f30e4bc663309224088ccac06f4ccbb08a78c Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:12:09 -0700 Subject: [PATCH 045/568] docs: record node runtime admission qualification and SQL handoff --- .../multi-raft/cell-groups/baseline.json | 4 +-- docs/design/multi-raft/cell-groups/handoff.md | 17 ++++++++--- .../design/multi-raft/cell-groups/progress.md | 30 +++++++++++++++++++ 3 files changed, 45 insertions(+), 6 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index a51cabfeb..4af0eec0d 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "865f253da0a14d730b62f77b17cefc6d1497890b", + "source_revision": "aec4029e77124848dc6b155c098f8eec9ccdacc4", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index bdab7f895..71f2fe4f6 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -13,9 +13,9 @@ node resources. A group spans voter nodes; it is not all cells on one node. ## Starting point Use branch `codex/shared-cell-groups` and the existing -[draft PR 160](https://github.com/crabbuild/cellar/pull/160). +[PR 160](https://github.com/crabbuild/cellar/pull/160). The shared-node runtime implementation and reviewed source baseline are at -`865f253d`; see the latest checkpoint in +`aec4029e`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; @@ -47,6 +47,15 @@ initialization, exact receipt settlement and coherent read serving. Preserve the node worker's physical completion and cancellation boundaries. Do not treat the shared scheduler's isolated tests as completion of the six-case CG-04 gate. +The remaining coupling is concrete: `execution::CommandAuthority` binds the +old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; +and `host_sql::Pool` keys execution owners by that scope. Adapt those owning +components to a cell history and exact manifest/image pin, with group authority +kept separately. Reuse the existing receipt transaction and callback machinery; +do not manufacture a synthetic cell-scoped group to satisfy the old interfaces. +A later mutation of another cell must not settle the first cell's pending output. + + ## First executor session Inspect `git status` and the existing diff before editing. The physical owner in @@ -82,9 +91,9 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The current partial CG-04 evidence is retained at -`.cellar/cell-groups-evidence/CG-04-runtime/checkpoint.json`. Transfer that +`.cellar/cell-groups-evidence/CG-04-runtime-admission/checkpoint.json`. Transfer that directory alongside the earlier owner checkpoint and failed gate. The runtime -record covers 59 selected Rust cases and eight verification commands, with +record covers 60 selected Rust cases and eight verification commands, with explicit limitations; it cannot stand in for the complete six-case CG-04 contract. ## Delivery sequence diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 0ab266486..bfbf7f998 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -8,6 +8,36 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 attachment admission and main integration + +Source: `aec4029e77124848dc6b155c098f8eec9ccdacc4`. + +Attachment now reserves ordinary node capacity before catalog revalidation and +RawNode construction, retaining it through dispatch or failure. A public-runtime +regression holds a real vote's engine sync and fills the sole request slot; a +second attachment refuses with overload before claiming an owner. Releasing sync +allows the queued protocol work and shutdown to finish at the same limits. +The first fixture attempted the normal multi-peer pump with one slot and refused +transport overload before the target assertion; the final fixture isolates the +startup boundary without relaxing admission. + +The branch includes `origin/main` at `8a93ffd7`. Its documentation consolidation +moves this package to `docs/design/multi-raft/cell-groups/`. Commands, links, +catalog entries, source-root discovery and the plan-only release-input exclusion +now follow that location. Validator tests exercise both the new design path and +historical `plans/` bookkeeping. Existing evidence retains its original source, +paths and hashes; moved documentation is not fabricated runtime qualification. + +All 60 selected Rust cases pass: 11 execution, 15 catalog, 15 active storage +and 19 active existing-runtime cases. All-target Clippy with warnings denied, +formatting (1,786 Rust files), layout, dependency isolation, documentation, +source preflight and all 37 plan-validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-runtime-admission/checkpoint.json` retains +165 source inputs, eight command logs, four executables and intermediate failure +logs. The larger source inventory includes main's documentation/tooling changes. +CG-04 remains IN_PROGRESS; this does not qualify per-cell SQL integration, SDK +serving, real RustFS recovery, performance or the final hard cutover. + ## CG-04 shared-node runtime Source: `865f253da0a14d730b62f77b17cefc6d1497890b`. From 5c7dfb53d0acb689cf0401c085a2741fa7539cd3 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:37:34 -0700 Subject: [PATCH 046/568] docs: deliver executable cell-group plan and preserve SQL draft handoff --- docs/design/multi-raft/cell-groups/README.md | 11 ++++---- docs/design/multi-raft/cell-groups/handoff.md | 23 +++++++++++++-- .../design/multi-raft/cell-groups/progress.md | 28 +++++++++++++++++-- docs/design/multi-raft/cell-groups/tasks.json | 6 +++- 4 files changed, 58 insertions(+), 10 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index e5c255d58..266f52284 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -9,11 +9,12 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The active goal implements this design end to end and hard cuts over the runtime. -See the [executor handoff](handoff.md) for the starting revision, shared-node -runtime checkpoint, reviewed main integration, commands and completion -boundary. Plan delivery or an intermediate task gate does not qualify the -complete shared-group runtime. +The requested deliverable is the executable design and implementation plan. +End-to-end implementation and hard cutover are the executor's target, with +separate task and release gates. See the [executor handoff](handoff.md) for the +starting revision, shared-node runtime checkpoint, unfinished working-tree draft, +reviewed main integration and commands. Plan delivery or an intermediate task +gate does not qualify the complete shared-group runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 71f2fe4f6..62c5069f2 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -2,8 +2,9 @@ # Cell-group implementation handoff -The active goal is end-to-end implementation and hard cutover from the executable -design. Continue through the full task and release gates. +The requested delivery is an executable design and implementation plan. This +handoff prepares subsequent end-to-end implementation and hard cutover; that +work must continue through the full task and release gates. This handoff does not claim completion of the shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many @@ -42,6 +43,16 @@ and joins submitted physical work on close. Native cases use independent SQLite images and actual node workers; see [progress](progress.md) for qualification. The prior uncommitted runtime draft has been incorporated into this source. +An unfinished local SQL draft follows the committed checkpoint. It adds +`cell_groups/execution_authority.rs`, `execution/cell.rs`, shared receipt +transaction helpers in `execution/command.rs`, and `tests/support/cell_group_sql.rs`, +with changes to physical image capture, owner/runtime authority and native tests. +It separates a selected cell image/version from group progress and attempts exact +receipt settlement in both image modes. It is not in the committed checkpoint +and has no completed source-bound qualification record. Review `git diff` and +untracked files before continuing; a fresh checkout of the PR will not contain +this draft. Do not infer a passing CG-04 gate from its presence. + The next integration is per-cell SQL ownership and fair admission, concurrent initialization, exact receipt settlement and coherent read serving. Preserve the node worker's physical completion and cancellation boundaries. Do not treat the @@ -67,6 +78,14 @@ integration remain incomplete. Preserve its persistence, completion and physical lease boundaries during integration; do not mistake the low-level owner cases for the full CG-04 gate. +Before extending the draft, qualify its changes to the existing execution paths +with `native_raft_execution` and `native_raft_rooted` as well as the shared-group +execution suite. In particular, rerun both cell receipt cases after the latest +callback/owner-close test change. Check authority revocation, exact committed +entry matching, duplicate callbacks, cleanup and compaction behavior. Those +selected regressions supplement, and do not replace, the full task gate printed +by `show CG-04`. Fair per-cell queues and registered SDK dispatch remain absent. + Run from the repository root with Python 3.11+ and Rust 1.98.1: ```sh diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index bfbf7f998..cb2a44248 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -2,12 +2,36 @@ # Cell-group implementation progress -The end-to-end implementation and hard-cutover goal remains active; see the -[executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; +The current delivery is the executable design and implementation plan; see the +[executor handoff](handoff.md) for subsequent implementation and hard cutover. +CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## Plan delivery and unfinished SQL draft + +The implementation package contains 16 dependency-ordered tasks, 86 required +cases, 14 invariants and nine qualification profiles. Each task specifies owning +paths, implementation steps, acceptance oracles and executable test commands. +The plan runner prints those commands and checks evidence; it does not execute +the implementation or provision infrastructure. + +The current working tree also contains an unfinished per-cell SQL producer, +leader-issued execution authority, shared receipt helpers and tests for both +image formats. These are preserved for the next implementation session. They +are not included in the committed runtime checkpoint below, and the latest +callback revocation extension has not completed its regression gate. The +handoff records the affected boundaries and next checks. No task status or +runtime qualification is advanced by this plan delivery. + +Plan delivery checks pass: package validation, reviewed source preflight, all +37 validator unit tests and repository documentation validation (113 documents, +2,314 local references). `show CG-04` emits the full executor brief. The release +command correctly refuses incomplete implementation evidence. The branch +contains the latest fetched `origin/main` at `8a93ffd7`; neither requested +MR-13/MR-14 progress JSON exclusion differs from main. + ## CG-04 attachment admission and main integration Source: `aec4029e77124848dc6b155c098f8eec9ccdacc4`. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 5e8ec88a2..3ed0f5c93 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -448,7 +448,11 @@ "crates/cellar/durability/raft/cell_groups/page_wire_tests.rs", "crates/cellar/durability/raft/cell_groups/runtime/mod.rs", "crates/cellar/durability/raft/cell_groups/runtime/worker.rs", - "crates/cellar/tests/support/cell_group_runtime.rs" + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/durability/raft/cell_groups/execution_authority.rs", + "crates/cellar/durability/raft/execution/cell.rs", + "crates/cellar/durability/raft/execution/command.rs", + "crates/cellar/tests/support/cell_group_sql.rs" ], "test_target": "native_cell_groups_execution", "steps": [ From f3d48bdc86186077cff96f66597a65c2c4bd5301 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:39:58 -0700 Subject: [PATCH 047/568] feat: execute cell-addressed SQL and settle exact group receipts --- crates/cellar/application/consensus/mod.rs | 3 +- crates/cellar/durability/raft/README.md | 12 + .../raft/cell_groups/execution_authority.rs | 111 ++++++ .../cellar/durability/raft/cell_groups/mod.rs | 3 + .../durability/raft/cell_groups/owner.rs | 50 ++- .../raft/cell_groups/runtime/mod.rs | 17 + .../raft/cell_groups/runtime/worker.rs | 5 + .../cellar/durability/raft/execution/cell.rs | 331 +++++++++++++++ .../durability/raft/execution/command.rs | 100 +++++ .../durability/raft/execution/images.rs | 160 ++++++-- .../cellar/durability/raft/execution/mod.rs | 114 ++---- .../raft/execution/rooted_working.rs | 29 +- crates/cellar/durability/raft/working.rs | 97 +++-- .../tests/native_cell_groups_execution.rs | 148 ++++++- .../tests/support/cell_group_runtime.rs | 8 + crates/cellar/tests/support/cell_group_sql.rs | 377 ++++++++++++++++++ 16 files changed, 1398 insertions(+), 167 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/execution_authority.rs create mode 100644 crates/cellar/durability/raft/execution/cell.rs create mode 100644 crates/cellar/durability/raft/execution/command.rs create mode 100644 crates/cellar/tests/support/cell_group_sql.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 06137294e..cdc46596d 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -157,7 +157,8 @@ pub mod verification { RuntimeEventKind, SnapshotCompletion, }; pub use crate::raft::execution::{ - query_registered, AppliedExecution, CommandAttempt, CommandAuthority, CommandRequest, + query_registered, AppliedExecution, CellCommittedReceipt, CellPreparedCommand, + CellPrivateExecutor, CellStageOutcome, CommandAttempt, CommandAuthority, CommandRequest, CommittedReceipt, ExecutionError, ExecutionLimits, ExecutionStatus, PreparedCommand, PrivateExecutor, PrivateOptions, RegisteredQueryOutput, RootPrivateOptions, StageOutcome, StoredReceipt, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 45fd2abf6..d39b21177 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -43,6 +43,18 @@ progress under full event admission, releasing the apply reservation before its Applied event is admitted. Shutdown stops admission, joins the submitted batch, then fences the engine before releasing exclusive runtime ownership. +`CellPrivateExecutor` now stages a cell-addressed mutation from an exact pinned +cell base, with materialized or extent-backed SQLite images. It reuses the existing +receipt admission, maintenance and atomic business/receipt transaction. The actual +leader issues `CellExecutionAuthority`; another cell may advance while this base +remains selected. Losing leadership or dropping the owner revokes the observation. +A callback must recheck it after capture, and settlement requires a fresh authority, +the exact committed entry and that cell's expected image/version. A receipt retry +reads the selected immutable cell image and does not invoke the callback again. +The synchronous executor must run on an owning blocking worker. It currently +reopens after settlement; it is not yet the production SQL pool or registered SDK +operation dispatcher. Receipt proof alone does not provide S3 coverage. + Per-cell SQL admission, SDK dispatch and the production hard cutover remain required; the existing host still uses its prior single-cell owner. Transport authentication and actual image preparation remain the caller's responsibility diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs new file mode 100644 index 000000000..858acd56b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -0,0 +1,111 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A private-execution observation, never an acknowledgement or query lease. + +use super::{CellImagePin, CellManifestLeaf, PreparedGroupManifest}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{cell_group::*, consensus::Term}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +/// Issued only by the actual leader after its current-term applied barrier. +/// It pins one cell, not every image in the group. Other cells may advance while +/// this cell prepares; its own selected base, route or leader term may not. +/// Proposal admission must still recheck authority after physical preparation. +pub struct CellExecutionAuthority { + group: GroupLog, + route: GroupRoute, + term: Term, + leader_term: Arc, + leaf: CellManifestLeaf, + image: CellImagePin, +} + +impl CellExecutionAuthority { + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: &CellHistory, + term: Term, + leader_term: Arc, + ) -> Result { + let (leaf, image) = current.pin_cell(&cell.cell).ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell execution history".into())); + } + let authority = Self { + group, + route: current.marker().route().clone(), + term, + leader_term, + leaf: leaf.clone(), + image, + }; + authority.check()?; + Ok(authority) + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.leaf + } + + pub fn image(&self) -> &CellImagePin { + &self.image + } + + pub(crate) fn check_committed_entry(&self, bytes: &[u8]) -> Result<(), RuntimeError> { + self.check()?; + let cache = self.group.read()?; + let position = self.leaf.last_mutation; + let entry = cache + .entries + .get(&position.index.get()) + .ok_or(RuntimeError::NotReady)?; + if entry.term != position.term.get() + || entry.data.as_ref() != bytes + || cache.hard_state.commit < position.index.get() + { + return Err(RuntimeError::StaleCompletion); + } + Ok(()) + } + + pub fn check(&self) -> Result<(), RuntimeError> { + if self.leader_term.load(Ordering::Acquire) != self.term.get() { + return Err(RuntimeError::NotReady); + } + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + self.route.history.namespace, + &self.leaf.version.history.cell, + ); + if cache.hard_state.term != self.term.get() + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .is_none_or(|(leaf, _)| leaf != &self.leaf) + || self.leader_term.load(Ordering::Acquire) != self.term.get() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index ca57a3aa4..e419cdaaa 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -2,6 +2,9 @@ //! Complete group-state storage. A synced page is data, not Raft authority. +mod execution_authority; +pub use execution_authority::CellExecutionAuthority; + mod enrollment; mod enrollment_apply; mod image_pin; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index ad71b5a4b..435e3608a 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -22,7 +22,7 @@ use protobuf::Message as _; use std::{ collections::VecDeque, sync::{ - atomic::{AtomicBool, Ordering}, + atomic::{AtomicBool, AtomicU64, Ordering}, Arc, }, }; @@ -146,6 +146,7 @@ enum Phase { } pub struct CellGroupOwner { + leader_term: Arc, lease: Arc, log: ReplicaLog, group: GroupLog, @@ -224,6 +225,7 @@ impl CellGroupOwner { Some(seed), )?; Ok(Self { + leader_term: Arc::new(AtomicU64::new(0)), lease, log: permit.log().clone(), group, @@ -242,6 +244,36 @@ impl CellGroupOwner { }) } + pub(super) fn revoke_execution(&self) { + self.leader_term.store(0, Ordering::Release); + } + + fn refresh_execution_fence(&self) { + let term = if self.node.raft.state == StateRole::Leader { + self.node.raft.term + } else { + 0 + }; + self.leader_term.store(term, Ordering::Release); + } + + pub fn execution_authority( + &self, + cell: &CellHistory, + ) -> Result { + let status = self.status()?; + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + super::CellExecutionAuthority::new( + self.group.clone(), + &self.current, + cell, + status.term, + self.leader_term.clone(), + ) + } + pub fn manifest(&self) -> &PreparedGroupManifest { &self.current } @@ -268,7 +300,9 @@ impl CellGroupOwner { if self.phase.is_some() { return Err(RuntimeError::NotReady); } - self.node.campaign()?; + let result = self.node.campaign(); + self.refresh_execution_fence(); + result?; Ok(()) } @@ -281,6 +315,7 @@ impl CellGroupOwner { return Err(RuntimeError::NotReady); } self.node.tick(); + self.refresh_execution_fence(); Ok(()) } @@ -293,7 +328,9 @@ impl CellGroupOwner { return Err(RuntimeError::NotReady); } self.validate_message(&message)?; - self.node.step(message)?; + let result = self.node.step(message); + self.refresh_execution_fence(); + result?; Ok(()) } @@ -407,6 +444,7 @@ impl CellGroupOwner { let result = self.poll_inner(collect, emit); if let Err(error) = &result { self.failed = Some(error.clone()); + self.revoke_execution(); } result } @@ -880,3 +918,9 @@ impl CellGroupOwner { Ok(()) } } + +impl Drop for CellGroupOwner { + fn drop(&mut self) { + self.leader_term.store(0, Ordering::Release); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 88325e667..8521274be 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -84,6 +84,7 @@ struct Handle { enum Command { Attach(Box), Campaign, + Execution(cellar_logic::cell_group::CellHistory), Step(Message), AdmitStep(Message), Propose(Vec), @@ -96,6 +97,7 @@ enum Command { enum Response { Unit, Proposal(CellGroupProposal), + Execution(Box), Status(CellGroupStatus), } @@ -301,6 +303,21 @@ impl CellGroupRuntime { .map(|_| ()) } + pub async fn execution_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Execution(cell), bytes) + .await? + { + Response::Execution(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol("execution authority reply".into())), + } + } + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { self.call(Some(history), Command::Campaign, 4096) .await diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index a523a13c7..c66b13904 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -48,6 +48,7 @@ impl Group { } fn fail(&mut self, error: RuntimeError) { + self.owner.revoke_execution(); self.failed = Some(error.clone()); self.pending = Some(Err(error.clone())); self.settled = None; @@ -320,6 +321,10 @@ impl Worker { Err(RuntimeError::Overloaded) } else { match command { + Command::Execution(cell) => group + .owner + .execution_authority(&cell) + .map(|authority| Response::Execution(Box::new(authority))), Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs new file mode 100644 index 000000000..545293e27 --- /dev/null +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -0,0 +1,331 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One cell's private SQL/receipt producer under a physical group's authority. +//! The node's SQL pool must own this synchronous object and join its callbacks. + +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, GroupLtx, GroupMutation, +}; +use crate::raft::cell_groups::{cell_image_history_digest, CellExecutionAuthority}; +use crate::raft::completion::RuntimeError; +use cellar_logic::cell_group::CellVersion; +use std::sync::Arc; + +#[derive(Clone)] +pub struct CellPreparedCommand { + bytes: Arc>, + proposal: ProposalId, +} + +impl CellPreparedCommand { + pub fn bytes(&self) -> &[u8] { + &self.bytes + } + + pub fn proposal(&self) -> ProposalId { + self.proposal + } +} + +/// A receipt read from the exact committed cell image. This is not S3 coverage. +pub struct CellCommittedReceipt { + pub receipt: StoredReceipt, + pub version: CellVersion, + pub position: LogPosition, +} + +pub enum CellStageOutcome { + Prepared(CellPreparedCommand), + Duplicate(Box), +} + +struct PendingCell { + prepared: CellPreparedCommand, + request: CommandRequest, + result: CellVersion, + _preview: Preview, + submitted: bool, +} + +pub struct CellPrivateExecutor { + options: Options, + authority: CellExecutionAuthority, + working: Option, + pending: Option, + poisoned: bool, + receipt_cursor: Option, + _lock: PrivateLock, +} + +impl CellPrivateExecutor { + pub fn open( + options: PrivateOptions, + authority: CellExecutionAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + pub fn open_rooted( + options: RootPrivateOptions, + authority: CellExecutionAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + fn open_image( + options: Options, + authority: CellExecutionAuthority, + ) -> Result { + options.limits.validate()?; + options.images.validate_workspace(options.limits)?; + authority.check().map_err(authority_failure)?; + if authority.route().schema_digest != options.application_digest { + return Err(ExecutionError::Invalid("cell execution schema")); + } + let filesystem = options.host.filesystem(); + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &options.directory) + .map_err(storage)?; + let lock = PrivateLock { + _lock: filesystem + .lock_file(&options.directory.join("working.lock")) + .map_err(storage)?, + _workspace: options.images.workspace().cloned(), + }; + let working = WorkingImage::open_cell( + &options, + authority.image(), + cell_image_history_digest(&authority.cell().version.history), + ) + .map_err(storage)?; + authority.check().map_err(authority_failure)?; + Ok(Self { + options, + authority, + working: Some(working), + pending: None, + poisoned: false, + receipt_cursor: None, + _lock: lock, + }) + } + + /// Stage on an owning blocking worker. Callback output is retained only in + /// the private receipt transaction; it cannot escape as a committed result. + /// The host serializes unresolved producers by CellHistory, independently + /// from the group's consensus owner and other cells' SQL work. + pub fn stage( + &mut self, + attempt: CommandAttempt, + allow_new: bool, + callback: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, + ) -> Result { + if self.pending.is_some() { + return Err(ExecutionError::Busy); + } + if self.poisoned { + return Err(ExecutionError::RebuildRequired); + } + self.authority.check().map_err(authority_failure)?; + let working = self + .working + .as_ref() + .ok_or(ExecutionError::RebuildRequired)?; + let maintenance = match command::prepare( + working.database().framework(), + self.options.limits, + &mut self.receipt_cursor, + &attempt, + allow_new, + )? { + command::Admission::Duplicate => { + return self + .read_receipt(&self.authority, &attempt.request) + .map(|receipt| CellStageOutcome::Duplicate(Box::new(receipt))) + } + command::Admission::New(maintenance) => maintenance, + }; + let sequence = self + .authority + .cell() + .version + .sequence + .checked_next() + .map_err(|_| ExecutionError::Invalid("cell mutation sequence exhausted"))?; + self.poisoned = true; + let working = self + .working + .as_mut() + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + command::apply( + working.database(), + self.options.limits, + &attempt, + maintenance, + callback, + )?; + let update = working + .capture_cell(&self.options, self.authority.image()) + .map_err(private_failure)?; + if cellar_logic::cell_group::CellImage::from(update.base) + != self.authority.cell().version.image + { + return Err(ExecutionError::Invalid("captured cell base")); + } + let route = self.authority.route(); + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: attempt.proposal, + }, + command: GroupCommand::Mutation(GroupMutation { + cell: self.authority.cell().version.history.clone(), + sequence, + base: update.base, + result: update.result, + files: update + .files + .iter() + .map(|file| GroupLtx { + min_txid: file.min_txid, + max_txid: file.max_txid, + page_count: file.page_count, + decoded_bytes: file.decoded_bytes, + bytes: &file.bytes, + }) + .collect(), + }), + }; + let bytes = entry + .encode( + route, + GroupEntryLimits { + envelope: self.options.limits.envelope, + ..Default::default() + }, + ) + .map_err(ExecutionError::Protocol)?; + let preview = self + .options + .images + .preview_cell( + &self.options, + self.authority.image(), + attempt.preview, + &update, + cell_image_history_digest(&self.authority.cell().version.history), + ) + .map_err(storage)?; + self.authority.check().map_err(authority_failure)?; + let prepared = CellPreparedCommand { + bytes: Arc::new(bytes), + proposal: attempt.proposal, + }; + self.pending = Some(PendingCell { + prepared: prepared.clone(), + request: attempt.request, + result: CellVersion { + history: self.authority.cell().version.history.clone(), + sequence, + image: update.result.into(), + }, + _preview: preview, + submitted: false, + }); + self.poisoned = false; + Ok(CellStageOutcome::Prepared(prepared)) + } + + /// Set before awaiting proposal submission. A dropped response never frees + /// this producer for another callback or establishes that its effect failed. + pub fn mark_submitted(&mut self, proposal: ProposalId) -> Result<(), ExecutionError> { + self.authority.check().map_err(authority_failure)?; + let pending = self + .pending + .as_mut() + .ok_or(ExecutionError::Invalid("no cell proposal"))?; + if pending.submitted || pending.prepared.proposal != proposal { + return Err(ExecutionError::Invalid("cell proposal submission")); + } + pending.submitted = true; + Ok(()) + } + + /// Settle only the exact committed entry and cell result, using a fresh + /// current-term authority from the actual owner. Another cell's progress, + /// no-op or selected image cannot settle this receipt. Compacted entries + /// require later receipt reconciliation, not an inferred proposal proof. + pub fn finish( + &mut self, + committed: CellExecutionAuthority, + ) -> Result { + let pending = self + .pending + .as_ref() + .ok_or(ExecutionError::Invalid("no cell proposal"))?; + if !pending.submitted + || committed.route() != self.authority.route() + || committed.cell().version != pending.result + { + return Err(ExecutionError::Invalid("committed cell proposal/image")); + } + committed + .check_committed_entry(pending.prepared.bytes()) + .map_err(authority_failure)?; + let receipt = self.read_receipt(&committed, &pending.request)?; + self.pending = None; + self.poisoned = true; + Ok(receipt) + } + + fn read_receipt( + &self, + authority: &CellExecutionAuthority, + request: &CommandRequest, + ) -> Result { + authority.check().map_err(authority_failure)?; + let database = self + .options + .images + .read_cell(&self.options, authority.image()) + .map_err(storage)?; + let receipt = receipts::preflight(database.connection(), request, &self.options.limits)? + .ok_or(ExecutionError::Invalid("committed cell receipt missing"))?; + authority.check().map_err(authority_failure)?; + Ok(CellCommittedReceipt { + receipt, + version: authority.cell().version.clone(), + position: authority.cell().last_mutation, + }) + } + + /// Join and close the disposable writer. It never changes the outcome of + /// an already submitted entry; recovery reads the committed cell image. + pub fn close(mut self) -> Result<(), ExecutionError> { + let store = self.options.images.root_store().cloned(); + if let Some(working) = self.working.take() { + working.close().map_err(storage)?; + } + drop(self.options); + drop(self._lock); + if let Some(store) = store { + if store.usage().map_err(storage)?.quarantined { + return Err(storage(cellar_ltx::internal::StoreError::Quarantined)); + } + } + Ok(()) + } +} + +fn authority_failure(error: RuntimeError) -> ExecutionError { + match error { + RuntimeError::NotReady | RuntimeError::NotLeader | RuntimeError::Closed => { + ExecutionError::Refused(CommandFailure::Unavailable) + } + other => storage(other), + } +} diff --git a/crates/cellar/durability/raft/execution/command.rs b/crates/cellar/durability/raft/execution/command.rs new file mode 100644 index 000000000..fcadeb28e --- /dev/null +++ b/crates/cellar/durability/raft/execution/command.rs @@ -0,0 +1,100 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt admission and one atomic business/receipt transaction for both owners. + +use super::*; + +pub(super) enum Admission { + Duplicate, + New(Option), +} + +pub(super) fn prepare( + connection: &rusqlite::Connection, + limits: ExecutionLimits, + cursor: &mut Option, + attempt: &CommandAttempt, + allow_new: bool, +) -> Result { + if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { + return Err(ExecutionError::Invalid("command operation")); + } + if receipts::preflight_identity(connection, &attempt.request, &limits)?.is_some() { + return Ok(Admission::Duplicate); + } + let count = receipts::retained_count(connection)?; + let ceiling = limits.max_retained_receipts; + let maintenance = match limits.receipt_maintenance { + Some(limits) if count >= ceiling - ceiling / 4 => { + let plan = crate::raft::receipt_maintenance::plan( + connection, + attempt.inputs.now_unix_millis, + limits, + *cursor, + )?; + *cursor = plan.next_cursor; + Some(plan) + } + _ => None, + }; + let removed = maintenance.as_ref().map_or(0, |plan| plan.rows_removed); + if count + .checked_sub(removed) + .is_none_or(|count| count >= ceiling) + { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + if attempt.request.retry_until_ms < attempt.inputs.now_unix_millis + || attempt + .request + .retry_until_ms + .checked_sub(attempt.inputs.now_unix_millis) + .is_none_or(|window| window > limits.max_retry_window_ms) + { + return Err(ExecutionError::Invalid("command retry window")); + } + // Duplicate receipt lookup above remains available under archive + // pressure. A known refusal has not mutated SQL or capture state and + // must not force the next request to reconstruct the private image. + if !allow_new { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + Ok(Admission::New(maintenance)) +} + +pub(super) fn apply( + database: &crate::storage::CommandDatabase, + limits: ExecutionLimits, + attempt: &CommandAttempt, + maintenance: Option, + command: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, +) -> Result<(), ExecutionError> { + database + .transaction(attempt.inputs, |connection, transaction| { + if let Some(maintenance) = maintenance { + maintenance + .apply(connection, attempt.inputs.now_unix_millis) + .map_err(anyhow::Error::new)?; + } + let result = command(transaction)?; + if result.len() > limits.max_result_bytes { + return Err(anyhow::Error::new(CommandFailure::Overloaded)); + } + receipts::record(connection, &attempt.request, result)?; + let limits = &limits; + if limits.receipt_reserve_bytes != 0 { + let usage = + crate::storage::CommandDatabase::capacity(connection, limits.max_image_bytes)?; + limits + .check_capacity( + usage.application_bytes, + usage.receipt_bytes, + usage.structural_reserve_bytes, + ) + .map_err(anyhow::Error::new)?; + } + Ok(()) + }) + .map_err(private_failure)?; + Ok(()) +} diff --git a/crates/cellar/durability/raft/execution/images.rs b/crates/cellar/durability/raft/execution/images.rs index 2140af734..8a311e44e 100644 --- a/crates/cellar/durability/raft/execution/images.rs +++ b/crates/cellar/durability/raft/execution/images.rs @@ -256,6 +256,48 @@ impl WorkingImage { } } + pub(super) fn open_cell( + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + history: [u8; 32], + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; + match (&options.images, base) { + (Images::Materialized(_), CellImagePin::Materialized(pin)) => { + Ok(Self::Materialized(Box::new( + crate::raft::working::Working::open_pin(&options.materialized()?, Some(pin))?, + ))) + } + (Images::Materialized(_), CellImagePin::Empty) => Ok(Self::Materialized(Box::new( + crate::raft::working::Working::open_pin(&options.materialized()?, None)?, + ))), + (Images::Extents { .. }, CellImagePin::Extents(pin)) => Ok(Self::Rooted(Box::new( + super::rooted_working::RootWorking::open_pin(options, Some(pin.clone()), history)?, + ))), + _ => anyhow::bail!("cell image storage differs"), + } + } + + pub(super) fn capture_cell( + &mut self, + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; + match (self, base) { + (Self::Materialized(working), CellImagePin::Materialized(pin)) => { + working.capture_pin(&options.materialized()?, Some(pin)) + } + (Self::Materialized(working), CellImagePin::Empty) => { + working.capture_pin(&options.materialized()?, None) + } + (Self::Rooted(working), CellImagePin::Extents(pin)) => { + working.capture_pin(options, Some(pin)) + } + _ => anyhow::bail!("cell image storage differs"), + } + } + pub fn database(&self) -> &crate::storage::CommandDatabase { match self { Self::Materialized(working) => &working.database, @@ -404,37 +446,91 @@ impl Images { update: &CellUpdate, history: [u8; 32], ) -> anyhow::Result { + let pin = base.map(|base| match base { + Published::Materialized(image) => { + crate::raft::cell_groups::CellImagePin::Materialized(image.pin()) + } + Published::Rooted(image) => { + crate::raft::cell_groups::CellImagePin::Extents(image.image()) + } + }); + self.preview_payload( + options, + pin.as_ref(), + id, + crate::raft::working::CapturedUpdateRef { + base: update.base, + result: update.result, + files: &update.files, + }, + history, + ) + } + + pub(super) fn preview_cell( + &self, + options: &Options, + base: &crate::raft::cell_groups::CellImagePin, + id: ImageId, + update: &crate::raft::working::CapturedUpdate, + history: [u8; 32], + ) -> anyhow::Result { + self.preview_payload(options, Some(base), id, update.as_ref(), history) + } + + fn preview_payload( + &self, + options: &Options, + base: Option<&crate::raft::cell_groups::CellImagePin>, + id: ImageId, + update: crate::raft::working::CapturedUpdateRef<'_>, + history: [u8; 32], + ) -> anyhow::Result { + let crate::raft::working::CapturedUpdateRef { + base: before, + result: after, + files, + } = update; match self { - Self::Materialized(_) => { - Ok(Preview::Materialized(crate::raft::working::verify_preview( + Self::Materialized(_) => Ok(Preview::Materialized( + crate::raft::working::verify_preview_pin( &options.materialized()?, - base.map(Published::flat).transpose()?, + match base { + None | Some(crate::raft::cell_groups::CellImagePin::Empty) => None, + Some(crate::raft::cell_groups::CellImagePin::Materialized(pin)) => { + Some(pin) + } + _ => anyhow::bail!("materialized base required"), + }, id, - update, - )?)) - } + before, + after, + files, + )?, + )), Self::Extents { store, .. } => { - let result = match update.result.commitment { + let result = match after.commitment { ImageCommitment::Extents(reference) => reference, _ => anyhow::bail!("extent result required"), }; - let before = match update.base.commitment { + let before = match before.commitment { ImageCommitment::Extents(reference) => reference, _ => anyhow::bail!("extent base required"), }; let header = cellar_ltx::ltx::Header::parse( - &update - .files + &files .last() .ok_or_else(|| anyhow::anyhow!("empty update"))? .bytes, )?; - let base = base - .map(Published::rooted) - .transpose()? - .map(RootPublishedImage::image); - let segments = update - .files + let base = match base { + Some(crate::raft::cell_groups::CellImagePin::Extents(pin)) => { + Some(pin.as_ref()) + } + None => None, + _ => anyhow::bail!("extent base required"), + }; + let segments = files .iter() .map(|file| { Ok(cellar_ltx::apply::LtxSegment { @@ -447,16 +543,16 @@ impl Images { }) .collect::>>()?; let prepared = store.prepare_image_root( - base.as_deref(), + base, &ImageRootUpdate { base: before, result, binding: ImageBinding { history, schema: options.application_digest, - epoch: update.result.epoch, - txid: update.result.txid, - checksum: update.result.checksum, + epoch: after.epoch, + txid: after.txid, + checksum: after.checksum, }, geometry: Geometry::new( header.page_size, @@ -477,10 +573,27 @@ impl Images { options: &Options, image: &Published, ) -> anyhow::Result { + let image = match image { + Published::Materialized(image) => { + crate::raft::cell_groups::CellImagePin::Materialized(image.pin()) + } + Published::Rooted(image) => { + crate::raft::cell_groups::CellImagePin::Extents(image.image()) + } + }; + self.read_cell(options, &image) + } + + pub(super) fn read_cell( + &self, + options: &Options, + image: &crate::raft::cell_groups::CellImagePin, + ) -> anyhow::Result { + use crate::raft::cell_groups::CellImagePin; match (self, image) { - (Self::Materialized(_), Published::Materialized(image)) => { + (Self::Materialized(_), CellImagePin::Materialized(image)) => { Ok(cellar_ltx::generation::open_immutable( - image.pin(), + image.clone(), options.query_services.clone(), )?) } @@ -491,10 +604,9 @@ impl Images { cache, .. }, - Published::Rooted(image), + CellImagePin::Extents(image), ) => Ok(cellar_ltx::internal::open_extent_image( image - .image() .view() .image(cache_directory.clone(), cache_pool.clone(), *cache)?, options.query_services.clone(), diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index 5168198ad..ac33ffd82 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -7,7 +7,10 @@ use super::apply::{marker, PublishedImage}; use super::cache::{DurableApply, GroupLog}; use super::completion::GroupStatus; use super::receipts; +mod cell; +mod command; mod images; +pub use cell::{CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellStageOutcome}; mod rooted_working; use crate::application::consensus::{ command_request_digest, CommittedVersion, EnvelopeLimits, OperationContract, OperationKind, @@ -821,102 +824,41 @@ impl PrivateExecutor { command: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result>, ) -> Result { self.ready()?; - if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { - return Err(ExecutionError::Invalid("command operation")); - } let working = self .working .as_ref() .ok_or(ExecutionError::RebuildRequired)?; - if receipts::preflight_identity( + let maintenance = match command::prepare( working.database().framework(), - &attempt.request, - &self.options.limits, - )? - .is_some() - { - let receipt = self.read_committed( - &attempt.request, - self.base - .as_ref() - .ok_or(ExecutionError::Invalid("receipt without base"))?, - )?; - return Ok(StageOutcome::Duplicate(receipt)); - } - let count = receipts::retained_count(working.database().framework())?; - let ceiling = self.options.limits.max_retained_receipts; - let maintenance = match self.options.limits.receipt_maintenance { - Some(limits) if count >= ceiling - ceiling / 4 => { - let plan = super::receipt_maintenance::plan( - working.database().framework(), - attempt.inputs.now_unix_millis, - limits, - self.receipt_cursor, + self.options.limits, + &mut self.receipt_cursor, + &attempt, + allow_new, + )? { + command::Admission::Duplicate => { + let receipt = self.read_committed( + &attempt.request, + self.base + .as_ref() + .ok_or(ExecutionError::Invalid("receipt without base"))?, )?; - self.receipt_cursor = plan.next_cursor; - Some(plan) + return Ok(StageOutcome::Duplicate(receipt)); } - _ => None, + command::Admission::New(maintenance) => maintenance, }; - let removed = maintenance.as_ref().map_or(0, |plan| plan.rows_removed); - if count - .checked_sub(removed) - .is_none_or(|count| count >= ceiling) - { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - if attempt.request.retry_until_ms < attempt.inputs.now_unix_millis - || attempt - .request - .retry_until_ms - .checked_sub(attempt.inputs.now_unix_millis) - .is_none_or(|window| window > self.options.limits.max_retry_window_ms) - { - return Err(ExecutionError::Invalid("command retry window")); - } - // Duplicate receipt lookup above remains available under archive - // pressure. A known refusal has not mutated SQL or capture state and - // must not force the next request to reconstruct the private image. - if !allow_new { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } self.rebuild_required = true; - self.working + let working = self + .working .as_mut() - .ok_or(ExecutionError::RebuildRequired)? - .begin_mutation(); - self.working - .as_ref() - .ok_or(ExecutionError::RebuildRequired)? - .database() - .transaction(attempt.inputs, |connection, transaction| { - if let Some(maintenance) = maintenance { - maintenance - .apply(connection, attempt.inputs.now_unix_millis) - .map_err(anyhow::Error::new)?; - } - let result = command(transaction)?; - if result.len() > self.options.limits.max_result_bytes { - return Err(anyhow::Error::new(CommandFailure::Overloaded)); - } - receipts::record(connection, &attempt.request, result)?; - let limits = &self.options.limits; - if limits.receipt_reserve_bytes != 0 { - let usage = crate::storage::CommandDatabase::capacity( - connection, - limits.max_image_bytes, - )?; - limits - .check_capacity( - usage.application_bytes, - usage.receipt_bytes, - usage.structural_reserve_bytes, - ) - .map_err(anyhow::Error::new)?; - } - Ok(()) - }) - .map_err(private_failure)?; + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + command::apply( + working.database(), + self.options.limits, + &attempt, + maintenance, + command, + )?; let prepared = self.freeze( attempt.proposal, attempt.preview, diff --git a/crates/cellar/durability/raft/execution/rooted_working.rs b/crates/cellar/durability/raft/execution/rooted_working.rs index 2af25995b..e4626b09e 100644 --- a/crates/cellar/durability/raft/execution/rooted_working.rs +++ b/crates/cellar/durability/raft/execution/rooted_working.rs @@ -37,6 +37,14 @@ impl RootWorking { options: &Options, base: Option<&crate::raft::rooted::RootPublishedImage>, history: [u8; 32], + ) -> Result { + Self::open_pin(options, base.map(|base| base.image()), history) + } + + pub fn open_pin( + options: &Options, + base: Option>, + history: [u8; 32], ) -> Result { let Images::Extents { store, @@ -56,7 +64,7 @@ impl RootWorking { let path = options.directory.join("working.sqlite"); crate::raft::working::clean(filesystem.as_ref(), &options.directory, &path)?; let image = match base { - Some(image) => image.image(), + Some(image) => image, None => Arc::new(store.seal_image_root( ExtentManifest::new( ImageBinding { @@ -170,9 +178,17 @@ impl RootWorking { proposal: ProposalId, base: Option<&crate::raft::rooted::RootPublishedImage>, ) -> Result { - let before = base - .map(|base| RootState::of(&base.image())) - .unwrap_or(self.initial); + Ok(self + .capture_pin(options, base.map(|base| base.image()).as_deref())? + .bind(scope.clone(), options.application_digest, proposal)) + } + + pub fn capture_pin( + &mut self, + options: &Options, + base: Option<&cellar_ltx::internal::SealedImageRoot>, + ) -> Result { + let before = base.map(RootState::of).unwrap_or(self.initial); let (position, files) = crate::raft::working::capture_update( &mut self.capture, &self.database, @@ -213,10 +229,7 @@ impl RootWorking { geometry: observed.manifest().geometry(), }); self.observation = Some(observed); - Ok(CellUpdate { - scope: scope.clone(), - application_digest: options.application_digest, - proposal, + Ok(crate::raft::working::CapturedUpdate { base: before.image, result, files, diff --git a/crates/cellar/durability/raft/working.rs b/crates/cellar/durability/raft/working.rs index d90aadad1..959cd46c2 100644 --- a/crates/cellar/durability/raft/working.rs +++ b/crates/cellar/durability/raft/working.rs @@ -5,7 +5,7 @@ use super::apply::{image_position, PublishedImage}; use super::execution::PrivateOptions; use super::receipts; -use crate::application::consensus::{CellUpdate, LtxUpdate}; +use crate::application::consensus::{CellUpdate, LtxImage, LtxUpdate}; use crate::storage::CommandDatabase; use anyhow::{bail, ensure, Context as _, Result}; use cellar_logic::consensus::{GroupScope, ProposalId}; @@ -15,6 +15,46 @@ use cellar_ltx::{ltx, Db, FileSystem, TXID}; use std::collections::BTreeMap; use std::path::{Path, PathBuf}; +/// A verified physical cut, independent of routing, consensus and cell identity. +/// The execution owner must bind it to the exact cell base before proposal. +pub(crate) struct CapturedUpdate { + pub base: LtxImage, + pub result: LtxImage, + pub files: Vec, +} + +pub(crate) struct CapturedUpdateRef<'a> { + pub base: LtxImage, + pub result: LtxImage, + pub files: &'a [LtxUpdate], +} + +impl CapturedUpdate { + pub fn as_ref(&self) -> CapturedUpdateRef<'_> { + CapturedUpdateRef { + base: self.base, + result: self.result, + files: &self.files, + } + } + + pub fn bind( + self, + scope: GroupScope, + application_digest: [u8; 32], + proposal: ProposalId, + ) -> CellUpdate { + CellUpdate { + scope, + application_digest, + proposal, + base: self.base, + result: self.result, + files: self.files, + } + } +} + pub(super) struct Working { pub database: CommandDatabase, capture: Db, @@ -23,17 +63,20 @@ pub(super) struct Working { impl Working { pub fn open(options: &PrivateOptions, base: Option<&PublishedImage>) -> Result { + Self::open_pin(options, base.map(PublishedImage::pin).as_ref()) + } + + pub fn open_pin(options: &PrivateOptions, base: Option<&GenerationPin>) -> Result { let filesystem = options.host.filesystem(); let path = options.directory.join("working.sqlite"); clean(filesystem.as_ref(), &options.directory, &path)?; if let Some(base) = base { - let pin = base.pin(); - let state = pin.state(); + let state = base.state(); ensure!( state.length <= options.limits.max_image_bytes, "command base exceeds image bound" ); - pin.private_copy(&filesystem, &path)?; + base.private_copy(&filesystem, &path)?; // Only the independently sealed base is a recovery authority. // This copy is discarded on every reopen, including after a crash. } @@ -49,7 +92,7 @@ impl Working { )?; receipts::install(database.framework())?; if let Some(base) = base { - let state = base.pin().state(); + let state = base.state(); // The zero-page baseline is local capture bookkeeping. Starting at // the WAL header includes every opening/control/schema write. capture @@ -86,8 +129,18 @@ impl Working { proposal: ProposalId, base: Option<&PublishedImage>, ) -> Result { + Ok(self + .capture_pin(options, base.map(PublishedImage::pin).as_ref())? + .bind(scope.clone(), options.application_digest, proposal)) + } + + pub fn capture_pin( + &mut self, + options: &PrivateOptions, + base: Option<&GenerationPin>, + ) -> Result { let base_state = base - .map(|b| b.pin().state()) + .map(GenerationPin::state) .unwrap_or(ImageState::empty(1)?); let (result_position, files) = capture_update( &mut self.capture, @@ -110,10 +163,7 @@ impl Working { length, digest, }; - let update = CellUpdate { - scope: scope.clone(), - application_digest: options.application_digest, - proposal, + let update = CapturedUpdate { base: image_position(base_state), result: image_position(result), files, @@ -248,28 +298,27 @@ pub(super) fn capture_update( Ok((result_position, files.into_values().collect())) } -pub(super) fn verify_preview( +pub(super) fn verify_preview_pin( options: &PrivateOptions, - base: Option<&PublishedImage>, + base_pin: Option<&GenerationPin>, id: ImageId, - update: &CellUpdate, + before: LtxImage, + after: LtxImage, + files: &[LtxUpdate], ) -> Result { - let header = ltx::Header::parse(&update.files.last().context("empty command cut")?.bytes)?; + let header = ltx::Header::parse(&files.last().context("empty command cut")?.bytes)?; let result = ImageState { - epoch: update.result.epoch, - txid: TXID(update.result.txid), - checksum: update.result.checksum, - digest: update.result.commitment.sha256()?, + epoch: after.epoch, + txid: TXID(after.txid), + checksum: after.checksum, + digest: after.commitment.sha256()?, page_size: header.page_size, length: header.commit as u64 * header.page_size as u64, }; - let base_pin = base.map(PublishedImage::pin); let base_state = base_pin - .as_ref() .map(GenerationPin::state) - .unwrap_or(ImageState::empty(update.base.epoch)?); - let segments = update - .files + .unwrap_or(ImageState::empty(before.epoch)?); + let segments = files .iter() .map(|file| { Ok(LtxSegment { @@ -283,7 +332,7 @@ pub(super) fn verify_preview( .collect::>>()?; Ok(options.images.prepare( id, - base_pin.as_ref(), + base_pin, &ImageUpdate { base: base_state, result, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 1eab0b2dc..4874b87fd 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -36,6 +36,16 @@ async fn node_runtime_attachment_reserves_before_owner() -> Result<()> { runtime_fixture::attachment_reserves_before_owner().await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_receipt_not_group_progress() -> Result<()> { + runtime_fixture::cell_receipts(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_receipt_not_group_progress() -> Result<()> { + runtime_fixture::cell_receipts(true).await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; @@ -61,6 +71,7 @@ struct Voter { owner: Option, pages: ManifestPageStore, images: GenerationStore, + extents: Option, tree: ManifestTree, gate: Arc, paused: bool, @@ -181,6 +192,7 @@ impl Cluster { owner: Some(owner), pages, images, + extents: None, tree, gate, paused: false, @@ -366,6 +378,7 @@ impl Cluster { tree, pages, images, + extents: None, gate, paused: false, held: None, @@ -431,6 +444,105 @@ fn apply_image( Ok(store.prepare(image_id(history, sequence)?, base, &update)?) } +fn prepare_cell_image( + images: &GenerationStore, + extents: Option<&cellar_ltx::internal::NodeExtentStore>, + cell: &CellHistory, + sequence: u64, + base: Option<&CellImagePin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + use cellar_ltx::internal::{ + ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, + }; + let (pin, geometry) = match image.commitment { + ImageCommitment::Sha256(_) => { + let base = match base { + Some(CellImagePin::Materialized(pin)) => Some(pin), + None => None, + _ => anyhow::bail!("materialized base"), + }; + let pin = apply_image(images, cell, sequence, base, image, files)?; + let geometry = CellImageGeometry { + length: pin.state().length, + page_size: pin.state().page_size, + }; + (CellImagePin::Materialized(pin), geometry) + } + ImageCommitment::Extents(result) => { + let store = extents.context("extent store")?; + let base = match base { + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + None => None, + _ => anyhow::bail!("extent base"), + }; + let binding = ImageBinding { + history: cell_image_history_digest(cell), + schema: [6; 32], + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0)?, + &[], + )?; + let before = base + .map(|pin| pin.reference()) + .unwrap_or(ExtentRootReference::new( + empty.root(), + empty.encode()?.len().try_into()?, + )?); + let header = + cellar_ltx::ltx::Header::parse(&files.last().context("extent LTX")?.bytes)?; + let geometry = Geometry::new( + header.page_size, + u64::from(header.page_size) * u64::from(header.commit), + )?; + let update = ImageRootUpdate { + base: before, + result, + binding, + geometry, + segments: files + .iter() + .map(|file| LtxSegment { + bytes: file.bytes, + min_txid: TXID(file.min_txid), + max_txid: TXID(file.max_txid), + page_count: file.page_count as usize, + decoded_bytes: file.decoded_bytes, + }) + .collect(), + }; + let prepared = store.prepare_image_root( + base, + &update, + &images.apply_limits(), + &mut vec![0; EXTENT_BYTES as usize], + )?; + ( + CellImagePin::Extents(Arc::new(prepared.image)), + CellImageGeometry { + length: geometry.length(), + page_size: geometry.page_size(), + }, + ) + } + }; + Ok(EnrollmentImage { + pin, + geometry, + effects: EffectSummary::default(), + }) +} + fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { if apply.entry.data.is_empty() { return Ok(voter.tree.prepare_unchanged( @@ -456,28 +568,25 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { - let (old, CellImagePin::Materialized(base)) = - apply.base.pin_cell(&mutation.cell.cell).context("cell")? - else { - anyhow::bail!("materialized base") - }; - let pin = apply_image( + let (old, base) = apply.base.pin_cell(&mutation.cell.cell).context("cell")?; + let prepared = prepare_cell_image( &voter.images, + voter.extents.as_ref(), &mutation.cell, mutation.sequence.get(), Some(&base), @@ -491,10 +600,7 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result Result anyhow::bail!("unexpected administration"), diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 9b13ea099..6222ea5ed 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -303,6 +303,7 @@ impl RuntimeCluster { owner: None, pages, images, + extents: None, tree, gate: self.cluster.voters[n].gate.clone(), paused: false, @@ -500,3 +501,10 @@ pub(super) async fn attachment_reserves_before_owner() -> Result<()> { drop(runtime); cluster.close().await } + +#[path = "cell_group_sql.rs"] +mod sql; + +pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { + sql::cell_receipts(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs new file mode 100644 index 000000000..ceac610ea --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -0,0 +1,377 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The actual cell SQL producer on domain workers with three persisted voters. + +use super::*; +use cellar_ltx::internal::{ + CacheLimits, ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRoot, + NodeExtentStore, EXTENT_BYTES, +}; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; + +fn cell_attempt(tag: u8) -> Result { + Ok(CommandAttempt { + request: CommandRequest { + id: CommandId::new(ClientId::new([81; 16])?, 1)?, + operation: "cell.increment".into(), + contract_digest: [6; 32], + request_digest: [9; 32], + durability: Durability::Quorum, + retry_until_ms: 1000, + }, + proposal: ProposalId::new([tag; 16])?, + preview: ImageId::new([tag; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }) +} + +impl RuntimeCluster { + async fn private_cell(&self, cell: &CellHistory) -> Result { + let authority = self.runtimes[0] + .execution_authority(self.cluster.birth.route().history, cell.clone()) + .await?; + let options = PrivateOptions { + directory: self + .cluster + ._root + .path() + .join(format!("private-{}", cell.cell.as_str())), + host: cellar_ltx::LtxHost::new( + || 10, + |_| Ok(Duration::ZERO), + |path| async move { std::fs::read(path) }, + |job| async move { + job(); + Ok(()) + }, + ) + .with_filesystem(Arc::new(DirectFileSystem)), + sqlite_vfs: None, + images: self.cluster.voters[0].images.clone(), + query_services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + application_digest: [6; 32], + limits: ExecutionLimits::default(), + }; + let extents = self.cluster.voters[0].extents.clone(); + Ok(self + .cluster + .domain + .blocking(move || -> Result<_> { + if let Some(store) = extents { + let rooted = RootPrivateOptions { + workspace: store.create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(options.limits)?, + )?, + host: options.host, + sqlite_vfs: None, + images: store, + query_services: options.query_services, + application_digest: options.application_digest, + limits: options.limits, + cache_directory: options.directory.with_extension("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }; + Ok(CellPrivateExecutor::open_rooted(rooted, authority)?) + } else { + Ok(CellPrivateExecutor::open(options, authority)?) + } + })? + .await??) + } +} + +pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let a = history(&route, "sql-a")?; + let b = history(&route, "sql-b")?; + for cell in [&a, &b] { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (0);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let bytes = if rooted { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(cell), + schema: route.schema_digest, + epoch: cut.result.epoch, + txid: cut.result.txid.0, + checksum: cut.result.checksum, + }, + Geometry::new(cut.result.page_size, cut.result.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + let encoded = request(&route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + anyhow::bail!("enrollment") + }; + image.commitment = ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len().try_into()?, + )?); + entry.encode(&route, GroupEntryLimits::default())? + } else { + request(&route, cell.clone(), 1, &cut)? + }; + let proposed = fixture.runtimes[0].propose(route.history, bytes).await?; + fixture.drive(Some(proposed.position.index)).await?; + } + assert!(matches!( + fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await, + Err(RuntimeError::NotReady) + )); + let mut wrong = a.clone(); + wrong.incarnation = IncarnationId::new([99; 16])?; + assert!(fixture.runtimes[0] + .execution_authority(route.history, wrong) + .await + .is_err()); + let mut ea = fixture.private_cell(&a).await?; + let mut eb = fixture.private_cell(&b).await?; + let (ea_out, pa) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = ea.stage(cell_attempt(82)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"a-result".to_vec()) + })? + else { + anyhow::bail!("new A receipt") + }; + ea.mark_submitted(prepared.proposal())?; + Ok((ea, prepared)) + })? + .await??; + ea = ea_out; + let (eb_out, pb) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = eb.stage(cell_attempt(83)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"b-result".to_vec()) + })? + else { + anyhow::bail!("new B receipt") + }; + eb.mark_submitted(prepared.proposal())?; + Ok((eb, prepared)) + })? + .await??; + eb = eb_out; + // Both use the same client/sequence, but B's committed image cannot settle A. + let proposed = fixture.runtimes[0] + .propose(route.history, pb.bytes().to_vec()) + .await?; + fixture.drive(Some(proposed.position.index)).await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let (ea_out, eb_out) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + assert!(ea.finish(cb).is_err()); + assert!(matches!( + ea.stage(cell_attempt(84)?, true, |_| panic!("unresolved A callback")), + Err(ExecutionError::Busy) + )); + Ok((ea, eb)) + })? + .await??; + ea = ea_out; + eb = eb_out; + // A prepared before B committed remains valid because its own base is unchanged. + let proposed = fixture.runtimes[0] + .propose(route.history, pa.bytes().to_vec()) + .await?; + fixture.drive(Some(proposed.position.index)).await?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let ra = ea.finish(ca)?; + let rb = eb.finish(cb)?; + assert_eq!(ra.receipt.result(), b"a-result"); + assert_eq!(rb.receipt.result(), b"b-result"); + assert_eq!(ra.version.sequence.get(), 2); + assert_eq!(rb.version.sequence.get(), 2); + assert_ne!(ra.version.history, rb.version.history); + ea.close()?; + eb.close()?; + Ok(()) + })? + .await??; + for (cell, expected, value) in [ + (&a, b"a-result".as_slice(), 1), + (&b, b"b-result".as_slice(), 7), + ] { + let mut executor = fixture.private_cell(cell).await?; + let expected = expected.to_vec(); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let CellStageOutcome::Duplicate(receipt) = + executor.stage(cell_attempt(85)?, false, |_| { + panic!("receipt retry must not execute callback") + })? + else { + anyhow::bail!("missing duplicate") + }; + assert_eq!(receipt.receipt.result(), expected); + executor.close()?; + Ok(()) + })? + .await??; + for voter in &fixture.cluster.voters { + let marker = voter + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected root")?; + let leaf = voter + .tree + .get(&voter.pages, marker.root(), &cell.cell)? + .context("selected cell")?; + let query = match leaf.version.image.commitment { + CellImageCommitment::Materialized(digest) => { + let pin = voter.images.adopt( + image_id(cell, 2)?, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }, + )?; + cellar_ltx::generation::open_immutable( + pin, + cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }, + )? + } + CellImageCommitment::Extents { + root, + manifest_bytes, + } => { + let store = voter.extents.as_ref().context("rooted store")?; + let view = store.open_root(ExtentRootReference::new( + ImageRoot::from_bytes(root), + manifest_bytes, + )?)?; + let cache = fixture + .cluster + ._root + .path() + .join(format!("verify-{}", cell.cell.as_str())); + std::fs::create_dir_all(&cache)?; + cellar_ltx::internal::open_extent_image( + view.image( + cache, + ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + )?, + cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }, + )? + } + }; + let observed: i64 = query + .connection() + .query_row("SELECT n FROM value", [], |r| r.get(0))?; + assert_eq!(observed, value); + } + } + let stale = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let mut executor = fixture.private_cell(&a).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let physical = fixture.cluster.domain.blocking(move || -> Result<()> { + let mut attempt = cell_attempt(86)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let result = executor.stage(attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+100", [])?; + let _ = entered.send(()); + wait.recv()?; + Ok(b"must-not-escape".to_vec()) + }); + assert!(matches!( + result, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Unavailable + )) + )); + executor.close()?; + Ok(()) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + fixture.runtimes[0].close().await?; + assert!(stale.check().is_err()); + release.send(())?; + physical.await??; + let marker = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected root")?; + let leaf = fixture.cluster.voters[0] + .tree + .get(&fixture.cluster.voters[0].pages, marker.root(), &a.cell)? + .context("selected cell")?; + assert_eq!(leaf.version, stale.cell().version); + drop(stale); + fixture.close().await +} From f807b2f70e01e3cd8c741557889e1965fd14ec49 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:47:37 -0700 Subject: [PATCH 048/568] test: simplify cell SQL fixture results for clippy --- crates/cellar/tests/native_cell_groups_execution.rs | 3 +-- crates/cellar/tests/support/cell_group_sql.rs | 5 ++--- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 4874b87fd..8026dada6 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -499,8 +499,7 @@ fn prepare_cell_image( empty.root(), empty.encode()?.len().try_into()?, )?); - let header = - cellar_ltx::ltx::Header::parse(&files.last().context("extent LTX")?.bytes)?; + let header = cellar_ltx::ltx::Header::parse(files.last().context("extent LTX")?.bytes)?; let geometry = Geometry::new( header.page_size, u64::from(header.page_size) * u64::from(header.commit), diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index ceac610ea..4a97d703c 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -59,8 +59,7 @@ impl RuntimeCluster { limits: ExecutionLimits::default(), }; let extents = self.cluster.voters[0].extents.clone(); - Ok(self - .cluster + self.cluster .domain .blocking(move || -> Result<_> { if let Some(store) = extents { @@ -90,7 +89,7 @@ impl RuntimeCluster { Ok(CellPrivateExecutor::open(options, authority)?) } })? - .await??) + .await? } } From b8ed4dd3dbaf495da9788d64cd849fca91e65717 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 18:54:41 -0700 Subject: [PATCH 049/568] docs: record cell SQL qualification and production pool handoff --- docs/design/multi-raft/cell-groups/README.md | 16 ++--- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 56 ++++++++++------- .../design/multi-raft/cell-groups/progress.md | 61 ++++++++++++++++--- 4 files changed, 96 insertions(+), 41 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 266f52284..d0b8db478 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -9,12 +9,11 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The requested deliverable is the executable design and implementation plan. -End-to-end implementation and hard cutover are the executor's target, with -separate task and release gates. See the [executor handoff](handoff.md) for the -starting revision, shared-node runtime checkpoint, unfinished working-tree draft, -reviewed main integration and commands. Plan delivery or an intermediate task -gate does not qualify the complete shared-group runtime. +The executable design and implementation plan has been delivered. The active +implementation goal continues through the end-to-end hard cutover, with separate +task and release gates. See the [executor handoff](handoff.md) for the current +source, qualification boundary, reviewed main integration and commands. Plan +delivery or an intermediate task gate does not qualify the complete runtime. This package implements the direction selected in the [density decision](../density-decision.md). The target is TiKV-style partitioned @@ -94,8 +93,9 @@ directory along with the earlier DONE-task evidence to the implementation executor. CG-04 now retains creation identity with the selected image and prepares ordered enrollment and exact retries. Its physical group owner consumes the catalog startup permit and verified selected image/manifest closure, and -obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. Per-cell SQL -execution, receipt settlement and serving integration remain required. Managed +obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. The per-cell SQL +producer and exact receipts now pass their partial regression gate. Fixed-pool +SQL ownership, fair admission and SDK serving integration remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 4af0eec0d..f08df121b 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "aec4029e77124848dc6b155c098f8eec9ccdacc4", + "source_revision": "f807b2f70e01e3cd8c741557889e1965fd14ec49", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 62c5069f2..5a244da27 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -2,9 +2,9 @@ # Cell-group implementation handoff -The requested delivery is an executable design and implementation plan. This -handoff prepares subsequent end-to-end implementation and hard cutover; that -work must continue through the full task and release gates. +The executable design and implementation plan is delivered. The active goal +continues through end-to-end implementation and hard cutover, including the full +task and release gates. This handoff does not claim completion of the shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many @@ -15,8 +15,8 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The shared-node runtime implementation and reviewed source baseline are at -`aec4029e`; see the latest checkpoint in +The reviewed source baseline is `f807b2f7`, qualifying the SQL producer after +the shared-node runtime at `aec4029e`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; @@ -43,15 +43,13 @@ and joins submitted physical work on close. Native cases use independent SQLite images and actual node workers; see [progress](progress.md) for qualification. The prior uncommitted runtime draft has been incorporated into this source. -An unfinished local SQL draft follows the committed checkpoint. It adds +The per-cell SQL producer is now committed at `f3d48bdc`. It adds `cell_groups/execution_authority.rs`, `execution/cell.rs`, shared receipt transaction helpers in `execution/command.rs`, and `tests/support/cell_group_sql.rs`, with changes to physical image capture, owner/runtime authority and native tests. -It separates a selected cell image/version from group progress and attempts exact -receipt settlement in both image modes. It is not in the committed checkpoint -and has no completed source-bound qualification record. Review `git diff` and -untracked files before continuing; a fresh checkout of the PR will not contain -this draft. Do not infer a passing CG-04 gate from its presence. +It separates a selected cell image/version from group progress and requires exact +receipt settlement in both image modes. The partial gate passes at `f807b2f7`: +133 native tests and eight verification commands. It does not complete CG-04. The next integration is per-cell SQL ownership and fair admission, concurrent initialization, exact receipt settlement and coherent read serving. Preserve the @@ -66,6 +64,17 @@ kept separately. Reuse the existing receipt transaction and callback machinery; do not manufacture a synthetic cell-scoped group to satisfy the old interfaces. A later mutation of another cell must not settle the first cell's pending output. +The registered callback boundary also needs a real identity conversion: +`application/consensus/class.rs::CommandContext::identity` currently exposes +`GroupScope`, and command identity validation/invocation passes that same scope +to inbox and system-effect authentication. Carry logical `CellHistory` through +this boundary and keep physical consensus proof separate. Do not enable effects +with a fabricated group scope; preserve the CG-08-before-CG-07 dependency. Reuse +`host_sql::Pool`'s fixed execution-domain workers and joined shutdown while making +cell admission, queued ordering and unresolved proposals key on cell history. +A cancelled caller cannot release physical callback ownership or allow a second +producer for that cell. + ## First executor session @@ -78,13 +87,14 @@ integration remain incomplete. Preserve its persistence, completion and physical lease boundaries during integration; do not mistake the low-level owner cases for the full CG-04 gate. -Before extending the draft, qualify its changes to the existing execution paths -with `native_raft_execution` and `native_raft_rooted` as well as the shared-group -execution suite. In particular, rerun both cell receipt cases after the latest -callback/owner-close test change. Check authority revocation, exact committed -entry matching, duplicate callbacks, cleanup and compaction behavior. Those -selected regressions supplement, and do not replace, the full task gate printed -by `show CG-04`. Fair per-cell queues and registered SDK dispatch remain absent. +The SQL checkpoint includes `native_raft_execution` and `native_raft_rooted` +as well as the shared-group execution suite. Both cell receipt cases include +the final callback/owner-close extension. Preserve authority revocation, exact +committed entry matching and duplicate callback suppression during integration. +Receipt reconciliation after compaction and preview/cache retention still need +the owning host. These regressions supplement, and do not replace, the full task +gate printed by `show CG-04`. Fair per-cell queues and registered SDK dispatch +remain absent. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -110,10 +120,12 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The current partial CG-04 evidence is retained at -`.cellar/cell-groups-evidence/CG-04-runtime-admission/checkpoint.json`. Transfer that -directory alongside the earlier owner checkpoint and failed gate. The runtime -record covers 60 selected Rust cases and eight verification commands, with -explicit limitations; it cannot stand in for the complete six-case CG-04 contract. +`.cellar/cell-groups-evidence/CG-04-cell-sql/checkpoint.json`. Transfer that entire +directory, including copied binaries, failed and successful logs, with the prior +runtime/owner and DONE-task evidence. The SQL checkpoint binds 173 source inputs, +six executables, 133 selected Rust cases and eight verification commands. It +cannot stand in for the complete six-case CG-04 contract. Three RustFS-specific +cases remain explicitly unrun at this checkpoint. ## Delivery sequence diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index cb2a44248..1fd547df8 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -2,14 +2,59 @@ # Cell-group implementation progress -The current delivery is the executable design and implementation plan; see the -[executor handoff](handoff.md) for subsequent implementation and hard cutover. +The executable design and implementation plan is delivered. The active goal +continues through end-to-end implementation and hard cutover; see the +[executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. -## Plan delivery and unfinished SQL draft +## CG-04 per-cell SQL producer + +Implementation source: `f3d48bdc`; qualified successor: +`f807b2f70e01e3cd8c741557889e1965fd14ec49`. + +The synchronous `CellPrivateExecutor` executes one cell's private SQLite and +receipt transaction, captures its physical LTX update and encodes the actual +cell-addressed group entry. Materialized and extent-backed images share the +existing receipt admission, maintenance and transaction implementation. Physical +capture no longer requires a synthetic cell-scoped group identity. + +The real leader issues an authority pin for the selected cell, route and term. +Another cell can advance without invalidating that base. Leadership loss, owner +failure or owner drop revokes the observation; the producer checks again after +capture. Settlement requires fresh authority, the exact committed entry bytes +and expected cell image/version. Duplicate receipts come from the selected +immutable cell image without another callback. An unresolved proposal remains +busy; closing its writer does not establish a failed proposal outcome. + +The two new real-three-voter cases use the same client/sequence in two distinct +cells, commit B before A, refuse settling A with B's authority, then verify each +receipt and independent SQLite value on all voters. Both image modes also stop +the consensus owner while a SQL callback is blocked, then require a typed +Unavailable refusal and unchanged selected cell version after release. + +All 133 selected native cases pass: 13 shared execution, 15 catalog, 15 active +storage, 21 prior execution, 50 rooted and 19 prior runtime cases. Four ignored +process helpers are invoked by their parent tests; three explicitly ignored +RustFS cases were not run. All-target Clippy with warnings denied, formatting +(1,790 Rust files), layout, native dependency isolation, docs, reviewed source +preflight and all 37 plan-validator tests pass. The partial record at +`.cellar/cell-groups-evidence/CG-04-cell-sql/checkpoint.json` binds 173 source inputs, +eight successful command logs and six copied executables. It retains preliminary +runs and the initial two-fixture Clippy failure; the successor removes only an +unnecessary result wrapper and borrow, without changing assertions. + +The fixed SQL pool, fair per-cell admission, concurrent initialization, +registered SDK dispatch and coherent read serving still require integration. +The producer currently reopens after settlement and conservatively refuses +exact-entry settlement after compaction; receipt reconciliation, preview +retention and cache lifecycle must be integrated with the owning host. CG-04 +remains IN_PROGRESS. No full cutover, RustFS drill or performance qualification +is claimed by this checkpoint. + +## Earlier plan delivery and SQL draft handoff The implementation package contains 16 dependency-ordered tasks, 86 required cases, 14 invariants and nine qualification profiles. Each task specifies owning @@ -17,13 +62,11 @@ paths, implementation steps, acceptance oracles and executable test commands. The plan runner prints those commands and checks evidence; it does not execute the implementation or provision infrastructure. -The current working tree also contains an unfinished per-cell SQL producer, +At plan delivery the working tree retained an unfinished per-cell SQL producer, leader-issued execution authority, shared receipt helpers and tests for both -image formats. These are preserved for the next implementation session. They -are not included in the committed runtime checkpoint below, and the latest -callback revocation extension has not completed its regression gate. The -handoff records the affected boundaries and next checks. No task status or -runtime qualification is advanced by this plan delivery. +image formats. That draft has since been committed and qualified at the partial +SQL checkpoint above. The plan delivery itself advanced no task status or +runtime qualification. Plan delivery checks pass: package validation, reviewed source preflight, all 37 validator unit tests and repository documentation validation (113 documents, From 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 19:22:32 -0700 Subject: [PATCH 050/568] feat: run cell SQL on bounded shared node workers --- crates/cellar/durability/raft/README.md | 35 +- crates/cellar/durability/raft/budget.rs | 39 ++ .../raft/cell_groups/execution_authority.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../raft/host_sql/cell_groups/admission.rs | 101 +++++ .../raft/host_sql/cell_groups/mod.rs | 416 ++++++++++++++++++ .../raft/host_sql/cell_groups/worker.rs | 221 ++++++++++ crates/cellar/durability/raft/host_sql/mod.rs | 98 ++++- crates/cellar/durability/raft/storage.rs | 20 + .../tests/native_cell_groups_execution.rs | 12 +- .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 64 ++- .../tests/support/cell_group_sql_pool.rs | 295 +++++++++++++ 13 files changed, 1279 insertions(+), 35 deletions(-) create mode 100644 crates/cellar/durability/raft/host_sql/cell_groups/admission.rs create mode 100644 crates/cellar/durability/raft/host_sql/cell_groups/mod.rs create mode 100644 crates/cellar/durability/raft/host_sql/cell_groups/worker.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_pool.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index d39b21177..18bfaadc5 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -52,13 +52,34 @@ A callback must recheck it after capture, and settlement requires a fresh author the exact committed entry and that cell's expected image/version. A receipt retry reads the selected immutable cell image and does not invoke the callback again. The synchronous executor must run on an owning blocking worker. It currently -reopens after settlement; it is not yet the production SQL pool or registered SDK -operation dispatcher. Receipt proof alone does not provide S3 coverage. - -Per-cell SQL admission, SDK dispatch and the production hard cutover remain -required; the existing host still uses its prior single-cell owner. Transport -authentication and actual image preparation remain the caller's responsibility -at this verification boundary. See the [implementation status](../../../../docs/design/multi-raft/cell-groups/progress.md) +reopens after settlement. Receipt proof alone does not provide S3 coverage. + +`CellGroupSqlPool` now sends cell-history requests through the existing fixed +`host_sql::Pool` workers and fair request queue. A stable cell key chooses its +worker; the node does not create a thread or task for each enrolled cell. One +admission lease covers queued SQL, the physical callback and its unresolved +proposal even when the waiting caller is cancelled. Node/group byte and item +reservations cover capture and retained outputs. Finished capture refunds its +transient buffer budget while prepared/receipt responses retain their payload +charge, including clones. Extent workspace creation and final cleanup run on the +admitted blocking worker. An exclusive engine lease prevents a second node SQL +pool until workers, the handle and retained outputs release their ownership. A lost stage response can recover the same prepared proposal without +another callback. Admission checks the authority's exact engine owner. + +Each worker has one reserved progress slot, independent of its ordinary queue. +Submission marking is idempotent for the retained proposal. Settlement still +requires the exact committed receipt; group progress cannot release another +cell's lease. Pool close stops admission and its domain cleanup owner joins every +physical callback/worker even if a close waiter is cancelled. Closing a writer or +pool does not establish a failed submitted outcome or authorize replay; the host +must reconcile pending commands against committed state before admitting again. + +Registered SDK dispatch, concurrent initialization, coherent query serving and +the production hard cutover remain required; the existing host still uses its +prior single-cell owner. The new pool's callback entry point remains a verification +boundary. Transport authentication and actual image preparation are also caller +responsibilities at that boundary. See the +[implementation status](../../../../docs/design/multi-raft/cell-groups/progress.md) for the exact qualification boundary. Managed empty hosts expose `RaftCells::inspect_peer_node` under rooted fleet diff --git a/crates/cellar/durability/raft/budget.rs b/crates/cellar/durability/raft/budget.rs index 9b158ab34..6f101ef71 100644 --- a/crates/cellar/durability/raft/budget.rs +++ b/crates/cellar/durability/raft/budget.rs @@ -63,6 +63,23 @@ pub(crate) struct Permit { bytes: usize, } +impl Permit { + /// Refund transient capture capacity while keeping the operation slot and + /// the retained payload charged. A reservation can never grow here. + pub(crate) fn shrink_to(&mut self, bytes: usize) -> bool { + if bytes > self.bytes { + return false; + } + let released = self.bytes - bytes; + self.bytes = bytes; + self.owner.bytes.fetch_sub(released, Ordering::AcqRel); + if released != 0 { + self.owner.wake.notify_one(); + } + true + } +} + impl Drop for Permit { fn drop(&mut self) { self.owner.bytes.fetch_sub(self.bytes, Ordering::AcqRel); @@ -70,3 +87,25 @@ impl Drop for Permit { self.owner.wake.notify_one(); } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn shrink_refunds_bytes_but_keeps_the_operation_slot() { + let budget = Budget::new(100, 2); + let mut capture = budget.acquire(100).unwrap(); + assert!(budget.acquire(1).is_none()); + assert!(capture.shrink_to(20)); + let next = budget.acquire(80).unwrap(); + assert_eq!(budget.usage(), (100, 2)); + assert!(budget.acquire(0).is_none()); + assert!(!capture.shrink_to(21)); + assert_eq!(budget.usage(), (100, 2)); + drop(capture); + assert_eq!(budget.usage(), (80, 1)); + drop(next); + assert_eq!(budget.usage(), (0, 0)); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index 858acd56b..facdc4011 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -48,6 +48,10 @@ impl CellExecutionAuthority { Ok(authority) } + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + pub fn route(&self) -> &GroupRoute { &self.route } diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e419cdaaa..414427be9 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -2,6 +2,11 @@ //! Complete group-state storage. A synced page is data, not Raft authority. +pub use super::host_sql::cell_groups::{ + CellGroupSqlPool, CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, + CellSqlReceipt, CellSqlRootOptions, CellSqlUsage, +}; + mod execution_authority; pub use execution_authority::CellExecutionAuthority; diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs new file mode 100644 index 000000000..82fcc4f76 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs @@ -0,0 +1,101 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{Mutex, Weak}; + +pub(super) struct Admission { + lease: Arc, + node: Arc, + limits: CellSqlLimits, + state: Mutex, +} + +#[derive(Default)] +struct State { + cells: BTreeSet, + groups: BTreeMap>, +} + +pub(super) struct Bytes { + _lease: Arc, + _node: Permit, + _group: Permit, +} + +impl Bytes { + pub fn shrink(bytes: &mut Arc, retained: usize) -> Result<(), ExecutionError> { + let unique = + Arc::get_mut(bytes).ok_or(ExecutionError::Invalid("shared capture admission"))?; + if !unique._node.shrink_to(retained) || !unique._group.shrink_to(retained) { + return Err(ExecutionError::Invalid("retained SQL output bound")); + } + Ok(()) + } +} + +pub(super) struct CellLease { + owner: Arc, + cell: CellHistory, +} + +impl Drop for CellLease { + fn drop(&mut self) { + if let Ok(mut state) = self.owner.state.lock() { + state.cells.remove(&self.cell); + } + } +} + +impl Admission { + pub fn new(limits: CellSqlLimits, lease: Arc) -> Arc { + Arc::new(Self { + lease, + node: Budget::new(limits.node_bytes, limits.max_pending_cells), + limits, + state: Mutex::new(State::default()), + }) + } + + pub fn reserve( + self: &Arc, + cell: &CellHistory, + group: GroupHistory, + bytes: usize, + ) -> Result<(CellLease, Arc), ExecutionError> { + let mut state = self.state.lock().map_err(|_| unavailable())?; + if state.cells.contains(cell) { + return Err(ExecutionError::Busy); + } + let node = self.node.acquire(bytes).ok_or_else(overloaded)?; + // Reclaim metadata only, never authority or pending physical ownership. + state.groups.retain(|_, budget| budget.strong_count() != 0); + let group_budget = state + .groups + .get(&group) + .and_then(Weak::upgrade) + .unwrap_or_else(|| { + Budget::new(self.limits.group_bytes, self.limits.group_pending_cells) + }); + let group_permit = group_budget.acquire(bytes).ok_or_else(overloaded)?; + state.groups.insert(group, Arc::downgrade(&group_budget)); + state.cells.insert(cell.clone()); + Ok(( + CellLease { + owner: self.clone(), + cell: cell.clone(), + }, + Arc::new(Bytes { + _lease: self.lease.clone(), + _node: node, + _group: group_permit, + }), + )) + } + + pub fn usage(&self) -> Result<(usize, usize, usize), ExecutionError> { + let cells = self.state.lock().map_err(|_| unavailable())?.cells.len(); + let (bytes, reservations) = self.node.usage(); + Ok((cells, bytes, reservations)) + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs new file mode 100644 index 000000000..acef70105 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -0,0 +1,416 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cell-history admission and commands on the existing fixed node SQL workers. + +mod admission; +mod worker; + +use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; +use crate::raft::cell_groups::{cell_image_history_digest, CellExecutionAuthority}; +use crate::raft::completion::RuntimeError; +use crate::raft::execution::{ + CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellStageOutcome, + CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, RootPrivateOptions, +}; +use crate::raft::runtime::Terminal; +use crate::raft::storage::ReplicaLog; +use crate::storage::CommandTransaction; +use admission::{Admission, Bytes, CellLease}; +use cellar_logic::cell_group::{CellHistory, GroupHistory}; +use cellar_logic::consensus::{CommandFailure, ProposalId}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, +}; +use tokio::sync::{oneshot, Notify}; +pub(super) use worker::{Cells, Request}; + +#[derive(Clone, Copy)] +pub struct CellSqlLimits { + pub workers: usize, + pub queue_per_worker: usize, + pub max_pending_cells: usize, + pub group_pending_cells: usize, + pub node_bytes: usize, + pub group_bytes: usize, +} + +impl CellSqlLimits { + fn validate(self) -> Result<(), ExecutionError> { + if !(1..=256).contains(&self.workers) + || !(1..=4096).contains(&self.queue_per_worker) + || !(1..=65_536).contains(&self.max_pending_cells) + || self.group_pending_cells == 0 + || self.group_pending_cells > self.max_pending_cells + || self.group_bytes == 0 + || self.group_bytes > self.node_bytes + { + return Err(ExecutionError::Invalid("cell SQL pool limits")); + } + Ok(()) + } +} + +/// Extent configuration without a live private workspace. The admitted worker +/// creates and destroys that physical owner; rejected/queued requests do no I/O. +#[derive(Clone)] +pub struct CellSqlRootOptions { + pub host: cellar_ltx::LtxHost, + pub sqlite_vfs: Option, + pub images: cellar_ltx::internal::NodeExtentStore, + pub query_services: cellar_ltx::generation::ReadServices, + pub application_digest: [u8; 32], + pub limits: ExecutionLimits, + pub cache_directory: std::path::PathBuf, + pub cache_pool: cellar_ltx::stream_snapshot::ScratchPool, + pub cache: cellar_ltx::internal::CacheLimits, +} + +#[derive(Clone)] +pub enum CellSqlOptions { + Materialized(PrivateOptions), + Extents(CellSqlRootOptions), +} + +impl CellSqlOptions { + fn limits(&self) -> ExecutionLimits { + match self { + Self::Materialized(o) => o.limits, + Self::Extents(o) => o.limits, + } + } + + fn open( + self, + authority: CellExecutionAuthority, + ) -> Result { + match self { + Self::Materialized(o) => CellPrivateExecutor::open(o, authority), + Self::Extents(o) => { + let workspace = o + .images + .create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(o.limits)?, + ) + .map_err(crate::raft::execution::storage)?; + CellPrivateExecutor::open_rooted( + RootPrivateOptions { + workspace, + host: o.host, + sqlite_vfs: o.sqlite_vfs, + images: o.images, + query_services: o.query_services, + application_digest: o.application_digest, + limits: o.limits, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + authority, + ) + } + } + } +} + +/// Clones retain both node and group admission through delivery and submission. +#[derive(Clone)] +pub struct CellSqlPrepared { + prepared: CellPreparedCommand, + _bytes: Arc, +} + +impl CellSqlPrepared { + pub fn bytes(&self) -> &[u8] { + self.prepared.bytes() + } + + pub fn proposal(&self) -> ProposalId { + self.prepared.proposal() + } +} + +pub struct CellSqlReceipt { + receipt: CellCommittedReceipt, + _bytes: Arc, +} + +impl CellSqlReceipt { + pub fn receipt(&self) -> &CellCommittedReceipt { + &self.receipt + } +} + +pub enum CellSqlOutcome { + Prepared(CellSqlPrepared), + Receipt(Box), + Submitted, +} + +pub struct CellSqlUsage { + pub active_cells: usize, + pub retained_bytes: usize, + pub retained_operations: usize, + pub running_workers: usize, + pub worker_capacity: usize, + pub queued_or_running: usize, + pub completion_reservations: usize, +} + +#[derive(Clone)] +pub struct CellGroupSqlPool { + inner: Arc, +} + +struct Handle { + engine_owner: Arc<()>, + workers: Vec, + resources: ResourceObserver, + admission: Arc, + closing: Arc, + stop: Arc, + terminal: Arc, +} + +impl Drop for Handle { + fn drop(&mut self) { + self.closing.store(true, Ordering::Release); + self.stop.notify_one(); + } +} + +fn unavailable() -> ExecutionError { + ExecutionError::Refused(CommandFailure::Unavailable) +} + +fn overloaded() -> ExecutionError { + ExecutionError::Refused(CommandFailure::Overloaded) +} + +impl CellGroupSqlPool { + pub fn start(log: &ReplicaLog, limits: CellSqlLimits) -> Result { + limits.validate()?; + log.cell_groups().map_err(crate::raft::execution::storage)?; + let lease = log + .claim_sql_pool() + .map_err(crate::raft::execution::storage)?; + let domain = log.domain(); + let mut pool = Pool::start( + domain.clone(), + limits.workers, + limits.queue_per_worker, + limits.max_pending_cells, + Budgets { + business: Budget::new(limits.node_bytes, limits.max_pending_cells), + archive_capture: Budget::new(1, 1), + query_pins: Budget::new(1, 1), + }, + log.performance(), + )?; + let workers = (0..limits.workers) + .map(|n| pool.worker(n)) + .collect::, _>>()?; + let resources = pool.resource_observer(); + let closing = pool.stopped.clone(); + let stop = Arc::new(Notify::new()); + let terminal = Arc::new(Terminal::new()); + let inner = Arc::new(Handle { + engine_owner: log.owner(), + workers, + resources, + admission: Admission::new(limits, lease.clone()), + closing: closing.clone(), + stop: stop.clone(), + terminal: terminal.clone(), + }); + let cleanup_domain = domain.clone(); + // Cleanup owns the join handles, so cancelling a close waiter cannot + // release physical workers. Domain shutdown also joins this cleanup. + domain.spawn_cleanup(async move { + struct Complete(Arc); + + impl Drop for Complete { + fn drop(&mut self) { self.0.complete(Err(RuntimeError::Closed)); } + } + let _complete = Complete(terminal.clone()); + let _physical_lease = lease; + crate::asyncrt::select! { _ = stop.notified() => (), _ = cleanup_domain.wait_for_work() => () } + closing.store(true, Ordering::Release); + let result = pool.join().await.map_err(|e| RuntimeError::Protocol(e.to_string())); + drop(pool); + drop(_physical_lease); + terminal.complete(result); + }).map_err(|_| unavailable())?; + Ok(Self { inner }) + } + + pub fn usage(&self) -> Result { + let (active_cells, retained_bytes, retained_operations) = self.inner.admission.usage()?; + let resources = self.inner.resources.capture().ok_or_else(unavailable)?; + Ok(CellSqlUsage { + active_cells, + retained_bytes, + retained_operations, + running_workers: resources.running_workers, + worker_capacity: resources.worker_capacity, + queued_or_running: resources.work_reservations, + completion_reservations: self + .inner + .workers + .iter() + .map(|w| 1 - w.completion_slots.available_permits()) + .sum(), + }) + } + + fn admit(&self) -> Result<(), ExecutionError> { + if self.inner.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + Ok(()) + } + + async fn call( + &self, + cell: CellHistory, + action: worker::Action, + ) -> Result { + self.admit()?; + let hash = cell_image_history_digest(&cell); + let index = u64::from_le_bytes( + hash[..8] + .try_into() + .map_err(|_| ExecutionError::Invalid("cell worker key"))?, + ) % self.inner.workers.len() as u64; + let worker = &self.inner.workers[index as usize]; + worker.domain.mono_ms().map_err(|_| unavailable())?; + let slots = if action.progress_priority() { + &worker.completion_slots + } else { + &worker.slots + }; + let slot = slots + .clone() + .try_acquire_owned() + .map_err(|error| match error { + tokio::sync::TryAcquireError::Closed => unavailable(), + tokio::sync::TryAcquireError::NoPermits => overloaded(), + })?; + self.admit()?; + let (reply, wait) = oneshot::channel(); + worker + .sender + .try_send(Envelope::Shared(Request { + cell, + action, + reply, + _slot: slot, + queued: worker + .performance + .timer(crate::raft::performance::Operation::SqlQueue), + })) + .map_err(|_| unavailable())?; + wait.await.map_err(|_| unavailable())? + } + + /// The owning worker retains the callback and admission even if this waiter + /// is cancelled. Caller-provided callbacks are the verification boundary; + /// the production host must dispatch registered operations here. + pub async fn stage( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result> + Send + 'static, + ) -> Result { + self.admit()?; + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + if attempt.request.operation.is_empty() || attempt.request.operation.len() > 128 { + return Err(ExecutionError::Invalid("command operation")); + } + let limits = options.limits(); + limits.validate()?; + // Bound simultaneous encoded capture, entry, preview verification and + // result buffers before queueing or opening SQLite. Disk/workspace + // reservation is separately enforced by the chosen image owner. + let bytes = limits + .envelope + .max_payload_bytes + .checked_mul(4) + .and_then(|n| n.checked_add(limits.max_result_bytes)) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)?; + let cell = authority.cell().version.history.clone(); + let (lease, bytes) = + self.inner + .admission + .reserve(&cell, authority.route().history, bytes)?; + self.call( + cell, + worker::Action::Stage(Box::new(worker::Stage { + options, + authority, + attempt, + allow_new, + callback: Box::new(callback), + lease, + bytes, + })), + ) + .await + } + + /// Recover a lost stage response without another callback or producer. + pub async fn resume( + &self, + cell: CellHistory, + proposal: ProposalId, + ) -> Result { + match self.call(cell, worker::Action::Resume(proposal)).await? { + CellSqlOutcome::Prepared(prepared) => Ok(prepared), + _ => Err(ExecutionError::Invalid("cell SQL resume response")), + } + } + + pub async fn mark_submitted( + &self, + cell: CellHistory, + proposal: ProposalId, + ) -> Result<(), ExecutionError> { + match self.call(cell, worker::Action::Submit(proposal)).await? { + CellSqlOutcome::Submitted => Ok(()), + _ => Err(ExecutionError::Invalid("cell SQL submit response")), + } + } + + pub async fn finish( + &self, + authority: CellExecutionAuthority, + ) -> Result { + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = authority.cell().version.history.clone(); + match self + .call(cell, worker::Action::Finish(Box::new(authority))) + .await? + { + CellSqlOutcome::Receipt(receipt) => Ok(*receipt), + _ => Err(ExecutionError::Invalid("cell SQL finish response")), + } + } + + pub async fn close(&self) -> Result<(), ExecutionError> { + self.inner.closing.store(true, Ordering::Release); + self.inner.stop.notify_one(); + self.inner + .terminal + .wait() + .await + .map_err(crate::raft::execution::storage) + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs new file mode 100644 index 000000000..4e51d423f --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -0,0 +1,221 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use std::collections::BTreeMap; + +type Callback = Box) -> anyhow::Result> + Send>; + +pub(super) struct Stage { + pub options: CellSqlOptions, + pub authority: CellExecutionAuthority, + pub attempt: CommandAttempt, + pub allow_new: bool, + pub callback: Callback, + pub lease: CellLease, + pub bytes: Arc, +} + +pub(super) enum Action { + Stage(Box), + Resume(ProposalId), + Submit(ProposalId), + Finish(Box), +} + +impl Action { + pub(super) fn progress_priority(&self) -> bool { + !matches!(self, Self::Stage(_)) + } +} + +pub(in crate::raft::host_sql) struct Request { + pub cell: CellHistory, + pub(super) queued: crate::raft::performance::Timer, + pub(super) action: Action, + pub(super) reply: oneshot::Sender>, + pub(super) _slot: tokio::sync::OwnedSemaphorePermit, +} + +impl Request { + pub(in crate::raft::host_sql) fn progress_priority(&self) -> bool { + self.action.progress_priority() + } +} + +struct Pending { + executor: CellPrivateExecutor, + prepared: CellSqlPrepared, + submitted: bool, + _lease: CellLease, +} + +#[derive(Default)] +pub(in crate::raft::host_sql) struct Cells { + pending: BTreeMap, +} + +impl Cells { + pub fn len(&self) -> usize { + self.pending.len() + } + + pub fn execute( + &mut self, + request: Request, + max: usize, + domain: &crate::asyncrt::DomainToken, + stopped: &AtomicBool, + performance: &Arc, + ) { + let Request { + cell, + action, + reply, + _slot, + queued, + } = request; + let operation = match &action { + Action::Stage(_) => crate::raft::performance::Operation::SqlStage, + Action::Finish(_) => crate::raft::performance::Operation::SqlPublish, + _ => crate::raft::performance::Operation::SqlOther, + }; + let timer = performance.timer(operation); + let result = if stopped.load(Ordering::Acquire) || domain.mono_ms().is_err() { + queued.finish(false); + Err(unavailable()) + } else { + queued.finish(true); + self.run(cell, action, max) + }; + // A lost receiver drops only the delivery. Pending owns both the + // private writer and per-cell reservation until exact settlement. + timer.finish(result.is_ok()); + let _ = reply.send(result); + } + + fn run( + &mut self, + cell: CellHistory, + action: Action, + max: usize, + ) -> Result { + match action { + Action::Stage(stage) => { + if self.pending.contains_key(&cell) { + return Err(ExecutionError::Busy); + } + if self.pending.len() >= max { + return Err(overloaded()); + } + let Stage { + options, + authority, + attempt, + allow_new, + callback, + lease, + mut bytes, + } = *stage; + if authority.cell().version.history != cell { + return Err(ExecutionError::Invalid("SQL cell history")); + } + let max_result = options.limits().max_result_bytes; + let mut executor = options.open(authority)?; + let result = executor.stage(attempt, allow_new, callback); + match result { + Ok(CellStageOutcome::Prepared(prepared)) => { + let retained = prepared + .bytes() + .len() + .checked_add(max_result) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + let prepared = CellSqlPrepared { + prepared, + _bytes: bytes, + }; + self.pending.insert( + cell, + Pending { + executor, + prepared: prepared.clone(), + submitted: false, + _lease: lease, + }, + ); + Ok(CellSqlOutcome::Prepared(prepared)) + } + Ok(CellStageOutcome::Duplicate(receipt)) => { + executor.close()?; + let retained = receipt + .receipt + .result() + .len() + .checked_add(4096) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { + receipt: *receipt, + _bytes: bytes, + }))) + } + Err(error) => { + executor.close()?; + Err(error) + } + } + } + Action::Resume(proposal) => { + let pending = self + .pending + .get(&cell) + .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + if pending.prepared.proposal() != proposal { + return Err(ExecutionError::Invalid("cell producer proposal")); + } + Ok(CellSqlOutcome::Prepared(pending.prepared.clone())) + } + Action::Submit(proposal) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + if pending.prepared.proposal() != proposal { + return Err(ExecutionError::Invalid("cell producer proposal")); + } + if !pending.submitted { + pending.executor.mark_submitted(proposal)?; + pending.submitted = true; + } + Ok(CellSqlOutcome::Submitted) + } + Action::Finish(authority) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + let receipt = pending.executor.finish(*authority)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Invalid("cell producer changed"))?; + pending.executor.close()?; + Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { + receipt, + _bytes: pending.prepared._bytes, + }))) + } + } + } + + pub fn close(&mut self) -> Result<(), ExecutionError> { + let mut result = Ok(()); + for (_, pending) in std::mem::take(&mut self.pending) { + if let Err(error) = pending.executor.close() { + result = Err(error); + } + } + result + } +} diff --git a/crates/cellar/durability/raft/host_sql/mod.rs b/crates/cellar/durability/raft/host_sql/mod.rs index d58b0c91d..e274a0d70 100644 --- a/crates/cellar/durability/raft/host_sql/mod.rs +++ b/crates/cellar/durability/raft/host_sql/mod.rs @@ -4,6 +4,7 @@ //! cancellation joins a running callback before these owners release locks. use super::apply::ImageApplier; +pub(crate) mod cell_groups; mod images; use super::archive::{ArchiveLimits, CommittedArchiveCut}; use super::budget::{Budget, Permit}; @@ -211,15 +212,53 @@ struct Request { _slot: tokio::sync::OwnedSemaphorePermit, } -struct RequestQueue { - receiver: mpsc::Receiver, - pending: VecDeque, +trait Scheduled { + fn same_cell(&self, other: &Self) -> bool; + + fn progress_priority(&self) -> bool; +} + +impl Scheduled for Request { + fn same_cell(&self, other: &Self) -> bool { + self.scope == other.scope + } + + fn progress_priority(&self) -> bool { + self.action.progress_priority() + } +} + +enum Envelope { + Legacy(Box), + Shared(cell_groups::Request), +} + +impl Scheduled for Envelope { + fn same_cell(&self, other: &Self) -> bool { + match (self, other) { + (Self::Legacy(a), Self::Legacy(b)) => a.same_cell(b), + (Self::Shared(a), Self::Shared(b)) => a.cell == b.cell, + _ => false, + } + } + + fn progress_priority(&self) -> bool { + match self { + Self::Legacy(request) => request.progress_priority(), + Self::Shared(request) => request.progress_priority(), + } + } +} + +struct RequestQueue { + receiver: mpsc::Receiver, + pending: VecDeque, capacity: usize, progress_streak: usize, } -impl RequestQueue { - fn new(receiver: mpsc::Receiver, capacity: usize) -> Self { +impl RequestQueue { + fn new(receiver: mpsc::Receiver, capacity: usize) -> Self { Self { receiver, pending: VecDeque::new(), @@ -228,7 +267,7 @@ impl RequestQueue { } } - fn next(&mut self) -> Result { + fn next(&mut self) -> Result { if self.pending.is_empty() { self.pending .push_back(self.receiver.recv_timeout(Duration::from_millis(10))?); @@ -250,12 +289,12 @@ impl RequestQueue { .iter() .enumerate() .find(|(index, request)| { - request.action.progress_priority() + request.progress_priority() && !self .pending .iter() .take(*index) - .any(|earlier| earlier.scope == request.scope) + .any(|earlier| earlier.same_cell(request)) }) .map_or(0, |(index, _)| index) } else { @@ -265,7 +304,7 @@ impl RequestQueue { .pending .remove(index) .ok_or(mpsc::RecvTimeoutError::Disconnected)?; - self.progress_streak = if request.action.progress_priority() { + self.progress_streak = if request.progress_priority() { self.progress_streak.saturating_add(1).min(3) } else { 0 @@ -291,7 +330,8 @@ impl Action { pub(crate) struct Pool { performance: Arc, domain: crate::asyncrt::DomainToken, - senders: Vec>, + senders: Vec>, + completion_slots: Vec>, slots: Vec>, workers: Vec>>, stopped: Arc, @@ -363,7 +403,8 @@ pub(crate) struct Budgets { pub(crate) struct Worker { performance: Arc, domain: crate::asyncrt::DomainToken, - sender: mpsc::SyncSender, + sender: mpsc::SyncSender, + completion_slots: Arc, slots: Arc, stopped: Arc, } @@ -388,6 +429,7 @@ impl Pool { performance: performance.clone(), domain: domain.clone(), senders: vec![], + completion_slots: vec![], slots: vec![], workers: vec![], stopped: stopped.clone(), @@ -396,7 +438,10 @@ impl Pool { images: images.clone(), }; for _ in 0..count { - let (sender, receiver) = mpsc::sync_channel::(queue); + let capacity = queue + .checked_add(1) + .ok_or(ExecutionError::Invalid("SQL queue capacity"))?; + let (sender, receiver) = mpsc::sync_channel::(capacity); let owner = domain.clone(); let performance = performance.clone(); let stopped = stopped.clone(); @@ -412,7 +457,8 @@ impl Pool { // retire this physical worker count. let _running = Running::enter(running); let mut cells = BTreeMap::::new(); - let mut requests = RequestQueue::new(receiver, queue); + let mut shared = cell_groups::Cells::default(); + let mut requests = RequestQueue::new(receiver, capacity); let result = loop { if stopped.load(Ordering::Acquire) || owner.mono_ms().is_err() { break Ok(()); @@ -422,6 +468,19 @@ impl Pool { Err(mpsc::RecvTimeoutError::Timeout) => continue, Err(mpsc::RecvTimeoutError::Disconnected) => break Ok(()), }; + let request = match request { + Envelope::Shared(request) => { + shared.execute( + request, + max_cells.saturating_sub(cells.len()), + &owner, + &stopped, + &performance, + ); + continue; + } + Envelope::Legacy(request) => *request, + }; let Request { queued, scope, @@ -484,7 +543,7 @@ impl Pool { let timer = performance.timer(measured); let result = Self::execute( &mut cells, - max_cells, + max_cells.saturating_sub(shared.len()), &scope, action, &query_pins, @@ -547,7 +606,7 @@ impl Pool { }; // Keep joining/closing every resource even when one reports an // error. Their locks are released on this blocking owner. - let mut cleanup = Ok(()); + let mut cleanup = shared.close(); for (_, mut cell) in cells { if let Some(private) = cell.private.take() { if let Err(error) = private.close() { @@ -560,6 +619,7 @@ impl Pool { .map_err(|_| ExecutionError::Refused(CommandFailure::Unavailable))?; pool.senders.push(sender); pool.slots.push(Arc::new(Semaphore::new(queue))); + pool.completion_slots.push(Arc::new(Semaphore::new(1))); pool.workers.push(task); } Ok(pool) @@ -599,6 +659,7 @@ impl Pool { performance: self.performance.clone(), domain: self.domain.clone(), sender, + completion_slots: self.completion_slots[worker].clone(), slots: self.slots[worker].clone(), stopped: self.stopped.clone(), }) @@ -606,7 +667,7 @@ impl Pool { pub fn stop(&self) { self.stopped.store(true, Ordering::Release); - for slots in &self.slots { + for slots in self.slots.iter().chain(&self.completion_slots) { slots.close(); } } @@ -764,6 +825,7 @@ impl Worker { sender, slots, stopped, + completion_slots: _, } = self.clone(); Box::pin(async move { let queued = performance.timer(Measured::SqlQueue); @@ -776,7 +838,7 @@ impl Worker { } let (reply, result) = oneshot::channel(); sender - .try_send(Request { + .try_send(Envelope::Legacy(Box::new(Request { queued, scope, action, @@ -784,7 +846,7 @@ impl Worker { _work: work, cancel, _slot: slot, - }) + }))) .map_err(|_| ExecutionError::Refused(CommandFailure::Unavailable))?; result .await diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 3bfc929c5..fbd0969a2 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -173,6 +173,7 @@ struct LogHandle { options: ReplicaLogOptions, closing: Arc, runtime_claimed: AtomicBool, + sql_claimed: AtomicBool, abort: Arc, closed: Arc, terminal: Arc, @@ -258,6 +259,15 @@ enum Work { /// Exclusive consensus ownership; normal release follows a final storage fence. pub(crate) struct RuntimeLease(Arc); +/// Held by the SQL owner, physical workers and retained response admission. +pub(crate) struct SqlPoolLease(Arc); + +impl Drop for SqlPoolLease { + fn drop(&mut self) { + self.0.sql_claimed.store(false, Ordering::Release); + } +} + impl RuntimeLease { /// Dropping an unfenced worker deliberately leaves the claim poisoned. pub(crate) fn release(self) { @@ -316,6 +326,15 @@ impl ReplicaLog { Ok(RuntimeLease(self.inner.clone())) } + pub(crate) fn claim_sql_pool(&self) -> Result, LogError> { + self.admit()?; + self.inner + .sql_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| LogError::AlreadyExists)?; + Ok(Arc::new(SqlPoolLease(self.inner.clone()))) + } + pub(crate) fn domain(&self) -> crate::asyncrt::DomainToken { self.inner.domain.clone() } @@ -390,6 +409,7 @@ impl ReplicaLog { options: options.clone(), closing: closing.clone(), runtime_claimed: AtomicBool::new(false), + sql_claimed: AtomicBool::new(false), abort: abort.clone(), closed: closed.clone(), terminal: terminal.clone(), diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 8026dada6..9415c4c66 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Real raft-rs owners, three independent engines and cell-addressed SQLite/LTX. -//! This boundary does not qualify SDK dispatch, worker fairness or snapshots. +//! Selected fixed-pool and consensus boundaries; not SDK or snapshot qualification. #![allow(clippy::disallowed_methods)] #[path = "support/raft_storage_fault.rs"] @@ -46,6 +46,16 @@ async fn rooted_cell_receipt_not_group_progress() -> Result<()> { runtime_fixture::cell_receipts(true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_group_fair_admission() -> Result<()> { + runtime_fixture::sql_pool_admission(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_group_fair_admission() -> Result<()> { + runtime_fixture::sql_pool_admission(true).await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 6222ea5ed..75e436878 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -508,3 +508,7 @@ mod sql; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await } + +pub(super) async fn sql_pool_admission(rooted: bool) -> Result<()> { + sql::pool_admission(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 4a97d703c..9207336fb 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -33,6 +33,41 @@ impl RuntimeCluster { let authority = self.runtimes[0] .execution_authority(self.cluster.birth.route().history, cell.clone()) .await?; + let options = self.cell_options(cell).await?; + self.cluster + .domain + .blocking(move || match options { + CellSqlOptions::Materialized(o) => CellPrivateExecutor::open(o, authority), + CellSqlOptions::Extents(o) => { + let workspace = o + .images + .create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(o.limits)?, + ) + .map_err(|e| ExecutionError::Storage(e.into()))?; + CellPrivateExecutor::open_rooted( + RootPrivateOptions { + workspace, + host: o.host, + sqlite_vfs: o.sqlite_vfs, + images: o.images, + query_services: o.query_services, + application_digest: o.application_digest, + limits: o.limits, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + authority, + ) + } + })? + .await? + .map_err(Into::into) + } + + async fn cell_options(&self, cell: &CellHistory) -> Result { let options = PrivateOptions { directory: self .cluster @@ -63,11 +98,7 @@ impl RuntimeCluster { .domain .blocking(move || -> Result<_> { if let Some(store) = extents { - let rooted = RootPrivateOptions { - workspace: store.create_workspace( - cell_image_history_digest(&authority.cell().version.history), - RootPrivateOptions::workspace_bounds(options.limits)?, - )?, + let rooted = CellSqlRootOptions { host: options.host, sqlite_vfs: None, images: store, @@ -84,16 +115,18 @@ impl RuntimeCluster { max_cached_extents: 1, }, }; - Ok(CellPrivateExecutor::open_rooted(rooted, authority)?) + Ok(CellSqlOptions::Extents(rooted)) } else { - Ok(CellPrivateExecutor::open(options, authority)?) + Ok(CellSqlOptions::Materialized(options)) } })? .await? } } -pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { +async fn enrolled_sql_cells( + rooted: bool, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let mut fixture = RuntimeCluster::new().await?; let route = fixture.cluster.birth.route().clone(); if rooted { @@ -110,7 +143,8 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { fixture.drive(None).await?; let a = history(&route, "sql-a")?; let b = history(&route, "sql-b")?; - for cell in [&a, &b] { + let c = history(&route, "sql-c")?; + for cell in [&a, &b, &c] { let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (0);")?; let cut = images::Cut::new(&[], &bytes, 0, true, false); let bytes = if rooted { @@ -144,6 +178,11 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { let proposed = fixture.runtimes[0].propose(route.history, bytes).await?; fixture.drive(Some(proposed.position.index)).await?; } + Ok((fixture, route, a, b)) +} + +pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; assert!(matches!( fixture.runtimes[1] .execution_authority(route.history, a.clone()) @@ -374,3 +413,10 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { drop(stale); fixture.close().await } + +#[path = "cell_group_sql_pool.rs"] +mod pool; + +pub(super) async fn pool_admission(rooted: bool) -> Result<()> { + pool::admission(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql_pool.rs b/crates/cellar/tests/support/cell_group_sql_pool.rs new file mode 100644 index 000000000..3297555e1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_pool.rs @@ -0,0 +1,295 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) async fn admission(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let c = history(&route, "sql-c")?; + let key = |cell: &CellHistory| -> u64 { + u64::from_le_bytes(cell_image_history_digest(cell)[..8].try_into().unwrap()) + }; + let workers = (2..=8) + .find(|n| key(&a) % n != key(&b) % n) + .context("distinct workers")? as usize; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 256 * 1024 * 1024, + }, + )?; + assert!(CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err()); + let oa = fixture.cell_options(&a).await?; + let ob = fixture.cell_options(&b).await?; + let oc = fixture.cell_options(&c).await?; + let foreign = CellGroupSqlPool::start( + &fixture.cluster.voters[1].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert!(matches!( + foreign + .stage(oa.clone(), ca, cell_attempt(90)?, true, |_| panic!( + "cross-engine authority" + )) + .await, + Err(ExecutionError::Invalid(_)) + )); + assert_eq!(foreign.usage()?.active_cells, 0); + foreign.close().await?; + drop(foreign); + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let options = oa.clone(); + let attempt = cell_attempt(91)?; + let pa = attempt.proposal; + let waiter = fixture.cluster.domain.spawn(async move { + caller + .stage(options, ca, attempt, true, move |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(b"a-pool".to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let retained = pool.usage()?; + assert_eq!(retained.active_cells, 1); + assert!(retained.retained_bytes > 0); + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.stage(oa.clone(), ca, cell_attempt(92)?, true, |_| panic!( + "cancelled waiter released cell ownership" + )) + .await, + Err(ExecutionError::Busy) + )); + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Prepared(pb) = tokio::time::timeout( + Duration::from_secs(5), + pool.stage(ob.clone(), cb, cell_attempt(93)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"b-pool".to_vec()) + }), + ) + .await?? + else { + anyhow::bail!("prepared B") + }; + assert_eq!(pool.usage()?.active_cells, 2); + assert!( + pool.usage()?.retained_bytes < 2 * retained.retained_bytes, + "completed capture retained its full transient allocation" + ); + let cc = fixture.runtimes[0] + .execution_authority(route.history, c.clone()) + .await?; + assert!(matches!( + pool.stage(oc, cc, cell_attempt(94)?, true, |_| panic!( + "group quota exceeded" + )) + .await, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded + )) + )); + // B completes through the same consensus group while A's physical SQL is held. + pool.mark_submitted(b.clone(), pb.proposal()).await?; + pool.mark_submitted(b.clone(), pb.proposal()).await?; + let proposal = fixture.runtimes[0] + .propose(route.history, pb.bytes().to_vec()) + .await?; + fixture.drive(Some(proposal.position.index)).await?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let rb = pool.finish(cb).await?; + assert_eq!(rb.receipt().receipt.result(), b"b-pool"); + assert_eq!(pool.usage()?.active_cells, 1); + drop(rb); + drop(pb); + assert_eq!(pool.usage()?.retained_bytes, retained.retained_bytes); + release.send(())?; + let pa = tokio::time::timeout(Duration::from_secs(5), pool.resume(a.clone(), pa)).await??; + pool.mark_submitted(a.clone(), pa.proposal()).await?; + let proposal = fixture.runtimes[0] + .propose(route.history, pa.bytes().to_vec()) + .await?; + fixture.drive(Some(proposal.position.index)).await?; + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let before = ca.cell().version.clone(); + let ra = pool.finish(ca).await?; + assert_eq!(ra.receipt().receipt.result(), b"a-pool"); + drop(ra); + drop(pa); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + let cb = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let duplicate = pool + .stage(ob, cb, cell_attempt(95)?, false, |_| { + panic!("receipt replayed callback") + }) + .await?; + let CellSqlOutcome::Receipt(receipt) = duplicate else { + anyhow::bail!("duplicate receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"b-pool"); + drop(receipt); + // Cancelling close's waiter must not detach the physical callback or its lock. + let ca = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let mut attempt = cell_attempt(96)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .stage(oa, ca, attempt, true, move |tx| { + tx.execute("UPDATE value SET n=n+100", [])?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(b"private-only".to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let closer = pool.clone(); + let mut closing = fixture + .cluster + .domain + .spawn(async move { closer.close().await })?; + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + closing.abort(); + assert!(closing.await.is_err()); + assert!(pool.usage()?.running_workers > 0); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.active_cells, 0); + assert_eq!(usage.retained_bytes, 0); + let after = fixture.runtimes[0] + .execution_authority(route.history, a) + .await?; + assert_eq!(after.cell().version, before); + drop(after); + drop(pool); + let replacement = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + let cb = fixture.runtimes[0] + .execution_authority(route.history, b) + .await?; + let options = fixture.cell_options(&cb.cell().version.history).await?; + let receipt = replacement + .stage(options, cb, cell_attempt(97)?, false, |_| { + panic!("restart receipt replayed callback") + }) + .await?; + assert!(matches!(receipt, CellSqlOutcome::Receipt(_))); + replacement.close().await?; + drop(replacement); + assert!( + CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + } + ) + .is_err(), + "retained response released its node admission owner" + ); + drop(receipt); + let replacement = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 128 * 1024 * 1024, + group_bytes: 128 * 1024 * 1024, + }, + )?; + replacement.close().await?; + drop(replacement); + fixture.close().await +} From 821ec0d17b0d0788a6de2dd3d1e13e248ff097d4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 19:43:34 -0700 Subject: [PATCH 051/568] docs: retain bounded SQL pool qualification and reconciliation handoff --- docs/design/multi-raft/cell-groups/README.md | 6 ++- .../multi-raft/cell-groups/baseline.json | 8 +-- docs/design/multi-raft/cell-groups/handoff.md | 54 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 50 +++++++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 6 ++- 5 files changed, 96 insertions(+), 28 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index d0b8db478..b64a40e5c 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -94,8 +94,10 @@ executor. CG-04 now retains creation identity with the selected image and prepares ordered enrollment and exact retries. Its physical group owner consumes the catalog startup permit and verified selected image/manifest closure, and obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. The per-cell SQL -producer and exact receipts now pass their partial regression gate. Fixed-pool -SQL ownership, fair admission and SDK serving integration remain required. Managed +producer and exact receipts now pass their partial regression gate, followed by +fixed-pool SQL ownership and admission at `51dff0e5`. Leadership/compaction +reconciliation, concurrent enrollment, coherent reads and registered SDK serving +integration remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index f08df121b..1aca6e5c3 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "f807b2f70e01e3cd8c741557889e1965fd14ec49", + "source_revision": "51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -43,8 +43,8 @@ }, "sql": { "path": "crates/cellar/durability/raft/host_sql/mod.rs", - "sha256": "afab717be888dc826ff558bca6898abeb7406a9c531baf36da1a644d2218a923", - "line": 802, + "sha256": "55484a37d1d4c00be0a1a83f27ec787cb3ce05aee53a4b9c419ccc044b865456", + "line": 864, "excerpt": "struct Cell {\n group: GroupLog,\n class: Arc,\n images: ImageOwner,\n snapshot: Option<(ApplyTicket, PreparedSnapshot, Permit)>,\n private: Option,\n active: Option,\n}\n" }, "runtime": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 5a244da27..aedd1d2ea 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,8 +15,8 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `f807b2f7`, qualifying the SQL producer after -the shared-node runtime at `aec4029e`; see the latest checkpoint in +The reviewed source baseline is `51dff0e5`, adding the fixed SQL pool after +the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; @@ -51,17 +51,26 @@ It separates a selected cell image/version from group progress and requires exac receipt settlement in both image modes. The partial gate passes at `f807b2f7`: 133 native tests and eight verification commands. It does not complete CG-04. -The next integration is per-cell SQL ownership and fair admission, concurrent -initialization, exact receipt settlement and coherent read serving. Preserve the -node worker's physical completion and cancellation boundaries. Do not treat the -shared scheduler's isolated tests as completion of the six-case CG-04 gate. +The fixed SQL pool now accepts cell-history jobs on the existing blocking workers. +Admission, pending proposals and retained outputs survive waiter cancellation; +node/group quotas and an exclusive engine lease bound their ownership. Extent +workspaces are created and closed on those workers. The partial gate passes at +`51dff0e5`: 184 selected Rust tests and ten verification commands. It does not +complete CG-04 or qualify the production dispatcher. + +The next integration is per-cell reconciliation across leadership changes and +compaction, then concurrent initialization, coherent reads and registered SDK +dispatch. Preview/cache lifecycle also remains required. Preserve the physical +completion and cancellation boundaries. These pool cases alone do not complete +the six-case CG-04 gate. The remaining coupling is concrete: `execution::CommandAuthority` binds the old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; -and `host_sql::Pool` keys execution owners by that scope. Adapt those owning -components to a cell history and exact manifest/image pin, with group authority -kept separately. Reuse the existing receipt transaction and callback machinery; -do not manufacture a synthetic cell-scoped group to satisfy the old interfaces. +and the old `host_sql::Cell` still keys its execution state by that scope. The +new pool path already keys by cell history and exact image pins; replace the +production dispatcher with that path while keeping group authority separate. +Reuse the existing receipt transaction and callback machinery; do not manufacture +a synthetic cell-scoped group to satisfy the old interfaces. A later mutation of another cell must not settle the first cell's pending output. The registered callback boundary also needs a real identity conversion: @@ -70,8 +79,8 @@ The registered callback boundary also needs a real identity conversion: to inbox and system-effect authentication. Carry logical `CellHistory` through this boundary and keep physical consensus proof separate. Do not enable effects with a fabricated group scope; preserve the CG-08-before-CG-07 dependency. Reuse -`host_sql::Pool`'s fixed execution-domain workers and joined shutdown while making -cell admission, queued ordering and unresolved proposals key on cell history. +the new `host_sql::Pool` cell-history path and joined shutdown; do not create a +second node worker pool or a task per enrolled cell. A cancelled caller cannot release physical callback ownership or allow a second producer for that cell. @@ -91,9 +100,12 @@ The SQL checkpoint includes `native_raft_execution` and `native_raft_rooted` as well as the shared-group execution suite. Both cell receipt cases include the final callback/owner-close extension. Preserve authority revocation, exact committed entry matching and duplicate callback suppression during integration. -Receipt reconciliation after compaction and preview/cache retention still need -the owning host. These regressions supplement, and do not replace, the full task -gate printed by `show CG-04`. Fair per-cell queues and registered SDK dispatch +Receipt reconciliation after leadership changes or compaction and preview/cache +retention still need the owning host. Reuse the old executor's current-term +applied-barrier rule when replacing it: a submitted producer cannot be discarded +from a sampled leader status, timeout or missing response. Resolve the original +command against the authoritative cell receipt before admitting another producer. These regressions supplement, and do not replace, the full task +gate printed by `show CG-04`. Registered SDK dispatch and strong-read serving remain absent. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -120,12 +132,12 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The current partial CG-04 evidence is retained at -`.cellar/cell-groups-evidence/CG-04-cell-sql/checkpoint.json`. Transfer that entire -directory, including copied binaries, failed and successful logs, with the prior -runtime/owner and DONE-task evidence. The SQL checkpoint binds 173 source inputs, -six executables, 133 selected Rust cases and eight verification commands. It -cannot stand in for the complete six-case CG-04 contract. Three RustFS-specific -cases remain explicitly unrun at this checkpoint. +`.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json`. Transfer that entire +directory, including copied binaries and intermediate/successful logs, with the +prior SQL-producer, runtime/owner and DONE-task evidence. The pool checkpoint +binds 178 source inputs, eight executables, 184 selected Rust cases and ten +verification commands. It cannot stand in for the complete six-case CG-04 +contract. Three RustFS-specific cases remain explicitly unrun at this checkpoint. ## Delivery sequence diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1fd547df8..78e18d034 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,56 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 fixed node SQL pool + +Qualified source: `51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1`. + +`CellGroupSqlPool` now uses the existing fixed blocking worker pool and fairness +queue. Requests have typed logical cell keys; the old host adapter remains +separate during the staged cutover. The node creates no worker per enrolled cell. +A cell's admission lease remains with queued work, its physical callback and its +unresolved proposal, including when a caller cancels. A lost stage response can +resume the retained proposal without running another callback. Exact settlement +releases that cell; unrelated group progress cannot do so. + +Node/group quotas reserve capture capacity before queueing. Once capture ends, +the reservation shrinks to the retained entry/result bound while keeping its +operation slot. Prepared-response clones and receipt delivery retain that charge. +A separate progress slot per worker permits settlement without needing ordinary +queue admission. Extent workspaces are created and destroyed on the admitted +blocking worker, so rejected requests cannot release their last physical workspace +on an async executor. + +The pool validates the exact engine behind execution authority and claims an +exclusive SQL lease on that engine. Its cleanup task owns every worker join; +cancelling a close waiter cannot release a blocked callback. The engine lease +also survives through retained response admission, preventing a replacement pool +from bypassing live output reservations. Queue/stage/publication timers use the +existing node performance registry. + +All 184 selected Rust tests pass: 168 native cases across seven suites, 11 SQL +queue/extent-worker unit tests and five budget tests. Both new pool cases use +actual three-voter replication and cover blocked/cancelled callbacks, independent +cell progress, group quota refusal, exact resume, duplicate receipts, engine +fencing, cancelled close and retained-response ownership. Four process helpers +are invoked by parent tests; three RustFS-specific cases remain explicitly unrun. + +All-target Clippy with warnings denied, formatting (1,794 Rust files), module +layout, dependency isolation, docs, reviewed source preflight and all 37 plan +validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json` binds 178 source inputs, +ten command logs and eight copied executables. Intermediate logs retain the +initial verification-export compilation error and direct-Tokio-select Clippy +failure, as well as passes preceding the final ownership refinements. Only the +final gate qualifies the committed source; no task status advances here. + +Registered SDK dispatch, concurrent enrollment, strong-read serving, preview/cache +lifecycle and receipt reconciliation after leadership changes or compaction +still need integration. +Closing SQL does not prove an ambiguous submitted command failed; the owning host +must reconcile against committed receipts before admitting a new producer. CG-04 +and the full hard-cutover goal remain incomplete. + ## CG-04 per-cell SQL producer Implementation source: `f3d48bdc`; qualified successor: diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 3ed0f5c93..c7659dfac 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -452,7 +452,11 @@ "crates/cellar/durability/raft/cell_groups/execution_authority.rs", "crates/cellar/durability/raft/execution/cell.rs", "crates/cellar/durability/raft/execution/command.rs", - "crates/cellar/tests/support/cell_group_sql.rs" + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/mod.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/admission.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/worker.rs", + "crates/cellar/tests/support/cell_group_sql_pool.rs" ], "test_target": "native_cell_groups_execution", "steps": [ From 8a0cf640ae70f3280f02e281f81b55a8823205fd Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 20:00:10 -0700 Subject: [PATCH 052/568] feat: reconcile cell SQL producers through applied term barriers --- crates/cellar/application/consensus/mod.rs | 8 +- crates/cellar/durability/raft/README.md | 20 +- .../raft/cell_groups/execution_authority.rs | 82 +++++ .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../durability/raft/cell_groups/owner.rs | 8 + .../raft/cell_groups/runtime/mod.rs | 19 ++ .../raft/cell_groups/runtime/worker.rs | 4 + .../cellar/durability/raft/execution/cell.rs | 68 +++- .../cellar/durability/raft/execution/mod.rs | 5 +- .../raft/host_sql/cell_groups/mod.rs | 40 ++- .../raft/host_sql/cell_groups/worker.rs | 37 ++- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 17 +- .../tests/support/cell_group_sql_reconcile.rs | 311 ++++++++++++++++++ 15 files changed, 613 insertions(+), 24 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_sql_reconcile.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index cdc46596d..bc2eaf11d 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -158,10 +158,10 @@ pub mod verification { }; pub use crate::raft::execution::{ query_registered, AppliedExecution, CellCommittedReceipt, CellPreparedCommand, - CellPrivateExecutor, CellStageOutcome, CommandAttempt, CommandAuthority, CommandRequest, - CommittedReceipt, ExecutionError, ExecutionLimits, ExecutionStatus, PreparedCommand, - PrivateExecutor, PrivateOptions, RegisteredQueryOutput, RootPrivateOptions, StageOutcome, - StoredReceipt, + CellPrivateExecutor, CellReconciliation, CellStageOutcome, CommandAttempt, + CommandAuthority, CommandRequest, CommittedReceipt, ExecutionError, ExecutionLimits, + ExecutionStatus, PreparedCommand, PrivateExecutor, PrivateOptions, RegisteredQueryOutput, + RootPrivateOptions, StageOutcome, StoredReceipt, }; pub use crate::raft::fleet_catalog::{ FleetBirthAuthorization, FleetBirthReservation, FleetCatalog, FleetCatalogError, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 18bfaadc5..8c727664b 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -67,13 +67,27 @@ pool until workers, the handle and retained outputs release their ownership. A l another callback. Admission checks the authority's exact engine owner. Each worker has one reserved progress slot, independent of its ordinary queue. -Submission marking is idempotent for the retained proposal. Settlement still -requires the exact committed receipt; group progress cannot release another -cell's lease. Pool close stops admission and its domain cleanup owner joins every +Submission marking is idempotent for the retained proposal. Exact-entry settlement +requires the committed receipt; group progress cannot release another cell's lease. Pool close stops admission and its domain cleanup owner joins every physical callback/worker even if a close waiter is cancelled. Closing a writer or pool does not establish a failed submitted outcome or authorize replay; the host must reconcile pending commands against committed state before admitting again. +`CellReconciliationAuthority` selects a verified cell pin from the durable +current-term applied group root, including on followers. It grants no write or +strong-read authority. The pool can reconcile its exact pending proposal against +the original command identity in that image even after a later cell mutation; +this lookup does not require retaining the original Raft entry. A matching receipt +settles the command. Absence can release the producer only after a strictly newer +term's applied barrier excludes its old proposal. Replicated expiry/conflict +refusals remain distinct from a known uncommitted outcome. Foreign engines, +changed routes/histories, stale cell pins and same-term absence refuse without +releasing the producer. Missing producers return uncertainty after a lost response. +Reconciliation uses the reserved worker progress slot and closes the disposable +writer before releasing cell admission; retained receipt outputs keep their byte +reservation. A retry still needs fresh leader authority and receipt admission. +Actual snapshot/compaction integration and production retry dispatch remain pending. + Registered SDK dispatch, concurrent initialization, coherent query serving and the production hard cutover remain required; the existing host still uses its prior single-cell owner. The new pool's callback entry point remains a verification diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index facdc4011..29e473485 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -25,6 +25,12 @@ pub struct CellExecutionAuthority { } impl CellExecutionAuthority { + pub(crate) fn accepts_reconciliation(&self, proof: &CellReconciliationAuthority) -> bool { + Arc::ptr_eq(&self.group.owner, &proof.group.owner) + && self.route == proof.route + && self.leaf.version.history == proof.leaf.version.history + } + pub(super) fn new( group: GroupLog, current: &PreparedGroupManifest, @@ -113,3 +119,79 @@ impl CellExecutionAuthority { Ok(()) } } + +/// A durable current-term applied cut, including on a follower. This permits +/// receipt inspection and retirement of older-term producers, never new SQL +/// execution, a linearizable query, or an S3 acknowledgement. +pub struct CellReconciliationAuthority { + group: GroupLog, + route: GroupRoute, + term: Term, + leaf: CellManifestLeaf, + image: CellImagePin, +} + +impl CellReconciliationAuthority { + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: &CellHistory, + ) -> Result { + let (leaf, image) = current.pin_cell(&cell.cell).ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell reconciliation history".into())); + } + let proof = Self { + group, + route: current.marker().route().clone(), + term: current.marker().position().term, + leaf: leaf.clone(), + image, + }; + proof.check()?; + Ok(proof) + } + + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.leaf + } + + pub fn image(&self) -> &CellImagePin { + &self.image + } + + pub fn check(&self) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + self.route.history.namespace, + &self.leaf.version.history.cell, + ); + if self.term.get() == 0 + || cache.hard_state.term != self.term.get() + || current.position().term != self.term + || current.position().index.get() > cache.hard_state.commit + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .is_none_or(|(leaf, _)| leaf != &self.leaf) + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 414427be9..83fbd5cdc 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -4,11 +4,11 @@ pub use super::host_sql::cell_groups::{ CellGroupSqlPool, CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, - CellSqlReceipt, CellSqlRootOptions, CellSqlUsage, + CellSqlReceipt, CellSqlReconciliation, CellSqlRootOptions, CellSqlUsage, }; mod execution_authority; -pub use execution_authority::CellExecutionAuthority; +pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; mod enrollment; mod enrollment_apply; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 435e3608a..c35cfab0b 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -278,6 +278,14 @@ impl CellGroupOwner { &self.current } + pub fn reconciliation_authority( + &self, + cell: &CellHistory, + ) -> Result { + self.live()?; + super::CellReconciliationAuthority::new(self.group.clone(), &self.current, cell) + } + pub fn status(&self) -> Result { self.live()?; let cache = self.group.read()?; diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 8521274be..e41de290b 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -85,6 +85,7 @@ enum Command { Attach(Box), Campaign, Execution(cellar_logic::cell_group::CellHistory), + Reconciliation(cellar_logic::cell_group::CellHistory), Step(Message), AdmitStep(Message), Propose(Vec), @@ -98,6 +99,7 @@ enum Response { Unit, Proposal(CellGroupProposal), Execution(Box), + Reconciliation(Box), Status(CellGroupStatus), } @@ -318,6 +320,23 @@ impl CellGroupRuntime { } } + pub async fn reconciliation_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Reconciliation(cell), bytes) + .await? + { + Response::Reconciliation(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol( + "reconciliation authority reply".into(), + )), + } + } + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { self.call(Some(history), Command::Campaign, 4096) .await diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index c66b13904..9a6a719eb 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -325,6 +325,10 @@ impl Worker { .owner .execution_authority(&cell) .map(|authority| Response::Execution(Box::new(authority))), + Command::Reconciliation(cell) => group + .owner + .reconciliation_authority(&cell) + .map(|authority| Response::Reconciliation(Box::new(authority))), Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 545293e27..72f7d6ae3 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -7,7 +7,9 @@ use super::*; use crate::application::consensus::group_protocol::{ CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, GroupLtx, GroupMutation, }; -use crate::raft::cell_groups::{cell_image_history_digest, CellExecutionAuthority}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellExecutionAuthority, CellReconciliationAuthority, +}; use crate::raft::completion::RuntimeError; use cellar_logic::cell_group::CellVersion; use std::sync::Arc; @@ -40,6 +42,18 @@ pub enum CellStageOutcome { Duplicate(Box), } +pub enum CellReconciliation { + /// The original request's receipt, from the inspected committed image. + /// Its position identifies that image, not necessarily the original entry. + Committed(Box), + /// A newer-term applied barrier excludes the old proposal and the original + /// request has no receipt. A retry must still pass fresh command admission. + NotCommitted, + /// A replicated receipt floor or conflicting identity prevents a retry. + /// Expiration does not establish whether the original command committed. + Refused(CommandFailure), +} + struct PendingCell { prepared: CellPreparedCommand, request: CommandRequest, @@ -282,6 +296,58 @@ impl CellPrivateExecutor { Ok(receipt) } + /// Resolve the original request without requiring the original log entry. + /// Missing receipts cannot retire an ambiguous same-term proposal. The + /// caller must close this disposable producer after successful resolution. + pub fn reconcile( + &mut self, + proposal: ProposalId, + authority: CellReconciliationAuthority, + ) -> Result { + let pending = self.pending.as_ref().ok_or(ExecutionError::Uncertain)?; + if pending.prepared.proposal != proposal + || !self.authority.accepts_reconciliation(&authority) + { + return Err(ExecutionError::Invalid("cell reconciliation identity")); + } + authority.check().map_err(authority_failure)?; + // Staging already established absence in this exact immutable base. + // In particular an enrollment image need not yet have receipt tables. + let receipt = if authority.cell().version == self.authority.cell().version { + Ok(None) + } else { + let database = self + .options + .images + .read_cell(&self.options, authority.image()) + .map_err(storage)?; + receipts::preflight_identity( + database.connection(), + &pending.request, + &self.options.limits, + ) + }; + let resolution = match receipt { + Ok(Some(receipt)) => CellReconciliation::Committed(Box::new(CellCommittedReceipt { + receipt, + version: authority.cell().version.clone(), + position: authority.cell().last_mutation, + })), + Ok(None) if authority.term() > self.authority.term() => { + CellReconciliation::NotCommitted + } + Err(ExecutionError::Refused( + failure @ (CommandFailure::CommandExpired | CommandFailure::CommandConflict), + )) if authority.term() > self.authority.term() => CellReconciliation::Refused(failure), + Err(error) => return Err(error), + Ok(None) => return Err(ExecutionError::Uncertain), + }; + authority.check().map_err(authority_failure)?; + self.pending = None; + self.poisoned = true; + Ok(resolution) + } + fn read_receipt( &self, authority: &CellExecutionAuthority, diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index ac33ffd82..090d47265 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -10,7 +10,10 @@ use super::receipts; mod cell; mod command; mod images; -pub use cell::{CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellStageOutcome}; +pub use cell::{ + CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellReconciliation, + CellStageOutcome, +}; mod rooted_working; use crate::application::consensus::{ command_request_digest, CommittedVersion, EnvelopeLimits, OperationContract, OperationKind, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index acef70105..0fb1866f2 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -6,11 +6,14 @@ mod admission; mod worker; use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; -use crate::raft::cell_groups::{cell_image_history_digest, CellExecutionAuthority}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellExecutionAuthority, CellReconciliationAuthority, +}; use crate::raft::completion::RuntimeError; use crate::raft::execution::{ - CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellStageOutcome, - CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, RootPrivateOptions, + CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellReconciliation, + CellStageOutcome, CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, + RootPrivateOptions, }; use crate::raft::runtime::Terminal; use crate::raft::storage::ReplicaLog; @@ -146,6 +149,13 @@ pub enum CellSqlOutcome { Prepared(CellSqlPrepared), Receipt(Box), Submitted, + Reconciled(CellSqlReconciliation), +} + +pub enum CellSqlReconciliation { + Committed(Box), + NotCommitted, + Refused(CommandFailure), } pub struct CellSqlUsage { @@ -404,6 +414,30 @@ impl CellGroupSqlPool { } } + /// Release a producer only after receipt resolution or a newer applied + /// term excludes its proposal. Cancellation leaves resolution on the worker; + /// a lost response must be reconciled against receipts, never inferred failed. + pub async fn reconcile( + &self, + proposal: ProposalId, + authority: CellReconciliationAuthority, + ) -> Result { + if !authority.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = authority.cell().version.history.clone(); + match self + .call( + cell, + worker::Action::Reconcile(proposal, Box::new(authority)), + ) + .await? + { + CellSqlOutcome::Reconciled(resolution) => Ok(resolution), + _ => Err(ExecutionError::Invalid("cell SQL reconciliation response")), + } + } + pub async fn close(&self) -> Result<(), ExecutionError> { self.inner.closing.store(true, Ordering::Release); self.inner.stop.notify_one(); diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index 4e51d423f..04360f1b0 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -20,6 +20,7 @@ pub(super) enum Action { Resume(ProposalId), Submit(ProposalId), Finish(Box), + Reconcile(ProposalId, Box), } impl Action { @@ -76,7 +77,9 @@ impl Cells { } = request; let operation = match &action { Action::Stage(_) => crate::raft::performance::Operation::SqlStage, - Action::Finish(_) => crate::raft::performance::Operation::SqlPublish, + Action::Finish(_) | Action::Reconcile(..) => { + crate::raft::performance::Operation::SqlPublish + } _ => crate::raft::performance::Operation::SqlOther, }; let timer = performance.timer(operation); @@ -167,10 +170,7 @@ impl Cells { } } Action::Resume(proposal) => { - let pending = self - .pending - .get(&cell) - .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + let pending = self.pending.get(&cell).ok_or(ExecutionError::Uncertain)?; if pending.prepared.proposal() != proposal { return Err(ExecutionError::Invalid("cell producer proposal")); } @@ -180,7 +180,7 @@ impl Cells { let pending = self .pending .get_mut(&cell) - .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + .ok_or(ExecutionError::Uncertain)?; if pending.prepared.proposal() != proposal { return Err(ExecutionError::Invalid("cell producer proposal")); } @@ -194,7 +194,7 @@ impl Cells { let pending = self .pending .get_mut(&cell) - .ok_or(ExecutionError::Invalid("unknown cell producer"))?; + .ok_or(ExecutionError::Uncertain)?; let receipt = pending.executor.finish(*authority)?; let pending = self .pending @@ -206,6 +206,29 @@ impl Cells { _bytes: pending.prepared._bytes, }))) } + Action::Reconcile(proposal, authority) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + let resolution = pending.executor.reconcile(proposal, *authority)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Invalid("cell producer changed"))?; + pending.executor.close()?; + let resolution = match resolution { + CellReconciliation::Committed(receipt) => { + CellSqlReconciliation::Committed(Box::new(CellSqlReceipt { + receipt: *receipt, + _bytes: pending.prepared._bytes, + })) + } + CellReconciliation::NotCommitted => CellSqlReconciliation::NotCommitted, + CellReconciliation::Refused(failure) => CellSqlReconciliation::Refused(failure), + }; + Ok(CellSqlOutcome::Reconciled(resolution)) + } } } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 9415c4c66..e6df23332 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -56,6 +56,16 @@ async fn rooted_shared_group_fair_admission() -> Result<()> { runtime_fixture::sql_pool_admission(true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_sql_reconciles_after_leadership_change() -> Result<()> { + runtime_fixture::reconcile_sql_producers(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_sql_reconciles_after_leadership_change() -> Result<()> { + runtime_fixture::reconcile_sql_producers(true).await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 75e436878..5f2781ff3 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -512,3 +512,7 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { pub(super) async fn sql_pool_admission(rooted: bool) -> Result<()> { sql::pool_admission(rooted).await } + +pub(super) async fn reconcile_sql_producers(rooted: bool) -> Result<()> { + sql::reconcile_producers(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 9207336fb..0af6bfd57 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -68,12 +68,16 @@ impl RuntimeCluster { } async fn cell_options(&self, cell: &CellHistory) -> Result { + self.cell_options_at(cell, 0).await + } + + async fn cell_options_at(&self, cell: &CellHistory, node: usize) -> Result { let options = PrivateOptions { directory: self .cluster ._root .path() - .join(format!("private-{}", cell.cell.as_str())), + .join(format!("private-{node}-{}", cell.cell.as_str())), host: cellar_ltx::LtxHost::new( || 10, |_| Ok(Duration::ZERO), @@ -85,7 +89,7 @@ impl RuntimeCluster { ) .with_filesystem(Arc::new(DirectFileSystem)), sqlite_vfs: None, - images: self.cluster.voters[0].images.clone(), + images: self.cluster.voters[node].images.clone(), query_services: cellar_ltx::generation::ReadServices { now_unix_millis: Arc::new(|| 10), fill_random: Arc::new(|bytes| bytes.fill(17)), @@ -93,7 +97,7 @@ impl RuntimeCluster { application_digest: [6; 32], limits: ExecutionLimits::default(), }; - let extents = self.cluster.voters[0].extents.clone(); + let extents = self.cluster.voters[node].extents.clone(); self.cluster .domain .blocking(move || -> Result<_> { @@ -420,3 +424,10 @@ mod pool; pub(super) async fn pool_admission(rooted: bool) -> Result<()> { pool::admission(rooted).await } + +#[path = "cell_group_sql_reconcile.rs"] +mod reconciliation; + +pub(super) async fn reconcile_producers(rooted: bool) -> Result<()> { + reconciliation::reconcile(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs new file mode 100644 index 000000000..ba279dd4c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -0,0 +1,311 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Result<()> { + let history = fixture.cluster.birth.route().history; + let term = fixture.runtimes[from].status(history).await?.term; + let mut message = Message::default(); + // The current leader's transfer directive triggers a real higher-term + // election. MsgTransferLeader itself is a local administration request and + // is deliberately rejected by the authenticated peer-message boundary. + message.set_msg_type(MessageType::MsgTimeoutNow); + message.from = from as u64 + 1; + message.to = to as u64 + 1; + message.term = term.get(); + fixture.runtimes[to].step(history, message).await?; + if from == 0 { + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[history], &mut held).await?; + if fixture.runtimes[to].status(history).await?.ready && held.contains_key(&history) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let a = super::history(fixture.cluster.birth.route(), "sql-a")?; + assert!( + matches!( + fixture.runtimes[0] + .reconciliation_authority(history, a) + .await, + Err(RuntimeError::NotReady) + ), + "a persisted newer term without its applied barrier cannot retire a producer" + ); + let work = held.remove(&history).context("held new-term barrier")?; + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(history, work, next) + .await?; + } + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let status = fixture.runtimes[to].status(history).await?; + if status.ready + && status.term > term + && fixture.applied[&history] + .iter() + .all(|index| *index >= status.applied.index) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? +} + +pub(super) async fn reconcile(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 2, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 3, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let mut prepared = Vec::new(); + for (cell, tag) in [(&a, 111), (&b, 112)] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(cell).await?, + authority, + cell_attempt(tag)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"reconciled".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("prepared") + }; + pool.mark_submitted(cell.clone(), command.proposal()) + .await?; + prepared.push(command); + } + // A's submission response is unknown. The same-term applied prefix is not + // enough to release it, even though its receipt is currently absent. + let same_term = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), same_term).await, + Err(ExecutionError::Uncertain) + )); + assert_eq!(pool.usage()?.active_cells, 2); + let stale_b = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + let submitted = fixture.runtimes[0] + .propose(route.history, prepared[1].bytes().to_vec()) + .await?; + fixture.drive(Some(submitted.position.index)).await?; + assert!( + stale_b.check().is_err(), + "an old cell pin cannot prove absence after commit" + ); + assert!(pool + .reconcile(prepared[1].proposal(), stale_b) + .await + .is_err()); + + // B committed; deliberately lose its exact-entry completion. Transfer real + // leadership so node 0 must reconcile while it is a follower. + transfer(&mut fixture, 0, 1).await?; + assert!(fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await + .is_err()); + // The new leader advances B again while the old node retains B's first + // unresolved producer. Reconciliation must find the original receipt in a + // later image, rather than demand that image equal the old private result. + let successor = CellGroupSqlPool::start( + &fixture.cluster.voters[1].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 256 * 1024 * 1024, + group_bytes: 256 * 1024 * 1024, + }, + )?; + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let mut attempt = cell_attempt(115)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, 2)?; + let CellSqlOutcome::Prepared(next) = successor + .stage( + fixture.cell_options_at(&b, 1).await?, + authority, + attempt, + true, + |tx| { + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"later-b".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("successor command") + }; + successor.mark_submitted(b.clone(), next.proposal()).await?; + let accepted = fixture.runtimes[1] + .propose(route.history, next.bytes().to_vec()) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + if fixture.applied[&route.history] + .iter() + .all(|index| *index >= accepted.position.index) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let later = successor.finish(authority).await?; + assert_eq!(later.receipt().receipt.result(), b"later-b"); + drop(later); + drop(next); + successor.close().await?; + drop(successor); + let foreign = fixture.runtimes[1] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), foreign).await, + Err(ExecutionError::Invalid(_)) + )); + let wrong = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), wrong).await, + Err(ExecutionError::Invalid(_)) + )); + assert_eq!(pool.usage()?.active_cells, 2); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[0].proposal(), proof).await?, + CellSqlReconciliation::NotCommitted + )); + assert_eq!(pool.usage()?.active_cells, 1); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + let CellSqlReconciliation::Committed(receipt) = + pool.reconcile(prepared[1].proposal(), proof).await? + else { + anyhow::bail!("committed receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"reconciled"); + assert_eq!(receipt.receipt().version.history, b); + assert_eq!(receipt.receipt().version.sequence.get(), 3); + assert_eq!(receipt.receipt().position, accepted.position); + assert_eq!(pool.usage()?.active_cells, 0); + assert!( + pool.usage()?.retained_bytes > 0, + "responses retain admission" + ); + // Losing a reconciliation response cannot be reported as definite failure. + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(prepared[1].proposal(), proof).await, + Err(ExecutionError::Uncertain) + )); + assert!(matches!( + pool.resume(b.clone(), prepared[1].proposal()).await, + Err(ExecutionError::Uncertain) + )); + drop(receipt); + drop(prepared); + assert_eq!(pool.usage()?.retained_bytes, 0); + + transfer(&mut fixture, 1, 0).await?; + // B retries through the committed receipt without invoking the callback. + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = pool + .stage( + fixture.cell_options(&b).await?, + authority, + cell_attempt(113)?, + false, + |_| panic!("duplicate callback"), + ) + .await? + else { + anyhow::bail!("retry receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"reconciled"); + drop(receipt); + // A can reopen from its committed base and execute the request exactly once. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(&a).await?, + authority, + cell_attempt(114)?, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 0, "abandoned private writes must not be reused"); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"retry-a".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("retry A") + }; + pool.mark_submitted(a.clone(), command.proposal()).await?; + let submitted = fixture.runtimes[0] + .propose(route.history, command.bytes().to_vec()) + .await?; + fixture.drive(Some(submitted.position.index)).await?; + // A positive same-term receipt resolves without requiring a term change. + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + let CellSqlReconciliation::Committed(receipt) = + pool.reconcile(command.proposal(), proof).await? + else { + anyhow::bail!("same-term committed receipt") + }; + assert_eq!(receipt.receipt().receipt.result(), b"retry-a"); + drop(receipt); + drop(command); + assert_eq!(pool.usage()?.retained_bytes, 0); + pool.close().await?; + drop(pool); + fixture.close().await +} From 902e2dac82ce0589f40c508646869e485c1bc0ec Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 20:07:48 -0700 Subject: [PATCH 053/568] docs: retain cell reconciliation qualification and remaining gates --- docs/design/multi-raft/cell-groups/README.md | 7 +-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/design.md | 13 ++++++ docs/design/multi-raft/cell-groups/handoff.md | 39 ++++++++++++---- .../design/multi-raft/cell-groups/progress.md | 45 +++++++++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- 6 files changed, 96 insertions(+), 15 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index b64a40e5c..230919077 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -95,9 +95,10 @@ prepares ordered enrollment and exact retries. Its physical group owner consumes the catalog startup permit and verified selected image/manifest closure, and obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. The per-cell SQL producer and exact receipts now pass their partial regression gate, followed by -fixed-pool SQL ownership and admission at `51dff0e5`. Leadership/compaction -reconciliation, concurrent enrollment, coherent reads and registered SDK serving -integration remain required. Managed +fixed-pool SQL ownership and admission at `51dff0e5`. Applied-barrier receipt +reconciliation is implemented at `8a0cf640`. Production retry and compaction +integration, concurrent enrollment, coherent reads and registered SDK serving +remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 1aca6e5c3..4a10e367d 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1", + "source_revision": "8a0cf640ae70f3280f02e281f81b55a8823205fd", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/design.md b/docs/design/multi-raft/cell-groups/design.md index 06f85c061..b5e10adfc 100644 --- a/docs/design/multi-raft/cell-groups/design.md +++ b/docs/design/multi-raft/cell-groups/design.md @@ -162,6 +162,19 @@ may finish out of order but cannot advance a hole in the durable applied prefix. Retain exact term/index proof for each pending output; an unrelated cell's progress cannot settle it. Storage sync failure quarantines the owning engine as today. +An unresolved SQL producer can reconcile the original request against the exact +cell's committed receipt after leadership changes or log compaction. Select the +cell pin from a verified durable current-term applied root; follower inspection +is permitted but grants no write or strong-read authority. A matching receipt +can resolve the request from a later image. Its returned position describes the +inspected image, not the original proposal. Absence retires the old producer only +after a strictly newer term's applied barrier excludes that proposal; a persisted +term, timeout or sampled leader role is insufficient. Receipt expiry remains an +explicit refusal with unknown historical outcome. Reject changed route/history, +foreign engine and stale pin proofs. Close the private writer before releasing +cell admission; retained responses keep their byte reservations. Lost resolution +responses require receipt lookup, never an inferred failure or blind callback replay. + ## Reads, caches and effects Linearizable reads obtain a fresh current-term group ReadIndex, wait for contiguous diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index aedd1d2ea..ec72476d1 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,8 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `51dff0e5`, adding the fixed SQL pool after +The reviewed source baseline is `8a0cf640`, adding applied-barrier receipt +reconciliation after the fixed SQL pool at `51dff0e5`, following the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, @@ -58,11 +59,22 @@ workspaces are created and closed on those workers. The partial gate passes at `51dff0e5`: 184 selected Rust tests and ten verification commands. It does not complete CG-04 or qualify the production dispatcher. -The next integration is per-cell reconciliation across leadership changes and -compaction, then concurrent initialization, coherent reads and registered SDK -dispatch. Preview/cache lifecycle also remains required. Preserve the physical -completion and cancellation boundaries. These pool cases alone do not complete -the six-case CG-04 gate. +Per-cell reconciliation now runs on the pool's reserved progress path. A verified +current-term applied cell pin can be inspected on a follower. Matching original +receipts resolve from later images; absence requires a strictly newer applied +term before closing the old producer. Foreign engines, wrong proposals, stale +cell pins and an unapplied new-term barrier refuse without releasing ownership. +Missing producers report uncertainty. Production retry dispatch and real +snapshot/compaction integration remain required. + +Next, separate reconciliation lookup from new-client capacity admission: +`receipts::preflight_identity` currently checks capacity for an absent new client, +so saturation can keep an older absent producer unresolved unnecessarily. Use a +read-only identity/receipt lookup while preserving all command admission limits, +and add saturated-client, expiry-floor and conflicting-identity cases. Then +integrate concurrent initialization, coherent reads and registered SDK dispatch. +Preview/cache lifecycle also remains required. Preserve physical completion and +cancellation boundaries. These cases alone do not complete the six-case CG-04 gate. The remaining coupling is concrete: `execution::CommandAuthority` binds the old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; @@ -100,8 +112,9 @@ The SQL checkpoint includes `native_raft_execution` and `native_raft_rooted` as well as the shared-group execution suite. Both cell receipt cases include the final callback/owner-close extension. Preserve authority revocation, exact committed entry matching and duplicate callback suppression during integration. -Receipt reconciliation after leadership changes or compaction and preview/cache -retention still need the owning host. Reuse the old executor's current-term +The pool now provides applied-barrier receipt reconciliation; production retry +dispatch, real compaction integration and preview/cache retention still need the +owning host. Reuse the old executor's current-term applied-barrier rule when replacing it: a submitted producer cannot be discarded from a sampled leader status, timeout or missing response. Resolve the original command against the authoritative cell receipt before admitting another producer. These regressions supplement, and do not replace, the full task @@ -131,7 +144,15 @@ The three DONE records require their retained evidence directories listed in another machine. Missing evidence must refuse; do not replace it with a pass flag or reset completed tasks merely to bypass validation. -The current partial CG-04 evidence is retained at +The latest partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-reconciliation/checkpoint.json` at `8a0cf640`: +181 selected Rust cases, nine successful commands, 179 source inputs and eight +copied executables. Transfer its entire directory along with the prior records. +It qualifies the new pool reconciliation boundary in both image modes, not actual +compaction, production retry dispatch or the full task gate. See [progress](progress.md) +for tested behavior and remaining receipt lookup/refusal work. + +The prior partial CG-04 evidence is retained at `.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json`. Transfer that entire directory, including copied binaries and intermediate/successful logs, with the prior SQL-producer, runtime/owner and DONE-task evidence. The pool checkpoint diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 78e18d034..92e50822d 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,51 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 applied-barrier receipt reconciliation + +Implementation source: `8a0cf640ae70f3280f02e281f81b55a8823205fd`. + +The existing SQL pool now reconciles the exact pending proposal against the +original request identity in a selected immutable cell image. The new +`CellReconciliationAuthority` is issued from a verified durable current-term +applied group root, including on followers. It grants no write, strong-read or +S3 authority. A matching receipt may resolve from a later cell image without +requiring the original log entry. Its position identifies the inspected image. +A missing receipt releases the old producer only behind a strictly newer applied +term. Expiry/conflict remain distinct refusals; missing producers report uncertainty. + +The worker validates engine, route, cell history and proposal identity, rechecks +pin freshness after receipt inspection, and closes the disposable writer before +releasing cell admission. Reconciliation uses reserved progress capacity; retained +receipt outputs keep their existing byte reservation. + +Both image-mode tests use real three-voter elections and SQLite/LTX. They hold +the new-term apply on the old leader and reject premature reconciliation; advance +B again on the new leader while its first producer remains on the old follower; +then recover B's original receipt and retire A's absent older-term proposal. +They also cover same-term absence refusal, stale and foreign proofs, mismatched +proposals, unknown producer uncertainty, retained-output budgets, duplicate +callback suppression and reopening A from its committed base. + +All 181 selected Rust tests pass: 170 native cases across seven suites and 11 +SQL queue/extent-worker unit tests. All-target Clippy with warnings denied, +formatting (1,795 Rust files), layout, dependency isolation, docs, reviewed source +preflight and all 37 plan-validator tests pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-reconciliation/checkpoint.json` binds 179 +source inputs, nine successful command logs and eight copied executables. +The initial fixture failure is retained: `MsgTransferLeader` is a local +administrative request, correctly rejected by the peer boundary. The corrected +fixture delivers the current leader's `MsgTimeoutNow` directive and drives the +real election, persistence and apply. Four process helpers run through parent +tests; three RustFS cases remain explicitly unrun. This remains partial CG-04 +implementation; no task status advances. + +Next, split read-only receipt lookup from new-client capacity admission and +qualify saturated identity tables, expiry floors and conflicting identities. +Then integrate real snapshot/compaction, production retry dispatch, concurrent +initialization, coherent query serving and preview/cache lifecycle. No full +CG-04, RustFS, performance or hard-cutover qualification is claimed. + ## CG-04 fixed node SQL pool Qualified source: `51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1`. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index c7659dfac..849e797ba 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -456,7 +456,8 @@ "crates/cellar/durability/raft/host_sql/cell_groups/mod.rs", "crates/cellar/durability/raft/host_sql/cell_groups/admission.rs", "crates/cellar/durability/raft/host_sql/cell_groups/worker.rs", - "crates/cellar/tests/support/cell_group_sql_pool.rs" + "crates/cellar/tests/support/cell_group_sql_pool.rs", + "crates/cellar/tests/support/cell_group_sql_reconcile.rs" ], "test_target": "native_cell_groups_execution", "steps": [ From 91c3ecfc020e0d42738e4a6395045eb8a44c51f9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 20:14:48 -0700 Subject: [PATCH 054/568] fix: reconcile receipts independently of client admission capacity --- crates/cellar/durability/raft/README.md | 5 +- .../cellar/durability/raft/execution/cell.rs | 2 +- crates/cellar/durability/raft/receipts.rs | 54 +++- .../cellar/durability/raft/receipts_tests.rs | 73 +++++ .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../tests/support/cell_group_sql_reconcile.rs | 285 ++++++++++++++++-- 8 files changed, 400 insertions(+), 37 deletions(-) diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 8c727664b..2a66af1a4 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -77,7 +77,10 @@ must reconcile pending commands against committed state before admitting again. current-term applied group root, including on followers. It grants no write or strong-read authority. The pool can reconcile its exact pending proposal against the original command identity in that image even after a later cell mutation; -this lookup does not require retaining the original Raft entry. A matching receipt +this lookup does not require retaining the original Raft entry. It validates +identity, sequence floors and result allocation bounds without admitting a new +client or receipt; a full identity table cannot obscure absence. New commands +still enforce both client and retained-receipt ceilings. A matching receipt settles the command. Absence can release the producer only after a strictly newer term's applied barrier excludes its old proposal. Replicated expiry/conflict refusals remain distinct from a known uncommitted outcome. Foreign engines, diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 72f7d6ae3..9a90f8f97 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -321,7 +321,7 @@ impl CellPrivateExecutor { .images .read_cell(&self.options, authority.image()) .map_err(storage)?; - receipts::preflight_identity( + receipts::lookup_identity( database.connection(), &pending.request, &self.options.limits, diff --git a/crates/cellar/durability/raft/receipts.rs b/crates/cellar/durability/raft/receipts.rs index cacb68e00..86376f150 100644 --- a/crates/cellar/durability/raft/receipts.rs +++ b/crates/cellar/durability/raft/receipts.rs @@ -90,6 +90,46 @@ pub(super) fn preflight_identity( request: &CommandRequest, limits: &ExecutionLimits, ) -> Result, ExecutionError> { + match inspect_identity(connection, request, limits)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { new_client } => { + if new_client { + let count: usize = connection + .query_row("SELECT count(*) FROM __sys_raft_clients", [], |r| r.get(0)) + .map_err(storage)?; + if count >= limits.max_client_identities { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + } + Ok(None) + } + } +} + +/// Inspect immutable committed state without admitting a new client or receipt. +/// Capacity cannot obscure absence; identity, expiry, sequence and result-size +/// checks still apply. The caller supplies the consensus proof for using absence. +pub(super) fn lookup_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, +) -> Result, ExecutionError> { + match inspect_identity(connection, request, limits)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { .. } => Ok(None), + } +} + +enum Identity { + Retained(StoredReceipt), + Vacant { new_client: bool }, +} + +fn inspect_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, +) -> Result { let state = client(connection, request.id.client)?; let (last, floor) = state.unwrap_or((0, 0)); let sequence = request.id.sequence(); @@ -133,7 +173,7 @@ pub(super) fn preflight_identity( 1 => Durability::S3, _ => return Err(corrupt()), }; - return Ok(Some(StoredReceipt { + return Ok(Identity::Retained(StoredReceipt { id: request.id, operation: row.0, contract_digest: contract, @@ -149,15 +189,9 @@ pub(super) fn preflight_identity( next, })); } - if state.is_none() { - let count: usize = connection - .query_row("SELECT count(*) FROM __sys_raft_clients", [], |r| r.get(0)) - .map_err(storage)?; - if count >= limits.max_client_identities { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - } - Ok(None) + Ok(Identity::Vacant { + new_client: state.is_none(), + }) } pub(super) fn record( diff --git a/crates/cellar/durability/raft/receipts_tests.rs b/crates/cellar/durability/raft/receipts_tests.rs index 3e56e4f22..f5e91fa4c 100644 --- a/crates/cellar/durability/raft/receipts_tests.rs +++ b/crates/cellar/durability/raft/receipts_tests.rs @@ -62,3 +62,76 @@ fn maximum_retry_deadline_is_never_expired() -> anyhow::Result<()> { assert_eq!(client(&connection, command.id.client)?, Some((1, 0))); Ok(()) } + +#[test] +fn committed_lookup_does_not_admit_new_client_at_capacity() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + install(&connection)?; + let limits = ExecutionLimits { + max_client_identities: 1, + max_retained_receipts: 1, + ..Default::default() + }; + let retained = request(1, 1000); + record(&connection, &retained, vec![7])?; + let mut absent = request(1, 1000); + absent.id = CommandId::new(ClientId::new([9; 16])?, 1)?; + assert!(lookup_identity(&connection, &absent, &limits)?.is_none()); + assert!(matches!( + preflight_identity(&connection, &absent, &limits), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!( + lookup_identity(&connection, &retained, &limits)? + .unwrap() + .result(), + &[7] + ); + // Existing client admission remains independent from the new-client bound; + // the separate retained-receipt ceiling still refuses a new command. + assert!(preflight_identity(&connection, &request(2, 1000), &limits)?.is_none()); + assert!(matches!( + preflight(&connection, &request(2, 1000), &limits), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(retained_count(&connection)?, 1); + assert!(client(&connection, absent.id.client)?.is_none()); + Ok(()) +} + +#[test] +fn committed_lookup_preserves_identity_floor_and_allocation_checks() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + install(&connection)?; + let limits = ExecutionLimits::default(); + let original = request(1, 1000); + record(&connection, &original, vec![7, 8])?; + let mut conflict = request(1, 1000); + conflict.request_digest = [99; 32]; + assert!(matches!( + lookup_identity(&connection, &conflict, &limits), + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + assert!(matches!( + lookup_identity(&connection, &request(3, 1000), &limits), + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 2 + })) + )); + let bounded = ExecutionLimits { + max_result_bytes: 1, + ..limits + }; + assert!(matches!( + lookup_identity(&connection, &original, &bounded), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + prune(&connection, original.id.client, 1, 1001)?; + assert!(matches!( + lookup_identity(&connection, &original, &limits), + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + assert!(lookup_identity(&connection, &request(2, 2000), &limits)?.is_none()); + assert_eq!(client(&connection, original.id.client)?, Some((1, 1))); + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index e6df23332..c7df17534 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -66,6 +66,16 @@ async fn rooted_cell_sql_reconciles_after_leadership_change() -> Result<()> { runtime_fixture::reconcile_sql_producers(true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_sql_reconciles_expired_and_conflicting_receipts() -> Result<()> { + runtime_fixture::reconcile_sql_refusals(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_sql_reconciles_expired_and_conflicting_receipts() -> Result<()> { + runtime_fixture::reconcile_sql_refusals(true).await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 5f2781ff3..8bb7d6a1f 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -516,3 +516,7 @@ pub(super) async fn sql_pool_admission(rooted: bool) -> Result<()> { pub(super) async fn reconcile_sql_producers(rooted: bool) -> Result<()> { sql::reconcile_producers(rooted).await } + +pub(super) async fn reconcile_sql_refusals(rooted: bool) -> Result<()> { + sql::reconcile_refusals(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 0af6bfd57..0c9919390 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -431,3 +431,7 @@ mod reconciliation; pub(super) async fn reconcile_producers(rooted: bool) -> Result<()> { reconciliation::reconcile(rooted).await } + +pub(super) async fn reconcile_refusals(rooted: bool) -> Result<()> { + reconciliation::refusals(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs index ba279dd4c..fa458416e 100644 --- a/crates/cellar/tests/support/cell_group_sql_reconcile.rs +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -61,6 +61,50 @@ async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Resul .await? } +fn execution_limits(options: &mut CellSqlOptions) -> &mut ExecutionLimits { + match options { + CellSqlOptions::Materialized(o) => &mut o.limits, + CellSqlOptions::Extents(o) => &mut o.limits, + } +} + +async fn apply_through(fixture: &mut RuntimeCluster, index: LogIndex) -> Result<()> { + let group = fixture.cluster.birth.route().history; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + if fixture.applied[&group] + .iter() + .all(|applied| *applied >= index) + { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await? +} + +async fn commit_on( + fixture: &mut RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + prepared: CellSqlPrepared, +) -> Result { + let group = fixture.cluster.birth.route().history; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = fixture.runtimes[node] + .propose(group, prepared.bytes().to_vec()) + .await?; + apply_through(fixture, accepted.position.index).await?; + let authority = fixture.runtimes[node] + .execution_authority(group, cell.clone()) + .await?; + Ok(pool.finish(authority).await?) +} + pub(super) async fn reconcile(rooted: bool) -> Result<()> { let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; let pool = CellGroupSqlPool::start( @@ -79,17 +123,13 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { let authority = fixture.runtimes[0] .execution_authority(route.history, cell.clone()) .await?; + let mut options = fixture.cell_options(cell).await?; + execution_limits(&mut options).max_client_identities = 1; let CellSqlOutcome::Prepared(command) = pool - .stage( - fixture.cell_options(cell).await?, - authority, - cell_attempt(tag)?, - true, - |tx| { - tx.execute("UPDATE value SET n=n+1", [])?; - Ok(b"reconciled".to_vec()) - }, - ) + .stage(options, authority, cell_attempt(tag)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"reconciled".to_vec()) + }) .await? else { anyhow::bail!("prepared") @@ -169,19 +209,7 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { let accepted = fixture.runtimes[1] .propose(route.history, next.bytes().to_vec()) .await?; - tokio::time::timeout(Duration::from_secs(20), async { - loop { - fixture.pump(&[], &mut BTreeMap::new()).await?; - if fixture.applied[&route.history] - .iter() - .all(|index| *index >= accepted.position.index) - { - return Ok::<_, anyhow::Error>(()); - } - tokio::task::yield_now().await; - } - }) - .await??; + apply_through(&mut fixture, accepted.position.index).await?; let authority = fixture.runtimes[1] .execution_authority(route.history, b.clone()) .await?; @@ -189,6 +217,29 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { assert_eq!(later.receipt().receipt.result(), b"later-b"); drop(later); drop(next); + // Fill A's client identity capacity with a different committed client. + // Its changed image forces receipt lookup, not the unchanged-base shortcut. + let authority = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let mut attempt = cell_attempt(116)?; + attempt.request.id = CommandId::new(ClientId::new([99; 16])?, 1)?; + let CellSqlOutcome::Prepared(next) = successor + .stage( + fixture.cell_options_at(&a, 1).await?, + authority, + attempt, + true, + |tx| { + tx.execute("UPDATE value SET n=n+5", [])?; + Ok(b"other-client".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("other client command") + }; + drop(commit_on(&mut fixture, &successor, 1, &a, next).await?); successor.close().await?; drop(successor); let foreign = fixture.runtimes[1] @@ -266,7 +317,22 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { }; assert_eq!(receipt.receipt().receipt.result(), b"reconciled"); drop(receipt); - // A can reopen from its committed base and execute the request exactly once. + // Read-only absence did not relax fresh command admission at capacity. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let mut full = fixture.cell_options(&a).await?; + execution_limits(&mut full).max_client_identities = 1; + assert!(matches!( + pool.stage(full, authority, cell_attempt(117)?, true, |_| panic!( + "new client at capacity" + )) + .await, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(pool.usage()?.active_cells, 0); + // With sufficient capacity, reopen from the committed image, not the old + // private write. The other client's committed value must remain visible. let authority = fixture.runtimes[0] .execution_authority(route.history, a.clone()) .await?; @@ -278,7 +344,7 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { true, |tx| { let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; - assert_eq!(value, 0, "abandoned private writes must not be reused"); + assert_eq!(value, 5, "abandoned private writes must not be reused"); tx.execute("UPDATE value SET n=n+1", [])?; Ok(b"retry-a".to_vec()) }, @@ -309,3 +375,172 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { drop(pool); fixture.close().await } + +pub(super) async fn refusals(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let limits = CellSqlLimits { + workers: 2, + queue_per_worker: 2, + max_pending_cells: 3, + group_pending_cells: 3, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits)?; + let mut pending = Vec::new(); + for (cell, tag) in [(&a, 121), (&b, 122)] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(cell).await?, + authority, + cell_attempt(tag)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"original".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("original command") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + pending.push(prepared); + } + // B commits but its response is lost; A's original proposal never commits. + let accepted = fixture.runtimes[0] + .propose(route.history, pending[1].bytes().to_vec()) + .await?; + apply_through(&mut fixture, accepted.position.index).await?; + transfer(&mut fixture, 0, 1).await?; + let successor = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits)?; + let authority = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let mut conflict = cell_attempt(123)?; + conflict.request.request_digest = [88; 32]; + let CellSqlOutcome::Prepared(prepared) = successor + .stage( + fixture.cell_options_at(&a, 1).await?, + authority, + conflict, + true, + |tx| { + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"different-request".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("conflicting identity winner") + }; + drop(commit_on(&mut fixture, &successor, 1, &a, prepared).await?); + // Preserve the latest receipt while replicated maintenance expires B's + // first receipt. At command 3 the two-row ceiling triggers bounded pruning. + for sequence in 2..=3 { + let authority = fixture.runtimes[1] + .execution_authority(route.history, b.clone()) + .await?; + let mut options = fixture.cell_options_at(&b, 1).await?; + execution_limits(&mut options).max_retained_receipts = 2; + let mut attempt = cell_attempt(123 + sequence as u8)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + if sequence == 3 { + attempt.inputs.now_unix_millis = 2000; + attempt.request.retry_until_ms = 3000; + } + let CellSqlOutcome::Prepared(prepared) = successor + .stage(options, authority, attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"later".to_vec()) + }) + .await? + else { + anyhow::bail!("maintenance command") + }; + drop(commit_on(&mut fixture, &successor, 1, &b, prepared).await?); + } + successor.close().await?; + drop(successor); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert!(matches!( + pool.reconcile(pending[0].proposal(), proof).await?, + CellSqlReconciliation::Refused(CommandFailure::CommandConflict) + )); + let proof = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.reconcile(pending[1].proposal(), proof).await?, + CellSqlReconciliation::Refused(CommandFailure::CommandExpired) + )); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(pool.usage()?.retained_bytes > 0); + drop(pending); + assert_eq!(pool.usage()?.retained_bytes, 0); + transfer(&mut fixture, 1, 0).await?; + for cell in [&a, &b] { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let error = pool + .stage( + fixture.cell_options(cell).await?, + authority, + cell_attempt(128)?, + true, + |_| panic!("refused receipt cannot replay the callback"), + ) + .await; + if cell == &a { + assert!(matches!( + error, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + } else { + assert!(matches!( + error, + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + } + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let mut next = cell_attempt(129)?; + next.request.id = CommandId::new(ClientId::new([81; 16])?, 4)?; + next.inputs.now_unix_millis = 2000; + next.request.retry_until_ms = 3000; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(&b).await?, + authority, + next, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 3); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"after-expiry".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("next command after refused retry") + }; + let receipt = commit_on(&mut fixture, &pool, 0, &b, prepared).await?; + assert_eq!(receipt.receipt().version.sequence.get(), 5); + assert_eq!(receipt.receipt().receipt.result(), b"after-expiry"); + drop(receipt); + pool.close().await?; + drop(pool); + fixture.close().await +} From fa406313324d654f3e46cbd6a323e28ddfe89b30 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 20:23:55 -0700 Subject: [PATCH 055/568] docs: qualify receipt lookup and outline shared enrollment integration --- docs/design/multi-raft/cell-groups/README.md | 7 ++-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/design.md | 3 ++ docs/design/multi-raft/cell-groups/handoff.md | 42 ++++++++++++++----- .../design/multi-raft/cell-groups/progress.md | 36 ++++++++++++++++ 5 files changed, 76 insertions(+), 16 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 230919077..2f2d8e6c3 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -96,9 +96,10 @@ the catalog startup permit and verified selected image/manifest closure, and obtains current-term quorum before protocol admission. The shared node scheduler now multiplexes these physical owners. The per-cell SQL producer and exact receipts now pass their partial regression gate, followed by fixed-pool SQL ownership and admission at `51dff0e5`. Applied-barrier receipt -reconciliation is implemented at `8a0cf640`. Production retry and compaction -integration, concurrent enrollment, coherent reads and registered SDK serving -remain required. Managed +reconciliation is implemented at `8a0cf640`, followed by read-only receipt lookup +and saturated-capacity/expiry/conflict cases at `91c3ecfc`. Production retry and +compaction integration, concurrent enrollment, coherent reads and registered SDK +serving remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 4a10e367d..33385b363 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "8a0cf640ae70f3280f02e281f81b55a8823205fd", + "source_revision": "91c3ecfc020e0d42738e4a6395045eb8a44c51f9", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/design.md b/docs/design/multi-raft/cell-groups/design.md index b5e10adfc..e44e4cce8 100644 --- a/docs/design/multi-raft/cell-groups/design.md +++ b/docs/design/multi-raft/cell-groups/design.md @@ -174,6 +174,9 @@ explicit refusal with unknown historical outcome. Reject changed route/history, foreign engine and stale pin proofs. Close the private writer before releasing cell admission; retained responses keep their byte reservations. Lost resolution responses require receipt lookup, never an inferred failure or blind callback replay. +Receipt inspection preserves identity/floor/sequence and allocation checks but does +not admit a new client or receipt. Full admission tables cannot obscure absence; +subsequent new commands still obey their client and receipt ceilings. ## Reads, caches and effects diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index ec72476d1..32bd2c16d 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,8 +15,9 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `8a0cf640`, adding applied-barrier receipt -reconciliation after the fixed SQL pool at `51dff0e5`, following +The reviewed source baseline is `91c3ecfc`, separating committed receipt lookup +from admission after applied-barrier reconciliation at `8a0cf640` and the fixed +SQL pool at `51dff0e5`, following the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`; see the latest checkpoint in [progress](progress.md) for qualification and remaining work. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, @@ -67,14 +68,26 @@ cell pins and an unapplied new-term barrier refuse without releasing ownership. Missing producers report uncertainty. Production retry dispatch and real snapshot/compaction integration remain required. -Next, separate reconciliation lookup from new-client capacity admission: -`receipts::preflight_identity` currently checks capacity for an absent new client, -so saturation can keep an older absent producer unresolved unnecessarily. Use a -read-only identity/receipt lookup while preserving all command admission limits, -and add saturated-client, expiry-floor and conflicting-identity cases. Then -integrate concurrent initialization, coherent reads and registered SDK dispatch. -Preview/cache lifecycle also remains required. Preserve physical completion and -cancellation boundaries. These cases alone do not complete the six-case CG-04 gate. +Receipt lookup now uses `receipts::lookup_identity`, preserving identity, floor, +sequence and allocation checks independently from new-client admission. Command +admission retains both ceilings. Native fixtures now cover saturated identity +tables, replicated expiry floors and conflicting identities in both image modes, +including callback suppression and later successful command admission. + +Next integrate concurrent initialization through the existing bounded SQL pool +and replicated enrollment decisions, then coherent reads and registered SDK +dispatch. Real compaction and production retry integration, plus preview/cache +lifecycle, remain required. Preserve physical completion and cancellation +boundaries. These cases alone do not complete the six-case CG-04 gate. + +For enrollment, `ManifestTree::prepare_enrollment` already preserves creation +identity at ordered apply; current native fixtures construct initial SQLite images +manually. Add actual-leader enrollment admission and initialization jobs on the +existing SQL workers. Reserve the logical cell name before choosing a new history, +so competing incarnations cannot bypass single-producer admission. Reuse current +node/group budgets and joined callback ownership. An existing committed enrollment +must return its retained creation record without rerunning initialization or +replacing the current image; do not manufacture a legacy `GroupScope` adapter. The remaining coupling is concrete: `execution::CommandAuthority` binds the old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; @@ -145,12 +158,19 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The latest partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-receipt-lookup/checkpoint.json` at `91c3ecfc`: +187 selected Rust cases, ten successful commands, 181 source inputs and eight +copied executables. Transfer its entire directory with the earlier checkpoints. +It qualifies saturated-capacity lookup, replicated expiry/conflict and existing +reconciliation paths in both image modes; it does not complete CG-04. + +The prior reconciliation record is `.cellar/cell-groups-evidence/CG-04-reconciliation/checkpoint.json` at `8a0cf640`: 181 selected Rust cases, nine successful commands, 179 source inputs and eight copied executables. Transfer its entire directory along with the prior records. It qualifies the new pool reconciliation boundary in both image modes, not actual compaction, production retry dispatch or the full task gate. See [progress](progress.md) -for tested behavior and remaining receipt lookup/refusal work. +for the historical tested boundary and subsequent lookup/refusal qualification. The prior partial CG-04 evidence is retained at `.cellar/cell-groups-evidence/CG-04-sql-pool/checkpoint.json`. Transfer that entire diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 92e50822d..17d73b26a 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,42 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 receipt lookup independent from admission + +Qualified source: `91c3ecfc020e0d42738e4a6395045eb8a44c51f9`. + +Reconciliation now uses a read-only identity lookup that preserves sequence/floor, +request identity and result-allocation checks independently from capacity for new +clients or retained receipts. The command path still enforces both ceilings and +uses the same identity inspection; no extra client lookup is added to normal +admission. A full identity table can no longer leave an absent older-term +producer unresolved merely because it cannot admit a new client. + +Both native image-mode fixtures fill the selected cell's identity capacity with +another committed client after leadership changes, then reconcile the original +absent request. The changed cell image forces actual receipt lookup. A subsequent +new-client command still refuses at the original limit; increasing its explicit +capacity allows a fresh command from the committed base. Separate cases replicate +bounded receipt maintenance through the real group and return CommandExpired for +the pruned original receipt, and CommandConflict for a different request that won +the same client sequence. They prove callback suppression, producer admission +release, retained-output accounting and a valid next command after the refusals. +Private SQLite tests preserve capacity admission, sequence gaps, floors and result +allocation bounds. + +All 187 selected Rust cases pass: 172 native cases across seven suites, 11 SQL +queue/extent-worker unit tests and four receipt unit tests. All-target Clippy +with warnings denied, formatting (1,795 Rust files), layout, dependency isolation, +docs, source preflight and all 37 plan-validator tests pass. The checkpoint at +`.cellar/cell-groups-evidence/CG-04-receipt-lookup/checkpoint.json` binds 181 source +inputs, ten successful command logs and eight copied executables. The preliminary +four-case native pass is retained separately. Four process helpers execute through +parent tests; three RustFS cases remain explicitly unrun. No task status advances. + +This remains partial CG-04 implementation. Concurrent initialization, coherent +reads, registered dispatch, production retry and actual compaction integration, +and preview/cache lifecycle remain required before its complete gate. + ## CG-04 applied-barrier receipt reconciliation Implementation source: `8a0cf640ae70f3280f02e281f81b55a8823205fd`. From 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 20:45:26 -0700 Subject: [PATCH 056/568] feat: initialize and enroll cells through bounded shared SQL workers --- crates/cellar/application/consensus/mod.rs | 11 +- crates/cellar/durability/raft/README.md | 27 +- .../raft/cell_groups/enrollment_authority.rs | 135 ++++++ .../cellar/durability/raft/cell_groups/mod.rs | 8 +- .../durability/raft/cell_groups/owner.rs | 34 ++ .../raft/cell_groups/runtime/mod.rs | 58 +++ .../raft/cell_groups/runtime/worker.rs | 12 + .../cellar/durability/raft/execution/cell.rs | 16 +- .../durability/raft/execution/enrollment.rs | 234 ++++++++++ .../durability/raft/execution/images.rs | 20 + .../cellar/durability/raft/execution/mod.rs | 3 + .../raft/host_sql/cell_groups/admission.rs | 17 +- .../raft/host_sql/cell_groups/mod.rs | 176 +++++-- .../raft/host_sql/cell_groups/worker.rs | 153 +++++- .../tests/native_cell_groups_execution.rs | 20 + .../tests/support/cell_group_runtime.rs | 8 + crates/cellar/tests/support/cell_group_sql.rs | 11 + .../support/cell_group_sql_enrollment.rs | 439 ++++++++++++++++++ .../tests/support/cell_group_sql_reconcile.rs | 8 +- 19 files changed, 1333 insertions(+), 57 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/enrollment_authority.rs create mode 100644 crates/cellar/durability/raft/execution/enrollment.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_enrollment.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index bc2eaf11d..0b62fa8f4 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -157,11 +157,12 @@ pub mod verification { RuntimeEventKind, SnapshotCompletion, }; pub use crate::raft::execution::{ - query_registered, AppliedExecution, CellCommittedReceipt, CellPreparedCommand, - CellPrivateExecutor, CellReconciliation, CellStageOutcome, CommandAttempt, - CommandAuthority, CommandRequest, CommittedReceipt, ExecutionError, ExecutionLimits, - ExecutionStatus, PreparedCommand, PrivateExecutor, PrivateOptions, RegisteredQueryOutput, - RootPrivateOptions, StageOutcome, StoredReceipt, + query_registered, AppliedExecution, CellCommittedReceipt, CellEnrollmentAttempt, + CellEnrollmentResolution, CellPreparedCommand, CellPrivateEnrollment, CellPrivateExecutor, + CellReconciliation, CellStageOutcome, CommandAttempt, CommandAuthority, CommandRequest, + CommittedReceipt, ExecutionError, ExecutionLimits, ExecutionStatus, PreparedCommand, + PrivateExecutor, PrivateOptions, RegisteredQueryOutput, RootPrivateOptions, StageOutcome, + StoredReceipt, }; pub use crate::raft::fleet_catalog::{ FleetBirthAuthorization, FleetBirthReservation, FleetCatalog, FleetCatalogError, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 2a66af1a4..75d7d096e 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -91,7 +91,32 @@ writer before releasing cell admission; retained receipt outputs keep their byte reservation. A retry still needs fresh leader authority and receipt admission. Actual snapshot/compaction integration and production retry dispatch remain pending. -Registered SDK dispatch, concurrent initialization, coherent query serving and +Enrollment now uses `CellEnrollmentAuthority`, issued by the actual leader after +its current-term committed prefix is applied. `initialize` runs one SQLite +initializer on the existing SQL workers, captures the full initial LTX image and +proposes a cell-addressed enrollment. Followers apply those bytes without running +the initializer. Admission keys the logical cluster/namespace/cell name, so another +incarnation cannot bypass an in-flight producer. Ordinary commands use the same +name reservation. Queued and running initialization retain it through cancellation. + +Prepared SQL output carries its original leader term. Submit it through +`propose_at_term`: the actual RawNode owner checks that fence atomically with +proposal admission. Merely sampling leader status before queueing is insufficient, +including when the original node later regains leadership. The unfenced `propose` +method remains a low-level protocol verification boundary. + +A committed creation retry returns the immutable enrollment record without opening +SQLite or rerunning the callback, even after later cell mutations. An enrollment +state selected from a durable current-term applied root also permits resolution +on a follower: an exact creation record proves commitment; absence/conflict can +retire an older producer only after a strictly newer applied term. Resolution +closes the writer before releasing name admission, while retained responses keep +their byte charge. Unknown producers remain uncertain. Pool shutdown joins an +entered initializer even if its waiter and close waiter were both cancelled. +As with commands, shutdown is not proof that a submitted enrollment failed. + +Registered SDK creation/command dispatch, catalog routing integration, configured +group capacity policy, coherent query serving and the production hard cutover remain required; the existing host still uses its prior single-cell owner. The new pool's callback entry point remains a verification boundary. Transport authentication and actual image preparation are also caller diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs new file mode 100644 index 000000000..7a7b86314 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -0,0 +1,135 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Enrollment observations bind logical names before a cell image exists. + +use super::{EnrollmentRecord, PreparedGroupManifest}; +use crate::application::consensus::cell_routing_key; +use crate::raft::{cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{cell_group::*, consensus::Term}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +/// A verified current-term applied enrollment observation, including on a +/// follower. It permits resolution of old producers, never new initialization. +pub struct CellEnrollmentState { + group: GroupLog, + route: GroupRoute, + cell: CellHistory, + term: Term, + record: Option, +} + +impl CellEnrollmentState { + pub(super) fn new( + group: GroupLog, + current: &PreparedGroupManifest, + cell: CellHistory, + ) -> Result { + let route = current.marker().route(); + if !route.history.same_namespace(&cell) + || !route + .range + .contains(&cell_routing_key(cell.namespace, &cell.cell)) + { + return Err(RuntimeError::Invalid("enrollment cell route".into())); + } + let state = Self { + record: current + .pin_cell(&cell.cell) + .map(|(leaf, _)| leaf.enrollment.clone()), + group, + route: route.clone(), + cell, + term: current.marker().position().term, + }; + state.check()?; + Ok(state) + } + + pub fn cell(&self) -> &CellHistory { + &self.cell + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn record(&self) -> Option<&EnrollmentRecord> { + self.record.as_ref() + } + + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, owner) + } + + pub(crate) fn same_binding(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.group.owner, &other.group.owner) + && self.route == other.route + && self.cell == other.cell + } + + pub fn check(&self) -> Result<(), RuntimeError> { + let cache = self.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key(self.cell.namespace, &self.cell.cell); + if self.term.get() == 0 + || cache.hard_state.term != self.term.get() + || current.position().term != self.term + || current.position().index.get() > cache.hard_state.commit + || current.route() != &self.route + || !cache.membership.permits_group(self.group.identity()) + || cache + .manifest_pins + .as_ref() + .ok_or(RuntimeError::NotReady)? + .get(&key) + .map(|(leaf, _)| &leaf.enrollment) + != self.record.as_ref() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} + +/// Issued by the actual leader only after its current-term committed prefix is +/// applied. A node SQL admission lease must additionally serialize the name. +pub struct CellEnrollmentAuthority { + state: CellEnrollmentState, + leader_term: Arc, +} + +impl CellEnrollmentAuthority { + pub(super) fn new( + state: CellEnrollmentState, + leader_term: Arc, + ) -> Result { + let authority = Self { state, leader_term }; + authority.check()?; + Ok(authority) + } + + pub fn state(&self) -> &CellEnrollmentState { + &self.state + } + + pub fn check(&self) -> Result<(), RuntimeError> { + if self.leader_term.load(Ordering::Acquire) != self.state.term.get() { + return Err(RuntimeError::NotReady); + } + self.state.check()?; + if self.leader_term.load(Ordering::Acquire) != self.state.term.get() { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 83fbd5cdc..8c9bd8d06 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -3,13 +3,17 @@ //! Complete group-state storage. A synced page is data, not Raft authority. pub use super::host_sql::cell_groups::{ - CellGroupSqlPool, CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, - CellSqlReceipt, CellSqlReconciliation, CellSqlRootOptions, CellSqlUsage, + CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, CellSqlLimits, + CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlReceipt, CellSqlReconciliation, + CellSqlRootOptions, CellSqlUsage, }; mod execution_authority; pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; +mod enrollment_authority; +pub use enrollment_authority::{CellEnrollmentAuthority, CellEnrollmentState}; + mod enrollment; mod enrollment_apply; mod image_pin; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index c35cfab0b..0b3f9adfc 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -278,6 +278,25 @@ impl CellGroupOwner { &self.current } + pub fn enrollment_state( + &self, + cell: CellHistory, + ) -> Result { + self.live()?; + super::CellEnrollmentState::new(self.group.clone(), &self.current, cell) + } + + pub fn enrollment_authority( + &self, + cell: CellHistory, + ) -> Result { + let status = self.status()?; + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + super::CellEnrollmentAuthority::new(self.enrollment_state(cell)?, self.leader_term.clone()) + } + pub fn reconciliation_authority( &self, cell: &CellHistory, @@ -344,6 +363,7 @@ impl CellGroupOwner { /// Protocol admission only. The shared cell executor must serialize each /// cell's preparation, reserve physical work and retain ambiguous outcomes. + /// SQL producers use `propose_at_term` with their captured preparation term. pub fn propose(&mut self, data: Vec) -> Result { if !self.status()?.ready { return Err(RuntimeError::NotReady); @@ -384,6 +404,20 @@ impl CellGroupOwner { }) } + /// Check the SQL preparation fence on the actual RawNode owner in the same + /// turn as proposal admission. A sampled status before queueing is not enough. + pub fn propose_at_term( + &mut self, + term: Term, + data: Vec, + ) -> Result { + self.live()?; + if term.get() != self.node.raft.term { + return Err(RuntimeError::StaleCompletion); + } + self.propose(data) + } + pub fn read_index(&mut self, context: Vec) -> Result<(), RuntimeError> { if !self.status()?.ready { return Err(RuntimeError::NotReady); diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index e41de290b..02c5fdd76 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -85,10 +85,13 @@ enum Command { Attach(Box), Campaign, Execution(cellar_logic::cell_group::CellHistory), + Enrollment(cellar_logic::cell_group::CellHistory), + EnrollmentState(cellar_logic::cell_group::CellHistory), Reconciliation(cellar_logic::cell_group::CellHistory), Step(Message), AdmitStep(Message), Propose(Vec), + ProposeAtTerm(cellar_logic::consensus::Term, Vec), ReadIndex(Vec), Apply(CellGroupTicket, Box), Status, @@ -99,6 +102,8 @@ enum Response { Unit, Proposal(CellGroupProposal), Execution(Box), + Enrollment(Box), + EnrollmentState(Box), Reconciliation(Box), Status(CellGroupStatus), } @@ -337,6 +342,36 @@ impl CellGroupRuntime { } } + pub async fn enrollment_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::Enrollment(cell), bytes) + .await? + { + Response::Enrollment(authority) => Ok(*authority), + _ => Err(RuntimeError::Protocol("enrollment authority reply".into())), + } + } + + pub async fn enrollment_state( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + ) -> Result { + let bytes = cell.cell.as_str().len() + 4096; + match self + .call(Some(history), Command::EnrollmentState(cell), bytes) + .await? + { + Response::EnrollmentState(state) => Ok(*state), + _ => Err(RuntimeError::Protocol("enrollment state reply".into())), + } + } + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { self.call(Some(history), Command::Campaign, 4096) .await @@ -371,6 +406,29 @@ impl CellGroupRuntime { } } + pub async fn propose_at_term( + &self, + history: GroupHistory, + term: cellar_logic::consensus::Term, + data: Vec, + ) -> Result { + let bytes = data.len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + match self + .call( + Some(history), + Command::ProposeAtTerm(term, data), + bytes + 4096, + ) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("fenced proposal reply".into())), + } + } + /// Queue an authenticated peer frame without waiting for protocol work. /// Node/group admission remains owned until execution or discard. Success /// is neither a Raft acknowledgement nor persistence/application proof. diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 9a6a719eb..7e6fa4853 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -325,6 +325,14 @@ impl Worker { .owner .execution_authority(&cell) .map(|authority| Response::Execution(Box::new(authority))), + Command::Enrollment(cell) => group + .owner + .enrollment_authority(cell) + .map(|authority| Response::Enrollment(Box::new(authority))), + Command::EnrollmentState(cell) => group + .owner + .enrollment_state(cell) + .map(|state| Response::EnrollmentState(Box::new(state))), Command::Reconciliation(cell) => group .owner .reconciliation_authority(&cell) @@ -338,6 +346,10 @@ impl Worker { Command::Propose(data) => { group.owner.propose(data).map(Response::Proposal) } + Command::ProposeAtTerm(term, data) => group + .owner + .propose_at_term(term, data) + .map(Response::Proposal), Command::ReadIndex(context) => { group.owner.read_index(context).map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 9a90f8f97..16bdf0b0a 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -18,9 +18,22 @@ use std::sync::Arc; pub struct CellPreparedCommand { bytes: Arc>, proposal: ProposalId, + term: Term, } impl CellPreparedCommand { + pub(super) fn new(bytes: Vec, proposal: ProposalId, term: Term) -> Self { + Self { + bytes: Arc::new(bytes), + proposal, + term, + } + } + + pub fn term(&self) -> Term { + self.term + } + pub fn bytes(&self) -> &[u8] { &self.bytes } @@ -238,6 +251,7 @@ impl CellPrivateExecutor { let prepared = CellPreparedCommand { bytes: Arc::new(bytes), proposal: attempt.proposal, + term: self.authority.term(), }; self.pending = Some(PendingCell { prepared: prepared.clone(), @@ -387,7 +401,7 @@ impl CellPrivateExecutor { } } -fn authority_failure(error: RuntimeError) -> ExecutionError { +pub(crate) fn authority_failure(error: RuntimeError) -> ExecutionError { match error { RuntimeError::NotReady | RuntimeError::NotLeader | RuntimeError::Closed => { ExecutionError::Refused(CommandFailure::Unavailable) diff --git a/crates/cellar/durability/raft/execution/enrollment.rs b/crates/cellar/durability/raft/execution/enrollment.rs new file mode 100644 index 000000000..6b1ac07e0 --- /dev/null +++ b/crates/cellar/durability/raft/execution/enrollment.rs @@ -0,0 +1,234 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Disposable SQLite initialization under a physical group's leader fence. + +use super::cell::authority_failure; +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, GroupLtx, +}; +use crate::raft::cell_groups::{ + cell_image_history_digest, CellEnrollmentAuthority, CellEnrollmentState, EnrollmentRecord, +}; + +pub struct CellEnrollmentAttempt { + pub creation: ProposalId, + pub preview: ImageId, + pub inputs: CommandInputs, +} + +pub enum CellEnrollmentResolution { + Committed(Box), + NotCommitted, + Conflict, +} + +struct Pending { + record: EnrollmentRecord, + _preview: Preview, + submitted: bool, +} + +pub struct CellPrivateEnrollment { + options: Options, + authority: CellEnrollmentAuthority, + working: Option, + pending: Option, + staged: bool, + lock: PrivateLock, +} + +impl CellPrivateEnrollment { + pub fn open( + options: PrivateOptions, + authority: CellEnrollmentAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + pub fn open_rooted( + options: RootPrivateOptions, + authority: CellEnrollmentAuthority, + ) -> Result { + Self::open_image(options.into(), authority) + } + + fn open_image( + options: Options, + authority: CellEnrollmentAuthority, + ) -> Result { + options.limits.validate()?; + options.images.validate_workspace(options.limits)?; + authority.check().map_err(authority_failure)?; + if authority.state().record().is_some() + || authority.state().route().schema_digest != options.application_digest + { + return Err(ExecutionError::Invalid("initialization binding")); + } + let filesystem = options.host.filesystem(); + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &options.directory) + .map_err(storage)?; + let lock = PrivateLock { + _lock: filesystem + .lock_file(&options.directory.join("working.lock")) + .map_err(storage)?, + _workspace: options.images.workspace().cloned(), + }; + let working = WorkingImage::open( + &options, + None, + cell_image_history_digest(authority.state().cell()), + ) + .map_err(storage)?; + authority.check().map_err(authority_failure)?; + Ok(Self { + options, + authority, + working: Some(working), + pending: None, + staged: false, + lock, + }) + } + + /// Run once on the admitted blocking worker. Followers consume the encoded + /// initial LTX image and never execute the initializer. + pub fn prepare( + &mut self, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()>, + ) -> Result { + if self.staged { + return Err(ExecutionError::Busy); + } + self.authority.check().map_err(authority_failure)?; + self.staged = true; + let working = self + .working + .as_mut() + .ok_or(ExecutionError::RebuildRequired)?; + working.begin_mutation(); + working + .database() + .transaction(attempt.inputs, |_, tx| initialize(tx)) + .map_err(private_failure)?; + let update = working + .capture_initial(&self.options) + .map_err(private_failure)?; + let state = self.authority.state(); + let route = state.route(); + let entry = CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: attempt.creation, + }, + command: GroupCommand::Enroll { + cell: state.cell().clone(), + image: update.result, + files: update + .files + .iter() + .map(|file| GroupLtx { + min_txid: file.min_txid, + max_txid: file.max_txid, + page_count: file.page_count, + decoded_bytes: file.decoded_bytes, + bytes: &file.bytes, + }) + .collect(), + }, + }; + let bytes = entry + .encode( + route, + GroupEntryLimits { + envelope: self.options.limits.envelope, + ..Default::default() + }, + ) + .map_err(ExecutionError::Protocol)?; + let record = EnrollmentRecord::new( + state.cell().clone(), + attempt.creation, + route.schema_digest, + update.result.into(), + ) + .map_err(storage)?; + let preview = self + .options + .images + .preview_initial( + &self.options, + attempt.preview, + &update, + cell_image_history_digest(state.cell()), + ) + .map_err(storage)?; + self.authority.check().map_err(authority_failure)?; + self.pending = Some(Pending { + record, + _preview: preview, + submitted: false, + }); + Ok(CellPreparedCommand::new( + bytes, + attempt.creation, + state.term(), + )) + } + + pub fn mark_submitted(&mut self, creation: ProposalId) -> Result<(), ExecutionError> { + self.authority.check().map_err(authority_failure)?; + let pending = self.pending.as_mut().ok_or(ExecutionError::Uncertain)?; + if pending.record.creation() != creation || pending.submitted { + return Err(ExecutionError::Invalid("enrollment submission")); + } + pending.submitted = true; + Ok(()) + } + + /// The immutable creation record survives later cell mutations and original + /// log compaction. Absence/conflict retires an old producer only behind a + /// strictly newer applied term. No callback or initial-image replay occurs. + pub fn resolve( + &mut self, + creation: ProposalId, + state: CellEnrollmentState, + ) -> Result { + let pending = self.pending.as_ref().ok_or(ExecutionError::Uncertain)?; + if pending.record.creation() != creation || !self.authority.state().same_binding(&state) { + return Err(ExecutionError::Invalid("enrollment resolution binding")); + } + state.check().map_err(authority_failure)?; + let outcome = match state.record() { + Some(record) if record == &pending.record => { + CellEnrollmentResolution::Committed(Box::new(record.clone())) + } + _ if state.term() <= self.authority.state().term() => { + return Err(ExecutionError::Uncertain) + } + Some(_) => CellEnrollmentResolution::Conflict, + None => CellEnrollmentResolution::NotCommitted, + }; + self.pending = None; + Ok(outcome) + } + + pub fn close(mut self) -> Result<(), ExecutionError> { + let store = self.options.images.root_store().cloned(); + if let Some(working) = self.working.take() { + working.close().map_err(storage)?; + } + drop(self.options); + drop(self.lock); + if let Some(store) = store { + if store.usage().map_err(storage)?.quarantined { + return Err(storage(cellar_ltx::internal::StoreError::Quarantined)); + } + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/execution/images.rs b/crates/cellar/durability/raft/execution/images.rs index 8a311e44e..6f9143f2f 100644 --- a/crates/cellar/durability/raft/execution/images.rs +++ b/crates/cellar/durability/raft/execution/images.rs @@ -298,6 +298,16 @@ impl WorkingImage { } } + pub(super) fn capture_initial( + &mut self, + options: &Options, + ) -> anyhow::Result { + match self { + Self::Materialized(working) => working.capture_pin(&options.materialized()?, None), + Self::Rooted(working) => working.capture_pin(options, None), + } + } + pub fn database(&self) -> &crate::storage::CommandDatabase { match self { Self::Materialized(working) => &working.database, @@ -478,6 +488,16 @@ impl Images { self.preview_payload(options, Some(base), id, update.as_ref(), history) } + pub(super) fn preview_initial( + &self, + options: &Options, + id: ImageId, + update: &crate::raft::working::CapturedUpdate, + history: [u8; 32], + ) -> anyhow::Result { + self.preview_payload(options, None, id, update.as_ref(), history) + } + fn preview_payload( &self, options: &Options, diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index 090d47265..fef0a22f3 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -8,7 +8,10 @@ use super::cache::{DurableApply, GroupLog}; use super::completion::GroupStatus; use super::receipts; mod cell; +pub(crate) use cell::authority_failure; mod command; +mod enrollment; +pub use enrollment::{CellEnrollmentAttempt, CellEnrollmentResolution, CellPrivateEnrollment}; mod images; pub use cell::{ CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellReconciliation, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs index 82fcc4f76..549ad63b1 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs @@ -1,9 +1,15 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; +use cellar_logic::{ + cell_group::CellId, + consensus::{ClusterId, NamespaceId}, +}; use std::collections::{BTreeMap, BTreeSet}; use std::sync::{Mutex, Weak}; +type LogicalCell = (ClusterId, NamespaceId, CellId); + pub(super) struct Admission { lease: Arc, node: Arc, @@ -13,7 +19,7 @@ pub(super) struct Admission { #[derive(Default)] struct State { - cells: BTreeSet, + cells: BTreeSet, groups: BTreeMap>, } @@ -36,7 +42,7 @@ impl Bytes { pub(super) struct CellLease { owner: Arc, - cell: CellHistory, + cell: LogicalCell, } impl Drop for CellLease { @@ -64,7 +70,8 @@ impl Admission { bytes: usize, ) -> Result<(CellLease, Arc), ExecutionError> { let mut state = self.state.lock().map_err(|_| unavailable())?; - if state.cells.contains(cell) { + let key = (cell.cluster, cell.namespace, cell.cell.clone()); + if state.cells.contains(&key) { return Err(ExecutionError::Busy); } let node = self.node.acquire(bytes).ok_or_else(overloaded)?; @@ -79,11 +86,11 @@ impl Admission { }); let group_permit = group_budget.acquire(bytes).ok_or_else(overloaded)?; state.groups.insert(group, Arc::downgrade(&group_budget)); - state.cells.insert(cell.clone()); + state.cells.insert(key.clone()); Ok(( CellLease { owner: self.clone(), - cell: cell.clone(), + cell: key, }, Arc::new(Bytes { _lease: self.lease.clone(), diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 0fb1866f2..a07481a8f 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -7,13 +7,14 @@ mod worker; use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; use crate::raft::cell_groups::{ - cell_image_history_digest, CellExecutionAuthority, CellReconciliationAuthority, + cell_image_history_digest, CellEnrollmentAuthority, CellEnrollmentState, + CellExecutionAuthority, CellReconciliationAuthority, EnrollmentRecord, }; use crate::raft::completion::RuntimeError; use crate::raft::execution::{ - CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellReconciliation, - CellStageOutcome, CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, - RootPrivateOptions, + CellCommittedReceipt, CellEnrollmentAttempt, CellEnrollmentResolution, CellPreparedCommand, + CellPrivateEnrollment, CellPrivateExecutor, CellReconciliation, CellStageOutcome, + CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, RootPrivateOptions, }; use crate::raft::runtime::Terminal; use crate::raft::storage::ReplicaLog; @@ -83,40 +84,80 @@ impl CellSqlOptions { } } + fn application_digest(&self) -> [u8; 32] { + match self { + Self::Materialized(o) => o.application_digest, + Self::Extents(o) => o.application_digest, + } + } + + fn open_enrollment( + self, + authority: CellEnrollmentAuthority, + ) -> Result { + match self { + Self::Materialized(o) => CellPrivateEnrollment::open(o, authority), + Self::Extents(o) => { + CellPrivateEnrollment::open_rooted(o.prepare(authority.state().cell())?, authority) + } + } + } + fn open( self, authority: CellExecutionAuthority, ) -> Result { match self { Self::Materialized(o) => CellPrivateExecutor::open(o, authority), - Self::Extents(o) => { - let workspace = o - .images - .create_workspace( - cell_image_history_digest(&authority.cell().version.history), - RootPrivateOptions::workspace_bounds(o.limits)?, - ) - .map_err(crate::raft::execution::storage)?; - CellPrivateExecutor::open_rooted( - RootPrivateOptions { - workspace, - host: o.host, - sqlite_vfs: o.sqlite_vfs, - images: o.images, - query_services: o.query_services, - application_digest: o.application_digest, - limits: o.limits, - cache_directory: o.cache_directory, - cache_pool: o.cache_pool, - cache: o.cache, - }, - authority, - ) - } + Self::Extents(o) => CellPrivateExecutor::open_rooted( + o.prepare(&authority.cell().version.history)?, + authority, + ), } } } +impl CellSqlRootOptions { + fn prepare(self, cell: &CellHistory) -> Result { + let workspace = self + .images + .create_workspace( + cell_image_history_digest(cell), + RootPrivateOptions::workspace_bounds(self.limits)?, + ) + .map_err(crate::raft::execution::storage)?; + Ok(RootPrivateOptions { + workspace, + host: self.host, + sqlite_vfs: self.sqlite_vfs, + images: self.images, + query_services: self.query_services, + application_digest: self.application_digest, + limits: self.limits, + cache_directory: self.cache_directory, + cache_pool: self.cache_pool, + cache: self.cache, + }) + } +} + +pub struct CellSqlEnrollment { + record: EnrollmentRecord, + _bytes: Arc, +} + +impl CellSqlEnrollment { + pub fn record(&self) -> &EnrollmentRecord { + &self.record + } +} + +pub enum CellSqlEnrollmentResolution { + Committed(Box), + NotCommitted, + Conflict, +} + /// Clones retain both node and group admission through delivery and submission. #[derive(Clone)] pub struct CellSqlPrepared { @@ -125,6 +166,10 @@ pub struct CellSqlPrepared { } impl CellSqlPrepared { + pub fn term(&self) -> cellar_logic::consensus::Term { + self.prepared.term() + } + pub fn bytes(&self) -> &[u8] { self.prepared.bytes() } @@ -150,6 +195,8 @@ pub enum CellSqlOutcome { Receipt(Box), Submitted, Reconciled(CellSqlReconciliation), + Enrollment(Box), + EnrollmentResolved(CellSqlEnrollmentResolution), } pub enum CellSqlReconciliation { @@ -374,6 +421,79 @@ impl CellGroupSqlPool { .await } + /// Initialize one logical cell on the existing blocking pool. The creation + /// ID is immutable retry identity; a committed retry returns its original + /// enrollment record without opening SQLite or rerunning the callback. + pub async fn initialize( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result { + self.admit()?; + if !authority + .state() + .belongs_to_engine(&self.inner.engine_owner) + { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + authority.check().map_err(crate::raft::execution::authority_failure)?; + if options.application_digest() != authority.state().route().schema_digest { + return Err(ExecutionError::Invalid("enrollment schema")); + } + let limits = options.limits(); + limits.validate()?; + let bytes = if authority.state().record().is_some() { + 4096 + } else { + limits + .envelope + .max_payload_bytes + .checked_mul(4) + .and_then(|n| n.checked_add(4096)) + .ok_or_else(overloaded)? + }; + let cell = authority.state().cell().clone(); + let (lease, bytes) = + self.inner + .admission + .reserve(&cell, authority.state().route().history, bytes)?; + self.call( + cell, + worker::Action::Initialize(Box::new(worker::Initialize { + options, + authority, + attempt, + callback: Box::new(initialize), + lease, + bytes, + })), + ) + .await + } + + pub async fn resolve_enrollment( + &self, + creation: ProposalId, + state: CellEnrollmentState, + ) -> Result { + if !state.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell SQL engine owner")); + } + let cell = state.cell().clone(); + match self + .call( + cell, + worker::Action::ResolveEnrollment(creation, Box::new(state)), + ) + .await? + { + CellSqlOutcome::EnrollmentResolved(resolution) => Ok(resolution), + _ => Err(ExecutionError::Invalid("enrollment resolution response")), + } + } + /// Recover a lost stage response without another callback or producer. pub async fn resume( &self, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index 04360f1b0..a312f7618 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -15,8 +15,21 @@ pub(super) struct Stage { pub bytes: Arc, } +type Initializer = Box) -> anyhow::Result<()> + Send>; + +pub(super) struct Initialize { + pub options: CellSqlOptions, + pub authority: CellEnrollmentAuthority, + pub attempt: CellEnrollmentAttempt, + pub callback: Initializer, + pub lease: CellLease, + pub bytes: Arc, +} + pub(super) enum Action { Stage(Box), + Initialize(Box), + ResolveEnrollment(ProposalId, Box), Resume(ProposalId), Submit(ProposalId), Finish(Box), @@ -25,7 +38,7 @@ pub(super) enum Action { impl Action { pub(super) fn progress_priority(&self) -> bool { - !matches!(self, Self::Stage(_)) + !matches!(self, Self::Stage(_) | Self::Initialize(_)) } } @@ -44,12 +57,40 @@ impl Request { } struct Pending { - executor: CellPrivateExecutor, + producer: Producer, prepared: CellSqlPrepared, submitted: bool, _lease: CellLease, } +enum Producer { + Command(Box), + Enrollment(Box), +} + +impl Producer { + fn mark_submitted(&mut self, proposal: ProposalId) -> Result<(), ExecutionError> { + match self { + Self::Command(executor) => executor.mark_submitted(proposal), + Self::Enrollment(executor) => executor.mark_submitted(proposal), + } + } + + fn command(&mut self) -> Result<&mut CellPrivateExecutor, ExecutionError> { + match self { + Self::Command(executor) => Ok(executor), + Self::Enrollment(_) => Err(ExecutionError::Invalid("command producer required")), + } + } + + fn close(self) -> Result<(), ExecutionError> { + match self { + Self::Command(executor) => (*executor).close(), + Self::Enrollment(executor) => (*executor).close(), + } + } +} + #[derive(Default)] pub(in crate::raft::host_sql) struct Cells { pending: BTreeMap, @@ -76,8 +117,10 @@ impl Cells { queued, } = request; let operation = match &action { - Action::Stage(_) => crate::raft::performance::Operation::SqlStage, - Action::Finish(_) | Action::Reconcile(..) => { + Action::Stage(_) | Action::Initialize(_) => { + crate::raft::performance::Operation::SqlStage + } + Action::Finish(_) | Action::Reconcile(..) | Action::ResolveEnrollment(..) => { crate::raft::performance::Operation::SqlPublish } _ => crate::raft::performance::Operation::SqlOther, @@ -141,7 +184,7 @@ impl Cells { self.pending.insert( cell, Pending { - executor, + producer: Producer::Command(Box::new(executor)), prepared: prepared.clone(), submitted: false, _lease: lease, @@ -169,6 +212,91 @@ impl Cells { } } } + Action::Initialize(request) => { + if self.pending.contains_key(&cell) { + return Err(ExecutionError::Busy); + } + if self.pending.len() >= max { + return Err(overloaded()); + } + let Initialize { + options, + authority, + attempt, + callback, + lease, + mut bytes, + } = *request; + if authority.state().cell() != &cell { + return Err(ExecutionError::Invalid("initialization cell history")); + } + authority.check().map_err(crate::raft::execution::authority_failure)?; + if let Some(record) = authority.state().record() { + if record.cell() != &cell || record.creation() != attempt.creation { + return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); + } + return Ok(CellSqlOutcome::Enrollment(Box::new(CellSqlEnrollment { + record: record.clone(), + _bytes: bytes, + }))); + } + let mut executor = options.open_enrollment(authority)?; + let prepared = match executor.prepare(attempt, callback) { + Ok(prepared) => prepared, + Err(error) => { + executor.close()?; + return Err(error); + } + }; + let retained = prepared + .bytes() + .len() + .checked_add(4096) + .ok_or_else(overloaded)?; + Bytes::shrink(&mut bytes, retained)?; + let prepared = CellSqlPrepared { + prepared, + _bytes: bytes, + }; + self.pending.insert( + cell, + Pending { + producer: Producer::Enrollment(Box::new(executor)), + prepared: prepared.clone(), + submitted: false, + _lease: lease, + }, + ); + Ok(CellSqlOutcome::Prepared(prepared)) + } + Action::ResolveEnrollment(creation, state) => { + let pending = self + .pending + .get_mut(&cell) + .ok_or(ExecutionError::Uncertain)?; + let Producer::Enrollment(executor) = &mut pending.producer else { + return Err(ExecutionError::Invalid("enrollment producer required")); + }; + let outcome = executor.resolve(creation, *state)?; + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Uncertain)?; + pending.producer.close()?; + let outcome = match outcome { + CellEnrollmentResolution::Committed(record) => { + CellSqlEnrollmentResolution::Committed(Box::new(CellSqlEnrollment { + record: *record, + _bytes: pending.prepared._bytes, + })) + } + CellEnrollmentResolution::NotCommitted => { + CellSqlEnrollmentResolution::NotCommitted + } + CellEnrollmentResolution::Conflict => CellSqlEnrollmentResolution::Conflict, + }; + Ok(CellSqlOutcome::EnrollmentResolved(outcome)) + } Action::Resume(proposal) => { let pending = self.pending.get(&cell).ok_or(ExecutionError::Uncertain)?; if pending.prepared.proposal() != proposal { @@ -185,7 +313,7 @@ impl Cells { return Err(ExecutionError::Invalid("cell producer proposal")); } if !pending.submitted { - pending.executor.mark_submitted(proposal)?; + pending.producer.mark_submitted(proposal)?; pending.submitted = true; } Ok(CellSqlOutcome::Submitted) @@ -195,12 +323,12 @@ impl Cells { .pending .get_mut(&cell) .ok_or(ExecutionError::Uncertain)?; - let receipt = pending.executor.finish(*authority)?; + let receipt = pending.producer.command()?.finish(*authority)?; let pending = self .pending .remove(&cell) .ok_or(ExecutionError::Invalid("cell producer changed"))?; - pending.executor.close()?; + pending.producer.close()?; Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { receipt, _bytes: pending.prepared._bytes, @@ -211,12 +339,15 @@ impl Cells { .pending .get_mut(&cell) .ok_or(ExecutionError::Uncertain)?; - let resolution = pending.executor.reconcile(proposal, *authority)?; + let resolution = pending + .producer + .command()? + .reconcile(proposal, *authority)?; let pending = self .pending .remove(&cell) .ok_or(ExecutionError::Invalid("cell producer changed"))?; - pending.executor.close()?; + pending.producer.close()?; let resolution = match resolution { CellReconciliation::Committed(receipt) => { CellSqlReconciliation::Committed(Box::new(CellSqlReceipt { @@ -235,7 +366,7 @@ impl Cells { pub fn close(&mut self) -> Result<(), ExecutionError> { let mut result = Ok(()); for (_, pending) in std::mem::take(&mut self.pending) { - if let Err(error) = pending.executor.close() { + if let Err(error) = pending.producer.close() { result = Err(error); } } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index c7df17534..735eef520 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -76,6 +76,26 @@ async fn rooted_cell_sql_reconciles_expired_and_conflicting_receipts() -> Result runtime_fixture::reconcile_sql_refusals(true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn concurrent_enrollment_one_cell() -> Result<()> { + runtime_fixture::initialize_cells(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_concurrent_enrollment_one_cell() -> Result<()> { + runtime_fixture::initialize_cells(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn enrollment_recovery_after_leadership_change() -> Result<()> { + runtime_fixture::recover_initialization(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_enrollment_recovery_after_leadership_change() -> Result<()> { + runtime_fixture::recover_initialization(true).await +} + #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 8bb7d6a1f..c62dd26da 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -520,3 +520,11 @@ pub(super) async fn reconcile_sql_producers(rooted: bool) -> Result<()> { pub(super) async fn reconcile_sql_refusals(rooted: bool) -> Result<()> { sql::reconcile_refusals(rooted).await } + +pub(super) async fn initialize_cells(rooted: bool) -> Result<()> { + sql::initialize_cells(rooted).await +} + +pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { + sql::recover_initialization(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 0c9919390..22bac0ded 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -428,6 +428,17 @@ pub(super) async fn pool_admission(rooted: bool) -> Result<()> { #[path = "cell_group_sql_reconcile.rs"] mod reconciliation; +#[path = "cell_group_sql_enrollment.rs"] +mod enrollment; + +pub(super) async fn initialize_cells(rooted: bool) -> Result<()> { + enrollment::concurrent(rooted).await +} + +pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { + enrollment::recovery(rooted).await +} + pub(super) async fn reconcile_producers(rooted: bool) -> Result<()> { reconciliation::reconcile(rooted).await } diff --git a/crates/cellar/tests/support/cell_group_sql_enrollment.rs b/crates/cellar/tests/support/cell_group_sql_enrollment.rs new file mode 100644 index 000000000..44184f383 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_enrollment.rs @@ -0,0 +1,439 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::{apply_through, commit_on, transfer}; +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +fn attempt(tag: u8) -> Result { + Ok(CellEnrollmentAttempt { + creation: ProposalId::new([tag; 16])?, + preview: ImageId::new([tag; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }) +} + +fn limits(workers: usize) -> CellSqlLimits { + CellSqlLimits { + workers, + queue_per_worker: 4, + max_pending_cells: 4, + group_pending_cells: 4, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + } +} + +async fn cluster(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { + let mut fixture = RuntimeCluster::new().await?; + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("enrollment-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + Ok((fixture, route)) +} + +async fn prepare_initial( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + tag: u8, + value: i64, +) -> Result { + let authority = fixture.runtimes[node] + .enrollment_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let outcome = pool + .initialize( + fixture.cell_options_at(cell, node).await?, + authority, + attempt(tag)?, + move |tx| { + tx.execute_batch("CREATE TABLE value(n);")?; + tx.execute("INSERT INTO value VALUES (?1)", [value])?; + Ok(()) + }, + ) + .await?; + let CellSqlOutcome::Prepared(prepared) = outcome else { + anyhow::bail!("prepared enrollment") + }; + Ok(prepared) +} + +async fn submit( + fixture: &mut RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + prepared: &CellSqlPrepared, +) -> Result<()> { + let group = fixture.cluster.birth.route().history; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = fixture.runtimes[node] + .propose_at_term(group, prepared.term(), prepared.bytes().to_vec()) + .await?; + apply_through(fixture, accepted.position.index).await +} + +async fn resolved( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + node: usize, + cell: &CellHistory, + creation: ProposalId, +) -> Result { + let state = fixture.runtimes[node] + .enrollment_state(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let CellSqlEnrollmentResolution::Committed(record) = + pool.resolve_enrollment(creation, state).await? + else { + anyhow::bail!("committed enrollment") + }; + Ok(*record) +} + +pub(super) async fn concurrent(rooted: bool) -> Result<()> { + let (mut fixture, route) = cluster(rooted).await?; + let a = history(&route, "sql-a")?; + let b = history(&route, "sql-b")?; + let c = history(&route, "sql-c")?; + let key = |cell: &CellHistory| { + u64::from_le_bytes(cell_image_history_digest(cell)[..8].try_into().unwrap()) + }; + let workers = (2..=8) + .find(|n| key(&a) % n != key(&b) % n) + .context("independent workers")? as usize; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits(workers))?; + assert!(fixture.runtimes[1] + .enrollment_authority(route.history, a.clone()) + .await + .is_err()); + let callbacks = Arc::new(AtomicUsize::new(0)); + let count = callbacks.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .initialize(options, authority, attempt(151)?, move |tx| { + count.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(()) + }) + .await + .map_err(anyhow::Error::new) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let mut other = a.clone(); + other.incarnation = IncarnationId::new([152; 16])?; + for cell in [&a, &other] { + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, cell.clone()) + .await?; + assert!(matches!( + pool.initialize( + fixture.cell_options(cell).await?, + authority, + attempt(152)?, + |_| panic!("concurrent creation callback") + ) + .await, + Err(ExecutionError::Busy) + )); + } + let pb = prepare_initial(&fixture, &pool, 0, &b, 153, 7).await?; + submit(&mut fixture, &pool, 0, &b, &pb).await?; + let rb = resolved(&fixture, &pool, 0, &b, pb.proposal()).await?; + assert_eq!(rb.record().cell(), &b); + drop(rb); + drop(pb); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + let pa = tokio::time::timeout( + Duration::from_secs(5), + pool.resume(a.clone(), ProposalId::new([151; 16])?), + ) + .await??; + let state = fixture.runtimes[0] + .enrollment_state(route.history, a.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pa.proposal(), state).await, + Err(ExecutionError::Uncertain) + )); + submit(&mut fixture, &pool, 0, &a, &pa).await?; + let ra = resolved(&fixture, &pool, 0, &a, pa.proposal()).await?; + let original = ra.record().clone(); + assert_eq!(callbacks.load(Ordering::SeqCst), 1); + drop(ra); + drop(pa); + let pc = prepare_initial(&fixture, &pool, 0, &c, 154, 9).await?; + submit(&mut fixture, &pool, 0, &c, &pc).await?; + drop(resolved(&fixture, &pool, 0, &c, pc.proposal()).await?); + drop(pc); + for voter in &fixture.cluster.voters { + assert_eq!( + voter.log.cell_groups()?.len(), + 1, + "enrollment must not create a physical Raft group" + ); + let marker = voter + .log + .cell_group(&route.history)? + .group_manifest()? + .context("root")?; + for cell in [&a, &b, &c] { + let leaf = voter + .tree + .get(&voter.pages, marker.root(), &cell.cell)? + .context("enrolled cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + } + // A real mutation must not erase or overwrite the immutable creation record. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = pool + .stage( + fixture.cell_options(&a).await?, + authority, + cell_attempt(155)?, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 1); + tx.execute("UPDATE value SET n=n+10", [])?; + Ok(b"after-init".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("mutation") + }; + let receipt = commit_on(&mut fixture, &pool, 0, &a, command).await?; + let version = receipt.receipt().version.clone(); + drop(receipt); + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Enrollment(record) = pool + .initialize( + fixture.cell_options(&a).await?, + authority, + attempt(151)?, + |_| panic!("committed creation must not initialize again"), + ) + .await? + else { + anyhow::bail!("creation retry") + }; + assert_eq!(record.record(), &original); + drop(record); + assert_eq!( + fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await? + .cell() + .version, + version + ); + for (cell, tag) in [(&a, 156), (&other, 151)] { + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, cell.clone()) + .await?; + assert!(matches!( + pool.initialize( + fixture.cell_options(cell).await?, + authority, + attempt(tag)?, + |_| panic!("conflicting creation") + ) + .await, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + } + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + // A cancelled initializer and cancelled close waiter cannot release the + // physical callback or its logical-name reservation before the worker exits. + let d = history(&route, "cancel-init")?; + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, d.clone()) + .await?; + let options = fixture.cell_options(&d).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .initialize(options, authority, attempt(157)?, move |tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (99);")?; + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + Ok(()) + }) + .await + .map_err(anyhow::Error::new) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + let caller = pool.clone(); + let mut closing = fixture + .cluster + .domain + .spawn(async move { caller.close().await })?; + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + closing.abort(); + assert!(closing.await.is_err()); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(pool.usage()?.running_workers > 0); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + pool.close().await?; + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + if let Some(store) = &fixture.cluster.voters[0].extents { + assert_eq!(store.usage()?.workspaces, 0); + } + assert!(fixture.runtimes[0] + .enrollment_state(route.history, d) + .await? + .record() + .is_none()); + drop(pool); + fixture.close().await +} + +pub(super) async fn recovery(rooted: bool) -> Result<()> { + let (mut fixture, route) = cluster(rooted).await?; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits(2))?; + // The existing election fixture holds the sql-a barrier on the old leader. + let a = history(&route, "sql-a")?; + let initial = prepare_initial(&fixture, &pool, 0, &a, 161, 1).await?; + submit(&mut fixture, &pool, 0, &a, &initial).await?; + drop(resolved(&fixture, &pool, 0, &a, initial.proposal()).await?); + drop(initial); + let d = history(&route, "absent-init")?; + let e = history(&route, "conflict-init")?; + let f = history(&route, "committed-init")?; + let pd = prepare_initial(&fixture, &pool, 0, &d, 162, 2).await?; + let pe = prepare_initial(&fixture, &pool, 0, &e, 163, 3).await?; + let pf = prepare_initial(&fixture, &pool, 0, &f, 164, 4).await?; + pool.mark_submitted(d.clone(), pd.proposal()).await?; + pool.mark_submitted(e.clone(), pe.proposal()).await?; + submit(&mut fixture, &pool, 0, &f, &pf).await?; + transfer(&mut fixture, 0, 1).await?; + let successor = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits(1))?; + let mut new_e = e.clone(); + new_e.incarnation = IncarnationId::new([165; 16])?; + let next = prepare_initial(&fixture, &successor, 1, &new_e, 165, 30).await?; + submit(&mut fixture, &successor, 1, &new_e, &next).await?; + drop(resolved(&fixture, &successor, 1, &new_e, next.proposal()).await?); + drop(next); + let foreign = fixture.runtimes[1] + .enrollment_state(route.history, d.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pd.proposal(), foreign).await, + Err(ExecutionError::Invalid(_)) + )); + let state = fixture.runtimes[0] + .enrollment_state(route.history, d.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pd.proposal(), state).await?, + CellSqlEnrollmentResolution::NotCommitted + )); + let state = fixture.runtimes[0] + .enrollment_state(route.history, e.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pe.proposal(), state).await?, + CellSqlEnrollmentResolution::Conflict + )); + let authority = fixture.runtimes[1] + .execution_authority(route.history, f.clone()) + .await?; + let CellSqlOutcome::Prepared(command) = successor + .stage( + fixture.cell_options_at(&f, 1).await?, + authority, + cell_attempt(166)?, + true, + |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"later".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("new leader mutation") + }; + drop(commit_on(&mut fixture, &successor, 1, &f, command).await?); + let record = resolved(&fixture, &pool, 0, &f, pf.proposal()).await?; + assert_eq!(record.record().creation(), pf.proposal()); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(pool.usage()?.retained_bytes > 0); + // Even after this node regains leadership, a retained old prepared response + // cannot be proposed under its new term. Neither sample-before-queue nor + // merely checking that the receiver is a leader would reject this race. + transfer(&mut fixture, 1, 0).await?; + assert!(matches!( + fixture.runtimes[0] + .propose_at_term(route.history, pe.term(), pe.bytes().to_vec()) + .await, + Err(RuntimeError::StaleCompletion) + )); + assert_eq!( + fixture.runtimes[0] + .enrollment_state(route.history, new_e.clone()) + .await? + .record() + .context("winner")? + .cell(), + &new_e + ); + drop(record); + drop(pd); + drop(pe); + drop(pf); + assert_eq!(pool.usage()?.retained_bytes, 0); + successor.close().await?; + drop(successor); + pool.close().await?; + drop(pool); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs index fa458416e..16ac5d924 100644 --- a/crates/cellar/tests/support/cell_group_sql_reconcile.rs +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -2,7 +2,7 @@ use super::*; -async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Result<()> { +pub(super) async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Result<()> { let history = fixture.cluster.birth.route().history; let term = fixture.runtimes[from].status(history).await?.term; let mut message = Message::default(); @@ -68,7 +68,7 @@ fn execution_limits(options: &mut CellSqlOptions) -> &mut ExecutionLimits { } } -async fn apply_through(fixture: &mut RuntimeCluster, index: LogIndex) -> Result<()> { +pub(super) async fn apply_through(fixture: &mut RuntimeCluster, index: LogIndex) -> Result<()> { let group = fixture.cluster.birth.route().history; tokio::time::timeout(Duration::from_secs(20), async { loop { @@ -85,7 +85,7 @@ async fn apply_through(fixture: &mut RuntimeCluster, index: LogIndex) -> Result< .await? } -async fn commit_on( +pub(super) async fn commit_on( fixture: &mut RuntimeCluster, pool: &CellGroupSqlPool, node: usize, @@ -96,7 +96,7 @@ async fn commit_on( pool.mark_submitted(cell.clone(), prepared.proposal()) .await?; let accepted = fixture.runtimes[node] - .propose(group, prepared.bytes().to_vec()) + .propose_at_term(group, prepared.term(), prepared.bytes().to_vec()) .await?; apply_through(fixture, accepted.position.index).await?; let authority = fixture.runtimes[node] From 82ae6877c1c727a3a7c68e463e99beefe71fef46 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 21:00:18 -0700 Subject: [PATCH 057/568] style: format shared cell enrollment authority checks --- crates/cellar/durability/raft/host_sql/cell_groups/mod.rs | 4 +++- crates/cellar/durability/raft/host_sql/cell_groups/worker.rs | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index a07481a8f..b2c18be32 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -438,7 +438,9 @@ impl CellGroupSqlPool { { return Err(ExecutionError::Invalid("cell SQL engine owner")); } - authority.check().map_err(crate::raft::execution::authority_failure)?; + authority + .check() + .map_err(crate::raft::execution::authority_failure)?; if options.application_digest() != authority.state().route().schema_digest { return Err(ExecutionError::Invalid("enrollment schema")); } diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index a312f7618..686dadcba 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -230,7 +230,9 @@ impl Cells { if authority.state().cell() != &cell { return Err(ExecutionError::Invalid("initialization cell history")); } - authority.check().map_err(crate::raft::execution::authority_failure)?; + authority + .check() + .map_err(crate::raft::execution::authority_failure)?; if let Some(record) = authority.state().record() { if record.cell() != &cell || record.creation() != attempt.creation { return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); From cb9ca07d51c9c5dc19f102517f925857f9aafbd9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 21:02:34 -0700 Subject: [PATCH 058/568] docs: finalize executable cell-group implementation handoff --- docs/design/multi-raft/cell-groups/README.md | 13 ++-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 66 ++++++++++++------- .../design/multi-raft/cell-groups/progress.md | 55 +++++++++++++++- .../multi-raft/cell-groups/protocols.md | 8 +++ docs/design/multi-raft/cell-groups/tasks.json | 13 ++-- 6 files changed, 122 insertions(+), 37 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 2f2d8e6c3..6f9c37531 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -9,9 +9,8 @@ The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. -The executable design and implementation plan has been delivered. The active -implementation goal continues through the end-to-end hard cutover, with separate -task and release gates. See the [executor handoff](handoff.md) for the current +This delivery is the executable design and implementation plan. End-to-end +implementation and hard cutover remain tracked as separate task and release gates. See the [executor handoff](handoff.md) for the current source, qualification boundary, reviewed main integration and commands. Plan delivery or an intermediate task gate does not qualify the complete runtime. @@ -97,9 +96,11 @@ obtains current-term quorum before protocol admission. The shared node scheduler producer and exact receipts now pass their partial regression gate, followed by fixed-pool SQL ownership and admission at `51dff0e5`. Applied-barrier receipt reconciliation is implemented at `8a0cf640`, followed by read-only receipt lookup -and saturated-capacity/expiry/conflict cases at `91c3ecfc`. Production retry and -compaction integration, concurrent enrollment, coherent reads and registered SDK -serving remain required. Managed +and saturated-capacity/expiry/conflict cases at `91c3ecfc`. Initialization and +concurrent enrollment now run on that pool at `6e09bbc7`, including original +creation retries, follower resolution and atomic preparation-term proposal checks. +Registered SDK/catalog routing, configured group capacity, coherent reads, +production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 33385b363..293fe1766 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "91c3ecfc020e0d42738e4a6395045eb8a44c51f9", + "source_revision": "82ae6877c1c727a3a7c68e463e99beefe71fef46", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 32bd2c16d..12eb3c7b2 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -2,9 +2,9 @@ # Cell-group implementation handoff -The executable design and implementation plan is delivered. The active goal -continues through end-to-end implementation and hard cutover, including the full -task and release gates. +The executable design and implementation plan is delivered. This handoff scopes +the remaining end-to-end implementation and hard cutover, including the full task +and release gates, for subsequent execution. This handoff does not claim completion of the shared runtime, a production cutover or measured TiKV performance parity. The target is the same partitioned multi-Raft pattern: many @@ -15,11 +15,12 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `91c3ecfc`, separating committed receipt lookup -from admission after applied-barrier reconciliation at `8a0cf640` and the fixed -SQL pool at `51dff0e5`, following -the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`; see the latest checkpoint in -[progress](progress.md) for qualification and remaining work. +The reviewed source baseline is `82ae6877`, a formatting-only successor to +`6e09bbc7`, which adds shared-worker initialization, +logical-name admission and prepared-term proposal fencing after receipt lookup at +`91c3ecfc`, applied-barrier reconciliation at `8a0cf640`, the fixed SQL pool at +`51dff0e5`, the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`. +See the latest checkpoint in [progress](progress.md) for qualification and limits. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; use the commands below. Historical evidence keeps its original tested source, @@ -74,20 +75,41 @@ admission retains both ceilings. Native fixtures now cover saturated identity tables, replicated expiry floors and conflicting identities in both image modes, including callback suppression and later successful command admission. -Next integrate concurrent initialization through the existing bounded SQL pool -and replicated enrollment decisions, then coherent reads and registered SDK -dispatch. Real compaction and production retry integration, plus preview/cache -lifecycle, remain required. Preserve physical completion and cancellation -boundaries. These cases alone do not complete the six-case CG-04 gate. - -For enrollment, `ManifestTree::prepare_enrollment` already preserves creation -identity at ordered apply; current native fixtures construct initial SQLite images -manually. Add actual-leader enrollment admission and initialization jobs on the -existing SQL workers. Reserve the logical cell name before choosing a new history, -so competing incarnations cannot bypass single-producer admission. Reuse current -node/group budgets and joined callback ownership. An existing committed enrollment -must return its retained creation record without rerunning initialization or -replacing the current image; do not manufacture a legacy `GroupScope` adapter. +Initialization now runs through `CellGroupSqlPool::initialize` and +`CellPrivateEnrollment`, using `CellEnrollmentAuthority` from the real leader. +Logical name admission covers both commands and initialization, including competing +incarnations. `CellEnrollmentState` reads verified current-term applied creation +state on leaders or followers; `resolve_enrollment` preserves the original record, +requires a newer applied term for absent/conflicting old producers, and never +reexecutes callbacks. Prepared outputs carry their term; use `propose_at_term` so +the actual owner rejects old output even after that node regains leadership. +The four native cases exercise materialized/extents, concurrent creation, other +cell progress, three cells in one group, retries after mutation, cancelled close, +and absent/conflicting/committed enrollment recovery. All 191 selected Rust cases +and all-target Clippy pass at `6e09bbc7`. The retained partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-initialization/checkpoint.json`; it includes the +initial formatting failure. Formatting-only successor verification is recorded in +`.cellar/cell-groups-evidence/CG-04-initialization-format/gate-commands.json`: all +six checks pass, including formatting, layout, dependencies, docs, preflight and +37 validator tests. These records do not mark CG-04 DONE. + +Next implement configured group enrollment capacity admission on the actual owner, +including pending accepted creations rather than only the applied cell count. The +manifest layer already enforces its bounds; admission must refuse before appending +a creation that cannot apply, rather than turn capacity pressure into group failure. +Also complete same-term proposal retry handling: `propose_at_term` fences old +terms but does not deduplicate a repeated mutation envelope. A lost proposal +acceptance response must reuse retained acceptance state or reconcile receipts; +blindly appending the same cell mutation twice can invalidate its second base. +Enrollment's exact retry is idempotent at apply, but its pending capacity accounting +must still count logical creations rather than repeated entries. + +Then integrate catalog routing sessions and registered SDK creation/command +callbacks, and coherent reads. Preserve the original request fingerprint, deadline +and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to +satisfy callback/effect identities. The callback APIs added here remain verification +boundaries, not registered SDK entry points. Real compaction/production retries +and preview/cache lifecycle remain required. These cases do not complete CG-04. The remaining coupling is concrete: `execution::CommandAuthority` binds the old `GroupLog`/`GroupStatus`; `execution::images::Published` exposes `GroupScope`; diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 17d73b26a..9a197a550 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -2,14 +2,65 @@ # Cell-group implementation progress -The executable design and implementation plan is delivered. The active goal -continues through end-to-end implementation and hard cutover; see the +The executable design and implementation plan is delivered. Remaining +end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). CG-01 through CG-03 have passed their scoped gates; CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 shared-worker initialization and enrollment + +Implementation source: `6e09bbc7`. + +`CellEnrollmentAuthority` now binds an actual leader, current-term applied prefix, +route and logical cell name before a cell image exists. Initialization uses the +existing fixed SQL workers and admission budgets. One logical name reservation +covers commands and initializers, so a competing incarnation cannot bypass a +running or unresolved producer. Worker-owned SQLite initialization captures and +verifies its complete initial LTX image; followers apply those bytes without +running the callback. Root workspace preparation is shared with ordinary commands. + +Prepared SQL outputs now carry their preparation term. `propose_at_term` checks +that fence on the real RawNode owner in the same turn as proposal admission. +Retained output cannot be appended under a newer term even after the original +node becomes leader again. The raw unfenced protocol API remains a verification +boundary; registered dispatch must use the fenced path. + +Committed retries return the immutable creation record without opening SQLite or +rerunning initialization after later mutations. `CellEnrollmentState` observes +verified current-term applied creation state on followers as well as leaders. +Exact records prove commitment; missing/conflicting records retire old producers +only after a strictly newer applied term. Resolution closes the writer before +releasing name admission, while retained outputs keep their byte reservation. +Cancelled request and close waiters cannot release an entered initializer. + +Four native cases cover both image modes: competing histories of one name, +cancelled initialization and exact resume, another cell committing while its peer +initializer is blocked, three independent cells under one physical group on each +voter, creation retries after mutation, conflicting creation refusal, cancelled +close and workspace release, follower resolution of absent/conflicting/committed +creations, and rejection of old prepared output after leadership returns. + +All 191 selected Rust cases pass at `6e09bbc7`: 176 native cases across seven +suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy passes with warnings denied. The retained partial checkpoint +at `.cellar/cell-groups-evidence/CG-04-initialization/checkpoint.json` binds 184 +source inputs, eight copied executables and the command logs. The gate stopped +at two formatting differences; that failure is retained. `82ae6877` changes only +line wrapping in those two authority checks. Successor verification is recorded +separately in `CG-04-initialization-format/gate-commands.json`: formatting +(1,798 Rust files), layout, dependency isolation, documentation, source preflight +and all 37 plan-validator tests pass. No Rust behavior changed and the Rust tests +were not repeated for line wrapping. Four process helpers execute through parent +tests; three RustFS cases remain explicitly unrun. No task status advances. + +Configured group capacity policy, catalog routing sessions, registered SDK +creation/command callbacks, coherent reads, +production retry/compaction integration and preview/cache lifecycle remain open. +This is partial CG-04 implementation, not the final hard cutover or release gate. + ## CG-04 receipt lookup independent from admission Qualified source: `91c3ecfc020e0d42738e4a6395045eb8a44c51f9`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 0aa2c11a1..3a2f8eb1d 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -39,6 +39,14 @@ later mutations and normal transfers; comparing a retry to the latest image woul confuse a successful earlier creation with a conflict. The initial commitment is provenance, not a requirement to retain the superseded initial image forever. Snapshots and archives carry the record with the selected current cell state. +Initialization runs on the bounded node SQL workers under actual-leader authority. +Reserve the logical cluster/namespace/cell name before entering the callback; +competing incarnations must not obtain independent producer leases. Capture initial +SQLite/LTX bytes once, carry the preparation term with the output, and check that +term on the RawNode owner atomically with proposal admission. A node that loses +leadership and later regains it must reject retained output from its older term. +Cancellation retains physical ownership until the callback and cleanup complete. + Admission decisions must be checked again in committed log order against the current lifecycle state. Recognizing an exact retry grants neither current-term authority nor permission to restore the initial image over newer mutations. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 849e797ba..08be0f164 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -457,7 +457,10 @@ "crates/cellar/durability/raft/host_sql/cell_groups/admission.rs", "crates/cellar/durability/raft/host_sql/cell_groups/worker.rs", "crates/cellar/tests/support/cell_group_sql_pool.rs", - "crates/cellar/tests/support/cell_group_sql_reconcile.rs" + "crates/cellar/tests/support/cell_group_sql_reconcile.rs", + "crates/cellar/durability/raft/cell_groups/enrollment_authority.rs", + "crates/cellar/durability/raft/execution/enrollment.rs", + "crates/cellar/tests/support/cell_group_sql_enrollment.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -469,13 +472,13 @@ ] }, { - "action": "Admit one unresolved proposal per cell with fair cell queues and group/node byte quotas. Allow independent cells to prepare concurrently, then publish committed entries through one contiguous group prefix with per-cell base/version checks.", + "action": "Admit one unresolved proposal per cell with fair cell queues and group/node byte quotas. Allow independent cells to prepare concurrently, then publish committed entries through one contiguous group prefix with per-cell base/version checks. Enforce configured group enrollment capacity on the actual owner before append, counting applied cells and pending accepted logical creations. Exact creation retries consume no additional slot; competing creations cannot exceed the limit before apply. Verify capacity refusal leaves existing cells available and release reservations only with proven resolution.", "tests": [ "shared_group_fair_admission" ] }, { - "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes.", + "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes. Check the prepared term atomically with proposal admission on the actual Raft owner. Bind same-term acceptance retries to the original proposal identity and bytes; a lost acceptance response must recover retained acceptance or reconcile the original receipt without appending the same mutation again. Reject changed payloads under a reused identity, and retain uncertainty across elections until an applied barrier proves resolution.", "tests": [ "cell_receipt_not_group_progress" ] @@ -500,14 +503,14 @@ }, { "name": "shared_group_fair_admission", - "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks.", + "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks. Pending distinct enrollments cannot exceed configured group capacity before apply; identical retries reserve once and capacity refusal does not fail the group.", "invariants": [ "I12" ] }, { "name": "cell_receipt_not_group_progress", - "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent.", + "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent. Lost proposal acceptance responses and exact retries in the same term append at most one mutation; changed payloads with the same identity refuse and old prepared terms cannot append after leadership returns.", "invariants": [ "I02", "I03" From 446dd9057f10c8221122d0d3a9a21254adeb448f Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 21:20:21 -0700 Subject: [PATCH 059/568] feat: bound shared-group proposal admission and creation capacity --- crates/cellar/durability/raft/README.md | 20 +- .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../durability/raft/cell_groups/owner.rs | 43 +++- .../raft/cell_groups/proposal_admission.rs | 160 +++++++++++++ .../raft/cell_groups/runtime/worker.rs | 7 +- crates/cellar/durability/raft/options.rs | 7 + .../tests/native_cell_groups_execution.rs | 23 ++ .../tests/support/cell_group_admission.rs | 224 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 100 +++++++- 9 files changed, 558 insertions(+), 27 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/proposal_admission.rs create mode 100644 crates/cellar/tests/support/cell_group_admission.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 75d7d096e..7b4324f7b 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -105,6 +105,23 @@ proposal admission. Merely sampling leader status before queueing is insufficien including when the original node later regains leadership. The unfenced `propose` method remains a low-level protocol verification boundary. +The owner retains a bounded acceptance record for each pending logical cell name. +An exact proposal retry returns its original position and digest; different bytes +under that pending identity refuse. Independent proposals for the same pending +name wait. `max_cells_per_group` counts applied cells plus accepted creations +before append, so concurrent enrollments cannot overfill the manifest. Existing +cells still accept mutations at capacity. Mutation admission checks its exact +base against the selected cell version before entering Raft. + +Acceptance records retain a separate node-wide budget of `max_requests` items and +`max_request_bytes`, charging 4 KiB per record without retaining LTX payloads or +image pins. This budget cannot block Ready or completion work. Records release +only after exact durable apply, role/term loss, terminal failure, or owner destruction. A new leader +cannot admit until its applied current-term barrier covers inherited entries. +After a record releases, stale mutation bases and existing enrollments return +`StaleCompletion`: callers must resolve the original receipt or creation record. +That refusal does not establish that the prior attempt failed. + A committed creation retry returns the immutable enrollment record without opening SQLite or rerunning the callback, even after later cell mutations. An enrollment state selected from a durable current-term applied root also permits resolution @@ -115,8 +132,7 @@ their byte charge. Unknown producers remain uncertain. Pool shutdown joins an entered initializer even if its waiter and close waiter were both cancelled. As with commands, shutdown is not proof that a submitted enrollment failed. -Registered SDK creation/command dispatch, catalog routing integration, configured -group capacity policy, coherent query serving and +Registered SDK creation/command dispatch, catalog routing integration, coherent query serving and the production hard cutover remain required; the existing host still uses its prior single-cell owner. The new pool's callback entry point remains a verification boundary. Transport authentication and actual image preparation are also caller diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 8c9bd8d06..94918ef75 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -22,6 +22,7 @@ mod owner; mod page_store; mod page_wire; mod pins; +mod proposal_admission; mod retention; mod root; mod runtime; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 0b3f9adfc..4fe9f6ccf 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -162,6 +162,8 @@ pub struct CellGroupOwner { next_ticket: u64, last_sync: u64, failed: Option, + proposals: super::proposal_admission::ProposalAdmission, + pub(super) proposal_budget: Arc, } impl CellGroupOwner { @@ -231,6 +233,10 @@ impl CellGroupOwner { group, node, current, + proposal_budget: crate::raft::budget::Budget::new( + options.max_request_bytes, + options.max_requests, + ), options, phase: None, waiting: None, @@ -241,20 +247,24 @@ impl CellGroupOwner { next_ticket: 0, last_sync: 0, failed: None, + proposals: Default::default(), }) } - pub(super) fn revoke_execution(&self) { + pub(super) fn revoke_execution(&mut self) { self.leader_term.store(0, Ordering::Release); + self.proposals.clear(); } - fn refresh_execution_fence(&self) { + fn refresh_execution_fence(&mut self) { let term = if self.node.raft.state == StateRole::Leader { self.node.raft.term } else { 0 }; - self.leader_term.store(term, Ordering::Release); + if self.leader_term.swap(term, Ordering::AcqRel) != term { + self.proposals.clear(); + } } pub fn execution_authority( @@ -377,14 +387,18 @@ impl CellGroupOwner { GroupEntryLimits::default(), ) .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - if !matches!( - entry.command, - GroupCommand::Enroll { .. } | GroupCommand::Mutation(_) - ) { - return Err(RuntimeError::Invalid( - "group administration is not attached".into(), - )); - } + let admission = self.proposals.check( + &entry, + &data, + &self.current, + self.options.max_cells_per_group, + self.options.max_requests, + &self.proposal_budget, + )?; + let (candidate, permit) = match admission { + super::proposal_admission::Admission::Existing(accepted) => return Ok(accepted), + super::proposal_admission::Admission::New(candidate, permit) => (candidate, permit), + }; let before = self.node.raft.raft_log.last_index(); self.node.propose(Vec::new(), data)?; let entry = self @@ -397,11 +411,13 @@ impl CellGroupOwner { if entry.index != before + 1 { return Err(RuntimeError::NotReady); } - Ok(CellGroupProposal { + let accepted = CellGroupProposal { group: self.group.identity().scope, position: entry_position(entry), digest: entry_digest(entry)?, - }) + }; + self.proposals.accept(candidate, accepted.clone(), permit); + Ok(accepted) } /// Check the SQL preparation fence on the actual RawNode owner in the same @@ -729,6 +745,7 @@ impl CellGroupOwner { self.apply_bytes -= entry.compute_size() as usize + 128; self.applying = None; self.current = *next; + self.proposals.applied(ticket.position); // Physical preparation and publication have settled. Its // reservation must not deadlock admission of Applied. ticket.admission = None; diff --git a/crates/cellar/durability/raft/cell_groups/proposal_admission.rs b/crates/cellar/durability/raft/cell_groups/proposal_admission.rs new file mode 100644 index 000000000..67d58b4c7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/proposal_admission.rs @@ -0,0 +1,160 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded, leader-local acceptance; never a commit or receipt proof. + +use super::{CellGroupProposal, PreparedGroupManifest}; +use crate::application::consensus::group_protocol::{CellGroupEntry, GroupCommand}; +use crate::raft::budget::{Budget, Permit}; +use crate::raft::completion::RuntimeError; +use cellar_logic::{ + cell_group::CellId, + consensus::{LogIndex, LogPosition, ProposalId}, +}; +use sha2::{Digest, Sha256}; +use std::collections::BTreeMap; +use std::sync::Arc; + +pub(super) struct Candidate { + cell: CellId, + proposal: ProposalId, + payload: [u8; 32], + enrollment: bool, +} + +struct Accepted { + candidate: Candidate, + acceptance: CellGroupProposal, + _permit: Permit, +} + +pub(super) enum Admission { + Existing(CellGroupProposal), + New(Candidate, Permit), +} + +/// At most one record per pending logical name, bounded by max_requests. No LTX +/// payload or image pin is retained. Clearing on role loss is safe because new +/// admission requires a current-term applied barrier covering inherited entries. +#[derive(Default)] +pub(super) struct ProposalAdmission { + cells: BTreeMap, + positions: BTreeMap, + enrollments: u32, +} + +impl ProposalAdmission { + pub(super) fn check( + &self, + entry: &CellGroupEntry<'_>, + bytes: &[u8], + current: &PreparedGroupManifest, + max_cells: u32, + max_pending: usize, + budget: &Arc, + ) -> Result { + let (cell, enrollment) = match &entry.command { + GroupCommand::Enroll { cell, .. } => (&cell.cell, true), + GroupCommand::Mutation(mutation) => (&mutation.cell.cell, false), + _ => { + return Err(RuntimeError::Invalid( + "group administration is not attached".into(), + )) + } + }; + let candidate = Candidate { + cell: cell.clone(), + proposal: entry.binding.proposal, + payload: Sha256::digest(bytes).into(), + enrollment, + }; + if let Some(accepted) = self.cells.get(cell) { + if accepted.candidate.proposal != candidate.proposal { + return Err(RuntimeError::NotReady); + } + if accepted.candidate.payload != candidate.payload { + return Err(RuntimeError::Invalid( + "proposal identity reused with different bytes".into(), + )); + } + return Ok(Admission::Existing(accepted.acceptance.clone())); + } + let existing = current.pin_cell(cell); + match &entry.command { + GroupCommand::Enroll { .. } => { + // Once applied, the immutable creation record is the retry + // authority. Do not append another enrollment, even at capacity. + if existing.is_some() { + return Err(RuntimeError::StaleCompletion); + } + if current + .marker() + .root() + .cells() + .saturating_add(self.enrollments) + >= max_cells + { + return Err(RuntimeError::Overloaded); + } + } + GroupCommand::Mutation(mutation) => { + let (leaf, _) = existing.ok_or(RuntimeError::StaleCompletion)?; + mutation + .check_base(&leaf.version) + .map_err(|_| RuntimeError::StaleCompletion)?; + } + _ => unreachable!("validated data command"), + } + if self.cells.len() >= max_pending { + return Err(RuntimeError::Overloaded); + } + // Three bounded cell-ID copies, both map nodes and acceptance metadata. + // Shared across runtime groups, separately from Ready/completion budgets + // so saturation cannot prevent accepted work from draining. + let permit = budget.acquire(4096).ok_or(RuntimeError::Overloaded)?; + Ok(Admission::New(candidate, permit)) + } + + pub(super) fn accept( + &mut self, + candidate: Candidate, + acceptance: CellGroupProposal, + permit: Permit, + ) { + self.enrollments += u32::from(candidate.enrollment); + self.positions + .insert(acceptance.position.index, candidate.cell.clone()); + self.cells.insert( + candidate.cell.clone(), + Accepted { + candidate, + acceptance, + _permit: permit, + }, + ); + } + + pub(super) fn applied(&mut self, position: LogPosition) { + let Some(cell) = self.positions.get(&position.index) else { + return; + }; + if self + .cells + .get(cell) + .is_none_or(|accepted| accepted.acceptance.position != position) + { + return; + } + let cell = self + .positions + .remove(&position.index) + .expect("accepted position"); + let accepted = self.cells.remove(&cell).expect("accepted cell"); + self.enrollments -= u32::from(accepted.candidate.enrollment); + } + + pub(super) fn clear(&mut self) { + self.cells.clear(); + self.positions.clear(); + self.enrollments = 0; + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 7e6fa4853..6def095bd 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -100,6 +100,7 @@ pub(super) struct Worker { events: mpsc::Sender, outbox: VecDeque, output: Arc, + proposals: Arc, inflight: Option>, inflight_bytes: usize, sequence: u64, @@ -119,6 +120,7 @@ impl Worker { Self { log, output: Budget::new(options.max_event_bytes, options.max_events), + proposals: Budget::new(options.max_request_bytes, options.max_requests), options, domain, groups: BTreeMap::new(), @@ -168,7 +170,7 @@ impl Worker { return; } let Request { - command: Command::Attach(owner), + command: Command::Attach(mut owner), reply: answer, _permit, .. @@ -176,6 +178,9 @@ impl Worker { else { unreachable!() }; + // attach constructs a fresh follower; it has no accepted proposals. + // All owners retain acceptance against the same node-wide budget. + owner.proposal_budget = self.proposals.clone(); let mut group = Group { owner, queued: false, diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index f1b641ad9..a7acbb856 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -122,6 +122,10 @@ pub struct RaftRuntimeOptions { pub max_proposal_bytes: usize, pub max_members_per_group: usize, pub max_pending_reads: usize, + /// Shared-group creation admission counts applied cells and pending accepted + /// enrollments. A lower limit does not prevent existing cells from serving. + /// max_requests also bounds leader-local pending cell acceptance records. + pub max_cells_per_group: u32, } impl Default for RaftRuntimeOptions { @@ -142,6 +146,7 @@ impl Default for RaftRuntimeOptions { max_proposal_bytes: 16 * 1024 * 1024, max_members_per_group: 7, max_pending_reads: 256, + max_cells_per_group: 1_000_000, } } } @@ -177,6 +182,8 @@ impl RaftRuntimeOptions { || self.max_members_per_group > 16 || self.max_pending_reads == 0 || self.max_pending_reads > 4096 + || self.max_cells_per_group == 0 + || self.max_cells_per_group > 1_000_000 { return Err(super::completion::RuntimeError::Invalid( "invalid runtime bounds".into(), diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 735eef520..d0e6beba5 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -11,6 +11,29 @@ mod fault; #[path = "support/cell_group_runtime.rs"] mod runtime_fixture; +#[path = "support/cell_group_admission.rs"] +mod admission_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn pending_enrollments_respect_group_capacity() -> Result<()> { + admission_fixture::capacity().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn proposal_acceptance_retries_do_not_append_twice() -> Result<()> { + admission_fixture::mutation_retries().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn inherited_enrollments_count_after_applied_term_barrier() -> Result<()> { + admission_fixture::restart_capacity().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_runtime_bounds_proposal_acceptance_across_groups() -> Result<()> { + runtime_fixture::shared_proposal_budget().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn node_runtime_replicates_independent_cells() -> Result<()> { runtime_fixture::replicated_cells().await diff --git a/crates/cellar/tests/support/cell_group_admission.rs b/crates/cellar/tests/support/cell_group_admission.rs new file mode 100644 index 000000000..27da4aa9e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_admission.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) async fn capacity() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_cells_per_group: 2, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let a = history(&route, "capacity-a")?; + let b = history(&route, "capacity-b")?; + let c = history(&route, "capacity-c")?; + let request_a = request(&route, a.clone(), 1, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let first = owner.propose_at_term(term, request_a.clone())?; + let repeated = owner.propose_at_term(term, request_a.clone())?; + assert_eq!(first.position, repeated.position); + assert_eq!(first.digest, repeated.digest); + let mut rival = a.clone(); + rival.incarnation = IncarnationId::new([99; 16])?; + assert!(matches!( + owner.propose_at_term(term, request(&route, rival, 1, &cut)?), + Err(RuntimeError::Invalid(_)) + )); + let second = owner.propose_at_term(term, request(&route, b.clone(), 1, &cut)?)?; + assert_eq!(second.position.index.get(), first.position.index.get() + 1); + assert_eq!(owner.manifest().marker().root().cells(), 0); + assert!(matches!( + owner.propose_at_term(term, request(&route, c.clone(), 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + cluster.drive(false).await?; + for voter in &cluster.voters { + assert_eq!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .root() + .cells(), + 2 + ); + } + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(matches!( + owner.propose_at_term(term, request_a), + Err(RuntimeError::StaleCompletion) + )); + assert!(matches!( + owner.propose_at_term(term, request(&route, c, 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + let updated = images::image(Some(&bytes), "UPDATE value SET n=n+1;")?; + let mutation = images::Cut::new(&bytes, &updated, 1, true, false); + let accepted = owner.propose_at_term(term, request(&route, a.clone(), 2, &mutation)?)?; + assert_eq!( + accepted.position.index.get(), + second.position.index.get() + 1 + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + let (leaf, _) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&a.cell) + .context("updated cell")?; + assert_eq!(leaf.version.sequence.get(), 2); + } + cluster.close().await +} + +pub(super) async fn mutation_retries() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_requests: 1, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let a = history(&route, "retry-a")?; + let b = history(&route, "retry-b")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + for cell in [&a, &b] { + cluster.voters[0].owner.as_mut().unwrap().propose(request( + &route, + cell.clone(), + 1, + &cut, + )?)?; + cluster.drive(false).await?; + } + let updated = images::image(Some(&bytes), "UPDATE value SET n=n+1;")?; + let cut = images::Cut::new(&bytes, &updated, 1, true, false); + let mutation_a = request(&route, a.clone(), 2, &cut)?; + let mutation_b = request(&route, b.clone(), 2, &cut)?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let accepted = owner.propose_at_term(term, mutation_a.clone())?; + // Model a lost acceptance response: the same bytes recover its original + // position without another append, even when acceptance capacity is full. + let retry = owner.propose_at_term(term, mutation_a.clone())?; + assert_eq!(accepted.position, retry.position); + assert_eq!(accepted.digest, retry.digest); + assert!(matches!( + owner.propose_at_term(term, mutation_b.clone()), + Err(RuntimeError::Overloaded) + )); + let different = images::image(Some(&bytes), "UPDATE value SET n=n+99;")?; + let cut = images::Cut::new(&bytes, &different, 1, true, false); + assert!(matches!( + owner.propose_at_term(term, request(&route, a.clone(), 2, &cut)?), + Err(RuntimeError::Invalid(_)) + )); + cluster.drive(false).await?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + // The applied image excludes the old base after its acceptance record is + // released. The caller must inspect its receipt; this is not a failed write. + assert!(matches!( + owner.propose_at_term(term, mutation_a), + Err(RuntimeError::StaleCompletion) + )); + let accepted_b = owner.propose_at_term(term, mutation_b)?; + assert_eq!( + accepted_b.position.index.get(), + accepted.position.index.get() + 1 + ); + cluster.drive(false).await?; + for voter in &cluster.voters { + for cell in [&a, &b] { + let (leaf, CellImagePin::Materialized(pin)) = voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&cell.cell) + .context("cell")? + else { + anyhow::bail!("materialized cell") + }; + assert_eq!(leaf.version.sequence.get(), 2); + assert_eq!(images::read(&pin)?, updated); + } + } + cluster.close().await +} + +pub(super) async fn restart_capacity() -> Result<()> { + let mut cluster = Cluster::with_options(RaftRuntimeOptions { + max_cells_per_group: 1, + ..Default::default() + }) + .await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let a = history(&route, "inherited")?; + let b = history(&route, "extra")?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + let term = owner.status()?.term; + let old = owner.propose_at_term(term, request(&route, a.clone(), 1, &cut)?)?; + cluster.drive(true).await?; + assert_eq!( + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .root() + .cells(), + 0 + ); + // Recover every voter from disk. Live followers retain the old leader's + // lease and may correctly reject an immediate pre-vote from a lone restart. + for n in 0..3 { + cluster.restart(n).await?; + } + cluster.voters[0].paused = true; + cluster.voters[0].owner.as_mut().unwrap().campaign()?; + cluster.drive(false).await?; + assert!(cluster.voters[0].held.is_some()); + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(owner.status()?.term > term); + assert!(!owner.status()?.ready); + assert!(matches!( + owner.propose(request(&route, b.clone(), 1, &cut)?), + Err(RuntimeError::NotReady) + )); + assert!(matches!( + owner.propose_at_term(term, request(&route, a.clone(), 1, &cut)?), + Err(RuntimeError::StaleCompletion) + )); + let voter = &mut cluster.voters[0]; + voter.paused = false; + let work = voter.held.take().context("held inherited apply")?; + assert_eq!(work.ticket.position(), old.position); + let next = prepare(voter, &work)?; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(work.ticket, next)?; + cluster.drive(false).await?; + let owner = cluster.voters[0].owner.as_mut().unwrap(); + assert!(owner.status()?.ready); + assert_eq!(owner.manifest().marker().root().cells(), 1); + assert!(matches!( + owner.propose(request(&route, b, 1, &cut)?), + Err(RuntimeError::Overloaded) + )); + cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index c62dd26da..fd473fb0c 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -11,14 +11,15 @@ struct RuntimeCluster { events: Vec, applied: BTreeMap>, extra: BTreeMap>, + network: VecDeque<(GroupHistory, Message)>, } impl RuntimeCluster { async fn new() -> Result { - Self::with_events(256).await + Self::with_limits(256, 256).await } - async fn with_events(max_events: usize) -> Result { + async fn with_limits(max_events: usize, max_requests: usize) -> Result { let mut cluster = Cluster::new().await?; let mut runtimes = Vec::new(); let mut events = Vec::new(); @@ -34,6 +35,7 @@ impl RuntimeCluster { tick_interval_ms: None, election_seed: Some([71; 32]), max_events, + max_requests, ..Default::default() }, ) @@ -49,6 +51,7 @@ impl RuntimeCluster { events, applied, extra: BTreeMap::new(), + network: VecDeque::new(), }) } @@ -58,7 +61,6 @@ impl RuntimeCluster { held: &mut BTreeMap, ) -> Result<()> { let primary = self.cluster.birth.route().history; - let mut messages = Vec::new(); for n in 0..3 { for _ in 0..64 { let Ok(event) = self.events[n].try_recv() else { @@ -66,7 +68,9 @@ impl RuntimeCluster { }; let scope = event.group(); match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { - CellGroupEffect::Message(message) => messages.push((scope, message.clone())), + CellGroupEffect::Message(message) => { + self.network.push_back((scope, message.clone())) + } CellGroupEffect::Apply(_) => { let work = event.retain_apply().context("retained apply")?; if n == 0 && hold.contains(&scope) { @@ -90,13 +94,27 @@ impl RuntimeCluster { } } } - for result in - futures_util::future::join_all(messages.into_iter().map(|(scope, message)| { - self.runtimes[message.to as usize - 1].admit_step(scope, message) - })) - .await + let messages = std::mem::take(&mut self.network); + let runtimes = &self.runtimes; + for (scope, message, result) in futures_util::future::join_all(messages.into_iter().map( + |(scope, message)| async move { + let result = runtimes[message.to as usize - 1] + .admit_step(scope, message.clone()) + .await; + (scope, message, result) + }, + )) + .await { - result?; + match result { + Ok(()) => (), + // Preserve the peer frame until bounded transport admission is + // available. Test deadlines still fail a permanently stuck peer. + Err(RuntimeError::Overloaded | RuntimeError::NotReady) => { + self.network.push_back((scope, message)) + } + Err(error) => return Err(error.into()), + } } Ok(()) } @@ -354,8 +372,68 @@ pub(super) async fn shared_batches() -> Result<()> { fixture.close().await } +pub(super) async fn shared_proposal_budget() -> Result<()> { + let mut fixture = RuntimeCluster::with_limits(256, 8).await?; + let first = fixture.cluster.birth.route().history; + let second = fixture.add_group(61).await?; + for scope in [first, second] { + fixture.runtimes[0].campaign(scope).await?; + fixture.settle(&[scope]).await?; + } + let route = fixture.cluster.birth.route().clone(); + let other_route = fixture.extra[&second][0] + .log + .cell_group(&second)? + .group_manifest()? + .context("other group")? + .route() + .clone(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut first_request = None; + let mut target = LogIndex::new(0); + for n in 0..8 { + let data = request(&route, history(&route, &format!("budget-{n}"))?, 1, &cut)?; + let accepted = fixture.runtimes[0].propose(first, data.clone()).await?; + if n == 0 { + first_request = Some((data, accepted.clone())); + } + target = accepted.position.index; + } + let other = request(&other_route, history(&other_route, "other-group")?, 1, &cut)?; + assert!(matches!( + fixture.runtimes[0].propose(second, other.clone()).await, + Err(RuntimeError::Overloaded) + )); + let (data, accepted) = first_request.context("first acceptance")?; + let retry = fixture.runtimes[0].propose(first, data).await?; + assert_eq!(accepted.position, retry.position); + assert_eq!(accepted.digest, retry.digest); + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&first].iter().any(|index| *index < target) { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let accepted = fixture.runtimes[0].propose(second, other).await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&second] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + fixture.close().await +} + pub(super) async fn bounded_apply_completion() -> Result<()> { - let mut fixture = RuntimeCluster::with_events(2).await?; + let mut fixture = RuntimeCluster::with_limits(2, 256).await?; let first = fixture.cluster.birth.route().history; let second = fixture.add_group(51).await?; let histories = [first, second]; From f4e418a627a369eb66d1fddadb9818d7b7765d19 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 21:35:09 -0700 Subject: [PATCH 060/568] docs: qualify group capacity and proposal acceptance gates --- docs/design/multi-raft/cell-groups/README.md | 6 ++- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 53 ++++++++++++------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 47 ++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 9 ++++ docs/design/multi-raft/cell-groups/tasks.json | 52 ++++++++++++++++-- 7 files changed, 145 insertions(+), 28 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 6f9c37531..a3d828249 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -99,8 +99,10 @@ reconciliation is implemented at `8a0cf640`, followed by read-only receipt looku and saturated-capacity/expiry/conflict cases at `91c3ecfc`. Initialization and concurrent enrollment now run on that pool at `6e09bbc7`, including original creation retries, follower resolution and atomic preparation-term proposal checks. -Registered SDK/catalog routing, configured group capacity, coherent reads, -production retry/compaction integration and cache lifecycle remain required. Managed +Bounded proposal acceptance and configured enrollment capacity are implemented at +`446dd905`, including pending creations, exact acceptance retries, stale-base +refusal and a shared node metadata budget. Registered SDK/catalog routing, coherent +reads, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 293fe1766..65477126f 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "82ae6877c1c727a3a7c68e463e99beefe71fef46", + "source_revision": "446dd9057f10c8221122d0d3a9a21254adeb448f", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 12eb3c7b2..524065914 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,11 +15,11 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `82ae6877`, a formatting-only successor to -`6e09bbc7`, which adds shared-worker initialization, -logical-name admission and prepared-term proposal fencing after receipt lookup at -`91c3ecfc`, applied-barrier reconciliation at `8a0cf640`, the fixed SQL pool at -`51dff0e5`, the SQL producer at `f807b2f7` and shared-node runtime at `aec4029e`. +The reviewed source baseline is `446dd905`, adding bounded proposal acceptance +and enrollment capacity. Earlier checkpoints cover shared-worker initialization +and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), +applied-barrier reconciliation (`8a0cf640`), the fixed SQL pool (`51dff0e5`), +the SQL producer (`f807b2f7`) and shared-node runtime (`aec4029e`). See the latest checkpoint in [progress](progress.md) for qualification and limits. The branch integrates `origin/main` at `8a93ffd7` (documentation consolidation, PR 161). The design package now lives in `docs/design/multi-raft/cell-groups/`; @@ -93,19 +93,27 @@ initial formatting failure. Formatting-only successor verification is recorded i six checks pass, including formatting, layout, dependencies, docs, preflight and 37 validator tests. These records do not mark CG-04 DONE. -Next implement configured group enrollment capacity admission on the actual owner, -including pending accepted creations rather than only the applied cell count. The -manifest layer already enforces its bounds; admission must refuse before appending -a creation that cannot apply, rather than turn capacity pressure into group failure. -Also complete same-term proposal retry handling: `propose_at_term` fences old -terms but does not deduplicate a repeated mutation envelope. A lost proposal -acceptance response must reuse retained acceptance state or reconcile receipts; -blindly appending the same cell mutation twice can invalidate its second base. -Enrollment's exact retry is idempotent at apply, but its pending capacity accounting -must still count logical creations rather than repeated entries. - -Then integrate catalog routing sessions and registered SDK creation/command -callbacks, and coherent reads. Preserve the original request fingerprint, deadline +The current admission change counts applied cells and accepted pending creations +against `max_cells_per_group` before append. Pending acceptance is keyed by logical +cell name, binds proposal identity to exact bytes, and returns the original +position on a retry. Conflicting bytes refuse; another proposal for the same +pending name waits. Independent cells remain eligible. Once applied, stale bases +and existing enrollments refuse another append and require receipt/creation +resolution. Acceptance metadata has a separate shared node budget so saturation +cannot prevent Ready or completion work. All 195 selected Rust cases and ten +verification commands pass at `446dd905`; the source-bound checkpoint is +`.cellar/cell-groups-evidence/CG-04-proposal-admission/checkpoint.json`, with 187 +source inputs and eight copied executables. See [progress](progress.md) for the +retained intermediate failures and qualification limits. This does not complete +CG-04. + +Next integrate catalog routing sessions and registered SDK creation/command +callbacks, and coherent reads. Admission errors do not yet settle a SQL producer +already marked submitted: an `Overloaded` response may come from a queue before +the owner sees a retry of an earlier unknown attempt. Do not release that producer +from a generic error or same-term receipt absence. Production dispatch needs an +identity-bound owner rejection proof when it can establish non-acceptance, or +must preserve uncertainty and reconcile through the existing applied-barrier path. Preserve the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to satisfy callback/effect identities. The callback APIs added here remain verification boundaries, not registered SDK entry points. Real compaction/production retries @@ -131,6 +139,15 @@ second node worker pool or a task per enrolled cell. A cancelled caller cannot release physical callback ownership or allow a second producer for that cell. +Query integration must also replace `application/consensus/class.rs::QueryContext`'s +`CommittedVersion`/`GroupScope` identity with the logical cell version. The shared +SQL options already carry query services, but the pool has no cell query operation. +`CellExecutionAuthority` and `CellReconciliationAuthority` explicitly grant no +query lease: do not repurpose them as strong-read proof. Bind a ReadIndex result +to its request context, current owner term and applied prefix, then pin the exact +target cell image; another cell's newer log index must not manufacture a cell +version. Preserve the existing registered query transaction restrictions. + ## First executor session diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 28a96c83b..3c1349a37 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 6 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 10 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 9a197a550..1ba6b2767 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,53 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 bounded proposal acceptance and enrollment capacity + +Qualified source: `446dd9057f10c8221122d0d3a9a21254adeb448f`. + +The actual group owner now counts pending accepted creations alongside applied +cells before appending a new +enrollment. Existing cells continue accepting mutations at the configured limit. +Leader-local acceptance records bind a proposal to its exact bytes and original +position; retries cannot append a pending mutation twice. Changed bytes under the +same pending identity refuse, and independent proposals for a pending logical +name wait. Mutation bases are checked against the selected cell image. After +apply, callers resolve receipts or creation records instead of resubmitting stale +images; a refusal does not prove that the original request failed. + +Acceptance retains a separate shared node budget until durable apply, role/term +loss, terminal failure or owner destruction. It retains no LTX payload or image pin and cannot +consume Ready/completion admission. A current-term applied barrier remains +mandatory before a new leader admits work, so inherited enrollments count even +when the previous process's pending map is gone. + +The preliminary native run passed 25 cases and exposed a restart-fixture issue: +restarting only the prior leader leaves live followers' leader leases intact, so +its immediate pre-vote need not elect it. The fixture now restarts all voters from +their retained disks before requiring inherited apply. A later run exposed an +extra async constructor layer overflowing the fixture's default stack; that +layer was removed. A larger-stack diagnostic is retained only for diagnosis, +not qualification. It also exposed the fixture transport discarding overloaded +peer admission. The fixture now retains and retries those exact frames under its +existing deadlines, as the direct-owner fixture already does. All 27 shared-group +execution cases now pass on the normal stack. + +All 195 selected Rust cases pass at the qualified source: 180 native cases across +seven suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy with warnings denied, formatting (1,800 Rust files), layout, +dependency isolation, docs, source preflight and all 37 plan-validator tests pass. +The checkpoint at +`.cellar/cell-groups-evidence/CG-04-proposal-admission/checkpoint.json` binds 187 +source inputs, ten successful command logs and eight copied executables, with +intermediate failures and the larger-stack diagnostic retained separately. Four +process helpers run through parent tests; three RustFS cases remain unrun. + +Four distinct admission cases are now required by the CG-04 executor gate, +bringing the package to 90 required cases. Their oracles are separate from the +existing SQL fairness and receipt-settlement cases. Registered dispatch, catalog +sessions, coherent reads, production retry/compaction and cache lifecycle remain +required. No task status or release claim advances. + ## CG-04 shared-worker initialization and enrollment Implementation source: `6e09bbc7`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 3a2f8eb1d..89cf1fa74 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -47,6 +47,15 @@ term on the RawNode owner atomically with proposal admission. A node that loses leadership and later regains it must reject retained output from its older term. Cancellation retains physical ownership until the callback and cleanup complete. +Before append, reserve group capacity against applied cells plus accepted pending +creations, keyed by logical name. Retrying one creation consumes no extra slot. +Retain bounded leader-local acceptance records independently from Ready/completion +budgets. An exact pending proposal retry returns its original acceptance, never a +second log entry; changed bytes under that identity refuse. After apply or leader +replacement, resolve the receipt/creation record rather than append an old base. +A stale-base refusal does not prove non-commitment. New leaders must apply their +current-term barrier before admitting creations against recovered capacity. + Admission decisions must be checked again in committed log order against the current lifecycle state. Recognizing an exact retry grants neither current-term authority nor permission to restore the initial image over newer mutations. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 08be0f164..ef935aa54 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -460,7 +460,9 @@ "crates/cellar/tests/support/cell_group_sql_reconcile.rs", "crates/cellar/durability/raft/cell_groups/enrollment_authority.rs", "crates/cellar/durability/raft/execution/enrollment.rs", - "crates/cellar/tests/support/cell_group_sql_enrollment.rs" + "crates/cellar/tests/support/cell_group_sql_enrollment.rs", + "crates/cellar/durability/raft/cell_groups/proposal_admission.rs", + "crates/cellar/tests/support/cell_group_admission.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -474,13 +476,17 @@ { "action": "Admit one unresolved proposal per cell with fair cell queues and group/node byte quotas. Allow independent cells to prepare concurrently, then publish committed entries through one contiguous group prefix with per-cell base/version checks. Enforce configured group enrollment capacity on the actual owner before append, counting applied cells and pending accepted logical creations. Exact creation retries consume no additional slot; competing creations cannot exceed the limit before apply. Verify capacity refusal leaves existing cells available and release reservations only with proven resolution.", "tests": [ - "shared_group_fair_admission" + "shared_group_fair_admission", + "pending_enrollments_respect_group_capacity", + "node_runtime_bounds_proposal_acceptance_across_groups" ] }, { "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes. Check the prepared term atomically with proposal admission on the actual Raft owner. Bind same-term acceptance retries to the original proposal identity and bytes; a lost acceptance response must recover retained acceptance or reconcile the original receipt without appending the same mutation again. Reject changed payloads under a reused identity, and retain uncertainty across elections until an applied barrier proves resolution.", "tests": [ - "cell_receipt_not_group_progress" + "cell_receipt_not_group_progress", + "proposal_acceptance_retries_do_not_append_twice", + "inherited_enrollments_count_after_applied_term_barrier" ] }, { @@ -503,14 +509,14 @@ }, { "name": "shared_group_fair_admission", - "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks. Pending distinct enrollments cannot exceed configured group capacity before apply; identical retries reserve once and capacity refusal does not fail the group.", + "oracle": "Concurrent cells receive bounded fair admission; cancellation retains physical accounting and busy SQL never blocks consensus ticks.", "invariants": [ "I12" ] }, { "name": "cell_receipt_not_group_progress", - "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent. Lost proposal acceptance responses and exact retries in the same term append at most one mutation; changed payloads with the same identity refuse and old prepared terms cannot append after leadership returns.", + "oracle": "A later update to B cannot settle or replace A; identical client/sequence in A and B remains independent.", "invariants": [ "I02", "I03" @@ -539,6 +545,42 @@ "I03", "I04" ] + }, + { + "name": "pending_enrollments_respect_group_capacity", + "oracle": "Before any apply, accepted distinct creations fill the configured group capacity; exact retries reserve once, conflicting incarnations refuse and a further creation is rejected before append. Existing cells can still mutate at capacity.", + "invariants": [ + "I01", + "I04", + "I12" + ] + }, + { + "name": "proposal_acceptance_retries_do_not_append_twice", + "oracle": "An exact pending mutation retry returns its original log position and digest at saturated acceptance capacity; changed bytes under that identity refuse. After apply, a stale base cannot append again, the original image advances once on every voter and released capacity admits another cell.", + "invariants": [ + "I02", + "I03", + "I04", + "I12" + ] + }, + { + "name": "inherited_enrollments_count_after_applied_term_barrier", + "oracle": "Restart all voters from retained disks with an uncommitted enrollment present only on the longest log. The recovered leader refuses admission until inherited history and its current-term barrier apply, then counts that cell against capacity. Old prepared terms refuse.", + "invariants": [ + "I01", + "I04", + "I12" + ] + }, + { + "name": "node_runtime_bounds_proposal_acceptance_across_groups", + "oracle": "Pending acceptance in one physical group exhausts the shared node budget and refuses a different group, while exact retries retain their original position. Peer persistence and durable apply still progress under saturation, releasing capacity for the other group on all voters.", + "invariants": [ + "I04", + "I12" + ] } ], "invariants": [ From 0cf89c403f4a500d18116499b8fc0f219195cd54 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 22:07:14 -0700 Subject: [PATCH 061/568] feat: seal rejected shared-group SQL submissions before cleanup --- crates/cellar/application/consensus/mod.rs | 8 +- crates/cellar/durability/raft/README.md | 28 +- .../raft/cell_groups/enrollment_authority.rs | 4 + .../raft/cell_groups/execution_authority.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 2 +- .../durability/raft/cell_groups/owner.rs | 92 ++++-- .../raft/cell_groups/runtime/mod.rs | 35 +++ .../raft/cell_groups/runtime/worker.rs | 3 + .../cellar/durability/raft/execution/cell.rs | 46 +-- .../durability/raft/execution/enrollment.rs | 2 + .../cellar/durability/raft/execution/mod.rs | 7 +- .../durability/raft/execution/submission.rs | 162 +++++++++++ .../raft/host_sql/cell_groups/mod.rs | 62 ++++- .../raft/host_sql/cell_groups/worker.rs | 26 +- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 28 +- crates/cellar/tests/support/cell_group_sql.rs | 35 ++- .../support/cell_group_sql_enrollment.rs | 13 +- .../tests/support/cell_group_sql_pool.rs | 8 +- .../tests/support/cell_group_sql_reconcile.rs | 16 +- .../support/cell_group_sql_submission.rs | 263 ++++++++++++++++++ 21 files changed, 739 insertions(+), 115 deletions(-) create mode 100644 crates/cellar/durability/raft/execution/submission.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_submission.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 0b62fa8f4..ca7128447 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -159,10 +159,10 @@ pub mod verification { pub use crate::raft::execution::{ query_registered, AppliedExecution, CellCommittedReceipt, CellEnrollmentAttempt, CellEnrollmentResolution, CellPreparedCommand, CellPrivateEnrollment, CellPrivateExecutor, - CellReconciliation, CellStageOutcome, CommandAttempt, CommandAuthority, CommandRequest, - CommittedReceipt, ExecutionError, ExecutionLimits, ExecutionStatus, PreparedCommand, - PrivateExecutor, PrivateOptions, RegisteredQueryOutput, RootPrivateOptions, StageOutcome, - StoredReceipt, + CellProposalRejection, CellReconciliation, CellStageOutcome, CellSubmissionOutcome, + CommandAttempt, CommandAuthority, CommandRequest, CommittedReceipt, ExecutionError, + ExecutionLimits, ExecutionStatus, PreparedCommand, PrivateExecutor, PrivateOptions, + RegisteredQueryOutput, RootPrivateOptions, StageOutcome, StoredReceipt, }; pub use crate::raft::fleet_catalog::{ FleetBirthAuthorization, FleetBirthReservation, FleetCatalog, FleetCatalogError, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 7b4324f7b..bfa8c6ccd 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -99,11 +99,24 @@ the initializer. Admission keys the logical cluster/namespace/cell name, so anot incarnation cannot bypass an in-flight producer. Ordinary commands use the same name reservation. Queued and running initialization retain it through cancellation. -Prepared SQL output carries its original leader term. Submit it through -`propose_at_term`: the actual RawNode owner checks that fence atomically with -proposal admission. Merely sampling leader status before queueing is insufficient, -including when the original node later regains leadership. The unfenced `propose` -method remains a low-level protocol verification boundary. +Prepared SQL output carries its original leader term and engine/group identity. +Use `CellSqlPrepared::submit` (or `CellGroupRuntime::submit` for standalone +verification producers). The actual RawNode owner checks those bindings before +new admission. Acceptance is remembered by that exact preparation, so a lost +response can return its original position even after apply; it never proves a +commit or S3 coverage. The first valid prepared submission irreversibly disables +byte-only `propose`/`propose_at_term` for that owner, including raw requests already +queued behind it. This prevents delayed byte-only retries bypassing a seal. + +Only a definitive owner capacity refusal seals the preparation as rejected. +Generic queue overload, stale authority and missing responses grant no such +proof. Every clone shares the seal and can recover it through `rejection()`, even +if the original rejection response was lost. Returning capacity cannot unseal +old output. The pool's `reject` progress operation matches the exact preparation, +closes its private writer, then releases logical-name admission. It rejects +foreign engines and proofs for another preparation, including fresh retries with +the same proposal ID. Retained prepared outputs, queued submissions and rejection +proofs keep their SQL byte reservations through cancellation and delivery. The owner retains a bounded acceptance record for each pending logical cell name. An exact proposal retry returns its original position and digest; different bytes @@ -118,8 +131,9 @@ Acceptance records retain a separate node-wide budget of `max_requests` items an image pins. This budget cannot block Ready or completion work. Records release only after exact durable apply, role/term loss, terminal failure, or owner destruction. A new leader cannot admit until its applied current-term barrier covers inherited entries. -After a record releases, stale mutation bases and existing enrollments return -`StaleCompletion`: callers must resolve the original receipt or creation record. +Without retained acceptance for that preparation, stale mutation bases and +existing enrollments return `StaleCompletion`: callers must resolve the original +receipt or creation record. That refusal does not establish that the prior attempt failed. A committed creation retry returns the immutable enrollment record without opening diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs index 7a7b86314..0c31e846a 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -22,6 +22,10 @@ pub struct CellEnrollmentState { } impl CellEnrollmentState { + pub(crate) fn engine_owner(&self) -> &Arc<()> { + &self.group.owner + } + pub(super) fn new( group: GroupLog, current: &PreparedGroupManifest, diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index 29e473485..74116d80d 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -25,6 +25,10 @@ pub struct CellExecutionAuthority { } impl CellExecutionAuthority { + pub(crate) fn engine_owner(&self) -> &Arc<()> { + &self.group.owner + } + pub(crate) fn accepts_reconciliation(&self, proof: &CellReconciliationAuthority) -> bool { Arc::ptr_eq(&self.group.owner, &proof.group.owner) && self.route == proof.route diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 94918ef75..e65781491 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -5,7 +5,7 @@ pub use super::host_sql::cell_groups::{ CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlReceipt, CellSqlReconciliation, - CellSqlRootOptions, CellSqlUsage, + CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, }; mod execution_authority; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 4fe9f6ccf..37c70c4d6 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -164,6 +164,7 @@ pub struct CellGroupOwner { failed: Option, proposals: super::proposal_admission::ProposalAdmission, pub(super) proposal_budget: Arc, + prepared_only: bool, } impl CellGroupOwner { @@ -248,6 +249,7 @@ impl CellGroupOwner { last_sync: 0, failed: None, proposals: Default::default(), + prepared_only: false, }) } @@ -373,8 +375,37 @@ impl CellGroupOwner { /// Protocol admission only. The shared cell executor must serialize each /// cell's preparation, reserve physical work and retain ambiguous outcomes. - /// SQL producers use `propose_at_term` with their captured preparation term. + /// The first prepared submission permanently closes this byte-only entry + /// point for this owner, including already queued raw retries. pub fn propose(&mut self, data: Vec) -> Result { + if self.prepared_only { + return Err(RuntimeError::Invalid("prepared submission required".into())); + } + self.try_propose(data)?.ok_or(RuntimeError::Overloaded) + } + + /// Seals a definitively refused preparation against every delayed clone. + /// Queue admission errors never manufacture a rejection proof. + pub fn submit( + &mut self, + prepared: crate::raft::execution::CellPreparedCommand, + ) -> Result { + let engine = self.group.owner.clone(); + let group = self.group.identity().scope; + if !prepared.belongs_to(&engine, group) { + return Err(RuntimeError::Invalid("prepared submission owner".into())); + } + self.prepared_only = true; + prepared.submit(&engine, group, || { + self.live()?; + if prepared.term().get() != self.node.raft.term { + return Err(RuntimeError::StaleCompletion); + } + self.try_propose(prepared.bytes().to_vec()) + }) + } + + fn try_propose(&mut self, data: Vec) -> Result, RuntimeError> { if !self.status()?.ready { return Err(RuntimeError::NotReady); } @@ -387,37 +418,60 @@ impl CellGroupOwner { GroupEntryLimits::default(), ) .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - let admission = self.proposals.check( + let admission = match self.proposals.check( &entry, &data, &self.current, self.options.max_cells_per_group, self.options.max_requests, &self.proposal_budget, - )?; + ) { + Ok(admission) => admission, + Err(RuntimeError::Overloaded) => return Ok(None), + Err(error) => return Err(error), + }; let (candidate, permit) = match admission { - super::proposal_admission::Admission::Existing(accepted) => return Ok(accepted), + super::proposal_admission::Admission::Existing(accepted) => return Ok(Some(accepted)), super::proposal_admission::Admission::New(candidate, permit) => (candidate, permit), }; let before = self.node.raft.raft_log.last_index(); - self.node.propose(Vec::new(), data)?; - let entry = self - .node - .raft - .raft_log - .unstable_entries() - .last() - .ok_or(RuntimeError::NotReady)?; - if entry.index != before + 1 { - return Err(RuntimeError::NotReady); + if let Err(error) = self.node.propose(Vec::new(), data) { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); } - let accepted = CellGroupProposal { - group: self.group.identity().scope, - position: entry_position(entry), - digest: entry_digest(entry)?, + let observed = (|| { + let entry = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol("missing proposed entry".into()))?; + if entry.index != before + 1 { + return Err(RuntimeError::Protocol("proposal append position".into())); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(entry), + digest: entry_digest(entry)?, + }) + })(); + let accepted = match observed { + Ok(accepted) => accepted, + Err(error) => { + // Append was attempted. Never turn an uncertain local result + // into permission to append again or a rejection proof. + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } }; self.proposals.accept(candidate, accepted.clone(), permit); - Ok(accepted) + Ok(Some(accepted)) } /// Check the SQL preparation fence on the actual RawNode owner in the same diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 02c5fdd76..d5fec918f 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -92,6 +92,10 @@ enum Command { AdmitStep(Message), Propose(Vec), ProposeAtTerm(cellar_logic::consensus::Term, Vec), + Submit( + crate::raft::execution::CellPreparedCommand, + Option>, + ), ReadIndex(Vec), Apply(CellGroupTicket, Box), Status, @@ -101,6 +105,7 @@ enum Command { enum Response { Unit, Proposal(CellGroupProposal), + Submission(crate::raft::execution::CellSubmissionOutcome), Execution(Box), Enrollment(Box), EnrollmentState(Box), @@ -118,6 +123,36 @@ struct Request { } impl CellGroupRuntime { + pub async fn submit( + &self, + prepared: crate::raft::execution::CellPreparedCommand, + ) -> Result { + self.submit_retained(prepared, None).await + } + + pub(crate) async fn submit_retained( + &self, + prepared: crate::raft::execution::CellPreparedCommand, + retained: Option>, + ) -> Result { + let bytes = prepared.bytes().len(); + if bytes > self.inner.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let group = prepared.group(); + match self + .call( + Some(group), + Command::Submit(prepared, retained), + bytes + 4096, + ) + .await? + { + Response::Submission(outcome) => Ok(outcome), + _ => Err(RuntimeError::Protocol("prepared submission reply".into())), + } + } + /// Start parked: no recovered group is attached without its verified root /// and catalog/member permit. The engine admits only one node runtime. pub async fn start( diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 6def095bd..ea1f5fdd0 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -355,6 +355,9 @@ impl Worker { .owner .propose_at_term(term, data) .map(Response::Proposal), + Command::Submit(prepared, _retained) => { + group.owner.submit(prepared).map(Response::Submission) + } Command::ReadIndex(context) => { group.owner.read_index(context).map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 16bdf0b0a..57f3a5d45 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -12,36 +12,6 @@ use crate::raft::cell_groups::{ }; use crate::raft::completion::RuntimeError; use cellar_logic::cell_group::CellVersion; -use std::sync::Arc; - -#[derive(Clone)] -pub struct CellPreparedCommand { - bytes: Arc>, - proposal: ProposalId, - term: Term, -} - -impl CellPreparedCommand { - pub(super) fn new(bytes: Vec, proposal: ProposalId, term: Term) -> Self { - Self { - bytes: Arc::new(bytes), - proposal, - term, - } - } - - pub fn term(&self) -> Term { - self.term - } - - pub fn bytes(&self) -> &[u8] { - &self.bytes - } - - pub fn proposal(&self) -> ProposalId { - self.proposal - } -} /// A receipt read from the exact committed cell image. This is not S3 coverage. pub struct CellCommittedReceipt { @@ -248,11 +218,13 @@ impl CellPrivateExecutor { ) .map_err(storage)?; self.authority.check().map_err(authority_failure)?; - let prepared = CellPreparedCommand { - bytes: Arc::new(bytes), - proposal: attempt.proposal, - term: self.authority.term(), - }; + let prepared = CellPreparedCommand::new( + bytes, + attempt.proposal, + self.authority.term(), + route.history, + self.authority.engine_owner().clone(), + ); self.pending = Some(PendingCell { prepared: prepared.clone(), request: attempt.request, @@ -276,7 +248,7 @@ impl CellPrivateExecutor { .pending .as_mut() .ok_or(ExecutionError::Invalid("no cell proposal"))?; - if pending.submitted || pending.prepared.proposal != proposal { + if pending.submitted || pending.prepared.proposal() != proposal { return Err(ExecutionError::Invalid("cell proposal submission")); } pending.submitted = true; @@ -319,7 +291,7 @@ impl CellPrivateExecutor { authority: CellReconciliationAuthority, ) -> Result { let pending = self.pending.as_ref().ok_or(ExecutionError::Uncertain)?; - if pending.prepared.proposal != proposal + if pending.prepared.proposal() != proposal || !self.authority.accepts_reconciliation(&authority) { return Err(ExecutionError::Invalid("cell reconciliation identity")); diff --git a/crates/cellar/durability/raft/execution/enrollment.rs b/crates/cellar/durability/raft/execution/enrollment.rs index 6b1ac07e0..355f7287a 100644 --- a/crates/cellar/durability/raft/execution/enrollment.rs +++ b/crates/cellar/durability/raft/execution/enrollment.rs @@ -177,6 +177,8 @@ impl CellPrivateEnrollment { bytes, attempt.creation, state.term(), + state.route().history, + state.engine_owner().clone(), )) } diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index fef0a22f3..52b900ce2 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -8,15 +8,14 @@ use super::cache::{DurableApply, GroupLog}; use super::completion::GroupStatus; use super::receipts; mod cell; +mod submission; pub(crate) use cell::authority_failure; +pub use submission::{CellPreparedCommand, CellProposalRejection, CellSubmissionOutcome}; mod command; mod enrollment; pub use enrollment::{CellEnrollmentAttempt, CellEnrollmentResolution, CellPrivateEnrollment}; mod images; -pub use cell::{ - CellCommittedReceipt, CellPreparedCommand, CellPrivateExecutor, CellReconciliation, - CellStageOutcome, -}; +pub use cell::{CellCommittedReceipt, CellPrivateExecutor, CellReconciliation, CellStageOutcome}; mod rooted_working; use crate::application::consensus::{ command_request_digest, CommittedVersion, EnvelopeLimits, OperationContract, OperationKind, diff --git a/crates/cellar/durability/raft/execution/submission.rs b/crates/cellar/durability/raft/execution/submission.rs new file mode 100644 index 000000000..6bb79e192 --- /dev/null +++ b/crates/cellar/durability/raft/execution/submission.rs @@ -0,0 +1,162 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A prepared SQL attempt is accepted once or permanently sealed before append. + +use crate::raft::{cell_groups::CellGroupProposal, completion::RuntimeError}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{ProposalId, Term}, +}; +use std::sync::{Arc, Mutex}; + +#[derive(Default)] +enum Decision { + #[default] + Open, + Submitting, + Accepted(CellGroupProposal), + Rejected, +} + +struct Submission { + engine: Arc<()>, + decision: Mutex, +} + +#[derive(Clone)] +pub struct CellPreparedCommand { + bytes: Arc>, + proposal: ProposalId, + term: Term, + group: GroupHistory, + submission: Arc, +} + +/// This preparation was sealed before append. It is not a claim about other +/// preparations of the same business request. All clones share the same seal. +#[derive(Clone)] +pub struct CellProposalRejection { + submission: Arc, +} + +pub enum CellSubmissionOutcome { + /// Acceptance remains provisional, never a commit or S3 acknowledgement. + Accepted(CellGroupProposal), + Rejected(CellProposalRejection), +} + +impl CellPreparedCommand { + pub(super) fn new( + bytes: Vec, + proposal: ProposalId, + term: Term, + group: GroupHistory, + engine: Arc<()>, + ) -> Self { + Self { + bytes: Arc::new(bytes), + proposal, + term, + group, + submission: Arc::new(Submission { + engine, + decision: Mutex::new(Decision::Open), + }), + } + } + + pub fn term(&self) -> Term { + self.term + } + + pub fn bytes(&self) -> &[u8] { + &self.bytes + } + + pub fn proposal(&self) -> ProposalId { + self.proposal + } + + pub(crate) fn group(&self) -> GroupHistory { + self.group + } + + pub(crate) fn belongs_to(&self, engine: &Arc<()>, group: GroupHistory) -> bool { + Arc::ptr_eq(engine, &self.submission.engine) && group == self.group + } + + pub(crate) fn submit( + &self, + engine: &Arc<()>, + group: GroupHistory, + attempt: impl FnOnce() -> Result, RuntimeError>, + ) -> Result { + if !self.belongs_to(engine, group) { + return Err(RuntimeError::Invalid("prepared submission owner".into())); + } + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + match &*decision { + Decision::Accepted(accepted) => { + return Ok(CellSubmissionOutcome::Accepted(accepted.clone())) + } + Decision::Rejected => return Ok(CellSubmissionOutcome::Rejected(self.rejected())), + Decision::Submitting => return Err(RuntimeError::NotReady), + Decision::Open => (), + } + *decision = Decision::Submitting; + drop(decision); + // No lock spans decoding, hashing, RawNode work or async operations. + // Only the owner's definitive pre-append capacity refusal returns None. + let outcome = attempt(); + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + match outcome { + Ok(Some(accepted)) => { + *decision = Decision::Accepted(accepted.clone()); + Ok(CellSubmissionOutcome::Accepted(accepted)) + } + Ok(None) => { + *decision = Decision::Rejected; + Ok(CellSubmissionOutcome::Rejected(self.rejected())) + } + Err(error) => { + *decision = Decision::Open; + Err(error) + } + } + } + + fn rejected(&self) -> CellProposalRejection { + CellProposalRejection { + submission: self.submission.clone(), + } + } + + /// Recover a lost rejection response without asking a closed or replaced + /// runtime. Open and accepted attempts never yield a rejection proof. + pub fn rejection(&self) -> Result, RuntimeError> { + let decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + Ok(matches!(*decision, Decision::Rejected).then(|| self.rejected())) + } + + pub(crate) fn matches_rejection(&self, rejection: &CellProposalRejection) -> bool { + Arc::ptr_eq(&self.submission, &rejection.submission) + } +} + +impl CellProposalRejection { + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(engine, &self.submission.engine) + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index b2c18be32..06e48d72c 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -13,8 +13,9 @@ use crate::raft::cell_groups::{ use crate::raft::completion::RuntimeError; use crate::raft::execution::{ CellCommittedReceipt, CellEnrollmentAttempt, CellEnrollmentResolution, CellPreparedCommand, - CellPrivateEnrollment, CellPrivateExecutor, CellReconciliation, CellStageOutcome, - CommandAttempt, ExecutionError, ExecutionLimits, PrivateOptions, RootPrivateOptions, + CellPrivateEnrollment, CellPrivateExecutor, CellProposalRejection, CellReconciliation, + CellStageOutcome, CellSubmissionOutcome, CommandAttempt, ExecutionError, ExecutionLimits, + PrivateOptions, RootPrivateOptions, }; use crate::raft::runtime::Terminal; use crate::raft::storage::ReplicaLog; @@ -166,6 +167,31 @@ pub struct CellSqlPrepared { } impl CellSqlPrepared { + pub async fn submit( + &self, + runtime: &crate::raft::cell_groups::CellGroupRuntime, + ) -> Result { + match runtime + .submit_retained(self.prepared.clone(), Some(self._bytes.clone())) + .await? + { + CellSubmissionOutcome::Accepted(accepted) => Ok(CellSqlSubmission::Accepted(accepted)), + CellSubmissionOutcome::Rejected(proof) => { + Ok(CellSqlSubmission::Rejected(CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + } + } + + pub fn rejection(&self) -> Result, RuntimeError> { + Ok(self.prepared.rejection()?.map(|proof| CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + pub fn term(&self) -> cellar_logic::consensus::Term { self.prepared.term() } @@ -179,6 +205,18 @@ impl CellSqlPrepared { } } +pub enum CellSqlSubmission { + Accepted(crate::raft::cell_groups::CellGroupProposal), + Rejected(CellSqlRejection), +} + +/// Retains the original SQL output budget through delivery and lost responses. +#[derive(Clone)] +pub struct CellSqlRejection { + proof: CellProposalRejection, + _bytes: Arc, +} + pub struct CellSqlReceipt { receipt: CellCommittedReceipt, _bytes: Arc, @@ -197,6 +235,7 @@ pub enum CellSqlOutcome { Reconciled(CellSqlReconciliation), Enrollment(Box), EnrollmentResolved(CellSqlEnrollmentResolution), + Rejected, } pub enum CellSqlReconciliation { @@ -519,6 +558,25 @@ impl CellGroupSqlPool { } } + /// Close the private writer only for this exact, irreversibly sealed + /// preparation. A generic error or another producer's proof cannot do this. + pub async fn reject( + &self, + cell: CellHistory, + rejection: CellSqlRejection, + ) -> Result<(), ExecutionError> { + if !rejection.proof.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("cell rejection engine")); + } + match self + .call(cell, worker::Action::Reject(Box::new(rejection))) + .await? + { + CellSqlOutcome::Rejected => Ok(()), + _ => Err(ExecutionError::Invalid("cell rejection response")), + } + } + pub async fn finish( &self, authority: CellExecutionAuthority, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index 686dadcba..48b59b289 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -34,6 +34,7 @@ pub(super) enum Action { Submit(ProposalId), Finish(Box), Reconcile(ProposalId, Box), + Reject(Box), } impl Action { @@ -120,9 +121,10 @@ impl Cells { Action::Stage(_) | Action::Initialize(_) => { crate::raft::performance::Operation::SqlStage } - Action::Finish(_) | Action::Reconcile(..) | Action::ResolveEnrollment(..) => { - crate::raft::performance::Operation::SqlPublish - } + Action::Finish(_) + | Action::Reconcile(..) + | Action::ResolveEnrollment(..) + | Action::Reject(_) => crate::raft::performance::Operation::SqlPublish, _ => crate::raft::performance::Operation::SqlOther, }; let timer = performance.timer(operation); @@ -306,6 +308,24 @@ impl Cells { } Ok(CellSqlOutcome::Prepared(pending.prepared.clone())) } + Action::Reject(rejection) => { + let pending = self.pending.get(&cell).ok_or(ExecutionError::Uncertain)?; + if !pending + .prepared + .prepared + .matches_rejection(&rejection.proof) + { + return Err(ExecutionError::Invalid("cell rejection preparation")); + } + let pending = self + .pending + .remove(&cell) + .ok_or(ExecutionError::Uncertain)?; + // The proof permanently seals every submission clone. Close + // physical SQL ownership before dropping logical-name admission. + pending.producer.close()?; + Ok(CellSqlOutcome::Rejected) + } Action::Submit(proposal) => { let pending = self .pending diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index d0e6beba5..f65a0120b 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -11,6 +11,16 @@ mod fault; #[path = "support/cell_group_runtime.rs"] mod runtime_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn sql_submission_rejection_seals_delayed_retries() -> Result<()> { + runtime_fixture::submission_rejections(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_sql_submission_rejection_seals_delayed_retries() -> Result<()> { + runtime_fixture::submission_rejections(true).await +} + #[path = "support/cell_group_admission.rs"] mod admission_fixture; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index fd473fb0c..e925975f0 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -16,10 +16,14 @@ struct RuntimeCluster { impl RuntimeCluster { async fn new() -> Result { - Self::with_limits(256, 256).await + Self::with_limits(256, 256, 1_000_000).await } - async fn with_limits(max_events: usize, max_requests: usize) -> Result { + async fn with_limits( + max_events: usize, + max_requests: usize, + max_cells_per_group: u32, + ) -> Result { let mut cluster = Cluster::new().await?; let mut runtimes = Vec::new(); let mut events = Vec::new(); @@ -36,6 +40,7 @@ impl RuntimeCluster { election_seed: Some([71; 32]), max_events, max_requests, + max_cells_per_group, ..Default::default() }, ) @@ -145,7 +150,16 @@ impl RuntimeCluster { tokio::time::timeout(Duration::from_secs(20), async { loop { self.pump(&[], &mut BTreeMap::new()).await?; - let status = self.runtimes[0].status(history).await?; + let status = match self.runtimes[0].status(history).await { + Ok(status) => status, + Err(RuntimeError::Overloaded) => { + // With one ordinary slot, an admitted peer frame can + // still own it. Keep pumping under the existing deadline. + tokio::task::yield_now().await; + continue; + } + Err(error) => return Err(error.into()), + }; let target = target.unwrap_or(status.applied.index); if status.ready && self.applied[&history].iter().all(|index| *index >= target) { return Ok::<_, anyhow::Error>(()); @@ -373,7 +387,7 @@ pub(super) async fn shared_batches() -> Result<()> { } pub(super) async fn shared_proposal_budget() -> Result<()> { - let mut fixture = RuntimeCluster::with_limits(256, 8).await?; + let mut fixture = RuntimeCluster::with_limits(256, 8, 1_000_000).await?; let first = fixture.cluster.birth.route().history; let second = fixture.add_group(61).await?; for scope in [first, second] { @@ -433,7 +447,7 @@ pub(super) async fn shared_proposal_budget() -> Result<()> { } pub(super) async fn bounded_apply_completion() -> Result<()> { - let mut fixture = RuntimeCluster::with_limits(2, 256).await?; + let mut fixture = RuntimeCluster::with_limits(2, 256, 1_000_000).await?; let first = fixture.cluster.birth.route().history; let second = fixture.add_group(51).await?; let histories = [first, second]; @@ -606,3 +620,7 @@ pub(super) async fn initialize_cells(rooted: bool) -> Result<()> { pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { sql::recover_initialization(rooted).await } + +pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { + sql::submission_rejections(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 22bac0ded..5b4c759e3 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -9,6 +9,26 @@ use cellar_ltx::internal::{ }; use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; +async fn accepted_sql( + runtime: &CellGroupRuntime, + prepared: &CellSqlPrepared, +) -> Result { + match prepared.submit(runtime).await? { + CellSqlSubmission::Accepted(accepted) => Ok(accepted), + CellSqlSubmission::Rejected(_) => anyhow::bail!("unexpected capacity rejection"), + } +} + +async fn accepted_private( + runtime: &CellGroupRuntime, + prepared: &CellPreparedCommand, +) -> Result { + match runtime.submit(prepared.clone()).await? { + CellSubmissionOutcome::Accepted(accepted) => Ok(accepted), + CellSubmissionOutcome::Rejected(_) => anyhow::bail!("unexpected private rejection"), + } +} + fn cell_attempt(tag: u8) -> Result { Ok(CommandAttempt { request: CommandRequest { @@ -234,9 +254,7 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { .await??; eb = eb_out; // Both use the same client/sequence, but B's committed image cannot settle A. - let proposed = fixture.runtimes[0] - .propose(route.history, pb.bytes().to_vec()) - .await?; + let proposed = accepted_private(&fixture.runtimes[0], &pb).await?; fixture.drive(Some(proposed.position.index)).await?; let cb = fixture.runtimes[0] .execution_authority(route.history, b.clone()) @@ -256,9 +274,7 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { ea = ea_out; eb = eb_out; // A prepared before B committed remains valid because its own base is unchanged. - let proposed = fixture.runtimes[0] - .propose(route.history, pa.bytes().to_vec()) - .await?; + let proposed = accepted_private(&fixture.runtimes[0], &pa).await?; fixture.drive(Some(proposed.position.index)).await?; let ca = fixture.runtimes[0] .execution_authority(route.history, a.clone()) @@ -446,3 +462,10 @@ pub(super) async fn reconcile_producers(rooted: bool) -> Result<()> { pub(super) async fn reconcile_refusals(rooted: bool) -> Result<()> { reconciliation::refusals(rooted).await } + +#[path = "cell_group_sql_submission.rs"] +mod submission; + +pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { + submission::rejections(rooted).await +} diff --git a/crates/cellar/tests/support/cell_group_sql_enrollment.rs b/crates/cellar/tests/support/cell_group_sql_enrollment.rs index 44184f383..d52063c56 100644 --- a/crates/cellar/tests/support/cell_group_sql_enrollment.rs +++ b/crates/cellar/tests/support/cell_group_sql_enrollment.rs @@ -43,7 +43,7 @@ async fn cluster(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { Ok((fixture, route)) } -async fn prepare_initial( +pub(super) async fn prepare_initial( fixture: &RuntimeCluster, pool: &CellGroupSqlPool, node: usize, @@ -79,16 +79,13 @@ async fn submit( cell: &CellHistory, prepared: &CellSqlPrepared, ) -> Result<()> { - let group = fixture.cluster.birth.route().history; pool.mark_submitted(cell.clone(), prepared.proposal()) .await?; - let accepted = fixture.runtimes[node] - .propose_at_term(group, prepared.term(), prepared.bytes().to_vec()) - .await?; + let accepted = accepted_sql(&fixture.runtimes[node], prepared).await?; apply_through(fixture, accepted.position.index).await } -async fn resolved( +pub(super) async fn resolved( fixture: &RuntimeCluster, pool: &CellGroupSqlPool, node: usize, @@ -412,9 +409,7 @@ pub(super) async fn recovery(rooted: bool) -> Result<()> { // merely checking that the receiver is a leader would reject this race. transfer(&mut fixture, 1, 0).await?; assert!(matches!( - fixture.runtimes[0] - .propose_at_term(route.history, pe.term(), pe.bytes().to_vec()) - .await, + pe.submit(&fixture.runtimes[0]).await, Err(RuntimeError::StaleCompletion) )); assert_eq!( diff --git a/crates/cellar/tests/support/cell_group_sql_pool.rs b/crates/cellar/tests/support/cell_group_sql_pool.rs index 3297555e1..66bb609da 100644 --- a/crates/cellar/tests/support/cell_group_sql_pool.rs +++ b/crates/cellar/tests/support/cell_group_sql_pool.rs @@ -131,9 +131,7 @@ pub(super) async fn admission(rooted: bool) -> Result<()> { // B completes through the same consensus group while A's physical SQL is held. pool.mark_submitted(b.clone(), pb.proposal()).await?; pool.mark_submitted(b.clone(), pb.proposal()).await?; - let proposal = fixture.runtimes[0] - .propose(route.history, pb.bytes().to_vec()) - .await?; + let proposal = accepted_sql(&fixture.runtimes[0], &pb).await?; fixture.drive(Some(proposal.position.index)).await?; let cb = fixture.runtimes[0] .execution_authority(route.history, b.clone()) @@ -147,9 +145,7 @@ pub(super) async fn admission(rooted: bool) -> Result<()> { release.send(())?; let pa = tokio::time::timeout(Duration::from_secs(5), pool.resume(a.clone(), pa)).await??; pool.mark_submitted(a.clone(), pa.proposal()).await?; - let proposal = fixture.runtimes[0] - .propose(route.history, pa.bytes().to_vec()) - .await?; + let proposal = accepted_sql(&fixture.runtimes[0], &pa).await?; fixture.drive(Some(proposal.position.index)).await?; let ca = fixture.runtimes[0] .execution_authority(route.history, a.clone()) diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs index 16ac5d924..6a48a7314 100644 --- a/crates/cellar/tests/support/cell_group_sql_reconcile.rs +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -95,9 +95,7 @@ pub(super) async fn commit_on( let group = fixture.cluster.birth.route().history; pool.mark_submitted(cell.clone(), prepared.proposal()) .await?; - let accepted = fixture.runtimes[node] - .propose_at_term(group, prepared.term(), prepared.bytes().to_vec()) - .await?; + let accepted = accepted_sql(&fixture.runtimes[node], &prepared).await?; apply_through(fixture, accepted.position.index).await?; let authority = fixture.runtimes[node] .execution_authority(group, cell.clone()) @@ -151,9 +149,7 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { let stale_b = fixture.runtimes[0] .reconciliation_authority(route.history, b.clone()) .await?; - let submitted = fixture.runtimes[0] - .propose(route.history, prepared[1].bytes().to_vec()) - .await?; + let submitted = accepted_sql(&fixture.runtimes[0], &prepared[1]).await?; fixture.drive(Some(submitted.position.index)).await?; assert!( stale_b.check().is_err(), @@ -354,9 +350,7 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { anyhow::bail!("retry A") }; pool.mark_submitted(a.clone(), command.proposal()).await?; - let submitted = fixture.runtimes[0] - .propose(route.history, command.bytes().to_vec()) - .await?; + let submitted = accepted_sql(&fixture.runtimes[0], &command).await?; fixture.drive(Some(submitted.position.index)).await?; // A positive same-term receipt resolves without requiring a term change. let proof = fixture.runtimes[0] @@ -412,9 +406,7 @@ pub(super) async fn refusals(rooted: bool) -> Result<()> { pending.push(prepared); } // B commits but its response is lost; A's original proposal never commits. - let accepted = fixture.runtimes[0] - .propose(route.history, pending[1].bytes().to_vec()) - .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &pending[1]).await?; apply_through(&mut fixture, accepted.position.index).await?; transfer(&mut fixture, 0, 1).await?; let successor = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits)?; diff --git a/crates/cellar/tests/support/cell_group_sql_submission.rs b/crates/cellar/tests/support/cell_group_sql_submission.rs new file mode 100644 index 000000000..cac2fb2f9 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_submission.rs @@ -0,0 +1,263 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::apply_through; +use super::*; + +async fn initial( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + cell: &CellHistory, + creation: u8, + preview: u8, +) -> Result { + let authority = fixture.runtimes[0] + .enrollment_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .initialize( + fixture.cell_options(cell).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([creation; 16])?, + preview: ImageId::new([preview; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + |tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES (0);")?; + Ok(()) + }, + ) + .await? + else { + anyhow::bail!("new enrollment") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + Ok(prepared) +} + +async fn mutation( + fixture: &RuntimeCluster, + pool: &CellGroupSqlPool, + cell: &CellHistory, + proposal: u8, + preview: u8, +) -> Result { + let authority = fixture.runtimes[0] + .execution_authority(fixture.cluster.birth.route().history, cell.clone()) + .await?; + let mut attempt = cell_attempt(proposal)?; + attempt.preview = ImageId::new([preview; 16])?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage( + fixture.cell_options(cell).await?, + authority, + attempt, + true, + |tx| { + let value: i64 = tx.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(value, 0, "rejected private effects must not be reused"); + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + ) + .await? + else { + anyhow::bail!("new mutation") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + Ok(prepared) +} + +pub(super) async fn rejections(rooted: bool) -> Result<()> { + let mut fixture = RuntimeCluster::with_limits(256, 1, 2).await?; + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("submission-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let limits = CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 4, + group_pending_cells: 4, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }; + let pool = CellGroupSqlPool::start(&fixture.cluster.voters[0].log, limits)?; + let a = history(&route, "submission-a")?; + let b = history(&route, "submission-b")?; + let c = history(&route, "submission-c")?; + let pa = initial(&fixture, &pool, &a, 181, 181).await?; + let pb = initial(&fixture, &pool, &b, 182, 182).await?; + + // Drop the acceptance response, then hold real engine sync so a queued + // ReadIndex occupies the only ordinary request slot. Queue overload must + // not produce a false owner rejection for B. + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + drop(pa.submit(&fixture.runtimes[0]).await?); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let mut waiting = Box::pin(fixture.runtimes[0].read_index(route.history, vec![77])); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + assert!(matches!( + pb.submit(&fixture.runtimes[0]).await, + Err(RuntimeError::Overloaded) + )); + assert!(pb.rejection()?.is_none()); + assert!(pa.rejection()?.is_none()); + assert_eq!(pool.usage()?.active_cells, 2); + drop(release); + tokio::time::timeout(Duration::from_secs(5), waiting).await??; + let accepted_a = accepted_sql(&fixture.runtimes[0], &pa).await?; + assert!(pa.rejection()?.is_none()); + let state = fixture.runtimes[0] + .enrollment_state(route.history, b.clone()) + .await?; + assert!(matches!( + pool.resolve_enrollment(pb.proposal(), state).await, + Err(ExecutionError::Uncertain) + )); + + // The actual owner now refuses B against A's retained acceptance slot. + // Discard the response and recover its irreversible seal from the producer. + drop(pb.submit(&fixture.runtimes[0]).await?); + let proof_b = pb.rejection()?.context("sealed B")?; + assert!(pool.reject(a.clone(), proof_b.clone()).await.is_err()); + assert_eq!(pool.usage()?.active_cells, 2); + let foreign = CellGroupSqlPool::start(&fixture.cluster.voters[1].log, limits)?; + assert!(foreign.reject(b.clone(), proof_b.clone()).await.is_err()); + foreign.close().await?; + assert!(matches!( + pb.submit(&fixture.runtimes[1]).await, + Err(RuntimeError::Invalid(_)) + )); + pool.reject(b.clone(), proof_b.clone()).await?; + assert_eq!(pool.usage()?.active_cells, 1); + assert!(pool.usage()?.retained_bytes > 0); + assert!(matches!( + fixture.runtimes[0] + .propose_at_term(route.history, pb.term(), pb.bytes().to_vec()) + .await, + Err(RuntimeError::Invalid(_)) + )); + apply_through(&mut fixture, accepted_a.position.index).await?; + drop(super::enrollment::resolved(&fixture, &pool, 0, &a, pa.proposal()).await?); + + // Retrying after capacity returns must still reject every old clone. A fresh + // producer of the same creation identity gets a distinct submission seal. + let fresh_b = initial(&fixture, &pool, &b, 182, 183).await?; + assert_eq!(pb.proposal(), fresh_b.proposal()); + assert!(pool.reject(b.clone(), proof_b.clone()).await.is_err()); + assert_eq!(pool.usage()?.active_cells, 1); + assert!(matches!( + pb.submit(&fixture.runtimes[0]).await?, + CellSqlSubmission::Rejected(_) + )); + let accepted_b = accepted_sql(&fixture.runtimes[0], &fresh_b).await?; + assert_eq!( + accepted_b.position.index.get(), + accepted_a.position.index.get() + 1 + ); + apply_through(&mut fixture, accepted_b.position.index).await?; + drop(super::enrollment::resolved(&fixture, &pool, 0, &b, fresh_b.proposal()).await?); + let pc = initial(&fixture, &pool, &c, 184, 184).await?; + drop(pc.submit(&fixture.runtimes[0]).await?); + pool.reject(c.clone(), pc.rejection()?.context("full group rejection")?) + .await?; + assert_eq!(pool.usage()?.active_cells, 0); + assert!(fixture.runtimes[0] + .enrollment_state(route.history, c) + .await? + .record() + .is_none()); + + // Completed/rejected producers release cell names, but their retained + // outputs still consume all four operation reservations. Release this + // phase's outputs before preparing the independent mutation phase. + assert_eq!(pool.usage()?.retained_operations, 4); + drop((pa, pb, fresh_b, pc, proof_b)); + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + + let ma = mutation(&fixture, &pool, &a, 185, 185).await?; + let mb = mutation(&fixture, &pool, &b, 186, 186).await?; + drop(ma.submit(&fixture.runtimes[0]).await?); + let accepted_ma = accepted_sql(&fixture.runtimes[0], &ma).await?; + drop(mb.submit(&fixture.runtimes[0]).await?); + let proof_mb = mb.rejection()?.context("mutation seal")?; + assert!(pool.reject(a.clone(), proof_mb.clone()).await.is_err()); + pool.reject(b.clone(), proof_mb.clone()).await?; + apply_through(&mut fixture, accepted_ma.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + assert_eq!( + pool.finish(authority).await?.receipt().receipt.result(), + b"committed" + ); + let cached = accepted_sql(&fixture.runtimes[0], &ma).await?; + assert_eq!(cached.position, accepted_ma.position); + assert_eq!(cached.digest, accepted_ma.digest); + assert!(ma.rejection()?.is_none()); + let fresh_mb = mutation(&fixture, &pool, &b, 186, 187).await?; + assert_eq!(mb.proposal(), fresh_mb.proposal()); + assert!(pool.reject(b.clone(), proof_mb.clone()).await.is_err()); + assert!(matches!( + mb.submit(&fixture.runtimes[0]).await?, + CellSqlSubmission::Rejected(_) + )); + let accepted_mb = accepted_sql(&fixture.runtimes[0], &fresh_mb).await?; + assert_eq!( + accepted_mb.position.index.get(), + accepted_ma.position.index.get() + 1 + ); + apply_through(&mut fixture, accepted_mb.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + assert_eq!( + pool.finish(authority).await?.receipt().receipt.result(), + b"committed" + ); + assert_eq!(pool.usage()?.active_cells, 0); + for node in 0..3 { + for cell in [&a, &b] { + let proof = fixture.runtimes[node] + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(proof.cell().version.sequence.get(), 2); + } + } + drop((ma, mb, fresh_mb, proof_mb)); + pool.close().await?; + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + if rooted { + assert_eq!( + fixture.cluster.voters[0] + .extents + .as_ref() + .context("extents")? + .usage()? + .workspaces, + 0 + ); + } + fixture.close().await +} From 4147ad479ceb4030ddc79c874203427931eb7385 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 22:20:07 -0700 Subject: [PATCH 062/568] docs: qualify prepared submission rejection and cleanup --- docs/design/multi-raft/cell-groups/README.md | 4 +- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 44 +++++++++++++------ .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 44 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 12 +++++ docs/design/multi-raft/cell-groups/tasks.json | 30 +++++++++++-- 7 files changed, 120 insertions(+), 20 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index a3d828249..b58776cdb 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -101,7 +101,9 @@ concurrent enrollment now run on that pool at `6e09bbc7`, including original creation retries, follower resolution and atomic preparation-term proposal checks. Bounded proposal acceptance and configured enrollment capacity are implemented at `446dd905`, including pending creations, exact acceptance retries, stale-base -refusal and a shared node metadata budget. Registered SDK/catalog routing, coherent +refusal and a shared node metadata budget. Prepared submission sealing and exact +rejection cleanup are implemented at `0cf89c40`; see the latest checkpoint for +qualification. Registered SDK/catalog routing, coherent reads, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 65477126f..f00859087 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "446dd9057f10c8221122d0d3a9a21254adeb448f", + "source_revision": "0cf89c403f4a500d18116499b8fc0f219195cd54", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 524065914..47d8e9d24 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,8 +15,9 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `446dd905`, adding bounded proposal acceptance -and enrollment capacity. Earlier checkpoints cover shared-worker initialization +The reviewed source baseline is `0cf89c40`, adding prepared submission sealing +and exact rejection cleanup after bounded proposal acceptance and enrollment +capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), applied-barrier reconciliation (`8a0cf640`), the fixed SQL pool (`51dff0e5`), the SQL producer (`f807b2f7`) and shared-node runtime (`aec4029e`). @@ -81,8 +82,9 @@ Logical name admission covers both commands and initialization, including compet incarnations. `CellEnrollmentState` reads verified current-term applied creation state on leaders or followers; `resolve_enrollment` preserves the original record, requires a newer applied term for absent/conflicting old producers, and never -reexecutes callbacks. Prepared outputs carry their term; use `propose_at_term` so -the actual owner rejects old output even after that node regains leadership. +reexecutes callbacks. Prepared outputs carry their term; the guarded submission +path checks it on the actual owner before append, including after that node +regains leadership. Cached acceptance grants no new authority. The four native cases exercise materialized/extents, concurrent creation, other cell progress, three cells in one group, retries after mutation, cancelled close, and absent/conflicting/committed enrollment recovery. All 191 selected Rust cases @@ -107,14 +109,20 @@ source inputs and eight copied executables. See [progress](progress.md) for the retained intermediate failures and qualification limits. This does not complete CG-04. +Prepared SQL submission now shares an irreversible decision across clones. Only +an actual owner's definitive pre-append capacity refusal seals the preparation; +a generic queue error leaves its outcome unresolved. The exact seal can be +recovered locally after a lost response, and the worker accepts it only for that +same preparation and engine. It closes the private producer before releasing +name admission. A fresh preparation with the same proposal ID cannot be closed by +the old proof. Accepted preparations retain their original position/digest even +after apply; this remains provisional acceptance, never commitment. Prepared +submission also closes byte-only proposal entry points on that owner. + Next integrate catalog routing sessions and registered SDK creation/command -callbacks, and coherent reads. Admission errors do not yet settle a SQL producer -already marked submitted: an `Overloaded` response may come from a queue before -the owner sees a retry of an earlier unknown attempt. Do not release that producer -from a generic error or same-term receipt absence. Production dispatch needs an -identity-bound owner rejection proof when it can establish non-acceptance, or -must preserve uncertainty and reconcile through the existing applied-barrier path. Preserve the original request fingerprint, deadline -and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to +callbacks, and coherent reads. Use `CellSqlPrepared::submit` and its typed result; +never release a producer from generic overload or same-term absence. Preserve +the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to satisfy callback/effect identities. The callback APIs added here remain verification boundaries, not registered SDK entry points. Real compaction/production retries and preview/cache lifecycle remain required. These cases do not complete CG-04. @@ -197,6 +205,15 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The latest partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` at `0cf89c40`: +197 selected Rust cases, ten successful commands, 189 source inputs and eight +copied executables. Transfer the entire directory, including intermediate failures, +with the prior evidence. Both image modes exercise rejection cleanup for creation +and mutation, exact proof matching, remembered acceptance and delayed retries. +This remains a partial CG-04 checkpoint; it does not qualify production dispatch, +coherent reads, RustFS or the hard cutover. + +The earlier receipt-lookup partial CG-04 record is `.cellar/cell-groups-evidence/CG-04-receipt-lookup/checkpoint.json` at `91c3ecfc`: 187 selected Rust cases, ten successful commands, 181 source inputs and eight copied executables. Transfer its entire directory with the earlier checkpoints. @@ -216,7 +233,7 @@ The prior partial CG-04 evidence is retained at directory, including copied binaries and intermediate/successful logs, with the prior SQL-producer, runtime/owner and DONE-task evidence. The pool checkpoint binds 178 source inputs, eight executables, 184 selected Rust cases and ten -verification commands. It cannot stand in for the complete six-case CG-04 +verification commands. It cannot stand in for the complete CG-04 contract. Three RustFS-specific cases remain explicitly unrun at this checkpoint. ## Delivery sequence @@ -236,7 +253,8 @@ an old-format migration reader; preserve marker 1 with explicit domain separatio and refuse unsupported persisted layouts. Never delete existing user data as part of the hard cutover. -CG-04 needs all six named real-Raft cases: `three_cells_one_rawnode_per_voter`, +CG-04 needs its six core real-Raft cases, plus the admission and submission cases +listed in `tasks.json`: `three_cells_one_rawnode_per_voter`, `concurrent_enrollment_one_cell`, `shared_group_fair_admission`, `cell_receipt_not_group_progress`, `strong_read_at_group_barrier` and `apply_prefix_never_skips_slow_cell`. Fixing a prototype's compilation or passing diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 3c1349a37..7e7629fcf 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 10 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 12 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1ba6b2767..4967e7e7c 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,50 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 prepared submission and rejection cleanup + +Qualified source: `0cf89c403f4a500d18116499b8fc0f219195cd54`. + +A prepared SQL attempt now carries its engine, group and preparation term, with +one submission decision shared by all clones. Exact accepted retries return the +original provisional position/digest, including after apply. Only an actual +owner's definitive pre-append capacity refusal seals the preparation and returns +a rejection proof. A queue overload remains uncertain. A lost rejection response +can be recovered from the local seal without contacting a closed runtime. +The first prepared submission closes raw byte-only proposal paths on that owner; +an uncertain append observation fails the owner closed rather than permitting +another append or manufacturing a rejection proof. + +SQL workers match rejection proofs to the exact preparation, not merely the +business proposal ID. Foreign engines, other cells and replacement producers +refuse old proofs. Cleanup closes the writer before releasing logical-name +admission. Queued submissions, prepared outputs and retained proofs keep their +original byte/operation reservations through cancellation and lost responses. +The real three-voter tests cover initialization and mutation in both image modes, +including a fresh preparation with the same proposal identity after rejection. + +Intermediate failures exposed two fixture accounting issues. A one-slot runtime +could refuse a read-only status poll while retaining admitted peer work; the +fixture now retries that poll while pumping the cluster under its existing +deadline. Completed enrollment outputs then correctly exhausted the SQL pool's +four retained-operation reservations. The fixture now asserts that accounting, +drops the completed phase's outputs, and asserts zero reservations/bytes before +preparing mutations. Resource limits and rejection assertions remain unchanged. +All 197 selected Rust cases pass at this source: 182 native cases across seven +suites, 11 SQL queue/extent-worker unit tests and four receipt unit tests. +All-target Clippy with warnings denied, formatting (1,802 Rust files), layout, +dependency isolation, docs, source preflight and all 37 plan-validator tests pass. +The checkpoint at `.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` +binds 189 source inputs, ten successful command logs and eight copied executables. +It retains the initial failures, diagnostic backtraces and repaired targeted run +separately. Four process helpers run through parent tests; three RustFS-specific +cases remain unrun. + +The two new cases raise the CG-04 contract to 12 required cases and the package +to 92. No task status advances. Registered SDK/catalog dispatch, coherent reads, +production retry/compaction integration and cache lifecycle remain required; +this checkpoint does not qualify the complete CG-04 or release gate. + ## CG-04 bounded proposal acceptance and enrollment capacity Qualified source: `446dd9057f10c8221122d0d3a9a21254adeb448f`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 89cf1fa74..00919f3fc 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -56,6 +56,18 @@ replacement, resolve the receipt/creation record rather than append an old base. A stale-base refusal does not prove non-commitment. New leaders must apply their current-term barrier before admitting creations against recovered capacity. +A definitive owner capacity refusal before append may retire a preparation only +with an exact preparation-bound rejection proof. Issuing that proof irreversibly +seals every clone against later submission, even after capacity returns. A generic +queue error, lost response or same-term receipt absence proves nothing about an +earlier attempt. Remember an accepted preparation's original position/digest; +returning that acceptance again grants no commit or current authority. Keep raw +byte submission closed once the owner receives prepared SQL work so queued raw +retries cannot bypass the seal. A proof for another engine, cell or replacement +preparation refuses even when the business proposal ID matches. Cleanup closes +the private writer on its worker before releasing logical-name admission; queued +submissions and retained outputs/proofs keep their byte reservations. + Admission decisions must be checked again in committed log order against the current lifecycle state. Recognizing an exact retry grants neither current-term authority nor permission to restore the initial image over newer mutations. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index ef935aa54..4545b4ce0 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -462,7 +462,9 @@ "crates/cellar/durability/raft/execution/enrollment.rs", "crates/cellar/tests/support/cell_group_sql_enrollment.rs", "crates/cellar/durability/raft/cell_groups/proposal_admission.rs", - "crates/cellar/tests/support/cell_group_admission.rs" + "crates/cellar/tests/support/cell_group_admission.rs", + "crates/cellar/durability/raft/execution/submission.rs", + "crates/cellar/tests/support/cell_group_sql_submission.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -482,11 +484,13 @@ ] }, { - "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes. Check the prepared term atomically with proposal admission on the actual Raft owner. Bind same-term acceptance retries to the original proposal identity and bytes; a lost acceptance response must recover retained acceptance or reconcile the original receipt without appending the same mutation again. Reject changed payloads under a reused identity, and retain uncertainty across elections until an applied barrier proves resolution.", + "action": "Implement exact per-cell output settlement and receipt retry. Group no-ops/configuration entries advance group apply without changing cell version. Reject stale term/activation/completion tickets and preserve unknown proposed outcomes. Check the prepared term atomically with proposal admission on the actual Raft owner. Bind same-term acceptance retries to the original proposal identity and bytes; a lost acceptance response must recover retained acceptance or reconcile the original receipt without appending the same mutation again. Reject changed payloads under a reused identity, and retain uncertainty across elections until an applied barrier proves resolution. A definitive pre-append owner refusal must irreversibly seal all clones of that preparation before returning an identity-bound rejection proof. Generic queue errors cannot retire a submitted SQL producer. Close rejected producers on their owning worker before releasing name admission; delayed retries and proofs from another preparation must not affect a replacement producer.", "tests": [ "cell_receipt_not_group_progress", "proposal_acceptance_retries_do_not_append_twice", - "inherited_enrollments_count_after_applied_term_barrier" + "inherited_enrollments_count_after_applied_term_barrier", + "sql_submission_rejection_seals_delayed_retries", + "rooted_sql_submission_rejection_seals_delayed_retries" ] }, { @@ -581,6 +585,26 @@ "I04", "I12" ] + }, + { + "name": "sql_submission_rejection_seals_delayed_retries", + "oracle": "Actual three-voter SQL initialization and mutation distinguish queue overload from definitive owner refusal. Lost rejection responses are recovered from the sealed preparation; foreign-engine, wrong-cell and same-proposal replacement proofs refuse. Old clones remain rejected after capacity returns, remembered acceptance never re-appends after apply, and worker cleanup releases names while retained outputs keep bounded reservations. Both image modes return all retained resources on close.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_sql_submission_rejection_seals_delayed_retries", + "oracle": "Actual three-voter SQL initialization and mutation distinguish queue overload from definitive owner refusal. Lost rejection responses are recovered from the sealed preparation; foreign-engine, wrong-cell and same-proposal replacement proofs refuse. Old clones remain rejected after capacity returns, remembered acceptance never re-appends after apply, and worker cleanup releases names while retained outputs keep bounded reservations. Both image modes return all retained resources on close.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From 88c4717d857fc52fd1d4f04e6d3fac871f8185a1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 22:45:49 -0700 Subject: [PATCH 063/568] feat: serve coherent cell reads through shared Raft and SQL workers --- crates/cellar/durability/raft/README.md | 36 +- crates/cellar/durability/raft/budget.rs | 36 ++ .../cellar/durability/raft/cell_groups/mod.rs | 6 +- .../durability/raft/cell_groups/owner.rs | 58 +++ .../raft/cell_groups/read_authority.rs | 114 ++++++ .../raft/cell_groups/runtime/mod.rs | 55 +++ .../raft/cell_groups/runtime/reads.rs | 197 ++++++++++ .../raft/cell_groups/runtime/worker.rs | 45 ++- crates/cellar/durability/raft/completion.rs | 1 + .../raft/host_sql/cell_groups/admission.rs | 43 ++- .../raft/host_sql/cell_groups/mod.rs | 3 + .../raft/host_sql/cell_groups/query.rs | 203 ++++++++++ .../raft/host_sql/cell_groups/worker.rs | 10 +- crates/cellar/durability/raft/options.rs | 3 + .../tests/native_cell_groups_execution.rs | 20 + .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 5 + .../tests/support/cell_group_sql_reads.rs | 365 ++++++++++++++++++ 18 files changed, 1183 insertions(+), 19 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/read_authority.rs create mode 100644 crates/cellar/durability/raft/cell_groups/runtime/reads.rs create mode 100644 crates/cellar/durability/raft/host_sql/cell_groups/query.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_reads.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index bfa8c6ccd..2dedceddb 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -146,7 +146,41 @@ their byte charge. Unknown producers remain uncertain. Pool shutdown joins an entered initializer even if its waiter and close waiter were both cancelled. As with commands, shutdown is not proof that a submitted enrollment failed. -Registered SDK creation/command dispatch, catalog routing integration, coherent query serving and +`CellGroupRuntime::read_authority` now selects one coherent cell image for an +invocation under an absolute execution-domain deadline. Linearizable selection +uses a fresh nonce/counter-correlated ReadIndex, current leader term and contiguous +applied group prefix. A committed update to B can delay a read of unchanged A, +but cannot change A's version. Raw read requests and forwarded read messages +cannot use the internal context namespace. The returned authority is consumed by +one query and rejects a changed route, lost leader term, foreign engine or expired +deadline. It is not a reusable leadership lease. + +Stale explicitly selects a local committed image, including on followers. AtLeast +compares logical cell history/sequence/image, not group log positions; the +production dispatcher must authenticate the token and schema/transfer lineage. +Neither path needs a new quorum. Future versions wait under the same original +deadline, while wrong histories or inconsistent equal-sequence images refuse. + +Waiting reads and returned pins use a separate node budget of `max_pending_reads` +items and `max_request_bytes` bytes; each group admits at most +`(max_pending_reads / 8).clamp(1, 32)` items under the same byte ceiling. Selecting +an image extends the reservation to its logical bytes plus metadata, refusing +before delivery if the ceiling is exceeded. This is a conservative pin-retention +bound, not a claim that every image byte is resident. Reads release ordinary peer +queue slots before waiting for quorum/apply. Cancellation, deadlines and shutdown +retire pending waiters; returned pins keep their reservations until consumed or +dropped. + +`CellGroupSqlPool::query` consumes that authority on the existing fixed workers. +It opens the immutable materialized/extent view without a private workspace or +schema initialization. Queries may read committed state while a private writer +is unresolved. They retain the read-only SQL authorizer, bounded output admission, +worker-owned pins and original deadline. SQLite's progress hook checks domain +expiry, waiter cancellation and shutdown; arbitrary callback work remains owned +and must finish before shutdown joins. Callback panics are contained. Result +accessors borrow bytes so their output reservation cannot be detached. + +Registered SDK creation/command/query dispatch, catalog routing integration and the production hard cutover remain required; the existing host still uses its prior single-cell owner. The new pool's callback entry point remains a verification boundary. Transport authentication and actual image preparation are also caller diff --git a/crates/cellar/durability/raft/budget.rs b/crates/cellar/durability/raft/budget.rs index 6f101ef71..377513fc3 100644 --- a/crates/cellar/durability/raft/budget.rs +++ b/crates/cellar/durability/raft/budget.rs @@ -64,6 +64,26 @@ pub(crate) struct Permit { } impl Permit { + /// Extend an admitted operation without consuming another item slot. + /// Failure leaves its existing reservation unchanged. + pub(crate) fn grow_to(&mut self, bytes: usize) -> bool { + let Some(extra) = bytes.checked_sub(self.bytes) else { + return false; + }; + if self + .owner + .bytes + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { + n.checked_add(extra).filter(|n| *n <= self.owner.max_bytes) + }) + .is_err() + { + return false; + } + self.bytes = bytes; + true + } + /// Refund transient capture capacity while keeping the operation slot and /// the retained payload charged. A reservation can never grow here. pub(crate) fn shrink_to(&mut self, bytes: usize) -> bool { @@ -92,6 +112,22 @@ impl Drop for Permit { mod tests { use super::*; + #[test] + fn grow_preserves_item_and_refuses_without_changing_accounting() { + let budget = Budget::new(100, 2); + let mut pin = budget.acquire(10).unwrap(); + let other = budget.acquire(70).unwrap(); + assert!(pin.grow_to(30)); + assert_eq!(budget.usage(), (100, 2)); + assert!(!pin.grow_to(31)); + assert!(!pin.grow_to(29)); + assert_eq!(budget.usage(), (100, 2)); + drop(other); + assert!(pin.grow_to(100)); + drop(pin); + assert_eq!(budget.usage(), (0, 0)); + } + #[test] fn shrink_refunds_bytes_but_keeps_the_operation_slot() { let budget = Budget::new(100, 2); diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e65781491..d1f8e6aad 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -4,8 +4,8 @@ pub use super::host_sql::cell_groups::{ CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, CellSqlLimits, - CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlReceipt, CellSqlReconciliation, - CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, + CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlQueryResult, CellSqlReceipt, + CellSqlReconciliation, CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, }; mod execution_authority; @@ -23,6 +23,8 @@ mod page_store; mod page_wire; mod pins; mod proposal_admission; +mod read_authority; +pub use read_authority::{CellReadAuthority, CellReadConsistency}; mod retention; mod root; mod runtime; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 37c70c4d6..3fcb449d3 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -290,6 +290,56 @@ impl CellGroupOwner { &self.current } + pub(super) fn select_read( + &self, + cell: &CellHistory, + consistency: &super::CellReadConsistency, + term: Option, + barrier: Option, + ) -> Result { + self.live()?; + let status = self.status()?; + let leader = if matches!(consistency, super::CellReadConsistency::Linearizable) { + if self.node.raft.state != StateRole::Leader || Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if !status.ready || status.applied.index < index { + return Err(RuntimeError::NotReady); + } + Some((status.term, self.leader_term.clone())) + } else { + None + }; + let (leaf, image) = self + .current + .pin_cell(&cell.cell) + .ok_or(RuntimeError::NotFound)?; + if &leaf.version.history != cell { + return Err(RuntimeError::Invalid("cell read history".into())); + } + if let super::CellReadConsistency::AtLeast(minimum) = consistency { + if !leaf + .version + .covers(minimum) + .map_err(|_| RuntimeError::Invalid("cell read version".into()))? + { + return Err(RuntimeError::NotReady); + } + } + Ok(super::read_authority::ReadSelection { + group: self.group.clone(), + route: self.current.marker().route().clone(), + cell: leaf.clone(), + image, + applied: status.applied, + barrier: term + .zip(barrier) + .map(|(term, index)| LogPosition { term, index }), + leader, + }) + } + pub fn enrollment_state( &self, cell: CellHistory, @@ -489,6 +539,13 @@ impl CellGroupOwner { } pub fn read_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if super::runtime::reads::reserved_context(&context) { + return Err(RuntimeError::Invalid("reserved cell read context".into())); + } + self.read_cell_index(context) + } + + pub(super) fn read_cell_index(&mut self, context: Vec) -> Result<(), RuntimeError> { if !self.status()?.ready { return Err(RuntimeError::NotReady); } @@ -1011,6 +1068,7 @@ impl CellGroupOwner { let entry = &message.entries[0]; if entry.data.is_empty() || entry.data.len() > 4096 + || super::runtime::reads::reserved_context(&entry.data) || entry.index != 0 || entry.term != 0 || !entry.context.is_empty() diff --git a/crates/cellar/durability/raft/cell_groups/read_authority.rs b/crates/cellar/durability/raft/cell_groups/read_authority.rs new file mode 100644 index 000000000..60670f825 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/read_authority.rs @@ -0,0 +1,114 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One invocation's coherent cell pin, never a reusable leadership lease. + +use super::{CellImagePin, CellManifestLeaf}; +use crate::raft::{budget::Permit, cache::GroupLog, completion::RuntimeError}; +use cellar_logic::{ + cell_group::*, + consensus::{LogPosition, Term}, +}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +#[derive(Clone)] +pub enum CellReadConsistency { + /// Explicitly allows a locally committed older image, including on followers. + Stale, + /// The dispatcher must first authenticate the version and its schema/lineage. + AtLeast(CellVersion), + Linearizable, +} + +pub(super) struct ReadSelection { + pub group: GroupLog, + pub route: GroupRoute, + pub cell: CellManifestLeaf, + pub image: CellImagePin, + pub applied: LogPosition, + pub barrier: Option, + pub leader: Option<(Term, Arc)>, +} + +/// Consumed by one SQL query. The image and its admission cannot be cloned out +/// of this proof. A strong selection requires a fresh correlated ReadIndex and +/// contiguous apply; unrelated cell progress never becomes a cell version. +pub struct CellReadAuthority { + selection: ReadSelection, + deadline: u64, + _node: Permit, + _group: Permit, +} + +impl CellReadAuthority { + pub(super) fn new( + selection: ReadSelection, + deadline: u64, + mut node: Permit, + mut group: Permit, + ) -> Result { + let bytes = usize::try_from(selection.cell.geometry.length) + .ok() + .and_then(|n| n.checked_add(4096)) + .ok_or(RuntimeError::Overloaded)?; + if !node.grow_to(bytes) || !group.grow_to(bytes) { + return Err(RuntimeError::Overloaded); + } + Ok(Self { + selection, + deadline, + _node: node, + _group: group, + }) + } + + pub fn cell(&self) -> &CellManifestLeaf { + &self.selection.cell + } + + pub fn route(&self) -> &GroupRoute { + &self.selection.route + } + + pub fn applied(&self) -> LogPosition { + self.selection.applied + } + + pub fn barrier(&self) -> Option { + self.selection.barrier + } + + pub fn deadline(&self) -> u64 { + self.deadline + } + + pub(crate) fn image(&self) -> &CellImagePin { + &self.selection.image + } + + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(engine, &self.selection.group.owner) + } + + pub(crate) fn check(&self, now: u64) -> Result<(), RuntimeError> { + if now >= self.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let s = &self.selection; + let cache = s.group.read()?; + if cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != &s.route) + || !cache.membership.permits_group(s.group.identity()) + || s.leader.as_ref().is_some_and(|(term, leader)| { + cache.hard_state.term != term.get() || leader.load(Ordering::Acquire) != term.get() + }) + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index d5fec918f..d2f6f35d3 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -2,6 +2,7 @@ //! One execution-domain worker multiplexes physical cell-group owners per node. +pub(super) mod reads; mod worker; use super::*; @@ -76,6 +77,7 @@ struct Handle { requests: mailbox::Sender, budget: Arc, completions: Arc, + read_pins: Arc, closing: Arc, terminal: Arc, stop: Arc, @@ -97,6 +99,7 @@ enum Command { Option>, ), ReadIndex(Vec), + Read(Box), Apply(CellGroupTicket, Box), Status, Tick, @@ -111,6 +114,7 @@ enum Response { EnrollmentState(Box), Reconciliation(Box), Status(CellGroupStatus), + Read(Box), } struct Request { @@ -176,6 +180,7 @@ impl CellGroupRuntime { domain: domain.clone(), budget: Budget::new(options.max_request_bytes, options.max_requests), completions: Budget::new(options.max_ready_bytes, COMPLETION_SLOTS), + read_pins: Budget::new(options.max_request_bytes, options.max_pending_reads), options: options.clone(), requests: sender, closing: closing.clone(), @@ -345,6 +350,56 @@ impl CellGroupRuntime { .map(|_| ()) } + /// Select one coherent image for this invocation. The deadline uses the + /// owning execution domain's monotonic clock and includes queue/barrier wait. + /// Returned pins retain separate node/group read admission until dropped. + pub async fn read_authority( + &self, + history: GroupHistory, + cell: cellar_logic::cell_group::CellHistory, + consistency: CellReadConsistency, + deadline: u64, + ) -> Result { + self.admit()?; + if self + .inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + let reservation = self + .inner + .read_pins + .acquire(4096) + .ok_or(RuntimeError::Overloaded)?; + let request = reads::ReadRequest { + cell, + consistency, + deadline, + reservation, + }; + let timer = self + .inner + .domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + crate::asyncrt::select! { + result = self.call(Some(history), Command::Read(Box::new(request)), 4096) => { + match result? { + Response::Read(authority) => { + authority.check(self.inner.domain.mono_ms().map_err(|_| RuntimeError::Closed)?)?; + Ok(*authority) + } + _ => Err(RuntimeError::Protocol("cell read authority reply".into())), + } + } + _ = timer => Err(RuntimeError::DeadlineExceeded), + } + } + pub async fn execution_authority( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs new file mode 100644 index 000000000..32144a855 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -0,0 +1,197 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation-owned reads. No pending read retains peer/apply capacity. + +use super::super::read_authority::ReadSelection; +use super::*; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{LogIndex, Term}, +}; +use std::{collections::BTreeMap, sync::Weak}; + +const PREFIX: &[u8] = b"cellar-cell-read-1\0"; + +pub(in crate::raft::cell_groups) fn reserved_context(context: &[u8]) -> bool { + context.starts_with(PREFIX) +} + +pub(super) struct ReadRequest { + pub cell: CellHistory, + pub consistency: CellReadConsistency, + pub deadline: u64, + pub reservation: Permit, +} + +struct Pending { + history: GroupHistory, + request: ReadRequest, + reply: oneshot::Sender>, + group: Permit, + term: Option, + barrier: Option, +} + +pub(super) struct Reads { + pending: BTreeMap, Pending>, + groups: BTreeMap>, + nonce: [u8; 16], + sequence: u64, + max_bytes: usize, + max_items: usize, +} + +impl Reads { + pub fn new(options: &RaftRuntimeOptions) -> Self { + use rand::RngCore as _; + let mut nonce = [0; 16]; + crate::asyncrt::rng("cell-group-read-context").fill_bytes(&mut nonce); + Self { + pending: BTreeMap::new(), + groups: BTreeMap::new(), + nonce, + sequence: 0, + max_bytes: options.max_request_bytes, + max_items: (options.max_pending_reads / 8).clamp(1, 32), + } + } + + pub fn start( + &mut self, + history: GroupHistory, + owner: &mut CellGroupOwner, + request: ReadRequest, + reply: oneshot::Sender>, + now: u64, + ) { + if reply.is_closed() { + return; + } + let begin = (|| { + if now >= request.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + if !history.same_namespace(&request.cell) { + return Err(RuntimeError::Invalid("cell read namespace".into())); + } + if let CellReadConsistency::AtLeast(minimum) = &request.consistency { + if minimum.history != request.cell { + return Err(RuntimeError::Invalid("cell read version history".into())); + } + } + self.groups.retain(|_, budget| budget.strong_count() != 0); + let budget = self + .groups + .get(&history) + .and_then(Weak::upgrade) + .unwrap_or_else(|| Budget::new(self.max_bytes, self.max_items)); + let group = budget.acquire(4096).ok_or(RuntimeError::Overloaded)?; + self.groups.insert(history, Arc::downgrade(&budget)); + self.sequence = self + .sequence + .checked_add(1) + .ok_or(RuntimeError::Overloaded)?; + let mut context = PREFIX.to_vec(); + context.extend_from_slice(&self.nonce); + context.extend_from_slice(&self.sequence.to_be_bytes()); + let term = if matches!(request.consistency, CellReadConsistency::Linearizable) { + let status = owner.status()?; + if !status.ready { + return Err(RuntimeError::NotReady); + } + owner.read_cell_index(context.clone())?; + Some(status.term) + } else { + None + }; + Ok((context, group, term)) + })(); + match begin { + Ok((context, group, term)) => { + self.pending.insert( + context, + Pending { + history, + request, + reply, + group, + term, + barrier: None, + }, + ); + } + Err(error) => { + let _ = reply.send(Err(error)); + } + } + } + + pub fn observe(&mut self, history: GroupHistory, term: Term, index: LogIndex, context: &[u8]) { + if let Some(pending) = self.pending.get_mut(context) { + if pending.history == history && pending.term == Some(term) { + pending.barrier = Some(index); + } + } + } + + pub fn deadline(&self) -> Option { + self.pending.values().map(|p| p.request.deadline).min() + } + + pub fn progress( + &mut self, + now: u64, + mut select: impl FnMut( + GroupHistory, + &ReadRequest, + Option, + Option, + ) -> Result, + ) -> bool { + let mut made = false; + self.pending.retain(|_, pending| { + // Taking a sender requires removing the entry; collect below instead. + if pending.reply.is_closed() { + made = true; + false + } else { + true + } + }); + let mut finished = Vec::new(); + for (context, pending) in &self.pending { + let result = if now >= pending.request.deadline { + Err(RuntimeError::DeadlineExceeded) + } else { + select( + pending.history, + &pending.request, + pending.term, + pending.barrier, + ) + }; + if !matches!(result, Err(RuntimeError::NotReady)) { + finished.push((context.clone(), result)); + } + } + for (context, result) in finished { + let pending = self + .pending + .remove(&context) + .expect("selected pending read"); + let result = result + .and_then(|selection| { + CellReadAuthority::new( + selection, + pending.request.deadline, + pending.request.reservation, + pending.group, + ) + }) + .map(|authority| Response::Read(Box::new(authority))); + let _ = pending.reply.send(result); + made = true; + } + made + } +} diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index ea1f5fdd0..1fdda291c 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -108,6 +108,7 @@ pub(super) struct Worker { ready: usize, apply: usize, reads: usize, + cell_reads: reads::Reads, } impl Worker { @@ -121,6 +122,7 @@ impl Worker { log, output: Budget::new(options.max_event_bytes, options.max_events), proposals: Budget::new(options.max_request_bytes, options.max_requests), + cell_reads: reads::Reads::new(&options), options, domain, groups: BTreeMap::new(), @@ -320,12 +322,26 @@ impl Worker { inbound, .. } = request; + let mut answer = Some(answer); let new_read = matches!(command, Command::ReadIndex(_)) + || matches!(&command, Command::Read(request) if matches!(request.consistency, CellReadConsistency::Linearizable)) || matches!(&command, Command::Step(m) if m.get_msg_type()==::raft::prelude::MessageType::MsgReadIndex); let result = if new_read && self.reads >= self.options.max_pending_reads { Err(RuntimeError::Overloaded) } else { match command { + Command::Read(request) => { + let now = + self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + self.cell_reads.start( + history, + &mut group.owner, + *request, + answer.take().expect("read reply"), + now, + ); + Ok(Response::Unit) + } Command::Execution(cell) => group .owner .execution_authority(&cell) @@ -366,7 +382,9 @@ impl Worker { }; drop(_permit); drop(inbound); - let _ = answer.send(result); + if let Some(answer) = answer { + let _ = answer.send(result); + } made = true; } } @@ -383,6 +401,14 @@ impl Worker { self.writes.push_back(history); made = true; } + Ok(Some(CellGroupEffect::ReadIndex { + term, + index, + context, + })) if reads::reserved_context(&context) => { + self.cell_reads.observe(history, term, index, &context); + made = true; + } Ok(Some(effect)) => { if group.pending.is_none() { group.pending = Some(Ok(effect)); @@ -587,13 +613,28 @@ impl Worker { if closing.load(Ordering::Acquire) { return Ok(()); } - let made = self.progress()? | self.start_batch()? | self.flush(); + let mut made = self.progress()? | self.start_batch()? | self.flush(); let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + made |= self + .cell_reads + .progress(now, |history, request, term, barrier| { + let group = self.groups.get(&history).ok_or(RuntimeError::NotFound)?; + if let Some(error) = &group.failed { + return Err(error.clone()); + } + group + .owner + .select_read(&request.cell, &request.consistency, term, barrier) + }); let deadline = if self.retry_at > now { tick_at.min(self.retry_at) } else { tick_at }; + let deadline = self + .cell_reads + .deadline() + .map_or(deadline, |read| deadline.min(read)); let timer = self .domain .sleep_until(deadline) diff --git a/crates/cellar/durability/raft/completion.rs b/crates/cellar/durability/raft/completion.rs index f2becc9c9..1a62ceb00 100644 --- a/crates/cellar/durability/raft/completion.rs +++ b/crates/cellar/durability/raft/completion.rs @@ -21,6 +21,7 @@ pub enum RuntimeError { Protocol(String), NotLeader, NotReady, + DeadlineExceeded, NotFound, AlreadyAttached, Overloaded, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs index 549ad63b1..d82633ce5 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs @@ -74,6 +74,32 @@ impl Admission { if state.cells.contains(&key) { return Err(ExecutionError::Busy); } + let bytes = self.reserve_bytes(&mut state, group, bytes)?; + state.cells.insert(key.clone()); + Ok(( + CellLease { + owner: self.clone(), + cell: key, + }, + bytes, + )) + } + + pub fn reserve_read( + &self, + group: GroupHistory, + bytes: usize, + ) -> Result, ExecutionError> { + let mut state = self.state.lock().map_err(|_| unavailable())?; + self.reserve_bytes(&mut state, group, bytes) + } + + fn reserve_bytes( + &self, + state: &mut State, + group: GroupHistory, + bytes: usize, + ) -> Result, ExecutionError> { let node = self.node.acquire(bytes).ok_or_else(overloaded)?; // Reclaim metadata only, never authority or pending physical ownership. state.groups.retain(|_, budget| budget.strong_count() != 0); @@ -86,18 +112,11 @@ impl Admission { }); let group_permit = group_budget.acquire(bytes).ok_or_else(overloaded)?; state.groups.insert(group, Arc::downgrade(&group_budget)); - state.cells.insert(key.clone()); - Ok(( - CellLease { - owner: self.clone(), - cell: key, - }, - Arc::new(Bytes { - _lease: self.lease.clone(), - _node: node, - _group: group_permit, - }), - )) + Ok(Arc::new(Bytes { + _lease: self.lease.clone(), + _node: node, + _group: group_permit, + })) } pub fn usage(&self) -> Result<(usize, usize, usize), ExecutionError> { diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 06e48d72c..1c1b0cff0 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -3,7 +3,9 @@ //! Cell-history admission and commands on the existing fixed node SQL workers. mod admission; +mod query; mod worker; +pub use query::CellSqlQueryResult; use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; use crate::raft::cell_groups::{ @@ -236,6 +238,7 @@ pub enum CellSqlOutcome { Enrollment(Box), EnrollmentResolved(CellSqlEnrollmentResolution), Rejected, + Queried(CellSqlQueryResult), } pub enum CellSqlReconciliation { diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs new file mode 100644 index 000000000..a159ec21f --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -0,0 +1,203 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable cell queries on the existing node SQL workers. + +use super::*; +use crate::raft::cell_groups::{CellImagePin, CellReadAuthority}; +use cellar_logic::cell_group::CellVersion; +use std::panic::AssertUnwindSafe; + +type Callback = Box< + dyn FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + Send, +>; + +pub(super) struct Query { + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, + cancelled: Arc, + closing: Arc, + bytes: Arc, +} + +pub struct CellSqlQueryResult { + result: Vec, + version: CellVersion, + _bytes: Arc, +} + +impl CellSqlQueryResult { + pub fn result(&self) -> &[u8] { + &self.result + } + + pub fn version(&self) -> &CellVersion { + &self.version + } +} + +struct Cancel(Arc); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupSqlPool { + /// Consume an invocation's coherent pin. Queries can coexist with an + /// unresolved private writer but read only the selected committed image. + /// The registered dispatcher must enforce operation contracts before here. + pub async fn query( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: impl FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + + Send + + 'static, + ) -> Result { + self.admit()?; + let domain = &self.inner.workers[0].domain; + if !authority.belongs_to_engine(&self.inner.engine_owner) + || options.application_digest() != authority.route().schema_digest + { + return Err(ExecutionError::Invalid("cell query engine/schema")); + } + authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure)?; + options.limits().validate()?; + let bytes = self.inner.admission.reserve_read( + authority.route().history, + options + .limits() + .max_result_bytes + .checked_add(4096) + .ok_or_else(overloaded)?, + )?; + let cell = authority.cell().version.history.clone(); + let deadline = authority.deadline(); + let cancelled = Arc::new(AtomicBool::new(false)); + let _cancel = Cancel(cancelled.clone()); + let query = Query { + options, + authority, + callback: Box::new(callback), + cancelled, + closing: self.inner.closing.clone(), + bytes, + }; + let timer = domain.sleep_until(deadline).map_err(|_| unavailable())?; + crate::asyncrt::select! { + result = self.call(cell, worker::Action::Query(Box::new(query))) => { + match result? { + CellSqlOutcome::Queried(output) => { + if domain.mono_ms().map_err(|_| unavailable())? >= deadline { + return Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)); + } + Ok(output) + } + _ => Err(ExecutionError::Invalid("cell query reply")), + } + } + _ = timer => Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)), + } + } +} + +// The SQLite wrapper catches progress-hook panics. This wrapper allows the +// domain clock to be captured without claiming that arbitrary clock backends +// themselves implement unwind-safety; a clock failure interrupts the query. +struct QueryClock(AssertUnwindSafe); + +impl QueryClock { + fn expired(&self, deadline: u64) -> bool { + self.0.mono_ms().map_or(true, |now| now >= deadline) + } +} + +pub(super) fn run( + mut query: Query, + domain: &crate::asyncrt::DomainToken, +) -> Result { + let check = |query: &Query| { + if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + query + .authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure) + }; + check(&query)?; + let database = match (&query.options, query.authority.image()) { + (CellSqlOptions::Materialized(options), CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin.clone(), + crate::storage::ImmutableQueryOptions { + services: options.query_services.clone(), + sqlite_vec: false, + }, + ) + } + (CellSqlOptions::Extents(options), CellImagePin::Extents(pin)) => { + let image = pin + .view() + .image( + options.cache_directory.clone(), + options.cache_pool.clone(), + options.cache, + ) + .map_err(crate::raft::execution::storage)?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services: options.query_services.clone(), + sqlite_vec: false, + }, + ) + } + _ => return Err(ExecutionError::Invalid("cell query image storage")), + } + .map_err(crate::raft::execution::storage)?; + let clock = QueryClock(AssertUnwindSafe(domain.clone())); + let deadline = query.authority.deadline(); + let cancelled = query.cancelled.clone(); + let closing = query.closing.clone(); + database + .connection() + .progress_handler( + 1000, + Some(move || { + cancelled.load(Ordering::Acquire) + || closing.load(Ordering::Acquire) + || clock.expired(deadline) + }), + ) + .map_err(crate::raft::execution::storage)?; + check(&query)?; + let version = query.authority.cell().version.clone(); + let result = + std::panic::catch_unwind(AssertUnwindSafe(|| (query.callback)(&database, &version))) + .unwrap_or_else(|panic| Err(crate::native_worker::panic_error(panic))) + .map_err(crate::raft::execution::storage)?; + // The callback has been consumed, so check the remaining owners explicitly. + if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { + return Err(unavailable()); + } + query + .authority + .check(domain.mono_ms().map_err(|_| unavailable())?) + .map_err(crate::raft::execution::authority_failure)?; + if result.len() > query.options.limits().max_result_bytes { + return Err(overloaded()); + } + drop(database); + drop(query.authority); + Bytes::shrink(&mut query.bytes, result.len() + 4096)?; + Ok(CellSqlOutcome::Queried(CellSqlQueryResult { + result, + version, + _bytes: query.bytes, + })) +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index 48b59b289..d6c4e3568 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -35,11 +35,12 @@ pub(super) enum Action { Finish(Box), Reconcile(ProposalId, Box), Reject(Box), + Query(Box), } impl Action { pub(super) fn progress_priority(&self) -> bool { - !matches!(self, Self::Stage(_) | Self::Initialize(_)) + !matches!(self, Self::Stage(_) | Self::Initialize(_) | Self::Query(_)) } } @@ -125,6 +126,7 @@ impl Cells { | Action::Reconcile(..) | Action::ResolveEnrollment(..) | Action::Reject(_) => crate::raft::performance::Operation::SqlPublish, + Action::Query(_) => crate::raft::performance::Operation::SqlQuery, _ => crate::raft::performance::Operation::SqlOther, }; let timer = performance.timer(operation); @@ -133,7 +135,10 @@ impl Cells { Err(unavailable()) } else { queued.finish(true); - self.run(cell, action, max) + match action { + Action::Query(query) => super::query::run(*query, domain), + action => self.run(cell, action, max), + } }; // A lost receiver drops only the delivery. Pending owns both the // private writer and per-cell reservation until exact settlement. @@ -148,6 +153,7 @@ impl Cells { max: usize, ) -> Result { match action { + Action::Query(_) => Err(ExecutionError::Invalid("query worker dispatch")), Action::Stage(stage) => { if self.pending.contains_key(&cell) { return Err(ExecutionError::Busy); diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index a7acbb856..2a731dc1e 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -121,6 +121,9 @@ pub struct RaftRuntimeOptions { pub max_message_bytes: usize, pub max_proposal_bytes: usize, pub max_members_per_group: usize, + /// Bounds protocol read states and a separate shared-cell read/pin budget. + /// That budget charges logical image bytes against max_request_bytes, with + /// at most (max_pending_reads / 8).clamp(1, 32) read pins per group. pub max_pending_reads: usize, /// Shared-group creation admission counts applied cells and pending accepted /// enrollments. A lower limit does not prevent existing cells from serving. diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index f65a0120b..a4fa1a2f5 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1186,3 +1186,23 @@ async fn physical_owner_retains_lease_through_cancelled_sync() -> Result<()> { cluster.voters[0].owner = Some(owner); cluster.close().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn strong_read_at_group_barrier() -> Result<()> { + runtime_fixture::coherent_reads(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_strong_read_at_group_barrier() -> Result<()> { + runtime_fixture::coherent_reads(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { + runtime_fixture::query_lifecycle(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { + runtime_fixture::query_lifecycle(true).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index e925975f0..e811c3348 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -624,3 +624,5 @@ pub(super) async fn recover_initialization(rooted: bool) -> Result<()> { pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { sql::submission_rejections(rooted).await } + +pub(super) use sql::{coherent_reads, query_lifecycle}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 5b4c759e3..a5daeb35b 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -469,3 +469,8 @@ mod submission; pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { submission::rejections(rooted).await } + +#[path = "cell_group_sql_reads.rs"] +mod reads; + +pub(crate) use reads::{coherent as coherent_reads, lifecycle as query_lifecycle}; diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs new file mode 100644 index 000000000..8f6253371 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -0,0 +1,365 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::reconciliation::apply_through; +use super::*; + +async fn deadline(fixture: &RuntimeCluster, ms: u64) -> Result { + fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await? + .checked_add(ms) + .context("deadline") +} + +fn pool(fixture: &RuntimeCluster, node: usize) -> Result { + Ok(CellGroupSqlPool::start( + &fixture.cluster.voters[node].log, + CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?) +} + +async fn selected( + fixture: &mut RuntimeCluster, + node: usize, + cell: &CellHistory, + consistency: CellReadConsistency, +) -> Result { + let runtime = fixture.runtimes[node].clone(); + let read = runtime.read_authority( + fixture.cluster.birth.route().history, + cell.clone(), + consistency, + deadline(fixture, 5000).await?, + ); + tokio::pin!(read); + loop { + tokio::select! { + result = &mut read => return result.map_err(Into::into), + result = async { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + Ok::<_, anyhow::Error>(()) + } => result?, + } + } +} + +async fn value( + pool: &CellGroupSqlPool, + options: CellSqlOptions, + authority: CellReadAuthority, +) -> Result { + Ok(pool + .query(options, authority, |db, version| { + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert!(db + .connection() + .execute("UPDATE value SET n=999", []) + .is_err()); + assert!(version.sequence.get() > 0); + Ok(n.to_le_bytes().to_vec()) + }) + .await?) +} + +pub(crate) async fn coherent(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 0)?; + let oa = fixture.cell_options(&a).await?; + let ob = fixture.cell_options(&b).await?; + let original = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let original_version = original.cell().version.clone(); + drop(value(&pool, oa.clone(), original).await?); + + // A ready leader without any new peer responses cannot reuse a cached cut. + assert!(fixture.runtimes[0].status(route.history).await?.ready); + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 50).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + fixture.drive(None).await?; + assert!(fixture.runtimes[0] + .read_index(route.history, b"cellar-cell-read-1\0forged".to_vec()) + .await + .is_err()); + + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage(ob.clone(), authority, cell_attempt(201)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"b".to_vec()) + }) + .await? + else { + anyhow::bail!("B preparation") + }; + let private_b = selected(&mut fixture, 0, &b, CellReadConsistency::Stale).await?; + assert_eq!( + value(&pool, ob.clone(), private_b).await?.result(), + 0_i64.to_le_bytes() + ); + let old_b = selected(&mut fixture, 0, &b, CellReadConsistency::Stale).await?; + let old_b_version = old_b.cell().version.clone(); + let encoded = CellGroupEntry::decode(prepared.bytes(), &route, GroupEntryLimits::default())?; + let GroupCommand::Mutation(mutation) = encoded.command else { + anyhow::bail!("B mutation") + }; + let expected_b = CellVersion { + history: b.clone(), + sequence: mutation.sequence, + image: mutation.result.into(), + }; + pool.mark_submitted(b.clone(), prepared.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + // A has not changed, but a strong A read must still wait for B's earlier + // committed entry to enter the contiguous group publication prefix. + let runtime = fixture.runtimes[0].clone(); + let pending = runtime.read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 5000).await?, + ); + tokio::pin!(pending); + let at_least = runtime.read_authority( + route.history, + b.clone(), + CellReadConsistency::AtLeast(expected_b.clone()), + deadline(&fixture, 5000).await?, + ); + tokio::pin!(at_least); + for _ in 0..30 { + assert!(tokio::time::timeout(Duration::from_millis(2), &mut pending) + .await + .is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(2), &mut at_least) + .await + .is_err() + ); + fixture.pump(&[route.history], &mut held).await?; + } + let work = held.remove(&route.history).context("held B apply")?; + assert_eq!(work.entry.index, accepted.position.index.get()); + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(route.history, work, next) + .await?; + apply_through(&mut fixture, accepted.position.index).await?; + let selected = tokio::time::timeout(Duration::from_secs(5), pending).await??; + let at_least = tokio::time::timeout(Duration::from_secs(5), at_least).await??; + assert_eq!(at_least.cell().version, expected_b); + drop(value(&pool, ob.clone(), at_least).await?); + let old_output = value(&pool, ob.clone(), old_b).await?; + assert_eq!(old_output.result(), 0_i64.to_le_bytes()); + assert_eq!(old_output.version(), &old_b_version); + drop(old_output); + assert_eq!(selected.cell().version, original_version); + assert!(selected.applied().index >= accepted.position.index); + assert!(selected.barrier().context("fresh barrier")?.index >= accepted.position.index); + assert_eq!( + value(&pool, oa.clone(), selected).await?.result(), + 0_i64.to_le_bytes() + ); + drop( + pool.finish( + fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?, + ) + .await?, + ); + drop(prepared); + + let version_b = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await? + .cell() + .version + .clone(); + let ready_b = self::selected( + &mut fixture, + 0, + &b, + CellReadConsistency::AtLeast(version_b.clone()), + ) + .await?; + let output = value(&pool, ob, ready_b).await?; + assert_eq!(output.result(), 1_i64.to_le_bytes()); + assert_eq!(output.version(), &version_b); + drop(output); + assert!(fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::AtLeast(version_b), + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + let mut future = original_version.clone(); + future.sequence = MutationSequence::new(2)?; + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::AtLeast(future), + deadline(&fixture, 30).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + let mut wrong = a.clone(); + wrong.incarnation = IncarnationId::new([202; 16])?; + assert!(fixture.runtimes[0] + .read_authority( + route.history, + wrong, + CellReadConsistency::Stale, + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + + let strong = self::selected(&mut fixture, 0, &a, CellReadConsistency::Linearizable).await?; + super::reconciliation::transfer(&mut fixture, 0, 1).await?; + assert!(pool + .query(oa.clone(), strong, |_, _| panic!("old leader query")) + .await + .is_err()); + assert!(fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 1000).await? + ) + .await + .is_err()); + let stale = self::selected( + &mut fixture, + 0, + &a, + CellReadConsistency::AtLeast(original_version), + ) + .await?; + drop(value(&pool, oa, stale).await?); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + pool.close().await?; + fixture.close().await +} + +pub(crate) async fn lifecycle(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 0)?; + let oa = fixture.cell_options(&a).await?; + let mut pins = Vec::new(); + for _ in 0..32 { + pins.push(selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?); + } + assert!(matches!( + fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::Overloaded) + )); + drop(pins); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let foreign = self::pool(&fixture, 1)?; + assert!(foreign + .query(oa.clone(), authority, |_, _| panic!("foreign query")) + .await + .is_err()); + foreign.close().await?; + + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline(&fixture, 100).await?, + ) + .await?; + assert!(pool.query(oa.clone(), authority, |db, _| { + let n: i64 = db.query_row("WITH RECURSIVE n(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }).await.is_err()); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, oa.clone(), authority).await?); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + assert!(pool + .query(oa.clone(), authority, |_, _| panic!( + "contained query panic" + )) + .await + .is_err()); + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, oa.clone(), authority).await?); + + let authority = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + caller + .query(oa, authority, move |db, _| { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.running_workers, 0); + fixture.close().await +} From 07ad47975c38fb9d01e8ec49deac61e049f7641b Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 23:02:50 -0700 Subject: [PATCH 064/568] docs: deliver source-bound cell-group execution handoff --- docs/design/multi-raft/cell-groups/README.md | 7 ++- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 49 ++++++++++++----- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 55 +++++++++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 40 +++++++++++++- 6 files changed, 134 insertions(+), 23 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index b58776cdb..fd862bebd 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -103,8 +103,11 @@ Bounded proposal acceptance and configured enrollment capacity are implemented a `446dd905`, including pending creations, exact acceptance retries, stale-base refusal and a shared node metadata budget. Prepared submission sealing and exact rejection cleanup are implemented at `0cf89c40`; see the latest checkpoint for -qualification. Registered SDK/catalog routing, coherent -reads, production retry/compaction integration and cache lifecycle remain required. Managed +qualification. Coherent shared read selection and immutable queries now exist +at `88c4717d`, whose read gate failed Clippy after 203 passing Rust cases. +The handoff records the repair steps; this source is not a qualified checkpoint. +Registered SDK/catalog routing, query/version-token integration, +production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index f00859087..b9d005d85 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "0cf89c403f4a500d18116499b8fc0f219195cd54", + "source_revision": "88c4717d857fc52fd1d4f04e6d3fac871f8185a1", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 47d8e9d24..b1c68dc1b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,9 +15,10 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `0cf89c40`, adding prepared submission sealing -and exact rejection cleanup after bounded proposal acceptance and enrollment -capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization +The reviewed source baseline is `88c4717d`, adding coherent read selection and +immutable cell queries. Earlier checkpoints cover prepared submission sealing +and exact rejection cleanup (`0cf89c40`) and bounded proposal acceptance and +enrollment capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), applied-barrier reconciliation (`8a0cf640`), the fixed SQL pool (`51dff0e5`), the SQL producer (`f807b2f7`) and shared-node runtime (`aec4029e`). @@ -120,7 +121,7 @@ after apply; this remains provisional acceptance, never commitment. Prepared submission also closes byte-only proposal entry points on that owner. Next integrate catalog routing sessions and registered SDK creation/command -callbacks, and coherent reads. Use `CellSqlPrepared::submit` and its typed result; +callbacks, and registered query/version-token serving. Use `CellSqlPrepared::submit` and its typed result; never release a producer from generic overload or same-term absence. Preserve the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to satisfy callback/effect identities. The callback APIs added here remain verification @@ -147,14 +148,24 @@ second node worker pool or a task per enrolled cell. A cancelled caller cannot release physical callback ownership or allow a second producer for that cell. -Query integration must also replace `application/consensus/class.rs::QueryContext`'s -`CommittedVersion`/`GroupScope` identity with the logical cell version. The shared -SQL options already carry query services, but the pool has no cell query operation. -`CellExecutionAuthority` and `CellReconciliationAuthority` explicitly grant no -query lease: do not repurpose them as strong-read proof. Bind a ReadIndex result -to its request context, current owner term and applied prefix, then pin the exact -target cell image; another cell's newer log index must not manufacture a cell -version. Preserve the existing registered query transaction restrictions. +Shared read selection and SQL execution now exist. Use +`CellGroupRuntime::read_authority` with the original absolute domain deadline, +then consume its pin through `CellGroupSqlPool::query`. Strong reads correlate +fresh quorum to that invocation and wait for contiguous apply; Stale/AtLeast +select logical cell versions. The immutable pin, node/group reservations and +worker-owned query survive publication and cancellation. A returned query result +borrows retained bytes and exposes its logical `CellVersion`. + +Registered query integration must still replace +`application/consensus/class.rs::QueryContext`'s `CommittedVersion`/`GroupScope` +identity with the logical cell version. Preserve operation/schema validation, +read-only callback restrictions, version-token authentication and installed +transfer lineage. `CellExecutionAuthority` and `CellReconciliationAuthority` +still grant no strong-read authority; use the new request-owned read path. +Do not synthesize a cell-scoped legacy group or treat the group barrier index +as the target cell's mutation position. SDK token issuance must retain the actual +cell decision witness through compaction, rather than invent a proposal identity +from the latest group progress. ## First executor session @@ -178,8 +189,16 @@ owning host. Reuse the old executor's current-term applied-barrier rule when replacing it: a submitted producer cannot be discarded from a sampled leader status, timeout or missing response. Resolve the original command against the authoritative cell receipt before admitting another producer. These regressions supplement, and do not replace, the full task -gate printed by `show CG-04`. Registered SDK dispatch and strong-read serving -remain absent. +gate printed by `show CG-04`. Registered SDK dispatch and registered strong-read serving +remain absent; the low-level read-authority and immutable-query paths exist. + +The first repair steps are recorded in [progress](progress.md): replace the +forbidden fixture `tokio::select!` macro and decouple logical image-pin capacity +from transport request bytes. The current read checkpoint passed 203 selected +Rust cases but failed Clippy; later commands in that gate were not run. Preserve +`.cellar/cell-groups-evidence/CG-04-reads/` as failed intermediate evidence. Rerun +the full source-bound gate after the repairs; do not carry forward a passing +qualification label from the previous submission checkpoint. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -204,7 +223,7 @@ The three DONE records require their retained evidence directories listed in another machine. Missing evidence must refuse; do not replace it with a pass flag or reset completed tasks merely to bypass validation. -The latest partial CG-04 record is +The latest fully passing partial CG-04 record is `.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` at `0cf89c40`: 197 selected Rust cases, ten successful commands, 189 source inputs and eight copied executables. Transfer the entire directory, including intermediate failures, diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 7e7629fcf..dea7fd58d 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 12 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 15 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 4967e7e7c..fbce25989 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,61 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 coherent cell reads and immutable SQL queries + +Implementation source: `88c4717d`. + +The shared node runtime now owns bounded read requests with original absolute +domain deadlines. Linearizable requests use unique nonce/counter contexts and +current-term ReadIndex, then wait for contiguous group apply before pinning the +target cell. Raw or forwarded read requests cannot borrow those internal contexts. +A minority leader cannot substitute a cached commit position. A read of unchanged +A waits behind committed B while retaining A's own version. Stale/AtLeast can +select committed follower images; AtLeast compares logical cell history and +sequence/image, never unrelated group indexes. + +Separate node/group admission retains waiting reads and delivered pins. Image +selection grows the existing item reservation to its logical byte size without +consuming ordinary peer/apply capacity. The pin is consumed once by an immutable +query on the existing SQL workers. Queries can coexist with an unresolved writer +and read only committed images; an older pin remains coherent across a later +publication of the same cell. SQL output retains its own reservation through +borrowed result accessors. Wrong engines, routes, histories, expired deadlines +and revoked strong-read terms refuse. + +SQL progress hooks check the domain deadline, waiter cancellation and shutdown. +Callbacks retain worker and pin ownership until they return; callback panics are +contained and close joins entered work. Deadline checks also cover queued replies, +so a ready result cannot bypass expiry when the receiver resumes late. + +Both image modes pass targeted strong-read and lifecycle cases. The first native +compile used an unavailable clock accessor; fixtures now run clock reads inside +the owning domain. Expanding the strong cases with a pending AtLeast read and +an older same-cell pin exposed unnecessary async forwarding layers overflowing +the default fixture stack. Reexports remove those layers; a visibility error in +that intermediate change was corrected. The enhanced cases pass on the normal +stack without weakening assertions. + +The retained gate at `.cellar/cell-groups-evidence/CG-04-reads/gate-commands.json` +completed four test commands: 203 selected Rust cases passed. The fifth command, +all-target Clippy, failed because `tests/support/cell_group_sql_reads.rs` uses the +forbidden `tokio::select!` macro. Subsequent commands in that gate did not run. +This is an unqualified source checkpoint; the latest fully passing partial +checkpoint remains `0cf89c40`. No task status advances. + +Before resuming qualification, replace that fixture macro with +`cellar::asyncrt::select!`. Also separate the logical read-image pin budget from +`RaftRuntimeOptions::max_request_bytes`: the current transport ceiling is 512 MiB, +whereas valid native images can reach 10,000,000,000 bytes. Add an explicit bounded +pin-byte option, validate its arithmetic and supported image envelope, and test +that pin configuration does not require raising transport limits. Preserve +node/group accounting, count limits and cancellation ownership. Retain the failed +gate, then run a fresh full gate against the repaired source revision. + +Registered SDK dispatch, callback identity and version-token integration, catalog +sessions, production retry/compaction and cache lifecycle remain required. This +is partial CG-04 implementation, not the production cutover or release gate. + ## CG-04 prepared submission and rejection cleanup Qualified source: `0cf89c403f4a500d18116499b8fc0f219195cd54`. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 4545b4ce0..937db301c 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -464,7 +464,11 @@ "crates/cellar/durability/raft/cell_groups/proposal_admission.rs", "crates/cellar/tests/support/cell_group_admission.rs", "crates/cellar/durability/raft/execution/submission.rs", - "crates/cellar/tests/support/cell_group_sql_submission.rs" + "crates/cellar/tests/support/cell_group_sql_submission.rs", + "crates/cellar/durability/raft/cell_groups/read_authority.rs", + "crates/cellar/durability/raft/cell_groups/runtime/reads.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/query.rs", + "crates/cellar/tests/support/cell_group_sql_reads.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -494,10 +498,13 @@ ] }, { - "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture.", + "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture. Correlate each strong read to its own runtime-owned request context and absolute domain deadline; retain separate bounded node/group pin admission through delivery and physical SQL execution. Configure logical image-pin bytes independently of transport message/request bytes; validate overflow and supported image bounds, and test independent capacity and reservation cleanup. Query callbacks consume immutable pins on existing workers, preserve read-only SQL policy and contain panics. Deadline/cancellation must interrupt SQLite work while arbitrary callback ownership remains retained until it finishes; shutdown joins that work.", "tests": [ "strong_read_at_group_barrier", - "apply_prefix_never_skips_slow_cell" + "apply_prefix_never_skips_slow_cell", + "rooted_strong_read_at_group_barrier", + "cell_query_pins_deadline_and_cancelled_cleanup", + "rooted_cell_query_pins_deadline_and_cancelled_cleanup" ] } ], @@ -605,6 +612,33 @@ "I03", "I12" ] + }, + { + "name": "rooted_strong_read_at_group_barrier", + "oracle": "Extent-backed strong reads require fresh quorum and contiguous apply behind another cell, preserve the target cell version, wait for AtLeast publication, retain older coherent pins and refuse wrong histories, expired deadlines and revoked leadership.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "cell_query_pins_deadline_and_cancelled_cleanup", + "oracle": "Real group read pins enforce capacity and engine binding. Immutable SQL refuses writes, interrupts deadline-bound work, contains callback panics and retains entered work after waiter cancellation; close joins it and returns all SQL reservations.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_cell_query_pins_deadline_and_cancelled_cleanup", + "oracle": "Real group read pins enforce capacity and engine binding. Immutable SQL refuses writes, interrupts deadline-bound work, contains callback panics and retains entered work after waiter cancellation; close joins it and returns all SQL reservations.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ From 3da4481e09c3767face30469ffc855d302a7c142 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 23:06:04 -0700 Subject: [PATCH 065/568] fix: separate cell read pin retention from transport capacity --- crates/cellar/durability/raft/README.md | 7 ++- .../raft/cell_groups/runtime/mod.rs | 2 +- .../raft/cell_groups/runtime/reads.rs | 2 +- crates/cellar/durability/raft/options.rs | 49 ++++++++++++++++++- .../tests/support/cell_group_sql_reads.rs | 2 +- 5 files changed, 55 insertions(+), 7 deletions(-) diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 2dedceddb..8b7079e4d 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -162,11 +162,14 @@ Neither path needs a new quorum. Future versions wait under the same original deadline, while wrong histories or inconsistent equal-sequence images refuse. Waiting reads and returned pins use a separate node budget of `max_pending_reads` -items and `max_request_bytes` bytes; each group admits at most +items and `max_read_pin_bytes` bytes; each group admits at most `(max_pending_reads / 8).clamp(1, 32)` items under the same byte ceiling. Selecting an image extends the reservation to its logical bytes plus metadata, refusing before delivery if the ceiling is exceeded. This is a conservative pin-retention -bound, not a claim that every image byte is resident. Reads release ordinary peer +bound, not a claim that every image byte is resident. The pin-byte limit defaults +to 64 MiB and can cover the native image-size envelope independently of transport +limits; configure it for the largest admitted images and concurrent reads. +Reads release ordinary peer queue slots before waiting for quorum/apply. Cancellation, deadlines and shutdown retire pending waiters; returned pins keep their reservations until consumed or dropped. diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index d2f6f35d3..b25f8e005 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -180,7 +180,7 @@ impl CellGroupRuntime { domain: domain.clone(), budget: Budget::new(options.max_request_bytes, options.max_requests), completions: Budget::new(options.max_ready_bytes, COMPLETION_SLOTS), - read_pins: Budget::new(options.max_request_bytes, options.max_pending_reads), + read_pins: Budget::new(options.max_read_pin_bytes, options.max_pending_reads), options: options.clone(), requests: sender, closing: closing.clone(), diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs index 32144a855..de75fb687 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -51,7 +51,7 @@ impl Reads { groups: BTreeMap::new(), nonce, sequence: 0, - max_bytes: options.max_request_bytes, + max_bytes: options.max_read_pin_bytes, max_items: (options.max_pending_reads / 8).clamp(1, 32), } } diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index 2a731dc1e..7441615fc 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -122,9 +122,13 @@ pub struct RaftRuntimeOptions { pub max_proposal_bytes: usize, pub max_members_per_group: usize, /// Bounds protocol read states and a separate shared-cell read/pin budget. - /// That budget charges logical image bytes against max_request_bytes, with - /// at most (max_pending_reads / 8).clamp(1, 32) read pins per group. + /// At most (max_pending_reads / 8).clamp(1, 32) read pins belong to one group. pub max_pending_reads: usize, + /// Logical image bytes plus 4 KiB per waiting read or selected pin, retained + /// through query execution. Separate from transport request capacity; both + /// node and group pin admission use this byte ceiling. No memory is allocated + /// merely by configuring this limit. + pub max_read_pin_bytes: usize, /// Shared-group creation admission counts applied cells and pending accepted /// enrollments. A lower limit does not prevent existing cells from serving. /// max_requests also bounds leader-local pending cell acceptance records. @@ -149,6 +153,7 @@ impl Default for RaftRuntimeOptions { max_proposal_bytes: 16 * 1024 * 1024, max_members_per_group: 7, max_pending_reads: 256, + max_read_pin_bytes: 64 * 1024 * 1024, max_cells_per_group: 1_000_000, } } @@ -185,6 +190,10 @@ impl RaftRuntimeOptions { || self.max_members_per_group > 16 || self.max_pending_reads == 0 || self.max_pending_reads > 4096 + || self.max_read_pin_bytes < 4096 + || self.max_read_pin_bytes as u128 + > (cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES as u128 + 4096) + * self.max_pending_reads as u128 || self.max_cells_per_group == 0 || self.max_cells_per_group > 1_000_000 { @@ -195,3 +204,39 @@ impl RaftRuntimeOptions { Ok(()) } } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn read_pin_capacity_is_independent_of_transport_and_bounds_images() { + let log = ReplicaLogOptions::new( + PathBuf::from("read-bounds"), + NodeId::new([1; 16]).unwrap(), + DiskLineage::new([2; 16]).unwrap(), + ProcessGeneration::new([3; 16]).unwrap(), + LogOpenMode::Create, + ); + let mut options = RaftRuntimeOptions::default(); + assert!(options.validate(&log).is_ok()); + let maximum_pin = cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES as u128 + 4096; + // On narrower platforms the accounting type itself bounds the image. + options.max_read_pin_bytes = usize::try_from(maximum_pin).unwrap_or(usize::MAX); + assert_eq!(options.max_request_bytes, 64 * 1024 * 1024); + assert!(options.validate(&log).is_ok()); + options.max_read_pin_bytes = 4095; + assert!(options.validate(&log).is_err()); + options.max_read_pin_bytes = 4096; + assert!(options.validate(&log).is_ok()); + let envelope = maximum_pin * options.max_pending_reads as u128; + if let Ok(envelope) = usize::try_from(envelope) { + options.max_read_pin_bytes = envelope; + assert!(options.validate(&log).is_ok()); + if let Some(too_large) = envelope.checked_add(1) { + options.max_read_pin_bytes = too_large; + assert!(options.validate(&log).is_err()); + } + } + } +} diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs index 8f6253371..4985a66ac 100644 --- a/crates/cellar/tests/support/cell_group_sql_reads.rs +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -42,7 +42,7 @@ async fn selected( ); tokio::pin!(read); loop { - tokio::select! { + cellar::asyncrt::select! { result = &mut read => return result.map_err(Into::into), result = async { fixture.pump(&[], &mut BTreeMap::new()).await?; From f83b74b520c7d2f9c6988734a7bac31d49b11f32 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 23:22:49 -0700 Subject: [PATCH 066/568] docs: qualify coherent cell reads and independent pin capacity --- docs/design/multi-raft/cell-groups/README.md | 5 +- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 32 +++++++++---- .../design/multi-raft/cell-groups/progress.md | 48 +++++++++++++++---- 4 files changed, 67 insertions(+), 22 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index fd862bebd..7b903ae85 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -104,8 +104,9 @@ Bounded proposal acceptance and configured enrollment capacity are implemented a refusal and a shared node metadata budget. Prepared submission sealing and exact rejection cleanup are implemented at `0cf89c40`; see the latest checkpoint for qualification. Coherent shared read selection and immutable queries now exist -at `88c4717d`, whose read gate failed Clippy after 203 passing Rust cases. -The handoff records the repair steps; this source is not a qualified checkpoint. +at `88c4717d`, repaired and qualified at `3da4481e`: 204 selected Rust cases +and all twelve verification commands pass. The retained checkpoint includes +the earlier failed Clippy gate; see the handoff for its exact scope. Registered SDK/catalog routing, query/version-token integration, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index b9d005d85..9ad5e357c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "88c4717d857fc52fd1d4f04e6d3fac871f8185a1", + "source_revision": "3da4481e09c3767face30469ffc855d302a7c142", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index b1c68dc1b..a148359df 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,8 +15,9 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `88c4717d`, adding coherent read selection and -immutable cell queries. Earlier checkpoints cover prepared submission sealing +The reviewed source baseline is `3da4481e`, qualifying coherent read selection +and immutable cell queries after decoupling pin capacity from transport limits +and repairing the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing and exact rejection cleanup (`0cf89c40`) and bounded proposal acceptance and enrollment capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), @@ -192,13 +193,18 @@ command against the authoritative cell receipt before admitting another producer gate printed by `show CG-04`. Registered SDK dispatch and registered strong-read serving remain absent; the low-level read-authority and immutable-query paths exist. -The first repair steps are recorded in [progress](progress.md): replace the -forbidden fixture `tokio::select!` macro and decouple logical image-pin capacity -from transport request bytes. The current read checkpoint passed 203 selected -Rust cases but failed Clippy; later commands in that gate were not run. Preserve -`.cellar/cell-groups-evidence/CG-04-reads/` as failed intermediate evidence. Rerun -the full source-bound gate after the repairs; do not carry forward a passing -qualification label from the previous submission checkpoint. +The read fixture macro and independent pin-byte capacity are repaired and +qualified at `3da4481e`; the earlier failed read gate is retained as intermediate +evidence. Next persist the selected version's complete commit witness in the +authenticated cell leaf and carry it through read/query results before registered +SDK token issuance. `raft/version.rs::CellVersionObservation` already requires +`CommitWitness`; the current leaf stores only `last_mutation` and the original +enrollment record, so a later mutation's proposal ID cannot be reconstructed +after log compaction. Validate the witness against the actual applied entry, +retain it across unchanged-cell group progress and restart, and exercise tokens +from both image modes. Keep physical apply position separate from provenance +when later implementing group transfer. Then integrate registered callbacks and +catalog invocation sessions without synthesizing legacy group identities. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -224,6 +230,14 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The latest fully passing partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-reads-final/checkpoint.json` at `3da4481e`: +204 selected Rust cases, twelve successful commands, 194 source inputs and eight +copied executables. Transfer the entire directory, including intermediate failures, +with the prior evidence. This qualifies low-level coherent reads and immutable +SQL in both image modes; it does not complete registered SDK query serving, +production compaction, CG-04 or the release gate. + +The preceding submission record is `.cellar/cell-groups-evidence/CG-04-submission/checkpoint.json` at `0cf89c40`: 197 selected Rust cases, ten successful commands, 189 source inputs and eight copied executables. Transfer the entire directory, including intermediate failures, diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index fbce25989..916afa722 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,38 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 read-pin capacity and repaired qualification + +Qualified source: `3da4481e09c3767face30469ffc855d302a7c142`. + +Read-image retention now uses `RaftRuntimeOptions::max_read_pin_bytes`, independently +of transport request capacity. The default remains 64 MiB. Node and group read +reservations use the explicit pin ceiling; ordinary peer/apply admission remains +separate. Validation bounds configured capacity by the native image-size envelope +and pending-read count using wide arithmetic. The new unit case admits the full +native image limit without raising transport limits, accepts the minimum metadata +reservation and upper envelope, and refuses out-of-range settings. The native +read fixture now uses `cellar::asyncrt::select!`. + +All 204 selected Rust cases pass: 186 native cases across seven suites, 11 SQL +queue/extent-worker tests, four receipt tests, two budget tests and one runtime +configuration test. All-target Clippy for Cellar/cellar-logic with warnings denied, +formatting (1,806 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator tests pass. The checkpoint at +`.cellar/cell-groups-evidence/CG-04-reads-final/checkpoint.json` binds 194 source +inputs, eight copied executables and twelve successful command logs. It retains +the failed full gate at `88c4717d` and earlier targeted compile/stack diagnostics +separately. No test limits or assertions were relaxed. Four process helpers run +through parent tests; three RustFS-specific cases remain unrun. + +This qualifies the low-level coherent-read and immutable-query boundary in both +image modes. CG-04 remains IN_PROGRESS. Registered SDK/catalog dispatch, callback +identity and version-token integration, real compaction/retry dispatch and cache +lifecycle remain required. In particular, the selected leaf retains its last +mutation position but not the corresponding mutation proposal ID: preserve a +complete authenticated cell commit witness before issuing SDK version observations +from that image after log compaction. Group progress is never that witness. + ## CG-04 coherent cell reads and immutable SQL queries Implementation source: `88c4717d`. @@ -49,17 +81,15 @@ The retained gate at `.cellar/cell-groups-evidence/CG-04-reads/gate-commands.jso completed four test commands: 203 selected Rust cases passed. The fifth command, all-target Clippy, failed because `tests/support/cell_group_sql_reads.rs` uses the forbidden `tokio::select!` macro. Subsequent commands in that gate did not run. -This is an unqualified source checkpoint; the latest fully passing partial -checkpoint remains `0cf89c40`. No task status advances. +This source was unqualified at that point; the repaired source and full gate are +recorded above. No task status advances. -Before resuming qualification, replace that fixture macro with -`cellar::asyncrt::select!`. Also separate the logical read-image pin budget from +The repair required replacing that fixture macro with +`cellar::asyncrt::select!` and separating the logical read-image pin budget from `RaftRuntimeOptions::max_request_bytes`: the current transport ceiling is 512 MiB, -whereas valid native images can reach 10,000,000,000 bytes. Add an explicit bounded -pin-byte option, validate its arithmetic and supported image envelope, and test -that pin configuration does not require raising transport limits. Preserve -node/group accounting, count limits and cancellation ownership. Retain the failed -gate, then run a fresh full gate against the repaired source revision. +whereas valid native images can reach 10,000,000,000 bytes. The repair above adds +the explicit bounded pin-byte option and validation test, preserves node/group accounting, count limits +and cancellation ownership, and retains this failed gate separately. Registered SDK dispatch, callback identity and version-token integration, catalog sessions, production retry/compaction and cache lifecycle remain required. This From ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 23:34:38 -0700 Subject: [PATCH 067/568] feat: retain cell version commit witnesses in durable manifests --- crates/cellar/durability/raft/README.md | 8 ++- .../durability/raft/cell_groups/README.md | 20 ++++++-- .../raft/cell_groups/enrollment_apply.rs | 5 ++ .../durability/raft/cell_groups/manifest.rs | 26 ++++++++++ .../durability/raft/cell_groups/owner.rs | 7 +++ .../durability/raft/cell_groups/page_wire.rs | 50 +++++++++++++++++++ .../raft/cell_groups/page_wire_tests.rs | 48 ++++++++++++++++++ .../durability/raft/cell_groups/root.rs | 2 + .../cellar/durability/raft/execution/cell.rs | 13 +++++ .../raft/host_sql/cell_groups/query.rs | 21 +++++--- .../tests/native_cell_groups_execution.rs | 32 ++++++++++++ .../tests/native_cell_groups_storage.rs | 5 ++ .../tests/support/cell_group_recovery.rs | 29 +++++++++++ .../tests/support/cell_group_sql_reads.rs | 36 +++++++++++-- 14 files changed, 286 insertions(+), 16 deletions(-) diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 8b7079e4d..2ed76a907 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -153,7 +153,13 @@ applied group prefix. A committed update to B can delay a read of unchanged A, but cannot change A's version. Raw read requests and forwarded read messages cannot use the internal context namespace. The returned authority is consumed by one query and rejects a changed route, lost leader term, foreign engine or expired -deadline. It is not a reusable leadership lease. +deadline. It is not a reusable leadership lease. Each selected leaf retains its complete +commit witness independently of the Raft entry cache. Immutable-query results +expose `observation()` for the exact version and witness; cell receipt results +expose the corresponding selected-version observation, which can cover an older +command receipt. A later mutation of another cell cannot change that observation. +Registered dispatch must still authenticate contracts and issue tokens only from +these authoritative results; manifest observation data alone is not authority. Stale explicitly selects a local committed image, including on followers. AtLeast compares logical cell history/sequence/image, not group log positions; the diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 290d0721d..5bc731b9a 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -15,9 +15,23 @@ JSON and a 16 KiB decode bound. Tree depth and cell count are explicit limits. These limits are admission ceilings, not qualified fleet capacities. Leaves bind logical cell history, mutation sequence, image commitment and -geometry, last group mutation position, schema, effect summary and immutable -original enrollment identity. A page without the required enrollment field -refuses at marker 1; no old-layout decoder or inferred creation record is used. Local image +geometry, last group installation/mutation position, schema, effect summary, +immutable original enrollment identity and the selected version's complete +`CommitWitness` (original group history, term/index and proposal ID). A page +without either the enrollment or witness field refuses at marker 1; no old-layout +decoder or inferred identity is used. The witness is part of the authenticated +page and survives unrelated group progress, restart and image collection. A local +installation position remains separate from original decision provenance for +future transfers. Shape validation of foreign-group provenance grants no transfer +authority; the transfer protocol must establish its lineage. + +Ordinary `prepare_update` requires the witness to name the current group and +its exact mutation position. The Raft owner additionally matches enrollment and +mutation witnesses to the actual committed entry's proposal before publishing. +An otherwise correct image cannot supply a fabricated proposal witness. Repeated +enrollment and unchanged entries retain the existing leaf and witness exactly. +`CellManifestLeaf::observation` exposes version/witness data; only an authoritative +selected root establishes its commitment. Local image filenames are excluded: replicas may store the same image under different local names and still derive the same logical manifest. A physical image adapter must resolve and pin each image by the selected logical leaf. Extent and materialized diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs index e5bd68f8f..bd7b724e4 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs @@ -110,6 +110,11 @@ impl ManifestTree { }, geometry: prepared.geometry, last_mutation: apply.position, + witness: CommitWitness { + group: self.route.history, + position: apply.position, + proposal: record.creation(), + }, effects: prepared.effects, }; let next = self.prepare_update(store, previous, leaf, None, prepared.pin)?; diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 26da49243..8598c3b0a 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -94,11 +94,24 @@ pub struct CellManifestLeaf { pub enrollment: super::EnrollmentRecord, pub version: CellVersion, pub geometry: CellImageGeometry, + /// Position that installed this image in the current group. pub last_mutation: LogPosition, + /// Original decision for this logical version, retained independently of + /// log entries and of later group progress or transfer installation. + pub witness: CommitWitness, pub effects: EffectSummary, } impl CellManifestLeaf { + /// Observation data from this leaf. Only an authoritative selected root + /// establishes commitment; constructing a leaf grants no serving authority. + pub fn observation(&self) -> crate::raft::version::CellVersionObservation { + crate::raft::version::CellVersionObservation { + version: self.version.clone(), + witness: self.witness, + } + } + pub(super) fn validate(&self, route: &GroupRoute) -> Result { let key = cell_routing_key(route.history.namespace, &self.version.history.cell); route @@ -110,6 +123,19 @@ impl CellManifestLeaf { if self.last_mutation.index.get() == 0 { return Err(ManifestError::Invalid("zero mutation index")); } + self.witness + .validate(&self.version.history) + .map_err(|_| ManifestError::Invalid("cell commit witness"))?; + if (self.witness.group == route.history + && (self.witness.position.index > self.last_mutation.index + || self.witness.position.term > self.last_mutation.term + || (self.witness.position.index == self.last_mutation.index + && self.witness.position.term != self.last_mutation.term))) + || (self.version.sequence.get() == 1 + && self.witness.proposal != self.enrollment.creation()) + { + return Err(ManifestError::Invalid("cell witness position/creation")); + } // The immutable record's constructor/decoder validates its bounded shape. // Do not allocate its canonical encoding on every tree lookup. let initial = self.enrollment.initial(); diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 3fcb449d3..33739078a 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -959,6 +959,11 @@ impl CellGroupOwner { GroupEntryLimits::default(), ) .map_err(|_| invalid())?; + let witness = cellar_logic::cell_group::CommitWitness { + group: self.current.marker().route().history, + position: entry_position(entry), + proposal: decoded.binding.proposal, + }; match decoded.command { GroupCommand::Enroll { ref cell, .. } => { let old = self.current.pin_cell(&cell.cell); @@ -976,6 +981,7 @@ impl CellGroupOwner { || leaf.version.sequence.get() != 1 || leaf.version.image != record.initial() || leaf.last_mutation != entry_position(entry) + || leaf.witness != witness { return Err(invalid()); } @@ -1001,6 +1007,7 @@ impl CellGroupOwner { || new.version.image != mutation.result.into() || new.enrollment != old.enrollment || new.last_mutation != entry_position(entry) + || new.witness != witness { return Err(invalid()); } diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs index 5605e1708..48e64a1e5 100644 --- a/crates/cellar/durability/raft/cell_groups/page_wire.rs +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -85,11 +85,59 @@ struct WirePage { geometry: CellImageGeometry, mutation_index: u64, mutation_term: u64, + witness: WireWitness, effects: EffectSummary, left: Option, right: Option, } +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct WireWitness { + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + index: u64, + term: u64, + proposal: [u8; 16], +} + +impl From for WireWitness { + fn from(witness: CommitWitness) -> Self { + Self { + cluster: witness.group.cluster.bytes(), + namespace: witness.group.namespace.bytes(), + group: witness.group.group.bytes(), + incarnation: witness.group.incarnation.bytes(), + index: witness.position.index.get(), + term: witness.position.term.get(), + proposal: witness.proposal.bytes(), + } + } +} + +impl TryFrom for CommitWitness { + type Error = ManifestError; + + fn try_from(wire: WireWitness) -> Result { + let invalid = |_| ManifestError::Invalid("cell commit witness"); + Ok(Self { + group: GroupHistory { + cluster: ClusterId::new(wire.cluster).map_err(invalid)?, + namespace: NamespaceId::new(wire.namespace).map_err(invalid)?, + group: GroupId::new(wire.group).map_err(invalid)?, + incarnation: IncarnationId::new(wire.incarnation).map_err(invalid)?, + }, + position: LogPosition { + index: LogIndex::new(wire.index), + term: Term::new(wire.term), + }, + proposal: ProposalId::new(wire.proposal).map_err(invalid)?, + }) + } +} + pub(super) fn digest(bytes: &[u8]) -> [u8; 32] { let mut hash = Sha256::new(); hash.update(b"cellar-cell-group-manifest-page-1\0"); @@ -122,6 +170,7 @@ pub(super) fn encode( geometry: leaf.geometry, mutation_index: leaf.last_mutation.index.get(), mutation_term: leaf.last_mutation.term.get(), + witness: leaf.witness.into(), effects: leaf.effects, left: page.left, right: page.right, @@ -181,6 +230,7 @@ pub(super) fn decode( index: LogIndex::new(wire.mutation_index), term: Term::new(wire.mutation_term), }, + witness: wire.witness.try_into()?, effects: wire.effects, }, left: wire.left, diff --git a/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs index a304ebd74..e3afe0d04 100644 --- a/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs +++ b/crates/cellar/durability/raft/cell_groups/page_wire_tests.rs @@ -35,6 +35,14 @@ fn fixture() -> (GroupRoute, Page) { initial, ) .unwrap(); + let witness = CommitWitness { + group: route.history, + position: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + proposal: enrollment.creation(), + }; ( route, Page { @@ -53,6 +61,7 @@ fn fixture() -> (GroupRoute, Page) { index: LogIndex::new(2), term: Term::new(1), }, + witness, effects: EffectSummary::default(), }, left: None, @@ -107,6 +116,8 @@ fn creation_binding_survives_later_versions_and_bounds_all_cell_names() { page.leaf.version.image.txid = 2; page.leaf.version.image.commitment = CellImageCommitment::Materialized([10; 32]); page.leaf.last_mutation.index = LogIndex::new(3); + page.leaf.witness.position = page.leaf.last_mutation; + page.leaf.witness.proposal = ProposalId::new([10; 16]).unwrap(); let bytes = encode(&page, &route, ManifestLimits::default()).unwrap(); let restored = decode(&bytes, &route, ManifestLimits::default()).unwrap(); assert_eq!(restored.leaf.enrollment, original); @@ -130,3 +141,40 @@ fn creation_binding_survives_later_versions_and_bounds_all_cell_names() { page.leaf ); } + +#[test] +fn commit_witness_is_required_and_bounded_at_marker_one() { + let (route, page) = fixture(); + let limits = ManifestLimits::default(); + let encoded = encode(&page, &route, limits).unwrap(); + let original = String::from_utf8(encoded[MAGIC.len()..].to_vec()).unwrap(); + let witness = serde_json::to_string(&WireWitness::from(page.leaf.witness)).unwrap(); + let replace = |field: &str, value: serde_json::Value| { + let parsed: serde_json::Value = serde_json::from_str(&witness).unwrap(); + witness.replacen( + &format!("\"{field}\":{}", parsed[field]), + &format!("\"{field}\":{value}"), + 1, + ) + }; + for changed in [ + None, + Some("null".to_owned()), + Some(replace("namespace", serde_json::json!(vec![80; 16]))), + Some(replace("group", serde_json::json!(vec![0; 16]))), + Some(replace("proposal", serde_json::json!(vec![0; 16]))), + Some(replace("proposal", serde_json::json!(vec![80; 16]))), + Some(replace("index", 0.into())), + Some(replace("index", 3.into())), + Some(replace("term", 2.into())), + ] { + let changed = match changed { + Some(value) => original.replacen(&witness, &value, 1), + None => original.replacen(&format!("\"witness\":{witness},"), "", 1), + }; + assert_ne!(changed, original); + let mut bytes = MAGIC.to_vec(); + bytes.extend(changed.as_bytes()); + assert!(decode(&bytes, &route, limits).is_err()); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index cd809267c..897960306 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -217,6 +217,8 @@ impl ManifestTree { if previous.marker.position.index.get().checked_add(1) != Some(leaf.last_mutation.index.get()) || leaf.last_mutation.term < previous.marker.position.term + || leaf.witness.group != self.route.history + || leaf.witness.position != leaf.last_mutation { return Err(ManifestError::Invalid("noncontiguous mutation")); } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 57f3a5d45..0e494592f 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -18,6 +18,17 @@ pub struct CellCommittedReceipt { pub receipt: StoredReceipt, pub version: CellVersion, pub position: LogPosition, + /// Decision for the selected version, which may cover a prior receipt. + pub witness: cellar_logic::cell_group::CommitWitness, +} + +impl CellCommittedReceipt { + pub fn observation(&self) -> crate::raft::version::CellVersionObservation { + crate::raft::version::CellVersionObservation { + version: self.version.clone(), + witness: self.witness, + } + } } pub enum CellStageOutcome { @@ -318,6 +329,7 @@ impl CellPrivateExecutor { receipt, version: authority.cell().version.clone(), position: authority.cell().last_mutation, + witness: authority.cell().witness, })), Ok(None) if authority.term() > self.authority.term() => { CellReconciliation::NotCommitted @@ -352,6 +364,7 @@ impl CellPrivateExecutor { receipt, version: authority.cell().version.clone(), position: authority.cell().last_mutation, + witness: authority.cell().witness, }) } diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index a159ec21f..2fac605fc 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -22,7 +22,7 @@ pub(super) struct Query { pub struct CellSqlQueryResult { result: Vec, - version: CellVersion, + observation: crate::raft::version::CellVersionObservation, _bytes: Arc, } @@ -31,8 +31,12 @@ impl CellSqlQueryResult { &self.result } + pub fn observation(&self) -> &crate::raft::version::CellVersionObservation { + &self.observation + } + pub fn version(&self) -> &CellVersion { - &self.version + &self.observation.version } } @@ -176,11 +180,12 @@ pub(super) fn run( ) .map_err(crate::raft::execution::storage)?; check(&query)?; - let version = query.authority.cell().version.clone(); - let result = - std::panic::catch_unwind(AssertUnwindSafe(|| (query.callback)(&database, &version))) - .unwrap_or_else(|panic| Err(crate::native_worker::panic_error(panic))) - .map_err(crate::raft::execution::storage)?; + let observation = query.authority.cell().observation(); + let result = std::panic::catch_unwind(AssertUnwindSafe(|| { + (query.callback)(&database, &observation.version) + })) + .unwrap_or_else(|panic| Err(crate::native_worker::panic_error(panic))) + .map_err(crate::raft::execution::storage)?; // The callback has been consumed, so check the remaining owners explicitly. if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { return Err(unavailable()); @@ -197,7 +202,7 @@ pub(super) fn run( Bytes::shrink(&mut query.bytes, result.len() + 4096)?; Ok(CellSqlOutcome::Queried(CellSqlQueryResult { result, - version, + observation, _bytes: query.bytes, })) } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index a4fa1a2f5..3b5de64fc 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -684,6 +684,11 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result Result<()> { .complete_apply(foreign, next.clone()), Err(RuntimeError::StaleCompletion) )); + // Correct image and apply position cannot authorize a fabricated proposal + // witness. Completion must match the exact committed cell-addressed entry. + let (leaf, image) = next.pin_cell(&expected[0].0.cell).context("changed cell")?; + let mut forged_leaf = leaf.clone(); + forged_leaf.witness.proposal = ProposalId::new([254; 16])?; + assert_ne!(forged_leaf.witness, leaf.witness); + let old = apply + .base + .pin_cell(&expected[0].0.cell) + .context("old cell")? + .0; + let forged = voter.tree.prepare_update( + &mut voter.pages, + &apply.base, + forged_leaf, + Some(&old.version), + image, + )?; + assert!(matches!( + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket.clone(), forged,), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(voter.owner.as_ref().unwrap().manifest().marker(), &before); let unchanged = voter .tree diff --git a/crates/cellar/tests/native_cell_groups_storage.rs b/crates/cellar/tests/native_cell_groups_storage.rs index cdbe5ad95..e75af0430 100644 --- a/crates/cellar/tests/native_cell_groups_storage.rs +++ b/crates/cellar/tests/native_cell_groups_storage.rs @@ -114,6 +114,11 @@ fn leaf(id: u8, pin: &GenerationPin, sequence: u64, index: u64) -> CellManifestL page_size: image.page_size, }, last_mutation: position(index), + witness: CommitWitness { + group: route().history, + position: position(index), + proposal: ProposalId::new([id; 16]).unwrap(), + }, effects: EffectSummary::default(), } } diff --git a/crates/cellar/tests/support/cell_group_recovery.rs b/crates/cellar/tests/support/cell_group_recovery.rs index 92009e632..3c92cf92d 100644 --- a/crates/cellar/tests/support/cell_group_recovery.rs +++ b/crates/cellar/tests/support/cell_group_recovery.rs @@ -160,6 +160,11 @@ impl Images { page_size: state.page_size, }, last_mutation: position(index), + witness: CommitWitness { + group: route().history, + position: position(index), + proposal: ProposalId::new([if state.txid.0 == 1 { cell } else { cell + 32 }; 16])?, + }, effects: EffectSummary::default(), }; let previous = base.map(|(leaf, _)| leaf.enrollment.clone()); @@ -529,6 +534,30 @@ async fn recover_and_check(path: &Path, mode: Mode, expected: Option) -> Re assert_eq!(leaf.enrollment.creation(), ProposalId::new([cell; 16])?); assert_eq!(leaf.enrollment.initial().txid, 1); assert_eq!(leaf.version.sequence.get(), sequence); + assert_eq!(leaf.witness.group, route().history); + assert_eq!( + leaf.witness.position, + position(if sequence == 2 { + 4 + } else { + u64::from(cell) + 1 + }) + ); + assert_eq!( + leaf.witness.proposal, + ProposalId::new([if sequence == 1 { cell } else { cell + 32 }; 16])? + ); + let keys = cellar::application::consensus::VersionKeys::new( + cellar::application::consensus::VersionSigningKey::new([72; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + let observation = leaf.observation(); + let token = keys.issue_cell(&observation, route().schema_digest, 100)?; + assert_eq!( + keys.verify_cell(&token, &history(cell), route().schema_digest, 101)?, + observation + ); assert_receipt(&pin, cell, sequence)?; leaves.push(leaf.clone()); } diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs index 4985a66ac..78f5a39d4 100644 --- a/crates/cellar/tests/support/cell_group_sql_reads.rs +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -71,6 +71,20 @@ async fn value( .await?) } +fn verify_observation(output: &CellSqlQueryResult, schema: [u8; 32]) -> Result<()> { + let keys = VersionKeys::new( + VersionSigningKey::new([73; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + let token = keys.issue_cell(output.observation(), schema, 100)?; + assert_eq!( + &keys.verify_cell(&token, &output.version().history, schema, 101)?, + output.observation(), + ); + Ok(()) +} + pub(crate) async fn coherent(rooted: bool) -> Result<()> { let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; let pool = pool(&fixture, 0)?; @@ -78,6 +92,7 @@ pub(crate) async fn coherent(rooted: bool) -> Result<()> { let ob = fixture.cell_options(&b).await?; let original = selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; let original_version = original.cell().version.clone(); + let original_observation = original.cell().observation(); drop(value(&pool, oa.clone(), original).await?); // A ready leader without any new peer responses cannot reuse a cached cut. @@ -118,6 +133,7 @@ pub(crate) async fn coherent(rooted: bool) -> Result<()> { ); let old_b = selected(&mut fixture, 0, &b, CellReadConsistency::Stale).await?; let old_b_version = old_b.cell().version.clone(); + let old_b_observation = old_b.cell().observation(); let encoded = CellGroupEntry::decode(prepared.bytes(), &route, GroupEntryLimits::default())?; let GroupCommand::Mutation(mutation) = encoded.command else { anyhow::bail!("B mutation") @@ -129,6 +145,11 @@ pub(crate) async fn coherent(rooted: bool) -> Result<()> { }; pool.mark_submitted(b.clone(), prepared.proposal()).await?; let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + let expected_witness = CommitWitness { + group: route.history, + position: accepted.position, + proposal: prepared.proposal(), + }; let mut held = BTreeMap::new(); tokio::time::timeout(Duration::from_secs(5), async { loop { @@ -182,22 +203,27 @@ pub(crate) async fn coherent(rooted: bool) -> Result<()> { let old_output = value(&pool, ob.clone(), old_b).await?; assert_eq!(old_output.result(), 0_i64.to_le_bytes()); assert_eq!(old_output.version(), &old_b_version); + assert_eq!(old_output.observation(), &old_b_observation); + verify_observation(&old_output, route.schema_digest)?; drop(old_output); assert_eq!(selected.cell().version, original_version); + assert_eq!(selected.cell().observation(), original_observation); assert!(selected.applied().index >= accepted.position.index); assert!(selected.barrier().context("fresh barrier")?.index >= accepted.position.index); assert_eq!( value(&pool, oa.clone(), selected).await?.result(), 0_i64.to_le_bytes() ); - drop( - pool.finish( + let receipt = pool + .finish( fixture.runtimes[0] .execution_authority(route.history, b.clone()) .await?, ) - .await?, - ); + .await?; + assert_eq!(receipt.receipt().witness, expected_witness); + assert_eq!(receipt.receipt().observation().version, expected_b); + drop(receipt); drop(prepared); let version_b = fixture.runtimes[0] @@ -216,6 +242,8 @@ pub(crate) async fn coherent(rooted: bool) -> Result<()> { let output = value(&pool, ob, ready_b).await?; assert_eq!(output.result(), 1_i64.to_le_bytes()); assert_eq!(output.version(), &version_b); + assert_eq!(output.observation().witness, expected_witness); + verify_observation(&output, route.schema_digest)?; drop(output); assert!(fixture.runtimes[0] .read_authority( From fd787d2956aacf4f0314aef37930e892023efb0a Mon Sep 17 00:00:00 2001 From: forhappy Date: Thu, 29 Oct 2026 23:52:16 -0700 Subject: [PATCH 068/568] docs: qualify durable cell witness observations --- docs/design/multi-raft/cell-groups/README.md | 6 ++- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/design.md | 8 +++- docs/design/multi-raft/cell-groups/handoff.md | 48 ++++++++++++------- .../design/multi-raft/cell-groups/progress.md | 38 +++++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 4 +- 6 files changed, 83 insertions(+), 25 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 7b903ae85..9a0ce3658 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -105,8 +105,10 @@ refusal and a shared node metadata budget. Prepared submission sealing and exact rejection cleanup are implemented at `0cf89c40`; see the latest checkpoint for qualification. Coherent shared read selection and immutable queries now exist at `88c4717d`, repaired and qualified at `3da4481e`: 204 selected Rust cases -and all twelve verification commands pass. The retained checkpoint includes -the earlier failed Clippy gate; see the handoff for its exact scope. +and all twelve verification commands pass. Durable cell-version witnesses and +query/receipt observations are implemented and qualified at `ed84df5d`: 214 +selected Rust cases and thirteen verification commands pass. See the handoff +for the exact scope and retained evidence. Registered SDK/catalog routing, query/version-token integration, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 9ad5e357c..21a9ca8ff 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "3da4481e09c3767face30469ffc855d302a7c142", + "source_revision": "ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/design.md b/docs/design/multi-raft/cell-groups/design.md index e44e4cce8..c2203ae95 100644 --- a/docs/design/multi-raft/cell-groups/design.md +++ b/docs/design/multi-raft/cell-groups/design.md @@ -149,7 +149,13 @@ cell's identity and base chain; batching is never a cross-cell transaction. The durable group root contains group history, applied position, routing/transition state, membership binding and an authenticated map of all cell image leaves. Each leaf contains cell history/version, immutable original enrollment identity, -last group mutation position and effect summary. Enrollment identity binds the +last group installation/mutation position, complete original `CommitWitness` +and effect summary. Keep original decision provenance separate from local +installation position through transfers. Bind each new witness to the actual +applied enrollment/mutation entry, retain it on unchanged-cell progress, and +recover it from the authenticated manifest without requiring compacted log +entries. Query and receipt observations carry this selected-version witness; +never manufacture a proposal ID from group progress. Enrollment identity binds the creation operation, original cell history, schema and initial image commitment; it is retained even after that original image is reclaimed. Use a content-addressed paged tree with bounded copy-on-write paths, not an O(cell-count) full manifest rewrite for every update. An engine transaction diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index a148359df..2423a2e6b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,9 +15,10 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `3da4481e`, qualifying coherent read selection -and immutable cell queries after decoupling pin capacity from transport limits -and repairing the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing +The reviewed source baseline is `ed84df5d`, retaining exact cell-version witnesses +in authenticated manifests and query/receipt observations. The preceding read +checkpoint at `3da4481e` decouples pin capacity from transport limits and repairs +the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing and exact rejection cleanup (`0cf89c40`) and bounded proposal acceptance and enrollment capacity (`446dd905`). Earlier checkpoints cover shared-worker initialization and term fencing (`6e09bbc7`, formatted at `82ae6877`), receipt lookup (`91c3ecfc`), @@ -164,9 +165,10 @@ read-only callback restrictions, version-token authentication and installed transfer lineage. `CellExecutionAuthority` and `CellReconciliationAuthority` still grant no strong-read authority; use the new request-owned read path. Do not synthesize a cell-scoped legacy group or treat the group barrier index -as the target cell's mutation position. SDK token issuance must retain the actual -cell decision witness through compaction, rather than invent a proposal identity -from the latest group progress. +as the target cell's mutation position. SDK token issuance should consume `CellSqlQueryResult::observation()` or +`CellCommittedReceipt::observation()`. These retain the selected version's actual +cell decision witness. Real compaction integration must preserve it; never use +the latest group progress or the retried command's proposal as a substitute. ## First executor session @@ -193,18 +195,21 @@ command against the authoritative cell receipt before admitting another producer gate printed by `show CG-04`. Registered SDK dispatch and registered strong-read serving remain absent; the low-level read-authority and immutable-query paths exist. -The read fixture macro and independent pin-byte capacity are repaired and -qualified at `3da4481e`; the earlier failed read gate is retained as intermediate -evidence. Next persist the selected version's complete commit witness in the -authenticated cell leaf and carry it through read/query results before registered -SDK token issuance. `raft/version.rs::CellVersionObservation` already requires -`CommitWitness`; the current leaf stores only `last_mutation` and the original -enrollment record, so a later mutation's proposal ID cannot be reconstructed -after log compaction. Validate the witness against the actual applied entry, -retain it across unchanged-cell group progress and restart, and exercise tokens -from both image modes. Keep physical apply position separate from provenance -when later implementing group transfer. Then integrate registered callbacks and -catalog invocation sessions without synthesizing legacy group identities. +The selected version's complete commit witness now persists in the authenticated +cell leaf and flows through query and receipt observations. The source-bound gate +at `ed84df5d` qualifies exact entry matching, unchanged-cell and older-pin +provenance, restart/image-collection recovery and token round-trips in both image +modes. Actual log-compaction and group-snapshot integration remain unqualified. + +Next integrate registered callbacks and catalog invocation sessions without +synthesizing legacy group identities. Preserve original invocation deadlines, +contracts and retry identities. Query callbacks must receive logical cell versions; +returned tokens must use the selected observation. The remaining production apply +adapter must consume `CellGroupEffect::Apply` on owning execution-domain workers, +prepare cell-addressed LTX images and complete manifests, and return exact apply +completions. Current native fixtures supply that preparation; do not put blocking +image work on the consensus loop or mistake fixture integration for host serving. +Keep physical apply position separate from provenance in later group transfers. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -230,6 +235,13 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The latest fully passing partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-witness/checkpoint.json` at `ed84df5d`: +214 selected Rust cases, thirteen successful commands, 194 source inputs and nine +copied executables. Transfer the full directory with prior evidence. It qualifies +durable selected-version witnesses and read/receipt observations; registered SDK +serving, real compaction, CG-04 and the release gate remain incomplete. + +The preceding coherent-read record is `.cellar/cell-groups-evidence/CG-04-reads-final/checkpoint.json` at `3da4481e`: 204 selected Rust cases, twelve successful commands, 194 source inputs and eight copied executables. Transfer the entire directory, including intermediate failures, diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 916afa722..60a4646b1 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,44 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 durable selected-version commit witnesses + +Qualified source: `ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2`. + +Authenticated cell manifest pages now require a complete selected-version +`CommitWitness`: original group history, term/index and proposal identity. The +local installation/mutation position remains separate for future group transfers. +Marker-1 pages without this required field refuse; there is no legacy decoder or +proposal inferred from later group progress. Shape checks bound IDs, namespace, +positions and initial enrollment identity. Ordinary update preparation requires +the current group and exact mutation position; the physical Raft owner additionally +matches the witness to the actual committed entry's proposal before publication. +Repeated enrollment and unchanged-cell progress preserve the selected witness. + +Immutable-query results expose their exact `CellVersionObservation`; committed +receipt results expose the selected-version observation, which can cover an older +command receipt. Both use the authenticated leaf rather than the entry cache. +Native tests reject a forged witness despite otherwise correct apply/image data, +preserve older pinned observations after a newer cell mutation, round-trip signed +tokens, and recover exact witnesses after restart and image collection in both +physical image modes. Codec tests reject missing/malformed witness fields. + +All 214 selected Rust cases pass: 193 native cases across eight suites and 21 +library cases covering SQL workers, receipts, budgets, options and manifest pages. +All-target Clippy for Cellar/cellar-logic with warnings denied, formatting (1,806 +Rust files), layout, dependency isolation, docs, source preflight and all 37 +plan-validator tests pass. The retained partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-witness/checkpoint.json` binds 194 source inputs, +nine copied executables and thirteen successful command logs. Pre-commit compile +and formatting logs are retained separately. Four process helpers run through +parent tests; three RustFS-specific cases remain unrun. + +CG-04 remains IN_PROGRESS. This proves witness persistence and observations at the +owning storage/apply/read boundaries, not actual log compaction, group snapshot +installation, registered SDK serving or S3 recovery. Next integrate registered +callback identity, query/version-token dispatch and catalog invocation sessions; +complete the production apply adapter, retry/compaction and cache lifecycle. + ## CG-04 read-pin capacity and repaired qualification Qualified source: `3da4481e09c3767face30469ffc855d302a7c142`. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 937db301c..851d6d5e9 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -498,7 +498,7 @@ ] }, { - "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture. Correlate each strong read to its own runtime-owned request context and absolute domain deadline; retain separate bounded node/group pin admission through delivery and physical SQL execution. Configure logical image-pin bytes independently of transport message/request bytes; validate overflow and supported image bounds, and test independent capacity and reservation cleanup. Query callbacks consume immutable pins on existing workers, preserve read-only SQL policy and contain panics. Deadline/cancellation must interrupt SQLite work while arbitrary callback ownership remains retained until it finishes; shutdown joins that work.", + "action": "Implement Stale, AtLeast and current-term ReadIndex for the target cell. Pin coherent manifest/image views and include cross-cell noninterference and slow-apply cases in the real-engine fixture. Correlate each strong read to its own runtime-owned request context and absolute domain deadline; retain separate bounded node/group pin admission through delivery and physical SQL execution. Configure logical image-pin bytes independently of transport message/request bytes; validate overflow and supported image bounds, and test independent capacity and reservation cleanup. Persist each selected version's full commit witness in the authenticated manifest and require exact proposal matching at apply. Keep original decision provenance separate from local installation position, retain it across unchanged-cell progress/restart and carry it through query and receipt observations. Missing witness fields must refuse at marker 1. Query callbacks consume immutable pins on existing workers, preserve read-only SQL policy and contain panics. Deadline/cancellation must interrupt SQLite work while arbitrary callback ownership remains retained until it finishes; shutdown joins that work.", "tests": [ "strong_read_at_group_barrier", "apply_prefix_never_skips_slow_cell", @@ -615,7 +615,7 @@ }, { "name": "rooted_strong_read_at_group_barrier", - "oracle": "Extent-backed strong reads require fresh quorum and contiguous apply behind another cell, preserve the target cell version, wait for AtLeast publication, retain older coherent pins and refuse wrong histories, expired deadlines and revoked leadership.", + "oracle": "Extent-backed strong reads require fresh quorum and contiguous apply behind another cell, preserve the target cell version, wait for AtLeast publication, retain older coherent pins and original cell witnesses for authenticated version tokens, and refuse wrong histories, expired deadlines and revoked leadership.", "invariants": [ "I01", "I05", From de97dd8b4607124f2256dfc7bc80cceab279d58b Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 00:14:16 -0700 Subject: [PATCH 069/568] docs: record executable cell-group plan delivery boundary --- docs/design/multi-raft/cell-groups/handoff.md | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 2423a2e6b..abdc038c1 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -173,6 +173,31 @@ the latest group progress or the retried command's proposal as a substitute. ## First executor session +### Design delivery check, 2026-10-30 + +The requested deliverable is the executable design and implementation plan. +The existing runtime checkpoints remain implementation evidence at their stated +boundaries; this delivery does not complete CG-04 or authorize a release claim. +Fetching origin confirmed that the branch contains `origin/main` at `8a93ffd7`. +The supplied TiKV checkout still matches the investigation pin above. + +Package validation passes: 16 tasks, 95 required cases, 14 invariants and nine +profiles. Repository documentation validation also passes. In the current working +tree, preflight deliberately refuses source drift in +`crates/cellar/durability/raft/host_sql/mod.rs`; consequently 36 of the 37 plan +validator tests pass and the actual-source preflight test errors. Do not refresh +the baseline merely to silence this refusal. + +The uncommitted draft adds `cell_groups/apply/` and +`host_sql/cell_groups/apply.rs`, with changes to the manifest, owner and worker +modules. It prepares cell images on the existing SQL workers and retains apply +admission. Preserve and review this draft before continuing CG-04. Its library +check completed, but it has no source-bound test or qualification checkpoint. +The untracked `diagram/` directory is separate existing work. Neither the draft +nor those diagrams forms part of this documentation-only delivery check. + +### Resume implementation + Inspect `git status` and the existing diff before editing. The physical owner in `crates/cellar/durability/raft/cell_groups/owner.rs` and its native execution fixture now use real three-voter Raft with complete group manifests and separate From 2085cc9697b462401bce1f49d4eda875bf01a7fd Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 00:39:21 -0700 Subject: [PATCH 070/568] feat: prepare committed cell-group images on shared workers --- crates/cellar-ltx/README.md | 7 + crates/cellar-ltx/src/generation.rs | 25 +- crates/cellar-ltx/tests/private_clone.rs | 73 +++++ .../raft/cell_groups/apply/images.rs | 260 ++++++++++++++++++ .../durability/raft/cell_groups/apply/mod.rs | 233 ++++++++++++++++ .../raft/cell_groups/apply/summary.rs | 51 ++++ .../raft/cell_groups/apply/summary_tests.rs | 36 +++ .../durability/raft/cell_groups/manifest.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 10 +- .../durability/raft/cell_groups/owner.rs | 2 + .../raft/host_sql/cell_groups/apply.rs | 128 +++++++++ .../raft/host_sql/cell_groups/mod.rs | 13 + crates/cellar/durability/raft/host_sql/mod.rs | 7 + .../tests/native_cell_groups_execution.rs | 33 ++- .../tests/support/cell_group_runtime.rs | 17 +- crates/cellar/tests/support/cell_group_sql.rs | 19 +- .../tests/support/cell_group_sql_apply.rs | 204 ++++++++++++++ .../support/cell_group_sql_enrollment.rs | 6 +- .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 39 ++- 20 files changed, 1146 insertions(+), 23 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/apply/images.rs create mode 100644 crates/cellar/durability/raft/cell_groups/apply/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/apply/summary.rs create mode 100644 crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs create mode 100644 crates/cellar/durability/raft/host_sql/cell_groups/apply.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_apply.rs diff --git a/crates/cellar-ltx/README.md b/crates/cellar-ltx/README.md index 75f3f7e5b..83c7c372f 100644 --- a/crates/cellar-ltx/README.md +++ b/crates/cellar-ltx/README.md @@ -37,6 +37,13 @@ identities, files, cached images and persistent storage live together in [`src/extent/`](src/extent/mod.rs), with the node inventory in `store/`. Public `Db`/`db` paths and the `internal` verification facade remain unchanged. +`GenerationStore::pin_prepared` reopens an exactly identified materialized image +for a preparation retry. It verifies cold image bytes and persistence barriers, +preserves the existing host retention flag, and refuses missing or mismatched +images. Its pin prevents reclamation while borrowed; it supplies no apply, +publication or serving authority. The caller must obtain the expected state from +its committed input and publish the resulting complete manifest separately. + `RootView::read_extent` supplies one complete dependency to an admitted blocking transfer owner. The caller reserves its at-most-4-MiB output. It refuses objects outside the pinned root and returns bytes only after the full extent identity diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index 9e9e86e48..7635d5c63 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -958,6 +958,27 @@ impl GenerationStore { &self, id: ImageId, expected: ImageState, + ) -> Result { + self.pin_existing(id, expected, true) + } + + /// Reopen an exactly identified prepared image without changing its host + /// retention flag. Cold images are independently verified and synced before + /// returning a pin. This is physical preparation only: the caller must prove + /// the expected image from its committed input and separately publish it. + pub fn pin_prepared( + &self, + id: ImageId, + expected: ImageState, + ) -> Result { + self.pin_existing(id, expected, false) + } + + fn pin_existing( + &self, + id: ImageId, + expected: ImageState, + retain: bool, ) -> Result { expected.validate(&self.inner.limits.apply)?; let token = { @@ -970,7 +991,7 @@ impl GenerationStore { if image.state != expected { return Err(GenerationError::Identity); } - entry.retained = true; + entry.retained |= retain; return Ok(GenerationPin { inner: self.inner.clone(), image: image.clone(), @@ -980,7 +1001,7 @@ impl GenerationStore { return Err(GenerationError::Missing); } entry.busy = true; - entry.retained = true; + entry.retained |= retain; entry.token.clone() }; let mut reservation = Reservation { diff --git a/crates/cellar-ltx/tests/private_clone.rs b/crates/cellar-ltx/tests/private_clone.rs index 10bcae2aa..376207f74 100644 --- a/crates/cellar-ltx/tests/private_clone.rs +++ b/crates/cellar-ltx/tests/private_clone.rs @@ -178,3 +178,76 @@ fn corrupt_clone_and_ambiguous_creation_never_publish_or_refund_uncertain_capaci .is_err()); assert_eq!(std::fs::read(original).unwrap(), bytes); } + +#[test] +fn prepared_image_retry_verifies_identity_without_restoring_retention() { + let root = tempfile::tempdir().unwrap(); + let original = root.path().join("source.sqlite"); + let bytes = source(&original); + let directory = root.path().join("images"); + let id = ImageId::new([1; 16]).unwrap(); + let expected = state(&bytes, 1); + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + drop(store.prepare_image_file(id, &original, expected).unwrap()); + store.retire(id).unwrap(); + assert_eq!(store.usage().unwrap().retained, 0); + let pin = store.pin_prepared(id, expected).unwrap(); + assert_eq!(store.usage().unwrap().retained, 0); + assert_eq!(store.reclaim(10).unwrap().files, 0); + assert!(matches!( + store.pin_prepared( + id, + ImageState { + txid: TXID(2), + ..expected + } + ), + Err(GenerationError::Identity) + )); + drop(pin); + drop(store); + // Restart must verify the file, not infer publication from directory order. + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + store.retire(id).unwrap(); + let pin = store.pin_prepared(id, expected).unwrap(); + assert_eq!(pin.state(), expected); + assert_eq!(store.usage().unwrap().retained, 0); + drop(pin); + assert!(store.reclaim(10).unwrap().files > 0); + assert!(matches!( + store.pin_prepared(id, expected), + Err(GenerationError::Missing) + )); + drop(store); + let store = GenerationStore::open( + directory.clone(), + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + drop(store.prepare_image_file(id, &original, expected).unwrap()); + drop(store); + let image = std::fs::read_dir(&directory) + .unwrap() + .map(|entry| entry.unwrap().path()) + .find(|path| path.extension().is_some_and(|ext| ext == "sqlite")) + .unwrap(); + std::fs::write(image, b"corrupt immutable image").unwrap(); + let store = GenerationStore::open( + directory, + Arc::new(DirectFileSystem), + GenerationLimits::default(), + ) + .unwrap(); + assert!(store.pin_prepared(id, expected).is_err()); +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs new file mode 100644 index 000000000..4676c88a4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -0,0 +1,260 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::application::consensus::{group_protocol::GroupLtx, ImageCommitment, LtxImage}; +use cellar_logic::cell_group::CellHistory; +use cellar_ltx::{ + apply::{ApplyLimits, ImageState, ImageUpdate, LtxSegment}, + generation::{GenerationStore, ImageId, ReadServices}, + internal::{ + CacheLimits, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, + NodeExtentStore, EXTENT_BYTES, + }, + stream_snapshot::ScratchPool, + TXID, +}; +use sha2::{Digest, Sha256}; +use std::path::PathBuf; + +#[derive(Clone)] +pub enum CellGroupApplyImages { + Materialized { + images: GenerationStore, + services: ReadServices, + }, + Extents { + images: NodeExtentStore, + limits: ApplyLimits, + services: ReadServices, + cache_directory: PathBuf, + cache_pool: ScratchPool, + cache: CacheLimits, + }, +} + +pub(super) struct ImageApply<'a> { + pub cell: &'a CellHistory, + pub sequence: u64, + pub schema: [u8; 32], + pub base: Option<&'a CellImagePin>, + pub image: LtxImage, + pub files: &'a [GroupLtx<'a>], + pub max_effect_rows: u32, +} + +impl CellGroupApplyImages { + pub(super) fn memory_bound(&self) -> usize { + let limits = match self { + Self::Materialized { images, .. } => images.apply_limits(), + Self::Extents { limits, .. } => *limits, + }; + // Limits validate decoded bytes to at most 256 MiB. Include one extent + // scratch buffer even for materialized mode to keep admission uniform. + limits.max_total_decoded_bytes as usize + EXTENT_BYTES as usize + } + + pub(super) fn validate(&self) -> Result<(), ExecutionError> { + match self { + Self::Materialized { images, .. } => images.apply_limits().validate().map_err(storage), + Self::Extents { + limits, + cache_directory, + .. + } => { + limits.validate().map_err(storage)?; + if cache_directory.as_os_str().is_empty() { + return Err(ExecutionError::Invalid("apply cache directory")); + } + Ok(()) + } + } + } + + pub(super) fn prepare( + &self, + request: ImageApply<'_>, + ) -> Result { + let ImageApply { + cell, + sequence, + schema, + base, + image, + files, + max_effect_rows, + } = request; + if image.txid == 0 { + if base.is_some() + || !files.is_empty() + || image.checksum != 0 + || image.commitment != ImageCommitment::Sha256(Sha256::digest([]).into()) + { + return Err(ExecutionError::Invalid("noncanonical empty cell image")); + } + return Ok(EnrollmentImage { + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + effects: EffectSummary::default(), + pin: CellImagePin::Empty, + }); + } + let header = cellar_ltx::ltx::Header::parse( + files + .last() + .ok_or(ExecutionError::Invalid("apply LTX missing"))? + .bytes, + ) + .map_err(storage)?; + let geometry = CellImageGeometry { + page_size: header.page_size, + length: u64::from(header.page_size) * u64::from(header.commit), + }; + let segments = files + .iter() + .map(|file| { + Ok(LtxSegment { + bytes: file.bytes, + min_txid: TXID(file.min_txid), + max_txid: TXID(file.max_txid), + page_count: usize::try_from(file.page_count) + .map_err(|_| ExecutionError::Invalid("apply page count"))?, + decoded_bytes: file.decoded_bytes, + }) + }) + .collect::, ExecutionError>>()?; + let pin = match (self, image.commitment) { + (Self::Materialized { images, .. }, ImageCommitment::Sha256(digest)) => { + let base = match base { + Some(CellImagePin::Materialized(pin)) => Some(pin), + Some(CellImagePin::Empty) | None => None, + _ => return Err(ExecutionError::Invalid("materialized apply base")), + }; + let mut id = Sha256::new(); + id.update(b"cellar-cell-apply-image-1\0"); + id.update(cell_image_history_digest(cell)); + for value in [sequence, image.epoch, image.txid, image.checksum] { + id.update(value.to_be_bytes()); + } + id.update(digest); + let bytes: [u8; 32] = id.finalize().into(); + let id = ImageId::new(bytes[..16].try_into().map_err(storage)?).map_err(storage)?; + let update = ImageUpdate { + base: match base { + Some(pin) => pin.state(), + None => ImageState::empty(image.epoch).map_err(storage)?, + }, + result: ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: geometry.page_size, + length: geometry.length, + }, + segments, + }; + CellImagePin::Materialized(match images.prepare(id, base, &update) { + Ok(pin) => pin, + Err(cellar_ltx::generation::GenerationError::Exists) => { + images.pin_prepared(id, update.result).map_err(storage)? + } + Err(error) => return Err(storage(error)), + }) + } + (Self::Extents { images, limits, .. }, ImageCommitment::Extents(result)) => { + let base = match base { + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + Some(CellImagePin::Empty) | None => None, + _ => return Err(ExecutionError::Invalid("extent apply base")), + }; + let binding = ImageBinding { + history: cell_image_history_digest(cell), + schema, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0).map_err(storage)?, + &[], + ) + .map_err(storage)?; + let before = match base { + Some(pin) => pin.reference(), + None => ExtentRootReference::new( + empty.root(), + empty + .encode() + .map_err(storage)? + .len() + .try_into() + .map_err(storage)?, + ) + .map_err(storage)?, + }; + let update = ImageRootUpdate { + base: before, + result, + binding, + geometry: Geometry::new(geometry.page_size, geometry.length) + .map_err(storage)?, + segments, + }; + let prepared = images + .prepare_image_root(base, &update, limits, &mut vec![0; EXTENT_BYTES as usize]) + .map_err(storage)?; + CellImagePin::Extents(Arc::new(prepared.image)) + } + _ => return Err(ExecutionError::Invalid("apply image representation")), + }; + let database = match (self, &pin) { + (Self::Materialized { services, .. }, CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin.clone(), + crate::storage::ImmutableQueryOptions { + services: services.clone(), + sqlite_vec: false, + }, + ) + } + ( + Self::Extents { + services, + cache_directory, + cache_pool, + cache, + .. + }, + CellImagePin::Extents(pin), + ) => { + let image = pin + .view() + .image(cache_directory.clone(), cache_pool.clone(), *cache) + .map_err(storage)?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services: services.clone(), + sqlite_vec: false, + }, + ) + } + _ => return Err(ExecutionError::Invalid("apply image query")), + } + .map_err(storage)?; + let effects = summary::read(database.connection(), max_effect_rows)?; + drop(database); + Ok(EnrollmentImage { + pin, + geometry, + effects, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs new file mode 100644 index 000000000..269437808 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -0,0 +1,233 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Worker-owned physical preparation; only the Raft owner can publish its result. + +mod images; +mod summary; +pub use images::CellGroupApplyImages; + +use super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryLimits, +}; +use crate::raft::{ + cache::GroupLog, + completion::{entry_digest, entry_position}, + execution::{storage, ExecutionError}, +}; +use cellar_logic::cell_group::{CommitWitness, GroupHistory}; +use std::sync::{Arc, Mutex}; + +struct Storage { + tree: ManifestTree, + pages: ManifestPageStore, + images: CellGroupApplyImages, + limits: GroupEntryLimits, + max_effect_rows: u32, +} + +/// Owns the already opened manifest directory and shared physical image stores. +/// Construction performs no I/O. Opening/recovering the stores belongs on the +/// owning execution domain; transfer the same page owner used for startup. +#[derive(Clone)] +pub struct CellGroupApplier { + group: GroupLog, + storage: Arc>, +} + +impl CellGroupApplier { + pub fn new( + group: GroupLog, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + limits.validate().map_err(storage)?; + images.validate()?; + if !(1..=1_000_000).contains(&max_effect_rows) { + return Err(ExecutionError::Invalid("apply effect summary limit")); + } + let marker = group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("group manifest missing"))?; + let tree = ManifestTree::new(marker.route().clone(), manifest_limits).map_err(storage)?; + Ok(Self { + group, + storage: Arc::new(Mutex::new(Storage { + tree, + pages, + images, + limits, + max_effect_rows, + })), + }) + } + + pub fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { + Arc::ptr_eq(&self.group.owner, engine) + } + + pub(crate) fn reservation_bytes(&self, work: &CellGroupApply) -> Result { + self.check(work)?; + let owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned group applier") + } + })?; + work.base + .reserved_bytes() + .checked_add(work.entry.data.len()) + .and_then(|n| n.checked_add(owner.images.memory_bound())) + .and_then(|n| n.checked_add(4096)) + .ok_or(ExecutionError::Invalid("apply reservation overflow")) + } + + fn check(&self, work: &CellGroupApply) -> Result<(), ExecutionError> { + if !Arc::ptr_eq(&self.group.owner, &work.base.engine_owner) + || work.base.marker().route().history != self.history() + || work.ticket.position() != entry_position(&work.entry) + || work.entry.get_entry_type() != ::raft::prelude::EntryType::EntryNormal + { + return Err(ExecutionError::Invalid("group apply identity/entry")); + } + let cache = self.group.read().map_err(storage)?; + if cache.group_manifest.as_ref() != Some(work.base.marker()) + || cache.hard_state.commit < work.entry.index + || cache.entries.get(&work.entry.index) != Some(&work.entry) + || !cache.membership.permits_group(self.group.identity()) + { + return Err(ExecutionError::Invalid("group apply committed base")); + } + drop(cache); + if work.ticket.entry_digest() != entry_digest(&work.entry).map_err(storage)? { + return Err(ExecutionError::Invalid("group apply entry digest")); + } + Ok(()) + } + + /// Called only by the shared blocking SQL worker. The try-lock detects an + /// invalid concurrent owner instead of blocking one worker behind another. + pub(crate) fn prepare( + &self, + work: &CellGroupApply, + ) -> Result { + self.check(work)?; + let mut owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned group applier") + } + })?; + let Storage { + tree, + pages, + images, + limits, + max_effect_rows, + } = &mut *owner; + // Check page owner and contiguous prefix before any image allocation/I/O. + let unchanged = tree + .prepare_unchanged(pages, &work.base, work.ticket.position()) + .map_err(storage)?; + if work.entry.data.is_empty() { + return Ok(unchanged); + } + let decoded = CellGroupEntry::decode(&work.entry.data, work.base.marker().route(), *limits) + .map_err(storage)?; + let witness = CommitWitness { + group: self.history(), + position: work.ticket.position(), + proposal: decoded.binding.proposal, + }; + let next = match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let decision = decide_enrollment( + tree.route(), + EnrollmentPhase::Active, + &work.entry.data, + work.base + .pin_cell(&cell.cell) + .map(|(leaf, _)| &leaf.enrollment), + *limits, + ); + match decision { + Ok(EnrollmentDecision::Existing(_)) | Err(EnrollmentError::Conflict) => { + unchanged + } + Ok(EnrollmentDecision::Propose(record)) => { + let prepared = images.prepare(images::ImageApply { + cell: &cell, + sequence: 1, + schema: tree.route().schema_digest, + base: None, + image, + files: &files, + max_effect_rows: *max_effect_rows, + })?; + let leaf = CellManifestLeaf { + enrollment: record, + version: cellar_logic::cell_group::CellVersion { + history: cell, + sequence: cellar_logic::cell_group::MutationSequence::new(1) + .map_err(storage)?, + image: image.into(), + }, + geometry: prepared.geometry, + last_mutation: work.ticket.position(), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, &work.base, leaf, None, prepared.pin) + .map_err(storage)? + } + Err(error) => return Err(storage(error)), + } + } + GroupCommand::Mutation(mutation) => { + let (old, base) = work + .base + .pin_cell(&mutation.cell.cell) + .ok_or(ExecutionError::Invalid("apply cell missing"))?; + mutation.check_base(&old.version).map_err(storage)?; + let prepared = images.prepare(images::ImageApply { + cell: &mutation.cell, + sequence: mutation.sequence.get(), + schema: tree.route().schema_digest, + base: Some(&base), + image: mutation.result, + files: &mutation.files, + max_effect_rows: *max_effect_rows, + })?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: cellar_logic::cell_group::CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: prepared.geometry, + last_mutation: work.ticket.position(), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, &work.base, leaf, Some(&old.version), prepared.pin) + .map_err(storage)? + } + _ => { + return Err(ExecutionError::Invalid( + "unsupported group apply administration", + )) + } + }; + self.check(work)?; + Ok(next) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary.rs b/crates/cellar/durability/raft/cell_groups/apply/summary.rs new file mode 100644 index 000000000..3afe46b97 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/summary.rs @@ -0,0 +1,51 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use rusqlite::OptionalExtension; + +pub(super) fn read( + connection: &rusqlite::Connection, + max: u32, +) -> Result { + let scan = |table: &str, column: &str| -> Result<(u64, Option), ExecutionError> { + let kind: Option = connection + .query_row( + "SELECT type FROM sqlite_schema WHERE name=?1", + [table], + |row| row.get(0), + ) + .optional() + .map_err(storage)?; + match kind.as_deref() { + None => return Ok((0, None)), + Some("table") => (), + _ => return Err(ExecutionError::Invalid("effect summary schema")), + } + // Only constant internal identifiers reach this function. Limit the scan + // before aggregation; corrupt/unbounded effect state cannot monopolize a worker. + let sql = format!("SELECT COUNT(*), MIN(value), COALESCE(SUM(invalid),0) FROM (SELECT {column} AS value, CASE WHEN typeof({column})='integer' AND {column}>=0 THEN 0 ELSE 1 END AS invalid FROM {table} LIMIT ?1)"); + let (count, earliest, invalid): (u64, Option, u64) = connection + .query_row(&sql, [u64::from(max) + 1], |row| { + Ok((row.get(0)?, row.get(1)?, row.get(2)?)) + }) + .map_err(storage)?; + if count > u64::from(max) || invalid != 0 { + return Err(ExecutionError::Invalid("effect summary bounds/state")); + } + Ok((count, earliest)) + }; + let (_, next_alarm_unix_millis) = scan("__sys_raft_alarms", "due")?; + let (pending_outbox, next_outbox_unix_millis) = scan("__sys_raft_outbox", "next_at")?; + Ok(EffectSummary { + next_alarm_unix_millis, + next_outbox_unix_millis, + pending_outbox, + // The current inbox is synchronous receipt/deduplication state, not an + // asynchronous work queue. Retained inbox receipts cannot prevent parking. + pending_inbox: 0, + }) +} + +#[cfg(test)] +#[path = "summary_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs new file mode 100644 index 000000000..b945a6002 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs @@ -0,0 +1,36 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[test] +fn effects_are_bounded_and_inbox_receipts_do_not_schedule_work() { + let db = rusqlite::Connection::open_in_memory().unwrap(); + assert_eq!(read(&db, 2).unwrap(), EffectSummary::default()); + db.execute_batch("CREATE TABLE __sys_raft_inbox(sequence); INSERT INTO __sys_raft_inbox VALUES(1); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(20),(0); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(30),(10);").unwrap(); + assert_eq!( + read(&db, 2).unwrap(), + EffectSummary { + next_alarm_unix_millis: Some(0), + next_outbox_unix_millis: Some(10), + pending_outbox: 2, + pending_inbox: 0, + } + ); + assert!(read(&db, 1).is_err()); +} + +#[test] +fn malformed_effect_state_refuses_instead_of_reporting_idle() { + for ddl in [ + "CREATE VIEW __sys_raft_alarms AS SELECT 10 AS due", + "CREATE TABLE __sys_raft_alarms(other)", + "CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(-1)", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(NULL)", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES('10')", + "CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(1.5)", + ] { + let db = rusqlite::Connection::open_in_memory().unwrap(); + db.execute_batch(ddl).unwrap(); + assert!(read(&db, 100).is_err(), "{ddl}"); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 8598c3b0a..9ae542f6d 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -202,6 +202,10 @@ pub struct ManifestTree { } impl ManifestTree { + pub(crate) fn route(&self) -> &GroupRoute { + &self.route + } + pub fn new(route: GroupRoute, limits: ManifestLimits) -> Result { limits.validate()?; for bound in [route.range.lower(), route.range.upper()] { diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index d1f8e6aad..e29e457ab 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -3,11 +3,15 @@ //! Complete group-state storage. A synced page is data, not Raft authority. pub use super::host_sql::cell_groups::{ - CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, CellSqlLimits, - CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlQueryResult, CellSqlReceipt, - CellSqlReconciliation, CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, + CellGroupPreparedApply, CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, + CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlQueryResult, + CellSqlReceipt, CellSqlReconciliation, CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, + CellSqlUsage, }; +mod apply; +pub use apply::{CellGroupApplier, CellGroupApplyImages}; + mod execution_authority; pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 33739078a..75e2eaba0 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -42,6 +42,7 @@ pub struct CellGroupTicket { position: LogPosition, digest: [u8; 32], pub(super) admission: Option>, + pub(crate) preparation: Option>, } impl CellGroupTicket { @@ -713,6 +714,7 @@ impl CellGroupOwner { position, digest, admission: None, + preparation: None, }) } diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs b/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs new file mode 100644 index 000000000..57b773c10 --- /dev/null +++ b/crates/cellar/durability/raft/host_sql/cell_groups/apply.rs @@ -0,0 +1,128 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Apply uses the existing workers' reserved progress lane, independently of SQL producers. + +use super::*; +use crate::raft::cell_groups::{ + CellGroupApplier, CellGroupApply, CellGroupRuntime, GroupManifest, PreparedGroupManifest, +}; +use sha2::{Digest, Sha256}; + +pub(in crate::raft::host_sql) struct ApplyRequest { + pub group: GroupHistory, + applier: CellGroupApplier, + work: CellGroupApply, + reply: oneshot::Sender>, + _slot: tokio::sync::OwnedSemaphorePermit, +} + +/// Retains physical preparation and its admission through delivery, retries and +/// owner completion. Successful submission is not a durable apply acknowledgement. +#[derive(Clone)] +pub struct CellGroupPreparedApply { + work: CellGroupApply, + prepared: PreparedGroupManifest, +} + +impl CellGroupPreparedApply { + pub fn manifest(&self) -> &GroupManifest { + self.prepared.marker() + } + + pub async fn complete(self, runtime: &CellGroupRuntime) -> Result<(), RuntimeError> { + runtime + .complete_apply( + self.prepared.marker().route().history, + self.work, + self.prepared, + ) + .await + } +} + +impl CellGroupSqlPool { + /// Runs image and manifest I/O on the existing fixed workers. A cancelled + /// waiter does not cancel physical work or release its ticket reservations. + pub async fn prepare_apply( + &self, + applier: &CellGroupApplier, + mut work: CellGroupApply, + ) -> Result { + self.admit()?; + if !applier.belongs_to_engine(&self.inner.engine_owner) { + return Err(ExecutionError::Invalid("apply pool engine")); + } + let bytes = applier.reservation_bytes(&work)?; + let reservation = self + .inner + .apply_admission + .acquire(bytes) + .ok_or_else(overloaded)?; + work.ticket.preparation = Some(Arc::new(reservation)); + let group = applier.history(); + let mut hash = Sha256::new(); + for id in [ + group.cluster.bytes(), + group.namespace.bytes(), + group.group.bytes(), + group.incarnation.bytes(), + ] { + hash.update(id); + } + let hash: [u8; 32] = hash.finalize().into(); + let index = u64::from_le_bytes( + hash[..8] + .try_into() + .map_err(|_| ExecutionError::Invalid("apply worker key"))?, + ) % self.inner.workers.len() as u64; + let worker = &self.inner.workers[index as usize]; + worker.domain.mono_ms().map_err(|_| unavailable())?; + let slot = worker + .completion_slots + .clone() + .try_acquire_owned() + .map_err(|_| overloaded())?; + self.admit()?; + let (reply, wait) = oneshot::channel(); + worker + .sender + .try_send(Envelope::GroupApply(Box::new(ApplyRequest { + group, + applier: applier.clone(), + work, + reply, + _slot: slot, + }))) + .map_err(|_| unavailable())?; + wait.await.map_err(|_| unavailable())? + } +} + +impl ApplyRequest { + pub fn execute(self, domain: &crate::asyncrt::DomainToken, stopped: &AtomicBool) { + let Self { + applier, + work, + reply, + _slot, + .. + } = self; + let result = if stopped.load(Ordering::Acquire) || domain.mono_ms().is_err() { + Err(unavailable()) + } else { + std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| applier.prepare(&work))) + .unwrap_or_else(|panic| { + Err(crate::raft::execution::storage( + crate::native_worker::panic_error(panic), + )) + }) + .and_then(|prepared| { + if stopped.load(Ordering::Acquire) || domain.mono_ms().is_err() { + return Err(unavailable()); + } + Ok(CellGroupPreparedApply { work, prepared }) + }) + }; + let _ = reply.send(result); + } +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 1c1b0cff0..bcd65b2d9 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -3,6 +3,9 @@ //! Cell-history admission and commands on the existing fixed node SQL workers. mod admission; +mod apply; +pub(in crate::raft::host_sql) use apply::ApplyRequest; +pub use apply::CellGroupPreparedApply; mod query; mod worker; pub use query::CellSqlQueryResult; @@ -38,6 +41,8 @@ pub struct CellSqlLimits { pub queue_per_worker: usize, pub max_pending_cells: usize, pub group_pending_cells: usize, + /// Each of the SQL producer and independent apply budgets has this ceiling. + /// Configure their combined envelope as twice this value. pub node_bytes: usize, pub group_bytes: usize, } @@ -248,6 +253,9 @@ pub enum CellSqlReconciliation { } pub struct CellSqlUsage { + /// Independent progress budget; not included in SQL producer retention. + pub apply_retained_bytes: usize, + pub apply_retained_operations: usize, pub active_cells: usize, pub retained_bytes: usize, pub retained_operations: usize, @@ -267,6 +275,7 @@ struct Handle { workers: Vec, resources: ResourceObserver, admission: Arc, + apply_admission: Arc, closing: Arc, stop: Arc, terminal: Arc, @@ -319,6 +328,7 @@ impl CellGroupSqlPool { workers, resources, admission: Admission::new(limits, lease.clone()), + apply_admission: Budget::new(limits.node_bytes, log.limits().max_groups), closing: closing.clone(), stop: stop.clone(), terminal: terminal.clone(), @@ -347,7 +357,10 @@ impl CellGroupSqlPool { pub fn usage(&self) -> Result { let (active_cells, retained_bytes, retained_operations) = self.inner.admission.usage()?; let resources = self.inner.resources.capture().ok_or_else(unavailable)?; + let (apply_retained_bytes, apply_retained_operations) = self.inner.apply_admission.usage(); Ok(CellSqlUsage { + apply_retained_bytes, + apply_retained_operations, active_cells, retained_bytes, retained_operations, diff --git a/crates/cellar/durability/raft/host_sql/mod.rs b/crates/cellar/durability/raft/host_sql/mod.rs index e274a0d70..8c211915d 100644 --- a/crates/cellar/durability/raft/host_sql/mod.rs +++ b/crates/cellar/durability/raft/host_sql/mod.rs @@ -231,6 +231,7 @@ impl Scheduled for Request { enum Envelope { Legacy(Box), Shared(cell_groups::Request), + GroupApply(Box), } impl Scheduled for Envelope { @@ -238,6 +239,7 @@ impl Scheduled for Envelope { match (self, other) { (Self::Legacy(a), Self::Legacy(b)) => a.same_cell(b), (Self::Shared(a), Self::Shared(b)) => a.cell == b.cell, + (Self::GroupApply(a), Self::GroupApply(b)) => a.group == b.group, _ => false, } } @@ -246,6 +248,7 @@ impl Scheduled for Envelope { match self { Self::Legacy(request) => request.progress_priority(), Self::Shared(request) => request.progress_priority(), + Self::GroupApply(_) => true, } } } @@ -469,6 +472,10 @@ impl Pool { Err(mpsc::RecvTimeoutError::Disconnected) => break Ok(()), }; let request = match request { + Envelope::GroupApply(request) => { + request.execute(&owner, &stopped); + continue; + } Envelope::Shared(request) => { shared.execute( request, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 3b5de64fc..5fb850bb1 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -152,7 +152,7 @@ use std::{collections::VecDeque, sync::Arc, time::Duration}; struct Voter { log: ReplicaLog, owner: Option, - pages: ManifestPageStore, + pages: Option, images: GenerationStore, extents: Option, tree: ManifestTree, @@ -273,7 +273,7 @@ impl Cluster { voters.push(Voter { log, owner: Some(owner), - pages, + pages: Some(pages), images, extents: None, tree, @@ -459,7 +459,7 @@ impl Cluster { log, owner: Some(owner), tree, - pages, + pages: Some(pages), images, extents: None, gate, @@ -628,7 +628,7 @@ fn prepare_cell_image( fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { if apply.entry.data.is_empty() { return Ok(voter.tree.prepare_unchanged( - &voter.pages, + voter.pages.as_ref().context("fixture page owner")?, &apply.base, apply.ticket.position(), )?); @@ -641,7 +641,7 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { let (next, _) = voter.tree.prepare_enrollment( - &mut voter.pages, + voter.pages.as_mut().context("fixture page owner")?, &apply.base, EnrollmentApply { position: apply.ticket.position(), @@ -692,7 +692,7 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result Result<()> { .context("old cell")? .0; let forged = voter.tree.prepare_update( - &mut voter.pages, + voter.pages.as_mut().context("fixture page owner")?, &apply.base, forged_leaf, Some(&old.version), @@ -1005,10 +1005,11 @@ async fn apply_prefix_never_skips_slow_cell() -> Result<()> { Err(RuntimeError::Invalid(_)) )); assert_eq!(voter.owner.as_ref().unwrap().manifest().marker(), &before); - let unchanged = - voter - .tree - .prepare_unchanged(&voter.pages, &apply.base, apply.ticket.position())?; + let unchanged = voter.tree.prepare_unchanged( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + apply.ticket.position(), + )?; assert!(matches!( voter .owner @@ -1238,3 +1239,13 @@ async fn cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { async fn rooted_cell_query_pins_deadline_and_cancelled_cleanup() -> Result<()> { runtime_fixture::query_lifecycle(true).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_workers_prepare_committed_cell_images() -> Result<()> { + runtime_fixture::apply_images(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_workers_prepare_committed_cell_images() -> Result<()> { + runtime_fixture::apply_images(true).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index e811c3348..a8a98893b 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -6,6 +6,7 @@ use super::*; use std::collections::BTreeMap; struct RuntimeCluster { + appliers: BTreeMap<(usize, GroupHistory), (CellGroupSqlPool, CellGroupApplier)>, cluster: Cluster, runtimes: Vec, events: Vec, @@ -51,6 +52,7 @@ impl RuntimeCluster { } let applied = BTreeMap::from([(cluster.birth.route().history, applied)]); Ok(Self { + appliers: BTreeMap::new(), cluster, runtimes, events, @@ -81,6 +83,13 @@ impl RuntimeCluster { if n == 0 && hold.contains(&scope) { assert!(held.insert(scope, work).is_none()); } else { + if let Some((pool, applier)) = self.appliers.get(&(n, scope)) { + pool.prepare_apply(applier, work) + .await? + .complete(&self.runtimes[n]) + .await?; + continue; + } let voter = if scope == primary { &mut self.cluster.voters[n] } else { @@ -171,6 +180,10 @@ impl RuntimeCluster { } async fn close(self) -> Result<()> { + for (pool, _) in self.appliers.values() { + pool.close().await?; + } + drop(self.appliers); for runtime in &self.runtimes { runtime.close().await?; } @@ -333,7 +346,7 @@ impl RuntimeCluster { voters.push(Voter { log, owner: None, - pages, + pages: Some(pages), images, extents: None, tree, @@ -626,3 +639,5 @@ pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { } pub(super) use sql::{coherent_reads, query_lifecycle}; + +pub(super) use sql::apply_images; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index a5daeb35b..8414e0b75 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -328,7 +328,11 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { .context("selected root")?; let leaf = voter .tree - .get(&voter.pages, marker.root(), &cell.cell)? + .get( + voter.pages.as_ref().context("fixture page owner")?, + marker.root(), + &cell.cell, + )? .context("selected cell")?; let query = match leaf.version.image.commitment { CellImageCommitment::Materialized(digest) => { @@ -427,7 +431,14 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { .context("selected root")?; let leaf = fixture.cluster.voters[0] .tree - .get(&fixture.cluster.voters[0].pages, marker.root(), &a.cell)? + .get( + fixture.cluster.voters[0] + .pages + .as_ref() + .context("fixture page owner")?, + marker.root(), + &a.cell, + )? .context("selected cell")?; assert_eq!(leaf.version, stale.cell().version); drop(stale); @@ -474,3 +485,7 @@ pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { mod reads; pub(crate) use reads::{coherent as coherent_reads, lifecycle as query_lifecycle}; + +#[path = "cell_group_sql_apply.rs"] +mod apply; +pub(crate) use apply::apply_images; diff --git a/crates/cellar/tests/support/cell_group_sql_apply.rs b/crates/cellar/tests/support/cell_group_sql_apply.rs new file mode 100644 index 000000000..4dfe62a6a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply.rs @@ -0,0 +1,204 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The production image adapter, exercised with actual committed three-voter work. +use super::*; + +fn pool(fixture: &RuntimeCluster, node: usize) -> Result { + Ok(CellGroupSqlPool::start( + &fixture.cluster.voters[node].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?) +} + +pub(crate) async fn apply_images(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + for node in 0..3 { + let config = match fixture.cell_options_at(&a, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let pool = pool(&fixture, node)?; + let voter = &mut fixture.cluster.voters[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&route.history)?, + voter.pages.take().context("page owner")?, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture + .appliers + .insert((node, route.history), (pool, applier)); + } + let (pool, applier) = fixture.appliers[&(0, route.history)].clone(); + let foreign = fixture.appliers[&(1, route.history)].0.clone(); + + // Enroll through the production adapter on all voters, including scheduling + // summaries derived from real SQLite state rather than fixture defaults. + let cell = history(&route, "effects")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(7); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(80),(40);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let encoded = request(&route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + if rooted { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(&cell), + schema: route.schema_digest, + epoch: cut.result.epoch, + txid: cut.result.txid.0, + checksum: cut.result.checksum, + }, + Geometry::new(cut.result.page_size, cut.result.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + anyhow::bail!("enroll") + }; + image.commitment = ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len().try_into()?, + )?); + } + let accepted = fixture.runtimes[0] + .propose( + route.history, + entry.encode(&route, GroupEntryLimits::default())?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(state.cell().effects.next_alarm_unix_millis, Some(30)); + assert_eq!(state.cell().effects.next_outbox_unix_millis, Some(40)); + assert_eq!(state.cell().effects.pending_outbox, 2); + } + + let options = fixture.cell_options(&a).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Prepared(sql) = pool + .stage(options, authority, cell_attempt(231)?, true, |tx| { + tx.execute("UPDATE value SET n=n+11", [])?; + Ok(b"applied".to_vec()) + }) + .await? + else { + anyhow::bail!("SQL preparation") + }; + // Business admission is full. Its retained producer cannot prevent apply. + assert_eq!(pool.usage()?.active_cells, 1); + let other = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + assert!(matches!( + pool.stage( + fixture.cell_options(&b).await?, + other, + cell_attempt(232)?, + true, + |_| panic!("saturated business admission") + ) + .await, + Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded + )) + )); + pool.mark_submitted(a.clone(), sql.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &sql).await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let work = held.remove(&route.history).context("committed work")?; + let before = work.base.marker().clone(); + let mut forged = work.clone(); + forged.entry.data = vec![1, 2, 3].into(); + assert!(pool.prepare_apply(&applier, forged).await.is_err()); + assert!(foreign.prepare_apply(&applier, work.clone()).await.is_err()); + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert_eq!(foreign.usage()?.apply_retained_operations, 0); + + let prepared = pool.prepare_apply(&applier, work.clone()).await?; + assert_eq!(prepared.manifest().position(), accepted.position); + let duplicate = pool.prepare_apply(&applier, work.clone()).await?; + assert_eq!(prepared.manifest(), duplicate.manifest()); + assert_eq!(pool.usage()?.apply_retained_operations, 2); + drop(duplicate); + let retained = prepared.clone(); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert!(pool.usage()?.apply_retained_bytes > 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + + // Completion only queues publication. Retention outlives the caller and a + // blocked physical log sync; the old root remains selected until sync ends. + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + prepared.complete(&fixture.runtimes[0]).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + drop(retained); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + drop(release); + fixture.drive(Some(accepted.position.index)).await?; + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert!(pool.prepare_apply(&applier, work).await.is_err()); + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 2); + assert_eq!(state.cell().witness.position, accepted.position); + assert_eq!(state.cell().witness.proposal, sql.proposal()); + } + drop(sql); + drop(pool); + drop(foreign); + drop(applier); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_enrollment.rs b/crates/cellar/tests/support/cell_group_sql_enrollment.rs index d52063c56..0fb39050d 100644 --- a/crates/cellar/tests/support/cell_group_sql_enrollment.rs +++ b/crates/cellar/tests/support/cell_group_sql_enrollment.rs @@ -204,7 +204,11 @@ pub(super) async fn concurrent(rooted: bool) -> Result<()> { for cell in [&a, &b, &c] { let leaf = voter .tree - .get(&voter.pages, marker.root(), &cell.cell)? + .get( + voter.pages.as_ref().context("fixture page owner")?, + marker.root(), + &cell.cell, + )? .context("enrolled cell")?; assert_eq!(leaf.version.sequence.get(), 1); } diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index dea7fd58d..81768a622 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 15 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 17 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 851d6d5e9..54233f962 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -439,7 +439,9 @@ "scope": [ "crates/cellar/durability/raft/", "crates/cellar/application/consensus/", - "crates/cellar-logic/" + "crates/cellar-logic/", + "crates/cellar-ltx/src/generation.rs", + "crates/cellar-ltx/tests/private_clone.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/owner.rs", @@ -468,7 +470,13 @@ "crates/cellar/durability/raft/cell_groups/read_authority.rs", "crates/cellar/durability/raft/cell_groups/runtime/reads.rs", "crates/cellar/durability/raft/host_sql/cell_groups/query.rs", - "crates/cellar/tests/support/cell_group_sql_reads.rs" + "crates/cellar/tests/support/cell_group_sql_reads.rs", + "crates/cellar/durability/raft/cell_groups/apply/mod.rs", + "crates/cellar/durability/raft/cell_groups/apply/images.rs", + "crates/cellar/durability/raft/cell_groups/apply/summary.rs", + "crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs", + "crates/cellar/durability/raft/host_sql/cell_groups/apply.rs", + "crates/cellar/tests/support/cell_group_sql_apply.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -506,6 +514,13 @@ "cell_query_pins_deadline_and_cancelled_cleanup", "rooted_cell_query_pins_deadline_and_cancelled_cleanup" ] + }, + { + "action": "Run physical group apply on the existing node SQL workers with independent bounded progress admission. Transfer the startup manifest-page owner, verify the exact committed entry/base, prepare either cell image representation and derive bounded scheduling summaries. Retain preparation through response cancellation and physical publication. Materialized retries may pin only the exact verified prepared image; they cannot infer publication or reset retirement. Test the physical image store retry across restart and corruption, and reject malformed effect tables rather than reporting idle. Keep membership/topology administrative apply explicitly refused until its owning tasks implement it.", + "tests": [ + "shared_workers_prepare_committed_cell_images", + "rooted_shared_workers_prepare_committed_cell_images" + ] } ], "test_cases": [ @@ -639,6 +654,26 @@ "I05", "I12" ] + }, + { + "name": "shared_workers_prepare_committed_cell_images", + "oracle": "The existing fixed SQL workers prepare committed enrollment and mutation images on all three voters. Wrong engines and forged entries refuse; exact duplicate preparations yield the same manifest; stale bases refuse after publication. Business admission saturation does not prevent apply. Derived scheduling hints match actual SQLite state. Preparation leaves the old root selected, and its independent reservation survives clones and blocked physical publication until the owner completion is consumed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_workers_prepare_committed_cell_images", + "oracle": "The existing fixed SQL workers prepare committed enrollment and mutation images on all three voters. Wrong engines and forged entries refuse; exact duplicate preparations yield the same manifest; stale bases refuse after publication. Business admission saturation does not prevent apply. Derived scheduling hints match actual SQLite state. Preparation leaves the old root selected, and its independent reservation survives clones and blocked physical publication until the owner completion is consumed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From c9100ec774d66f30c14eae009f9a9c76d2bc0c33 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 00:45:11 -0700 Subject: [PATCH 071/568] test: bound shared apply future in recovery fixtures --- crates/cellar/tests/support/cell_group_runtime.rs | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index a8a98893b..b741090a4 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -84,10 +84,16 @@ impl RuntimeCluster { assert!(held.insert(scope, work).is_none()); } else { if let Some((pool, applier)) = self.appliers.get(&(n, scope)) { - pool.prepare_apply(applier, work) - .await? - .complete(&self.runtimes[n]) - .await?; + // Keep the physical apply future out of every + // caller's inline transport/recovery state. + Box::pin(async { + pool.prepare_apply(applier, work) + .await? + .complete(&self.runtimes[n]) + .await?; + Ok::<_, anyhow::Error>(()) + }) + .await?; continue; } let voter = if scope == primary { From d65a3e3ccb66e5d05b8e3aa08894fe4643933352 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 00:53:23 -0700 Subject: [PATCH 072/568] docs: qualify shared apply and prioritize remaining CG gaps --- docs/design/multi-raft/cell-groups/README.md | 6 ++ .../multi-raft/cell-groups/baseline.json | 8 +-- docs/design/multi-raft/cell-groups/handoff.md | 66 ++++++++++++++----- .../design/multi-raft/cell-groups/progress.md | 48 ++++++++++++++ 4 files changed, 109 insertions(+), 19 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 9a0ce3658..d24a07e34 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -109,6 +109,12 @@ and all twelve verification commands pass. Durable cell-version witnesses and query/receipt observations are implemented and qualified at `ed84df5d`: 214 selected Rust cases and thirteen verification commands pass. See the handoff for the exact scope and retained evidence. +Committed image preparation now runs through the shared SQL workers at `2085cc96`, +with its fixture repair and full partial gate at `c9100ec7`: 254 selected Rust +cases and fifteen commands pass. Transfer both +`.cellar/cell-groups-evidence/CG-04-apply-final/` and the retained failed +`.cellar/cell-groups-evidence/CG-04-apply/` directory with the prior evidence. +Direct cancellation/crash cuts during image preparation remain to be qualified. Registered SDK/catalog routing, query/version-token integration, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 21a9ca8ff..e3eef7579 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2", + "source_revision": "c9100ec774d66f30c14eae009f9a9c76d2bc0c33", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -43,8 +43,8 @@ }, "sql": { "path": "crates/cellar/durability/raft/host_sql/mod.rs", - "sha256": "55484a37d1d4c00be0a1a83f27ec787cb3ce05aee53a4b9c419ccc044b865456", - "line": 864, + "sha256": "bb822ccd381c8d5532641047aeb61f4b04872fabbcaa2611dcc891cd71473640", + "line": 871, "excerpt": "struct Cell {\n group: GroupLog,\n class: Arc,\n images: ImageOwner,\n snapshot: Option<(ApplyTicket, PreparedSnapshot, Permit)>,\n private: Option,\n active: Option,\n}\n" }, "runtime": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index abdc038c1..5daf376fb 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,8 +15,10 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `ed84df5d`, retaining exact cell-version witnesses -in authenticated manifests and query/receipt observations. The preceding read +The reviewed source baseline is `c9100ec7`, qualifying the committed image +preparation introduced at `2085cc96` on the existing SQL workers. The successor +bounds the shared fixture's inline async state without increasing test stacks. The preceding `ed84df5d` checkpoint retains exact +cell-version witnesses in authenticated manifests and query/receipt observations. The preceding read checkpoint at `3da4481e` decouples pin capacity from transport limits and repairs the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing and exact rejection cleanup (`0cf89c40`) and bounded proposal acceptance and @@ -188,13 +190,12 @@ tree, preflight deliberately refuses source drift in validator tests pass and the actual-source preflight test errors. Do not refresh the baseline merely to silence this refusal. -The uncommitted draft adds `cell_groups/apply/` and -`host_sql/cell_groups/apply.rs`, with changes to the manifest, owner and worker -modules. It prepares cell images on the existing SQL workers and retains apply -admission. Preserve and review this draft before continuing CG-04. Its library -check completed, but it has no source-bound test or qualification checkpoint. -The untracked `diagram/` directory is separate existing work. Neither the draft -nor those diagrams forms part of this documentation-only delivery check. +At that design-delivery check, `cell_groups/apply/` and +`host_sql/cell_groups/apply.rs` were uncommitted drafts. Subsequent execution +completed that adapter at `2085cc96` and repaired the regression fixture at +`c9100ec7`; see the current checkpoint in [progress](progress.md). The untracked +`diagram/` directory remains separate existing work and is excluded from this +implementation checkpoint. ### Resume implementation @@ -229,12 +230,14 @@ modes. Actual log-compaction and group-snapshot integration remain unqualified. Next integrate registered callbacks and catalog invocation sessions without synthesizing legacy group identities. Preserve original invocation deadlines, contracts and retry identities. Query callbacks must receive logical cell versions; -returned tokens must use the selected observation. The remaining production apply -adapter must consume `CellGroupEffect::Apply` on owning execution-domain workers, -prepare cell-addressed LTX images and complete manifests, and return exact apply -completions. Current native fixtures supply that preparation; do not put blocking -image work on the consensus loop or mistake fixture integration for host serving. -Keep physical apply position separate from provenance in later group transfers. +returned tokens must use the selected observation. `CellGroupApplier` now consumes the startup manifest-page owner, and +`CellGroupSqlPool::prepare_apply` prepares committed cell images and complete +manifests on the existing workers. Consume its retained output with +`CellGroupPreparedApply::complete`; that queues owner publication and is not a +commit acknowledgement. The new native fixture uses this adapter on all three +voters. Registered host dispatch must now consume these APIs instead of copying +fixture image preparation. Keep physical apply position separate from provenance +in later group transfers. Run from the repository root with Python 3.11+ and Rust 1.98.1: @@ -260,6 +263,15 @@ another machine. Missing evidence must refuse; do not replace it with a pass fla or reset completed tasks merely to bypass validation. The latest fully passing partial CG-04 record is +`.cellar/cell-groups-evidence/CG-04-apply-final/checkpoint.json` at `c9100ec7`: +254 selected Rust cases, fifteen successful commands, 202 source inputs and +12 copied executables. Transfer the entire directory and its failed predecessor +`CG-04-apply` with the prior evidence. It qualifies committed image preparation +on the existing SQL workers and retention through held publication, not direct +image-I/O cancellation/crash cuts or registered host serving. See the priorities +below before continuing CG-04. + +The preceding durable-witness record is `.cellar/cell-groups-evidence/CG-04-witness/checkpoint.json` at `ed84df5d`: 214 selected Rust cases, thirteen successful commands, 194 source inputs and nine copied executables. Transfer the full directory with prior evidence. It qualifies @@ -306,6 +318,30 @@ binds 178 source inputs, eight executables, 184 selected Rust cases and ten verification commands. It cannot stand in for the complete CG-04 contract. Three RustFS-specific cases remain explicitly unrun at this checkpoint. +## Next execution priorities + +Deliver CG tasks in dependency order and keep the status at its actual qualified +boundary. At the end of each turn, record the completed behavior, evidence and +limits, then identify the most critical remaining gap and the concrete next step. +Do not let plan maintenance replace implementation. + +1. Complete the new apply adapter's failure qualification: cancel its receiver + and close the pool while image or manifest I/O is held, then prove that physical + work and reservations remain owned until the workers join. Inject preparation + and restart failures; publication-sync retention alone does not cover these cuts. +2. Integrate catalog invocation sessions and registered command/query callbacks + with logical cell identity and version observations. Preserve request deadlines, + receipt identity and uncertainty across redirects; never fabricate legacy + `GroupScope` values. This is the most important remaining serving gap. +3. Complete CG-04's production retry/cache lifecycle and configuration-entry + handling at the appropriate owner boundary, then run its entire gate before + marking it DONE. Full snapshot/membership-change qualification follows in CG-05; + archive/reclamation in CG-06 must preserve complete group recovery closure. + +The opportunity is to reuse the now-shared consensus, SQL, read and apply paths in +one dispatcher. Additional per-cell workers, a second production mode or a demo +that bypasses the registered API would increase integration work. + ## Delivery sequence | Milestone | Tasks | Required outcome | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 60a4646b1..56084c7c0 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,54 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 committed image preparation on shared workers + +Qualified source: `c9100ec774d66f30c14eae009f9a9c76d2bc0c33`. Runtime implementation: `2085cc96`. + +`CellGroupApplier` owns the same manifest-page store used to verify startup. It +checks the exact persisted committed entry, engine, group and preceding root +before preparing cell-addressed images. Enrollment and mutation work support +materialized and extent storage, derive bounded alarm/outbox hints from the actual +SQLite image, preserve logical versions and exact commit witnesses, and refuse +malformed scheduling state. Membership/topology administration remains refused +pending its owner integration. + +`CellGroupSqlPool::prepare_apply` uses the existing fixed SQL workers and their +reserved progress lane. Apply has a separate byte/item budget, so retained SQL +producers cannot consume its admission. `CellSqlLimits::node_bytes` bounds each +of the producer and apply budgets independently; account for their combined +ceiling. Usage exposes both budgets separately. The returned preparation retains +its reservation across clones, delivery and owner completion. Completion queues +publication; only durable owner apply establishes the new selected root. + +Materialized duplicate preparation now reopens the exact verified image through +`GenerationStore::pin_prepared`. It does not reset retirement or infer publication. +Cold images require independent byte verification and persistence barriers. +Regression tests cover restart, corruption, mismatched expected state, reader pins +and reclamation. Both real three-voter cases cover enrollment, mutation, derived +hints, forged/foreign/stale work, duplicate preparation, saturated command admission, +old-root selection before publication and retention during a held log sync. + +All 254 selected Rust cases pass: 227 native cases across ten suites, 23 +Cellar library cases and four LTX image-store cases. All-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,812 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases pass. The partial checkpoint at +`.cellar/cell-groups-evidence/CG-04-apply-final/checkpoint.json` binds 202 source +inputs, 12 copied executables and fifteen successful commands. The failed gate +at `CG-04-apply` is retained: the newly inlined apply branch overflowed an existing +recovery fixture's stack. The branch is now heap allocated; normal stack limits +and assertions remain unchanged. Earlier fixture, materialized retry and Clippy +failures are retained too. Six process helpers run through parent tests; three +RustFS-specific cases remain unrun. + +CG-04 remains IN_PROGRESS. The most critical next validation is cancellation, +shutdown and crash injection while this new image/manifest preparation is held; +the held publication sync does not qualify those earlier cuts. Then integrate the +registered SDK/catalog dispatcher with logical cell identities and observations. +Production retry/cache lifecycle and configuration-entry integration remain open. +No complete shared-group snapshot, compaction, RustFS or performance claim is made. + ## CG-04 durable selected-version commit witnesses Qualified source: `ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2`. From 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 01:20:00 -0700 Subject: [PATCH 073/568] feat(raft): recover selected cell-group images and qualify apply faults --- .../raft/cell_groups/apply/images.rs | 28 +- .../durability/raft/cell_groups/apply/mod.rs | 1 + .../raft/cell_groups/apply/recovery.rs | 135 +++++++++ .../tests/native_cell_groups_execution.rs | 53 +++- .../tests/support/cell_group_runtime.rs | 20 +- crates/cellar/tests/support/cell_group_sql.rs | 24 +- .../tests/support/cell_group_sql_apply.rs | 111 ++++--- .../support/cell_group_sql_apply_faults.rs | 183 ++++++++++++ .../support/cell_group_sql_apply_recovery.rs | 279 ++++++++++++++++++ .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 76 ++++- 11 files changed, 843 insertions(+), 69 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/apply/recovery.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_apply_faults.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_apply_recovery.rs diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs index 4676c88a4..aa235d5bb 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/images.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -131,15 +131,6 @@ impl CellGroupApplyImages { Some(CellImagePin::Empty) | None => None, _ => return Err(ExecutionError::Invalid("materialized apply base")), }; - let mut id = Sha256::new(); - id.update(b"cellar-cell-apply-image-1\0"); - id.update(cell_image_history_digest(cell)); - for value in [sequence, image.epoch, image.txid, image.checksum] { - id.update(value.to_be_bytes()); - } - id.update(digest); - let bytes: [u8; 32] = id.finalize().into(); - let id = ImageId::new(bytes[..16].try_into().map_err(storage)?).map_err(storage)?; let update = ImageUpdate { base: match base { Some(pin) => pin.state(), @@ -155,6 +146,7 @@ impl CellGroupApplyImages { }, segments, }; + let id = image_id(cell, sequence, update.result)?; CellImagePin::Materialized(match images.prepare(id, base, &update) { Ok(pin) => pin, Err(cellar_ltx::generation::GenerationError::Exists) => { @@ -258,3 +250,21 @@ impl CellGroupApplyImages { }) } } + +// Names are physical lookup hints. The selected manifest and exact image state +// still have to verify; a filename never supplies publication authority. +pub(super) fn image_id( + cell: &CellHistory, + sequence: u64, + image: ImageState, +) -> Result { + let mut id = Sha256::new(); + id.update(b"cellar-cell-apply-image-1\0"); + id.update(cell_image_history_digest(cell)); + for value in [sequence, image.epoch, image.txid.0, image.checksum] { + id.update(value.to_be_bytes()); + } + id.update(image.digest); + let bytes: [u8; 32] = id.finalize().into(); + ImageId::new(bytes[..16].try_into().map_err(storage)?).map_err(storage) +} diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 269437808..7ef0bde98 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -3,6 +3,7 @@ //! Worker-owned physical preparation; only the Raft owner can publish its result. mod images; +mod recovery; mod summary; pub use images::CellGroupApplyImages; diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs new file mode 100644 index 000000000..6eb91882a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -0,0 +1,135 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recover only the complete root selected by the persisted engine marker. +use super::*; +use cellar_logic::cell_group::CellImageCommitment; +use cellar_ltx::{ + apply::ImageState, + internal::{ExtentRootReference, ImageBinding, ImageRoot, EXTENT_BYTES}, + TXID, +}; + +impl CellGroupApplier { + /// Verify the persisted selected root and all physical cell images. Run on + /// the owning blocking execution domain, before starting its Raft owner. + /// This grants no quorum or serving authority: startup still consumes the + /// catalog permit and must establish current-term quorum before admission. + /// Missing/corrupt closure refuses; unselected newer files are not recovery. + pub fn recover( + group: GroupLog, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result<(Self, PreparedGroupManifest), ExecutionError> { + let applier = Self::new( + group, + pages, + images, + manifest_limits, + limits, + max_effect_rows, + )?; + let marker = applier + .group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("recovery manifest missing"))?; + let prepared = { + let mut owner = applier + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + let Storage { + tree, + pages, + images, + .. + } = &mut *owner; + tree.prepare_complete( + &applier.group, + pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + images + .recover_image(leaf, marker.route().schema_digest) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }, + ) + .map_err(storage)? + }; + if applier.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) { + return Err(ExecutionError::Invalid("recovery selection changed")); + } + Ok((applier, prepared)) + } +} + +impl CellGroupApplyImages { + fn recover_image( + &self, + leaf: &CellManifestLeaf, + schema: [u8; 32], + ) -> Result { + let image = leaf.version.image; + let pin = if image.txid == 0 { + CellImagePin::Empty + } else { + match (self, image.commitment) { + (Self::Materialized { images, .. }, CellImageCommitment::Materialized(digest)) => { + let expected = ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: leaf.geometry.page_size, + length: leaf.geometry.length, + }; + let id = images::image_id( + &leaf.version.history, + leaf.version.sequence.get(), + expected, + )?; + CellImagePin::Materialized(images.pin_prepared(id, expected).map_err(storage)?) + } + ( + Self::Extents { images, .. }, + CellImageCommitment::Extents { + root, + manifest_bytes, + }, + ) => { + let reference = + ExtentRootReference::new(ImageRoot::from_bytes(root), manifest_bytes) + .map_err(storage)?; + let view = images.open_root(reference).map_err(storage)?; + let manifest = view.manifest(); + let expected = ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + schema, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + if manifest.binding() != expected + || manifest.geometry().length() != leaf.geometry.length + || manifest.geometry().page_size() != leaf.geometry.page_size + { + return Err(ExecutionError::Invalid("recovery image binding")); + } + CellImagePin::Extents(Arc::new( + images + .seal_image_root(manifest.clone(), &mut vec![0; EXTENT_BYTES as usize]) + .map_err(storage)?, + )) + } + _ => return Err(ExecutionError::Invalid("recovery image representation")), + } + }; + pin.check(leaf, schema).map_err(storage)?; + Ok(pin) + } +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 5fb850bb1..5f34c151b 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -178,7 +178,19 @@ impl Cluster { } async fn with_options(options: RaftRuntimeOptions) -> Result { - let root = tempfile::tempdir()?; + Box::pin(Self::with_storage( + options, + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + )) + .await + } + + async fn with_storage( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { cluster: ClusterId::new([1; 16])?, @@ -243,7 +255,7 @@ impl Cluster { let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; let mut pages = ManifestPageStore::open( &base.join("pages"), - Arc::new(DirectFileSystem), + filesystem.clone(), 1024, 16 * 1024 * 1024, 16, @@ -265,11 +277,8 @@ impl Cluster { [n; 32], )) .await??; - let images = GenerationStore::open( - base.join("images"), - Arc::new(DirectFileSystem), - Default::default(), - )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; voters.push(Voter { log, owner: Some(owner), @@ -1249,3 +1258,33 @@ async fn shared_workers_prepare_committed_cell_images() -> Result<()> { async fn rooted_shared_workers_prepare_committed_cell_images() -> Result<()> { runtime_fixture::apply_images(true).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_cancelled_close_joins_preparation() -> Result<()> { + runtime_fixture::apply_cancellation(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_cancelled_close_joins_preparation() -> Result<()> { + runtime_fixture::apply_cancellation(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_sync_failure_preserves_selected_root() -> Result<()> { + runtime_fixture::apply_failure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_sync_failure_preserves_selected_root() -> Result<()> { + runtime_fixture::apply_failure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_apply_recovers_selected_images_before_replay() -> Result<()> { + runtime_fixture::recover_apply(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_shared_apply_recovers_selected_images_before_replay() -> Result<()> { + runtime_fixture::recover_apply(true).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index b741090a4..d9f98a629 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -25,7 +25,21 @@ impl RuntimeCluster { max_requests: usize, max_cells_per_group: u32, ) -> Result { - let mut cluster = Cluster::new().await?; + Box::pin(Self::from_cluster( + Cluster::new().await?, + max_events, + max_requests, + max_cells_per_group, + )) + .await + } + + async fn from_cluster( + mut cluster: Cluster, + max_events: usize, + max_requests: usize, + max_cells_per_group: u32, + ) -> Result { let mut runtimes = Vec::new(); let mut events = Vec::new(); let mut applied = Vec::new(); @@ -647,3 +661,7 @@ pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { pub(super) use sql::{coherent_reads, query_lifecycle}; pub(super) use sql::apply_images; + +pub(super) use sql::{apply_cancellation, apply_failure}; + +pub(super) use sql::recover_apply; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 8414e0b75..975750de1 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -151,13 +151,25 @@ impl RuntimeCluster { async fn enrolled_sql_cells( rooted: bool, ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { - let mut fixture = RuntimeCluster::new().await?; + Box::pin(enroll_sql_cells( + RuntimeCluster::new().await?, + rooted, + Arc::new(DirectFileSystem), + )) + .await +} + +async fn enroll_sql_cells( + mut fixture: RuntimeCluster, + rooted: bool, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let route = fixture.cluster.birth.route().clone(); if rooted { for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { voter.extents = Some(NodeExtentStore::open( fixture.cluster._root.path().join(format!("rooted-{n}")), - Arc::new(DirectFileSystem), + filesystem.clone(), Default::default(), )?); } @@ -489,3 +501,11 @@ pub(crate) use reads::{coherent as coherent_reads, lifecycle as query_lifecycle} #[path = "cell_group_sql_apply.rs"] mod apply; pub(crate) use apply::apply_images; + +#[path = "cell_group_sql_apply_faults.rs"] +mod apply_faults; +pub(crate) use apply_faults::{cancellation as apply_cancellation, failure as apply_failure}; + +#[path = "cell_group_sql_apply_recovery.rs"] +mod apply_recovery; +pub(crate) use apply_recovery::recover as recover_apply; diff --git a/crates/cellar/tests/support/cell_group_sql_apply.rs b/crates/cellar/tests/support/cell_group_sql_apply.rs index 4dfe62a6a..d0aba067e 100644 --- a/crates/cellar/tests/support/cell_group_sql_apply.rs +++ b/crates/cellar/tests/support/cell_group_sql_apply.rs @@ -3,7 +3,7 @@ //! The production image adapter, exercised with actual committed three-voter work. use super::*; -fn pool(fixture: &RuntimeCluster, node: usize) -> Result { +pub(super) fn pool(fixture: &RuntimeCluster, node: usize) -> Result { Ok(CellGroupSqlPool::start( &fixture.cluster.voters[node].log, CellSqlLimits { @@ -17,51 +17,55 @@ fn pool(fixture: &RuntimeCluster, node: usize) -> Result { )?) } -pub(crate) async fn apply_images(rooted: bool) -> Result<()> { - let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; - for node in 0..3 { - let config = match fixture.cell_options_at(&a, node).await? { - CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { - images: o.images, - services: o.query_services, - }, - CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { - images: o.images, - services: o.query_services, - limits: Default::default(), - cache_directory: o.cache_directory, - cache_pool: o.cache_pool, - cache: o.cache, - }, - }; - let pool = pool(&fixture, node)?; - let voter = &mut fixture.cluster.voters[node]; - let applier = CellGroupApplier::new( - voter.log.cell_group(&route.history)?, - voter.pages.take().context("page owner")?, - config, - ManifestLimits::default(), - GroupEntryLimits::default(), - 100, - )?; - fixture - .appliers - .insert((node, route.history), (pool, applier)); - } - let (pool, applier) = fixture.appliers[&(0, route.history)].clone(); - let foreign = fixture.appliers[&(1, route.history)].0.clone(); +pub(super) async fn install( + fixture: &mut RuntimeCluster, + cell: &CellHistory, + node: usize, +) -> Result<()> { + let route = fixture.cluster.birth.route().clone(); + let config = match fixture.cell_options_at(cell, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let pool = pool(fixture, node)?; + let voter = &mut fixture.cluster.voters[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&route.history)?, + voter.pages.take().context("page owner")?, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture + .appliers + .insert((node, route.history), (pool, applier)); + Ok(()) +} - // Enroll through the production adapter on all voters, including scheduling - // summaries derived from real SQLite state rather than fixture defaults. - let cell = history(&route, "effects")?; - let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(7); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(80),(40);")?; - let cut = images::Cut::new(&[], &bytes, 0, true, false); - let encoded = request(&route, cell.clone(), 1, &cut)?; - let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; +pub(super) fn enrollment_bytes( + route: &GroupRoute, + cell: &CellHistory, + bytes: &[u8], + rooted: bool, +) -> Result> { + let cut = images::Cut::new(&[], bytes, 0, true, false); + let encoded = request(route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, route, GroupEntryLimits::default())?; if rooted { let manifest = ExtentManifest::new( ImageBinding { - history: cell_image_history_digest(&cell), + history: cell_image_history_digest(cell), schema: route.schema_digest, epoch: cut.result.epoch, txid: cut.result.txid.0, @@ -81,12 +85,23 @@ pub(crate) async fn apply_images(rooted: bool) -> Result<()> { manifest.encode()?.len().try_into()?, )?); } - let accepted = fixture.runtimes[0] - .propose( - route.history, - entry.encode(&route, GroupEntryLimits::default())?, - ) - .await?; + Ok(entry.encode(route, GroupEntryLimits::default())?) +} + +pub(crate) async fn apply_images(rooted: bool) -> Result<()> { + let (mut fixture, route, a, b) = enrolled_sql_cells(rooted).await?; + for node in 0..3 { + install(&mut fixture, &a, node).await?; + } + let (pool, applier) = fixture.appliers[&(0, route.history)].clone(); + let foreign = fixture.appliers[&(1, route.history)].0.clone(); + + // Enroll through the production adapter on all voters, including scheduling + // summaries derived from real SQLite state rather than fixture defaults. + let cell = history(&route, "effects")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(7); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(80),(40);")?; + let encoded = enrollment_bytes(&route, &cell, &bytes, rooted)?; + let accepted = fixture.runtimes[0].propose(route.history, encoded).await?; fixture.drive(Some(accepted.position.index)).await?; for runtime in &fixture.runtimes { let state = runtime diff --git a/crates/cellar/tests/support/cell_group_sql_apply_faults.rs b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs new file mode 100644 index 000000000..1d9a29afc --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs @@ -0,0 +1,183 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical preparation ownership and refusal at real image/manifest barriers. +use super::*; +use std::sync::atomic::Ordering; + +#[path = "raft_ltx_fault.rs"] +#[allow(dead_code)] +mod physical; + +async fn pending( + rooted: bool, + manifest: bool, +) -> Result<( + RuntimeCluster, + Arc, + CellGroupApply, + CellHistory, +)> { + let root = tempfile::tempdir()?; + let directory = if manifest { + root.path().join("node-1/pages") + } else if rooted { + root.path().join("rooted-0") + } else { + root.path().join("node-1/images") + }; + let faults = physical::Faults::new(directory, None); + let filesystem: Arc = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let cluster = + Cluster::with_storage(RaftRuntimeOptions::default(), root, filesystem.clone()).await?; + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000).await?; + let (mut fixture, route, a, _) = enroll_sql_cells(fixture, rooted, filesystem).await?; + apply::install(&mut fixture, &a, 0).await?; + let cell = history(&route, "faulted-apply")?; + // Raw replicated enrollment ensures no leader SQL preview has already + // prepared this result, including in the content-addressed extent store. + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(11);")?; + fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + Ok(( + fixture, + faults, + held.remove(&route.history).context("committed work")?, + cell, + )) +} + +pub(crate) async fn cancellation(rooted: bool) -> Result<()> { + for manifest in [false, true] { + let (mut fixture, faults, work, cell) = Box::pin(pending(rooted, manifest)).await?; + let history = fixture.cluster.birth.route().history; + let before = work.base.marker().clone(); + let position = work.ticket.position(); + let (pool, applier) = fixture.appliers.remove(&(0, history)).context("applier")?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10)) + .expect("release physical apply barrier"); + })); + let caller = pool.clone(); + let worker = applier.clone(); + let input = work.clone(); + let waiter = fixture + .cluster + .domain + .spawn(async move { caller.prepare_apply(&worker, input).await })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert!(pool.usage()?.apply_retained_bytes > 0); + assert_eq!(pool.usage()?.completion_reservations, 1); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + // Cancel a close waiter too. Cleanup still owns the entered worker and + // the exclusive SQL-pool lease until the physical barrier is released. + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + assert_eq!(pool.usage()?.running_workers, 1); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.apply_retained_operations, 0); + assert_eq!(usage.apply_retained_bytes, 0); + assert_eq!(usage.completion_reservations, 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + drop(pool); + // Shutdown did not publish a result or settle this committed entry. + // A fresh pool can repeat exact preparation, then the owner publishes it. + let replacement = apply::pool(&fixture, 0)?; + replacement + .prepare_apply(&applier, work) + .await? + .complete(&fixture.runtimes[0]) + .await?; + fixture.drive(Some(position.index)).await?; + for runtime in &fixture.runtimes { + let state = runtime + .reconciliation_authority(history, cell.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 1); + assert_eq!(state.cell().witness.position, position); + } + replacement.close().await?; + drop(replacement); + drop(applier); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn failure(rooted: bool) -> Result<()> { + for manifest in [false, true] { + let (fixture, faults, work, _) = Box::pin(pending(rooted, manifest)).await?; + let history = fixture.cluster.birth.route().history; + let before = work.base.marker().clone(); + let (pool, applier) = &fixture.appliers[&(0, history)]; + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + assert!(pool.prepare_apply(applier, work).await.is_err()); + assert_eq!( + faults.mode.load(Ordering::Acquire), + 0, + "fault must reach actual preparation I/O" + ); + assert_eq!(pool.usage()?.apply_retained_operations, 0); + assert_eq!(pool.usage()?.apply_retained_bytes, 0); + assert_eq!(pool.usage()?.completion_reservations, 0); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + let status = fixture.runtimes[0].status(history).await?; + assert_eq!(status.applied, before.position()); + assert!(status.committed > before.position().index); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs new file mode 100644 index 000000000..46b57de8a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs @@ -0,0 +1,279 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Cold selected-root verification and replay of committed, unpublished work. +use super::*; + +pub(crate) async fn recover(rooted: bool) -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let a = history(&route, "recover-a")?; + let b = history(&route, "recover-b")?; + apply::install(&mut fixture, &a, 0).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(29);")?; + let accepted_a = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &a, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted_a.position.index)).await?; + let selected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected A")?; + let accepted_b = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &b, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(5), async { + loop { + fixture.pump(&[route.history], &mut held).await?; + if held.contains_key(&route.history) { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await??; + let work = held.remove(&route.history).context("unpublished B")?; + let (pool, applier) = &fixture.appliers[&(0, route.history)]; + let prepared = pool.prepare_apply(applier, work).await?; + assert_eq!(prepared.manifest().position(), accepted_b.position); + drop(prepared); + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&selected) + ); + let directory = fixture.cluster.directory.clone(); + let birth = fixture.cluster.birth.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + fixture.close().await?; + + // All prior engine, page, image, worker and runtime owners are gone. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-1"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([31; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let group = log.cell_group(&route.history)?; + let image_directory = if rooted { + disk.path().join("rooted-0") + } else { + base.join("images") + }; + let config = open_images(&image_directory, rooted)?; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + assert_eq!(recovered.pinned_images(), 1); + assert_eq!( + recovered.pin_cell(&a.cell).context("A")?.0.witness.position, + accepted_a.position + ); + assert!( + recovered.pin_cell(&b.cell).is_none(), + "unselected files became authoritative" + ); + let (runtime, mut events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered).await?; + let pool = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + // Replay durable committed B with only this node present. It is apply, not + // renewed serving authority; messages are deliberately not delivered. + tokio::time::timeout(Duration::from_secs(5), async { + loop { + let event = events.recv().await.context("runtime event")?; + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("replay")?; + pool.prepare_apply(&applier, work) + .await? + .complete(&runtime) + .await?; + } + CellGroupEffect::Applied { manifest, .. } + if manifest.position() == accepted_b.position => + { + break + } + CellGroupEffect::Message(_) | CellGroupEffect::Applied { .. } => (), + _ => anyhow::bail!("unexpected recovery event"), + } + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert_eq!( + group.group_manifest()?.context("B marker")?.root().cells(), + 2 + ); + assert!(!runtime.status(route.history).await?.ready); + assert!(runtime + .execution_authority(route.history, b.clone()) + .await + .is_err()); + pool.close().await?; + runtime.close().await?; + drop(events); + drop(runtime); + drop(pool); + drop(applier); + let selected = group.group_manifest()?.context("replayed root")?; + // The open engine cache pins its published page/image closure too. A real + // restart closes that owner before reopening the same directories. + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + + // Missing physical closure must fail even though the engine marker remains + // intact. Only this test's disposable directory is moved. + std::fs::rename(&image_directory, image_directory.with_extension("saved"))?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([32; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let group = log.cell_group(&route.history)?; + let config = open_images(&image_directory, rooted)?; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let recovering = group.clone(); + assert!(domain + .blocking(move || CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100 + ))? + .await? + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&selected)); + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + drop(disk); + Ok(()) +} + +fn open_images(directory: &std::path::Path, rooted: bool) -> Result { + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + Ok(if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + directory.into(), + Arc::new(DirectFileSystem), + Default::default(), + )?, + limits: Default::default(), + services, + cache_directory: directory.with_extension("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open( + directory.into(), + Arc::new(DirectFileSystem), + Default::default(), + )?, + services, + } + }) +} diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 81768a622..c068865a0 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 17 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 23 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 54233f962..845555e9a 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -476,7 +476,10 @@ "crates/cellar/durability/raft/cell_groups/apply/summary.rs", "crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs", "crates/cellar/durability/raft/host_sql/cell_groups/apply.rs", - "crates/cellar/tests/support/cell_group_sql_apply.rs" + "crates/cellar/tests/support/cell_group_sql_apply.rs", + "crates/cellar/durability/raft/cell_groups/apply/recovery.rs", + "crates/cellar/tests/support/cell_group_sql_apply_faults.rs", + "crates/cellar/tests/support/cell_group_sql_apply_recovery.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -521,6 +524,17 @@ "shared_workers_prepare_committed_cell_images", "rooted_shared_workers_prepare_committed_cell_images" ] + }, + { + "action": "Qualify image and manifest I/O ownership at actual held/failing directory barriers. Recover complete cell-group images through the same stable physical naming and verified extent bindings used by apply; accept only the engine-selected root and retain its exact cell versions/witnesses. Cold restart may replay a committed unpublished entry but never substitutes selected files or inherited commit state for fresh serving authority. Missing closure refuses before owner admission. Keep whole-process kill/power-loss qualification distinct from joined cold restart.", + "tests": [ + "shared_apply_cancelled_close_joins_preparation", + "rooted_shared_apply_cancelled_close_joins_preparation", + "shared_apply_sync_failure_preserves_selected_root", + "rooted_shared_apply_sync_failure_preserves_selected_root", + "shared_apply_recovers_selected_images_before_replay", + "rooted_shared_apply_recovers_selected_images_before_replay" + ] } ], "test_cases": [ @@ -674,6 +688,66 @@ "I03", "I12" ] + }, + { + "name": "shared_apply_cancelled_close_joins_preparation", + "oracle": "Hold actual image and manifest directory barriers separately. Cancel the apply receiver and a close waiter; entered work, bytes, progress slot and exclusive SQL lease remain retained until release. Joined close refunds the reservations without publishing; a fresh pool repeats exact preparation and the real Raft owner publishes it.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_cancelled_close_joins_preparation", + "oracle": "Hold actual image and manifest directory barriers separately. Cancel the apply receiver and a close waiter; entered work, bytes, progress slot and exclusive SQL lease remain retained until release. Joined close refunds the reservations without publishing; a fresh pool repeats exact preparation and the real Raft owner publishes it.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_apply_sync_failure_preserves_selected_root", + "oracle": "Inject actual image and manifest directory-sync failures separately. Preparation fails, refunds its admission, and never advances the selected root or applied index despite the entry being committed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_sync_failure_preserves_selected_root", + "oracle": "Inject actual image and manifest directory-sync failures separately. Preparation fails, refunds its admission, and never advances the selected root or applied index despite the entry being committed.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "shared_apply_recovers_selected_images_before_replay", + "oracle": "Close all previous runtime, engine, manifest and image owners. Recover the engine-selected manifest and physical image closure in a new process generation, ignoring an already prepared but unpublished cell. Replay that committed entry without inventing fresh quorum/serving authority. After another cold restart, missing selected image files must refuse recovery without changing the persisted marker.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_shared_apply_recovers_selected_images_before_replay", + "oracle": "Close all previous runtime, engine, manifest and image owners. Recover the engine-selected manifest and physical image closure in a new process generation, ignoring an already prepared but unpublished cell. Replay that committed entry without inventing fresh quorum/serving authority. After another cold restart, missing selected image files must refuse recovery without changing the persisted marker.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From f9593e9f18b0554837a93c9ae7dacbe08818712a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 01:27:33 -0700 Subject: [PATCH 074/568] docs(raft): record apply recovery qualification and next CG priorities --- docs/design/multi-raft/cell-groups/README.md | 9 +++- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 39 +++++++++------ .../design/multi-raft/cell-groups/progress.md | 48 +++++++++++++++++++ 4 files changed, 82 insertions(+), 18 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index d24a07e34..6480071a5 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -114,7 +114,14 @@ with its fixture repair and full partial gate at `c9100ec7`: 254 selected Rust cases and fifteen commands pass. Transfer both `.cellar/cell-groups-evidence/CG-04-apply-final/` and the retained failed `.cellar/cell-groups-evidence/CG-04-apply/` directory with the prior evidence. -Direct cancellation/crash cuts during image preparation remain to be qualified. +Selected-image recovery and physical apply fault coverage are implemented at +`1ec2eeaa`. Both image modes cover cancelled preparation/close while real image +or manifest directory sync is held, failed-sync old-root preservation, and cold +selected-root recovery with committed replay and no renewed serving authority. +All 260 selected Rust cases and fifteen commands pass; transfer the retained +`.cellar/cell-groups-evidence/CG-04-apply-recovery/` directory with prior evidence. +Whole-process kill and power-loss cuts remain unqualified. See the newest +[progress checkpoint](progress.md) for the committed-source gate and evidence. Registered SDK/catalog routing, query/version-token integration, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index e3eef7579..acda4c20c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "c9100ec774d66f30c14eae009f9a9c76d2bc0c33", + "source_revision": "1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 5daf376fb..db4b87442 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,9 +15,13 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `c9100ec7`, qualifying the committed image -preparation introduced at `2085cc96` on the existing SQL workers. The successor -bounds the shared fixture's inline async state without increasing test stacks. The preceding `ed84df5d` checkpoint retains exact +The reviewed source baseline is `1ec2eeaa`, adding selected-image recovery and +physical apply fault coverage in both image modes. Its retained checkpoint is +`.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`: 260 selected +Rust cases, fifteen commands, 205 source inputs and twelve executables. Transfer +that entire directory, including failed intermediate output, with prior evidence. +This qualifies joined cold restart, not whole-process kill or power loss. The preceding `c9100ec7` +checkpoint qualifies committed preparation on shared workers (`2085cc96`). The preceding `ed84df5d` checkpoint retains exact cell-version witnesses in authenticated manifests and query/receipt observations. The preceding read checkpoint at `3da4481e` decouples pin capacity from transport limits and repairs the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing @@ -325,18 +329,23 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Complete the new apply adapter's failure qualification: cancel its receiver - and close the pool while image or manifest I/O is held, then prove that physical - work and reservations remain owned until the workers join. Inject preparation - and restart failures; publication-sync retention alone does not cover these cuts. -2. Integrate catalog invocation sessions and registered command/query callbacks - with logical cell identity and version observations. Preserve request deadlines, - receipt identity and uncertainty across redirects; never fabricate legacy - `GroupScope` values. This is the most important remaining serving gap. -3. Complete CG-04's production retry/cache lifecycle and configuration-entry - handling at the appropriate owner boundary, then run its entire gate before - marking it DONE. Full snapshot/membership-change qualification follows in CG-05; - archive/reclamation in CG-06 must preserve complete group recovery closure. +1. Integrate catalog invocation sessions and registered command/query callbacks + with logical cell identity and version observations. Start at + `application/consensus/class.rs`: command/query contexts still expose legacy + `GroupScope`, and queries expose `CommittedVersion`. Carry `CellHistory` and + selected cell observations through callbacks, receipts and effect validation; + keep physical quorum proof separate. Preserve original deadlines, request + fingerprints and the total retry budget. Never fabricate legacy group scopes. +2. Complete CG-04's production retry/cache lifecycle and configuration-entry + handling at the owning boundary. The new `CellGroupApplier::recover` provides + selected-image startup with the same page owner; use it in the dispatcher. + The actual image/manifest directory-sync cancellation and failure cuts now pass. + Add whole-process kill/restart coverage before claiming crash qualification; + joined cold reopen does not establish power-loss behavior. +3. Run the entire CG-04 gate before marking it DONE. Full snapshot/membership + qualification follows in CG-05; archive/reclamation in CG-06 must preserve the + complete group recovery closure. Preserve CG-08-before-CG-07 for effect identity + and public cutover. The opportunity is to reuse the now-shared consensus, SQL, read and apply paths in one dispatcher. Additional per-cell workers, a second production mode or a demo diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 56084c7c0..29779a3d3 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,54 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 physical apply faults and selected-image recovery + +Qualified source: `1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1`. + +`CellGroupApplier::recover` reconstructs only the engine-selected complete root. +It verifies every selected physical image, logical binding and geometry, and +retains the same manifest-page owner for subsequent apply. Materialized IDs use +the existing deterministic derivation; filenames supply no authority. Extent +roots verify the full cell history and schema. Missing or corrupt selected closure +refuses. Startup must still consume the catalog permit and establish current-term +quorum before serving. + +Six new native cases exercise both storage modes. At actual image and manifest +directory-sync cuts, cancelling preparation and then cancelling close retains +worker, budget and completion ownership. A replacement pool refuses until cleanup +joins; exact preparation can then retry. Injected directory-sync failure preserves +the old selected root and releases preparation reservations. Cold restart closes +all prior engines, stores and execution domains, selects the published cell A, +ignores prepared but unpublished B, and replays B from the durable committed log. +It refuses serving authority without fresh quorum and refuses a missing selected +image directory without changing the marker. + +This is joined cold restart, not whole-process kill or power-loss qualification. +The first recovery run hit an ownership refusal because the fixture still held +engine-owned manifest pins. Closing the complete owner before reopening fixed the +fixture; ownership rules and assertions were preserved. + +All 260 selected Rust cases pass: 233 native cases across ten suites, 23 Cellar +library cases and four LTX image-store cases. Fifteen commands pass, including +all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, formatting +(1,815 Rust files), layout, dependency isolation, docs, source preflight and all 37 +plan-validator tests. The partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`, binding 205 +source inputs and twelve copied executables. Intermediate failures and passing +targeted runs are retained. Six process helpers run via parent cases; three +RustFS-specific cases remain unrun. The branch contains latest fetched +`origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The most critical gap is the registered SDK/catalog +dispatcher: callback contexts still expose legacy cell-scoped group identity. +Next carry logical cell identity and selected-version observations through command, +query, receipt and effect validation, then use the existing shared worker path with +original deadlines, request fingerprints and retry budgets. Production cache/retry +lifecycle, configuration-entry handling and process-kill cuts remain open. Group +snapshot/compaction, RustFS recovery, performance and public cutover remain later +gates. The opportunity is to connect the existing owners, reads and recovery into +one production dispatcher rather than build another runtime. + ## CG-04 committed image preparation on shared workers Qualified source: `c9100ec774d66f30c14eae009f9a9c76d2bc0c33`. Runtime implementation: `2085cc96`. From b28078afbd243c18ac7c671a481a3e369c086ab2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 01:54:15 -0700 Subject: [PATCH 075/568] feat(raft): dispatch registered commands with logical cell identity --- crates/cellar-logic/cell_group.rs | 17 ++ crates/cellar/application/consensus/class.rs | 16 +- crates/cellar/application/consensus/client.rs | 10 +- crates/cellar/application/consensus/mod.rs | 1 + crates/cellar/durability/raft/README.md | 28 +++ .../cellar/durability/raft/effect_delivery.rs | 55 +++--- .../durability/raft/effect_delivery_tests.rs | 71 ++++++- .../cellar/durability/raft/effect_history.rs | 53 +++++ .../cellar/durability/raft/effect_outbox.rs | 2 +- crates/cellar/durability/raft/effect_scan.rs | 4 +- crates/cellar/durability/raft/effects.rs | 11 +- .../cellar/durability/raft/execution/cell.rs | 26 +++ .../cellar/durability/raft/execution/mod.rs | 47 ++--- .../durability/raft/execution/registered.rs | 63 ++++++ crates/cellar/durability/raft/host_effects.rs | 3 +- .../raft/host_sql/cell_groups/mod.rs | 54 ++++- .../raft/host_sql/cell_groups/worker.rs | 20 +- .../cellar/durability/raft/host_sql/images.rs | 3 +- .../tests/native_cell_groups_execution.rs | 24 ++- crates/cellar/tests/native_raft_execution.rs | 2 +- crates/cellar/tests/native_raft_extents.rs | 4 +- .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 6 +- .../support/cell_group_sql_registered.rs | 186 ++++++++++++++++++ crates/cellar/tests/support/raft_node.rs | 6 +- .../tests/support/raft_outbox_loopback.rs | 2 +- .../tests/support/raft_outbox_native.rs | 4 +- .../design/multi-raft/application-contract.md | 4 +- .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 32 ++- examples/raft-effects/main.rs | 2 +- 31 files changed, 656 insertions(+), 104 deletions(-) create mode 100644 crates/cellar/durability/raft/effect_history.rs create mode 100644 crates/cellar/durability/raft/execution/registered.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_registered.rs diff --git a/crates/cellar-logic/cell_group.rs b/crates/cellar-logic/cell_group.rs index 048c24214..fe19a4b4b 100644 --- a/crates/cellar-logic/cell_group.rs +++ b/crates/cellar-logic/cell_group.rs @@ -59,6 +59,23 @@ pub struct CellHistory { pub incarnation: IncarnationId, } +/// Projects the actual logical binding from the temporary single-cell host. +/// No physical group identity becomes part of the logical history. This does +/// not migrate stored state or confer execution authority. +impl TryFrom<&crate::consensus::GroupScope> for CellHistory { + type Error = Refusal; + + fn try_from(scope: &crate::consensus::GroupScope) -> Result { + scope.validate()?; + Ok(Self { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new(&scope.cell)?, + incarnation: scope.incarnation, + }) + } +} + /// A physical Raft history; deliberately contains no cell identifier. #[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)] pub struct GroupHistory { diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index d8197e5f3..07131e7ff 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -4,7 +4,7 @@ use super::{CommandId, CommittedVersion, Durability, GroupScope}; use crate::storage::{CommandInputs, CommandTransaction, ImmutableQuery}; -use cellar_logic::NamespaceMode; +use cellar_logic::{cell_group::CellHistory, NamespaceMode}; use sha2::{Digest, Sha256}; use std::{ collections::{BTreeMap, BTreeSet}, @@ -105,7 +105,7 @@ impl std::error::Error for RegistrationError {} /// fn socket(context: &CommandContext<'_>) { let _ = context.websocket_pair(); } /// ``` pub struct CommandContext<'a> { - scope: &'a GroupScope, + scope: &'a CellHistory, command: CommandId, transaction: &'a CommandTransaction<'a>, blobs: Option, @@ -118,7 +118,7 @@ pub struct CommandContext<'a> { } impl<'a> CommandContext<'a> { - pub fn identity(&self) -> &GroupScope { + pub fn identity(&self) -> &CellHistory { self.scope } @@ -585,7 +585,7 @@ impl RaftClass { pub(crate) fn validate_command_identity( &self, - scope: &GroupScope, + scope: &CellHistory, contract: &OperationContract, id: CommandId, durability: Durability, @@ -735,7 +735,7 @@ impl RaftClass { pub(crate) fn invoke_command( &self, contract: &OperationContract, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -754,7 +754,7 @@ impl RaftClass { pub(crate) fn invoke_system_command( &self, contract: &OperationContract, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -773,7 +773,7 @@ impl RaftClass { fn invoke_registered_command( &self, registration: RegisteredCommand<'_>, - scope: &GroupScope, + scope: &CellHistory, command: CommandId, durability: Durability, transaction: &CommandTransaction<'_>, @@ -849,7 +849,7 @@ impl RaftClass { fn bind_command_manifest(&mut self) { let mut hash = Sha256::new(); - hash.update(b"cellar-raft-class-1"); + hash.update(b"cellar-cell-class-1"); hash.update(self.base_schema_digest); if let Some(limits) = self.blobs { hash.update(b"immutable-blobs-1"); diff --git a/crates/cellar/application/consensus/client.rs b/crates/cellar/application/consensus/client.rs index 56e0ebe65..7f4016cd5 100644 --- a/crates/cellar/application/consensus/client.rs +++ b/crates/cellar/application/consensus/client.rs @@ -858,7 +858,8 @@ impl RaftCellHandle { self.binding .class .validate_command_identity( - &self.binding.scope, + &cellar_logic::cell_group::CellHistory::try_from(&self.binding.scope) + .map_err(|_| CommandFailure::CorruptState)?, contract, options.id, options.durability, @@ -883,7 +884,12 @@ impl RaftCellHandle { .validate_system_operation(contract) .is_err() || options.id.client - != crate::raft::effects::system_client(&self.binding.scope)? + != crate::raft::effects::system_client( + &cellar_logic::cell_group::CellHistory::try_from( + &self.binding.scope, + ) + .map_err(|_| CommandFailure::CorruptState)?, + )? { return Err(CommandFailure::UnsupportedCapability); } diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index ca7128447..56b2c7b58 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -123,6 +123,7 @@ pub use client::{ RaftCloseReport, StaleReadPolicy, UnfinishedCommand, }; +pub use cellar_logic::cell_group::{CellHistory, CellId}; pub use cellar_logic::consensus::{ ClientId, ClusterId, CommandFailure, CommandId, CommittedVersion, DiskLineage, Durability, EngineInstanceId, GroupId, GroupScope, IncarnationId, LogIndex, LogPosition, MemberId, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 2ed76a907..bd18ef86c 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -1760,3 +1760,31 @@ require their fleet guard. Earlier roots/state have no compatibility reader. Use a fresh experimental namespace/prefix and local state. The earlier MR-10 evidence retains its exact historical protocol and source scope. + +## Shared-group registered command boundary + +`CellGroupSqlPool::stage_registered` dispatches a namespace's compiled `RaftClass` +on the existing fixed SQL workers. The selected route and private-image options +must agree on the full class schema. Operation contract, normalized input digest, +inbox authentication and reserved system-client checks precede receipt lookup, +including duplicate retries. The callback receives the exact logical `CellHistory` +from the execution authority. Prepared output follows the existing proposal, +publication and receipt-settlement path; it is not a committed response. +Registered input bytes are retained within the producer budget while queued or +executing. Cancelling the waiter does not release the physical producer. + +`CommandContext::identity()` and `RaftEffectTarget` now use `CellHistory` without a +physical group ID. Alarm clients, outbox lane identities and signed requests use +that logical history; normal group reassignment preserves them, while a new cell +incarnation separates them. Signed effects use canonical marker-1 logical-history +records and distinct `cellar-cell-delivery-*` domains. Old group-bearing records +and old signature domains refuse. The full class manifest uses the +`cellar-cell-class-1` domain, so this identity change cannot silently reuse a +previous namespace schema. There is no migration reader. + +The temporary single-cell host projects its actual logical binding into +`CellHistory`; the shared command path does not manufacture `GroupScope` values. +Registered shared queries and catalog/SDK routing still require integration; +`QueryContext` continues to expose its existing committed version until that +boundary is replaced. Shared effect scheduling and transfer qualification remain +separate work. These APIs do not by themselves complete the production cutover. diff --git a/crates/cellar/durability/raft/effect_delivery.rs b/crates/cellar/durability/raft/effect_delivery.rs index 8f511c98e..a25c016aa 100644 --- a/crates/cellar/durability/raft/effect_delivery.rs +++ b/crates/cellar/durability/raft/effect_delivery.rs @@ -3,12 +3,14 @@ //! Authenticated, immutable delivery envelopes. Transport success alone cannot //! settle an intent: the destination signs its transaction result and releases //! it through the ordinary command durability gate. +use super::history::WireHistory; use crate::application::consensus::{ - protocol::WireScope, ClientId, CommandFailure, CommandId, Durability, GroupScope, - OperationContract, OperationKind, RaftCellHandle, + ClientId, CommandFailure, CommandId, Durability, OperationContract, OperationKind, + RaftCellHandle, }; use anyhow::{ensure, Result}; use base64::{engine::general_purpose::URL_SAFE_NO_PAD, Engine as _}; +use cellar_logic::cell_group::CellHistory; use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; use futures_util::future::BoxFuture; use serde::{de::DeserializeOwned, Deserialize, Serialize}; @@ -27,12 +29,12 @@ pub(crate) const MAX_REQUEST: usize = 60 * 1024; pub(crate) const MAX_RECEIPT: usize = 16 * 1024; pub(crate) const MAX_PAYLOAD: usize = 32 * 1024; pub(crate) const MAX_RESULT: usize = 4 * 1024; -const REQUEST_DOMAIN: &[u8] = b"cellar-raft-delivery-request-1"; -const RECEIPT_DOMAIN: &[u8] = b"cellar-raft-delivery-receipt-1"; +const REQUEST_DOMAIN: &[u8] = b"cellar-cell-delivery-request-1"; +const RECEIPT_DOMAIN: &[u8] = b"cellar-cell-delivery-receipt-1"; const CLIENT_PREFIX: &[u8; 4] = b"RFX1"; pub(crate) fn protocol_identity() -> [u8; 32] { - Sha256::digest(b"cellar-raft-delivery-1/definition-target-1").into() + Sha256::digest(b"cellar-cell-delivery-1/definition-target-1").into() } /// Deployment-owned material shared by compatible replicas. Debug never @@ -80,7 +82,7 @@ impl RaftEffectVerificationKey { /// compiled schema; that manifest can contain self/cyclic delivery bindings. #[derive(Clone, Debug, PartialEq, Eq)] pub struct RaftEffectTarget { - pub(crate) scope: GroupScope, + pub(crate) scope: CellHistory, pub(crate) definition: [u8; 32], pub(crate) contract: OperationContract, pub(crate) key: RaftEffectVerificationKey, @@ -89,13 +91,12 @@ pub struct RaftEffectTarget { impl RaftEffectTarget { pub fn new( - scope: GroupScope, + scope: CellHistory, definition: [u8; 32], contract: OperationContract, receipt_key: RaftEffectVerificationKey, durability: Durability, ) -> Result { - scope.validate()?; ensure!( definition != [0; 32] && contract.kind == OperationKind::Command @@ -114,7 +115,7 @@ impl RaftEffectTarget { }) } - pub fn scope(&self) -> &GroupScope { + pub fn scope(&self) -> &CellHistory { &self.scope } @@ -132,7 +133,7 @@ impl RaftEffectTarget { pub(crate) fn wire(&self) -> Target { Target { - scope: WireScope::from(&self.scope), + scope: WireHistory::from(&self.scope), definition: self.definition, operation: self.contract.operation.clone(), contract: self.contract.digest, @@ -194,11 +195,12 @@ impl RaftEffectSourceGrant { fn accepts(&self, request: &Request) -> Result { let scope = request.origin.clone().decode()?; - scope.validate()?; + Ok(scope.cluster == self.cluster && scope.namespace == self.namespace && scope .cell + .as_str() .split_once(':') .is_some_and(|(class, _)| class == self.class) && request.binding == self.binding @@ -283,7 +285,7 @@ fn verify( #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct Target { - scope: WireScope, + scope: WireHistory, definition: [u8; 32], operation: String, contract: [u8; 32], @@ -295,7 +297,7 @@ pub(crate) struct Target { #[serde(deny_unknown_fields)] pub(crate) struct Request { marker: u8, - pub(crate) origin: WireScope, + origin: WireHistory, origin_schema: [u8; 32], pub(crate) binding: String, binding_digest: [u8; 32], @@ -330,7 +332,7 @@ impl fmt::Debug for RaftDeliveryRequest { } pub(crate) struct DeliveryOrigin<'a> { - pub scope: &'a GroupScope, + pub scope: &'a CellHistory, pub schema: [u8; 32], pub binding: &'a str, pub digest: [u8; 32], @@ -351,7 +353,7 @@ impl RaftDeliveryRequest { digest, sequence, } = origin; - origin.validate()?; + ensure!( schema != [0; 32] && valid_name(binding) @@ -362,7 +364,7 @@ impl RaftDeliveryRequest { ); let request = Request { marker: 1, - origin: WireScope::from(origin), + origin: WireHistory::from(origin), origin_schema: schema, binding: binding.into(), binding_digest: digest, @@ -446,7 +448,7 @@ fn identity(request: &Request) -> Result<[u8; 32]> { &request.binding_digest, &request.target, ))?; - Ok(Sha256::digest(message(b"cellar-raft-delivery-lane-1", &body)).into()) + Ok(Sha256::digest(message(b"cellar-cell-delivery-lane-1", &body)).into()) } fn command_id(request: &Request) -> Result { @@ -473,7 +475,7 @@ pub(crate) struct Incoming { pub(crate) fn authenticate( bytes: &[u8], grants: &[RaftEffectSourceGrant], - target: &GroupScope, + target: &CellHistory, definition: [u8; 32], contract: &OperationContract, key: RaftEffectVerificationKey, @@ -568,7 +570,7 @@ pub struct RaftNativeEffectSink { #[derive(Clone)] struct NativeEndpoint { handle: Weak, - scope: GroupScope, + scope: CellHistory, } impl RaftNativeEffectSink { @@ -589,11 +591,13 @@ impl RaftNativeEffectSink { Ok(Arc::new(Self { endpoints: endpoints .into_iter() - .map(|endpoint| NativeEndpoint { - handle: Arc::downgrade(&endpoint.handle), - scope: endpoint.scope().clone(), + .map(|endpoint| { + Ok(NativeEndpoint { + handle: Arc::downgrade(&endpoint.handle), + scope: CellHistory::try_from(endpoint.scope())?, + }) }) - .collect(), + .collect::>>()?, timeout, retry_window_ms, })) @@ -620,14 +624,15 @@ impl RaftEffectSink for RaftNativeEffectSink { }; let binding = match handle .bindings - .get(&(hint.scope.namespace, hint.scope.cell)) + .get(&(hint.scope.namespace, hint.scope.cell.as_str().to_owned())) { Ok(binding) => binding, Err(CommandFailure::Unavailable) => continue, Err(error) => return Err(error), }; let endpoint = RaftCellHandle { handle, binding }; - if endpoint.scope() != request.inner.target.scope() + if CellHistory::try_from(endpoint.scope()).as_ref().ok() + != Some(request.inner.target.scope()) || endpoint.binding.class.definition_digest() != request.inner.target.definition { continue; diff --git a/crates/cellar/durability/raft/effect_delivery_tests.rs b/crates/cellar/durability/raft/effect_delivery_tests.rs index fab1d381d..5881f6813 100644 --- a/crates/cellar/durability/raft/effect_delivery_tests.rs +++ b/crates/cellar/durability/raft/effect_delivery_tests.rs @@ -1,20 +1,19 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; -use crate::application::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; +use crate::application::consensus::{ClusterId, IncarnationId, NamespaceId}; -fn scope(id: u8, class: &str) -> Result { - Ok(GroupScope { +fn scope(id: u8, class: &str) -> Result { + Ok(CellHistory { cluster: ClusterId::new([id; 16])?, namespace: NamespaceId::new([id + 1; 16])?, - cell: format!("{class}:one"), - group: GroupId::new([id + 2; 16])?, + cell: cellar_logic::cell_group::CellId::new(format!("{class}:one"))?, incarnation: IncarnationId::new([id + 3; 16])?, }) } struct Fixture { - origin: GroupScope, + origin: CellHistory, target: RaftEffectTarget, source: RaftEffectSigningKey, receiver: RaftEffectSigningKey, @@ -222,3 +221,63 @@ fn delivery_bounds_and_secret_debug() -> Result<()> { ensure!(reserved_client(f.request(1, b"first")?.command_id().client)); Ok(()) } + +#[test] +fn logical_effect_identity_survives_group_reassignment() -> Result<()> { + use cellar_logic::consensus::{GroupId, GroupScope}; + let mut physical = GroupScope { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: "Source:one".into(), + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let before = CellHistory::try_from(&physical)?; + physical.group = GroupId::new([8; 16])?; + let after = CellHistory::try_from(&physical)?; + let mut f = Fixture::new()?; + f.origin = before.clone(); + let first = f.request(1, b"same")?; + f.origin = after.clone(); + assert_eq!(first.encoded(), f.request(1, b"same")?.encoded()); + assert_eq!( + super::super::system_client(&before)?, + super::super::system_client(&after)? + ); + f.origin.incarnation = IncarnationId::new([7; 16])?; + assert_ne!(first.command_id(), f.request(1, b"same")?.command_id()); + assert_ne!( + super::super::system_client(&before)?, + super::super::system_client(&f.origin)? + ); + f.origin = before; + f.origin.cell = cellar_logic::cell_group::CellId::new("Source:two")?; + assert_ne!(first.command_id(), f.request(1, b"same")?.command_id()); + Ok(()) +} + +#[test] +fn cell_effect_wire_refuses_group_scope_and_old_signature_domain() -> Result<()> { + let f = Fixture::new()?; + let request = f.request(1, b"same")?; + let signed: Signed = parse(request.encoded(), MAX_REQUEST)?; + let old_domain = sign( + signed.body.clone(), + &f.source, + b"cellar-raft-delivery-request-1", + MAX_REQUEST, + )?; + assert!(f.incoming(&old_domain).is_err()); + for field in ["origin", "target"] { + let mut body = serde_json::to_value(&signed.body)?; + let scope = if field == "origin" { + &mut body["origin"] + } else { + &mut body["target"]["scope"] + }; + scope["group"] = hex::encode([3; 16]).into(); + let bytes = sign(body, &f.source, REQUEST_DOMAIN, MAX_REQUEST)?; + assert!(f.incoming(&bytes).is_err()); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/effect_history.rs b/crates/cellar/durability/raft/effect_history.rs new file mode 100644 index 000000000..a78dabbad --- /dev/null +++ b/crates/cellar/durability/raft/effect_history.rs @@ -0,0 +1,53 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical logical history for signed effects; physical placement is absent. +use anyhow::{ensure, Result}; +use cellar_logic::{ + cell_group::{CellHistory, CellId}, + consensus::{ClusterId, IncarnationId, NamespaceId}, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct WireHistory { + cluster: String, + namespace: String, + cell: String, + incarnation: String, +} + +impl From<&CellHistory> for WireHistory { + fn from(history: &CellHistory) -> Self { + Self { + cluster: hex::encode(history.cluster.bytes()), + namespace: hex::encode(history.namespace.bytes()), + cell: history.cell.as_str().to_owned(), + incarnation: hex::encode(history.incarnation.bytes()), + } + } +} + +impl WireHistory { + pub(super) fn decode(self) -> Result { + Ok(CellHistory { + cluster: ClusterId::new(identity(&self.cluster)?)?, + namespace: NamespaceId::new(identity(&self.namespace)?)?, + cell: CellId::new(self.cell)?, + incarnation: IncarnationId::new(identity(&self.incarnation)?)?, + }) + } +} + +fn identity(value: &str) -> Result<[u8; 16]> { + ensure!( + value.len() == 32 + && value + .bytes() + .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)), + "noncanonical effect identity" + ); + let mut bytes = [0; 16]; + hex::decode_to_slice(value, &mut bytes)?; + Ok(bytes) +} diff --git a/crates/cellar/durability/raft/effect_outbox.rs b/crates/cellar/durability/raft/effect_outbox.rs index 0d44ef8e7..36ccbb44a 100644 --- a/crates/cellar/durability/raft/effect_outbox.rs +++ b/crates/cellar/durability/raft/effect_outbox.rs @@ -536,7 +536,7 @@ pub(crate) struct ClaimedDelivery { pub(crate) fn claimed( outbox: &RaftOutbox, - scope: &crate::application::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, schema: [u8; 32], selected: &Selected, bytes: &[u8], diff --git a/crates/cellar/durability/raft/effect_scan.rs b/crates/cellar/durability/raft/effect_scan.rs index b584b1f45..e8ecd4a6a 100644 --- a/crates/cellar/durability/raft/effect_scan.rs +++ b/crates/cellar/durability/raft/effect_scan.rs @@ -63,7 +63,7 @@ pub(crate) fn scan( )?; scan_connection( database.connection(), - image.scope(), + &cellar_logic::cell_group::CellHistory::try_from(image.scope())?, image.position(), class, now, @@ -74,7 +74,7 @@ pub(crate) fn scan( pub(crate) fn scan_connection( connection: &rusqlite::Connection, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, position: LogPosition, class: &RaftClass, now: u64, diff --git a/crates/cellar/durability/raft/effects.rs b/crates/cellar/durability/raft/effects.rs index 24a25b215..59c2aecc7 100644 --- a/crates/cellar/durability/raft/effects.rs +++ b/crates/cellar/durability/raft/effects.rs @@ -11,6 +11,8 @@ pub(crate) mod alarm; mod blob; #[path = "effect_delivery.rs"] pub(crate) mod delivery; +#[path = "effect_history.rs"] +mod history; #[path = "effect_inbox.rs"] mod inbox; #[path = "effect_outbox.rs"] @@ -32,21 +34,20 @@ pub use options::RaftEffectOptions; /// One stable private command client per history, independent of node boot and /// leadership. Public commands cannot consume it in a class with system work. pub(crate) fn system_client( - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::CellHistory, ) -> Result { use sha2::{Digest, Sha256}; let mut hash = Sha256::new(); - hash.update(b"cellar-raft-system-client-1"); + hash.update(b"cellar-cell-system-client-1"); for identity in [ scope.cluster.bytes(), scope.namespace.bytes(), - scope.group.bytes(), scope.incarnation.bytes(), ] { hash.update(identity); } - hash.update((scope.cell.len() as u64).to_be_bytes()); - hash.update(scope.cell.as_bytes()); + hash.update((scope.cell.as_str().len() as u64).to_be_bytes()); + hash.update(scope.cell.as_str().as_bytes()); let digest: [u8; 32] = hash.finalize().into(); let mut id = [0; 16]; id.copy_from_slice(&digest[..16]); diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 0e494592f..66b3b6d0e 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -118,6 +118,32 @@ impl CellPrivateExecutor { }) } + /// Invoke a registered application operation with this cell's logical + /// identity. Validation precedes receipt lookup, including duplicate retries. + pub fn stage_registered( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + ) -> Result { + let history = self.authority.cell().version.history.clone(); + let contract = super::registered::validate( + self.options.limits, + self.options.application_digest, + &history, + &attempt, + class, + input, + false, + )?; + let command = attempt.request.id; + let durability = attempt.request.durability; + self.stage(attempt, allow_new, |transaction| { + class.invoke_command(&contract, &history, command, durability, transaction, input) + }) + } + /// Stage on an owning blocking worker. Callback output is retained only in /// the private receipt transaction; it cannot escape as a committed result. /// The host serializes unresolved producers by CellHistory, independently diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index 52b900ce2..88cc8ffde 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -13,6 +13,7 @@ pub(crate) use cell::authority_failure; pub use submission::{CellPreparedCommand, CellProposalRejection, CellSubmissionOutcome}; mod command; mod enrollment; +mod registered; pub use enrollment::{CellEnrollmentAttempt, CellEnrollmentResolution, CellPrivateEnrollment}; mod images; pub use cell::{CellCommittedReceipt, CellPrivateExecutor, CellReconciliation, CellStageOutcome}; @@ -633,43 +634,19 @@ impl PrivateExecutor { allow_new: bool, system: bool, ) -> Result { - let contract = OperationContract { - operation: attempt.request.operation.clone(), - kind: OperationKind::Command, - digest: attempt.request.contract_digest, - }; - if system { - class.validate_system_operation(&contract) - } else { - class.validate_operation(&contract) - } - .map_err(ExecutionError::Registration)?; - if class.schema_digest() != self.options.application_digest { - return Err(ExecutionError::Registration( - RegistrationError::ContractMismatch, - )); - } - if input.len() > self.options.limits.envelope.max_payload_bytes { - return Err(ExecutionError::Refused(CommandFailure::Overloaded)); - } - if command_request_digest(input) != attempt.request.request_digest { - return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); - } - let scope = self.authority.replica.scope.clone(); + let scope = cellar_logic::cell_group::CellHistory::try_from(&self.authority.replica.scope) + .map_err(storage)?; + let contract = registered::validate( + self.options.limits, + self.options.application_digest, + &scope, + &attempt, + class, + input, + system, + )?; let command = attempt.request.id; let durability = attempt.request.durability; - if !system { - class - .validate_command_identity(&scope, &contract, command, durability, input) - .map_err(|error| { - ExecutionError::Refused( - error - .downcast_ref::() - .cloned() - .unwrap_or(CommandFailure::UnsupportedCapability), - ) - })?; - } self.stage_admitted(attempt, allow_new, |transaction| { if system { class.invoke_system_command( diff --git a/crates/cellar/durability/raft/execution/registered.rs b/crates/cellar/durability/raft/execution/registered.rs new file mode 100644 index 000000000..479d4131e --- /dev/null +++ b/crates/cellar/durability/raft/execution/registered.rs @@ -0,0 +1,63 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registration and request identity are checked even for a receipt retry. +use super::*; +use cellar_logic::cell_group::CellHistory; + +pub(super) fn validate( + limits: ExecutionLimits, + schema: [u8; 32], + history: &CellHistory, + attempt: &CommandAttempt, + class: &RaftClass, + input: &[u8], + system: bool, +) -> Result { + let contract = OperationContract { + operation: attempt.request.operation.clone(), + kind: OperationKind::Command, + digest: attempt.request.contract_digest, + }; + if system { + class.validate_system_operation(&contract) + } else { + class.validate_operation(&contract) + } + .map_err(ExecutionError::Registration)?; + if class.schema_digest() != schema { + return Err(ExecutionError::Registration( + RegistrationError::ContractMismatch, + )); + } + if input.len() > limits.envelope.max_payload_bytes { + return Err(ExecutionError::Refused(CommandFailure::Overloaded)); + } + if command_request_digest(input) != attempt.request.request_digest { + return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); + } + if !system { + class + .validate_command_identity( + history, + &contract, + attempt.request.id, + attempt.request.durability, + input, + ) + .map_err(|error| { + ExecutionError::Refused( + error + .downcast_ref::() + .cloned() + .unwrap_or(CommandFailure::UnsupportedCapability), + ) + })?; + } else if attempt.request.id.client + != crate::raft::effects::system_client(history).map_err(ExecutionError::Refused)? + { + return Err(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )); + } + Ok(contract) +} diff --git a/crates/cellar/durability/raft/host_effects.rs b/crates/cellar/durability/raft/host_effects.rs index 0bfcec3a0..489a5965e 100644 --- a/crates/cellar/durability/raft/host_effects.rs +++ b/crates/cellar/durability/raft/host_effects.rs @@ -342,7 +342,8 @@ impl Driver { sink, }) = super::super::effects::outbox::claimed( &definition, - &scope, + &cellar_logic::cell_group::CellHistory::try_from(&scope) + .map_err(|_| CommandFailure::CorruptState)?, local.binding.class.schema_digest(), &selected, &receipt.result, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index bcd65b2d9..b70c85d71 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -436,6 +436,53 @@ impl CellGroupSqlPool { attempt: CommandAttempt, allow_new: bool, callback: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result> + Send + 'static, + ) -> Result { + self.stage_callback( + options, + authority, + attempt, + allow_new, + worker::Callback::Verification(Box::new(callback)), + ) + .await + } + + /// Dispatch an authorized namespace's compiled class on the same fixed + /// workers and receipt path. The class is checked against the selected route + /// schema; contract, input digest and inbox identity are checked on the worker + /// before receipt lookup. Cancellation retains the same physical ownership. + pub async fn stage_registered( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + allow_new: bool, + ) -> Result { + if input.len() > options.limits().envelope.max_payload_bytes { + return Err(overloaded()); + } + self.stage_callback( + options, + authority, + attempt, + allow_new, + worker::Callback::Registered { + class, + input: input.into_boxed_slice(), + }, + ) + .await + } + + async fn stage_callback( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: worker::Callback, ) -> Result { self.admit()?; if !authority.belongs_to_engine(&self.inner.engine_owner) { @@ -446,6 +493,10 @@ impl CellGroupSqlPool { } let limits = options.limits(); limits.validate()?; + let input_bytes = match &callback { + worker::Callback::Verification(_) => 0, + worker::Callback::Registered { input, .. } => input.len(), + }; // Bound simultaneous encoded capture, entry, preview verification and // result buffers before queueing or opening SQLite. Disk/workspace // reservation is separately enforced by the chosen image owner. @@ -453,6 +504,7 @@ impl CellGroupSqlPool { .envelope .max_payload_bytes .checked_mul(4) + .and_then(|n| n.checked_add(input_bytes)) .and_then(|n| n.checked_add(limits.max_result_bytes)) .and_then(|n| n.checked_add(4096)) .ok_or_else(overloaded)?; @@ -468,7 +520,7 @@ impl CellGroupSqlPool { authority, attempt, allow_new, - callback: Box::new(callback), + callback, lease, bytes, })), diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index d6c4e3568..c136f7712 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -3,7 +3,16 @@ use super::*; use std::collections::BTreeMap; -type Callback = Box) -> anyhow::Result> + Send>; +type VerificationCallback = + Box) -> anyhow::Result> + Send>; + +pub(super) enum Callback { + Verification(VerificationCallback), + Registered { + class: Arc, + input: Box<[u8]>, + }, +} pub(super) struct Stage { pub options: CellSqlOptions, @@ -175,7 +184,14 @@ impl Cells { } let max_result = options.limits().max_result_bytes; let mut executor = options.open(authority)?; - let result = executor.stage(attempt, allow_new, callback); + let result = match callback { + Callback::Verification(callback) => { + executor.stage(attempt, allow_new, callback) + } + Callback::Registered { class, input } => { + executor.stage_registered(attempt, &class, &input, allow_new) + } + }; match result { Ok(CellStageOutcome::Prepared(prepared)) => { let retained = prepared diff --git a/crates/cellar/durability/raft/host_sql/images.rs b/crates/cellar/durability/raft/host_sql/images.rs index cdebac2ec..8a41af95b 100644 --- a/crates/cellar/durability/raft/host_sql/images.rs +++ b/crates/cellar/durability/raft/host_sql/images.rs @@ -505,7 +505,8 @@ impl ImageOwner { let database = options.database(&image)?; scan::scan_connection( database.connection(), - image.scope(), + &cellar_logic::cell_group::CellHistory::try_from(image.scope()) + .map_err(storage)?, image.position(), class, now, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 5f34c151b..edb4e14dd 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -190,6 +190,18 @@ impl Cluster { options: RaftRuntimeOptions, root: tempfile::TempDir, filesystem: Arc, + ) -> Result { + Box::pin(Self::with_storage_schema( + options, root, filesystem, [6; 32], + )) + .await + } + + async fn with_storage_schema( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { @@ -207,7 +219,7 @@ impl Cluster { history.cluster, history.namespace, cellar_logic::NamespaceMode::Raft, - [6; 32], + schema, ))) .await??; let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; @@ -1288,3 +1300,13 @@ async fn shared_apply_recovers_selected_images_before_replay() -> Result<()> { async fn rooted_shared_apply_recovers_selected_images_before_replay() -> Result<()> { runtime_fixture::recover_apply(true).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn registered_commands_use_cell_identity_and_receipts() -> Result<()> { + runtime_fixture::registered_commands(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_registered_commands_use_cell_identity_and_receipts() -> Result<()> { + runtime_fixture::registered_commands(true).await +} diff --git a/crates/cellar/tests/native_raft_execution.rs b/crates/cellar/tests/native_raft_execution.rs index 9ef850e5d..cbf03eb47 100644 --- a/crates/cellar/tests/native_raft_execution.rs +++ b/crates/cellar/tests/native_raft_execution.rs @@ -656,7 +656,7 @@ async fn registered_command_uses_exact_image_receipt_and_read_only_query() -> Re let class = RaftClass::new("Counter", 1, [22; 32])? .command("Increment", [23; 32], move |context, input| { called.fetch_add(1, Ordering::SeqCst); - assert_eq!(context.identity(), &scope()); + assert_eq!(context.identity(), &cellar_logic::cell_group::CellHistory::try_from(&scope())?); assert_eq!(context.command().sequence(), 1); assert_eq!(context.inputs().now_unix_millis, 10); assert_eq!(context.inputs().random, [17; 32]); diff --git a/crates/cellar/tests/native_raft_extents.rs b/crates/cellar/tests/native_raft_extents.rs index 822564f26..cb2708168 100644 --- a/crates/cellar/tests/native_raft_extents.rs +++ b/crates/cellar/tests/native_raft_extents.rs @@ -78,12 +78,12 @@ struct Callbacks { } impl Callbacks { - fn record(&self, scope: &GroupScope) { + fn record(&self, scope: &cellar_logic::cell_group::CellHistory) { *self .by_cell .lock() .unwrap() - .entry(scope.cell.clone()) + .entry(scope.cell.as_str().to_owned()) .or_default() += 1; } diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index d9f98a629..7002bcb49 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -665,3 +665,5 @@ pub(super) use sql::apply_images; pub(super) use sql::{apply_cancellation, apply_failure}; pub(super) use sql::recover_apply; + +pub(super) use sql::registered_commands; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 975750de1..bf7c9d145 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -114,7 +114,7 @@ impl RuntimeCluster { now_unix_millis: Arc::new(|| 10), fill_random: Arc::new(|bytes| bytes.fill(17)), }, - application_digest: [6; 32], + application_digest: self.cluster.birth.route().schema_digest, limits: ExecutionLimits::default(), }; let extents = self.cluster.voters[node].extents.clone(); @@ -509,3 +509,7 @@ pub(crate) use apply_faults::{cancellation as apply_cancellation, failure as app #[path = "cell_group_sql_apply_recovery.rs"] mod apply_recovery; pub(crate) use apply_recovery::recover as recover_apply; + +#[path = "cell_group_sql_registered.rs"] +mod registered; +pub(crate) use registered::commands as registered_commands; diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs new file mode 100644 index 000000000..4c9f518a8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -0,0 +1,186 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered callbacks, logical identity and receipts through real shared Raft. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn commands(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [41; 32])? + .alarm("Wake", [42; 32], |_, _| Ok(()))? + .command("Increment", [43; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + assert_eq!(context.command().sequence(), 1); + assert_eq!(context.inputs().random, [17; 32]); + assert_eq!(context.identity().incarnation, IncarnationId::new([8; 16])?); + let delta = i64::from_be_bytes(input.try_into()?); + context.sql().execute("UPDATE value SET n=n+?1", [delta])?; + context.set_alarm("Wake", 50, b"wake")?; + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + })?, + ); + let cluster = Cluster::with_storage_schema( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + class.schema_digest(), + ) + .await?; + let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let a = history(&route, "registered-a")?; + let b = history(&route, "registered-b")?; + for node in 0..3 { + apply::install(&mut fixture, &a, node).await?; + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + for cell in [&a, &b] { + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, cell, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + } + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let input = 7_i64.to_be_bytes().to_vec(); + let attempt = |tag| -> Result { + let mut attempt = cell_attempt(tag)?; + attempt.request.operation = "Increment".into(); + attempt.request.contract_digest = [43; 32]; + attempt.request.request_digest = command_request_digest(&input); + Ok(attempt) + }; + for (n, cell) in [&a, &b].into_iter().enumerate() { + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(150 + n as u8)?, + class.clone(), + input.clone(), + true, + ) + .await? + else { + anyhow::bail!("prepared registered command") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!( + receipt.receipt().receipt.result(), + cell.cell.as_str().as_bytes() + ); + assert_eq!(receipt.receipt().version.history, *cell); + assert_eq!(receipt.receipt().version.sequence.get(), 2); + drop(receipt); + drop(prepared); + for runtime in &fixture.runtimes { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().effects.next_alarm_unix_millis, Some(50)); + assert_eq!(selected.cell().version.sequence.get(), 2); + } + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + // A's exact receipt remains valid after B advances the group; no callback reruns. + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = pool + .stage_registered( + fixture.cell_options(&a).await?, + authority, + attempt(155)?, + class.clone(), + input.clone(), + false, + ) + .await? + else { + anyhow::bail!("registered duplicate") + }; + assert_eq!( + receipt.receipt().receipt.result(), + a.cell.as_str().as_bytes() + ); + drop(receipt); + for fault in 0..4 { + let mut request = attempt(156 + fault)?; + let mut supplied = input.clone(); + let mut definition = class.clone(); + match fault { + 0 => request.request.contract_digest = [44; 32], + 1 => supplied[0] ^= 1, + 2 => request.request.operation = "Missing".into(), + _ => { + definition = Arc::new(RaftClass::new("Counter", 1, [45; 32])?.command( + "Increment", + [43; 32], + |_, _| panic!("wrong schema callback"), + )?) + } + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let result = pool + .stage_registered( + fixture.cell_options(&a).await?, + authority, + request, + definition, + supplied, + false, + ) + .await; + match fault { + 0 | 3 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::ContractMismatch + )) + )), + 1 => assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::CommandConflict)) + )), + 2 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::UnknownOperation(_) + )) + )), + _ => unreachable!(), + } + assert_eq!(pool.usage()?.active_cells, 0); + } + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(pool.usage()?.retained_bytes, 0); + drop(pool); + fixture.close().await +} diff --git a/crates/cellar/tests/support/raft_node.rs b/crates/cellar/tests/support/raft_node.rs index b2a6aa856..3c61ae3f3 100644 --- a/crates/cellar/tests/support/raft_node.rs +++ b/crates/cellar/tests/support/raft_node.rs @@ -425,7 +425,7 @@ pub fn counter_with_blobs( gate: Arc, blobs: bool, ) -> Result { - let observe = move |_: &GroupScope| { + let observe = move |_: &cellar_logic::cell_group::CellHistory| { calls.fetch_add(1, std::sync::atomic::Ordering::SeqCst); Ok(()) }; @@ -438,14 +438,14 @@ pub fn counter_with_blobs( /// Attribute private execution to its group without changing counter SQL or gates. pub fn counter_observed( - observe: impl Fn(&GroupScope) -> Result<()> + Send + Sync + 'static, + observe: impl Fn(&cellar_logic::cell_group::CellHistory) -> Result<()> + Send + Sync + 'static, gate: Arc, ) -> Result { counter_observed_with_blobs(observe, gate, false) } fn counter_observed_with_blobs( - observe: impl Fn(&GroupScope) -> Result<()> + Send + Sync + 'static, + observe: impl Fn(&cellar_logic::cell_group::CellHistory) -> Result<()> + Send + Sync + 'static, gate: Arc, blobs: bool, ) -> Result { diff --git a/crates/cellar/tests/support/raft_outbox_loopback.rs b/crates/cellar/tests/support/raft_outbox_loopback.rs index 518b329af..f2032c4d2 100644 --- a/crates/cellar/tests/support/raft_outbox_loopback.rs +++ b/crates/cellar/tests/support/raft_outbox_loopback.rs @@ -45,7 +45,7 @@ pub async fn self_delivery() -> Result<()> { digest: [98; 32], }; let target = RaftEffectTarget::new( - scope.clone(), + cellar_logic::cell_group::CellHistory::try_from(&scope.clone())?, definition, contract.clone(), receiver.verification_key(), diff --git a/crates/cellar/tests/support/raft_outbox_native.rs b/crates/cellar/tests/support/raft_outbox_native.rs index 165e5c5b6..7a4d8b5d4 100644 --- a/crates/cellar/tests/support/raft_outbox_native.rs +++ b/crates/cellar/tests/support/raft_outbox_native.rs @@ -424,7 +424,9 @@ impl Fixture { ) .await?; let target = RaftEffectTarget::new( - destination.registration.birth.scope.clone(), + cellar_logic::cell_group::CellHistory::try_from( + &destination.registration.birth.scope.clone(), + )?, destination.definitions[0].definition, destination.definitions[0].write.clone(), receiver_key.verification_key(), diff --git a/docs/design/multi-raft/application-contract.md b/docs/design/multi-raft/application-contract.md index fe9cfa9e8..9e22e7e1a 100644 --- a/docs/design/multi-raft/application-contract.md +++ b/docs/design/multi-raft/application-contract.md @@ -51,13 +51,13 @@ Register operations by `(class, operation, kind, contract_digest)`. Ingress bind The internal MR-06 transaction facade supplies a captured Unix-millisecond value and 32 captured random bytes per attempt. Its SQL authorizer refuses ambient clock/random functions; use these captured values as SQL parameters. Additional input facilities require explicit host adapters before becoming SDK capabilities. - `RaftClass::schema_digest()` first binds the class name, explicit schema version and supplied schema-definition digest under the `cellar-raft-schema-1` SHA-256 domain. -- It then binds that base digest and the sorted required command names and contract digests under `cellar-raft-class-1`, with explicit count/name lengths. +- It then binds that base digest and the sorted required command names and contract digests under `cellar-cell-class-1`, with explicit count/name lengths. - Read this identity after registering commands and use it for the namespace root and physical update application digest. - Omitting, renaming or changing a command cannot preserve voting compatibility by repeating the schema version/digest. - Delivery targets use `RaftClass::definition_digest()` plus an exact operation contract. - That declaration identity binds the class name, schema version and supplied definition digest before adding bindings, allowing self-delivery and cyclic class declarations without recursive manifest hashes. - It does not replace the complete `schema_digest()` check at namespace admission. -- Full target history selects the immutable namespace and its required compiled manifest. +- Full logical target history (`CellHistory`) selects the immutable namespace and its required compiled manifest. Physical group placement is absent from command and effect identity; see the [shared command boundary](../../../crates/cellar/durability/raft/README.md#shared-group-registered-command-boundary). - Declaration order does not affect the identity. - Query contracts remain separate: a replica with incompatible/missing query code can still vote with the matching required commands, but refuses that query before executing it. - Changing required command contracts requires a fresh immutable namespace identity; this is a marker-1 hard cutover without a compatibility reader or in-place conversion. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index c068865a0..67cb8bfce 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 23 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 25 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 845555e9a..a5b41923c 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -479,7 +479,10 @@ "crates/cellar/tests/support/cell_group_sql_apply.rs", "crates/cellar/durability/raft/cell_groups/apply/recovery.rs", "crates/cellar/tests/support/cell_group_sql_apply_faults.rs", - "crates/cellar/tests/support/cell_group_sql_apply_recovery.rs" + "crates/cellar/tests/support/cell_group_sql_apply_recovery.rs", + "crates/cellar/durability/raft/execution/registered.rs", + "crates/cellar/durability/raft/effect_history.rs", + "crates/cellar/tests/support/cell_group_sql_registered.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -535,6 +538,13 @@ "shared_apply_recovers_selected_images_before_replay", "rooted_shared_apply_recovers_selected_images_before_replay" ] + }, + { + "action": "Use logical CellHistory at registered command and signed-effect identity boundaries. Dispatch namespace-authorized classes on the existing shared SQL workers; validate operation, schema, input and authenticated inbox identity before receipt lookup. Preserve exact proposal submission/settlement and cancellation ownership. Group placement is absent from effect identity; fresh cell incarnations remain fenced. Marker-1 old group-bearing effect wire and old signature domains refuse. Registered query/version and catalog dispatch remain separate integration work.", + "tests": [ + "registered_commands_use_cell_identity_and_receipts", + "rooted_registered_commands_use_cell_identity_and_receipts" + ] } ], "test_cases": [ @@ -748,6 +758,26 @@ "I03", "I12" ] + }, + { + "name": "registered_commands_use_cell_identity_and_receipts", + "oracle": "Dispatch a real registered class through the shared SQL pool and production image applier on three voters. Two logical cells in one group receive their exact histories, mutate independently and retain alarm hints. After B advances group progress, A returns its original receipt without rerunning its callback. Wrong contract, input digest, operation and class schema refuse even on a receipt retry, releasing physical admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_registered_commands_use_cell_identity_and_receipts", + "oracle": "Dispatch a real registered class through the shared SQL pool and production image applier on three voters. Two logical cells in one group receive their exact histories, mutate independently and retain alarm hints. After B advances group progress, A returns its original receipt without rerunning its callback. Wrong contract, input digest, operation and class schema refuse even on a receipt retry, releasing physical admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ diff --git a/examples/raft-effects/main.rs b/examples/raft-effects/main.rs index 6cef860a6..02e67be37 100644 --- a/examples/raft-effects/main.rs +++ b/examples/raft-effects/main.rs @@ -206,7 +206,7 @@ async fn main() -> Result<()> { let mut definitions=Vec::new();let mut listeners=Vec::new();let mut peers=Vec::new(); for index in 0..3u8 { let destination=effects::receiver(sender.verification_key(),receiver.clone())?; - let target=RaftEffectTarget::new(effects::scope(true)?,destination.definition_digest(),destination.contract("Receive").context("Receive")?,receiver.verification_key(),Durability::S3)?; + let target=RaftEffectTarget::new(cellar::sdk::consensus::CellHistory::try_from(&effects::scope(true)?)?,destination.definition_digest(),destination.contract("Receive").context("Receive")?,receiver.verification_key(),Durability::S3)?; let source=effects::producer(sender.clone(),target,routing.clone())?; let contracts=(source.contract("Prime").context("source Prime")?,source.contract("Emit").context("Emit")?,source.contract("Get").context("source Get")?,destination.contract("Prime").context("receiver Prime")?,destination.contract("Get").context("receiver Get")?); let schemas=[source.schema_digest(),destination.schema_digest()]; From e039029e339b753baec1c4a5e0d0f9edd4fe470b Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 02:08:38 -0700 Subject: [PATCH 076/568] docs(raft): qualify registered commands and record next CG gaps --- docs/design/multi-raft/cell-groups/README.md | 9 ++ .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 92 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 51 ++++++++++ 4 files changed, 117 insertions(+), 41 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 6480071a5..49566001a 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -122,6 +122,15 @@ All 260 selected Rust cases and fifteen commands pass; transfer the retained `.cellar/cell-groups-evidence/CG-04-apply-recovery/` directory with prior evidence. Whole-process kill and power-loss cuts remain unqualified. See the newest [progress checkpoint](progress.md) for the committed-source gate and evidence. +Registered command callbacks now run on shared SQL workers at `b28078af` with +logical cell identity, pre-receipt validation and exact receipt settlement. +Command/effect identities exclude the physical group; old signed effect layouts +refuse under the marker-1 cutover. Shared query context, effect scheduling and +production catalog/SDK dispatch remain incomplete. All 295 selected Rust cases +and seventeen commands pass; transfer +`.cellar/cell-groups-evidence/CG-04-registered/` with the previous evidence. Its +checkpoint binds 221 source inputs and fifteen selected executables. See the newest +progress record for scope and retained failures. Registered SDK/catalog routing, query/version-token integration, production retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index acda4c20c..73d22fd31 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1", + "source_revision": "b28078afbd243c18ac7c671a481a3e369c086ab2", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -103,7 +103,7 @@ }, "effects": { "path": "crates/cellar/durability/raft/host_effects.rs", - "sha256": "292eb1b101eeb0c250a1e095d8be7d3c345166af335b9139eedc10a6c710dda0", + "sha256": "f588b3e8dd17d3e313ee24502c2910e1d38b7f88288a3df3c6fe16ec4be5432d", "line": 8, "excerpt": "pub(super) struct Scheduler {\n options: RaftEffectOptions,\n cursor: Option,\n next_at: u64,\n sequence: u64,\n busy: Option,\n enabled: bool,\n memory: Arc,\n}" }, diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index db4b87442..7cfdbb8cf 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,13 +15,19 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `1ec2eeaa`, adding selected-image recovery and -physical apply fault coverage in both image modes. Its retained checkpoint is -`.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`: 260 selected -Rust cases, fifteen commands, 205 source inputs and twelve executables. Transfer -that entire directory, including failed intermediate output, with prior evidence. -This qualifies joined cold restart, not whole-process kill or power loss. The preceding `c9100ec7` -checkpoint qualifies committed preparation on shared workers (`2085cc96`). The preceding `ed84df5d` checkpoint retains exact +The reviewed source baseline is `b28078af`, adding registered commands on shared +SQL workers and logical command/effect identities. Its partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered/checkpoint.json`: 295 selected Rust +cases, seventeen successful commands, 221 source inputs and fifteen selected +executables. Transfer its entire directory with prior evidence. It does not +complete CG-04, qualify shared effect scheduling or implement the public cutover. +The preceding `1ec2eeaa` +checkpoint qualifies selected-image recovery and physical apply fault coverage. +Its retained record is `.cellar/cell-groups-evidence/CG-04-apply-recovery/checkpoint.json`: +260 selected Rust cases, fifteen commands, 205 source inputs and twelve executables. +Transfer that entire directory with prior evidence. It qualifies joined cold +restart, not whole-process kill or power loss. The preceding `c9100ec7` checkpoint +qualifies committed preparation on shared workers (`2085cc96`). The preceding `ed84df5d` checkpoint retains exact cell-version witnesses in authenticated manifests and query/receipt observations. The preceding read checkpoint at `3da4481e` decouples pin capacity from transport limits and repairs the fixture macro. The initial read implementation is `88c4717d`. Earlier checkpoints cover prepared submission sealing @@ -128,12 +134,12 @@ the old proof. Accepted preparations retain their original position/digest even after apply; this remains provisional acceptance, never commitment. Prepared submission also closes byte-only proposal entry points on that owner. -Next integrate catalog routing sessions and registered SDK creation/command -callbacks, and registered query/version-token serving. Use `CellSqlPrepared::submit` and its typed result; +Next connect catalog routing and SDK creation/commands to the registered worker +entry point, and integrate registered query/version-token serving. Use `CellSqlPrepared::submit` and its typed result; never release a producer from generic overload or same-term absence. Preserve the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to -satisfy callback/effect identities. The callback APIs added here remain verification -boundaries, not registered SDK entry points. Real compaction/production retries +satisfy callback/effect identities. The arbitrary-callback `stage` API remains a verification boundary; use +`stage_registered` for compiled commands. Public SDK routing is still pending. Real compaction/production retries and preview/cache lifecycle remain required. These cases do not complete CG-04. The remaining coupling is concrete: `execution::CommandAuthority` binds the @@ -145,14 +151,21 @@ Reuse the existing receipt transaction and callback machinery; do not manufactur a synthetic cell-scoped group to satisfy the old interfaces. A later mutation of another cell must not settle the first cell's pending output. -The registered callback boundary also needs a real identity conversion: -`application/consensus/class.rs::CommandContext::identity` currently exposes -`GroupScope`, and command identity validation/invocation passes that same scope -to inbox and system-effect authentication. Carry logical `CellHistory` through -this boundary and keep physical consensus proof separate. Do not enable effects -with a fabricated group scope; preserve the CG-08-before-CG-07 dependency. Reuse -the new `host_sql::Pool` cell-history path and joined shutdown; do not create a -second node worker pool or a task per enrolled cell. +The registered command boundary now carries `CellHistory` through +`CommandContext`, inbox authentication, effect targets and system-client identity. +`CellGroupSqlPool::stage_registered` validates the operation, full class schema, +normalized input digest and authenticated identity before receipt lookup, then +uses the existing shared worker and exact proposal settlement path. Signed effect +wire and the class manifest use distinct marker-1 domains; there is no old-wire +reader. The temporary single-cell host projects its actual logical binding. +The shared command path derives identity directly from cell execution authority. + +Registered query contexts still expose `CommittedVersion`/`GroupScope`. Replace +that boundary with the selected logical cell version and witness before the public +cutover; never infer a mutation sequence from group progress. The catalog/SDK +adapter must select the authorized class and preserve invocation deadline, +fingerprint and retry budget. Full shared effect scheduling remains CG-08 work +before CG-07. Reuse the existing SQL workers and joined shutdown. A cancelled caller cannot release physical callback ownership or allow a second producer for that cell. @@ -231,8 +244,8 @@ at `ed84df5d` qualifies exact entry matching, unchanged-cell and older-pin provenance, restart/image-collection recovery and token round-trips in both image modes. Actual log-compaction and group-snapshot integration remain unqualified. -Next integrate registered callbacks and catalog invocation sessions without -synthesizing legacy group identities. Preserve original invocation deadlines, +Next integrate registered query callbacks and connect catalog invocation sessions +to the registered command workers without synthesizing legacy group identities. Preserve original invocation deadlines, contracts and retry identities. Query callbacks must receive logical cell versions; returned tokens must use the selected observation. `CellGroupApplier` now consumes the startup manifest-page owner, and `CellGroupSqlPool::prepare_apply` prepares committed cell images and complete @@ -329,23 +342,26 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Integrate catalog invocation sessions and registered command/query callbacks - with logical cell identity and version observations. Start at - `application/consensus/class.rs`: command/query contexts still expose legacy - `GroupScope`, and queries expose `CommittedVersion`. Carry `CellHistory` and - selected cell observations through callbacks, receipts and effect validation; - keep physical quorum proof separate. Preserve original deadlines, request - fingerprints and the total retry budget. Never fabricate legacy group scopes. -2. Complete CG-04's production retry/cache lifecycle and configuration-entry - handling at the owning boundary. The new `CellGroupApplier::recover` provides - selected-image startup with the same page owner; use it in the dispatcher. - The actual image/manifest directory-sync cancellation and failure cuts now pass. - Add whole-process kill/restart coverage before claiming crash qualification; - joined cold reopen does not establish power-loss behavior. -3. Run the entire CG-04 gate before marking it DONE. Full snapshot/membership - qualification follows in CG-05; archive/reclamation in CG-06 must preserve the - complete group recovery closure. Preserve CG-08-before-CG-07 for effect identity - and public cutover. +1. Integrate registered query contexts with logical cell versions and selected + decision witnesses. `QueryContext` still exposes the old `CommittedVersion`; + the shared read path already provides `CellVersionObservation`. Preserve + read-only callback restrictions and version-token authentication. Do not map + group log progress to a cell mutation sequence. The remaining old constructors + are `execution/mod.rs::query_registered_controlled` and + `rooted/query.rs::query_registered_rooted_controlled`; both construct only a + group-position `CommittedVersion`. The new worker lives in + `host_sql/cell_groups/query.rs` and already retains the exact observation. +2. Connect catalog invocation sessions and the production SDK dispatcher to + `CellGroupSqlPool::stage_registered`, initialization, coherent queries and exact + receipt settlement. Select classes from authorized namespace bindings, and + preserve original deadlines, request fingerprints and total retry budgets. + Do not manufacture legacy group scopes or create another worker pool. +3. Complete CG-04's production retry/cache lifecycle and configuration-entry + handling, then its full gate before DONE. Use the selected-root recovery API + at startup. Whole-process kill/power-loss remains distinct from the qualified + directory-sync faults and joined cold restart. Full snapshots/membership follow + in CG-05, archive/reclamation in CG-06, and shared effect scheduling/cache + lifecycle in CG-08 before the CG-07 public cutover. The opportunity is to reuse the now-shared consensus, SQL, read and apply paths in one dispatcher. Additional per-cell workers, a second production mode or a demo diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 29779a3d3..98509a32e 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,57 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 registered commands and logical effect identity + +Qualified source: `b28078afbd243c18ac7c671a481a3e369c086ab2`. + +`CellGroupSqlPool::stage_registered` now dispatches a real compiled `RaftClass` +on the existing fixed workers. Its retained input is bounded by the producer +budget. The worker verifies operation kind/contract, full class schema, normalized +input digest and authenticated inbox/reserved client identity before receipt +lookup, including duplicates. It uses the existing private receipt transaction, +prepared submission and exact committed settlement paths. The shared path obtains +logical identity from `CellExecutionAuthority`; it never constructs `GroupScope`. + +`CommandContext::identity()` and `RaftEffectTarget` carry `CellHistory`. System +clients and signed effect lanes/requests bind the logical cell, so normal physical +group reassignment preserves identity and a fresh cell incarnation separates it. +Canonical marker-1 records omit physical groups and use distinct signature/lane +and compiled-class domains. Old group-bearing payloads and old signatures refuse; +there is no compatibility reader. The temporary old host projects its actual +logical binding into the same command/effect interfaces. + +The new real three-voter tests use two cells and production image preparation in +both storage modes. Callbacks see the exact cell history and persist alarm state. +A's receipt survives B advancing group progress; duplicates do not run callbacks. +Wrong contract, input digest, operation and schema produce their specific typed +errors even on duplicate requests, without retaining cell admission. Unit cases +cover placement-independent signed identity, new-incarnation separation, and old +wire/domain refusal. This does not qualify live transfers or shared effect dispatch. + +All 295 selected Rust cases pass: 256 native cases across twelve suites, 30 Cellar +library cases, five pure cell-group core cases and four LTX image-store cases. +The native gate includes the existing alarm/outbox/inbox, extent-host and host +regressions. All seventeen commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,818 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The retained checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered/checkpoint.json`, binding 221 source +inputs and fifteen selected executables. Failed compile, missing test delimiter, +Clippy type-alias and omitted test submission-transition attempts are retained +with the corrected targeted runs. Six process helpers run through parent tests; +three RustFS-specific cases remain unrun. The branch contains latest fetched +`origin/main` at `8a93ffd7`. + +CG-04 remains IN_PROGRESS and CG-08 remains TODO. The most critical next gap is +registered query context and version observations, followed by catalog/SDK routing +into these workers with original deadline, fingerprint and retry budgets. Query +contexts still carry the old committed group version; never invent a cell sequence +from that index. Production cache/retry and configuration handling, process-kill +cuts, shared effects and later snapshot/archive/RustFS/performance gates remain. +The opportunity is to complete one dispatcher over the existing consensus, SQL, +read and recovery owners, then remove the superseded production host. + ## CG-04 physical apply faults and selected-image recovery Qualified source: `1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1`. From 7a79511d4b8606d11177778a3fe8adf71a8d901a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 02:27:10 -0700 Subject: [PATCH 077/568] feat(raft): dispatch registered queries with exact cell observations --- crates/cellar/application/consensus/class.rs | 59 +++- crates/cellar/application/consensus/mod.rs | 5 +- crates/cellar/durability/raft/README.md | 30 +- .../cellar/durability/raft/execution/mod.rs | 2 +- .../raft/host_sql/cell_groups/query.rs | 87 +++++- crates/cellar/durability/raft/rooted/query.rs | 2 +- crates/cellar/sdk.rs | 3 +- .../tests/native_cell_groups_execution.rs | 10 + crates/cellar/tests/native_raft_execution.rs | 5 +- crates/cellar/tests/native_raft_reads.rs | 5 +- .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../tests/support/cell_group_sql_reads.rs | 6 +- .../support/cell_group_sql_registered.rs | 78 ++--- .../cell_group_sql_registered_queries.rs | 267 ++++++++++++++++++ .../design/multi-raft/application-contract.md | 1 + .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 34 ++- 18 files changed, 537 insertions(+), 65 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_sql_registered_queries.rs diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index 07131e7ff..269e6a0db 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -2,7 +2,8 @@ //! Compiled, synchronous Raft operations with no ordinary mutable cell factory. -use super::{CommandId, CommittedVersion, Durability, GroupScope}; +use super::{CommandId, CommittedVersion, Durability}; +use crate::raft::version::CellVersionObservation; use crate::storage::{CommandInputs, CommandTransaction, ImmutableQuery}; use cellar_logic::{cell_group::CellHistory, NamespaceMode}; use sha2::{Digest, Sha256}; @@ -210,10 +211,21 @@ impl CommandContext<'_> { } } +/// Exact observation supplied by the owning query path. A group cut carries no +/// cell mutation sequence and must never be relabeled as a cell version. +#[derive(Clone, Copy, Debug)] +pub enum QueryVersion<'a> { + Cell(&'a CellVersionObservation), + /// Temporary single-cell host boundary, removed with that host in CG-16. + /// This is an in-memory observation, not an old-format recovery reader. + GroupCut(&'a CommittedVersion), +} + /// A pinned, read-only committed image. The host establishes consistency before /// constructing this context. Query code cannot open or migrate writable state. pub struct QueryContext<'a> { - version: &'a CommittedVersion, + identity: &'a CellHistory, + version: QueryVersion<'a>, database: &'a ImmutableQuery, blobs: Option, outbox: Option<&'a super::RaftOutbox>, @@ -229,11 +241,11 @@ impl QueryContext<'_> { ) } - pub fn identity(&self) -> &GroupScope { - &self.version.scope + pub fn identity(&self) -> &CellHistory { + self.identity } - pub fn version(&self) -> &CommittedVersion { + pub fn version(&self) -> QueryVersion<'_> { self.version } @@ -815,12 +827,46 @@ impl RaftClass { ) } - pub(crate) fn invoke_query( + pub(crate) fn invoke_group_query( &self, contract: &OperationContract, version: &CommittedVersion, database: &ImmutableQuery, input: &[u8], + ) -> anyhow::Result> { + let identity = CellHistory::try_from(&version.scope)?; + self.invoke_query( + contract, + &identity, + QueryVersion::GroupCut(version), + database, + input, + ) + } + + pub(crate) fn invoke_cell_query( + &self, + contract: &OperationContract, + observation: &CellVersionObservation, + database: &ImmutableQuery, + input: &[u8], + ) -> anyhow::Result> { + self.invoke_query( + contract, + &observation.version.history, + QueryVersion::Cell(observation), + database, + input, + ) + } + + fn invoke_query( + &self, + contract: &OperationContract, + identity: &CellHistory, + version: QueryVersion<'_>, + database: &ImmutableQuery, + input: &[u8], ) -> anyhow::Result> { self.validate_operation(contract)?; anyhow::ensure!( @@ -836,6 +882,7 @@ impl RaftClass { let _guard = CallbackGuard::enter()?; callback( &QueryContext { + identity, version, database, blobs: self.blobs, diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 56b2c7b58..23903fdcc 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -115,7 +115,7 @@ pub use crate::raft::version::{ pub use crate::storage::{CommandInputs, CommandTransaction}; pub use class::{ command_request_digest, CommandContext, OperationContract, OperationKind, QueryContext, - RaftClass, RegistrationError, + QueryVersion, RaftClass, RegistrationError, }; pub use client::{ CellStatus, CommandOptions, CommandReceipt, LearnerActivation, LearnerOptions, LearnerReceipt, @@ -123,7 +123,8 @@ pub use client::{ RaftCloseReport, StaleReadPolicy, UnfinishedCommand, }; -pub use cellar_logic::cell_group::{CellHistory, CellId}; +pub use crate::raft::version::CellVersionObservation; +pub use cellar_logic::cell_group::{CellHistory, CellId, CellVersion}; pub use cellar_logic::consensus::{ ClientId, ClusterId, CommandFailure, CommandId, CommittedVersion, DiskLineage, Durability, EngineInstanceId, GroupId, GroupScope, IncarnationId, LogIndex, LogPosition, MemberId, diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index bd18ef86c..8e307d28b 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -1784,7 +1784,29 @@ previous namespace schema. There is no migration reader. The temporary single-cell host projects its actual logical binding into `CellHistory`; the shared command path does not manufacture `GroupScope` values. -Registered shared queries and catalog/SDK routing still require integration; -`QueryContext` continues to expose its existing committed version until that -boundary is replaced. Shared effect scheduling and transfer qualification remain -separate work. These APIs do not by themselves complete the production cutover. +Registered shared queries use the boundary below. Catalog/SDK routing, shared +effect scheduling and transfer qualification remain separate work. These APIs do not by themselves complete the production cutover. + +## Shared-group registered query boundary + +`CellGroupSqlPool::query_registered` consumes the invocation's `CellReadAuthority` +and dispatches a compiled query on the existing workers. Operation names and +input are bounded before queueing; retained input and output use read admission. +The worker checks the operation kind, contract and full class schema before +opening the selected immutable image. The same owner retains deadline, SQL +interruption, cancellation, panic containment and joined shutdown behavior. +A cancelled receiver cannot release a still-running callback or its image pins. + +`QueryContext::identity()` exposes logical `CellHistory`. On this shared path, +`version()` yields `QueryVersion::Cell` containing the exact selected +`CellVersionObservation`: mutation sequence, image commitment and decision witness. +The returned `CellSqlQueryResult` retains that same observation for token issuance. +Advancing another cell's group log position cannot relabel the selected version. +A previously pinned image may remain a coherent historical read after publication. + +The temporary old host invokes `invoke_group_query` and supplies an explicit +`QueryVersion::GroupCut`. It cannot supply or infer a logical mutation sequence. +This in-memory staging observation is not a persisted-format reader or a final +runtime choice. CG-16 must remove that variant and constructor with the old host, +leaving the selected cell observation directly exposed by the final query context. +Catalog/SDK dispatch into these workers remains required before public cutover. diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index 88cc8ffde..a7efcf215 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -192,7 +192,7 @@ pub(crate) fn query_registered_controlled( position: published.position(), }; let result = class - .invoke_query(contract, &version, &database, input) + .invoke_group_query(contract, &version, &database, input) .map_err(storage)?; if result.len() > limits.max_result_bytes { return Err(ExecutionError::Refused(CommandFailure::Overloaded)); diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index 2fac605fc..85c65f3c7 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -7,10 +7,19 @@ use crate::raft::cell_groups::{CellImagePin, CellReadAuthority}; use cellar_logic::cell_group::CellVersion; use std::panic::AssertUnwindSafe; -type Callback = Box< +type VerificationCallback = Box< dyn FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + Send, >; +enum Callback { + Verification(VerificationCallback), + Registered { + class: Arc, + contract: crate::application::consensus::OperationContract, + input: Box<[u8]>, + }, +} + pub(super) struct Query { options: CellSqlOptions, authority: CellReadAuthority, @@ -59,6 +68,52 @@ impl CellGroupSqlPool { callback: impl FnOnce(&crate::storage::ImmutableQuery, &CellVersion) -> anyhow::Result> + Send + 'static, + ) -> Result { + self.query_callback( + options, + authority, + Callback::Verification(Box::new(callback)), + ) + .await + } + + /// Invoke the namespace's compiled query against the exact selected cell + /// image and version witness. The existing query owner enforces the original + /// deadline, cancellation, read-only SQL, output budget and joined cleanup. + pub async fn query_registered( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + mut contract: crate::application::consensus::OperationContract, + input: Vec, + ) -> Result { + if contract.operation.len() > 128 { + return Err(ExecutionError::Registration( + crate::application::consensus::RegistrationError::Invalid("class/operation name"), + )); + } + contract.operation = contract.operation.into_boxed_str().into_string(); + if input.len() > options.limits().envelope.max_payload_bytes { + return Err(overloaded()); + } + self.query_callback( + options, + authority, + Callback::Registered { + class, + contract, + input: input.into_boxed_slice(), + }, + ) + .await + } + + async fn query_callback( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, ) -> Result { self.admit()?; let domain = &self.inner.workers[0].domain; @@ -71,12 +126,17 @@ impl CellGroupSqlPool { .check(domain.mono_ms().map_err(|_| unavailable())?) .map_err(crate::raft::execution::authority_failure)?; options.limits().validate()?; + let input_bytes = match &callback { + Callback::Verification(_) => 0, + Callback::Registered { input, .. } => input.len(), + }; let bytes = self.inner.admission.reserve_read( authority.route().history, options .limits() .max_result_bytes .checked_add(4096) + .and_then(|n| n.checked_add(input_bytes)) .ok_or_else(overloaded)?, )?; let cell = authority.cell().version.history.clone(); @@ -86,7 +146,7 @@ impl CellGroupSqlPool { let query = Query { options, authority, - callback: Box::new(callback), + callback, cancelled, closing: self.inner.closing.clone(), bytes, @@ -134,6 +194,20 @@ pub(super) fn run( .map_err(crate::raft::execution::authority_failure) }; check(&query)?; + if let Callback::Registered { + class, + contract, + input, + } = &query.callback + { + crate::raft::execution::validate_registered_query( + class, + contract, + input, + query.authority.route().schema_digest, + query.options.limits(), + )?; + } let database = match (&query.options, query.authority.image()) { (CellSqlOptions::Materialized(options), CellImagePin::Materialized(pin)) => { crate::storage::open_immutable_generation( @@ -181,8 +255,13 @@ pub(super) fn run( .map_err(crate::raft::execution::storage)?; check(&query)?; let observation = query.authority.cell().observation(); - let result = std::panic::catch_unwind(AssertUnwindSafe(|| { - (query.callback)(&database, &observation.version) + let result = std::panic::catch_unwind(AssertUnwindSafe(|| match query.callback { + Callback::Verification(callback) => callback(&database, &observation.version), + Callback::Registered { + class, + contract, + input, + } => class.invoke_cell_query(&contract, &observation, &database, &input), })) .unwrap_or_else(|panic| Err(crate::native_worker::panic_error(panic))) .map_err(crate::raft::execution::storage)?; diff --git a/crates/cellar/durability/raft/rooted/query.rs b/crates/cellar/durability/raft/rooted/query.rs index fe0584c1a..e71a5081b 100644 --- a/crates/cellar/durability/raft/rooted/query.rs +++ b/crates/cellar/durability/raft/rooted/query.rs @@ -69,7 +69,7 @@ pub(crate) fn query_registered_rooted_controlled( position: published.position(), }; let result = class - .invoke_query(contract, &version, &database, input) + .invoke_group_query(contract, &version, &database, input) .map_err(ExecutionError::Storage)?; if result.len() > options.execution.max_result_bytes { return Err(ExecutionError::Refused( diff --git a/crates/cellar/sdk.rs b/crates/cellar/sdk.rs index 31b5ef5a0..0f92508e0 100644 --- a/crates/cellar/sdk.rs +++ b/crates/cellar/sdk.rs @@ -186,7 +186,8 @@ //! ``` pub use crate::actor::{Actor, ActorIdentity, ActorOptions, ActorServices}; pub use crate::application::consensus::{ - CommandContext, OperationContract, OperationKind, QueryContext, RaftClass, RegistrationError, + CommandContext, OperationContract, OperationKind, QueryContext, QueryVersion, RaftClass, + RegistrationError, }; pub use crate::application::filesystem::FilesystemLimits; pub use crate::application::limits::{ diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index edb4e14dd..7e782615f 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1310,3 +1310,13 @@ async fn registered_commands_use_cell_identity_and_receipts() -> Result<()> { async fn rooted_registered_commands_use_cell_identity_and_receipts() -> Result<()> { runtime_fixture::registered_commands(true).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn registered_queries_keep_cell_observation_and_ownership() -> Result<()> { + Box::pin(runtime_fixture::registered_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_registered_queries_keep_cell_observation_and_ownership() -> Result<()> { + Box::pin(runtime_fixture::registered_queries(true)).await +} diff --git a/crates/cellar/tests/native_raft_execution.rs b/crates/cellar/tests/native_raft_execution.rs index cbf03eb47..704eab5dc 100644 --- a/crates/cellar/tests/native_raft_execution.rs +++ b/crates/cellar/tests/native_raft_execution.rs @@ -667,8 +667,9 @@ async fn registered_command_uses_exact_image_receipt_and_read_only_query() -> Re Ok(value.to_be_bytes().to_vec()) })? .query("Get", [24; 32], |context, _| { - assert_eq!(context.identity(), &scope()); - assert!(context.version().position.index.get() > 0); + assert_eq!(context.identity(), &cellar_logic::cell_group::CellHistory::try_from(&scope())?); + let cellar::sdk::consensus::QueryVersion::GroupCut(version) = context.version() else { anyhow::bail!("single-cell adapter observation") }; + assert!(version.position.index.get() > 0); for sql in [ "UPDATE counter SET value=999 RETURNING value", "PRAGMA user_version=8", diff --git a/crates/cellar/tests/native_raft_reads.rs b/crates/cellar/tests/native_raft_reads.rs index 9b122b5a1..bd79202ea 100644 --- a/crates/cellar/tests/native_raft_reads.rs +++ b/crates/cellar/tests/native_raft_reads.rs @@ -1107,7 +1107,10 @@ async fn stale_is_committed_prefix() -> Result<()> { let query = Arc::new(move |context: &QueryContext<'_>, input: &[u8]| { let mut value = get(context, input)?; if input == b"hold" { - let index = context.version().position.index.get(); + let QueryVersion::GroupCut(version) = context.version() else { + anyhow::bail!("single-cell adapter observation") + }; + let index = version.position.index.get(); gate.wait()?; assert_eq!(get(context, input)?, value); value.extend_from_slice(&index.to_be_bytes()); diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 7002bcb49..04602bfde 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -667,3 +667,5 @@ pub(super) use sql::{apply_cancellation, apply_failure}; pub(super) use sql::recover_apply; pub(super) use sql::registered_commands; + +pub(super) use sql::registered_queries; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index bf7c9d145..c79cacf20 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -513,3 +513,7 @@ pub(crate) use apply_recovery::recover as recover_apply; #[path = "cell_group_sql_registered.rs"] mod registered; pub(crate) use registered::commands as registered_commands; + +#[path = "cell_group_sql_registered_queries.rs"] +mod registered_queries; +pub(crate) use registered_queries::queries as registered_queries; diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs index 78f5a39d4..d2fe2f57f 100644 --- a/crates/cellar/tests/support/cell_group_sql_reads.rs +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -3,7 +3,7 @@ use super::reconciliation::apply_through; use super::*; -async fn deadline(fixture: &RuntimeCluster, ms: u64) -> Result { +pub(super) async fn deadline(fixture: &RuntimeCluster, ms: u64) -> Result { fixture .cluster .domain @@ -27,7 +27,7 @@ fn pool(fixture: &RuntimeCluster, node: usize) -> Result { )?) } -async fn selected( +pub(super) async fn selected( fixture: &mut RuntimeCluster, node: usize, cell: &CellHistory, @@ -71,7 +71,7 @@ async fn value( .await?) } -fn verify_observation(output: &CellSqlQueryResult, schema: [u8; 32]) -> Result<()> { +pub(super) fn verify_observation(output: &CellSqlQueryResult, schema: [u8; 32]) -> Result<()> { let keys = VersionKeys::new( VersionSigningKey::new([73; 32], 100_000)?, vec![], diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index 4c9f518a8..9dd84a9b3 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -21,41 +21,7 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { Ok(context.identity().cell.as_str().as_bytes().to_vec()) })?, ); - let cluster = Cluster::with_storage_schema( - RaftRuntimeOptions::default(), - tempfile::tempdir()?, - Arc::new(DirectFileSystem), - class.schema_digest(), - ) - .await?; - let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; - let route = fixture.cluster.birth.route().clone(); - if rooted { - for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { - voter.extents = Some(NodeExtentStore::open( - fixture.cluster._root.path().join(format!("rooted-{n}")), - Arc::new(DirectFileSystem), - Default::default(), - )?); - } - } - let a = history(&route, "registered-a")?; - let b = history(&route, "registered-b")?; - for node in 0..3 { - apply::install(&mut fixture, &a, node).await?; - } - fixture.runtimes[0].campaign(route.history).await?; - fixture.drive(None).await?; - let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; - for cell in [&a, &b] { - let accepted = fixture.runtimes[0] - .propose( - route.history, - apply::enrollment_bytes(&route, cell, &bytes, rooted)?, - ) - .await?; - fixture.drive(Some(accepted.position.index)).await?; - } + let (mut fixture, route, a, b) = Box::pin(setup(rooted, &class)).await?; let pool = fixture.appliers[&(0, route.history)].0.clone(); let input = 7_i64.to_be_bytes().to_vec(); let attempt = |tag| -> Result { @@ -184,3 +150,45 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { drop(pool); fixture.close().await } + +pub(super) async fn setup( + rooted: bool, + class: &RaftClass, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let cluster = Cluster::with_storage_schema( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + class.schema_digest(), + ) + .await?; + let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + let route = fixture.cluster.birth.route().clone(); + if rooted { + for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + fixture.cluster._root.path().join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let a = history(&route, "registered-a")?; + let b = history(&route, "registered-b")?; + for node in 0..3 { + apply::install(&mut fixture, &a, node).await?; + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + for cell in [&a, &b] { + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, cell, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + } + Ok((fixture, route, a, b)) +} diff --git a/crates/cellar/tests/support/cell_group_sql_registered_queries.rs b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs new file mode 100644 index 000000000..28bef7150 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs @@ -0,0 +1,267 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered queries keep actual cell versions, immutable images and ownership. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn verify( + output: &CellSqlQueryResult, + value: i64, + expected: &CellVersionObservation, + schema: [u8; 32], +) -> Result<()> { + let bytes = output.result(); + assert_eq!(bytes.len(), 24); + assert_eq!(i64::from_le_bytes(bytes[..8].try_into()?), value); + assert_eq!( + u64::from_le_bytes(bytes[8..16].try_into()?), + expected.version.sequence.get() + ); + assert_eq!( + u64::from_le_bytes(bytes[16..].try_into()?), + expected.witness.position.index.get() + ); + assert_eq!(output.observation(), expected); + reads::verify_observation(output, schema) +} + +pub(crate) async fn queries(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let wait = Mutex::new(wait); + let class = Arc::new(RaftClass::new("Counter", 1, [51; 32])? + .command("Increment", [52; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+7", [])?; + Ok(vec![7]) + })? + .query("Get", [53; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { anyhow::bail!("shared query received a group cut") }; + assert_eq!(context.identity(), &observation.version.history); + for sql in ["UPDATE value SET n=999 RETURNING n", "PRAGMA user_version=8", "ATTACH DATABASE ':memory:' AS other", "SELECT count(*) FROM __sys_raft_receipts"] { + assert!(context.query_row(sql, [], |row| row.get::<_, i64>(0)).is_err(), "{sql}"); + } + if input == b"panic" { panic!("contained registered query panic"); } + if input == b"spin" { + let _: i64 = context.query_row("WITH RECURSIVE n(x) AS (VALUES(0) UNION ALL SELECT x+1 FROM n WHERE x<1000000000) SELECT sum(x) FROM n", [], |row| row.get(0))?; + } + if input.starts_with(b"hold") { + entered.lock().unwrap().take().context("one held invocation")?.send(()).map_err(|_| anyhow::anyhow!("held receiver"))?; + wait.lock().unwrap().recv_timeout(Duration::from_secs(10))?; + } + let value: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let mut result = value.to_le_bytes().to_vec(); + result.extend_from_slice(&observation.version.sequence.get().to_le_bytes()); + result.extend_from_slice(&observation.witness.position.index.get().to_le_bytes()); + Ok(result) + })?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let contract = class.contract("Get").context("Get")?; + let options = fixture.cell_options(&a).await?; + let old = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let original = old.cell().observation(); + for (n, cell) in [&a, &b].into_iter().enumerate() { + let mut attempt = cell_attempt(170 + n as u8)?; + attempt.request.operation = "Increment".into(); + attempt.request.contract_digest = [52; 32]; + attempt.request.request_digest = command_request_digest(&[]); + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt, + class.clone(), + vec![], + true, + ) + .await? + else { + anyhow::bail!("command preparation") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + drop(pool.finish(authority).await?); + drop(prepared); + } + let output = pool + .query_registered( + options.clone(), + old, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 0, &original, route.schema_digest)?; + drop(output); + let fresh = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Linearizable).await?; + let latest = fresh.cell().observation(); + assert_eq!(latest.version.sequence.get(), 2); + assert!( + latest.witness.position.index + < fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + ); + let output = pool + .query_registered( + options.clone(), + fresh, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 7, &latest, route.schema_digest)?; + drop(output); + let before = calls.load(Ordering::SeqCst); + for fault in 0..6 { + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let mut request = contract.clone(); + let mut definition = class.clone(); + let mut supplied = vec![]; + let mut config = options.clone(); + match fault { + 0 => request.digest = [54; 32], + 1 => request.kind = OperationKind::Command, + 2 => request.operation = "Missing".into(), + 3 => { + definition = Arc::new(RaftClass::new("Counter", 1, [55; 32])?.query( + "Get", + [53; 32], + |_, _| panic!("wrong class query"), + )?) + } + 4 => { + match &mut config { + CellSqlOptions::Materialized(o) => o.limits.envelope.max_payload_bytes = 1024, + CellSqlOptions::Extents(o) => o.limits.envelope.max_payload_bytes = 1024, + } + supplied = vec![0; 1025]; + } + _ => request.operation = "x".repeat(129), + } + let result = pool + .query_registered(config, authority, definition, request, supplied) + .await; + match fault { + 0 | 3 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::ContractMismatch + )) + )), + 1 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::KindMismatch + )) + )), + 2 => assert!(matches!( + result, + Err(ExecutionError::Registration( + RegistrationError::UnknownOperation(_) + )) + )), + 4 => assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )), + _ => assert!(matches!( + result, + Err(ExecutionError::Registration(RegistrationError::Invalid( + "class/operation name" + ))) + )), + } + assert_eq!(calls.load(Ordering::SeqCst), before); + assert_eq!(pool.usage()?.retained_operations, 0); + } + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + assert!(matches!( + pool.query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + b"panic".to_vec() + ) + .await, + Err(ExecutionError::Storage(_)) + )); + let deadline = reads::deadline(&fixture, 100).await?; + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + assert!(tokio::time::timeout( + Duration::from_secs(3), + pool.query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + b"spin".to_vec() + ) + ) + .await? + .is_err()); + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let output = pool + .query_registered( + options.clone(), + authority, + class.clone(), + contract.clone(), + vec![], + ) + .await?; + verify(&output, 7, &latest, route.schema_digest)?; + drop(output); + let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; + let caller = pool.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + let mut input = vec![0; 256 * 1024]; + input[..4].copy_from_slice(b"hold"); + caller + .query_registered(options, authority, class, contract, input) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(pool.usage()?.retained_bytes >= 256 * 1024); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + drop(pool); + fixture.close().await +} diff --git a/docs/design/multi-raft/application-contract.md b/docs/design/multi-raft/application-contract.md index 9e22e7e1a..6ffd75713 100644 --- a/docs/design/multi-raft/application-contract.md +++ b/docs/design/multi-raft/application-contract.md @@ -58,6 +58,7 @@ The internal MR-06 transaction facade supplies a captured Unix-millisecond value - That declaration identity binds the class name, schema version and supplied definition digest before adding bindings, allowing self-delivery and cyclic class declarations without recursive manifest hashes. - It does not replace the complete `schema_digest()` check at namespace admission. - Full logical target history (`CellHistory`) selects the immutable namespace and its required compiled manifest. Physical group placement is absent from command and effect identity; see the [shared command boundary](../../../crates/cellar/durability/raft/README.md#shared-group-registered-command-boundary). +- Registered shared queries expose their exact `CellVersionObservation` through `QueryVersion::Cell`; `QueryContext::identity()` is logical cell history. The temporary old host uses a distinct `GroupCut` observation with no mutation sequence. CG-16 removes that staging branch; see the [registered query boundary](../../../crates/cellar/durability/raft/README.md#shared-group-registered-query-boundary). - Declaration order does not affect the identity. - Query contracts remain separate: a replica with incompatible/missing query code can still vote with the matching required commands, but refuses that query before executing it. - Changing required command contracts requires a fresh immutable namespace identity; this is a marker-1 hard cutover without a compatibility reader or in-place conversion. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 67cb8bfce..4509d86cd 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 25 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 27 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index a5b41923c..f56eef7ab 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -482,7 +482,8 @@ "crates/cellar/tests/support/cell_group_sql_apply_recovery.rs", "crates/cellar/durability/raft/execution/registered.rs", "crates/cellar/durability/raft/effect_history.rs", - "crates/cellar/tests/support/cell_group_sql_registered.rs" + "crates/cellar/tests/support/cell_group_sql_registered.rs", + "crates/cellar/tests/support/cell_group_sql_registered_queries.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -540,11 +541,18 @@ ] }, { - "action": "Use logical CellHistory at registered command and signed-effect identity boundaries. Dispatch namespace-authorized classes on the existing shared SQL workers; validate operation, schema, input and authenticated inbox identity before receipt lookup. Preserve exact proposal submission/settlement and cancellation ownership. Group placement is absent from effect identity; fresh cell incarnations remain fenced. Marker-1 old group-bearing effect wire and old signature domains refuse. Registered query/version and catalog dispatch remain separate integration work.", + "action": "Use logical CellHistory at registered command and signed-effect identity boundaries. Dispatch namespace-authorized classes on the existing shared SQL workers; validate operation, schema, input and authenticated inbox identity before receipt lookup. Preserve exact proposal submission/settlement and cancellation ownership. Group placement is absent from effect identity; fresh cell incarnations remain fenced. Marker-1 old group-bearing effect wire and old signature domains refuse. Catalog/SDK dispatch remains separate integration work.", "tests": [ "registered_commands_use_cell_identity_and_receipts", "rooted_registered_commands_use_cell_identity_and_receipts" ] + }, + { + "action": "Dispatch compiled queries through the existing coherent query worker and callback guard with the actual CellVersionObservation, bounded operation/input admission and original read deadline. Keep legacy group cuts explicitly typed during integration; never fabricate a cell sequence or witness. Remove the temporary group-cut observation and old constructor with CG-16; final serving uses the common cell-group path.", + "tests": [ + "registered_queries_keep_cell_observation_and_ownership", + "rooted_registered_queries_keep_cell_observation_and_ownership" + ] } ], "test_cases": [ @@ -778,6 +786,24 @@ "I03", "I12" ] + }, + { + "name": "registered_queries_keep_cell_observation_and_ownership", + "oracle": "Run a registered read-only callback on the existing worker with a real selected cell observation. An older A pin remains readable after A and B mutate; a fresh linearizable A query returns A's sequence and decision witness, not B's group progress, and round-trips its signed token. Invalid contract/kind/operation/schema/input refuses before callback execution. Panic and deadline interruption leave the worker usable. Cancelled query/close retain input, image and physical callback ownership until joined cleanup.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_registered_queries_keep_cell_observation_and_ownership", + "oracle": "Run a registered read-only callback on the existing worker with a real selected cell observation. An older A pin remains readable after A and B mutate; a fresh linearizable A query returns A's sequence and decision witness, not B's group progress, and round-trips its signed token. Invalid contract/kind/operation/schema/input refuses before callback execution. Panic and deadline interruption leave the worker usable. Cancelled query/close retain input, image and physical callback ownership until joined cleanup.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ @@ -2209,7 +2235,7 @@ "test_target": "native_cell_groups_release", "steps": [ { - "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code.", + "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code. Remove QueryVersion::GroupCut and RaftClass::invoke_group_query along with the old query adapters; final QueryContext exposes the selected cell observation directly.", "tests": [ "no_legacy_per_cell_consensus_path" ] @@ -2236,7 +2262,7 @@ "test_cases": [ { "name": "no_legacy_per_cell_consensus_path", - "oracle": "Public configuration cannot select a legacy one-cell host; physical group counts follow grouping, including occupancy one.", + "oracle": "Public configuration cannot select a legacy one-cell host; physical group counts follow grouping, including occupancy one. No temporary GroupCut query observation or invoke_group_query constructor remains; registered queries expose the actual selected cell version and witness.", "invariants": [ "I01", "I14" From fa609dcfb19f4823c2019258aa92d9c64061e985 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 02:40:12 -0700 Subject: [PATCH 078/568] docs(raft): record registered query qualification and next CG gap --- docs/design/multi-raft/cell-groups/README.md | 15 +++-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 59 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 52 ++++++++++++++++ 4 files changed, 98 insertions(+), 32 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 49566001a..65f33fef8 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -125,14 +125,21 @@ Whole-process kill and power-loss cuts remain unqualified. See the newest Registered command callbacks now run on shared SQL workers at `b28078af` with logical cell identity, pre-receipt validation and exact receipt settlement. Command/effect identities exclude the physical group; old signed effect layouts -refuse under the marker-1 cutover. Shared query context, effect scheduling and -production catalog/SDK dispatch remain incomplete. All 295 selected Rust cases +refuse under the marker-1 cutover. At that checkpoint shared query context, effect scheduling and +production catalog/SDK dispatch were incomplete. All 295 selected Rust cases and seventeen commands pass; transfer `.cellar/cell-groups-evidence/CG-04-registered/` with the previous evidence. Its checkpoint binds 221 source inputs and fifteen selected executables. See the newest progress record for scope and retained failures. -Registered SDK/catalog routing, query/version-token integration, -production retry/compaction integration and cache lifecycle remain required. Managed +Registered shared query callbacks now receive exact logical cell observations at +`7a79511d`, preserving selected witnesses, read-only SQL and retained physical +ownership through cancellation and joined shutdown. The old adapters expose a +distinct temporary group cut that CG-16 must delete. See [progress](progress.md) +for this checkpoint's qualification: all 319 selected Rust cases and seventeen +commands pass. Transfer `.cellar/cell-groups-evidence/CG-04-registered-queries/` +with prior evidence; its checkpoint binds 225 source inputs and sixteen selected +executables. Registered SDK/catalog routing, production +retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 73d22fd31..0b8e980b4 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "b28078afbd243c18ac7c671a481a3e369c086ab2", + "source_revision": "7a79511d4b8606d11177778a3fe8adf71a8d901a", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 7cfdbb8cf..a676673cf 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,15 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `b28078af`, adding registered commands on shared +The reviewed source baseline is `7a79511d`, adding registered queries with exact +cell observations on the existing workers. Its partial checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered-queries/checkpoint.json`: 319 selected +Rust cases, seventeen successful commands, 225 source inputs and sixteen selected +executables. Transfer its entire directory with prior evidence. Catalog/SDK +integration remains the next critical gap; CG-04 remains IN_PROGRESS. The +explicit temporary `QueryVersion::GroupCut` and `invoke_group_query` must be +removed with the old host by CG-16. +The preceding baseline is `b28078af`, adding registered commands on shared SQL workers and logical command/effect identities. Its partial checkpoint is `.cellar/cell-groups-evidence/CG-04-registered/checkpoint.json`: 295 selected Rust cases, seventeen successful commands, 221 source inputs and fifteen selected @@ -134,8 +142,8 @@ the old proof. Accepted preparations retain their original position/digest even after apply; this remains provisional acceptance, never commitment. Prepared submission also closes byte-only proposal entry points on that owner. -Next connect catalog routing and SDK creation/commands to the registered worker -entry point, and integrate registered query/version-token serving. Use `CellSqlPrepared::submit` and its typed result; +Next connect catalog routing and SDK creation/commands/queries to the registered +worker entry points and their exact cell-version observations. Use `CellSqlPrepared::submit` and its typed result; never release a producer from generic overload or same-term absence. Preserve the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to satisfy callback/effect identities. The arbitrary-callback `stage` API remains a verification boundary; use @@ -235,8 +243,8 @@ owning host. Reuse the old executor's current-term applied-barrier rule when replacing it: a submitted producer cannot be discarded from a sampled leader status, timeout or missing response. Resolve the original command against the authoritative cell receipt before admitting another producer. These regressions supplement, and do not replace, the full task -gate printed by `show CG-04`. Registered SDK dispatch and registered strong-read serving -remain absent; the low-level read-authority and immutable-query paths exist. +gate printed by `show CG-04`. Registered SDK dispatch remains absent; registered query callbacks now consume +the existing read-authority and immutable-query paths. The selected version's complete commit witness now persists in the authenticated cell leaf and flows through query and receipt observations. The source-bound gate @@ -244,8 +252,8 @@ at `ed84df5d` qualifies exact entry matching, unchanged-cell and older-pin provenance, restart/image-collection recovery and token round-trips in both image modes. Actual log-compaction and group-snapshot integration remain unqualified. -Next integrate registered query callbacks and connect catalog invocation sessions -to the registered command workers without synthesizing legacy group identities. Preserve original invocation deadlines, +Next connect catalog invocation sessions to registered command and query workers +without synthesizing legacy group identities. Preserve original invocation deadlines, contracts and retry identities. Query callbacks must receive logical cell versions; returned tokens must use the selected observation. `CellGroupApplier` now consumes the startup manifest-page owner, and `CellGroupSqlPool::prepare_apply` prepares committed cell images and complete @@ -342,26 +350,25 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Integrate registered query contexts with logical cell versions and selected - decision witnesses. `QueryContext` still exposes the old `CommittedVersion`; - the shared read path already provides `CellVersionObservation`. Preserve - read-only callback restrictions and version-token authentication. Do not map - group log progress to a cell mutation sequence. The remaining old constructors - are `execution/mod.rs::query_registered_controlled` and - `rooted/query.rs::query_registered_rooted_controlled`; both construct only a - group-position `CommittedVersion`. The new worker lives in - `host_sql/cell_groups/query.rs` and already retains the exact observation. -2. Connect catalog invocation sessions and the production SDK dispatcher to - `CellGroupSqlPool::stage_registered`, initialization, coherent queries and exact - receipt settlement. Select classes from authorized namespace bindings, and +1. Connect catalog invocation sessions and the production SDK dispatcher to + `CellGroupSqlPool::stage_registered`, `query_registered`, initialization and + exact receipt settlement. Select classes from authorized namespace bindings; preserve original deadlines, request fingerprints and total retry budgets. - Do not manufacture legacy group scopes or create another worker pool. -3. Complete CG-04's production retry/cache lifecycle and configuration-entry - handling, then its full gate before DONE. Use the selected-root recovery API - at startup. Whole-process kill/power-loss remains distinct from the qualified - directory-sync faults and joined cold restart. Full snapshots/membership follow - in CG-05, archive/reclamation in CG-06, and shared effect scheduling/cache - lifecycle in CG-08 before the CG-07 public cutover. + Start with one actual catalog-routed invocation spanning enrollment, a command + and a linearizable query in the real three-voter fixture. Exercise a lost + response and catalog-confirmed redirect without replenishing the deadline or + attempt budget. Retain physical command producers after waiter cancellation. +2. Complete CG-04's production retry/cache lifecycle and configuration-entry + handling, then its full gate before DONE. Use selected-root recovery at startup. + Whole-process kill/power-loss remains distinct from the qualified directory-sync + faults and joined cold restart. Full snapshots/membership follow in CG-05, + archive/reclamation in CG-06, and shared effect scheduling/cache lifecycle in + CG-08 before the CG-07 public cutover. +3. Remove `QueryVersion::GroupCut` and `RaftClass::invoke_group_query` with the old + host by CG-16. They explicitly preserve the old in-memory group observation + during integration; they must not become a permanent alternate serving mode. + Final `QueryContext` exposes the actual cell observation directly. Never derive + a cell sequence or witness from a group log position. The opportunity is to reuse the now-shared consensus, SQL, read and apply paths in one dispatcher. Additional per-cell workers, a second production mode or a demo diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 98509a32e..38b89367b 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 registered queries with exact cell observations + +Qualified source: `7a79511d4b8606d11177778a3fe8adf71a8d901a`. + +`CellGroupSqlPool::query_registered` dispatches compiled queries through the +existing coherent query owner and fixed SQL workers. It validates operation +kind/contract and full schema before opening the selected image, bounds retained +operation/input storage, and preserves the original authority deadline. The +callback receives the exact `CellHistory` and `CellVersionObservation` returned +in its result, including the selected cell's decision witness. It never derives +a mutation sequence from group log progress. + +The two new three-voter cases exercise materialized and extent images. An older +A pin returns its historical state after A and B mutate; a fresh linearizable A +query returns A's sequence and witness independently of B's later group progress. +Tokens round-trip the selected observation. Invalid contract, kind, operation, +schema and input refuse before callback execution. Read-only restrictions, +contained panics and SQL deadline interruption remain enforced. Cancelled waiters +retain input/image/worker ownership, and shutdown joins entered callbacks before +releasing reservations. + +The temporary old adapters explicitly use `QueryVersion::GroupCut` and +`RaftClass::invoke_group_query`. These preserve the actual old in-memory group cut +without fabricating a cell sequence. CG-16 requires removing both with the old +host; the final context exposes the selected cell observation directly. This is +an integration checkpoint, not a second permanent production mode. + +All 319 selected Rust cases pass: 280 native cases across thirteen suites, +30 Cellar library cases, five pure cell-group core cases and four LTX image-store +cases. The existing native read suite additionally covers current-term barriers, +minority refusal, follower catch-up, delayed replies and version authentication. +Six process helpers run through parent cases; three RustFS-specific cases remain +unrun. These results do not qualify the new shared dispatcher or infrastructure. + +All seventeen commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,819 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The retained checkpoint is +`.cellar/cell-groups-evidence/CG-04-registered-queries/checkpoint.json`, binding +225 source inputs and sixteen selected executables. Pre-commit library, targeted +query, format and Clippy logs are retained separately. The branch contains latest +fetched `origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain +unchanged from main. + +CG-04 remains IN_PROGRESS. The most critical gap is catalog invocation sessions +and production SDK dispatch into registered enrollment, command and query workers, +with namespace-authorized classes, exact receipt settlement and original deadline, +fingerprint and attempt budgets. Next qualify that actual routed path under lost +responses and redirects. Reuse the existing owners instead of adding a worker +pool. Retry/cache and configuration handling, shared effects, complete-group +snapshots/archive, process-kill cuts and real RustFS/load gates remain unfinished. + ## CG-04 registered commands and logical effect identity Qualified source: `b28078afbd243c18ac7c671a481a3e369c086ab2`. From 8723db2df7436478cbc5efc9147d65d3b7651247 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 02:55:41 -0700 Subject: [PATCH 079/568] feat(raft): route registered cell queries through catalog sessions --- crates/cellar/durability/raft/README.md | 31 ++ .../raft/cell_groups/directory/mod.rs | 4 + .../raft/cell_groups/directory/routing.rs | 39 +- .../durability/raft/cell_groups/dispatch.rs | 375 ++++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../raft/cell_groups/runtime/mod.rs | 4 + .../raft/host_sql/cell_groups/mod.rs | 4 + .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../tests/support/cell_group_sql_dispatch.rs | 402 ++++++++++++++++++ .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 29 +- 13 files changed, 904 insertions(+), 7 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch.rs create mode 100644 crates/cellar/tests/support/cell_group_sql_dispatch.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 8e307d28b..3bbfd6a08 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -1810,3 +1810,34 @@ This in-memory staging observation is not a persisted-format reader or a final runtime choice. CG-16 must remove that variant and constructor with the old host, leaving the selected cell observation directly exposed by the final query context. Catalog/SDK dispatch into these workers remains required before public cutover. + +### Catalog-bound query dispatch + +`CellGroupDispatcher` binds one registered namespace directory, compiled class, +version-key set and execution domain. Its clones share a bounded invocation bank; +there are no per-cell worker pools. `query` freezes the operation, input and +consistency before discovery, normalizes retained input capacity and charges both +the retained request and one physical input copy. The canonical fingerprint binds +logical cell identity, invocation ID, contract, input, consistency/token and routing +limits. A non-cloneable `CellQueryInvocation` uses the existing +`CellRoutingSession` for initial lookup, cached routes, retries and confirmed +redirects. The original absolute deadline and total attempt count never reset. + +Each attempt is consumed before asynchronous execution. A caller must explicitly +retry after losing or cancelling a response. The local destination must use the +same execution-domain clock, and its read authority must match the entire +catalog-confirmed route and logical cell. The existing SQL pool checks engine and +schema ownership. AtLeast tokens are verified against the configured keys, exact +cell history, schema and current captured-domain wall time. The response retains +SQL output admission and signs the actual selected observation. Catalog mapping +alone cannot manufacture a leader, group attachment or read authority. + +Cancellation can release the invocation's reservation; entered SQL still owns its +input, image pin, worker and independent admission until physical cleanup joins. +A timeout includes discovery, authority selection, callback execution and output +signing. Cross-host transports must translate a remaining deadline through their +own bounded request protocol; unrelated monotonic timestamps are refused here. +This is the local query dispatch integration boundary. Command/enrollment +invocations, transport selection and replacing the public SDK's old host binding +remain required; it does not complete CG-04 or the public cutover. Group topology +fencing is still owned by the later transition tasks, not by directory metadata. diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index aa918e6c3..efb57a512 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -133,6 +133,10 @@ impl DirectoryPublication { } impl RangeDirectory { + pub(crate) fn namespace_config(&self) -> &crate::raft::catalog::NamespaceConfig { + self.namespace.config() + } + pub fn new(namespace: NamespaceCatalog, limits: DirectoryLimits) -> Result { namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; if !(1..=1_000_000).contains(&limits.max_groups) diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs index f2018c719..f71ae1213 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/routing.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -133,6 +133,25 @@ impl RangeDirectory { request: CellRouteRequest, limits: CellRoutingLimits, cached: Option, + ) -> Result { + let domain = domain.services().domain_token().ok_or(CellRoutingError { + reason: CellRoutingFailure::Closed, + ambiguous_command: None, + })?; + let started_at = domain.mono_ms().map_err(|_| CellRoutingError { + reason: CellRoutingFailure::Closed, + ambiguous_command: None, + })?; + self.routing_session_in(domain, request, limits, cached, started_at) + } + + pub(crate) fn routing_session_in( + &self, + domain: DomainToken, + request: CellRouteRequest, + limits: CellRoutingLimits, + cached: Option, + started_at: u64, ) -> Result { let error = |reason| CellRoutingError { reason, @@ -152,16 +171,18 @@ impl RangeDirectory { { return Err(error(CellRoutingFailure::InvalidRequest)); } - let domain = domain - .services() - .domain_token() - .ok_or_else(|| error(CellRoutingFailure::Closed))?; let now = domain .mono_ms() .map_err(|_| error(CellRoutingFailure::Closed))?; - let deadline = now + if started_at > now { + return Err(error(CellRoutingFailure::InvalidRequest)); + } + let deadline = started_at .checked_add(limits.timeout.as_millis() as u64) .ok_or_else(|| error(CellRoutingFailure::InvalidRequest))?; + if now >= deadline { + return Err(error(CellRoutingFailure::Deadline)); + } Ok(CellRoutingSession { directory: self.clone(), domain, @@ -178,6 +199,14 @@ impl RangeDirectory { } impl CellRoutingSession { + pub(crate) fn validate_attempt( + &self, + attempt: &CellRouteAttempt, + ) -> Result<(), CellRoutingError> { + self.check_attempt(attempt)?; + self.live() + } + fn error(&self, reason: CellRoutingFailure) -> CellRoutingError { CellRoutingError { reason, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch.rs b/crates/cellar/durability/raft/cell_groups/dispatch.rs new file mode 100644 index 000000000..4926972fa --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch.rs @@ -0,0 +1,375 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-bound invocations over the existing consensus and SQL owners. +//! Routing observations never replace destination authority or ingress grants. + +use super::*; +use crate::application::consensus::{ + OperationContract, OperationKind, RaftClass, RegistrationError, +}; +use crate::asyncrt::{DomainToken, ExecutionDomain}; +use crate::raft::budget::{Budget, Permit}; +use crate::raft::completion::RuntimeError; +use crate::raft::execution::ExecutionError; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionError, VersionKeys}; +use cellar_logic::cell_group::{CellHistory, GroupRoute}; +use cellar_logic::consensus::ProposalId; +use sha2::{Digest, Sha256}; +use std::{fmt, sync::Arc}; + +#[derive(Clone, Copy)] +pub struct CellDispatchLimits { + pub max_invocations: usize, + pub retained_bytes: usize, + pub max_input_bytes: usize, +} + +#[derive(Clone, Debug)] +pub enum CellQueryConsistency { + Stale, + AtLeast(CellVersionToken), + Linearizable, +} + +pub struct CellQueryRequest { + pub invocation: ProposalId, + pub contract: OperationContract, + pub input: Vec, + pub consistency: CellQueryConsistency, +} + +#[derive(Debug)] +pub enum CellDispatchError { + Invalid(&'static str), + Routing(CellRoutingError), + Registration(RegistrationError), + Runtime(RuntimeError), + Execution(ExecutionError), + Version(VersionError), +} + +impl fmt::Display for CellDispatchError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell dispatch: {self:?}") + } +} + +impl std::error::Error for CellDispatchError {} + +/// One authorized namespace/class registration. Clones share invocation admission; +/// no cell owns a worker, timer loop or independent admission bank. +#[derive(Clone)] +pub struct CellGroupDispatcher { + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + domain: DomainToken, + limits: CellDispatchLimits, + budget: Arc, +} + +impl CellGroupDispatcher { + pub fn new( + domain: &ExecutionDomain, + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + limits: CellDispatchLimits, + ) -> Result { + if class.schema_digest() != directory.namespace_config().schema_digest { + return Err(CellDispatchError::Registration( + RegistrationError::ContractMismatch, + )); + } + if !(1..=65_536).contains(&limits.max_invocations) + || limits.max_input_bytes == 0 + || limits + .max_input_bytes + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .is_none_or(|n| n > limits.retained_bytes) + { + return Err(CellDispatchError::Invalid("invocation limits")); + } + let domain = domain.services().domain_token().ok_or_else(closed)?; + Ok(Self { + directory, + class, + keys, + domain, + limits, + budget: Budget::new(limits.retained_bytes, limits.max_invocations), + }) + } + + pub fn usage(&self) -> (usize, usize) { + self.budget.usage() + } + + /// Freeze the complete request before discovery. Input copies for a physical + /// attempt are included in this reservation; SQL additionally retains its + /// own admission if the invocation waiter or router is dropped. + pub fn query( + &self, + cell: CellHistory, + mut request: CellQueryRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + let started_at = self.domain.mono_ms().map_err(|_| closed())?; + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(CellDispatchError::Runtime(RuntimeError::Overloaded)); + } + if request.contract.kind != OperationKind::Query { + return Err(CellDispatchError::Registration( + RegistrationError::KindMismatch, + )); + } + self.class + .validate_operation(&request.contract) + .map_err(CellDispatchError::Registration)?; + let bytes = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let admission = self + .budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + request.input = request.input.into_boxed_slice().into_vec(); + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Query(request.invocation), + fingerprint: fingerprint(&cell, &request, limits), + }, + limits, + cached, + started_at, + ) + .map_err(CellDispatchError::Routing)?; + Ok(CellQueryInvocation { + dispatcher: self.clone(), + request, + session, + attempt: None, + dispatched: false, + _admission: admission, + }) + } +} + +/// A non-cloneable invocation. Each ticket is consumed before entering async +/// destination work; retries and redirects share one original deadline/budget. +pub struct CellQueryInvocation { + dispatcher: CellGroupDispatcher, + request: CellQueryRequest, + session: CellRoutingSession, + attempt: Option, + dispatched: bool, + _admission: Permit, +} + +impl CellQueryInvocation { + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub async fn begin(&mut self) -> Result<(), CellDispatchError> { + self.dispatched = true; + let attempt = self + .session + .begin() + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellDispatchError> { + let previous = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.dispatched = true; + let attempt = self + .session + .retry(previous) + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellDispatchError> { + let previous = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.dispatched = true; + let attempt = self + .session + .redirect(previous, hint) + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + /// Execute on a local destination with the same captured clock owner. Remote + /// transports must preserve the remaining deadline through their own protocol; + /// monotonic timestamps from unrelated hosts are never interchangeable. + pub async fn execute( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed route attempt")); + } + self.dispatched = true; + let domain = &self.dispatcher.domain; + if !runtime.uses_domain(domain) || !sql.uses_domain(domain) { + return Err(CellDispatchError::Invalid("foreign execution domain")); + } + let timer = domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| closed())?; + crate::asyncrt::select! { + result = self.execute_at(attempt, runtime, sql, options) => { + let result = result?; + self.session.validate_attempt(attempt).map_err(CellDispatchError::Routing)?; + Ok(result) + } + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + } + } + + async fn execute_at( + &self, + attempt: &CellRouteAttempt, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let cell = &attempt.request().cell; + let schema = self.dispatcher.class.schema_digest(); + let consistency = match &self.request.consistency { + CellQueryConsistency::Stale => CellReadConsistency::Stale, + CellQueryConsistency::Linearizable => CellReadConsistency::Linearizable, + CellQueryConsistency::AtLeast(token) => { + let observation = self + .dispatcher + .keys + .verify_cell(token, cell, schema, self.now().await?) + .map_err(CellDispatchError::Version)?; + CellReadConsistency::AtLeast(observation.version) + } + }; + let authority = runtime + .read_authority( + attempt.route().history, + cell.clone(), + consistency, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() || authority.cell().version.history != *cell { + return Err(CellDispatchError::Invalid("destination route binding")); + } + let selected = sql + .query_registered( + options, + authority, + self.dispatcher.class.clone(), + self.request.contract.clone(), + self.request.input.clone(), + ) + .await + .map_err(CellDispatchError::Execution)?; + let token = self + .dispatcher + .keys + .issue_cell(selected.observation(), schema, self.now().await?) + .map_err(CellDispatchError::Version)?; + Ok(CellQueryResponse { selected, token }) + } + + async fn now(&self) -> Result { + self.dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed()) + } +} + +pub struct CellQueryResponse { + selected: CellSqlQueryResult, + token: CellVersionToken, +} + +impl CellQueryResponse { + pub fn result(&self) -> &[u8] { + self.selected.result() + } + + pub fn observation(&self) -> &CellVersionObservation { + self.selected.observation() + } + + pub fn token(&self) -> &CellVersionToken { + &self.token + } +} + +fn closed() -> CellDispatchError { + CellDispatchError::Runtime(RuntimeError::Closed) +} + +fn fingerprint( + cell: &CellHistory, + request: &CellQueryRequest, + limits: CellRoutingLimits, +) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-query-invocation-1\0"); + hash.update(cell_image_history_digest(cell)); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([limits.max_attempts]); + hash.update(request.invocation.bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update((request.input.len() as u64).to_be_bytes()); + hash.update(&request.input); + match &request.consistency { + CellQueryConsistency::Stale => hash.update([0]), + CellQueryConsistency::Linearizable => hash.update([1]), + CellQueryConsistency::AtLeast(token) => { + hash.update([2]); + hash.update(token.as_str()); + } + } + hash.finalize().into() +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e29e457ab..d53d24da0 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -10,7 +10,12 @@ pub use super::host_sql::cell_groups::{ }; mod apply; +mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; +pub use dispatch::{ + CellDispatchError, CellDispatchLimits, CellGroupDispatcher, CellQueryConsistency, + CellQueryInvocation, CellQueryRequest, CellQueryResponse, +}; mod execution_authority; pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index b25f8e005..735942999 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -127,6 +127,10 @@ struct Request { } impl CellGroupRuntime { + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { + self.inner.domain.same_owner(domain) + } + pub async fn submit( &self, prepared: crate::raft::execution::CellPreparedCommand, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index b70c85d71..445416d7d 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -297,6 +297,10 @@ fn overloaded() -> ExecutionError { } impl CellGroupSqlPool { + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { + self.inner.workers[0].domain.same_owner(domain) + } + pub fn start(log: &ReplicaLog, limits: CellSqlLimits) -> Result { limits.validate()?; log.cell_groups().map_err(crate::raft::execution::storage)?; diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 7e782615f..01021e126 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1320,3 +1320,13 @@ async fn registered_queries_keep_cell_observation_and_ownership() -> Result<()> async fn rooted_registered_queries_keep_cell_observation_and_ownership() -> Result<()> { Box::pin(runtime_fixture::registered_queries(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_routed_queries_preserve_invocation_and_authority() -> Result<()> { + Box::pin(runtime_fixture::routed_queries(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_routed_queries_preserve_invocation_and_authority() -> Result<()> { + Box::pin(runtime_fixture::routed_queries(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 04602bfde..01561f4fc 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -669,3 +669,5 @@ pub(super) use sql::recover_apply; pub(super) use sql::registered_commands; pub(super) use sql::registered_queries; + +pub(super) use sql::routed_queries; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c79cacf20..f0d15f23c 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -517,3 +517,7 @@ pub(crate) use registered::commands as registered_commands; #[path = "cell_group_sql_registered_queries.rs"] mod registered_queries; pub(crate) use registered_queries::queries as registered_queries; + +#[path = "cell_group_sql_dispatch.rs"] +mod dispatch; +pub(crate) use dispatch::routed as routed_queries; diff --git a/crates/cellar/tests/support/cell_group_sql_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_dispatch.rs new file mode 100644 index 000000000..5191c8732 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_dispatch.rs @@ -0,0 +1,402 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-selected invocations use real Raft reads and registered SQL workers. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + } +} + +fn request(class: &RaftClass, consistency: CellQueryConsistency) -> Result { + Ok(CellQueryRequest { + invocation: ProposalId::new([201; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![7], + consistency, + }) +} + +async fn execute( + fixture: &mut RuntimeCluster, + query: &mut CellQueryInvocation, + cell: &CellHistory, + node: usize, +) -> Result { + let runtime = fixture.runtimes[node].clone(); + let pool = fixture.appliers[&(node, fixture.cluster.birth.route().history)] + .0 + .clone(); + let options = fixture.cell_options_at(cell, node).await?; + let work = query.execute(&runtime, &pool, options); + tokio::pin!(work); + loop { + cellar::asyncrt::select! { + result = &mut work => return result.map_err(Into::into), + result = async { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + Ok::<_, anyhow::Error>(()) + } => result?, + } + } +} + +pub(crate) async fn routed(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, waiting) = std::sync::mpsc::sync_channel(1); + let waiting = Mutex::new(waiting); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("group cut") + }; + assert_eq!(context.identity(), &observation.version.history); + if input.starts_with(b"hold") { + entered + .lock() + .unwrap() + .take() + .context("one held query")? + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + waiting + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(10))?; + } + let n: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let keys = VersionKeys::new( + VersionSigningKey::new([83; 32], u64::MAX)?, + vec![], + Duration::from_secs(10), + )?; + let bounds = CellDispatchLimits { + max_invocations: 1, + retained_bytes: 1024 * 1024, + max_input_bytes: 256 * 1024, + }; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let mut query = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Linearizable)?, + limits(), + None, + )?; + assert!(matches!( + dispatcher.clone().query( + b.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + None + ), + Err(CellDispatchError::Runtime(RuntimeError::Overloaded)) + )); + query.begin().await?; + let first = query.attempt().context("attempt")?.clone(); + assert_eq!(first.route(), &route); + let response = execute(&mut fixture, &mut query, &a, 0).await?; + assert_eq!(response.result(), &0_i64.to_le_bytes()); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys.verify_cell(response.token(), &a, route.schema_digest, now)?, + *response.observation() + ); + let token = response.token().clone(); + let expected = response.observation().clone(); + drop(response); // Lost response: retry the same immutable query and budget. + let pool = fixture.appliers[&(0, route.history)].0.clone(); + assert!(matches!( + query + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Invalid("consumed route attempt")) + )); + query.retry().await?; + let second = query.attempt().context("second")?; + assert_eq!(second.request(), first.request()); + assert_eq!(second.deadline_ms(), first.deadline_ms()); + assert_eq!(second.ordinal(), 2); + let repeated = execute(&mut fixture, &mut query, &a, 0).await?; + assert_eq!(repeated.observation(), &expected); + drop(repeated); + assert!(matches!( + query.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::AttemptsExhausted, + .. + })) + )); + drop(query); + assert_eq!(dispatcher.usage(), (0, 0)); + + let mut at_least = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::AtLeast(token.clone()))?, + limits(), + Some(first.cached_route()), + )?; + at_least.begin().await?; + let output = execute(&mut fixture, &mut at_least, &a, 1).await?; + assert_eq!(output.observation(), &expected); + drop(output); + drop(at_least); + let before = calls.load(Ordering::SeqCst); + let mut wrong = dispatcher.query( + b.clone(), + request(&class, CellQueryConsistency::AtLeast(token))?, + limits(), + None, + )?; + wrong.begin().await?; + assert!(matches!( + wrong + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&b).await?) + .await, + Err(CellDispatchError::Version( + cellar::application::consensus::VersionError::WrongHistory + )) + )); + drop(wrong); + assert_eq!(calls.load(Ordering::SeqCst), before); + + let mut wrong_engine = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + wrong_engine.begin().await?; + let other_pool = fixture.appliers[&(1, route.history)].0.clone(); + assert!(matches!( + wrong_engine + .execute( + &fixture.runtimes[0], + &other_pool, + fixture.cell_options(&a).await? + ) + .await, + Err(CellDispatchError::Execution(ExecutionError::Invalid( + "cell query engine/schema" + ))) + )); + drop(wrong_engine); + drop(other_pool); + let mut invalid_hint = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + invalid_hint.begin().await?; + let mut hint = route.clone(); + hint.epoch = RouteEpoch::new(2)?; + hint.schema_digest = [86; 32]; + assert!(matches!( + invalid_hint.redirect(&hint).await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::InvalidHint, + .. + })) + )); + assert!(matches!( + invalid_hint + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::StaleAttempt, + .. + })) + )); + drop(invalid_hint); + assert_eq!(calls.load(Ordering::SeqCst), before); + + let foreign_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let foreign = CellGroupDispatcher::new( + &foreign_domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let mut wrong = foreign.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + None, + )?; + wrong.begin().await?; + assert!(matches!( + wrong + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Invalid("foreign execution domain")) + )); + drop(wrong); + drop(foreign); + foreign_domain.shutdown().await; + let bad_class = Arc::new(RaftClass::new("Counter", 1, [84; 32])?); + assert!(matches!( + CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + bad_class, + keys, + bounds + ), + Err(CellDispatchError::Registration( + RegistrationError::ContractMismatch + )) + )); + + let mut expired = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + CellRoutingLimits { + timeout: Duration::from_millis(10), + max_attempts: 2, + }, + Some(first.cached_route()), + )?; + expired.begin().await?; + let original_deadline = expired.attempt().context("deadline")?.deadline_ms(); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(matches!( + expired.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + })) + )); + assert_eq!( + expired.attempt().context("same deadline")?.deadline_ms(), + original_deadline + ); + drop(expired); + assert_eq!(calls.load(Ordering::SeqCst), before); + + // Confirm a replacement against the real directory, but do not invent its + // physical owner or claim this metadata-only fixture performs a live move. + let directory = fixture.cluster.directory.clone(); + let replacement = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([203; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([204; 16])?, + voters: fixture.cluster.birth.voters().to_vec(), + learners: vec![], + })?; + fixture + .cluster + .domain + .run(directory.reserve_group_birth(&replacement)) + .await??; + let mut relocated = dispatcher.query( + a.clone(), + request(&class, CellQueryConsistency::Stale)?, + limits(), + Some(first.cached_route()), + )?; + relocated.begin().await?; + let original = relocated.attempt().context("original")?.clone(); + let view = fixture.cluster.domain.run(directory.observe()).await??; + let publish = directory.prepare_replacement( + &view, + ProposalId::new([205; 16])?, + std::slice::from_ref(&route), + &[replacement.route().clone()], + )?; + fixture + .cluster + .domain + .run(directory.publish(&publish)) + .await??; + relocated.redirect(replacement.route()).await?; + assert_eq!( + relocated.attempt().context("redirect")?.deadline_ms(), + original.deadline_ms() + ); + assert_eq!( + relocated.attempt().context("redirect")?.request(), + original.request() + ); + assert_eq!( + relocated.attempt().context("redirect")?.route(), + replacement.route() + ); + assert!(matches!( + relocated + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Runtime(RuntimeError::NotFound)) + )); + drop(relocated); + assert_eq!(calls.load(Ordering::SeqCst), before); + + // A known route needs no new catalog lookup; actual destination authority is + // still mandatory. Here the old physical owner has not been transitioned. + let mut held_request = request(&class, CellQueryConsistency::Stale)?; + held_request.input = vec![0; 256 * 1024]; + held_request.input[..4].copy_from_slice(b"hold"); + let mut held = dispatcher.query( + a.clone(), + held_request, + limits(), + Some(first.cached_route()), + )?; + held.begin().await?; + assert!(dispatcher.usage().0 >= 512 * 1024); + let runtime = fixture.runtimes[0].clone(); + let sql = pool.clone(); + let options = fixture.cell_options(&a).await?; + let waiter = fixture + .cluster + .domain + .spawn(async move { held.execute(&runtime, &sql, options).await })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + waiter.abort(); + assert!(waiter.await.is_err()); + assert_eq!(dispatcher.usage(), (0, 0)); + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(pool.usage()?.retained_bytes >= 256 * 1024); + assert!( + tokio::time::timeout(Duration::from_millis(30), pool.close()) + .await + .is_err() + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), pool.close()).await??; + assert_eq!(pool.usage()?.retained_bytes, 0); + drop(pool); + drop(dispatcher); + fixture.close().await +} diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 4509d86cd..309d538aa 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 27 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 29 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index f56eef7ab..b4b86239b 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -483,7 +483,9 @@ "crates/cellar/durability/raft/execution/registered.rs", "crates/cellar/durability/raft/effect_history.rs", "crates/cellar/tests/support/cell_group_sql_registered.rs", - "crates/cellar/tests/support/cell_group_sql_registered_queries.rs" + "crates/cellar/tests/support/cell_group_sql_registered_queries.rs", + "crates/cellar/durability/raft/cell_groups/dispatch.rs", + "crates/cellar/tests/support/cell_group_sql_dispatch.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -553,6 +555,13 @@ "registered_queries_keep_cell_observation_and_ownership", "rooted_registered_queries_keep_cell_observation_and_ownership" ] + }, + { + "action": "Connect catalog routing sessions to compiled query execution with a namespace-bound class and captured input/consistency. Share bounded invocation admission across dispatcher clones, preserve the original domain deadline and total retry budget, consume each attempt before entering asynchronous work, authenticate AtLeast cell tokens and return a token for the selected observation. Exact destination route and engine checks remain mandatory; a confirmed catalog redirect cannot invent a physical owner. Dropping the routing invocation must leave entered SQL owned until joined cleanup. Command/enrollment dispatch and final public SDK replacement remain required.", + "tests": [ + "catalog_routed_queries_preserve_invocation_and_authority", + "rooted_catalog_routed_queries_preserve_invocation_and_authority" + ] } ], "test_cases": [ @@ -804,6 +813,24 @@ "I05", "I12" ] + }, + { + "name": "catalog_routed_queries_preserve_invocation_and_authority", + "oracle": "Resolve a real catalog route and execute a registered linearizable query on three voters. A dropped response retries the same fingerprint/deadline, an authenticated AtLeast token reads on a follower, and a foreign cell token/domain/class refuses before callback execution. Invocation admission is shared across clones and total attempts/deadlines cannot reset. A catalog-confirmed replacement preserves request identity but refuses execution without its physical owner. Cancelled routing releases only its own reservation; held SQL remains charged and joined.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "rooted_catalog_routed_queries_preserve_invocation_and_authority", + "oracle": "Resolve a real catalog route and execute a registered linearizable query on three voters. A dropped response retries the same fingerprint/deadline, an authenticated AtLeast token reads on a follower, and a foreign cell token/domain/class refuses before callback execution. Invocation admission is shared across clones and total attempts/deadlines cannot reset. A catalog-confirmed replacement preserves request identity but refuses execution without its physical owner. Cancelled routing releases only its own reservation; held SQL remains charged and joined.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ From 9f48a660e215ea5135d55a391db2586725ceac96 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:04:33 -0700 Subject: [PATCH 080/568] docs(raft): qualify catalog-routed queries and prioritize command dispatch --- docs/design/multi-raft/cell-groups/README.md | 10 ++++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 48 +++++++++++++------ .../design/multi-raft/cell-groups/progress.md | 48 +++++++++++++++++++ 4 files changed, 93 insertions(+), 17 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 65f33fef8..cdcd82074 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -143,6 +143,16 @@ retry/compaction integration and cache lifecycle remain required. Managed fleet integration remains required by CG-07. A birth metadata observation alone cannot authorize voter startup. +Local query invocations now connect the catalog to registered execution at +`8723db2d`: namespace/class bindings, immutable request fingerprints, original +deadlines/attempt budgets, authenticated AtLeast tokens and destination authority +checks share existing workers. All 103 native cases across four affected suites +and eight commands pass. Transfer +`.cellar/cell-groups-evidence/CG-04-routed-queries/` with prior evidence; it binds +227 source inputs and four selected executables. Command/enrollment routing and +production controller/SDK integration are the next critical gaps. Metadata-only +redirect tests do not qualify live transfer. See [progress](progress.md). + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 0b8e980b4..d7bc5174c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "7a79511d4b8606d11177778a3fe8adf71a8d901a", + "source_revision": "8723db2df7436478cbc5efc9147d65d3b7651247", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index a676673cf..a3e2afd47 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,12 +15,21 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `7a79511d`, adding registered queries with exact +The reviewed source baseline is `8723db2d`, adding catalog-bound query +invocations over the existing runtime/SQL owners. Its partial record is +`.cellar/cell-groups-evidence/CG-04-routed-queries/checkpoint.json`: 103 native +cases across four affected suites, eight successful commands, 227 source inputs +and four selected executables. Transfer the complete directory with prior evidence. +This qualifies local routed queries, including authenticated cell tokens, +unchanged deadlines/attempt budgets and cancellation ownership. It does not +qualify live movement, command/enrollment routing, remote transports or the public +SDK/controller cutover. CG-04 remains IN_PROGRESS. +The preceding source baseline is `7a79511d`, adding registered queries with exact cell observations on the existing workers. Its partial checkpoint is `.cellar/cell-groups-evidence/CG-04-registered-queries/checkpoint.json`: 319 selected Rust cases, seventeen successful commands, 225 source inputs and sixteen selected -executables. Transfer its entire directory with prior evidence. Catalog/SDK -integration remains the next critical gap; CG-04 remains IN_PROGRESS. The +executables. Transfer its entire directory with prior evidence. That checkpoint +preceded the local catalog-routed query integration; CG-04 remains IN_PROGRESS. The explicit temporary `QueryVersion::GroupCut` and `invoke_group_query` must be removed with the old host by CG-16. The preceding baseline is `b28078af`, adding registered commands on shared @@ -142,8 +151,9 @@ the old proof. Accepted preparations retain their original position/digest even after apply; this remains provisional acceptance, never commitment. Prepared submission also closes byte-only proposal entry points on that owner. -Next connect catalog routing and SDK creation/commands/queries to the registered -worker entry points and their exact cell-version observations. Use `CellSqlPrepared::submit` and its typed result; +Next connect catalog routing for SDK creation/commands and the production host +controller to the registered workers. Local catalog-routed query invocations now +consume exact cell-version observations. Use `CellSqlPrepared::submit` and its typed result; never release a producer from generic overload or same-term absence. Preserve the original request fingerprint, deadline and attempt budget across redirects. Do not synthesize a legacy `GroupScope` to satisfy callback/effect identities. The arbitrary-callback `stage` API remains a verification boundary; use @@ -252,8 +262,9 @@ at `ed84df5d` qualifies exact entry matching, unchanged-cell and older-pin provenance, restart/image-collection recovery and token round-trips in both image modes. Actual log-compaction and group-snapshot integration remain unqualified. -Next connect catalog invocation sessions to registered command and query workers -without synthesizing legacy group identities. Preserve original invocation deadlines, +Local query invocations now connect catalog sessions to registered query workers. +Next connect command/enrollment routing and the public host without synthesizing +legacy group identities. Preserve original invocation deadlines, contracts and retry identities. Query callbacks must receive logical cell versions; returned tokens must use the selected observation. `CellGroupApplier` now consumes the startup manifest-page owner, and `CellGroupSqlPool::prepare_apply` prepares committed cell images and complete @@ -350,14 +361,21 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Connect catalog invocation sessions and the production SDK dispatcher to - `CellGroupSqlPool::stage_registered`, `query_registered`, initialization and - exact receipt settlement. Select classes from authorized namespace bindings; - preserve original deadlines, request fingerprints and total retry budgets. - Start with one actual catalog-routed invocation spanning enrollment, a command - and a linearizable query in the real three-voter fixture. Exercise a lost - response and catalog-confirmed redirect without replenishing the deadline or - attempt budget. Retain physical command producers after waiter cancellation. +1. Extend `CellGroupDispatcher` from its completed local query boundary to + registered command/enrollment invocations. Capture namespace minimum durability, + original command/creation identity, input fingerprint, deadline and total attempt + budget. Use existing prepared submission, exact rejection proofs and receipt or + creation reconciliation; neither a lost response nor a catalog redirect proves + an old producer uncommitted. Accepted proposals are provisional: return a durable + result only after exact publication/receipt settlement and the requested Quorum + or S3 proof. Do not weaken a namespace's minimum durability while archive + integration is pending. Qualify actual catalog-routed enrollment, command, + receipt retry and linearizable query together. The query path now preserves + these routing limits and authenticates selected-version tokens in both modes. + Public `RaftCellHandle` still uses its old host binding. Its final replacement + also needs the production node controller to consume shared-runtime transport, + apply and completion events; current real-engine fixtures pump those events. + Reuse the existing transport and worker owners, with joined cancellation. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 38b89367b..fa1c7bf0e 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,54 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 catalog-routed query invocations + +Qualified source: `8723db2df7436478cbc5efc9147d65d3b7651247`. + +`CellGroupDispatcher` binds a registered class to its configured namespace, +version keys and execution domain. Queries freeze bounded input, operation and +consistency before discovery. Clones share invocation admission, and the canonical +fingerprint includes logical history and routing limits. Deadline capture precedes +validation and hashing. Existing catalog sessions perform lookup, cached routing, +member retries and exact redirect confirmation without replenishing deadline or +attempt count. Each attempt is consumed before asynchronous destination work. + +Local execution authenticates AtLeast tokens, checks the destination's full route +and logical history, and reuses existing runtime read authority and registered SQL +workers. Foreign domain clocks and engines refuse. The response token signs the +selected cell observation. Dropped routing releases only invocation admission; +entered SQL retains its independent input, image and worker ownership through +joined shutdown. Directory metadata cannot create a physical group owner. + +The two new three-voter cases cover both image modes, real directory lookup, +linearizable execution, a lost response retry, follower AtLeast, foreign +cell/domain/class/engine rejection, invalid redirect ticket invalidation, original +timeout/attempt exhaustion, a confirmed replacement with no physical owner, and +cancelled routing while a callback is held. The replacement case tests only +metadata confirmation and refusal; it does not qualify live group transfer. + +All 103 native cases pass across the shared-group execution, catalog, existing +read and runtime suites. The runtime process helper is invoked by its parent. +This targeted gate complements the prior 319-case registered-query checkpoint at +its original source; it does not rerun the complete previous selection. +All eight commands pass, including all-target Clippy for +Cellar/cellar-logic/cellar-ltx with warnings denied, formatting (1,821 Rust files), +layout, dependency isolation, docs, source preflight and all 37 plan-validator +cases. The partial record is +`.cellar/cell-groups-evidence/CG-04-routed-queries/checkpoint.json`, binding +227 source inputs and four selected executables. The initial missing type +qualification, corrected targeted run and formatter logs are retained separately. +No RustFS or performance qualification was run. The branch contains latest fetched +`origin/main` at `8a93ffd7`; the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The most critical gap is routed command/enrollment +execution with retained ambiguous outcomes and exact receipt/creation settlement. +Next extend the same dispatcher and qualify enrollment, command, lost response and +linearizable query together. Production node-controller event/transport integration +and replacing the public SDK's old host binding remain required; the native +fixture still pumps shared-runtime events. Retry/cache/configuration, shared effects, +snapshots/archive, physical fault and real RustFS/load gates remain unfinished. + ## CG-04 registered queries with exact cell observations Qualified source: `7a79511d4b8606d11177778a3fe8adf71a8d901a`. From f93c690cc70346ae7bdf1cc61aceca0103eb8768 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:29:38 -0700 Subject: [PATCH 081/568] feat(raft): retain routed command producers through exact settlement --- crates/cellar/durability/raft/README.md | 37 + .../raft/cell_groups/dispatch/command.rs | 656 ++++++++++++++++++ .../{dispatch.rs => dispatch/mod.rs} | 6 + .../raft/cell_groups/execution_authority.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 6 +- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../cell_group_sql_command_dispatch.rs | 331 +++++++++ .../tests/support/cell_group_sql_reconcile.rs | 30 +- .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 33 +- 12 files changed, 1111 insertions(+), 10 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/command.rs rename crates/cellar/durability/raft/cell_groups/{dispatch.rs => dispatch/mod.rs} (98%) create mode 100644 crates/cellar/tests/support/cell_group_sql_command_dispatch.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 3bbfd6a08..e5fb8b088 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -1841,3 +1841,40 @@ This is the local query dispatch integration boundary. Command/enrollment invocations, transport selection and replacing the public SDK's old host binding remain required; it does not complete CG-04 or the public cutover. Group topology fencing is still owned by the later transition tasks, not by directory metadata. + +### Catalog-bound command phases + +`CellGroupDispatcher::command` captures a complete immutable command invocation, +including the caller's stable command identity/retry window and domain-owned time +and randomness. Its fingerprint and original deadline include that capture. The +same namespace/class binding and shared invocation admission used by queries also +bound commands. Current shared-group command dispatch requires both the request +and namespace minimum to be Quorum. S3 refuses with `UnsupportedCapability` until +CG-06 supplies a real complete-group archive proof; no acknowledgement is lowered. + +`CellCommandInvocation` exposes preparation, provisional submission and settlement +as distinct phases for the production controller. Before entering SQL it retains +its originating runtime, pool, exact route and fresh preparation IDs. Cancelling +preparation loses only the response. Resuming retrieves that same worker-owned +producer, even if a routing retry selects another member. A catalog redirect never +releases an unresolved producer or runs its callback at a second destination. +The controller must retain invocations independently from ingress response waiters. + +Submission uses the existing shared acceptance/seal state and original SQL +ownership. Repeated acceptance cannot append twice. An exact pre-append rejection +seals every clone before cleanup. Settlement reads the original receipt through +current applied authority; same-term absence stays uncertain. Only the existing +strictly newer applied-term proof can produce `NotCommitted`. A subsequent +preparation uses fresh physical IDs with the same logical command and captured +inputs. Lost duplicate/settlement responses may use registered receipt lookup with +new command execution disabled. On a follower, a missing local producer still +requires routing to a leader for that fallback; it is never evidence of absence. + +Returned `CellCommandResponse` retains SQL output admission and signs the selected +cell observation. Acceptance itself is never a response carrying durability. +Deadline checks cover each phase and output delivery. Errors retain the command +ID and an explicit ambiguity bit; a deadline, lost reply or redirect is not a +non-commit proof. Dropping the invocation never releases its worker's unresolved +producer. The node controller still needs to own these phases through ingress +cancellation and drive shared transport/apply/completion events. Enrollment routing, +that production integration and final public SDK replacement remain unfinished. diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command.rs new file mode 100644 index 000000000..3c35b8f17 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command.rs @@ -0,0 +1,656 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Routed commands retain their originating producer until exact resolution. +//! The node controller drives publication; acceptance is only provisional. +use super::*; +use crate::application::consensus::command_request_digest; +use crate::raft::execution::{CommandAttempt, CommandRequest}; +use crate::storage::CommandInputs; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; +use cellar_ltx::generation::ImageId; + +pub struct CellCommandRequest { + pub id: CommandId, + pub contract: OperationContract, + pub input: Vec, + pub durability: Durability, + pub retry_until_ms: u64, +} + +#[derive(Debug)] +pub struct CellCommandError { + pub command: CommandId, + /// Conservatively sticky after SQL entry. Definitive resolution is returned + /// separately; an error is never a non-commit proof. Retry the same identity. + pub ambiguous: bool, + pub cause: CellDispatchError, +} + +impl fmt::Display for CellCommandError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "cell command {:?} (ambiguous={}): {}", + self.command, self.ambiguous, self.cause + ) + } +} + +impl std::error::Error for CellCommandError {} + +pub enum CellCommandPreparation { + Prepared, + Committed(CellCommandResponse), +} + +pub enum CellCommandSubmission { + Accepted(CellGroupProposal), + /// An exact shared preparation was irreversibly sealed before append. + Rejected, +} + +pub enum CellCommandSettlement { + Committed(CellCommandResponse), + NotCommitted, + Refused(CommandFailure), +} + +pub struct CellCommandResponse { + receipt: Box, + token: CellVersionToken, +} + +impl CellCommandResponse { + pub fn receipt(&self) -> &crate::raft::execution::CellCommittedReceipt { + self.receipt.receipt() + } + + pub fn token(&self) -> &CellVersionToken { + &self.token + } + + pub fn durability(&self) -> Durability { + Durability::Quorum + } +} + +struct Origin { + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + options: CellSqlOptions, + route: GroupRoute, + proposal: ProposalId, + preview: ImageId, + prepared: Option, + submitted: bool, +} + +/// The controller retains this non-cloneable invocation across waiter cancellation. +/// Dropping it never releases the SQL worker's unresolved physical producer. +pub struct CellCommandInvocation { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + request: CellCommandRequest, + inputs: CommandInputs, + session: CellRoutingSession, + attempt: Option, + dispatched: bool, + ambiguous: bool, + origin: Option, + _admission: Permit, +} + +impl CellGroupDispatcher { + pub async fn command( + &self, + cell: CellHistory, + mut request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + let error = |cause| CellCommandError { + command: request.id, + ambiguous: false, + cause, + }; + let started = self.domain.mono_ms().map_err(|_| error(closed()))?; + // Shared-group archival coverage is not integrated yet. Never silently + // lower either the request or the immutable namespace minimum. + if request.durability != Durability::Quorum + || self.directory.namespace_config().minimum_durability != Durability::Quorum + { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(error(CellDispatchError::Runtime(RuntimeError::Overloaded))); + } + if request.contract.kind != OperationKind::Command { + return Err(error(CellDispatchError::Registration( + RegistrationError::KindMismatch, + ))); + } + self.class + .validate_operation(&request.contract) + .map_err(|e| error(CellDispatchError::Registration(e)))?; + let size = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + let admission = self + .budget + .acquire(size) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + request.input = request.input.into_boxed_slice().into_vec(); + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + let inputs = self + .domain + .run(async { + use rand::RngCore as _; + let mut random = [0; 32]; + crate::asyncrt::rng("cell-group-command-inputs").fill_bytes(&mut random); + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + } + }) + .await + .map_err(|_| error(closed()))?; + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Command(request.id), + fingerprint: fingerprint(&cell, &request, inputs, limits), + }, + limits, + cached, + started, + ) + .map_err(|e| error(CellDispatchError::Routing(e)))?; + Ok(CellCommandInvocation { + dispatcher: self.clone(), + cell, + request, + inputs, + session, + attempt: None, + dispatched: false, + ambiguous: false, + origin: None, + _admission: admission, + }) + } +} + +impl CellCommandInvocation { + fn error(&self, cause: CellDispatchError) -> CellCommandError { + CellCommandError { + command: self.request.id, + ambiguous: self.ambiguous, + cause, + } + } + + fn checked( + &self, + result: Result, + deadline: u64, + ) -> Result { + let output = result.map_err(|e| self.error(e))?; + if self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))? + >= deadline + { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + Ok(output) + } + + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub async fn begin(&mut self) -> Result<(), CellCommandError> { + self.dispatched = true; + let result = self + .session + .begin() + .await + .map_err(CellDispatchError::Routing); + self.attempt = Some(result.map_err(|e| self.error(e))?); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellCommandError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let result = self + .session + .retry(&previous) + .await + .map_err(CellDispatchError::Routing); + self.attempt = Some(result.map_err(|e| self.error(e))?); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellCommandError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let result = self + .session + .redirect(&previous, hint) + .await + .map_err(CellDispatchError::Routing); + self.attempt = Some(result.map_err(|e| self.error(e))?); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellCommandError> { + self.attempt + .as_ref() + .ok_or_else(|| self.error(CellDispatchError::Invalid("missing route"))) + } + + fn command_attempt(&self, origin: &Origin) -> CommandAttempt { + CommandAttempt { + request: CommandRequest { + id: self.request.id, + operation: self.request.contract.operation.clone(), + contract_digest: self.request.contract.digest, + request_digest: command_request_digest(&self.request.input), + durability: self.request.durability, + retry_until_ms: self.request.retry_until_ms, + }, + proposal: origin.proposal, + preview: origin.preview, + inputs: self.inputs, + } + } + + /// Retain the origin before entering SQL. A cancelled or lost preparation + /// response resumes that producer, never a second destination's callback. + pub async fn prepare( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched { + return Err(self.error(CellDispatchError::Invalid("consumed route attempt"))); + } + self.dispatched = true; + if self.origin.is_some() { + return self.resume().await; + } + let domain = self.dispatcher.domain.clone(); + if !runtime.uses_domain(&domain) || !sql.uses_domain(&domain) { + return Err(self.error(CellDispatchError::Invalid("foreign execution domain"))); + } + let timer = domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.prepare_at(&attempt, runtime, sql, options) => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, attempt.deadline_ms()) + } + + async fn prepare_at( + &mut self, + attempt: &CellRouteAttempt, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let authority = runtime + .execution_authority(attempt.route().history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() { + return Err(CellDispatchError::Invalid("destination route binding")); + } + let (proposal, preview) = self + .dispatcher + .domain + .run(async { + use rand::RngCore as _; + let mut proposal = [0; 16]; + let mut preview = [0; 16]; + let mut rng = crate::asyncrt::rng("cell-group-command-preparation"); + rng.fill_bytes(&mut proposal); + rng.fill_bytes(&mut preview); + (ProposalId::new(proposal), ImageId::new(preview)) + }) + .await + .map_err(|_| closed())?; + let proposal = proposal.map_err(|_| CellDispatchError::Invalid("proposal ID"))?; + let preview = preview.map_err(|_| CellDispatchError::Invalid("preview ID"))?; + self.session + .mark_submitted(attempt) + .map_err(CellDispatchError::Routing)?; + self.ambiguous = true; + self.origin = Some(Origin { + runtime: runtime.clone(), + sql: sql.clone(), + options: options.clone(), + route: attempt.route().clone(), + proposal, + preview, + prepared: None, + submitted: false, + }); + let command = self.command_attempt( + self.origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?, + ); + let outcome = sql + .stage_registered( + options, + authority, + command, + self.dispatcher.class.clone(), + self.request.input.clone(), + true, + ) + .await; + match outcome { + Ok(outcome) => self.preparation(outcome).await, + Err( + error @ (ExecutionError::Refused(_) + | ExecutionError::Registration(_) + | ExecutionError::Busy), + ) => { + // These stage responses did not install this preparation. A + // retry may seek fresh authority; lost/closed replies do not + // take this path and keep their originating producer binding. + self.origin = None; + Err(CellDispatchError::Execution(error)) + } + Err(error) => Err(CellDispatchError::Execution(error)), + } + } + + async fn preparation( + &mut self, + outcome: CellSqlOutcome, + ) -> Result { + match outcome { + CellSqlOutcome::Prepared(prepared) => { + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))? + .prepared = Some(prepared); + Ok(CellCommandPreparation::Prepared) + } + CellSqlOutcome::Receipt(receipt) => { + let response = self.response(receipt).await?; + self.origin = None; + Ok(CellCommandPreparation::Committed(response)) + } + _ => Err(CellDispatchError::Invalid("command preparation response")), + } + } + + pub async fn resume(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.resume_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn resume_at(&mut self) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + match origin.sql.resume(self.cell.clone(), origin.proposal).await { + Ok(prepared) => self.preparation(CellSqlOutcome::Prepared(prepared)).await, + Err(ExecutionError::Uncertain) => self.lookup().await, + Err(error) => Err(CellDispatchError::Execution(error)), + } + } + + // A lost duplicate/settlement response can leave no pending producer. The + // registered path with allow_new=false may return its receipt, never execute. + async fn lookup(&mut self) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + let authority = origin + .runtime + .execution_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &origin.route { + return Err(CellDispatchError::Invalid("origin route binding")); + } + let outcome = origin + .sql + .stage_registered( + origin.options.clone(), + authority, + self.command_attempt(origin), + self.dispatcher.class.clone(), + self.request.input.clone(), + false, + ) + .await + .map_err(CellDispatchError::Execution)?; + self.preparation(outcome).await + } + + pub async fn submit(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.submit_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn submit_at(&mut self) -> Result { + let origin = self + .origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + let prepared = origin + .prepared + .as_ref() + .ok_or(CellDispatchError::Invalid("missing preparation"))?; + if let Some(proof) = prepared.rejection().map_err(CellDispatchError::Runtime)? { + return self.reject(proof).await; + } + origin.submitted = true; + origin + .sql + .mark_submitted(self.cell.clone(), origin.proposal) + .await + .map_err(CellDispatchError::Execution)?; + match prepared + .submit(&origin.runtime) + .await + .map_err(CellDispatchError::Runtime)? + { + CellSqlSubmission::Accepted(accepted) => Ok(CellCommandSubmission::Accepted(accepted)), + CellSqlSubmission::Rejected(proof) => self.reject(proof).await, + } + } + + async fn reject( + &mut self, + proof: CellSqlRejection, + ) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + match origin.sql.reject(self.cell.clone(), proof).await { + // The retained seal proves this exact preparation can never append; + // Uncertain here means its worker cleanup has already removed it. + Ok(()) | Err(ExecutionError::Uncertain) => (), + Err(error) => return Err(CellDispatchError::Execution(error)), + } + self.origin = None; + Ok(CellCommandSubmission::Rejected) + } + + /// Call after durable apply progress. Same-term absence remains uncertain; + /// only the existing newer-term barrier can establish NotCommitted. + pub async fn settle(&mut self) -> Result { + let deadline = self.current()?.deadline_ms(); + let domain = self.dispatcher.domain.clone(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| self.error(closed()))?; + let result = crate::asyncrt::select! { + result = self.settle_at() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.checked(result, deadline) + } + + async fn settle_at(&mut self) -> Result { + let origin = self + .origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing origin"))?; + if !origin.submitted { + return Err(CellDispatchError::Invalid("command not submitted")); + } + let authority = origin + .runtime + .reconciliation_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &origin.route { + return Err(CellDispatchError::Invalid("origin route binding")); + } + match origin.sql.reconcile(origin.proposal, authority).await { + Ok(CellSqlReconciliation::Committed(receipt)) => { + let response = self.response(receipt).await?; + self.origin = None; + Ok(CellCommandSettlement::Committed(response)) + } + Ok(CellSqlReconciliation::NotCommitted) => { + self.origin = None; + Ok(CellCommandSettlement::NotCommitted) + } + Ok(CellSqlReconciliation::Refused(failure)) => { + self.origin = None; + Ok(CellCommandSettlement::Refused(failure)) + } + Err(ExecutionError::Uncertain) => { + match origin.sql.resume(self.cell.clone(), origin.proposal).await { + Ok(_) => return Err(CellDispatchError::Execution(ExecutionError::Uncertain)), + Err(ExecutionError::Uncertain) => (), + Err(error) => return Err(CellDispatchError::Execution(error)), + } + match self.lookup().await? { + CellCommandPreparation::Committed(receipt) => { + Ok(CellCommandSettlement::Committed(receipt)) + } + CellCommandPreparation::Prepared => { + Err(CellDispatchError::Invalid("lookup executed a command")) + } + } + } + Err(error) => Err(CellDispatchError::Execution(error)), + } + } + + async fn response( + &self, + receipt: Box, + ) -> Result { + if receipt.receipt().receipt.command() != self.request.id + || receipt.receipt().version.history != self.cell + { + return Err(CellDispatchError::Invalid("command receipt binding")); + } + if receipt.receipt().receipt.minimum_durability() != Durability::Quorum { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + let now = self + .dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed())?; + let token = self + .dispatcher + .keys + .issue_cell( + &receipt.receipt().observation(), + self.dispatcher.class.schema_digest(), + now, + ) + .map_err(CellDispatchError::Version)?; + Ok(CellCommandResponse { receipt, token }) + } +} + +fn fingerprint( + cell: &CellHistory, + request: &CellCommandRequest, + inputs: CommandInputs, + limits: CellRoutingLimits, +) -> [u8; 32] { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-command-invocation-1\0"); + hash.update(cell_image_history_digest(cell)); + hash.update(request.id.client.bytes()); + hash.update(request.id.sequence().to_be_bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update(command_request_digest(&request.input)); + hash.update([1]); // Quorum is the only supported request/minimum until CG-06. + hash.update(request.retry_until_ms.to_be_bytes()); + hash.update(inputs.now_unix_millis.to_be_bytes()); + hash.update(inputs.random); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([limits.max_attempts]); + hash.finalize().into() +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs similarity index 98% rename from crates/cellar/durability/raft/cell_groups/dispatch.rs rename to crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 4926972fa..283d8b36b 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -3,6 +3,12 @@ //! Catalog-bound invocations over the existing consensus and SQL owners. //! Routing observations never replace destination authority or ingress grants. +mod command; +pub use command::{ + CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, + CellCommandResponse, CellCommandSettlement, CellCommandSubmission, +}; + use super::*; use crate::application::consensus::{ OperationContract, OperationKind, RaftClass, RegistrationError, diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index 74116d80d..193305a01 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -136,6 +136,10 @@ pub struct CellReconciliationAuthority { } impl CellReconciliationAuthority { + pub fn route(&self) -> &GroupRoute { + &self.route + } + pub(super) fn new( group: GroupLog, current: &PreparedGroupManifest, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index d53d24da0..654b20c04 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -13,8 +13,10 @@ mod apply; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; pub use dispatch::{ - CellDispatchError, CellDispatchLimits, CellGroupDispatcher, CellQueryConsistency, - CellQueryInvocation, CellQueryRequest, CellQueryResponse, + CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, + CellCommandResponse, CellCommandSettlement, CellCommandSubmission, CellDispatchError, + CellDispatchLimits, CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, + CellQueryRequest, CellQueryResponse, }; mod execution_authority; diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 01021e126..f15e67be8 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1330,3 +1330,13 @@ async fn catalog_routed_queries_preserve_invocation_and_authority() -> Result<() async fn rooted_catalog_routed_queries_preserve_invocation_and_authority() -> Result<()> { Box::pin(runtime_fixture::routed_queries(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> { + Box::pin(runtime_fixture::routed_commands(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> { + Box::pin(runtime_fixture::routed_commands(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 01561f4fc..599e025f5 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -671,3 +671,5 @@ pub(super) use sql::registered_commands; pub(super) use sql::registered_queries; pub(super) use sql::routed_queries; + +pub(super) use sql::routed_commands; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f0d15f23c..48796f0d2 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -521,3 +521,7 @@ pub(crate) use registered_queries::queries as registered_queries; #[path = "cell_group_sql_dispatch.rs"] mod dispatch; pub(crate) use dispatch::routed as routed_queries; + +#[path = "cell_group_sql_command_dispatch.rs"] +mod command_dispatch; +pub(crate) use command_dispatch::commands as routed_commands; diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs new file mode 100644 index 000000000..81442548a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -0,0 +1,331 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +fn request( + class: &RaftClass, + sequence: u64, + retry: u64, + input: &[u8], +) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([211; 16])?, sequence)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry, + }) +} + +async fn accepted(invocation: &mut CellCommandInvocation) -> Result { + match invocation.submit().await? { + CellCommandSubmission::Accepted(accepted) => Ok(accepted), + CellCommandSubmission::Rejected => anyhow::bail!("unexpected sealed rejection"), + } +} + +async fn committed(invocation: &mut CellCommandInvocation) -> Result { + match invocation.settle().await? { + CellCommandSettlement::Committed(receipt) => Ok(receipt), + _ => anyhow::bail!("expected committed receipt"), + } +} + +pub(crate) async fn commands(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let captured = Arc::new(Mutex::new(Vec::new())); + let count = calls.clone(); + let inputs = captured.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let wait = Mutex::new(wait); + let class = Arc::new( + RaftClass::new("Counter", 1, [91; 32])? + .command("Add", [92; 32], move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + inputs.lock().unwrap().push(( + context.identity().cell.clone(), + context.command(), + context.inputs().now_unix_millis, + context.inputs().random, + )); + if input == b"hold" && entered.lock().unwrap().is_some() { + entered + .lock() + .unwrap() + .take() + .unwrap() + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + wait.lock().unwrap().recv_timeout(Duration::from_secs(10))?; + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + })? + .query("Get", [93; 32], |context, _| { + let n: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let keys = VersionKeys::new( + VersionSigningKey::new([94; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let bounds = CellDispatchLimits { + max_invocations: 3, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + bounds, + )?; + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let runtime = fixture.runtimes[0].clone(); + let options = fixture.cell_options(&a).await?; + let mut command = dispatcher + .command( + a.clone(), + request(&class, 1, now + 60_000, b"hold")?, + limits(), + None, + ) + .await?; + command.begin().await?; + let first = command.attempt().context("first route")?.clone(); + let mut preparation = Box::pin(command.prepare(&runtime, &pool, options.clone())); + tokio::select! { + result = &mut preparation => anyhow::bail!("held callback finished: {}", result.is_ok()), + result = started => result?, + } + drop(preparation); // Cancelled response, not physical SQL ownership. + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + command.retry().await?; + assert_eq!( + command.attempt().context("retry")?.request(), + first.request() + ); + assert_eq!( + command.attempt().context("retry")?.deadline_ms(), + first.deadline_ms() + ); + let other_pool = fixture.appliers[&(1, route.history)].0.clone(); + assert!(matches!( + command + .prepare( + &fixture.runtimes[1], + &other_pool, + fixture.cell_options_at(&a, 1).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(other_pool.usage()?.active_cells, 0); + let first_acceptance = accepted(&mut command).await?; + let again = accepted(&mut command).await?; // Lost acceptance response. + assert_eq!(first_acceptance.position, again.position); + assert_eq!(first_acceptance.digest, again.digest); + let absent = command.settle().await.err().context("not yet applied")?; + assert!(absent.ambiguous); + assert!(matches!( + absent.cause, + CellDispatchError::Execution(ExecutionError::Uncertain) + )); + assert_eq!(pool.usage()?.active_cells, 1); + fixture.drive(Some(again.position.index)).await?; + let receipt = committed(&mut command).await?; + assert_eq!( + receipt.receipt().receipt.result(), + a.cell.as_str().as_bytes() + ); + assert_eq!(receipt.receipt().version.sequence.get(), 2); + assert_eq!(receipt.durability(), Durability::Quorum); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys.verify_cell(receipt.token(), &a, route.schema_digest, now)?, + receipt.receipt().observation() + ); + drop(receipt); + drop(command); + + let mut other = dispatcher + .command( + b.clone(), + request(&class, 1, now + 60_000, b"ordinary")?, + limits(), + None, + ) + .await?; + other.begin().await?; + assert!(matches!( + other + .prepare(&runtime, &pool, fixture.cell_options(&b).await?) + .await?, + CellCommandPreparation::Prepared + )); + let b_acceptance = accepted(&mut other).await?; + fixture.drive(Some(b_acceptance.position.index)).await?; + drop(committed(&mut other).await?); + drop(other); + let mut duplicate = dispatcher + .command( + a.clone(), + request(&class, 1, now + 60_000, b"hold")?, + limits(), + Some(first.cached_route()), + ) + .await?; + duplicate.begin().await?; + let CellCommandPreparation::Committed(receipt) = + duplicate.prepare(&runtime, &pool, options.clone()).await? + else { + anyhow::bail!("duplicate executed") + }; + assert_eq!( + receipt.receipt().receipt.result(), + a.cell.as_str().as_bytes() + ); + assert_eq!(calls.load(Ordering::SeqCst), 2); + drop(receipt); + drop(duplicate); + + // An actual higher-term applied barrier can exclude a prepared old leader + // command. Retrying preserves host-captured time/randomness and command ID. + let mut retry = dispatcher + .command( + a.clone(), + request(&class, 2, now + 60_000, b"ordinary")?, + limits(), + Some(first.cached_route()), + ) + .await?; + retry.begin().await?; + let retry_first = retry.attempt().context("retry origin")?.clone(); + assert!(matches!( + retry.prepare(&runtime, &pool, options.clone()).await?, + CellCommandPreparation::Prepared + )); + reconciliation::transfer_cell(&mut fixture, 0, 1, a.clone()).await?; + let error = retry.submit().await.err().context("old leader refused")?; + assert!(error.ambiguous); + assert!(matches!( + retry.settle().await?, + CellCommandSettlement::NotCommitted + )); + assert_eq!(pool.usage()?.active_cells, 0); + retry.retry().await?; + assert_eq!( + retry.attempt().context("retry new leader")?.request(), + retry_first.request() + ); + assert!(matches!( + retry + .prepare( + &fixture.runtimes[1], + &other_pool, + fixture.cell_options_at(&a, 1).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + let retry_acceptance = accepted(&mut retry).await?; + reconciliation::apply_through(&mut fixture, retry_acceptance.position.index).await?; + let receipt = committed(&mut retry).await?; + assert_eq!(receipt.receipt().version.sequence.get(), 3); + drop(receipt); + drop(retry); + let captured = captured.lock().unwrap().clone(); + assert_eq!(captured.len(), 4); + assert_eq!(captured[2], captured[3]); + + let mut s3 = request(&class, 3, now + 60_000, b"ordinary")?; + s3.durability = Durability::S3; + let error = dispatcher + .command(a.clone(), s3, limits(), None) + .await + .err() + .context("S3 unavailable")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause, + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); + let catalog = Catalog::new( + Arc::new(object_store::memory::InMemory::new()), + CatalogOptions::default(), + )?; + let mut config = NamespaceConfig::new( + a.cluster, + a.namespace, + cellar_logic::NamespaceMode::Raft, + class.schema_digest(), + ); + config.minimum_durability = Durability::S3; + let namespace = fixture + .cluster + .domain + .run(catalog.open_namespace(config)) + .await??; + let required = CellGroupDispatcher::new( + &fixture.cluster.domain, + RangeDirectory::new(namespace, DirectoryLimits::default())?, + class.clone(), + keys, + bounds, + )?; + assert!(matches!( + required + .command( + a.clone(), + request(&class, 3, now + 60_000, b"ordinary")?, + limits(), + None + ) + .await, + Err(CellCommandError { + ambiguous: false, + cause: CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )), + .. + }) + )); + assert_eq!(calls.load(Ordering::SeqCst), 4); + assert_eq!(dispatcher.usage(), (0, 0)); + drop(required); + drop(dispatcher); + drop(pool); + drop(other_pool); + drop(runtime); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs index 6a48a7314..1606eed9e 100644 --- a/crates/cellar/tests/support/cell_group_sql_reconcile.rs +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -3,6 +3,16 @@ use super::*; pub(super) async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usize) -> Result<()> { + let cell = super::history(fixture.cluster.birth.route(), "sql-a")?; + Box::pin(transfer_cell(fixture, from, to, cell)).await +} + +pub(super) async fn transfer_cell( + fixture: &mut RuntimeCluster, + from: usize, + to: usize, + cell: CellHistory, +) -> Result<()> { let history = fixture.cluster.birth.route().history; let term = fixture.runtimes[from].status(history).await?.term; let mut message = Message::default(); @@ -27,21 +37,31 @@ pub(super) async fn transfer(fixture: &mut RuntimeCluster, from: usize, to: usiz } }) .await??; - let a = super::history(fixture.cluster.birth.route(), "sql-a")?; assert!( matches!( fixture.runtimes[0] - .reconciliation_authority(history, a) + .reconciliation_authority(history, cell) .await, Err(RuntimeError::NotReady) ), "a persisted newer term without its applied barrier cannot retire a producer" ); let work = held.remove(&history).context("held new-term barrier")?; - let next = prepare(&mut fixture.cluster.voters[0], &work)?; - fixture.runtimes[0] - .complete_apply(history, work, next) + if let Some((pool, applier)) = fixture.appliers.get(&(0, history)) { + Box::pin(async { + pool.prepare_apply(applier, work) + .await? + .complete(&fixture.runtimes[0]) + .await?; + Ok::<_, anyhow::Error>(()) + }) .await?; + } else { + let next = prepare(&mut fixture.cluster.voters[0], &work)?; + fixture.runtimes[0] + .complete_apply(history, work, next) + .await?; + } } tokio::time::timeout(Duration::from_secs(20), async { loop { diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 309d538aa..bd929cb22 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 29 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 31 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index b4b86239b..0f587d255 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -484,8 +484,10 @@ "crates/cellar/durability/raft/effect_history.rs", "crates/cellar/tests/support/cell_group_sql_registered.rs", "crates/cellar/tests/support/cell_group_sql_registered_queries.rs", - "crates/cellar/durability/raft/cell_groups/dispatch.rs", - "crates/cellar/tests/support/cell_group_sql_dispatch.rs" + "crates/cellar/durability/raft/cell_groups/dispatch/mod.rs", + "crates/cellar/tests/support/cell_group_sql_dispatch.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command.rs", + "crates/cellar/tests/support/cell_group_sql_command_dispatch.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -562,6 +564,13 @@ "catalog_routed_queries_preserve_invocation_and_authority", "rooted_catalog_routed_queries_preserve_invocation_and_authority" ] + }, + { + "action": "Route immutable registered command invocations through the same dispatcher and catalog sessions. Capture host time/randomness once, retain the original runtime/SQL producer across cancelled preparation, and resume it before considering a new destination. Submission acceptance is provisional; settle only an exact committed receipt or the existing newer-term absence proof. Preserve command identity, deadline and attempt budget across retries, and refuse S3 requests or S3 namespace minima until a real archive proof is integrated. The production controller must retain active invocations and drive publication; enrollment and the public SDK/controller cutover remain required.", + "tests": [ + "catalog_routed_commands_preserve_producer_and_receipts", + "rooted_catalog_routed_commands_preserve_producer_and_receipts" + ] } ], "test_cases": [ @@ -831,6 +840,26 @@ "I05", "I12" ] + }, + { + "name": "catalog_routed_commands_preserve_producer_and_receipts", + "oracle": "A real catalog-routed registered command on three voters resumes a held/cancelled preparation without another callback or a foreign-node producer. Repeated submission retains the original acceptance; same-term receipt absence stays ambiguous and charged. Exact publication returns the original cell receipt and token, including a duplicate after another cell advances group progress. Real leader transfer permits NotCommitted only behind the newer applied term, then retry preserves captured time/randomness and command identity. S3 request and namespace minima refuse without weakening durability.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_catalog_routed_commands_preserve_producer_and_receipts", + "oracle": "A real catalog-routed registered command on three voters resumes a held/cancelled preparation without another callback or a foreign-node producer. Repeated submission retains the original acceptance; same-term receipt absence stays ambiguous and charged. Exact publication returns the original cell receipt and token, including a duplicate after another cell advances group progress. Real leader transfer permits NotCommitted only behind the newer applied term, then retry preserves captured time/randomness and command identity. S3 request and namespace minima refuse without weakening durability.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From 7d5e01da39408308945ab80dff74d286dadaf14c Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:35:17 -0700 Subject: [PATCH 082/568] fix(raft): bound routed command error return size --- .../durability/raft/cell_groups/dispatch/command.rs | 6 +++--- .../tests/support/cell_group_sql_command_dispatch.rs | 12 ++++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command.rs index 3c35b8f17..34a12f124 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command.rs @@ -23,7 +23,7 @@ pub struct CellCommandError { /// Conservatively sticky after SQL entry. Definitive resolution is returned /// separately; an error is never a non-commit proof. Retry the same identity. pub ambiguous: bool, - pub cause: CellDispatchError, + pub cause: Box, } impl fmt::Display for CellCommandError { @@ -111,7 +111,7 @@ impl CellGroupDispatcher { let error = |cause| CellCommandError { command: request.id, ambiguous: false, - cause, + cause: Box::new(cause), }; let started = self.domain.mono_ms().map_err(|_| error(closed()))?; // Shared-group archival coverage is not integrated yet. Never silently @@ -195,7 +195,7 @@ impl CellCommandInvocation { CellCommandError { command: self.request.id, ambiguous: self.ambiguous, - cause, + cause: Box::new(cause), } } diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs index 81442548a..b7890f3fd 100644 --- a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -152,7 +152,7 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { let absent = command.settle().await.err().context("not yet applied")?; assert!(absent.ambiguous); assert!(matches!( - absent.cause, + absent.cause.as_ref(), CellDispatchError::Execution(ExecutionError::Uncertain) )); assert_eq!(pool.usage()?.active_cells, 1); @@ -275,7 +275,7 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { .context("S3 unavailable")?; assert!(!error.ambiguous); assert!(matches!( - error.cause, + error.cause.as_ref(), CellDispatchError::Execution(ExecutionError::Refused( CommandFailure::UnsupportedCapability )) @@ -314,11 +314,11 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { .await, Err(CellCommandError { ambiguous: false, - cause: CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability - )), + cause, .. - }) + }) if matches!(cause.as_ref(), CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + ))) )); assert_eq!(calls.load(Ordering::SeqCst), 4); assert_eq!(dispatcher.usage(), (0, 0)); From 1f014f42264c73b1d5e3cd387f65ffed45140ecb Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:41:18 -0700 Subject: [PATCH 083/568] test(raft): use domain-aware command cancellation polling --- crates/cellar/tests/support/cell_group_sql_command_dispatch.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs index b7890f3fd..a7b2e2ae1 100644 --- a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -116,7 +116,7 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { command.begin().await?; let first = command.attempt().context("first route")?.clone(); let mut preparation = Box::pin(command.prepare(&runtime, &pool, options.clone())); - tokio::select! { + cellar::asyncrt::select! { result = &mut preparation => anyhow::bail!("held callback finished: {}", result.is_ok()), result = started => result?, } From f8f09514b19d33032715ad9735e8ea89185791f6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:44:36 -0700 Subject: [PATCH 084/568] docs(raft): qualify routed commands and prioritize controller integration --- docs/design/multi-raft/cell-groups/README.md | 10 ++++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 46 +++++++++++------ .../design/multi-raft/cell-groups/progress.md | 50 +++++++++++++++++++ 4 files changed, 92 insertions(+), 18 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index cdcd82074..344e46dc8 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -153,6 +153,16 @@ and eight commands pass. Transfer production controller/SDK integration are the next critical gaps. Metadata-only redirect tests do not qualify live transfer. See [progress](progress.md). +Catalog-bound command phases now retain original SQL ownership through cancelled +preparation, recover repeated acceptance and settle exact receipts at `1f014f42`. +A real leader-transfer retry preserves captured inputs; S3 minima explicitly refuse +until archive proof integration. All 105 native cases and eight commands pass. +Transfer `.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/` and the failed +`CG-04-routed-commands/` and `CG-04-routed-commands-final/` gates with prior evidence. The newest record binds 229 +source inputs and four executables. Production controller ownership after ingress +cancellation and physical-group transport integration are the next critical gaps, +followed by routed enrollment and the public SDK cutover. See [progress](progress.md). + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index d7bc5174c..8ff71f314 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "8723db2df7436478cbc5efc9147d65d3b7651247", + "source_revision": "1f014f42264c73b1d5e3cd387f65ffed45140ecb", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index a3e2afd47..b0e59c8b7 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,17 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The reviewed source baseline is `8723db2d`, adding catalog-bound query +The newest reviewed source baseline is `1f014f42`, adding retained routed +command phases over the existing catalog, consensus and SQL owners. Its partial +record is `.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/checkpoint.json`: +105 native cases, eight commands, 229 source inputs and four selected executables. +Transfer that directory and the retained failed `CG-04-routed-commands` and +`CG-04-routed-commands-final` gates with prior evidence. Cancellation tests retain the invocation; production controller +ownership after ingress cancellation remains unqualified. The command phases +preserve captured inputs across proven-non-commit retry and never equate proposal +acceptance with durability. S3 minima refuse pending real shared archive proof. +CG-04 remains IN_PROGRESS; follow the next execution priorities below. +The preceding source baseline is `8723db2d`, adding catalog-bound query invocations over the existing runtime/SQL owners. Its partial record is `.cellar/cell-groups-evidence/CG-04-routed-queries/checkpoint.json`: 103 native cases across four affected suites, eight successful commands, 227 source inputs @@ -361,21 +371,25 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Extend `CellGroupDispatcher` from its completed local query boundary to - registered command/enrollment invocations. Capture namespace minimum durability, - original command/creation identity, input fingerprint, deadline and total attempt - budget. Use existing prepared submission, exact rejection proofs and receipt or - creation reconciliation; neither a lost response nor a catalog redirect proves - an old producer uncommitted. Accepted proposals are provisional: return a durable - result only after exact publication/receipt settlement and the requested Quorum - or S3 proof. Do not weaken a namespace's minimum durability while archive - integration is pending. Qualify actual catalog-routed enrollment, command, - receipt retry and linearizable query together. The query path now preserves - these routing limits and authenticates selected-version tokens in both modes. - Public `RaftCellHandle` still uses its old host binding. Its final replacement - also needs the production node controller to consume shared-runtime transport, - apply and completion events; current real-engine fixtures pump those events. - Reuse the existing transport and worker owners, with joined cancellation. +1. Integrate the phased command invocation into the production node controller. + Retain it independently from ingress waiters and drive existing shared-runtime + transport, apply and completion events through the existing owners. The current + `RaftTransport` routes by old `GroupScope`; adapt its route/authentication boundary + to physical `GroupHistory` instead of manufacturing a cell-scoped identity. Qualify + caller disconnect after SQL entry and after provisional acceptance: the + controller must resolve the exact receipt or a proven non-commit and release + bounded ownership without re-executing an uncertain command. The current test + cancels a preparation future while retaining the invocation; it does not + qualify controller ownership after ingress cancellation. Missing origin + producers on followers still need a safe leader receipt-lookup route; never + infer non-commit from missing local state. Keep original deadlines for responses + while giving the controller an explicit bounded reconciliation lifecycle. + Then add catalog-routed enrollment with stable creation identity and exact + creation settlement. Qualify enrollment, command, lost response and linearizable + query together. Public `RaftCellHandle` still uses its old host binding; the + real-engine fixtures still pump transport/apply events. Reuse existing worker + and transport ownership with joined shutdown. Requested and namespace S3 + durability currently refuse until CG-06 supplies real archive proofs. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index fa1c7bf0e..5fc7171ca 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,56 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 catalog-routed command phases + +Qualified source: `1f014f42264c73b1d5e3cd387f65ffed45140ecb`. + +`CellGroupDispatcher::command` freezes logical command identity, input, namespace +minimum durability, original routing budget and domain-captured time/randomness. +Preparation retains its original runtime, SQL pool, exact route and physical +preparation IDs before entering SQL. A cancelled response or routing retry resumes +that producer instead of executing a second destination's callback. Submission +reuses exact shared acceptance and pre-append seals. Acceptance is provisional; +settlement requires the exact receipt, while absence requires the existing newer +applied-term barrier. Errors preserve command identity and conservative ambiguity. +Returned receipts retain their output admission and sign the cell observation. +Both request and namespace S3 minima refuse until real shared archive proof exists. + +Two new three-voter cases cover both image modes, held-callback cancellation, +retry through a follower while retaining the original producer, duplicate proposal +acceptance, same-term uncertainty, exact receipt recovery after another cell +advances the group, and a real leader transfer. A proven non-commit can retry with +the same captured time/randomness. S3 refusal precedes callback execution. +The leadership fixture checks the caller's actual cell and uses the production +applier where installed; the held newer-term barrier assertion remains unchanged. + +All 105 native cases pass across execution, catalog, read and runtime suites; +one runtime helper is invoked by its parent. All eight commands pass, including +all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting, layout, dependency isolation, docs, source preflight and all 37 +plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/checkpoint.json` binds +229 source inputs and four selected executables. The failed gate at +`CG-04-routed-commands` is retained: all native cases passed, then Clippy rejected +the large command error. Its typed cause is now boxed, preserving all assertions. +A second gate at `CG-04-routed-commands-final` retained the passing native tests +and fixture lint failure; the direct Tokio select now uses Cellar's required +fair-polling macro. Earlier compile and fixture failures are retained too. No RustFS +or performance qualification was run. Latest fetched `origin/main` at `8a93ffd7` is included; +the requested MR-13/MR-14 JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The critical next gap is production controller ownership +of these phases after ingress cancellation, including an explicit bounded +reconciliation lifecycle beyond the response deadline. The current tests retain +the invocation while cancelling its preparation future; they do not qualify a +controller after the caller drops. Adapt the existing transport's cell-scoped +identity to physical group history, drive transport/apply/completion events, and +qualify exact resolution after caller disconnect. Then complete routed enrollment +and public SDK binding. Missing-origin follower receipt routing, configuration and +cache lifecycle remain unfinished. Reuse existing workers and resource owners; +complete-group snapshots/archive, effects, topology, real RustFS/load and the final +hard cutover remain later task gates. + ## CG-04 catalog-routed query invocations Qualified source: `8723db2df7436478cbc5efc9147d65d3b7651247`. From eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 03:58:28 -0700 Subject: [PATCH 085/568] feat(raft): replicate physical cell groups over shared TLS transport --- crates/cellar/durability/raft/README.md | 34 ++ .../durability/raft/cell_groups/owner.rs | 12 + .../raft/cell_groups/runtime/mod.rs | 36 +- .../raft/cell_groups/runtime/worker.rs | 4 + .../durability/raft/inspection_transport.rs | 1 + .../cellar/durability/raft/query_transport.rs | 8 +- crates/cellar/durability/raft/transport.rs | 120 ++++-- .../durability/raft/transport_directory.rs | 20 +- .../cellar/durability/raft/transport_frame.rs | 100 ++++- .../durability/raft/transport_frame_tests.rs | 55 +++ .../raft/transport_pipeline_tests.rs | 13 +- .../durability/raft/transport_resources.rs | 2 +- .../durability/raft/transport_runtime.rs | 96 +++++ .../cellar/durability/raft/transport_send.rs | 22 +- .../durability/raft/transport_shared.rs | 357 ++++++++++++++++++ .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_runtime.rs | 24 ++ crates/cellar/tests/support/cell_group_sql.rs | 4 + .../cell_group_sql_command_dispatch.rs | 7 + crates/cellar/tests/support/cell_group_tcp.rs | 221 +++++++++++ .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 44 ++- 22 files changed, 1132 insertions(+), 65 deletions(-) create mode 100644 crates/cellar/durability/raft/transport_runtime.rs create mode 100644 crates/cellar/durability/raft/transport_shared.rs create mode 100644 crates/cellar/tests/support/cell_group_tcp.rs diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index e5fb8b088..9982a2b98 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -1878,3 +1878,37 @@ non-commit proof. Dropping the invocation never releases its worker's unresolved producer. The node controller still needs to own these phases through ingress cancellation and drive shared transport/apply/completion events. Enrollment routing, that production integration and final public SDK replacement remain unfinished. + +### Physical-group peer transport + +`RaftTransport::start_shared` starts directly on `CellGroupRuntime`; it never +constructs the old cell-scoped runtime. Physical routes use +`GroupRoute` and `RaftPeer`. Installing a route checks +an existing group, selected schema, committed member/disk bindings, local process +and listener addresses. It creates no storage or Raft membership and grants no +serving authority. Unknown groups and unsupported complete-group snapshots refuse. + +Physical groups share the existing per-node TLS connections, control/data lanes, +pipelining, bounded byte/item banks and joined shutdown. Group routes share the +same configured group limit with temporary old/query routes. Node-view refresh, +fleet revocation and pool pruning include physical routes. Query/inspection grants +cannot replace conflicting physical-group endpoints. Queue admission, I/O and +observer delivery recheck exact route/process and committed membership. A delivery +response acknowledges receiver queue admission, never append, commit or apply. +Failed sends report bounded unreachable hints without fabricating Raft responses. + +Marker-1 peer frames now carry an explicitly typed history under a new protocol +digest. Physical frames contain group history without a cell key. Old untagged +scope frames refuse; there is no compatibility reader. Both directions bind full +history, member, node, disk, process, schema, sequence, payload digest and TLS +exporter through the existing authenticated frame machinery. The temporary +`TransportRuntime::Legacy` and typed cell-history frame arm support the old host +only during integration; CG-16 must remove them with that host. They are not a +configuration switch or an alternate final serving architecture. + +The transport retains its exclusive runtime claim until physical workers and +sockets exit, including cancelled close waiters. A replacement may start only +after that join. Complete-group snapshot streaming, shared query RPC routing and +production controller ownership of apply/command settlement remain unfinished. +The TLS fixtures still drive runtime events themselves; they do not qualify +controller ownership after an ingress caller disconnects. diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 75e2eaba0..e53a34205 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -409,6 +409,18 @@ impl CellGroupOwner { Ok(()) } + pub(super) fn report_unreachable(&mut self, member: u64) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() { + return Err(RuntimeError::NotReady); + } + if member == 0 || self.node.raft.prs().get(member).is_none() { + return Err(RuntimeError::Protocol("unknown unreachable member".into())); + } + self.node.report_unreachable(member); + Ok(()) + } + pub fn step(&mut self, message: Message) -> Result<(), RuntimeError> { self.live()?; if self.phase.is_some() { diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 735942999..49cf49198 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -78,6 +78,7 @@ struct Handle { budget: Arc, completions: Arc, read_pins: Arc, + transport_claimed: AtomicBool, closing: Arc, terminal: Arc, stop: Arc, @@ -99,6 +100,7 @@ enum Command { Option>, ), ReadIndex(Vec), + Unreachable(u64), Read(Box), Apply(CellGroupTicket, Box), Status, @@ -126,7 +128,28 @@ struct Request { inbound: Option, } +pub(crate) struct TransportClaim(Arc); + +impl Drop for TransportClaim { + fn drop(&mut self) { + self.0.transport_claimed.store(false, Ordering::Release); + } +} + impl CellGroupRuntime { + pub(crate) fn log(&self) -> &ReplicaLog { + &self.inner.log + } + + pub(crate) fn claim_transport(&self) -> Result { + self.admit()?; + self.inner + .transport_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached)?; + Ok(TransportClaim(self.inner.clone())) + } + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { self.inner.domain.same_owner(domain) } @@ -187,6 +210,7 @@ impl CellGroupRuntime { read_pins: Budget::new(options.max_read_pin_bytes, options.max_pending_reads), options: options.clone(), requests: sender, + transport_claimed: AtomicBool::new(false), closing: closing.clone(), terminal: terminal.clone(), stop: stop.clone(), @@ -225,7 +249,7 @@ impl CellGroupRuntime { Ok((handle, CellGroupRuntimeEvents { receiver })) } - fn admit(&self) -> Result<(), RuntimeError> { + pub(crate) fn admit(&self) -> Result<(), RuntimeError> { if self.inner.closing.load(Ordering::Acquire) || self.inner.domain.mono_ms().is_err() { return Err(RuntimeError::Closed); } @@ -540,6 +564,16 @@ impl CellGroupRuntime { .map(|_| ()) } + pub(crate) async fn report_unreachable( + &self, + history: GroupHistory, + member: u64, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::Unreachable(member), 4096) + .await + .map(|_| ()) + } + pub async fn read_index( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 1fdda291c..17cb888f5 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -374,6 +374,10 @@ impl Worker { Command::Submit(prepared, _retained) => { group.owner.submit(prepared).map(Response::Submission) } + Command::Unreachable(member) => group + .owner + .report_unreachable(member) + .map(|_| Response::Unit), Command::ReadIndex(context) => { group.owner.read_index(context).map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/inspection_transport.rs b/crates/cellar/durability/raft/inspection_transport.rs index b84c676d1..8e42e3f8d 100644 --- a/crates/cellar/durability/raft/inspection_transport.rs +++ b/crates/cellar/durability/raft/inspection_transport.rs @@ -142,6 +142,7 @@ impl RaftTransport { let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; if let Some(old) = registry.inspection_origins.get(&key) { if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { return Ok(()); diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 9884bd667..31d9aa28a 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -70,6 +70,7 @@ impl RaftTransport { let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; if let Some(old) = registry.origins.get(&key) { if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { return Ok(()); @@ -198,6 +199,7 @@ impl RaftTransport { state.admit(®istry)?; for peer in &physical { registry.check_node_view(peer)?; + registry.check_physical_endpoint(peer)?; } if let Some(old) = registry.query_groups.get(&route.scope) { if **old == route { @@ -206,7 +208,11 @@ impl RaftTransport { if old.revision >= route.revision || old.schema_digest != route.schema_digest { return Err(TransportError::StalePeer); } - } else if registry.query_groups.len() + registry.groups.len() >= state.options.max_groups { + } else if registry.query_groups.len() + + registry.physical_groups.len() + + registry.groups.len() + >= state.options.max_groups + { return Err(TransportError::Overloaded); } for peer in &physical { diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index 8c3285cc5..db0f89078 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -8,6 +8,12 @@ use super::completion::RuntimeError; use super::mailbox; use super::performance::{Operation, Timer}; use super::runtime::RaftRuntime; +use cellar_logic::cell_group::GroupHistory; +#[path = "transport_runtime.rs"] +mod runtime_adapter; +use runtime_adapter::TransportRuntime; +#[path = "transport_shared.rs"] +mod shared; use super::transport_frame::{self as frame, Disposition, FrameHead, RequestHeader, Route}; use super::transport_options::{NamespaceQueryOrigin, RaftPeerNode, RaftTls, TransportOptions}; use crate::asyncrt::DomainToken; @@ -16,6 +22,7 @@ use ::raft::prelude::{Message, MessageType}; use cellar_logic::consensus::{ClusterId, GroupScope, NamespaceId, NodeId, ReplicaIdentity}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; +use shared::SendRoute; use std::collections::{BTreeMap, BTreeSet}; use std::future::Future; use std::io::IoSlice; @@ -104,8 +111,8 @@ impl From for TransportError { } #[derive(Clone, Debug, PartialEq, Eq)] -pub struct RaftPeer { - pub replica: ReplicaIdentity, +pub struct RaftPeer { + pub replica: ReplicaIdentity, pub label: String, pub server_name: String, pub control_address: SocketAddr, @@ -113,7 +120,7 @@ pub struct RaftPeer { pub query_address: SocketAddr, } -impl RaftPeer { +impl RaftPeer { fn same_host(&self, other: &Self) -> bool { self.replica.node == other.replica.node && self.replica.disk == other.replica.disk @@ -126,8 +133,8 @@ impl RaftPeer { } } -impl From<&RaftPeer> for RaftPeerNode { - fn from(peer: &RaftPeer) -> Self { +impl From<&RaftPeer> for RaftPeerNode { + fn from(peer: &RaftPeer) -> Self { Self { node: peer.replica.node, disk: peer.replica.disk, @@ -164,7 +171,7 @@ impl RaftPeerNode { } } - fn same_host(&self, peer: &RaftPeer) -> bool { + fn same_host(&self, peer: &RaftPeer) -> bool { self.node == peer.replica.node && self.disk == peer.replica.disk && self.process == peer.replica.process @@ -179,11 +186,11 @@ impl RaftPeerNode { /// An already authorized routing view, installed by the membership controller. /// Installing it never creates storage or changes Raft's configuration. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct GroupRoute { - pub scope: GroupScope, +pub struct GroupRoute { + pub scope: S, pub schema_digest: [u8; 32], pub revision: u64, - pub peers: Vec, + pub peers: Vec>, } /// Message admission only. No durability, commit or applied position is implied. @@ -239,9 +246,8 @@ impl Banks { } struct SendRequest { - scope: GroupScope, - route: Arc, - peer: RaftPeer, + route: SendRoute, + member: cellar_logic::consensus::MemberId, header: RequestHeader, body: Vec, snapshot: bool, @@ -272,6 +278,7 @@ struct Registry { closed: bool, tasks: usize, groups: BTreeMap>, + physical_groups: BTreeMap>>, query_groups: BTreeMap>, origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, @@ -289,6 +296,11 @@ impl Registry { .values() .chain(self.query_groups.values()) .flat_map(|route| route.peers.iter().map(|peer| peer.replica.node)) + .chain( + self.physical_groups + .values() + .flat_map(|route| route.peers.iter().map(|peer| peer.replica.node)), + ) .chain( self.origins .values() @@ -309,7 +321,7 @@ impl Registry { struct State { performance: Arc, domain: DomainToken, - runtime: RaftRuntime, + runtime: TransportRuntime, auth: Arc, tls: RaftTls, options: TransportOptions, @@ -326,7 +338,7 @@ struct State { accepted: AtomicU64, refused: AtomicU64, failed: AtomicU64, - claim: Mutex>, + claim: Mutex>>, snapshot_source: Mutex>>, prepared_snapshot: super::snapshot::prepared::Cache, snapshot_source_budget: Arc, @@ -403,6 +415,48 @@ impl RaftTransport { data: TcpListener, query: TcpListener, options: TransportOptions, + ) -> Result { + Self::start_runtime( + TransportRuntime::Legacy(runtime), + auth, + tls, + control, + data, + query, + options, + ) + .await + } + + pub async fn start_shared( + runtime: super::cell_groups::CellGroupRuntime, + auth: Arc, + tls: RaftTls, + control: TcpListener, + data: TcpListener, + query: TcpListener, + options: TransportOptions, + ) -> Result { + Self::start_runtime( + TransportRuntime::Shared(runtime), + auth, + tls, + control, + data, + query, + options, + ) + .await + } + + async fn start_runtime( + runtime: TransportRuntime, + auth: Arc, + tls: RaftTls, + control: TcpListener, + data: TcpListener, + query: TcpListener, + options: TransportOptions, ) -> Result { options.validate()?; let local_control = control.local_addr()?; @@ -437,6 +491,7 @@ impl RaftTransport { closed: false, tasks: 0, groups: BTreeMap::new(), + physical_groups: BTreeMap::new(), query_groups: BTreeMap::new(), origins: BTreeMap::new(), inspection_origins: BTreeMap::new(), @@ -570,6 +625,7 @@ impl RaftTransport { state.admit(®istry)?; for peer in &route.peers { registry.check_node_view(&RaftPeerNode::from(peer))?; + registry.check_physical_endpoint(&RaftPeerNode::from(peer))?; } if registry.query_groups.contains_key(&route.scope) { return Err(TransportError::Identity); @@ -594,7 +650,11 @@ impl RaftTransport { } } } - } else if registry.groups.len() + registry.query_groups.len() >= state.options.max_groups { + } else if registry.groups.len() + + registry.physical_groups.len() + + registry.query_groups.len() + >= state.options.max_groups + { return Err(TransportError::Overloaded); } // Existing peers' physical addresses/process must agree across groups. @@ -745,9 +805,12 @@ impl RaftTransport { header.set_sequence(sequence); let (result, wait) = oneshot::channel(); let request = SendRequest { - scope: scope.clone(), - route, - peer, + route: SendRoute::Legacy { + scope: scope.clone(), + route, + peer: peer.clone(), + }, + member: peer.replica.member, header, body, snapshot: message.get_msg_type() == MessageType::MsgSnapshot, @@ -1026,7 +1089,7 @@ impl State { query_items: qi + rqi, tasks: registry.tasks, peers: registry.pools.len(), - groups: registry.groups.len(), + groups: registry.groups.len() + registry.physical_groups.len(), accepted: self.accepted.load(Ordering::Acquire), refused: self.refused.load(Ordering::Acquire), failed: self.failed.load(Ordering::Acquire), @@ -1034,7 +1097,10 @@ impl State { } fn current(&self, request: &SendRequest) -> bool { - self.current_route(&request.scope, &request.route, &request.peer) + match &request.route { + SendRoute::Legacy { scope, route, peer } => self.current_route(scope, route, peer), + SendRoute::Physical { route, peer } => self.current_physical(route, peer), + } } fn current_route(&self, scope: &GroupScope, route: &Arc, peer: &RaftPeer) -> bool { @@ -1072,6 +1138,9 @@ impl State { } fn authorize_route(&self, wire: &Route) -> Result, TransportError> { + if wire.is_group() { + return self.authorize_physical(wire); + } let (source, target) = wire.decode()?; let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; self.admit(®istry)?; @@ -1327,13 +1396,7 @@ impl State { if header.lane != lane || header.sequence()? <= sequence { return Err(TransportError::Replay); } - let (source, _) = header.route.decode()?; - let host = ( - source.node, - source.disk, - source.process, - header.route.source_label().to_owned(), - ); + let host = header.route.source_host()?; if source_host.as_ref().is_some_and(|h| *h != host) { return Err(TransportError::StalePeer); } @@ -1377,7 +1440,6 @@ impl State { let message = header.verify_body(&body, &self.options)?; // Routing/configuration may have changed while reading the body. self.authorize(&header)?; - let scope = source.scope.clone(); let admission = self.performance.timer(match lane { Lane::Control => Operation::ControlAdmit, Lane::Data => Operation::DataAdmit, @@ -1385,7 +1447,7 @@ impl State { let result = self .bounded(Some(&pool), async { self.runtime - .admit_step(&scope, message) + .admit_route(&header.route, message) .await .map_err(TransportError::from) }) diff --git a/crates/cellar/durability/raft/transport_directory.rs b/crates/cellar/durability/raft/transport_directory.rs index ba220c088..dab5a62d1 100644 --- a/crates/cellar/durability/raft/transport_directory.rs +++ b/crates/cellar/durability/raft/transport_directory.rs @@ -102,6 +102,16 @@ impl RaftTransport { return Err(TransportError::Identity); } } + for old in registry + .physical_groups + .values() + .flat_map(|route| &route.peers) + .filter(|old| old.replica.node == peer.node) + { + if !same_registration(&RaftPeerNode::from(old), peer) { + return Err(TransportError::Identity); + } + } for old in registry .origins .values() @@ -129,6 +139,10 @@ impl RaftTransport { .values() .chain(registry.query_groups.values()) .any(|route| route.peers.iter().any(|old| old.replica.node == peer.node)) + || registry + .physical_groups + .values() + .any(|route| route.peers.iter().any(|old| old.replica.node == peer.node)) || registry .origins .values() @@ -142,6 +156,7 @@ impl RaftTransport { // Reservations include old workers until physical completion. A full // budget refuses the refresh without partially changing routing. let starts = state.prepare_pools(&mut registry, physical)?; + refresh_routes(&mut registry.physical_groups, peer); refresh_routes(&mut registry.groups, peer); refresh_routes(&mut registry.query_groups, peer); let Registry { @@ -188,7 +203,10 @@ impl RaftTransport { } } -fn refresh_routes(routes: &mut BTreeMap>, peer: &RaftPeerNode) { +fn refresh_routes( + routes: &mut BTreeMap>>, + peer: &RaftPeerNode, +) { for route in routes.values_mut() { if !route .peers diff --git a/crates/cellar/durability/raft/transport_frame.rs b/crates/cellar/durability/raft/transport_frame.rs index e9ddbd3ba..343f86000 100644 --- a/crates/cellar/durability/raft/transport_frame.rs +++ b/crates/cellar/durability/raft/transport_frame.rs @@ -8,8 +8,10 @@ use super::transport_options::TransportOptions; use crate::application::consensus::protocol::WireScope; use crate::peer_auth::PeerAuth; use ::raft::prelude::{Message, MessageType}; +use cellar_logic::cell_group::GroupHistory; use cellar_logic::consensus::{ - DiskLineage, GroupScope, MemberId, NodeId, ProcessGeneration, ReplicaIdentity, + ClusterId, DiskLineage, GroupId, IncarnationId, MemberId, NamespaceId, NodeId, + ProcessGeneration, ReplicaIdentity, }; use protobuf::Message as _; use serde::{Deserialize, Serialize}; @@ -31,10 +33,19 @@ struct WirePeer { label: String, } +// Both temporary cell adapters and physical groups use this marker-1 typed +// envelope. There is no reader for the old untagged scope layout. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "identity", deny_unknown_fields)] +enum History { + Cell(WireScope), + Group([[u8; 16]; 4]), +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(crate) struct Route { - scope: WireScope, + scope: History, schema: [u8; 32], source: WirePeer, target: WirePeer, @@ -56,7 +67,7 @@ impl Route { return Err(TransportError::Identity); } Ok(Self { - scope: WireScope::from(&source.scope), + scope: History::Cell(WireScope::from(&source.scope)), schema, source: WirePeer::new(source, source_label), target: WirePeer::new(target, target_label), @@ -64,8 +75,10 @@ impl Route { } pub fn decode(&self) -> Result<(ReplicaIdentity, ReplicaIdentity), TransportError> { - let scope = self - .scope + let History::Cell(scope) = &self.scope else { + return Err(TransportError::Identity); + }; + let scope = scope .clone() .decode() .map_err(|_| TransportError::Identity)?; @@ -76,6 +89,71 @@ impl Route { )) } + pub fn new_group( + source: &ReplicaIdentity, + source_label: &str, + target: &ReplicaIdentity, + target_label: &str, + schema: [u8; 32], + ) -> Result { + if source.scope != target.scope { + return Err(TransportError::Identity); + } + let history = source.scope; + Ok(Self { + scope: History::Group([ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + ]), + schema, + source: WirePeer::new(source, source_label), + target: WirePeer::new(target, target_label), + }) + } + + pub fn is_group(&self) -> bool { + matches!(self.scope, History::Group(_)) + } + + pub fn decode_group( + &self, + ) -> Result<(ReplicaIdentity, ReplicaIdentity), TransportError> + { + let History::Group(ids) = self.scope else { + return Err(TransportError::Identity); + }; + let scope = GroupHistory { + cluster: ClusterId::new(ids[0]).map_err(|_| TransportError::Identity)?, + namespace: NamespaceId::new(ids[1]).map_err(|_| TransportError::Identity)?, + group: GroupId::new(ids[2]).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(ids[3]).map_err(|_| TransportError::Identity)?, + }; + Ok((self.source.decode(scope)?, self.target.decode(scope)?)) + } + + fn validate(&self) -> Result<(), TransportError> { + if self.is_group() { + self.decode_group()?; + } else { + self.decode()?; + } + Ok(()) + } + + pub fn source_host( + &self, + ) -> Result<(NodeId, DiskLineage, ProcessGeneration, String), TransportError> { + let peer = self.source.decode(())?; + Ok(( + peer.node, + peer.disk, + peer.process, + self.source.label.clone(), + )) + } + pub fn schema(&self) -> [u8; 32] { self.schema } @@ -99,7 +177,7 @@ impl Route { } impl WirePeer { - fn new(replica: &ReplicaIdentity, label: &str) -> Self { + fn new(replica: &ReplicaIdentity, label: &str) -> Self { Self { node: replica.node.bytes(), disk: replica.disk.bytes(), @@ -109,7 +187,7 @@ impl WirePeer { } } - fn decode(&self, scope: GroupScope) -> Result { + fn decode(&self, scope: S) -> Result, TransportError> { crate::machine::validate_node_name(&self.label).map_err(|_| TransportError::Identity)?; Ok(ReplicaIdentity { scope, @@ -159,7 +237,7 @@ impl RequestHeader { return Err(TransportError::Protocol); } self.sequence()?; - self.route.decode()?; + self.route.validate()?; Ok(()) } @@ -179,7 +257,9 @@ impl RequestHeader { // number of decoded Entry objects. scan_message(bytes, options.max_entries)?; let message = Message::parse_from_bytes(bytes).map_err(|_| TransportError::Protocol)?; - let (source, target) = self.route.decode()?; + self.route.validate()?; + let source = self.route.source.decode(())?; + let target = self.route.target.decode(())?; if message.from != source.member.get() || message.to != target.member.get() || classify(&message)? != self.lane @@ -395,7 +475,7 @@ pub(crate) fn classify(message: &Message) -> Result { fn protocol_digest() -> [u8; 32] { Sha256::digest( - b"cellar-raft-peer-1:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", + b"cellar-raft-peer-1:typed-history:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", ) .into() } diff --git a/crates/cellar/durability/raft/transport_frame_tests.rs b/crates/cellar/durability/raft/transport_frame_tests.rs index 587c9fb2f..9d8bcd5bb 100644 --- a/crates/cellar/durability/raft/transport_frame_tests.rs +++ b/crates/cellar/durability/raft/transport_frame_tests.rs @@ -205,3 +205,58 @@ fn strict_header_identity_sequence_and_protocol() { bytes.insert_str(1, "\"sequence\":\"2\","); assert!(serde_json::from_str::(&bytes).is_err()); } + +#[test] +fn physical_history_is_typed_authenticated_and_refuses_old_wire() { + let physical = |n| { + let peer = replica(n); + ReplicaIdentity { + scope: GroupHistory::from(&peer.scope), + member: peer.member, + node: peer.node, + disk: peer.disk, + process: peer.process, + } + }; + let route = Route::new_group(&physical(1), "node-1", &physical(2), "node-2", [5; 32]).unwrap(); + assert!(route.decode().is_err()); + assert_eq!(route.decode_group().unwrap(), (physical(1), physical(2))); + let (mut request, body) = request(); + request.route = route.clone(); + let auth = PeerAuth::new([8; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let exporter = [7; 32]; + let head = request_head(&auth, &exporter, &request, body.len(), &options).unwrap(); + let decoded = head.request(&auth, &exporter).unwrap(); + assert!(decoded.verify_body(&body, &options).is_ok()); + let response = response_head( + &auth, + &exporter, + &request, + &head.tag, + Disposition::Accepted, + &options, + ) + .unwrap(); + let mut changed = request.clone(); + let mut target = physical(2); + target.scope.incarnation = IncarnationId::new([77; 16]).unwrap(); + let mut source = physical(1); + source.scope = target.scope; + changed.route = Route::new_group(&source, "node-1", &target, "node-2", [5; 32]).unwrap(); + assert_eq!( + response.response(&auth, &exporter, &changed, &head.tag), + Err(TransportError::Authentication) + ); + let mut invalid = route.clone(); + invalid.scope = History::Group([[0; 16]; 4]); + assert!(invalid.validate().is_err()); + let mut old = serde_json::to_value(route).unwrap(); + old["scope"] = serde_json::to_value(WireScope::from(&replica(1).scope)).unwrap(); + assert!(serde_json::from_value::(old).is_err()); + request.protocol = Sha256::digest( + b"cellar-raft-peer-1:tls13-exporter:hmac-sha256:protobuf2:bounded-strict:extent-stream", + ) + .into(); + assert_eq!(request.validate(), Err(TransportError::Protocol)); +} diff --git a/crates/cellar/durability/raft/transport_pipeline_tests.rs b/crates/cellar/durability/raft/transport_pipeline_tests.rs index d5a145613..24ff0c2f8 100644 --- a/crates/cellar/durability/raft/transport_pipeline_tests.rs +++ b/crates/cellar/durability/raft/transport_pipeline_tests.rs @@ -597,9 +597,12 @@ async fn control_window_drives_both_directions_with_one_byte_buffers() -> anyhow ); let charge = body.len() + state.options.max_header_bytes + frame::PREFIX + frame::TAG; requests.push(SendRequest { - scope, - route, - peer, + route: SendRoute::Legacy { + scope, + route, + peer: peer.clone(), + }, + member: peer.replica.member, header, body, snapshot: false, @@ -615,7 +618,9 @@ async fn control_window_drives_both_directions_with_one_byte_buffers() -> anyhow let (mut client, mut server) = tokio::io::duplex(1); let exporter = [7; 32]; let (sender, mut inbox) = mailbox::channel(WINDOW); - let pool = state.registry.lock().unwrap().pools[&requests[0].peer.replica.node].clone(); + let pool = state.registry.lock().unwrap().pools + [&requests[0].header.route.decode()?.1.node] + .clone(); for request in requests.drain(..) { sender .try_send(request) diff --git a/crates/cellar/durability/raft/transport_resources.rs b/crates/cellar/durability/raft/transport_resources.rs index 9bcef7b48..17fa0f033 100644 --- a/crates/cellar/durability/raft/transport_resources.rs +++ b/crates/cellar/durability/raft/transport_resources.rs @@ -100,7 +100,7 @@ impl ResourceObserver { ( registry.tasks, registry.pools.len(), - registry.groups.len(), + registry.groups.len() + registry.physical_groups.len(), registry.query_groups.len(), connections, limits, diff --git a/crates/cellar/durability/raft/transport_runtime.rs b/crates/cellar/durability/raft/transport_runtime.rs new file mode 100644 index 000000000..aac228432 --- /dev/null +++ b/crates/cellar/durability/raft/transport_runtime.rs @@ -0,0 +1,96 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Temporary adapter while the public host cuts over. A shared transport starts +//! directly on its physical runtime; it never constructs a legacy runtime. +use super::*; +use crate::raft::{cell_groups::CellGroupRuntime, storage::ReplicaLog}; + +pub(super) enum TransportRuntime { + Legacy(RaftRuntime), + Shared(CellGroupRuntime), +} + +impl TransportRuntime { + pub(super) fn log(&self) -> &ReplicaLog { + match self { + Self::Legacy(r) => r.log(), + Self::Shared(r) => r.log(), + } + } + + pub(super) fn admit(&self) -> Result<(), RuntimeError> { + match self { + Self::Legacy(r) => r.admit(), + Self::Shared(r) => r.admit(), + } + } + + pub(super) fn claim_transport(&self) -> Result, RuntimeError> { + match self { + Self::Legacy(r) => Ok(Box::new(r.claim_transport()?)), + Self::Shared(r) => Ok(Box::new(r.claim_transport()?)), + } + } + + pub(super) fn shared(&self) -> Result<&CellGroupRuntime, TransportError> { + match self { + Self::Shared(r) => Ok(r), + Self::Legacy(_) => Err(TransportError::Identity), + } + } + + pub(super) async fn admit_route( + &self, + route: &Route, + message: Message, + ) -> Result<(), RuntimeError> { + match self { + Self::Shared(runtime) => { + if message.get_msg_type() == MessageType::MsgSnapshot { + return Err(RuntimeError::Protocol( + "complete-group snapshot transport unavailable".into(), + )); + } + let (source, _) = route + .decode_group() + .map_err(|_| RuntimeError::Protocol("physical route".into()))?; + runtime.admit_step(source.scope, message).await + } + Self::Legacy(runtime) => { + let (source, _) = route + .decode() + .map_err(|_| RuntimeError::Protocol("legacy route".into()))?; + runtime.admit_step(&source.scope, message).await + } + } + } + + pub(super) async fn delivery_hint( + &self, + route: &SendRoute, + member: u64, + delivered: bool, + snapshot: bool, + ) -> Result<(), RuntimeError> { + match (self, route) { + (Self::Legacy(runtime), SendRoute::Legacy { scope, .. }) => { + if !delivered { + runtime.admit_unreachable(scope, member).await?; + } + if snapshot { + runtime + .admit_snapshot_status(scope, member, delivered) + .await?; + } + Ok(()) + } + (Self::Shared(runtime), SendRoute::Physical { route, .. }) if !snapshot => { + if !delivered { + runtime.report_unreachable(route.scope, member).await?; + } + Ok(()) + } + _ => Err(RuntimeError::Protocol("transport/runtime identity".into())), + } + } +} diff --git a/crates/cellar/durability/raft/transport_send.rs b/crates/cellar/durability/raft/transport_send.rs index 1de168d61..c12486c9a 100644 --- a/crates/cellar/durability/raft/transport_send.rs +++ b/crates/cellar/durability/raft/transport_send.rs @@ -190,17 +190,13 @@ impl State { let _ = self .bounded(Some(pool), async { self.runtime - .admit_unreachable(&request.scope, request.peer.replica.member.get()) + .delivery_hint( + &request.route, + request.member.get(), + false, + request.snapshot, + ) .await?; - if request.snapshot { - self.runtime - .admit_snapshot_status( - &request.scope, - request.peer.replica.member.get(), - false, - ) - .await?; - } Ok(()) }) .await; @@ -212,11 +208,7 @@ impl State { let _ = self .bounded(Some(pool), async { self.runtime - .admit_snapshot_status( - &request.scope, - request.peer.replica.member.get(), - true, - ) + .delivery_hint(&request.route, request.member.get(), true, true) .await?; Ok(()) }) diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs new file mode 100644 index 000000000..79fcfee1c --- /dev/null +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -0,0 +1,357 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical group routing on the existing node transport's queues and sockets. +use super::*; +use crate::raft::cache::GroupLog; + +type PhysicalRoute = GroupRoute; +type PhysicalPeer = RaftPeer; + +pub(super) enum SendRoute { + Legacy { + scope: GroupScope, + route: Arc, + peer: RaftPeer, + }, + Physical { + route: Arc, + peer: PhysicalPeer, + }, +} + +fn permitted( + group: &GroupLog, + peers: &[&ReplicaIdentity], +) -> Result<(), TransportError> { + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + let conf = &cache.conf_state; + for peer in peers { + if !cache.membership.permits_group(peer) + || !conf + .voters + .iter() + .chain(&conf.voters_outgoing) + .chain(&conf.learners) + .chain(&conf.learners_next) + .any(|member| *member == peer.member.get()) + { + return Err(TransportError::StalePeer); + } + } + Ok(()) +} + +impl RaftTransport { + /// Install already-authorized physical membership. This creates no group, + /// grants no serving authority and changes no committed configuration. + pub fn install_physical_group(&self, route: PhysicalRoute) -> Result<(), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + let group = runtime + .log() + .cell_group(&route.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if route.revision == 0 || route.peers.is_empty() || route.peers.len() > 128 { + return Err(TransportError::Invalid("physical group route".into())); + } + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + if cache + .group_manifest + .as_ref() + .is_none_or(|root| root.route().schema_digest != route.schema_digest) + { + return Err(TransportError::Identity); + } + drop(cache); + let local = group.identity(); + let mut members = BTreeSet::new(); + let mut nodes = BTreeSet::new(); + for peer in &route.peers { + if peer.replica.scope != route.scope + || !members.insert(peer.replica.member) + || !nodes.insert(peer.replica.node) + { + return Err(TransportError::Identity); + } + RaftPeerNode::from(peer).validate_endpoint()?; + permitted(&group, &[&peer.replica])?; + } + let own = route + .peers + .iter() + .find(|p| p.replica == *local) + .ok_or(TransportError::Identity)?; + if own.label != state.auth.source() + || own.control_address != state.local_control + || own.data_address != state.local_data + || own.query_address != state.local_query + { + return Err(TransportError::Identity); + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + for peer in &route.peers { + registry.check_node_view(&RaftPeerNode::from(peer))?; + } + if let Some(old) = registry.physical_groups.get(&route.scope) { + if **old == route { + return Ok(()); + } + if old.revision >= route.revision || old.schema_digest != route.schema_digest { + return Err(TransportError::StalePeer); + } + for peer in &route.peers { + if old.peers.iter().any(|p| { + p.replica.member == peer.replica.member + && (p.replica.node != peer.replica.node + || p.replica.disk != peer.replica.disk) + }) { + return Err(TransportError::Identity); + } + } + } else if registry.groups.len() + + registry.query_groups.len() + + registry.physical_groups.len() + >= state.options.max_groups + { + return Err(TransportError::Overloaded); + } + let physical: Vec<_> = route + .peers + .iter() + .filter(|p| p.replica.node != local.node) + .map(RaftPeerNode::from) + .collect(); + for peer in &physical { + if registry + .physical_groups + .values() + .filter(|view| view.scope != route.scope) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .origins + .values() + .chain(registry.inspection_origins.values()) + .filter_map(|g| g.peer.as_ref()) + .any(|other| other.node == peer.node && other != peer) + { + return Err(TransportError::StalePeer); + } + } + let starts = state.prepare_pools(&mut registry, &physical)?; + registry + .physical_groups + .insert(route.scope, Arc::new(route)); + registry.prune_pools(); + drop(registry); + state.start_pools(starts) + } + + pub fn remove_physical_group(&self, history: &GroupHistory) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry.physical_groups.remove(history); + registry.prune_pools(); + Ok(()) + } + + /// Queue a physical-group frame on existing bounded per-node lanes. Delivery + /// acknowledges receiver admission only. Complete group snapshots require CG-05. + pub fn submit_physical( + &self, + history: &GroupHistory, + message: Message, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if message.get_msg_type() == MessageType::MsgSnapshot { + return Err(TransportError::Refused); + } + let lane = frame::classify(&message)?; + let size = message.compute_size() as usize; + if size > frame::payload_limit(lane, &state.options) { + return Err(TransportError::Limit); + } + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let route = registry + .physical_groups + .get(history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + let group = runtime + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + let local = group.identity().clone(); + if message.from != local.member.get() { + return Err(TransportError::Identity); + } + let peer = route + .peers + .iter() + .find(|p| p.replica.member.get() == message.to) + .ok_or(TransportError::Identity)? + .clone(); + if peer.replica.node == local.node { + return Err(TransportError::Identity); + } + permitted(&group, &[&local, &peer.replica])?; + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + drop(registry); + let charge = size + state.options.max_header_bytes + frame::PREFIX + frame::TAG; + let node = state + .outgoing + .lane(lane) + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_permit = pool + .outgoing + .lane(lane) + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let body = message + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + frame::scan_message(&body, state.options.max_entries)?; + let wire = Route::new_group( + &local, + state.auth.source(), + &peer.replica, + &peer.label, + route.schema_digest, + )?; + let mut header = RequestHeader::new(wire, 1, lane, &body); + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if !registry + .physical_groups + .get(history) + .is_some_and(|r| Arc::ptr_eq(r, &route)) + || !registry + .pools + .get(&peer.replica.node) + .is_some_and(|p| Arc::ptr_eq(p, &pool)) + { + return Err(TransportError::StalePeer); + } + let sequence = state + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + + 1; + header.set_sequence(sequence); + let (result, wait) = oneshot::channel(); + let request = SendRequest { + member: peer.replica.member, + route: SendRoute::Physical { route, peer }, + header, + body, + snapshot: false, + result: Some(result), + queued: Some(state.performance.timer(match lane { + Lane::Control => Operation::ControlQueue, + Lane::Data => Operation::DataQueue, + })), + _node: node, + _peer: peer_permit, + }; + pool.tx[lane_index(lane)].try_send(request).map_err(|e| { + let error = match &e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + }; + if let Some(timer) = e.into_inner().queued.take() { + timer.finish(false); + } + error + })?; + Ok(Delivery { result: wait }) + } +} + +impl State { + pub(super) fn current_physical(&self, route: &Arc, peer: &PhysicalPeer) -> bool { + self.registry.lock().is_ok_and(|r| { + self.admit(&r).is_ok() + && r.physical_groups + .get(&route.scope) + .is_some_and(|current| Arc::ptr_eq(current, route)) + && r.pools + .get(&peer.replica.node) + .is_some_and(|pool| pool.peer.same_host(peer)) + && self.runtime.shared().is_ok_and(|runtime| { + runtime.log().cell_group(&route.scope).is_ok_and(|group| { + permitted(&group, &[group.identity(), &peer.replica]).is_ok() + }) + }) + }) + } + + pub(super) fn authorize_physical(&self, wire: &Route) -> Result, TransportError> { + let runtime = self.runtime.shared()?; + let (source, target) = wire.decode_group()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + let peer = route + .peers + .iter() + .find(|p| p.replica == source && p.label == wire.source_label()) + .ok_or(TransportError::StalePeer)?; + permitted(&group, &[&source, &target])?; + registry + .pools + .get(&source.node) + .filter(|pool| pool.peer.same_host(peer)) + .cloned() + .ok_or(TransportError::StalePeer) + } +} + +impl Registry { + pub(super) fn check_physical_endpoint( + &self, + peer: &RaftPeerNode, + ) -> Result<(), TransportError> { + if self + .physical_groups + .values() + .flat_map(|route| &route.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + { + return Err(TransportError::StalePeer); + } + Ok(()) + } +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index f15e67be8..e4354e5c4 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1340,3 +1340,18 @@ async fn catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> async fn rooted_catalog_routed_commands_preserve_producer_and_receipts() -> Result<()> { Box::pin(runtime_fixture::routed_commands(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn physical_group_transport_multiplexes_real_raft() -> Result<()> { + Box::pin(runtime_fixture::physical_transport()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn catalog_commands_replicate_over_physical_tls() -> Result<()> { + Box::pin(runtime_fixture::commands_over_tcp(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_catalog_commands_replicate_over_physical_tls() -> Result<()> { + Box::pin(runtime_fixture::commands_over_tcp(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 599e025f5..229c5e5ac 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -5,7 +5,12 @@ use super::*; use std::collections::BTreeMap; +#[path = "cell_group_tcp.rs"] +mod tcp; +pub(super) use tcp::physical_transport; + struct RuntimeCluster { + transports: Vec, appliers: BTreeMap<(usize, GroupHistory), (CellGroupSqlPool, CellGroupApplier)>, cluster: Cluster, runtimes: Vec, @@ -66,6 +71,7 @@ impl RuntimeCluster { } let applied = BTreeMap::from([(cluster.birth.route().history, applied)]); Ok(Self { + transports: Vec::new(), appliers: BTreeMap::new(), cluster, runtimes, @@ -129,6 +135,16 @@ impl RuntimeCluster { } } let messages = std::mem::take(&mut self.network); + if !self.transports.is_empty() { + for (history, message) in messages { + let transport = &self.transports[message.from as usize - 1]; + transport + .submit_physical(&history, message)? + .accepted() + .await?; + } + return Ok(()); + } let runtimes = &self.runtimes; for (scope, message, result) in futures_util::future::join_all(messages.into_iter().map( |(scope, message)| async move { @@ -200,6 +216,10 @@ impl RuntimeCluster { } async fn close(self) -> Result<()> { + for transport in &self.transports { + transport.close().await; + } + drop(self.transports); for (pool, _) in self.appliers.values() { pool.close().await?; } @@ -673,3 +693,7 @@ pub(super) use sql::registered_queries; pub(super) use sql::routed_queries; pub(super) use sql::routed_commands; + +pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { + Box::pin(sql::commands_over_tcp(rooted)).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 48796f0d2..c0de46950 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -525,3 +525,7 @@ pub(crate) use dispatch::routed as routed_queries; #[path = "cell_group_sql_command_dispatch.rs"] mod command_dispatch; pub(crate) use command_dispatch::commands as routed_commands; + +pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { + Box::pin(command_dispatch::commands_with_transport(rooted, true)).await +} diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs index a7b2e2ae1..e4b2f47dd 100644 --- a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -43,6 +43,10 @@ async fn committed(invocation: &mut CellCommandInvocation) -> Result Result<()> { + Box::pin(commands_with_transport(rooted, false)).await +} + +pub(crate) async fn commands_with_transport(rooted: bool, tcp: bool) -> Result<()> { let calls = Arc::new(AtomicUsize::new(0)); let captured = Arc::new(Mutex::new(Vec::new())); let count = calls.clone(); @@ -80,6 +84,9 @@ pub(crate) async fn commands(rooted: bool) -> Result<()> { })?, ); let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + if tcp { + fixture.connect_tcp().await?; + } let keys = VersionKeys::new( VersionSigningKey::new([94; 32], u64::MAX)?, vec![], diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs new file mode 100644 index 000000000..a56458156 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -0,0 +1,221 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Three physical runtimes use the production transport and real TLS sockets. +use super::*; +use cellar::application::consensus::verification::GroupRoute as TransportRoute; +use cellar::peer_auth::PeerAuth; +use tokio::net::TcpListener; + +fn tls() -> Result { + Ok(RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} + +async fn start(runtime: &CellGroupRuntime, n: usize) -> Result { + RaftTransport::start_shared( + runtime.clone(), + Arc::new( + PeerAuth::new([17; 32], format!("node-{}", n + 1)) + .map_err(|_| TransportError::Identity)?, + ), + tls().map_err(|e| TransportError::Invalid(e.to_string()))?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TransportOptions::default(), + ) + .await +} + +impl RuntimeCluster { + pub(super) async fn connect_tcp(&mut self) -> Result<()> { + assert!(self.transports.is_empty()); + assert!(self.network.is_empty()); + for (n, runtime) in self.runtimes.iter().enumerate() { + self.transports.push(start(runtime, n).await?); + } + for history in self.applied.keys() { + let route = self.tcp_route(*history)?; + for transport in &self.transports { + transport.install_physical_group(route.clone())?; + } + } + Ok(()) + } + + fn tcp_route(&self, history: GroupHistory) -> Result> { + let marker = self.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()? + .context("manifest")?; + Ok(TransportRoute { + scope: history, + schema_digest: marker.route().schema_digest, + revision: 1, + peers: self + .transports + .iter() + .enumerate() + .map(|(n, transport)| -> Result<_> { + Ok(RaftPeer { + replica: self.cluster.voters[n] + .log + .cell_group(&history)? + .identity() + .clone(), + label: format!("node-{}", n + 1), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }) + }) + .collect::>>()?, + }) + } +} + +pub(crate) async fn physical_transport() -> Result<()> { + let mut fixture = RuntimeCluster::new().await?; + let second = fixture.add_group(31).await?; + let histories = [fixture.cluster.birth.route().history, second]; + fixture.connect_tcp().await?; + assert!(matches!( + start(&fixture.runtimes[0], 0).await, + Err(TransportError::Runtime(RuntimeError::AlreadyAttached)) + )); + for transport in &fixture.transports { + assert_eq!(transport.usage().groups, 2); + assert_eq!(transport.usage().peers, 2); + } + for history in histories { + fixture.runtimes[0].campaign(history).await?; + } + fixture.settle(&histories).await?; + for group in histories { + let route = fixture.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")? + .route() + .clone(); + for name in ["tls-a", "tls-b", "tls-c"] { + let cell = history(&route, name)?; + let bytes = + images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES (1);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose(group, request(&route, cell, 1, &cut)?) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + while fixture.applied[&group] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + for voter in &fixture.cluster.voters { + assert_eq!( + voter + .log + .cell_group(&group)? + .group_manifest()? + .context("applied")? + .root() + .cells(), + 3 + ); + } + } + let route = fixture.tcp_route(histories[0])?; + let mut heartbeat = Message { + from: 1, + to: 2, + term: fixture.runtimes[0].status(histories[0]).await?.term.get(), + ..Default::default() + }; + heartbeat.set_msg_type(MessageType::MsgHeartbeat); + let mut snapshot = heartbeat.clone(); + snapshot.set_msg_type(MessageType::MsgSnapshot); + assert!(matches!( + fixture.transports[0].submit_physical(&histories[0], snapshot), + Err(TransportError::Refused) + )); + let mut unknown = histories[0]; + unknown.group = GroupId::new([222; 16])?; + assert!(matches!( + fixture.transports[0].submit_physical(&unknown, heartbeat.clone()), + Err(TransportError::UnknownGroup) + )); + let mut wrong = route.clone(); + wrong.schema_digest[0] ^= 1; + assert!(matches!( + fixture.transports[0].install_physical_group(wrong), + Err(TransportError::Identity) + )); + let mut wrong = route.clone(); + wrong.revision += 1; + wrong.peers[0].replica.process = ProcessGeneration::new([222; 16])?; + assert!(matches!( + fixture.transports[0].install_physical_group(wrong), + Err(TransportError::Identity) + )); + let mut conflicting = RaftPeerNode::from(&route.peers[1]); + conflicting.process = ProcessGeneration::new([223; 16])?; + assert!(matches!( + fixture.transports[0].grant_query_origin(NamespaceQueryOrigin { + cluster: histories[0].cluster, + namespace: histories[0].namespace, + revision: 1, + peer: conflicting.clone(), + }), + Err(TransportError::StalePeer) + )); + assert!(matches!( + fixture.transports[0].grant_inspection_origin(NamespaceInspectionOrigin { + cluster: histories[0].cluster, + namespace: histories[0].namespace, + revision: 1, + peer: conflicting, + }), + Err(TransportError::StalePeer) + )); + fixture.transports[1].remove_physical_group(&histories[0])?; + assert_eq!(fixture.transports[1].usage().groups, 1); + assert_eq!(fixture.transports[1].usage().peers, 2); + assert!(fixture.transports[0] + .submit_physical(&histories[0], heartbeat.clone())? + .accepted() + .await + .is_err()); + fixture.transports[1].install_physical_group(route.clone())?; + fixture.transports[0] + .submit_physical(&histories[0], heartbeat)? + .accepted() + .await?; + for transport in &fixture.transports { + assert!(transport.usage().accepted > 0); + } + fixture.transports[0].close().await; + assert_eq!(fixture.transports[0].usage().tasks, 0); + let replacement = start(&fixture.runtimes[0], 0).await?; + replacement.close().await; + fixture.close().await +} diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index bd929cb22..73ad6a446 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 31 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 34 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 0f587d255..6a9e50346 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -487,7 +487,10 @@ "crates/cellar/durability/raft/cell_groups/dispatch/mod.rs", "crates/cellar/tests/support/cell_group_sql_dispatch.rs", "crates/cellar/durability/raft/cell_groups/dispatch/command.rs", - "crates/cellar/tests/support/cell_group_sql_command_dispatch.rs" + "crates/cellar/tests/support/cell_group_sql_command_dispatch.rs", + "crates/cellar/durability/raft/transport_runtime.rs", + "crates/cellar/durability/raft/transport_shared.rs", + "crates/cellar/tests/support/cell_group_tcp.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -571,6 +574,14 @@ "catalog_routed_commands_preserve_producer_and_receipts", "rooted_catalog_routed_commands_preserve_producer_and_receipts" ] + }, + { + "action": "Connect physical group histories to the existing node-owned peer transport without constructing the legacy runtime. Share TLS pools, lanes, byte/item banks, revocation and joined shutdown. Bind typed marker-1 history and full replica identity to authenticated requests/responses; refuse old untagged wire layouts and unsupported group snapshots. Include physical routes in all resource and endpoint-conflict checks. The temporary legacy runtime/frame adapter must be deleted with the old host in CG-16. Qualify actual multi-group replication and routed command settlement over TLS before integrating the production controller.", + "tests": [ + "physical_group_transport_multiplexes_real_raft", + "catalog_commands_replicate_over_physical_tls", + "rooted_catalog_commands_replicate_over_physical_tls" + ] } ], "test_cases": [ @@ -860,6 +871,35 @@ "I03", "I12" ] + }, + { + "name": "physical_group_transport_multiplexes_real_raft", + "oracle": "Two physical Raft groups on three real TLS endpoints replicate three independent SQLite cells each using the same bounded per-peer connection pools. Unknown groups, wrong schema/local process, conflicting query/inspection endpoints and unsupported snapshots refuse. Removing one group preserves the other group and peer pools; restored routing delivers again. Exclusive transport ownership refuses a second service until joined shutdown.", + "invariants": [ + "I01", + "I03", + "I12" + ] + }, + { + "name": "catalog_commands_replicate_over_physical_tls", + "oracle": "Run the catalog-routed registered command cancellation, duplicate acceptance, exact receipt and real leader-transfer retry assertions over production physical-group TLS delivery in this image mode. Preserve captured command inputs and original producer ownership; no fixture-direct peer delivery occurs after transport attachment. The fixture still drives runtime apply and completion events, so this does not qualify production controller ownership.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_catalog_commands_replicate_over_physical_tls", + "oracle": "Run the catalog-routed registered command cancellation, duplicate acceptance, exact receipt and real leader-transfer retry assertions over production physical-group TLS delivery in this image mode. Preserve captured command inputs and original producer ownership; no fixture-direct peer delivery occurs after transport attachment. The fixture still drives runtime apply and completion events, so this does not qualify production controller ownership.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ @@ -2291,7 +2331,7 @@ "test_target": "native_cell_groups_release", "steps": [ { - "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code. Remove QueryVersion::GroupCut and RaftClass::invoke_group_query along with the old query adapters; final QueryContext exposes the selected cell observation directly.", + "action": "Audit every production registration, scope codec, image marker, archive, SDK and operator path for the old cell-equals-group assumption. Remove temporary staging adapters and legacy runtime choices; leave explicit one-cell occupancy on the common code. Remove QueryVersion::GroupCut and RaftClass::invoke_group_query along with the old query adapters; final QueryContext exposes the selected cell observation directly. Remove TransportRuntime::Legacy, the typed cell-history transport frame arm and old transport entry points; the final shared transport must not retain a legacy serving mode.", "tests": [ "no_legacy_per_cell_consensus_path" ] From 8b70fa97f88992d1cddcf11edcb0877f1904b1ac Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 04:17:07 -0700 Subject: [PATCH 086/568] docs(raft): qualify physical group transport and prioritize command ownership --- docs/design/multi-raft/cell-groups/README.md | 10 ++++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 25 ++++++--- .../design/multi-raft/cell-groups/progress.md | 55 +++++++++++++++++++ 4 files changed, 85 insertions(+), 9 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 344e46dc8..4d8071870 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -163,6 +163,16 @@ source inputs and four executables. Production controller ownership after ingres cancellation and physical-group transport integration are the next critical gaps, followed by routed enrollment and the public SDK cutover. See [progress](progress.md). +Physical-group peer transport now starts directly on the shared runtime at +`eb99a242` and reuses existing TLS pools, pipelines, budgets and shutdown ownership. +Typed group histories refuse old untagged frames; unsupported group snapshots +refuse. All 192 native cases, 91 transport-filtered library cases and nine +commands pass. Transfer `.cellar/cell-groups-evidence/CG-04-physical-transport/` +with prior evidence (241 source inputs, eight executables). The next critical +gap is production controller ownership and reconciliation after caller disconnect, +followed by routed enrollment and SDK/shared-query RPC integration. The fixtures +still drive runtime events. See [progress](progress.md) for exact limits. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 8ff71f314..52aa47b06 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "1f014f42264c73b1d5e3cd387f65ffed45140ecb", + "source_revision": "eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index b0e59c8b7..ba5aca3a5 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,16 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `1f014f42`, adding retained routed +The newest reviewed source baseline is `eb99a242`, connecting physical groups +to the existing production TLS transport without constructing the legacy runtime. +Its partial record is `.cellar/cell-groups-evidence/CG-04-physical-transport/checkpoint.json`: +192 native cases, 91 library cases, nine commands, 241 source inputs and +eight selected executables. Transfer that directory with prior evidence. The +three new native cases qualify shared peer pools and routed command settlement +over TLS in both image modes. They still use fixture-driven runtime events, not +a production controller after ingress cancellation. Complete-group snapshots +explicitly refuse. CG-04 remains IN_PROGRESS. +The preceding reviewed source baseline is `1f014f42`, adding retained routed command phases over the existing catalog, consensus and SQL owners. Its partial record is `.cellar/cell-groups-evidence/CG-04-routed-commands-qualified/checkpoint.json`: 105 native cases, eight commands, 229 source inputs and four selected executables. @@ -373,9 +382,9 @@ Do not let plan maintenance replace implementation. 1. Integrate the phased command invocation into the production node controller. Retain it independently from ingress waiters and drive existing shared-runtime - transport, apply and completion events through the existing owners. The current - `RaftTransport` routes by old `GroupScope`; adapt its route/authentication boundary - to physical `GroupHistory` instead of manufacturing a cell-scoped identity. Qualify + transport, apply and completion events through the existing owners. Physical + `GroupHistory` routing now works over `RaftTransport::start_shared`; use that + transport directly in the production event loop. Qualify caller disconnect after SQL entry and after provisional acceptance: the controller must resolve the exact receipt or a proven non-commit and release bounded ownership without re-executing an uncertain command. The current test @@ -385,7 +394,8 @@ Do not let plan maintenance replace implementation. infer non-commit from missing local state. Keep original deadlines for responses while giving the controller an explicit bounded reconciliation lifecycle. Then add catalog-routed enrollment with stable creation identity and exact - creation settlement. Qualify enrollment, command, lost response and linearizable + creation settlement, and integrate shared query RPC. Qualify enrollment, + command, lost response and linearizable query together. Public `RaftCellHandle` still uses its old host binding; the real-engine fixtures still pump transport/apply events. Reuse existing worker and transport ownership with joined shutdown. Requested and namespace S3 @@ -396,8 +406,9 @@ Do not let plan maintenance replace implementation. faults and joined cold restart. Full snapshots/membership follow in CG-05, archive/reclamation in CG-06, and shared effect scheduling/cache lifecycle in CG-08 before the CG-07 public cutover. -3. Remove `QueryVersion::GroupCut` and `RaftClass::invoke_group_query` with the old - host by CG-16. They explicitly preserve the old in-memory group observation +3. Remove `TransportRuntime::Legacy`, the typed cell-scoped frame arm and old + transport entry points in CG-16. Also remove `QueryVersion::GroupCut` and + `RaftClass::invoke_group_query` with the old host by CG-16. They explicitly preserve the old in-memory group observation during integration; they must not become a permanent alternate serving mode. Final `QueryContext` exposes the actual cell observation directly. Never derive a cell sequence or witness from a group log position. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 5fc7171ca..7cf0bb55c 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,61 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 physical-group TLS transport + +Qualified source: `eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa`. + +`RaftTransport::start_shared` starts directly on the physical-group runtime. +`GroupRoute` and typed peers share the existing node-owned TLS +connections, control/data pipelines, byte/item budgets and joined shutdown. +Route installation requires an existing selected group, its schema, committed +member/disk binding and exact local process/listeners. Authorization rechecks +current routes and membership through queueing, I/O and response delivery. +Physical routes participate in node-view refresh, pool revocation/pruning and +combined group limits. Conflicting query/inspection endpoints refuse. + +Marker-1 frames use typed histories and a new protocol digest. Physical consensus +has no cell key; old untagged wire layouts refuse. Requests and responses bind +full replica identity, schema, sequence, payload and TLS channel. The exclusive +runtime transport claim survives until physical tasks exit. Delivery is only +receiver admission, not a Raft persistence or commit proof. Unsupported +complete-group snapshot messages refuse. The temporary legacy runtime/frame arms +are explicitly scheduled for deletion with the old host in CG-16. + +Three new native cases use real TLS endpoints. Two groups replicate three +independent SQLite cells each on the same peer pools. Unknown groups, wrong +schema/local process, conflicting grants and unsupported snapshots refuse. +Removing one group's route retains the other group and its peer pools; restoring +it delivers again. A second transport refuses until joined close releases its +claim. Existing routed command cancellation, exact receipt and leader-transfer +assertions also pass over TLS in both image modes. A new frame case verifies +history/incarnation binding and refusal of old wire/protocol layouts. + +All 283 selected Rust cases pass: 192 native cases across seven suites and +91 library cases selected by the transport filter. All nine commands pass, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting (1,826 Rust files), layout, dependency isolation, docs, source preflight +and all 37 plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-physical-transport/checkpoint.json` binds +241 source inputs and eight selected executables. Initial private-import and +legacy-membership-validator compile failures, the corrected targeted run and +formatter log are retained. Four process helpers run through their parent cases; +the RustFS-specific seed case is skipped. Prior checkpoints retain their original +source/scope. These local endpoints share a process; they do not qualify independent failure +domains, RustFS, power loss or load. Latest fetched `origin/main` is included and +the requested MR-13/MR-14 progress JSON files remain excluded. + +CG-04 remains IN_PROGRESS. The critical next gap is the production controller: +retain command invocations independently from callers, drive existing shared +transport/apply/completion events, and maintain bounded reconciliation after the +response deadline. The fixtures still pump events, so a cancelled preparation +future is not evidence of controller ownership after ingress disconnect. Reuse +the now-working physical transport and SQL/apply owners; qualify disconnects after +SQL entry and provisional acceptance without replaying ambiguous callbacks. Then +complete routed enrollment and public SDK/shared-query RPC integration. +Configuration/cache lifecycle, complete-group snapshots/archive, effects, +topology, real RustFS/load and final legacy removal remain later requirements. + ## CG-04 catalog-routed command phases Qualified source: `1f014f42264c73b1d5e3cd387f65ffed45140ecb`. From 61aa775206fdc842848952ffb042a7e1ca0d310e Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 04:38:55 -0700 Subject: [PATCH 087/568] feat: drive shared cell-group events with a node controller --- .../raft/cell_groups/controller/mod.rs | 240 ++++++++++ .../raft/cell_groups/controller/worker.rs | 205 +++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../raft/cell_groups/runtime/mod.rs | 12 +- .../raft/host_sql/cell_groups/mod.rs | 4 + .../durability/raft/transport_shared.rs | 9 + .../tests/native_cell_groups_execution.rs | 30 ++ .../tests/support/cell_group_controller.rs | 413 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 3 + crates/cellar/tests/support/cell_group_sql.rs | 6 + 10 files changed, 926 insertions(+), 1 deletion(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/worker.rs create mode 100644 crates/cellar/tests/support/cell_group_controller.rs diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs new file mode 100644 index 000000000..957dfb13b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -0,0 +1,240 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owns a fixed roster's consensus events, transport dispatch and physical apply. +//! This infrastructure owner does not yet own ingress command invocations. + +mod worker; + +use super::{CellGroupApplier, CellGroupRuntime, CellGroupRuntimeEvents, CellGroupSqlPool}; +use crate::raft::{ + completion::RuntimeError, + execution::ExecutionError, + transport::{RaftTransport, TransportError}, +}; +use futures_util::FutureExt; +use std::collections::BTreeMap; +use std::sync::{ + atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, + Arc, Mutex, +}; +use tokio::sync::Notify; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupControllerLimits { + pub max_groups: usize, + pub max_parallel_apply: usize, + /// One node timer backs off transient SQL admission, never one timer per cell. + pub retry_interval_ms: u64, +} + +#[derive(Clone, Debug)] +pub enum CellGroupControllerError { + Invalid(&'static str), + Runtime(RuntimeError), + Apply(Arc), + Transport(TransportError), + Closed, +} + +impl std::fmt::Display for CellGroupControllerError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group controller: {self:?}") + } +} + +impl std::error::Error for CellGroupControllerError {} + +impl From for CellGroupControllerError { + fn from(error: RuntimeError) -> Self { + Self::Runtime(error) + } +} + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupControllerUsage { + pub groups: usize, + pub active_apply: usize, + pub pending_apply: usize, + pub applied: u64, + pub apply_retries: u64, + /// Immediate queue/route refusals. The transport tracks later I/O failures. + pub send_refusals: u64, + pub closing: bool, +} + +struct Shared { + groups: usize, + active_apply: AtomicUsize, + pending_apply: AtomicUsize, + applied: AtomicU64, + apply_retries: AtomicU64, + send_refusals: AtomicU64, + closing: AtomicBool, + stop: Notify, + terminal: Mutex>>, + finished: Notify, +} + +impl Shared { + fn stop(&self) { + self.closing.store(true, Ordering::Release); + self.stop.notify_one(); + } + + fn complete(&self, result: Result<(), CellGroupControllerError>) { + let mut terminal = self.terminal.lock().unwrap_or_else(|e| e.into_inner()); + if terminal.is_none() { + *terminal = Some(result); + } + drop(terminal); + self.finished.notify_waiters(); + } + + async fn wait(&self) -> Result<(), CellGroupControllerError> { + loop { + let finished = self.finished.notified(); + tokio::pin!(finished); + finished.as_mut().enable(); + let result = self + .terminal + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clone(); + if let Some(result) = result { + return result; + } + finished.await; + } + } +} + +struct Handle(Arc); + +impl Drop for Handle { + fn drop(&mut self) { + self.0.stop(); + } +} + +/// One worker per node, with a bounded fixed roster and no cell-specific tasks. +/// Dropping the final public handle requests joined shutdown. An explicit close +/// is required to observe its result. Runtime/transport/pool clones remain +/// usable during operation; closing this owner closes all three services. +#[derive(Clone)] +pub struct CellGroupController { + handle: Arc, +} + +impl CellGroupController { + /// Consume the exclusive event receiver after validating exact engine and + /// runtime ownership. Every listed group must already be attached. Later + /// attachment requires a future roster API; an unknown event fails closed. + pub async fn start( + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + appliers: Vec, + limits: CellGroupControllerLimits, + ) -> Result { + if limits.max_groups == 0 + || limits.max_groups > runtime.log().limits().max_groups + || limits.max_parallel_apply == 0 + || limits.max_parallel_apply > limits.max_groups.min(256) + || !(1..=60_000).contains(&limits.retry_interval_ms) + || appliers.is_empty() + || appliers.len() > limits.max_groups + { + return Err(CellGroupControllerError::Invalid( + "controller limits/roster", + )); + } + if !events.belongs_to(&runtime) + || !transport.uses_shared_runtime(&runtime) + || !sql.belongs_to_engine(&runtime.log().owner()) + { + return Err(CellGroupControllerError::Invalid("controller owner")); + } + let mut roster = BTreeMap::new(); + for applier in appliers { + if !applier.belongs_to_engine(&runtime.log().owner()) + || roster.insert(applier.history(), applier).is_some() + { + return Err(CellGroupControllerError::Invalid("controller applier")); + } + } + for history in roster.keys() { + runtime.status(*history).await?; + } + runtime.admit()?; + let shared = Arc::new(Shared { + groups: roster.len(), + active_apply: AtomicUsize::new(0), + pending_apply: AtomicUsize::new(0), + applied: AtomicU64::new(0), + apply_retries: AtomicU64::new(0), + send_refusals: AtomicU64::new(0), + closing: AtomicBool::new(false), + stop: Notify::new(), + terminal: Mutex::new(None), + finished: Notify::new(), + }); + let handle = Self { + handle: Arc::new(Handle(shared.clone())), + }; + let domain = runtime.log().domain(); + let worker_domain = domain.clone(); + // Infrastructure cleanup is not cancelled with application tasks. The + // worker retains tickets/appliers until all physical owners have joined. + domain + .spawn_cleanup(async move { + let mut worker = worker::Worker::new( + runtime, + events, + transport, + sql, + roster, + limits, + shared.clone(), + ); + let result = std::panic::AssertUnwindSafe(worker.run(&worker_domain)) + .catch_unwind() + .await + .unwrap_or(Err(CellGroupControllerError::Runtime( + RuntimeError::Protocol("controller worker panicked".into()), + ))); + shared.stop(); + let joined = worker.finish().await; + drop(worker); + shared.active_apply.store(0, Ordering::Release); + shared.pending_apply.store(0, Ordering::Release); + shared.complete(result.and(joined)); + }) + .map_err(|_| CellGroupControllerError::Closed)?; + Ok(handle) + } + + pub fn usage(&self) -> CellGroupControllerUsage { + let shared = &self.handle.0; + CellGroupControllerUsage { + groups: shared.groups, + active_apply: shared.active_apply.load(Ordering::Acquire), + pending_apply: shared.pending_apply.load(Ordering::Acquire), + applied: shared.applied.load(Ordering::Relaxed), + apply_retries: shared.apply_retries.load(Ordering::Relaxed), + send_refusals: shared.send_refusals.load(Ordering::Relaxed), + closing: shared.closing.load(Ordering::Acquire), + } + } + + /// Wait for terminal failure or joined shutdown without requesting a stop. + pub async fn wait(&self) -> Result<(), CellGroupControllerError> { + self.handle.0.wait().await + } + + /// Cancelling this waiter does not cancel transport, persistence or SQL joins. + pub async fn close(&self) -> Result<(), CellGroupControllerError> { + self.handle.0.stop(); + self.wait().await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs new file mode 100644 index 000000000..a27cbde2f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -0,0 +1,205 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::{CellGroupApply, CellGroupEffect, CellGroupPreparedApply}; +use cellar_logic::{cell_group::GroupHistory, consensus::CommandFailure}; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; +use std::collections::{BTreeSet, VecDeque}; + +enum Work { + Prepare(Box), + Publish(Box), +} + +struct Pending { + group: GroupHistory, + work: Work, + retry_at: u64, +} + +type Completed = (GroupHistory, Result, CellGroupControllerError>); + +pub(super) struct Worker { + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + roster: BTreeMap, + limits: CellGroupControllerLimits, + shared: Arc, + pending: VecDeque, + active: BTreeSet, + jobs: FuturesUnordered>, + hints: FuturesUnordered>, +} + +impl Worker { + pub(super) fn new( + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + sql: CellGroupSqlPool, + roster: BTreeMap, + limits: CellGroupControllerLimits, + shared: Arc, + ) -> Self { + Self { + runtime, + events, + transport, + sql, + roster, + limits, + shared, + pending: VecDeque::new(), + active: BTreeSet::new(), + jobs: FuturesUnordered::new(), + hints: FuturesUnordered::new(), + } + } + + fn launch(&mut self, now: u64) { + // Rotate the bounded roster queue. A busy group cannot monopolize the + // next available apply slot, and successors never run concurrently. + for _ in 0..self.pending.len() { + if self.jobs.len() == self.limits.max_parallel_apply { + break; + } + let Some(job) = self.pending.pop_front() else { + break; + }; + if job.retry_at > now || self.active.contains(&job.group) { + self.pending.push_back(job); + continue; + } + self.active.insert(job.group); + let sql = self.sql.clone(); + let applier = self.roster[&job.group].clone(); + let runtime = self.runtime.clone(); + self.jobs.push(Box::pin(async move { + let result = apply(sql, applier, runtime, job.work).await; + (job.group, result) + })); + } + self.shared + .active_apply + .store(self.jobs.len(), Ordering::Release); + self.shared + .pending_apply + .store(self.pending.len(), Ordering::Release); + } + + pub(super) async fn run( + &mut self, + domain: &crate::asyncrt::DomainToken, + ) -> Result<(), CellGroupControllerError> { + loop { + if self.shared.closing.load(Ordering::Acquire) { + return Ok(()); + } + let now = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + self.launch(now); + let wake_at = now.saturating_add(self.limits.retry_interval_ms); + let timer = domain + .sleep_until(wake_at) + .map_err(|_| CellGroupControllerError::Closed)?; + crate::asyncrt::select! { + _ = self.shared.stop.notified() => return Ok(()), + _ = domain.wait_for_work() => return Err(CellGroupControllerError::Closed), + _ = timer => (), + Some((group, result)) = self.jobs.next(), if !self.jobs.is_empty() => { + self.active.remove(&group); + if let Some(work) = result? { + self.shared.apply_retries.fetch_add(1, Ordering::Relaxed); + self.pending.push_back(Pending { group, work, retry_at: wake_at }); + } + } + Some(()) = self.hints.next(), if !self.hints.is_empty() => (), + event = self.events.recv() => { + let event = event.ok_or(CellGroupControllerError::Closed)?; + let group = event.group(); + if !self.roster.contains_key(&group) { + return Err(CellGroupControllerError::Invalid("unregistered group event")); + } + match event.effect().map_err(Clone::clone)? { + CellGroupEffect::Message(message) => { + if let Err(error) = self.transport.submit_physical(&group, message.clone()) { + if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer) { + return Err(CellGroupControllerError::Transport(error)); + } + self.shared.send_refusals.fetch_add(1, Ordering::Relaxed); + // A hint is advisory, never delivery or commit evidence. + // Drop excess hints; Raft's own ticks retry lost traffic. + if self.hints.len() < self.limits.max_groups { + let runtime = self.runtime.clone(); + let member = message.to; + self.hints.push(Box::pin(async move { + let _ = runtime.report_unreachable(group, member).await; + })); + } + } + } + CellGroupEffect::Apply(_) => { + // One pending successor per group is possible before a + // completed publication future has been polled here. + if self.pending.iter().any(|job| job.group == group) { + return Err(CellGroupControllerError::Invalid("duplicate group apply")); + } + let work = event.retain_apply().ok_or(CellGroupControllerError::Invalid("apply ticket"))?; + self.pending.push_back(Pending { group, work: Work::Prepare(Box::new(work)), retry_at: now }); + } + CellGroupEffect::Applied { .. } => { + // Only this post-sync event advances the published count. + self.shared.applied.fetch_add(1, Ordering::Relaxed); + } + CellGroupEffect::ReadIndex { .. } => (), // Runtime-owned read authority. + CellGroupEffect::Persist(_) => return Err(CellGroupControllerError::Invalid("escaped persistence")), + } + } + } + } + } + + pub(super) async fn finish(&mut self) -> Result<(), CellGroupControllerError> { + self.transport.close().await; + let sql = self + .sql + .close() + .await + .map_err(|error| CellGroupControllerError::Apply(Arc::new(error))); + // Retain the runtime's engine claim while an apply worker can still + // write. A replacement runtime must not overlap physical preparation. + let runtime = self + .runtime + .close() + .await + .map_err(CellGroupControllerError::Runtime); + // Keep all unfinished tickets and appliers until these physical joins. + sql.and(runtime) + } +} + +async fn apply( + sql: CellGroupSqlPool, + applier: CellGroupApplier, + runtime: CellGroupRuntime, + work: Work, +) -> Result, CellGroupControllerError> { + let prepared = match work { + Work::Prepare(work) => match sql.prepare_apply(&applier, (*work).clone()).await { + Ok(prepared) => Box::new(prepared), + Err(ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded)) => { + return Ok(Some(Work::Prepare(work))) + } + Err(error) => return Err(CellGroupControllerError::Apply(Arc::new(error))), + }, + Work::Publish(prepared) => prepared, + }; + match (*prepared).clone().complete(&runtime).await { + Ok(()) => Ok(None), + Err(RuntimeError::Overloaded) => Ok(Some(Work::Publish(prepared))), + Err(error) => Err(error.into()), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 654b20c04..9422db7a0 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -10,6 +10,11 @@ pub use super::host_sql::cell_groups::{ }; mod apply; +mod controller; +pub use controller::{ + CellGroupController, CellGroupControllerError, CellGroupControllerLimits, + CellGroupControllerUsage, +}; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; pub use dispatch::{ diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 49cf49198..9489085e6 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -53,9 +53,14 @@ impl CellGroupRuntimeEvent { pub struct CellGroupRuntimeEvents { receiver: mpsc::Receiver, + owner: std::sync::Weak, } impl CellGroupRuntimeEvents { + pub(crate) fn belongs_to(&self, runtime: &CellGroupRuntime) -> bool { + self.owner.ptr_eq(&Arc::downgrade(&runtime.inner)) + } + pub async fn recv(&mut self) -> Option { self.receiver.recv().await } @@ -137,6 +142,10 @@ impl Drop for TransportClaim { } impl CellGroupRuntime { + pub(crate) fn same_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.inner, &other.inner) + } + pub(crate) fn log(&self) -> &ReplicaLog { &self.inner.log } @@ -246,7 +255,8 @@ impl CellGroupRuntime { .map_err(|_| RuntimeError::Closed)?; wait.await.map_err(|_| RuntimeError::Closed)?; handle.admit()?; - Ok((handle, CellGroupRuntimeEvents { receiver })) + let owner = Arc::downgrade(&handle.inner); + Ok((handle, CellGroupRuntimeEvents { receiver, owner })) } pub(crate) fn admit(&self) -> Result<(), RuntimeError> { diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 445416d7d..be8d8d825 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -297,6 +297,10 @@ fn overloaded() -> ExecutionError { } impl CellGroupSqlPool { + pub(crate) fn belongs_to_engine(&self, owner: &Arc<()>) -> bool { + Arc::ptr_eq(&self.inner.engine_owner, owner) + } + pub(crate) fn uses_domain(&self, domain: &crate::asyncrt::DomainToken) -> bool { self.inner.workers[0].domain.same_owner(domain) } diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 79fcfee1c..8166a682d 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -3,6 +3,7 @@ //! Physical group routing on the existing node transport's queues and sockets. use super::*; use crate::raft::cache::GroupLog; +use crate::raft::cell_groups::CellGroupRuntime; type PhysicalRoute = GroupRoute; type PhysicalPeer = RaftPeer; @@ -44,6 +45,14 @@ fn permitted( } impl RaftTransport { + pub(crate) fn uses_shared_runtime(&self, runtime: &CellGroupRuntime) -> bool { + self.handle + .0 + .runtime + .shared() + .is_ok_and(|owner| owner.same_owner(runtime)) + } + /// Install already-authorized physical membership. This creates no group, /// grants no serving authority and changes no committed configuration. pub fn install_physical_group(&self, route: PhysicalRoute) -> Result<(), TransportError> { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index e4354e5c4..17fb7b2fc 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -11,6 +11,36 @@ mod fault; #[path = "support/cell_group_runtime.rs"] mod runtime_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_drives_tls_apply_and_bounded_retry() -> Result<()> { + runtime_fixture::controller_replication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_group_controller_drives_tls_apply_and_bounded_retry() -> Result<()> { + runtime_fixture::controller_replication(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_checks_exact_owners() -> Result<()> { + runtime_fixture::controller_owners().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_cancelled_close_joins_sql() -> Result<()> { + runtime_fixture::controller_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_controller_apply_failure_and_shutdown_preserve_selected_root() -> Result<()> { + runtime_fixture::controller_apply_faults(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root() -> Result<()> { + runtime_fixture::controller_apply_faults(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn sql_submission_rejection_seals_delayed_retries() -> Result<()> { runtime_fixture::submission_rejections(false).await diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs new file mode 100644 index 000000000..303d7a824 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -0,0 +1,413 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! No fixture event pumping after the production controllers start. +use super::*; +#[path = "raft_ltx_fault.rs"] +#[allow(dead_code)] +mod physical; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn limits() -> CellGroupControllerLimits { + CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + } +} + +async fn start(fixture: &mut RuntimeCluster) -> Result> { + fixture.connect_tcp().await?; + let mut controllers = Vec::new(); + for (node, events) in fixture.events.drain(..).enumerate() { + let selected: Vec<_> = fixture + .appliers + .iter() + .filter(|((n, _), _)| *n == node) + .map(|(_, (_, applier))| applier.clone()) + .collect(); + controllers.push( + CellGroupController::start( + fixture.runtimes[node].clone(), + events, + fixture.transports[node].clone(), + fixture.appliers[&(node, fixture.cluster.birth.route().history)] + .0 + .clone(), + selected, + limits(), + ) + .await?, + ); + } + assert!(fixture.events.is_empty()); + Ok(controllers) +} + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +async fn applied(fixture: &RuntimeCluster, group: GroupHistory, index: LogIndex) -> Result<()> { + wait_for(async || { + for runtime in &fixture.runtimes { + if runtime.status(group).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +pub(crate) async fn controller_replication(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, hold) = std::sync::mpsc::sync_channel(1); + let hold = Mutex::new(hold); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + count.fetch_add(1, Ordering::SeqCst); + let _ = entered + .lock() + .unwrap() + .take() + .context("one callback")? + .send(()); + hold.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = fixture.add_group(31).await?; + for node in 0..3 { + let pool = fixture.appliers[&(node, route.history)].0.clone(); + let voter = &mut fixture.extra.get_mut(&second).context("extra")?[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&second)?, + voter.pages.take().context("pages")?, + CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture.appliers.insert((node, second), (pool, applier)); + } + let controllers = start(&mut fixture).await?; + fixture.runtimes[0].campaign(second).await?; + wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let mut attempt = cell_attempt(179)?; + attempt.request.operation = "Add".into(); + attempt.request.contract_digest = [82; 32]; + attempt.request.request_digest = command_request_digest(b""); + let caller = pool.clone(); + let preparation = fixture.cluster.domain.spawn(async move { + caller + .stage_registered(options, authority, attempt, class, vec![], true) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage()?.active_cells, 1); + // The sole SQL worker is physically occupied. Two committed groups must + // share its one reserved apply slot without losing either retained ticket. + let mut accepted = Vec::new(); + for (group, rooted) in [(route.history, rooted), (second, false)] { + let marker = fixture.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")?; + let cell = history(marker.route(), "controller-enrolled")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + accepted.push(( + group, + fixture.runtimes[0] + .propose( + group, + apply::enrollment_bytes(marker.route(), &cell, &bytes, rooted)?, + ) + .await?, + )); + } + wait_for(async || Ok(controllers[0].usage().apply_retries > 0)).await?; + assert!(!controllers[0].usage().closing); + assert!(controllers[0].usage().active_apply <= 2); + assert!(controllers[0].usage().pending_apply <= 2); + assert_eq!(pool.usage()?.completion_reservations, 1); + release.send(())?; + let CellSqlOutcome::Prepared(prepared) = preparation.await?? else { + anyhow::bail!("preparation"); + }; + for (group, receipt) in accepted { + applied(&fixture, group, receipt.position.index).await?; + } + pool.mark_submitted(a.clone(), prepared.proposal()).await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + applied(&fixture, route.history, accepted.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!(receipt.receipt().receipt.result(), b"committed"); + assert_eq!(receipt.receipt().version.history, a); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(receipt); + drop(prepared); + for (n, controller) in controllers.iter().enumerate() { + wait_for(async || Ok(controller.usage().applied >= 3)).await?; + assert_eq!(controller.usage().groups, 2); + assert_eq!(fixture.transports[n].usage().peers, 2); + controller.close().await?; + assert_eq!(controller.usage().active_apply, 0); + assert_eq!(controller.usage().pending_apply, 0); + assert_eq!( + fixture.appliers[&(n, route.history)] + .0 + .usage()? + .running_workers, + 0 + ); + } + drop(controllers); + fixture.close().await +} + +pub(crate) async fn controller_owners() -> Result<()> { + for fault in 0..6 { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + fixture.connect_tcp().await?; + // All three engines share an execution domain: domain equality is not + // sufficient to authorize any of these borrowed owners. + let events = fixture.events.remove(if fault == 0 { 1 } else { 0 }); + let transport = fixture.transports[usize::from(fault == 1)].clone(); + let pool = fixture.appliers[&(usize::from(fault == 2), route.history)] + .0 + .clone(); + let applier = fixture.appliers[&(usize::from(fault == 3), route.history)] + .1 + .clone(); + let appliers = if fault == 4 { + vec![applier.clone(), applier] + } else { + vec![applier] + }; + let mut bounds = limits(); + if fault == 5 { + bounds.max_parallel_apply = 0; + } + assert!(matches!( + CellGroupController::start( + fixture.runtimes[0].clone(), + events, + transport, + pool, + appliers, + bounds + ) + .await, + Err(CellGroupControllerError::Invalid(_)) + )); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn controller_close() -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, _) = Box::pin(registered::setup(false, &class)).await?; + let controllers = start(&mut fixture).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let options = fixture.cell_options(&a).await?; + let caller = pool.clone(); + let producer = fixture.cluster.domain.spawn(async move { + caller + .stage( + options, + authority, + cell_attempt(179).unwrap(), + true, + move |tx| { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(10))?; + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(vec![]) + }, + ) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + producer.abort(); + assert!(producer.await.is_err()); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(pool.usage()?.running_workers, 1); + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.appliers.clear(); + let replacement = apply::pool(&fixture, 0)?; + replacement.close().await?; + drop(replacement); + fixture.close().await +} + +pub(crate) async fn controller_apply_faults(rooted: bool) -> Result<()> { + for mode in 0..3 { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem: Arc = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let cluster = + Cluster::with_storage(RaftRuntimeOptions::default(), root, filesystem.clone()).await?; + let fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + let (mut fixture, route, a, _) = + Box::pin(enroll_sql_cells(fixture, rooted, filesystem)).await?; + for node in 0..3 { + apply::install(&mut fixture, &a, node).await?; + } + let mut controllers = start(&mut fixture).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let before = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected")?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, held) = std::sync::mpsc::sync_channel(1); + if mode == 0 { + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + } else { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + held.recv_timeout(Duration::from_secs(15)) + .expect("physical apply release"); + })); + } + let cell = history(&route, "controller-faulted")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + if mode == 0 { + let result = + tokio::time::timeout(Duration::from_secs(10), controllers[0].wait()).await?; + assert!( + matches!(result, Err(CellGroupControllerError::Apply(ref error)) if matches!(error.as_ref(), ExecutionError::Storage(_))) + ); + assert_eq!( + faults.mode.load(Ordering::Acquire), + 0, + "real manifest sync fault consumed" + ); + assert!(controllers[0].usage().closing); + } else { + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage()?.apply_retained_operations, 1); + assert_eq!(controllers[0].usage().active_apply, 1); + if mode == 1 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + } else { + // The worker cannot retain a public handle and prevent its own + // final-handle drop from initiating physical shutdown. + drop(controllers.remove(0)); + wait_for(async || Ok(fixture.transports[0].usage().tasks == 0)).await?; + } + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + assert_eq!(pool.usage()?.running_workers, 1); + assert_eq!(pool.usage()?.apply_retained_operations, 1); + release.send(())?; + if mode == 1 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + } else { + wait_for(async || { + Ok(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::Closed) + )) + }) + .await?; + } + } + assert_eq!(pool.usage()?.running_workers, 0); + // A failed or abandoned preparation must not advance engine authority. + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + if mode < 2 { + drop(controllers.remove(0)); + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 229c5e5ac..aee3f68b4 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -649,6 +649,9 @@ pub(super) async fn attachment_reserves_before_owner() -> Result<()> { #[path = "cell_group_sql.rs"] mod sql; +pub(super) use sql::{ + controller_apply_faults, controller_close, controller_owners, controller_replication, +}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c0de46950..ce5c8dd29 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -526,6 +526,12 @@ pub(crate) use dispatch::routed as routed_queries; mod command_dispatch; pub(crate) use command_dispatch::commands as routed_commands; +#[path = "cell_group_controller.rs"] +mod controller; +pub(crate) use controller::{ + controller_apply_faults, controller_close, controller_owners, controller_replication, +}; + pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { Box::pin(command_dispatch::commands_with_transport(rooted, true)).await } From ee684c3394ffccbfa84bfc5d94876de281f68b0b Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 04:54:44 -0700 Subject: [PATCH 088/568] test: reuse the shared physical apply fault fixture --- crates/cellar/tests/support/cell_group_controller.rs | 4 +--- crates/cellar/tests/support/cell_group_sql_apply_faults.rs | 2 +- 2 files changed, 2 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs index 303d7a824..64e529634 100644 --- a/crates/cellar/tests/support/cell_group_controller.rs +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -1,10 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! No fixture event pumping after the production controllers start. +use super::apply_faults::physical; use super::*; -#[path = "raft_ltx_fault.rs"] -#[allow(dead_code)] -mod physical; use std::sync::{ atomic::{AtomicUsize, Ordering}, Mutex, diff --git a/crates/cellar/tests/support/cell_group_sql_apply_faults.rs b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs index 1d9a29afc..688285069 100644 --- a/crates/cellar/tests/support/cell_group_sql_apply_faults.rs +++ b/crates/cellar/tests/support/cell_group_sql_apply_faults.rs @@ -6,7 +6,7 @@ use std::sync::atomic::Ordering; #[path = "raft_ltx_fault.rs"] #[allow(dead_code)] -mod physical; +pub(super) mod physical; async fn pending( rooted: bool, From 4e9063e889bc6290f937036c29427bbf5492c220 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 05:00:27 -0700 Subject: [PATCH 089/568] docs: record the CG-04 event controller checkpoint --- docs/design/multi-raft/cell-groups/README.md | 21 +++++-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 54 +++++++++------- .../design/multi-raft/cell-groups/progress.md | 61 +++++++++++++++++++ 4 files changed, 113 insertions(+), 27 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 4d8071870..a221e246b 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -168,10 +168,23 @@ Physical-group peer transport now starts directly on the shared runtime at Typed group histories refuse old untagged frames; unsupported group snapshots refuse. All 192 native cases, 91 transport-filtered library cases and nine commands pass. Transfer `.cellar/cell-groups-evidence/CG-04-physical-transport/` -with prior evidence (241 source inputs, eight executables). The next critical -gap is production controller ownership and reconciliation after caller disconnect, -followed by routed enrollment and SDK/shared-query RPC integration. The fixtures -still drive runtime events. See [progress](progress.md) for exact limits. +with prior evidence (241 source inputs, eight executables). At that checkpoint, +production controller ownership and reconciliation after caller disconnect remained +required, followed by routed enrollment and SDK/shared-query RPC integration. +Those fixtures still drive runtime events. See [progress](progress.md) for exact limits. + +At `ee684c33`, a fixed-roster `CellGroupController` drives shared TLS and SQL +apply without fixture pumping. Exact owner bindings, bounded per-group apply, +physical sync failure and joined shutdown pass six new native cases. The partial +gate passes 58 execution cases and eight commands; transfer +`.cellar/cell-groups-evidence/CG-04-event-controller-qualified/` (244 source inputs, one +executable) and the preceding `CG-04-event-controller/` regression gate with prior +evidence. That earlier run passed 289 Rust cases; the successor only corrects a +duplicated test fixture import identified by Clippy. The critical next gap is +caller-independent +command ownership and bounded reconciliation after the response deadline, +followed by routed enrollment and SDK/shared-query RPC integration. Fixed roster +and node-wide failure handling also remain incomplete. See [progress](progress.md). The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 52aa47b06..6cc037e0c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa", + "source_revision": "ee684c3394ffccbfa84bfc5d94876de281f68b0b", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index ba5aca3a5..86efd7e41 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,19 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `eb99a242`, connecting physical groups +The newest reviewed source baseline is `ee684c33`, adding a fixed-roster node +controller that drives existing shared TLS/apply owners without fixture pumping. +Its partial record is `.cellar/cell-groups-evidence/CG-04-event-controller-qualified/checkpoint.json`: +58 shared-execution cases, eight commands, 244 source inputs and +one executable. The preceding `61aa7752` gate passed 289 Rust regressions before +Clippy found a duplicated test fixture import; only that test import changed, and +the affected execution suite was rerun. Transfer both gates with prior evidence. +Six new cases +cover two groups on one SQL worker, both image modes, exact ownership, real +manifest sync failure, cancelled close and final-handle cleanup. Commands still +require caller-retained invocations; post-response resolution is not implemented. +Dynamic roster and per-group failure isolation remain gaps. CG-04 stays IN_PROGRESS. +The preceding reviewed source baseline is `eb99a242`, connecting physical groups to the existing production TLS transport without constructing the legacy runtime. Its partial record is `.cellar/cell-groups-evidence/CG-04-physical-transport/checkpoint.json`: 192 native cases, 91 library cases, nine commands, 241 source inputs and @@ -380,26 +392,26 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Integrate the phased command invocation into the production node controller. - Retain it independently from ingress waiters and drive existing shared-runtime - transport, apply and completion events through the existing owners. Physical - `GroupHistory` routing now works over `RaftTransport::start_shared`; use that - transport directly in the production event loop. Qualify - caller disconnect after SQL entry and after provisional acceptance: the - controller must resolve the exact receipt or a proven non-commit and release - bounded ownership without re-executing an uncertain command. The current test - cancels a preparation future while retaining the invocation; it does not - qualify controller ownership after ingress cancellation. Missing origin - producers on followers still need a safe leader receipt-lookup route; never - infer non-commit from missing local state. Keep original deadlines for responses - while giving the controller an explicit bounded reconciliation lifecycle. - Then add catalog-routed enrollment with stable creation identity and exact - creation settlement, and integrate shared query RPC. Qualify enrollment, - command, lost response and linearizable - query together. Public `RaftCellHandle` still uses its old host binding; the - real-engine fixtures still pump transport/apply events. Reuse existing worker - and transport ownership with joined shutdown. Requested and namespace S3 - durability currently refuse until CG-06 supplies real archive proofs. +1. Move fresh phased command invocations into `CellGroupController`, independently + from ingress response waiters. The fixed-roster event controller now drives + physical `GroupHistory` transport and SQL apply without fixture pumping; extend + that owner rather than introducing cell-specific tasks. Keep the original + response deadline while retaining bounded resolution ownership afterward. + Qualify caller disconnect/timeout after SQL entry and provisional acceptance: + resolve the exact receipt or proven non-commit without replaying uncertain + callbacks. Existing routed command tests still retain their invocation manually. + Before cancellation cleanup, extend `execution/submission.rs`: an attempted + submission error currently resets to Open; that is not a pre-append proof. + Seal only never-attempted preparations and retain an explicit uncertain state + after attempted submission errors. Serialize the seal with delayed clones and + prove physical cleanup with exact rejection capabilities. + Missing origin producers on followers still need a safe leader receipt-lookup + route; never infer non-commit from missing local state. Then add catalog-routed + enrollment with stable creation identity, shared query RPC and the public + `RaftCellHandle` binding. Requested and namespace S3 durability continue to + refuse until CG-06 supplies real archive proofs. The event controller currently + has a fixed roster and closes the node on fatal apply errors; add dynamic group + registration and per-group failure isolation before claiming runtime completion. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 7cf0bb55c..1ee1dd0ec 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,67 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 autonomous event controller + +Qualified source: `ee684c3394ffccbfa84bfc5d94876de281f68b0b`. + +`CellGroupController` owns a fixed roster's exclusive runtime event stream and +drives existing physical TLS transport, SQL preparation and runtime publication +without fixture pumping. Exact runtime/event/transport and engine/pool/applier +bindings refuse foreign owners even when they share an execution domain. One +node cleanup task runs bounded apply jobs, serializes each group's work and +rotates pending groups. Retained tickets preserve event and physical admission +through SQL backpressure; a shared node timer backs off retries. Only runtime +post-sync `Applied` events advance published progress. Transport queue admission +is not commit evidence. Immediate transient send refusals produce bounded advisory +hints; fatal transport/runtime/apply errors close the node and remain observable. + +Close requests and final-public-handle drop stop the controller. Cleanup joins +transport and physical SQL work before releasing the runtime's engine claim, +retaining outstanding tickets/appliers until all joins complete. Cancelling a +close waiter cannot release a blocked physical producer or permit replacement. +This controller has a fixed startup roster and currently fails the entire node +on a fatal group apply error; dynamic registration and per-group failure isolation +remain required. It does not yet own command invocations or replace the SDK host. + +Six new native cases qualify autonomous TLS replication in both image modes, +two groups sharing peer pools and one SQL worker, admission retry while that +worker is blocked, exact receipt settlement and same-domain foreign-owner refusal. +Real manifest directory-sync faults leave the engine-selected root unchanged. +Cancelled close and final-handle drop while physical sync is blocked retain +ownership until release; no interrupted preparation is published. The initial +shutdown test failed because the harness retained old pool handles while attempting +replacement. It now releases those handles after verifying physical joins; the +production lease checks and assertions remain intact. + +All 58 shared-execution cases and eight gate commands pass at this revision, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +format (1,829 Rust files), layout, dependencies, docs, source preflight and 37 +plan-validator cases. The preceding `61aa7752` run passed 289 Rust cases (198 +native across seven suites and 91 transport-filtered library cases), then Clippy +rejected duplicate loading of the physical fault fixture. This successor reuses +the existing module; production code and assertions are unchanged. The affected +execution suite was rerun; other regressions retain their original revision. +The partial record +`.cellar/cell-groups-evidence/CG-04-event-controller-qualified/checkpoint.json` +binds 244 source inputs and the execution test binary. Transfer it with the +retained `CG-04-event-controller/` gate and prior evidence. Four ignored process +helpers ran through their parents in the earlier regression run; its RustFS seed +case was skipped. Local TLS endpoints share a process. This is not RustFS, +independent-failure-domain, power-loss or load qualification. + +CG-04 remains IN_PROGRESS. The most critical next gap is command ownership after +caller disconnect: move fresh invocations into this controller and distinguish +response deadlines from bounded resolution ownership. Preserve exact origin and +inputs through uncertainty; never replay ambiguous callbacks or treat missing +local state as non-commit. Before adding cancellation cleanup, seal only a +preparation that has never attempted submission: the current submission state +resets to Open after an error, which cannot establish that append did not occur. +Retain a distinct uncertain-attempt state and qualify timeout/disconnect after SQL +entry and provisional acceptance. Then complete routed enrollment, shared query +RPC and SDK integration. Configuration/cache lifecycle, snapshots/archive, +effects, topology, real RustFS/load and final legacy removal remain later gates. + ## CG-04 physical-group TLS transport Qualified source: `eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa`. From 8c615ac91bc3c13e6c25da4f4ab584a73e609207 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 05:39:46 -0700 Subject: [PATCH 090/568] feat: retain shared commands beyond caller response lifetime --- .../raft/cell_groups/controller/commands.rs | 208 ++++++++ .../raft/cell_groups/controller/mod.rs | 38 +- .../raft/cell_groups/controller/worker.rs | 38 +- .../raft/cell_groups/directory/routing.rs | 4 + .../dispatch/{command.rs => command/mod.rs} | 22 +- .../cell_groups/dispatch/command/owned.rs | 182 +++++++ .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../durability/raft/execution/submission.rs | 36 +- .../raft/execution/submission_tests.rs | 73 +++ .../raft/host_sql/cell_groups/mod.rs | 101 +++- .../tests/native_cell_groups_execution.rs | 35 ++ .../tests/support/cell_group_controller.rs | 13 +- .../support/cell_group_controller_commands.rs | 481 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 1 + crates/cellar/tests/support/cell_group_sql.rs | 3 + .../support/cell_group_sql_registered.rs | 21 +- .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 159 +++++- 18 files changed, 1374 insertions(+), 47 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/commands.rs rename crates/cellar/durability/raft/cell_groups/dispatch/{command.rs => command/mod.rs} (97%) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs create mode 100644 crates/cellar/durability/raft/execution/submission_tests.rs create mode 100644 crates/cellar/tests/support/cell_group_controller_commands.rs diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs new file mode 100644 index 000000000..4815ae092 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -0,0 +1,208 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation ownership is independent of response-waiter lifetime. +use super::*; +use crate::raft::budget::{Budget, Permit}; +use crate::raft::cell_groups::{ + CellCommandError, CellCommandInvocation, CellCommandResponse, CellDispatchError, CellSqlOptions, +}; +use cellar_logic::consensus::CommandId; +use tokio::sync::{mpsc, oneshot, watch}; + +pub(super) struct Request { + invocation: CellCommandInvocation, + options: CellSqlOptions, + reply: oneshot::Sender, + permit: Arc, +} + +struct Delivery { + result: Result>, + _permit: Arc, +} + +pub(super) struct Completed { + pub invocation: CellCommandInvocation, + pub error: Option>, + _permit: Arc, +} + +/// Dropping or timing out this response never cancels the admitted command. +/// An unpolled reply retains node admission, including error-only replies. +pub struct CellCommandCall { + receiver: oneshot::Receiver, + command: CommandId, + deadline: u64, + domain: crate::asyncrt::DomainToken, +} + +/// Admission did not transfer ownership. In particular, a rejected non-fresh +/// invocation is returned with its original recovery state intact. +pub struct CellCommandAdmissionError { + pub error: CellCommandError, + pub invocation: Box, +} + +impl std::fmt::Debug for CellCommandAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellCommandAdmissionError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellCommandAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellCommandAdmissionError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.error) + } +} + +fn error(command: CommandId, ambiguous: bool, reason: RuntimeError) -> CellCommandError { + CellCommandError { + command, + ambiguous, + cause: Box::new(CellDispatchError::Runtime(reason)), + } +} + +impl CellCommandCall { + pub async fn response(self) -> Result> { + let failure = |reason| Arc::new(error(self.command, true, reason)); + let now = self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))?; + if now >= self.deadline { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + let timer = self + .domain + .sleep_until(self.deadline) + .map_err(|_| failure(RuntimeError::Closed))?; + let delivery = crate::asyncrt::select! { + result = self.receiver => result.map_err(|_| failure(RuntimeError::Closed))?, + _ = timer => return Err(failure(RuntimeError::DeadlineExceeded)), + }; + if self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))? + >= self.deadline + { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + delivery.result + } +} + +fn option_bytes(options: &mut CellSqlOptions) -> Result { + let (path, vfs) = match options { + CellSqlOptions::Materialized(options) => (&mut options.directory, &mut options.sqlite_vfs), + CellSqlOptions::Extents(options) => (&mut options.cache_directory, &mut options.sqlite_vfs), + }; + if path.as_os_str().len() > 4096 || vfs.as_ref().is_some_and(|vfs| vfs.len() > 128) { + return Err(RuntimeError::Invalid("command SQL option bounds".into())); + } + *path = std::mem::take(path).into_boxed_path().into_path_buf(); + *vfs = vfs.take().map(|vfs| vfs.into_boxed_str().into_string()); + Ok(2 * (path.as_os_str().len() + vfs.as_ref().map_or(0, String::len))) +} + +impl CellGroupController { + /// Admit a fresh local invocation. Discovery and destination authority still + /// run inside the owner; catalog routing never grants leader authority. + pub fn command( + &self, + invocation: CellCommandInvocation, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let admitted = (|| { + let (command, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let bytes = bytes + .checked_add(option_bytes(&mut options).map_err(|e| error(command, false, e))?) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + Ok((command, deadline, permit)) + })(); + let (command, deadline, permit) = match admitted { + Ok(admitted) => admitted, + Err(error) => { + return Err(CellCommandAdmissionError { + error, + invocation: Box::new(invocation), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(Request { + invocation, + options, + reply, + permit: Arc::new(permit), + }) + .map_err(|e| { + let (request, reason) = match e { + mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), + mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), + }; + CellCommandAdmissionError { + error: error(command, false, reason), + invocation: Box::new(request.invocation), + } + })?; + Ok(CellCommandCall { + receiver, + command, + deadline, + domain: shared.domain.clone(), + }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + mut progress: watch::Receiver<()>, +) -> Completed { + let Request { + mut invocation, + options, + reply, + permit, + } = request; + let result = invocation + .run_owned(&runtime, &sql, options, &shared.groups, &mut progress) + .await + .map_err(Arc::new); + let error = result.as_ref().err().cloned(); + let _ = reply.send(Delivery { + result, + _permit: permit.clone(), + }); + Completed { + invocation, + error, + _permit: permit, + } +} + +pub(super) fn budget(limits: CellGroupControllerLimits) -> Arc { + Budget::new(limits.max_command_bytes, limits.max_commands) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 957dfb13b..d6107674b 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -1,9 +1,11 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Owns a fixed roster's consensus events, transport dispatch and physical apply. -//! This infrastructure owner does not yet own ingress command invocations. +//! Admitted command ownership survives response cancellation and expiry. +mod commands; mod worker; +pub use commands::{CellCommandAdmissionError, CellCommandCall}; use super::{CellGroupApplier, CellGroupRuntime, CellGroupRuntimeEvents, CellGroupSqlPool}; use crate::raft::{ @@ -12,7 +14,7 @@ use crate::raft::{ transport::{RaftTransport, TransportError}, }; use futures_util::FutureExt; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use std::sync::{ atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, Arc, Mutex, @@ -25,6 +27,11 @@ pub struct CellGroupControllerLimits { pub max_parallel_apply: usize, /// One node timer backs off transient SQL admission, never one timer per cell. pub retry_interval_ms: u64, + /// Includes queued, resolving and unconsumed command responses. + pub max_commands: usize, + /// Invocation/input/options retention. SQL capture, apply and result bytes + /// retain the SQL pool's independent admission as well. + pub max_command_bytes: usize, } #[derive(Clone, Debug)] @@ -33,6 +40,7 @@ pub enum CellGroupControllerError { Runtime(RuntimeError), Apply(Arc), Transport(TransportError), + Command(Arc), Closed, } @@ -60,10 +68,16 @@ pub struct CellGroupControllerUsage { /// Immediate queue/route refusals. The transport tracks later I/O failures. pub send_refusals: u64, pub closing: bool, + pub retained_commands: usize, + pub command_bytes: usize, } struct Shared { - groups: usize, + groups: BTreeSet, + limits: CellGroupControllerLimits, + domain: crate::asyncrt::DomainToken, + commands: Arc, + requests: tokio::sync::mpsc::Sender, active_apply: AtomicUsize, pending_apply: AtomicUsize, applied: AtomicU64, @@ -144,6 +158,8 @@ impl CellGroupController { || !(1..=60_000).contains(&limits.retry_interval_ms) || appliers.is_empty() || appliers.len() > limits.max_groups + || !(1..=65_536).contains(&limits.max_commands) + || limits.max_command_bytes < 8192 { return Err(CellGroupControllerError::Invalid( "controller limits/roster", @@ -167,8 +183,14 @@ impl CellGroupController { runtime.status(*history).await?; } runtime.admit()?; + let domain = runtime.log().domain(); + let (requests, receiver) = tokio::sync::mpsc::channel(limits.max_commands); let shared = Arc::new(Shared { - groups: roster.len(), + groups: roster.keys().copied().collect(), + limits, + domain: domain.clone(), + commands: commands::budget(limits), + requests, active_apply: AtomicUsize::new(0), pending_apply: AtomicUsize::new(0), applied: AtomicU64::new(0), @@ -182,7 +204,6 @@ impl CellGroupController { let handle = Self { handle: Arc::new(Handle(shared.clone())), }; - let domain = runtime.log().domain(); let worker_domain = domain.clone(); // Infrastructure cleanup is not cancelled with application tasks. The // worker retains tickets/appliers until all physical owners have joined. @@ -194,7 +215,7 @@ impl CellGroupController { transport, sql, roster, - limits, + receiver, shared.clone(), ); let result = std::panic::AssertUnwindSafe(worker.run(&worker_domain)) @@ -216,8 +237,11 @@ impl CellGroupController { pub fn usage(&self) -> CellGroupControllerUsage { let shared = &self.handle.0; + let (command_bytes, retained_commands) = shared.commands.usage(); CellGroupControllerUsage { - groups: shared.groups, + groups: shared.groups.len(), + command_bytes, + retained_commands, active_apply: shared.active_apply.load(Ordering::Acquire), pending_apply: shared.pending_apply.load(Ordering::Acquire), applied: shared.applied.load(Ordering::Relaxed), diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index a27cbde2f..989456802 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -31,6 +31,10 @@ pub(super) struct Worker { active: BTreeSet, jobs: FuturesUnordered>, hints: FuturesUnordered>, + requests: tokio::sync::mpsc::Receiver, + commands: FuturesUnordered>, + failed_command: Option, + progress: tokio::sync::watch::Sender<()>, } impl Worker { @@ -40,7 +44,7 @@ impl Worker { transport: RaftTransport, sql: CellGroupSqlPool, roster: BTreeMap, - limits: CellGroupControllerLimits, + requests: tokio::sync::mpsc::Receiver, shared: Arc, ) -> Self { Self { @@ -49,12 +53,16 @@ impl Worker { transport, sql, roster, - limits, + limits: shared.limits, shared, pending: VecDeque::new(), active: BTreeSet::new(), jobs: FuturesUnordered::new(), hints: FuturesUnordered::new(), + requests, + commands: FuturesUnordered::new(), + failed_command: None, + progress: tokio::sync::watch::channel(()).0, } } @@ -93,6 +101,10 @@ impl Worker { &mut self, domain: &crate::asyncrt::DomainToken, ) -> Result<(), CellGroupControllerError> { + let mut wake_at = domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)? + .saturating_add(self.limits.retry_interval_ms); loop { if self.shared.closing.load(Ordering::Acquire) { return Ok(()); @@ -101,19 +113,34 @@ impl Worker { .mono_ms() .map_err(|_| CellGroupControllerError::Closed)?; self.launch(now); - let wake_at = now.saturating_add(self.limits.retry_interval_ms); let timer = domain .sleep_until(wake_at) .map_err(|_| CellGroupControllerError::Closed)?; crate::asyncrt::select! { _ = self.shared.stop.notified() => return Ok(()), _ = domain.wait_for_work() => return Err(CellGroupControllerError::Closed), - _ = timer => (), + _ = timer => { + self.progress.send_replace(()); + wake_at = domain.mono_ms().map_err(|_| CellGroupControllerError::Closed)? + .saturating_add(self.limits.retry_interval_ms); + }, + Some(request) = self.requests.recv() => { + self.commands.push(Box::pin(commands::drive(request, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))); + } + Some(completed) = self.commands.next(), if !self.commands.is_empty() => { + if completed.invocation.unresolved() { + let error = completed.error.clone().map_or(CellGroupControllerError::Invalid("unresolved successful command"), CellGroupControllerError::Command); + // Retain the invocation and its permit through physical + // shutdown even after delivering the error to its caller. + self.failed_command = Some(completed); + return Err(error); + } + } Some((group, result)) = self.jobs.next(), if !self.jobs.is_empty() => { self.active.remove(&group); if let Some(work) = result? { self.shared.apply_retries.fetch_add(1, Ordering::Relaxed); - self.pending.push_back(Pending { group, work, retry_at: wake_at }); + self.pending.push_back(Pending { group, work, retry_at: now.saturating_add(self.limits.retry_interval_ms) }); } } Some(()) = self.hints.next(), if !self.hints.is_empty() => (), @@ -153,6 +180,7 @@ impl Worker { CellGroupEffect::Applied { .. } => { // Only this post-sync event advances the published count. self.shared.applied.fetch_add(1, Ordering::Relaxed); + self.progress.send_replace(()); } CellGroupEffect::ReadIndex { .. } => (), // Runtime-owned read authority. CellGroupEffect::Persist(_) => return Err(CellGroupControllerError::Invalid("escaped persistence")), diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs index f71ae1213..ed04eaa01 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/routing.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -199,6 +199,10 @@ impl RangeDirectory { } impl CellRoutingSession { + pub(crate) fn deadline_ms(&self) -> u64 { + self.deadline + } + pub(crate) fn validate_attempt( &self, attempt: &CellRouteAttempt, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs similarity index 97% rename from crates/cellar/durability/raft/cell_groups/dispatch/command.rs rename to crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 34a12f124..ff9e059f5 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -2,6 +2,8 @@ //! Routed commands retain their originating producer until exact resolution. //! The node controller drives publication; acceptance is only provisional. +mod owned; + use super::*; use crate::application::consensus::command_request_digest; use crate::raft::execution::{CommandAttempt, CommandRequest}; @@ -364,7 +366,7 @@ impl CellCommandInvocation { .ok_or(CellDispatchError::Invalid("missing origin"))?, ); let outcome = sql - .stage_registered( + .stage_registered_owned( options, authority, command, @@ -375,18 +377,14 @@ impl CellCommandInvocation { .await; match outcome { Ok(outcome) => self.preparation(outcome).await, - Err( - error @ (ExecutionError::Refused(_) - | ExecutionError::Registration(_) - | ExecutionError::Busy), - ) => { - // These stage responses did not install this preparation. A - // retry may seek fresh authority; lost/closed replies do not - // take this path and keep their originating producer binding. - self.origin = None; - Err(CellDispatchError::Execution(error)) + Err(failure) => { + // Admission refusals and received Stage errors completed without + // installing this producer. Lost replies retain the exact origin. + if !failure.unresolved { + self.origin = None; + } + Err(CellDispatchError::Execution(failure.error)) } - Err(error) => Err(CellDispatchError::Execution(error)), } } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs new file mode 100644 index 000000000..6fa39fdbe --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs @@ -0,0 +1,182 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Controller-only execution: response expiry never cancels a physical phase. +use super::*; +use std::collections::BTreeSet; +use tokio::sync::watch; + +impl CellCommandInvocation { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(CommandId, u64, usize), CellCommandError> { + if !self.dispatcher.domain.same_owner(domain) + || self.attempt.is_some() + || self.dispatched + || self.origin.is_some() + { + return Err(self.error(CellDispatchError::Invalid( + "controller requires a fresh local invocation", + ))); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| self.error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + Ok((self.request.id, deadline, bytes)) + } + + pub(crate) fn unresolved(&self) -> bool { + self.origin.is_some() + } + + fn expired(&self) -> Result { + Ok(self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms()) + } + + async fn cancel_unsubmitted( + &mut self, + progress: &mut watch::Receiver<()>, + ) -> Result { + let prepared = self + .origin + .as_ref() + .and_then(|origin| origin.prepared.as_ref()) + .ok_or(CellDispatchError::Invalid("missing controller preparation"))?; + let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + else { + return Ok(false); + }; + loop { + match self.reject(proof.clone()).await { + Ok(_) => return Ok(true), + Err(CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded), + )) => { + progress.changed().await.map_err(|_| closed())?; + } + Err(error) => return Err(error), + } + } + } + + /// The caller must keep this future/owner until resolution or joined node + /// shutdown. It never resets routing deadlines or re-executes a callback. + pub(crate) async fn run_owned( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &BTreeSet, + progress: &mut watch::Receiver<()>, + ) -> Result { + self.begin().await?; + let result = self.run_at(runtime, sql, options, roster, progress).await; + result.map_err(|error| self.error(error)) + } + + async fn run_at( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &BTreeSet, + progress: &mut watch::Receiver<()>, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))? + .clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if !roster.contains(&attempt.route().history) { + return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); + } + self.dispatched = true; + // Unlike the response-facing phase API, do not drop this SQL waiter at + // the response deadline. It carries the exact preparation or refusal. + match self.prepare_at(&attempt, runtime, sql, options).await? { + CellCommandPreparation::Committed(response) => return Ok(response), + CellCommandPreparation::Prepared => (), + } + loop { + if self.expired()? { + if self.cancel_unsubmitted(progress).await? { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + break; + } + match self.submit_at().await { + Ok(CellCommandSubmission::Accepted(_)) => break, + Ok(CellCommandSubmission::Rejected) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))) + } + Err(CellDispatchError::Runtime( + RuntimeError::NotReady | RuntimeError::Overloaded, + )) + | Err(CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded), + )) => { + // Retry only the exact prepared submission; no new callback. + progress.changed().await.map_err(|_| closed())?; + } + Err(CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::StaleCompletion, + )) => break, + Err(error) => { + // An untouched preparation can be sealed even if the SQL + // mark/submission mailbox refused. Other errors stay owned. + self.cancel_unsubmitted(progress).await?; + return Err(error); + } + } + } + // Submission may have entered the owner despite a lost/error response. + // Only an exact receipt or newer applied term can retire this producer. + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing origin"))? + .submitted = true; + loop { + match self.settle_at().await { + Ok(CellCommandSettlement::Committed(response)) => return Ok(response), + Ok(CellCommandSettlement::NotCommitted) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Unavailable, + ))) + } + Ok(CellCommandSettlement::Refused(reason)) => { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + reason, + ))) + } + Err(CellDispatchError::Runtime( + RuntimeError::NotReady + | RuntimeError::NotLeader + | RuntimeError::Overloaded + | RuntimeError::StaleCompletion, + )) + | Err(CellDispatchError::Execution( + ExecutionError::Uncertain + | ExecutionError::Busy + | ExecutionError::Refused(CommandFailure::Overloaded), + )) => (), + Err(error) => return Err(error), + } + progress.changed().await.map_err(|_| closed())?; + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 9422db7a0..517c5ba4f 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -12,8 +12,8 @@ pub use super::host_sql::cell_groups::{ mod apply; mod controller; pub use controller::{ - CellGroupController, CellGroupControllerError, CellGroupControllerLimits, - CellGroupControllerUsage, + CellCommandAdmissionError, CellCommandCall, CellGroupController, CellGroupControllerError, + CellGroupControllerLimits, CellGroupControllerUsage, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; diff --git a/crates/cellar/durability/raft/execution/submission.rs b/crates/cellar/durability/raft/execution/submission.rs index 6bb79e192..f3b32759f 100644 --- a/crates/cellar/durability/raft/execution/submission.rs +++ b/crates/cellar/durability/raft/execution/submission.rs @@ -14,6 +14,7 @@ enum Decision { #[default] Open, Submitting, + Uncertain, Accepted(CellGroupProposal), Rejected, } @@ -99,13 +100,14 @@ impl CellPreparedCommand { .decision .lock() .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + let uncertain = matches!(*decision, Decision::Uncertain); match &*decision { Decision::Accepted(accepted) => { return Ok(CellSubmissionOutcome::Accepted(accepted.clone())) } Decision::Rejected => return Ok(CellSubmissionOutcome::Rejected(self.rejected())), Decision::Submitting => return Err(RuntimeError::NotReady), - Decision::Open => (), + Decision::Open | Decision::Uncertain => (), } *decision = Decision::Submitting; drop(decision); @@ -122,12 +124,17 @@ impl CellPreparedCommand { *decision = Decision::Accepted(accepted.clone()); Ok(CellSubmissionOutcome::Accepted(accepted)) } - Ok(None) => { + Ok(None) if !uncertain => { *decision = Decision::Rejected; Ok(CellSubmissionOutcome::Rejected(self.rejected())) } + Ok(None) => { + // This refusal says nothing about an earlier attempted append. + *decision = Decision::Uncertain; + Err(RuntimeError::Overloaded) + } Err(error) => { - *decision = Decision::Open; + *decision = Decision::Uncertain; Err(error) } } @@ -139,8 +146,25 @@ impl CellPreparedCommand { } } + /// Only a never-attempted preparation may be cancelled locally. The same + /// lock serializes the seal with every delayed or currently entering clone. + pub(crate) fn cancel_before_submission( + &self, + ) -> Result, RuntimeError> { + let mut decision = self + .submission + .decision + .lock() + .map_err(|_| RuntimeError::Protocol("submission state poisoned".into()))?; + if matches!(*decision, Decision::Open) { + *decision = Decision::Rejected; + } + Ok(matches!(*decision, Decision::Rejected).then(|| self.rejected())) + } + /// Recover a lost rejection response without asking a closed or replaced - /// runtime. Open and accepted attempts never yield a rejection proof. + /// runtime. Open, entering, uncertain and accepted attempts never yield a + /// rejection proof without an explicit seal of a never-attempted preparation. pub fn rejection(&self) -> Result, RuntimeError> { let decision = self .submission @@ -155,6 +179,10 @@ impl CellPreparedCommand { } } +#[cfg(test)] +#[path = "submission_tests.rs"] +mod tests; + impl CellProposalRejection { pub(crate) fn belongs_to_engine(&self, engine: &Arc<()>) -> bool { Arc::ptr_eq(engine, &self.submission.engine) diff --git a/crates/cellar/durability/raft/execution/submission_tests.rs b/crates/cellar/durability/raft/execution/submission_tests.rs new file mode 100644 index 000000000..ae6499c3d --- /dev/null +++ b/crates/cellar/durability/raft/execution/submission_tests.rs @@ -0,0 +1,73 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{ + ClusterId, GroupId, IncarnationId, LogIndex, LogPosition, NamespaceId, +}; + +fn prepared() -> CellPreparedCommand { + CellPreparedCommand::new( + vec![1], + ProposalId::new([1; 16]).unwrap(), + Term::new(1), + GroupHistory { + cluster: ClusterId::new([2; 16]).unwrap(), + namespace: NamespaceId::new([3; 16]).unwrap(), + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }, + Arc::new(()), + ) +} + +#[test] +fn cancellation_seals_every_delayed_submission_clone() { + let command = prepared(); + let delayed = command.clone(); + let proof = command.cancel_before_submission().unwrap().unwrap(); + assert!(delayed.matches_rejection(&proof)); + assert!(matches!( + delayed.submit(&command.submission.engine, command.group, || panic!( + "sealed clone reached append" + )), + Ok(CellSubmissionOutcome::Rejected(_)) + )); +} + +#[test] +fn attempted_error_cannot_turn_into_cancellation_or_capacity_proof() { + let command = prepared(); + assert!(command + .submit(&command.submission.engine, command.group, || Err( + RuntimeError::Protocol("uncertain append".into()) + )) + .is_err()); + assert!(command.cancel_before_submission().unwrap().is_none()); + assert!(matches!( + command.submit(&command.submission.engine, command.group, || Ok(None)), + Err(RuntimeError::Overloaded) + )); + assert!(command.rejection().unwrap().is_none()); + assert!(command.cancel_before_submission().unwrap().is_none()); +} + +#[test] +fn entering_or_accepted_submission_cannot_be_cancelled() { + let command = prepared(); + let accepted = CellGroupProposal { + group: command.group, + position: LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + digest: [9; 32], + }; + assert!(matches!( + command.submit(&command.submission.engine, command.group, || { + assert!(command.cancel_before_submission().unwrap().is_none()); + Ok(Some(accepted)) + }), + Ok(CellSubmissionOutcome::Accepted(_)) + )); + assert!(command.cancel_before_submission().unwrap().is_none()); +} diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index be8d8d825..d379fad36 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -174,6 +174,18 @@ pub struct CellSqlPrepared { } impl CellSqlPrepared { + pub(crate) fn cancel_before_submission( + &self, + ) -> Result, RuntimeError> { + Ok(self + .prepared + .cancel_before_submission()? + .map(|proof| CellSqlRejection { + proof, + _bytes: self._bytes.clone(), + })) + } + pub async fn submit( &self, runtime: &crate::raft::cell_groups::CellGroupRuntime, @@ -235,6 +247,25 @@ impl CellSqlReceipt { } } +// A stage error after a received worker reply cannot hide an installed producer: +// Stage inserts pending ownership only on its successful Prepared branch. +// A lost reply is deliberately different and requires recovery/joined shutdown. +pub(crate) struct CellSqlStageFailure { + pub error: ExecutionError, + pub unresolved: bool, +} + +impl CellSqlStageFailure { + fn known(error: ExecutionError) -> Self { + Self { + error, + unresolved: false, + } + } +} + +type SqlReply = Result; + pub enum CellSqlOutcome { Prepared(CellSqlPrepared), Receipt(Box), @@ -391,11 +422,15 @@ impl CellGroupSqlPool { Ok(()) } - async fn call( + async fn call(&self, cell: CellHistory, action: worker::Action) -> SqlReply { + self.send(cell, action)?.await.map_err(|_| unavailable())? + } + + fn send( &self, cell: CellHistory, action: worker::Action, - ) -> Result { + ) -> Result, ExecutionError> { self.admit()?; let hash = cell_image_history_digest(&cell); let index = u64::from_le_bytes( @@ -431,7 +466,7 @@ impl CellGroupSqlPool { .timer(crate::raft::performance::Operation::SqlQueue), })) .map_err(|_| unavailable())?; - wait.await.map_err(|_| unavailable())? + Ok(wait) } /// The owning worker retains the callback and admission even if this waiter @@ -468,10 +503,24 @@ impl CellGroupSqlPool { input: Vec, allow_new: bool, ) -> Result { + self.stage_registered_owned(options, authority, attempt, class, input, allow_new) + .await + .map_err(|failure| failure.error) + } + + pub(crate) async fn stage_registered_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + allow_new: bool, + ) -> Result { if input.len() > options.limits().envelope.max_payload_bytes { - return Err(overloaded()); + return Err(CellSqlStageFailure::known(overloaded())); } - self.stage_callback( + self.stage_owned( options, authority, attempt, @@ -491,7 +540,42 @@ impl CellGroupSqlPool { attempt: CommandAttempt, allow_new: bool, callback: worker::Callback, - ) -> Result { + ) -> SqlReply { + self.stage_owned(options, authority, attempt, allow_new, callback) + .await + .map_err(|failure| failure.error) + } + + async fn stage_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: worker::Callback, + ) -> Result { + let (cell, action) = self + .stage_request(options, authority, attempt, allow_new, callback) + .map_err(CellSqlStageFailure::known)?; + let wait = self + .send(cell, action) + .map_err(CellSqlStageFailure::known)?; + wait.await + .map_err(|_| CellSqlStageFailure { + error: unavailable(), + unresolved: true, + })? + .map_err(CellSqlStageFailure::known) + } + + fn stage_request( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: worker::Callback, + ) -> Result<(CellHistory, worker::Action), ExecutionError> { self.admit()?; if !authority.belongs_to_engine(&self.inner.engine_owner) { return Err(ExecutionError::Invalid("cell SQL engine owner")); @@ -521,7 +605,7 @@ impl CellGroupSqlPool { self.inner .admission .reserve(&cell, authority.route().history, bytes)?; - self.call( + Ok(( cell, worker::Action::Stage(Box::new(worker::Stage { options, @@ -532,8 +616,7 @@ impl CellGroupSqlPool { lease, bytes, })), - ) - .await + )) } /// Initialize one logical cell on the existing blocking pool. The creation diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 17fb7b2fc..7de193745 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -11,6 +11,41 @@ mod fault; #[path = "support/cell_group_runtime.rs"] mod runtime_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_survive_sql_disconnect() -> Result<()> { + runtime_fixture::owned_commands(0, 0).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_seal_late_preparations() -> Result<()> { + runtime_fixture::owned_commands(0, 1).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_survive_accepted_disconnect() -> Result<()> { + runtime_fixture::owned_commands(1, 0).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_resolve_after_response_deadline() -> Result<()> { + runtime_fixture::owned_commands(1, 1).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_join_cancelled_shutdown() -> Result<()> { + runtime_fixture::owned_commands(0, 2).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_release_completed_callback_errors() -> Result<()> { + runtime_fixture::owned_command_failure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_owned_commands_join_a_lost_sql_reply() -> Result<()> { + runtime_fixture::owned_command_failure(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_controller_drives_tls_apply_and_bounded_retry() -> Result<()> { runtime_fixture::controller_replication(false).await diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs index 64e529634..a870efdac 100644 --- a/crates/cellar/tests/support/cell_group_controller.rs +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -8,15 +8,24 @@ use std::sync::{ Mutex, }; -fn limits() -> CellGroupControllerLimits { +pub(super) fn limits() -> CellGroupControllerLimits { CellGroupControllerLimits { max_groups: 2, max_parallel_apply: 2, retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, } } async fn start(fixture: &mut RuntimeCluster) -> Result> { + start_with_limits(fixture, limits()).await +} + +pub(super) async fn start_with_limits( + fixture: &mut RuntimeCluster, + bounds: CellGroupControllerLimits, +) -> Result> { fixture.connect_tcp().await?; let mut controllers = Vec::new(); for (node, events) in fixture.events.drain(..).enumerate() { @@ -35,7 +44,7 @@ async fn start(fixture: &mut RuntimeCluster) -> Result> .0 .clone(), selected, - limits(), + bounds, ) .await?, ); diff --git a/crates/cellar/tests/support/cell_group_controller_commands.rs b/crates/cellar/tests/support/cell_group_controller_commands.rs new file mode 100644 index 000000000..04d2daa80 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_commands.rs @@ -0,0 +1,481 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Response lifetimes through real TLS, Raft Engine, SQLite and physical apply. +use super::apply_faults::physical; +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +async fn invocation( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + timeout: Duration, + retry_until_ms: u64, +) -> Result { + Ok(dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"owned".to_vec(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?) +} + +// phase=0 holds callback execution; phase=1 holds the committed manifest sync. +// mode=0 disconnects, mode=1 expires the response, mode=2 cancels node close. +pub(crate) async fn owned_commands(phase: u8, mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let callback_entered = entered.clone(); + let callback_held = held.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 && phase == 0 { + let _ = callback_entered + .lock() + .unwrap() + .take() + .context("callback signal")? + .send(()); + callback_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"owned-result".to_vec()) + }, + )?); + let (mut fixture, route, a, b) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let bounds = CellGroupControllerLimits { + max_commands: 1, + ..controller::limits() + }; + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + assert!(fixture.events.is_empty()); // No fixture pumping below this point. + let keys = VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys.clone(), + CellDispatchLimits { + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + if phase == 1 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply signal") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("apply release"); + })); + } + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let mut used = invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?; + used.begin().await?; + let original_deadline = used.attempt().context("routed invocation")?.deadline_ms(); + let refused = controllers[0] + .command(used, fixture.cell_options(&b).await?) + .err() + .context("fresh invocation required")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Invalid(_) + )); + assert_eq!( + refused + .invocation + .attempt() + .context("returned original owner")? + .deadline_ms(), + original_deadline + ); + drop(refused); + let mut oversized = fixture.cell_options(&b).await?; + match &mut oversized { + CellSqlOptions::Materialized(o) => { + o.directory = std::path::PathBuf::from("x".repeat(4097)) + } + CellSqlOptions::Extents(o) => { + o.cache_directory = std::path::PathBuf::from("x".repeat(4097)) + } + } + let refused = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?, + oversized, + ) + .err() + .context("option admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + drop(refused); + assert_eq!(dispatcher.usage().1, 0); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let timeout = if mode == 1 { + Duration::from_millis(400) + } else { + Duration::from_secs(15) + }; + let call = controllers[0].command( + invocation(&dispatcher, &class, &a, timeout, retry_until).await?, + fixture.cell_options(&a).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(dispatcher.usage().1, 1); + if phase == 1 { + let status = fixture.runtimes[0].status(route.history).await?; + assert!(status.committed > status.applied.index); + } + let rejected = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(10), + retry_until, + ) + .await?, + fixture.cell_options(&b).await?, + ) + .err() + .context("bounded admission")?; + assert!(!rejected.error.ambiguous); + assert!(matches!( + rejected.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + assert!(rejected.invocation.attempt().is_none()); + assert_eq!(dispatcher.usage().1, 2); + drop(rejected); + assert_eq!(dispatcher.usage().1, 1); + if mode == 1 { + let error = call.response().await.err().context("response deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + if mode == 2 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + assert!(CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + } + // Neither a lost waiter nor its deadline can free the entered producer. + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(dispatcher.usage().1, 1); + release.send(())?; + if mode == 2 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + } else { + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 + && pool.usage()?.active_cells == 0 + && dispatcher.usage().1 == 0) + }) + .await?; + let sealed = phase == 0 && mode == 1; + let state = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!( + state.cell().version.sequence.get(), + if sealed { 1 } else { 2 } + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + // Retrying the same identity reads a committed receipt, or executes + // once after the exact untouched preparation was sealed and closed. + let retry = controllers[0].command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )?; + let response = retry.response().await?; + assert_eq!(response.receipt().receipt.result(), b"owned-result"); + assert_eq!(response.receipt().version.history, a); + assert_eq!(response.receipt().version.sequence.get(), 2); + assert_eq!(calls.load(Ordering::SeqCst), if sealed { 2 } else { 1 }); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + assert_eq!( + keys.verify_cell(response.token(), &a, route.schema_digest, now)?, + response.receipt().observation() + ); + drop(response); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + // Successful, unpolled replies also retain node admission. + let unpolled = controllers[0].command( + invocation( + &dispatcher, + &class, + &b, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&b).await?, + )?; + wait_for(async || Ok(dispatcher.usage().1 == 0 && pool.usage()?.active_cells == 0)) + .await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + let refused = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + ) + .err() + .context("unpolled result admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + assert_eq!(unpolled.response().await?.receipt().version.history, b); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + } + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn owned_command_failure(lost_reply: bool) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let directory = root + .path() + .join(if rooted { "rooted-0" } else { "node-1/images" }); + let faults = physical::Faults::new(directory, None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + if count.fetch_add(1, Ordering::SeqCst) == 0 { + if lost_reply { + // Application panics are contained by the transaction. + // Lose the worker reply during subsequent physical capture. + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(|| { + panic!("injected lost SQL stage reply during image capture"); + })); + } else { + anyhow::bail!("injected callback refusal"); + } + } + Ok(b"retry-result".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let call = controllers[0].command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )?; + let error = call.response().await.err().context("failed stage")?; + assert!(error.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 1); + if lost_reply { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Unavailable)) + )); + assert!(matches!( + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await?, + Err(CellGroupControllerError::Command(_)) + )); + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + assert!(pool.close().await.is_err()); + fixture.appliers.remove(&(0, route.history)); + } else { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + assert!(!controllers[0].usage().closing); + assert_eq!(pool.usage()?.active_cells, 0); + let state = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(state.cell().version.sequence.get(), 1); + let retried = controllers[0] + .command( + invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry_until, + ) + .await?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!(retried.receipt().receipt.result(), b"retry-result"); + assert_eq!(retried.receipt().version.sequence.get(), 2); + assert_eq!(calls.load(Ordering::SeqCst), 2); + drop(retried); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index aee3f68b4..0510999e0 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -652,6 +652,7 @@ mod sql; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; +pub(super) use sql::{owned_command_failure, owned_commands}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index ce5c8dd29..6e7974a8d 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -528,9 +528,12 @@ pub(crate) use command_dispatch::commands as routed_commands; #[path = "cell_group_controller.rs"] mod controller; +#[path = "cell_group_controller_commands.rs"] +mod controller_commands; pub(crate) use controller::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; +pub(crate) use controller_commands::{owned_command_failure, owned_commands}; pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { Box::pin(command_dispatch::commands_with_transport(rooted, true)).await diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index 9dd84a9b3..1bc14b8af 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -155,10 +155,25 @@ pub(super) async fn setup( rooted: bool, class: &RaftClass, ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { - let cluster = Cluster::with_storage_schema( - RaftRuntimeOptions::default(), + setup_in( + rooted, + class, tempfile::tempdir()?, Arc::new(DirectFileSystem), + ) + .await +} + +pub(super) async fn setup_in( + rooted: bool, + class: &RaftClass, + root: tempfile::TempDir, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let cluster = Cluster::with_storage_schema( + RaftRuntimeOptions::default(), + root, + filesystem.clone(), class.schema_digest(), ) .await?; @@ -168,7 +183,7 @@ pub(super) async fn setup( for (n, voter) in fixture.cluster.voters.iter_mut().enumerate() { voter.extents = Some(NodeExtentStore::open( fixture.cluster._root.path().join(format!("rooted-{n}")), - Arc::new(DirectFileSystem), + filesystem.clone(), Default::default(), )?); } diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 73ad6a446..274646fb1 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 34 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 47 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 6a9e50346..b66ca569e 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -486,11 +486,18 @@ "crates/cellar/tests/support/cell_group_sql_registered_queries.rs", "crates/cellar/durability/raft/cell_groups/dispatch/mod.rs", "crates/cellar/tests/support/cell_group_sql_dispatch.rs", - "crates/cellar/durability/raft/cell_groups/dispatch/command.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs", "crates/cellar/tests/support/cell_group_sql_command_dispatch.rs", "crates/cellar/durability/raft/transport_runtime.rs", "crates/cellar/durability/raft/transport_shared.rs", - "crates/cellar/tests/support/cell_group_tcp.rs" + "crates/cellar/tests/support/cell_group_tcp.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs", + "crates/cellar/durability/raft/cell_groups/controller/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/worker.rs", + "crates/cellar/durability/raft/cell_groups/controller/commands.rs", + "crates/cellar/durability/raft/execution/submission_tests.rs", + "crates/cellar/tests/support/cell_group_controller.rs", + "crates/cellar/tests/support/cell_group_controller_commands.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -582,6 +589,24 @@ "catalog_commands_replicate_over_physical_tls", "rooted_catalog_commands_replicate_over_physical_tls" ] + }, + { + "action": "Own the exclusive shared event stream and bounded per-group apply jobs in one node controller. Preserve exact owner bindings and retain physical tickets through backpressure and joined shutdown. Admit fresh command invocations independently from response waiters; retain bounded ownership beyond disconnect/deadline until exact receipt, proven non-commit or joined shutdown. Seal only never-attempted preparations across all clones. Distinguish completed stage errors from lost worker replies and never automatically replay uncertain callbacks.", + "tests": [ + "group_controller_drives_tls_apply_and_bounded_retry", + "rooted_group_controller_drives_tls_apply_and_bounded_retry", + "group_controller_checks_exact_owners", + "group_controller_cancelled_close_joins_sql", + "group_controller_apply_failure_and_shutdown_preserve_selected_root", + "rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root", + "controller_owned_commands_survive_sql_disconnect", + "controller_owned_commands_seal_late_preparations", + "controller_owned_commands_survive_accepted_disconnect", + "controller_owned_commands_resolve_after_response_deadline", + "controller_owned_commands_join_cancelled_shutdown", + "controller_owned_commands_release_completed_callback_errors", + "controller_owned_commands_join_a_lost_sql_reply" + ] } ], "test_cases": [ @@ -900,6 +925,136 @@ "I03", "I12" ] + }, + { + "name": "group_controller_drives_tls_apply_and_bounded_retry", + "oracle": "Two groups replicate over real TLS using one SQL worker per node. The controller exclusively drives events, preserves per-group apply order and retries SQL capacity without fixture pumping.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_group_controller_drives_tls_apply_and_bounded_retry", + "oracle": "Repeat autonomous multi-group TLS replication, bounded SQL backoff and exact receipt publication with extent images.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_checks_exact_owners", + "oracle": "Refuse foreign runtime/event/transport and engine/SQL/applier combinations even within the same execution domain.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_cancelled_close_joins_sql", + "oracle": "Cancel a close waiter or drop the final public controller handle while physical work is blocked. Retain claims and tickets until physical SQL and runtime joins complete.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "group_controller_apply_failure_and_shutdown_preserve_selected_root", + "oracle": "A real manifest sync failure closes the controller without advancing the selected root. Interrupted physical preparation cannot publish or allow premature replacement.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root", + "oracle": "Repeat manifest sync failure and joined controller shutdown with extent images; preserve the engine-selected root.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_survive_sql_disconnect", + "oracle": "Drop the response while a registered SQL callback is physically held. Retain bounded controller/producer/invocation admission, then settle the exact receipt without callback replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_seal_late_preparations", + "oracle": "Expire the response while a callback is held. Seal the eventual never-submitted preparation, join private cleanup and leave the selected cell unchanged. An explicit retry may execute once, in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_survive_accepted_disconnect", + "oracle": "Drop the response after commitment while manifest sync is held. Preserve command ownership until exact publication, and return the original receipt on identity retry without replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_resolve_after_response_deadline", + "oracle": "Expire the original response deadline after commitment while manifest sync is held. Continue bounded resolution and settle the exact receipt without replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_join_cancelled_shutdown", + "oracle": "Cancel a close waiter during a physically blocked callback. Retain the invocation and engine/SQL claims until joined shutdown; release all command admission only afterward, in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_release_completed_callback_errors", + "oracle": "A received callback error discards its private mutation and releases command ownership without closing the node. Explicit identity retry succeeds exactly once; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_owned_commands_join_a_lost_sql_reply", + "oracle": "An injected physical image-capture panic loses the stage reply. Preserve ambiguous invocation ownership through controller failure and physical SQL joins, then release admission; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From 10617346819a6fa052d6c35748fdbc52ec8b6c10 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 05:51:51 -0700 Subject: [PATCH 091/568] docs: record qualified CG-04 command ownership checkpoint --- docs/design/multi-raft/cell-groups/README.md | 21 +++---- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 47 ++++++++------- .../design/multi-raft/cell-groups/progress.md | 60 +++++++++++++++++++ 4 files changed, 97 insertions(+), 35 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index a221e246b..42683ef5c 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -173,18 +173,15 @@ production controller ownership and reconciliation after caller disconnect remai required, followed by routed enrollment and SDK/shared-query RPC integration. Those fixtures still drive runtime events. See [progress](progress.md) for exact limits. -At `ee684c33`, a fixed-roster `CellGroupController` drives shared TLS and SQL -apply without fixture pumping. Exact owner bindings, bounded per-group apply, -physical sync failure and joined shutdown pass six new native cases. The partial -gate passes 58 execution cases and eight commands; transfer -`.cellar/cell-groups-evidence/CG-04-event-controller-qualified/` (244 source inputs, one -executable) and the preceding `CG-04-event-controller/` regression gate with prior -evidence. That earlier run passed 289 Rust cases; the successor only corrects a -duplicated test fixture import identified by Clippy. The critical next gap is -caller-independent -command ownership and bounded reconciliation after the response deadline, -followed by routed enrollment and SDK/shared-query RPC integration. Fixed roster -and node-wide failure handling also remain incomplete. See [progress](progress.md). +At `8c615ac9`, the shared controller owns fresh commands independently of +response disconnect/deadline. Exact receipt resolution, never-attempted preparation +sealing, bounded retained responses and joined physical shutdown pass seven new +native cases in both image modes plus three submission unit cases. The partial +gate passes 83 Rust cases and nine commands; transfer +`.cellar/cell-groups-evidence/CG-04-command-owner/` (248 source inputs, three +executables) with prior evidence. The next critical delivery is routed enrollment, +then remote shared command/query dispatch and public SDK integration. Fixed roster +and node-wide failure handling also remain gaps. See [progress](progress.md). The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 6cc037e0c..f9a35af80 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ee684c3394ffccbfa84bfc5d94876de281f68b0b", + "source_revision": "8c615ac91bc3c13e6c25da4f4ab584a73e609207", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 86efd7e41..cf295de84 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,19 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `ee684c33`, adding a fixed-roster node +The newest reviewed source baseline is `8c615ac9`, adding bounded command +ownership to the shared node controller. Response disconnect/deadline cannot drop +entered SQL or ambiguous submission ownership. Only never-attempted preparations +can be sealed; exact receipts or newer applied-term proof resolve submitted work. +Received stage errors release completed ownership, while lost replies require +resolution or joined shutdown. The partial record is +`.cellar/cell-groups-evidence/CG-04-command-owner/checkpoint.json`: 83 selected Rust +cases, nine commands, 248 source inputs and three executables. Seven new +native cases each cover both image modes; three unit cases cover submission seals. +Transfer this directory with earlier evidence. CG-04 remains IN_PROGRESS; routed +enrollment, remote dispatch, SDK integration, dynamic roster and per-group failure +isolation remain gaps. The public host has not cut over; RustFS/load is unqualified. +The preceding reviewed source baseline is `ee684c33`, adding a fixed-roster node controller that drives existing shared TLS/apply owners without fixture pumping. Its partial record is `.cellar/cell-groups-evidence/CG-04-event-controller-qualified/checkpoint.json`: 58 shared-execution cases, eight commands, 244 source inputs and @@ -392,26 +404,19 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Move fresh phased command invocations into `CellGroupController`, independently - from ingress response waiters. The fixed-roster event controller now drives - physical `GroupHistory` transport and SQL apply without fixture pumping; extend - that owner rather than introducing cell-specific tasks. Keep the original - response deadline while retaining bounded resolution ownership afterward. - Qualify caller disconnect/timeout after SQL entry and provisional acceptance: - resolve the exact receipt or proven non-commit without replaying uncertain - callbacks. Existing routed command tests still retain their invocation manually. - Before cancellation cleanup, extend `execution/submission.rs`: an attempted - submission error currently resets to Open; that is not a pre-append proof. - Seal only never-attempted preparations and retain an explicit uncertain state - after attempted submission errors. Serialize the seal with delayed clones and - prove physical cleanup with exact rejection capabilities. - Missing origin producers on followers still need a safe leader receipt-lookup - route; never infer non-commit from missing local state. Then add catalog-routed - enrollment with stable creation identity, shared query RPC and the public - `RaftCellHandle` binding. Requested and namespace S3 durability continue to - refuse until CG-06 supplies real archive proofs. The event controller currently - has a fixed roster and closes the node on fatal apply errors; add dynamic group - registration and per-group failure isolation before claiming runtime completion. +1. Complete catalog-routed enrollment with a stable creation identity on the + shared controller and SQL pool. Reuse the existing enrollment authority, + original producer and applied-term reconciliation; preserve cancellation and + ambiguous-result ownership just qualified for commands. Then connect remote + shared command/query dispatch and the public `RaftCellHandle` binding. Fresh + local commands now belong to `CellGroupController`, independently of response + lifetime. Preserve their original deadline and bounded resolution admission; + do not add per-cell tasks or an expiry that drops uncertainty. Missing origin + producers on followers still need safe leader receipt lookup; missing local + state is never proof of non-commit. Requested and namespace S3 durability must + continue to refuse until CG-06 supplies real archive proofs. The controller + still has a fixed roster and node-wide fatal-error handling; implement dynamic + registration and per-group failure isolation before runtime completion. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1ee1dd0ec..1b910807c 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,66 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 controller-owned commands + +Qualified source: `8c615ac91bc3c13e6c25da4f4ab584a73e609207`. + +`CellGroupController::command` transfers a fresh local invocation into the +existing node worker. The separate response handle preserves the original +deadline; dropping or timing out that handle does not cancel SQL preparation, +submission or receipt resolution. Count/byte admission covers queued commands, +active resolution and unconsumed responses, including errors. SQL capture/apply +and result bytes keep their separate pool budgets. Admission refusal returns the +original invocation, preserving any recovery state. One shared progress timer +and applied-event notifications drive retries; continuous traffic cannot reset +the timer and indefinitely postpone progress. + +A late preparation that has never attempted submission is sealed across all +clones and closed through its exact rejection capability. Entering, accepted and +uncertain attempts cannot produce that seal. An attempted submission error stays +uncertain; a later capacity refusal cannot prove the earlier attempt absent. +After submission, resolution requires the exact receipt or the existing newer +applied-term absence proof. It never replays a callback automatically. Retention +is bounded by admission, not by a timeout that discards uncertain state. On a +fatal unresolved command error, the controller retains the invocation through +physical SQL/runtime joins before releasing ownership. + +SQL staging now distinguishes a received completed error from a lost worker +reply. The former cannot hide an installed producer and releases this invocation +without closing the node; the latter remains ambiguous and retained. The public +stage API keeps its existing error type. Physical image capture, callbacks and +manifest publication continue to run on the existing SQL workers. + +Seven new native cases each run in both materialized and extent modes over three +real in-process TLS endpoints. They hold callbacks or committed manifest sync, +disconnect or expire responses, assert bounded ownership and exact identity retry, +verify unconsumed-response charging, and cancel shutdown while physical work is +held. Completed callback errors permit an explicit successful retry. A physical +image-capture panic loses the stage reply and requires joined cleanup. Three unit +cases prove clone sealing, entering/accepted refusal and uncertainty after an +attempted error. The initial fixture used an invalid retry horizon; the corrected +fixture preserves one valid horizon. An initial callback-panic injection was +contained by Cellar's transaction and therefore did not simulate a lost reply; +the final injection occurs during subsequent physical capture. Failed logs remain +with the checkpoint and are not counted as qualification. + +The gate passes 83 selected Rust cases (65 execution, 15 catalog, three submission) +and nine commands: affected tests, all-target Clippy for Cellar/cellar-logic/ +cellar-ltx with warnings denied, formatting, layout, dependencies, docs, source +preflight and 37 plan-validator cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-command-owner/checkpoint.json` binds 248 +source inputs and three copied test binaries. Transfer its complete directory +with prior evidence. This qualifies local shared-controller command ownership, +not real RustFS, separate-process crash, power-loss or load behavior. + +CG-04 remains IN_PROGRESS. The critical next delivery is catalog-routed enrollment +with stable creation identity, then remote shared command/query dispatch and the +public SDK binding to this owner. Missing origin producers on followers still +need a safe leader receipt-lookup route. Dynamic roster, per-group failure +isolation and configuration/cache lifecycle are unresolved runtime gaps. +Complete-group snapshots/archive, effects, topology, RustFS/load qualification and +legacy removal remain later gates; the public host has not cut over. + ## CG-04 autonomous event controller Qualified source: `ee684c3394ffccbfa84bfc5d94876de281f68b0b`. From bd3069d7723a8d4ec118bf465fb65633f762e3dd Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 06:06:43 -0700 Subject: [PATCH 092/568] feat: enroll cells through the shared node controller --- crates/cellar/application/consensus/class.rs | 64 +++ .../consensus/class_initializer_tests.rs | 31 ++ .../raft/cell_groups/controller/calls.rs | 49 ++ .../raft/cell_groups/controller/commands.rs | 61 +-- .../cell_groups/controller/enrollments.rs | 169 +++++++ .../raft/cell_groups/controller/mod.rs | 10 +- .../raft/cell_groups/controller/requests.rs | 49 ++ .../raft/cell_groups/controller/worker.rs | 13 +- .../raft/cell_groups/dispatch/enrollment.rs | 431 ++++++++++++++++ .../raft/cell_groups/dispatch/mod.rs | 3 + .../cellar/durability/raft/cell_groups/mod.rs | 10 +- .../raft/host_sql/cell_groups/mod.rs | 40 +- .../tests/native_cell_groups_execution.rs | 40 ++ .../cell_group_controller_enrollment.rs | 470 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 6 + .../cell-groups/implementation-plan.md | 2 +- .../multi-raft/cell-groups/protocols.md | 4 + docs/design/multi-raft/cell-groups/tasks.json | 101 +++- 19 files changed, 1495 insertions(+), 62 deletions(-) create mode 100644 crates/cellar/application/consensus/class_initializer_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/calls.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/enrollments.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/requests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs create mode 100644 crates/cellar/tests/support/cell_group_controller_enrollment.rs diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index 269e6a0db..bf09bcf30 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -273,6 +273,13 @@ type CommandCallback = dyn for<'a> Fn(&CommandContext<'a>, &[u8]) -> anyhow::Result> + Send + Sync; type QueryCallback = dyn for<'a> Fn(&QueryContext<'a>, &[u8]) -> anyhow::Result> + Send + Sync; +type InitializerCallback = + dyn Fn(&CellHistory, &CommandTransaction<'_>) -> anyhow::Result<()> + Send + Sync; + +struct Initializer { + digest: [u8; 32], + callback: Arc, +} struct Command { digest: [u8; 32], @@ -298,6 +305,7 @@ pub struct RaftClass { base_schema_digest: [u8; 32], schema_digest: [u8; 32], commands: BTreeMap, + initializer: Option, system_commands: BTreeMap, queries: BTreeMap, blobs: Option, @@ -337,6 +345,7 @@ impl RaftClass { base_schema_digest: schema_digest, schema_digest, commands: BTreeMap::new(), + initializer: None, system_commands: BTreeMap::new(), queries: BTreeMap::new(), blobs: None, @@ -373,6 +382,49 @@ impl RaftClass { self.base_schema_digest } + /// Register schema initialization for shared-group enrollment. The digest + /// participates in voting compatibility. Initialization runs once on a + /// private SQL transaction; only its committed creation record is success. + /// The old cell-scoped host refuses this registration until its replacement. + pub fn initializer( + mut self, + digest: [u8; 32], + callback: impl Fn(&CellHistory, &CommandTransaction<'_>) -> anyhow::Result<()> + + Send + + Sync + + 'static, + ) -> Result { + if digest == [0; 32] || self.initializer.is_some() { + return Err(RegistrationError::Invalid("initializer registration")); + } + self.initializer = Some(Initializer { + digest, + callback: Arc::new(callback), + }); + self.bind_command_manifest(); + Ok(self) + } + + pub(crate) fn require_initializer(&self) -> Result<(), RegistrationError> { + self.initializer + .as_ref() + .map(|_| ()) + .ok_or(RegistrationError::Invalid("missing initializer")) + } + + pub(crate) fn invoke_initializer( + &self, + cell: &CellHistory, + transaction: &CommandTransaction<'_>, + ) -> anyhow::Result<()> { + let initializer = self + .initializer + .as_ref() + .ok_or(RegistrationError::Invalid("missing initializer"))?; + let _guard = CallbackGuard::enter()?; + (initializer.callback)(cell, transaction) + } + /// Opt in to bounded immutable bytes replicated with the cell. This changes /// the class identity and requires a fresh compatible namespace. pub fn immutable_blobs( @@ -470,6 +522,10 @@ impl RaftClass { &self, host: &crate::raft::host::RaftHostOptions, ) -> anyhow::Result<()> { + anyhow::ensure!( + self.initializer.is_none(), + "registered initializers require the shared cell-group controller" + ); if let Some(outbox) = &self.outbox { anyhow::ensure!(host.effects.is_some_and(|options|options.metadata_bytes>=512*1024) && host.max_input_bytes>=32*1024 @@ -898,6 +954,10 @@ impl RaftClass { let mut hash = Sha256::new(); hash.update(b"cellar-cell-class-1"); hash.update(self.base_schema_digest); + if let Some(initializer) = &self.initializer { + hash.update(b"initializer-1"); + hash.update(initializer.digest); + } if let Some(limits) = self.blobs { hash.update(b"immutable-blobs-1"); for size in [ @@ -971,3 +1031,7 @@ pub(super) fn validate_name(name: &str) -> Result<(), RegistrationError> { } Ok(()) } + +#[cfg(test)] +#[path = "class_initializer_tests.rs"] +mod initializer_tests; diff --git a/crates/cellar/application/consensus/class_initializer_tests.rs b/crates/cellar/application/consensus/class_initializer_tests.rs new file mode 100644 index 000000000..56f1ae306 --- /dev/null +++ b/crates/cellar/application/consensus/class_initializer_tests.rs @@ -0,0 +1,31 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[test] +fn initializer_contract_binds_schema_and_refuses_old_host() -> anyhow::Result<()> { + let base = || RaftClass::new("Counter", 1, [81; 32]); + let ordinary = base()?.command("Add", [82; 32], |_, _| Ok(vec![]))?; + let initial = + base()? + .initializer([90; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + let reordered = base()? + .command("Add", [82; 32], |_, _| Ok(vec![]))? + .initializer([90; 32], |_, _| Ok(()))?; + let different = + base()? + .initializer([91; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + assert_eq!(initial.schema_digest(), reordered.schema_digest()); + assert_ne!(initial.schema_digest(), ordinary.schema_digest()); + assert_ne!(initial.schema_digest(), different.schema_digest()); + ordinary.validate_host(&Default::default())?; + assert!(initial.validate_host(&Default::default()).is_err()); + assert!(base()?.initializer([0; 32], |_, _| Ok(())).is_err()); + assert!(base()? + .initializer([90; 32], |_, _| Ok(()))? + .initializer([90; 32], |_, _| Ok(())) + .is_err()); + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/calls.rs b/crates/cellar/durability/raft/cell_groups/controller/calls.rs new file mode 100644 index 000000000..66d4fe6d5 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/calls.rs @@ -0,0 +1,49 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Response deadlines do not own the lifetime of admitted mutations. +use super::*; +use crate::raft::budget::Permit; +use tokio::sync::oneshot; + +pub(super) struct Delivery { + pub result: Result>, + pub _permit: Arc, +} + +pub(super) struct Call { + pub receiver: oneshot::Receiver>, + pub identity: I, + pub deadline: u64, + pub domain: crate::asyncrt::DomainToken, + pub failure: fn(I, RuntimeError) -> E, +} + +impl Call { + pub async fn response(self) -> Result> { + let failure = |reason| Arc::new((self.failure)(self.identity, reason)); + let now = self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))?; + if now >= self.deadline { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + let timer = self + .domain + .sleep_until(self.deadline) + .map_err(|_| failure(RuntimeError::Closed))?; + let delivery = crate::asyncrt::select! { + result = self.receiver => result.map_err(|_| failure(RuntimeError::Closed))?, + _ = timer => return Err(failure(RuntimeError::DeadlineExceeded)), + }; + if self + .domain + .mono_ms() + .map_err(|_| failure(RuntimeError::Closed))? + >= self.deadline + { + return Err(failure(RuntimeError::DeadlineExceeded)); + } + delivery.result + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs index 4815ae092..7f42dd903 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/commands.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -1,6 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Bounded invocation ownership is independent of response-waiter lifetime. +use super::calls::Delivery; use super::*; use crate::raft::budget::{Budget, Permit}; use crate::raft::cell_groups::{ @@ -12,15 +13,10 @@ use tokio::sync::{mpsc, oneshot, watch}; pub(super) struct Request { invocation: CellCommandInvocation, options: CellSqlOptions, - reply: oneshot::Sender, + reply: oneshot::Sender>, permit: Arc, } -struct Delivery { - result: Result>, - _permit: Arc, -} - pub(super) struct Completed { pub invocation: CellCommandInvocation, pub error: Option>, @@ -30,10 +26,7 @@ pub(super) struct Completed { /// Dropping or timing out this response never cancels the admitted command. /// An unpolled reply retains node admission, including error-only replies. pub struct CellCommandCall { - receiver: oneshot::Receiver, - command: CommandId, - deadline: u64, - domain: crate::asyncrt::DomainToken, + inner: calls::Call, } /// Admission did not transfer ownership. In particular, a rejected non-fresh @@ -73,35 +66,11 @@ fn error(command: CommandId, ambiguous: bool, reason: RuntimeError) -> CellComma impl CellCommandCall { pub async fn response(self) -> Result> { - let failure = |reason| Arc::new(error(self.command, true, reason)); - let now = self - .domain - .mono_ms() - .map_err(|_| failure(RuntimeError::Closed))?; - if now >= self.deadline { - return Err(failure(RuntimeError::DeadlineExceeded)); - } - let timer = self - .domain - .sleep_until(self.deadline) - .map_err(|_| failure(RuntimeError::Closed))?; - let delivery = crate::asyncrt::select! { - result = self.receiver => result.map_err(|_| failure(RuntimeError::Closed))?, - _ = timer => return Err(failure(RuntimeError::DeadlineExceeded)), - }; - if self - .domain - .mono_ms() - .map_err(|_| failure(RuntimeError::Closed))? - >= self.deadline - { - return Err(failure(RuntimeError::DeadlineExceeded)); - } - delivery.result + self.inner.response().await } } -fn option_bytes(options: &mut CellSqlOptions) -> Result { +pub(super) fn option_bytes(options: &mut CellSqlOptions) -> Result { let (path, vfs) = match options { CellSqlOptions::Materialized(options) => (&mut options.directory, &mut options.sqlite_vfs), CellSqlOptions::Extents(options) => (&mut options.cache_directory, &mut options.sqlite_vfs), @@ -149,12 +118,12 @@ impl CellGroupController { let (reply, receiver) = oneshot::channel(); shared .requests - .try_send(Request { + .try_send(requests::Request::Command(Box::new(Request { invocation, options, reply, permit: Arc::new(permit), - }) + }))) .map_err(|e| { let (request, reason) = match e { mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), @@ -162,14 +131,20 @@ impl CellGroupController { }; CellCommandAdmissionError { error: error(command, false, reason), - invocation: Box::new(request.invocation), + invocation: match request { + requests::Request::Command(request) => Box::new(request.invocation), + _ => unreachable!("command admission envelope"), + }, } })?; Ok(CellCommandCall { - receiver, - command, - deadline, - domain: shared.domain.clone(), + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, }) } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs new file mode 100644 index 000000000..509af5910 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs @@ -0,0 +1,169 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded invocation ownership is independent of response-waiter lifetime. +use super::calls::Delivery; +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{ + CellDispatchError, CellEnrollmentError, CellEnrollmentInvocation, CellSqlEnrollment, + CellSqlOptions, +}; +use cellar_logic::consensus::ProposalId; +use tokio::sync::{mpsc, oneshot, watch}; + +pub(super) struct Request { + invocation: CellEnrollmentInvocation, + options: CellSqlOptions, + reply: oneshot::Sender>, + permit: Arc, +} + +pub(super) struct Completed { + pub invocation: CellEnrollmentInvocation, + pub error: Option>, + _permit: Arc, +} + +/// Dropping or timing out this response never cancels the admitted enrollment. +/// An unpolled reply retains node admission, including error-only replies. +pub struct CellEnrollmentCall { + inner: calls::Call, +} + +/// Admission did not transfer ownership. In particular, a rejected non-fresh +/// invocation is returned with its original recovery state intact. +pub struct CellEnrollmentAdmissionError { + pub error: CellEnrollmentError, + pub invocation: Box, +} + +impl std::fmt::Debug for CellEnrollmentAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellEnrollmentAdmissionError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellEnrollmentAdmissionError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellEnrollmentAdmissionError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(&self.error) + } +} + +fn error(command: ProposalId, ambiguous: bool, reason: RuntimeError) -> CellEnrollmentError { + CellEnrollmentError { + creation: command, + ambiguous, + cause: Box::new(CellDispatchError::Runtime(reason)), + } +} + +impl CellEnrollmentCall { + pub async fn response(self) -> Result> { + self.inner.response().await + } +} + +impl CellGroupController { + /// Admit a fresh local invocation. Discovery and destination authority still + /// run inside the owner; catalog routing never grants leader authority. + pub fn enroll( + &self, + invocation: CellEnrollmentInvocation, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let admitted = (|| { + let (command, deadline) = invocation.controller_admission(&shared.domain)?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let bytes = 8192_usize + .checked_add( + commands::option_bytes(&mut options).map_err(|e| error(command, false, e))?, + ) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + Ok((command, deadline, permit)) + })(); + let (command, deadline, permit) = match admitted { + Ok(admitted) => admitted, + Err(error) => { + return Err(CellEnrollmentAdmissionError { + error, + invocation: Box::new(invocation), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Enrollment(Box::new(Request { + invocation, + options, + reply, + permit: Arc::new(permit), + }))) + .map_err(|e| { + let (request, reason) = match e { + mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), + mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), + }; + CellEnrollmentAdmissionError { + error: error(command, false, reason), + invocation: match request { + requests::Request::Enrollment(request) => Box::new(request.invocation), + _ => unreachable!("enrollment admission envelope"), + }, + } + })?; + Ok(CellEnrollmentCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + mut progress: watch::Receiver<()>, +) -> Completed { + let Request { + mut invocation, + options, + reply, + permit, + } = request; + let result = invocation + .run_owned(&runtime, &sql, options, &shared.groups, &mut progress) + .await + .map_err(Arc::new); + let error = result.as_ref().err().cloned(); + let _ = reply.send(Delivery { + result, + _permit: permit.clone(), + }); + Completed { + invocation, + error, + _permit: permit, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index d6107674b..0cd4f13fc 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -3,7 +3,11 @@ //! Owns a fixed roster's consensus events, transport dispatch and physical apply. //! Admitted command ownership survives response cancellation and expiry. +mod calls; mod commands; +mod enrollments; +mod requests; +pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; mod worker; pub use commands::{CellCommandAdmissionError, CellCommandCall}; @@ -27,7 +31,8 @@ pub struct CellGroupControllerLimits { pub max_parallel_apply: usize, /// One node timer backs off transient SQL admission, never one timer per cell. pub retry_interval_ms: u64, - /// Includes queued, resolving and unconsumed command responses. + /// Shared by commands and enrollments, including queued work, resolution + /// and unconsumed responses. pub max_commands: usize, /// Invocation/input/options retention. SQL capture, apply and result bytes /// retain the SQL pool's independent admission as well. @@ -41,6 +46,7 @@ pub enum CellGroupControllerError { Apply(Arc), Transport(TransportError), Command(Arc), + Enrollment(Arc), Closed, } @@ -77,7 +83,7 @@ struct Shared { limits: CellGroupControllerLimits, domain: crate::asyncrt::DomainToken, commands: Arc, - requests: tokio::sync::mpsc::Sender, + requests: tokio::sync::mpsc::Sender, active_apply: AtomicUsize, pending_apply: AtomicUsize, applied: AtomicU64, diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs new file mode 100644 index 000000000..e6fa4ceae --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -0,0 +1,49 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! All mutations share the same node queue, admission and physical shutdown owner. +use super::*; + +pub(super) enum Request { + Command(Box), + Enrollment(Box), +} + +pub(super) enum Completed { + Command(Box), + Enrollment(Box), +} + +impl Completed { + pub fn failure(&self) -> Option { + match self { + Self::Command(done) if done.invocation.unresolved() => Some(done.error.clone().map_or( + CellGroupControllerError::Invalid("unresolved successful command"), + CellGroupControllerError::Command, + )), + Self::Enrollment(done) if done.invocation.unresolved() => { + Some(done.error.clone().map_or( + CellGroupControllerError::Invalid("unresolved successful enrollment"), + CellGroupControllerError::Enrollment, + )) + } + _ => None, + } + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, + progress: tokio::sync::watch::Receiver<()>, +) -> Completed { + match request { + Request::Command(request) => Completed::Command(Box::new( + commands::drive(*request, runtime, sql, shared, progress).await, + )), + Request::Enrollment(request) => Completed::Enrollment(Box::new( + enrollments::drive(*request, runtime, sql, shared, progress).await, + )), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 989456802..23d6fdf14 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -31,9 +31,9 @@ pub(super) struct Worker { active: BTreeSet, jobs: FuturesUnordered>, hints: FuturesUnordered>, - requests: tokio::sync::mpsc::Receiver, - commands: FuturesUnordered>, - failed_command: Option, + requests: tokio::sync::mpsc::Receiver, + commands: FuturesUnordered>, + failed_command: Option, progress: tokio::sync::watch::Sender<()>, } @@ -44,7 +44,7 @@ impl Worker { transport: RaftTransport, sql: CellGroupSqlPool, roster: BTreeMap, - requests: tokio::sync::mpsc::Receiver, + requests: tokio::sync::mpsc::Receiver, shared: Arc, ) -> Self { Self { @@ -125,11 +125,10 @@ impl Worker { .saturating_add(self.limits.retry_interval_ms); }, Some(request) = self.requests.recv() => { - self.commands.push(Box::pin(commands::drive(request, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))); + self.commands.push(Box::pin(requests::drive(request, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))); } Some(completed) = self.commands.next(), if !self.commands.is_empty() => { - if completed.invocation.unresolved() { - let error = completed.error.clone().map_or(CellGroupControllerError::Invalid("unresolved successful command"), CellGroupControllerError::Command); + if let Some(error) = completed.failure() { // Retain the invocation and its permit through physical // shutdown even after delivering the error to its caller. self.failed_command = Some(completed); diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs new file mode 100644 index 000000000..b6708e028 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -0,0 +1,431 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Routed creation identity and controller-owned initialization, never a cell task. +use super::*; +use crate::raft::execution::CellEnrollmentAttempt; +use crate::storage::CommandInputs; +use cellar_logic::consensus::{CommandFailure, Durability}; +use cellar_ltx::generation::ImageId; +use std::collections::BTreeSet; +use tokio::sync::watch; + +#[derive(Clone, Copy)] +pub struct CellEnrollmentRequest { + pub creation: ProposalId, + pub durability: Durability, +} + +#[derive(Debug)] +pub struct CellEnrollmentError { + pub creation: ProposalId, + pub ambiguous: bool, + pub cause: Box, +} + +impl fmt::Display for CellEnrollmentError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "cell enrollment: {} (ambiguous={})", + self.cause, self.ambiguous + ) + } +} + +impl std::error::Error for CellEnrollmentError {} + +struct Origin { + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + route: GroupRoute, + prepared: Option, +} + +/// Captured once before discovery and transferred into the node controller. +/// The registered class initializer is shared; no per-request closure is retained. +pub struct CellEnrollmentInvocation { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + creation: ProposalId, + preview: ImageId, + inputs: CommandInputs, + session: CellRoutingSession, + started: bool, + ambiguous: bool, + origin: Option, + _admission: Permit, +} + +impl CellGroupDispatcher { + pub async fn enroll( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + let error = |cause| CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(cause), + }; + let started = self.domain.mono_ms().map_err(|_| error(closed()))?; + if request.durability != Durability::Quorum + || self.directory.namespace_config().minimum_durability != Durability::Quorum + { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } + self.class + .require_initializer() + .map_err(|e| error(CellDispatchError::Registration(e)))?; + let admission = self + .budget + .acquire(8192) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + let (inputs, preview) = self + .domain + .run(async { + use rand::RngCore as _; + let mut rng = crate::asyncrt::rng("cell-group-enrollment-inputs"); + let mut random = [0; 32]; + let mut preview = [0; 16]; + rng.fill_bytes(&mut random); + rng.fill_bytes(&mut preview); + ( + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + }, + ImageId::new(preview), + ) + }) + .await + .map_err(|_| error(closed()))?; + let preview = + preview.map_err(|_| error(CellDispatchError::Invalid("enrollment preview")))?; + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-enrollment-invocation-1\0"); + hash.update(cell_image_history_digest(&cell)); + hash.update(request.creation.bytes()); + hash.update(self.class.schema_digest()); + hash.update(inputs.now_unix_millis.to_be_bytes()); + hash.update(inputs.random); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + hash.update([limits.max_attempts, 1]); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Enrollment(request.creation), + fingerprint: hash.finalize().into(), + }, + limits, + cached, + started, + ) + .map_err(|e| error(CellDispatchError::Routing(e)))?; + Ok(CellEnrollmentInvocation { + dispatcher: self.clone(), + cell, + creation: request.creation, + preview, + inputs, + session, + started: false, + ambiguous: false, + origin: None, + _admission: admission, + }) + } +} + +impl CellEnrollmentInvocation { + fn error(&self, cause: CellDispatchError) -> CellEnrollmentError { + CellEnrollmentError { + creation: self.creation, + ambiguous: self.ambiguous, + cause: Box::new(cause), + } + } + + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(ProposalId, u64), CellEnrollmentError> { + if self.started || self.origin.is_some() || !self.dispatcher.domain.same_owner(domain) { + return Err(self.error(CellDispatchError::Invalid( + "controller requires fresh local enrollment", + ))); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| self.error(closed()))? >= deadline { + return Err(self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))); + } + Ok((self.creation, deadline)) + } + + pub(crate) fn unresolved(&self) -> bool { + self.origin.is_some() + } + + pub(crate) async fn run_owned( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &BTreeSet, + progress: &mut watch::Receiver<()>, + ) -> Result { + self.started = true; + self.run_at(runtime, sql, options, roster, progress) + .await + .map_err(|e| self.error(e)) + } + + async fn run_at( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + roster: &BTreeSet, + progress: &mut watch::Receiver<()>, + ) -> Result { + let attempt = self + .session + .begin() + .await + .map_err(CellDispatchError::Routing)?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if !roster.contains(&attempt.route().history) { + return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); + } + let authority = runtime + .enrollment_authority(attempt.route().history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.state().route() != attempt.route() { + return Err(CellDispatchError::Invalid("enrollment destination route")); + } + self.session + .mark_submitted(&attempt) + .map_err(CellDispatchError::Routing)?; + self.ambiguous = true; + self.origin = Some(Origin { + runtime: runtime.clone(), + sql: sql.clone(), + route: attempt.route().clone(), + prepared: None, + }); + let class = self.dispatcher.class.clone(); + let cell = self.cell.clone(); + let staged = sql + .initialize_owned( + options, + authority, + CellEnrollmentAttempt { + creation: self.creation, + preview: self.preview, + inputs: self.inputs, + }, + move |tx| class.invoke_initializer(&cell, tx), + ) + .await; + match staged { + Ok(CellSqlOutcome::Enrollment(enrolled)) => return self.committed(*enrolled), + Ok(CellSqlOutcome::Prepared(prepared)) => self.origin_mut()?.prepared = Some(prepared), + Ok(_) => { + return Err(CellDispatchError::Invalid( + "enrollment preparation response", + )) + } + Err(failure) => { + if !failure.unresolved { + self.origin = None; + } + return Err(CellDispatchError::Execution(failure.error)); + } + } + loop { + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() + { + let prepared = self.prepared()?; + if let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + break; + } + let prepared = self.prepared()?; + if let Some(proof) = prepared.rejection().map_err(CellDispatchError::Runtime)? { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))); + } + let origin = self.origin_ref()?; + let submitted = async { + origin + .sql + .mark_submitted(self.cell.clone(), self.creation) + .await + .map_err(CellDispatchError::Execution)?; + prepared + .submit(&origin.runtime) + .await + .map_err(CellDispatchError::Runtime) + } + .await; + match submitted { + Ok(CellSqlSubmission::Accepted(_)) => break, + Ok(CellSqlSubmission::Rejected(proof)) => { + self.reject(proof, progress).await?; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded, + ))); + } + Err(CellDispatchError::Runtime( + RuntimeError::NotLeader | RuntimeError::StaleCompletion, + )) => break, + Err(error) if transient(&error) => { + progress.changed().await.map_err(|_| closed())? + } + Err(error) => { + if let Some(proof) = prepared + .cancel_before_submission() + .map_err(CellDispatchError::Runtime)? + { + self.reject(proof, progress).await?; + } + return Err(error); + } + } + } + loop { + let origin = self.origin_ref()?; + let resolved = async { + let state = origin + .runtime + .enrollment_state(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if state.route() != &origin.route { + return Err(CellDispatchError::Invalid("enrollment resolution route")); + } + origin + .sql + .resolve_enrollment(self.creation, state) + .await + .map_err(CellDispatchError::Execution) + } + .await; + match resolved { + Ok(CellSqlEnrollmentResolution::Committed(enrolled)) => { + return self.committed(*enrolled) + } + Ok(result) => { + self.origin = None; + return Err(CellDispatchError::Execution(ExecutionError::Refused( + match result { + CellSqlEnrollmentResolution::Conflict => { + CommandFailure::CommandConflict + } + _ => CommandFailure::Unavailable, + }, + ))); + } + Err(error) + if transient(&error) + || matches!( + error, + CellDispatchError::Execution(ExecutionError::Uncertain) + ) => + { + () + } + Err(error) => return Err(error), + } + progress.changed().await.map_err(|_| closed())?; + } + } + + fn origin_ref(&self) -> Result<&Origin, CellDispatchError> { + self.origin + .as_ref() + .ok_or(CellDispatchError::Invalid("missing enrollment origin")) + } + + fn origin_mut(&mut self) -> Result<&mut Origin, CellDispatchError> { + self.origin + .as_mut() + .ok_or(CellDispatchError::Invalid("missing enrollment origin")) + } + + fn prepared(&self) -> Result { + self.origin_ref()? + .prepared + .clone() + .ok_or(CellDispatchError::Invalid("missing enrollment preparation")) + } + + fn committed( + &mut self, + enrolled: CellSqlEnrollment, + ) -> Result { + let record = enrolled.record(); + if record.cell() != &self.cell + || record.creation() != self.creation + || record.schema() != self.dispatcher.class.schema_digest() + { + return Err(CellDispatchError::Invalid("enrollment receipt binding")); + } + self.origin = None; + Ok(enrolled) + } + + async fn reject( + &mut self, + proof: CellSqlRejection, + progress: &mut watch::Receiver<()>, + ) -> Result<(), CellDispatchError> { + loop { + match self + .origin_ref()? + .sql + .reject(self.cell.clone(), proof.clone()) + .await + { + Ok(()) | Err(ExecutionError::Uncertain) => { + self.origin = None; + return Ok(()); + } + Err(ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded)) => { + progress.changed().await.map_err(|_| closed())?; + } + Err(error) => return Err(CellDispatchError::Execution(error)), + } + } + } +} + +fn transient(error: &CellDispatchError) -> bool { + matches!( + error, + CellDispatchError::Runtime( + RuntimeError::NotReady + | RuntimeError::Overloaded + | RuntimeError::NotLeader + | RuntimeError::StaleCompletion + ) | CellDispatchError::Execution( + ExecutionError::Busy | ExecutionError::Refused(CommandFailure::Overloaded) + ) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 283d8b36b..ecc2bfa5d 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -3,6 +3,9 @@ //! Catalog-bound invocations over the existing consensus and SQL owners. //! Routing observations never replace destination authority or ingress grants. +mod enrollment; +pub use enrollment::{CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest}; + mod command; pub use command::{ CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 517c5ba4f..0fd3ca577 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -12,16 +12,18 @@ pub use super::host_sql::cell_groups::{ mod apply; mod controller; pub use controller::{ - CellCommandAdmissionError, CellCommandCall, CellGroupController, CellGroupControllerError, - CellGroupControllerLimits, CellGroupControllerUsage, + CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, + CellGroupController, CellGroupControllerError, CellGroupControllerLimits, + CellGroupControllerUsage, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; pub use dispatch::{ CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, CellCommandResponse, CellCommandSettlement, CellCommandSubmission, CellDispatchError, - CellDispatchLimits, CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, - CellQueryRequest, CellQueryResponse, + CellDispatchLimits, CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest, + CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, CellQueryRequest, + CellQueryResponse, }; mod execution_authority; diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index d379fad36..815df9db1 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -248,7 +248,7 @@ impl CellSqlReceipt { } // A stage error after a received worker reply cannot hide an installed producer: -// Stage inserts pending ownership only on its successful Prepared branch. +// Stage and Initialize insert pending ownership only on successful Prepared branches. // A lost reply is deliberately different and requires recovery/joined shutdown. pub(crate) struct CellSqlStageFailure { pub error: ExecutionError, @@ -629,6 +629,39 @@ impl CellGroupSqlPool { attempt: CellEnrollmentAttempt, initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, ) -> Result { + self.initialize_owned(options, authority, attempt, initialize) + .await + .map_err(|failure| failure.error) + } + + pub(crate) async fn initialize_owned( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result { + let (cell, action) = self + .initialize_request(options, authority, attempt, initialize) + .map_err(CellSqlStageFailure::known)?; + let wait = self + .send(cell, action) + .map_err(CellSqlStageFailure::known)?; + wait.await + .map_err(|_| CellSqlStageFailure { + error: unavailable(), + unresolved: true, + })? + .map_err(CellSqlStageFailure::known) + } + + fn initialize_request( + &self, + options: CellSqlOptions, + authority: CellEnrollmentAuthority, + attempt: CellEnrollmentAttempt, + initialize: impl FnOnce(&CommandTransaction<'_>) -> anyhow::Result<()> + Send + 'static, + ) -> Result<(CellHistory, worker::Action), ExecutionError> { self.admit()?; if !authority .state() @@ -659,7 +692,7 @@ impl CellGroupSqlPool { self.inner .admission .reserve(&cell, authority.state().route().history, bytes)?; - self.call( + Ok(( cell, worker::Action::Initialize(Box::new(worker::Initialize { options, @@ -669,8 +702,7 @@ impl CellGroupSqlPool { lease, bytes, })), - ) - .await + )) } pub async fn resolve_enrollment( diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 7de193745..bd2e76811 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1420,3 +1420,43 @@ async fn catalog_commands_replicate_over_physical_tls() -> Result<()> { async fn rooted_catalog_commands_replicate_over_physical_tls() -> Result<()> { Box::pin(runtime_fixture::commands_over_tcp(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_survives_initializer_disconnect() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_seals_late_preparation() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_survives_accepted_disconnect() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(1, 0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_resolves_after_deadline() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(1, 1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_joins_cancelled_shutdown() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment(0, 2)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_releases_completed_initializer_errors() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment_failure(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_joins_lost_worker_reply() -> Result<()> { + Box::pin(runtime_fixture::owned_enrollment_failure(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_enrollment_requires_registered_initializer() -> Result<()> { + Box::pin(runtime_fixture::enrollment_requires_registration()).await +} diff --git a/crates/cellar/tests/support/cell_group_controller_enrollment.rs b/crates/cellar/tests/support/cell_group_controller_enrollment.rs new file mode 100644 index 000000000..834769ee3 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_enrollment.rs @@ -0,0 +1,470 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered creation on the shared controller, SQL pool and physical TLS peers. +use super::apply_faults::physical; +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +async fn enroll( + dispatcher: &CellGroupDispatcher, + cell: &CellHistory, + tag: u8, + timeout: Duration, +) -> Result { + Ok(dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([tag; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?) +} + +fn dispatcher(fixture: &RuntimeCluster, class: Arc) -> Result { + Ok(CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class, + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +// phase 0 holds initialization; phase 1 holds committed manifest publication. +// mode 0 disconnects; 1 expires the response; 2 cancels physical shutdown. +pub(crate) async fn owned_enrollment(phase: u8, mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let seen = Arc::new(Mutex::new(Vec::new())); + let observed = seen.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let init_entered = entered.clone(); + let init_held = held.clone(); + let base = || RaftClass::new("Counter", 1, [81; 32]); + let missing = base()?; + let class = Arc::new( + base()? + .initializer([90; 32], move |cell, tx| { + let first = count.fetch_add(1, Ordering::SeqCst) == 0; + observed.lock().unwrap().push((cell.clone(), tx.inputs())); + if first && phase == 0 { + let _ = init_entered + .lock() + .unwrap() + .take() + .context("one initializer")? + .send(()); + init_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"updated".to_vec()) + })?, + ); + assert_ne!(missing.schema_digest(), class.schema_digest()); + assert!(base()?.initializer([0; 32], |_, _| Ok(())).is_err()); + assert!(base()? + .initializer([90; 32], |_, _| Ok(()))? + .initializer([90; 32], |_, _| Ok(())) + .is_err()); + let different = + base()? + .initializer([91; 32], |_, _| Ok(()))? + .command("Add", [82; 32], |_, _| Ok(vec![]))?; + assert_ne!(different.schema_digest(), class.schema_digest()); + let (mut fixture, route, existing, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_commands: 1, + ..controller::limits() + }, + ) + .await?; + let dispatcher = dispatcher(&fixture, class.clone())?; + let cell = history(&route, "new-enrollment")?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let rejected = dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([170; 16])?, + durability: Durability::S3, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .err() + .context("S3 refusal")?; + assert!(!rejected.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 0); + if phase == 1 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply entered") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("apply released"); + })); + } + let timeout = if mode == 1 { + Duration::from_millis(400) + } else { + Duration::from_secs(15) + }; + let call = controllers[0].enroll( + enroll(&dispatcher, &cell, 170, timeout).await?, + fixture.cell_options(&cell).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(seen.lock().unwrap()[0].0, cell); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(dispatcher.usage().1, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + // Enrollment consumes the same controller budget as ordinary commands. + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let command = dispatcher + .command( + existing.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([171; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let refused = controllers[0] + .command(command, fixture.cell_options(&existing).await?) + .err() + .context("shared admission")?; + assert!(matches!( + refused.error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + if mode == 1 { + let error = call.response().await.err().context("response deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + if mode == 2 { + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(apply::pool(&fixture, 0).is_err()); + } + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + if mode == 2 { + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + } else { + wait_for(async || { + Ok( + controllers[0].usage().retained_commands == 0 + && pool.usage()?.active_cells == 0, + ) + }) + .await?; + let state = fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await?; + if phase == 0 && mode == 1 { + assert!(state.record().is_none()); + } else { + assert_eq!( + state.record().context("committed creation")?.creation(), + ProposalId::new([170; 16])? + ); + } + let original = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 170, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await?; + let record = original.record().clone(); + drop(original); + assert_eq!( + calls.load(Ordering::SeqCst), + if phase == 0 && mode == 1 { 2 } else { 1 } + ); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let command = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([171; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let updated = controllers[0] + .command(command, fixture.cell_options(&cell).await?)? + .response() + .await?; + assert_eq!(updated.receipt().version.sequence.get(), 2); + drop(updated); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let duplicate = controllers[0].enroll( + enroll(&dispatcher, &cell, 170, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )?; + wait_for(async || Ok(pool.usage()?.active_cells == 0 && dispatcher.usage().1 == 0)) + .await?; + assert_eq!(controllers[0].usage().retained_commands, 1); // Unconsumed creation response remains charged. + let duplicate = duplicate.response().await?; + assert_eq!(duplicate.record(), &record); + drop(duplicate); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + for incarnation in [false, true] { + let mut conflict = cell.clone(); + if incarnation { + conflict.incarnation = IncarnationId::new([172; 16])?; + } + let rejected = controllers[0] + .enroll( + enroll(&dispatcher, &conflict, 173, Duration::from_secs(15)).await?, + fixture.cell_options(&conflict).await?, + )? + .response() + .await + .err() + .context("creation conflict")?; + assert!(matches!( + rejected.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + )) + )); + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + } + assert_eq!( + calls.load(Ordering::SeqCst), + if phase == 0 && mode == 1 { 2 } else { 1 } + ); + assert!(!controllers[0].usage().closing); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn owned_enrollment_failure(lost_reply: bool) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new( + root.path() + .join(if rooted { "rooted-0" } else { "node-1/images" }), + None, + ); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.initializer( + [90; 32], + move |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + if count.fetch_add(1, Ordering::SeqCst) == 0 { + if lost_reply { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(|| { + panic!("injected initialization capture reply loss") + })); + } else { + anyhow::bail!("injected initializer failure"); + } + } + Ok(()) + }, + )?); + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatcher = dispatcher(&fixture, class)?; + let cell = history(&route, "failed-enrollment")?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let error = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 174, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await + .err() + .context("failed initializer")?; + assert!(error.ambiguous); + assert_eq!(calls.load(Ordering::SeqCst), 1); + if lost_reply { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Unavailable)) + )); + assert!(matches!( + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await?, + Err(CellGroupControllerError::Enrollment(_)) + )); + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(dispatcher.usage().1, 0); + assert!(pool.close().await.is_err()); + fixture.appliers.remove(&(0, route.history)); + } else { + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + wait_for(async || { + Ok(controllers[0].usage().retained_commands == 0 && dispatcher.usage().1 == 0) + }) + .await?; + assert!(!controllers[0].usage().closing); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await? + .record() + .is_none()); + let result = controllers[0] + .enroll( + enroll(&dispatcher, &cell, 174, Duration::from_secs(15)).await?, + fixture.cell_options(&cell).await?, + )? + .response() + .await?; + assert_eq!(result.record().creation(), ProposalId::new([174; 16])?); + drop(result); + assert_eq!(calls.load(Ordering::SeqCst), 2); + controllers[0].close().await?; + } + for controller in &controllers[1..] { + controller.close().await?; + } + drop(controllers); + drop(pool); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn enrollment_requires_registration() -> Result<()> { + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?); + let (fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let dispatcher = dispatcher(&fixture, class)?; + let error = dispatcher + .enroll( + history(&route, "unregistered")?, + CellEnrollmentRequest { + creation: ProposalId::new([175; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .err() + .context("initializer required")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Registration(RegistrationError::Invalid("missing initializer")) + )); + assert_eq!(dispatcher.usage().1, 0); + drop(dispatcher); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 0510999e0..2704a6968 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -701,3 +701,7 @@ pub(super) use sql::routed_commands; pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { Box::pin(sql::commands_over_tcp(rooted)).await } + +pub(super) use sql::{ + enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, +}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 6e7974a8d..5709b9dc3 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -538,3 +538,9 @@ pub(crate) use controller_commands::{owned_command_failure, owned_commands}; pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { Box::pin(command_dispatch::commands_with_transport(rooted, true)).await } + +#[path = "cell_group_controller_enrollment.rs"] +mod controller_enrollment; +pub(crate) use controller_enrollment::{ + enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, +}; diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 274646fb1..2f35e67eb 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 47 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 55 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 00919f3fc..dc7afcdb7 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -40,6 +40,10 @@ confuse a successful earlier creation with a conflict. The initial commitment is provenance, not a requirement to retain the superseded initial image forever. Snapshots and archives carry the record with the selected current cell state. Initialization runs on the bounded node SQL workers under actual-leader authority. +The compiled initializer has an explicit contract digest in the class voting +schema. It receives the logical cell history and captured host inputs through its +private transaction. A host that cannot execute that contract must refuse the +registration instead of silently omitting initialization. Reserve the logical cluster/namespace/cell name before entering the callback; competing incarnations must not obtain independent producer leases. Capture initial SQLite/LTX bytes once, carry the preparation term with the output, and check that diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index b66ca569e..b0c0b52d2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -497,7 +497,13 @@ "crates/cellar/durability/raft/cell_groups/controller/commands.rs", "crates/cellar/durability/raft/execution/submission_tests.rs", "crates/cellar/tests/support/cell_group_controller.rs", - "crates/cellar/tests/support/cell_group_controller_commands.rs" + "crates/cellar/tests/support/cell_group_controller_commands.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs", + "crates/cellar/durability/raft/cell_groups/controller/enrollments.rs", + "crates/cellar/durability/raft/cell_groups/controller/calls.rs", + "crates/cellar/durability/raft/cell_groups/controller/requests.rs", + "crates/cellar/application/consensus/class_initializer_tests.rs", + "crates/cellar/tests/support/cell_group_controller_enrollment.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -607,6 +613,19 @@ "controller_owned_commands_release_completed_callback_errors", "controller_owned_commands_join_a_lost_sql_reply" ] + }, + { + "action": "Register initialization with an explicit digest in the class voting schema. Route immutable creation IDs and captured host inputs through the existing catalog session into the node controller. Share command/enrollment queues and admission, preserve original response deadlines, seal only never-attempted preparations and resolve the original immutable creation record after commitment or later mutations. Distinguish completed initialization errors from lost replies, join physical shutdown, and refuse unsupported S3 durability. The old public host must refuse registered initialization until its shared-runtime cutover; do not silently ignore the callback.", + "tests": [ + "controller_enrollment_survives_initializer_disconnect", + "controller_enrollment_seals_late_preparation", + "controller_enrollment_survives_accepted_disconnect", + "controller_enrollment_resolves_after_deadline", + "controller_enrollment_joins_cancelled_shutdown", + "controller_enrollment_releases_completed_initializer_errors", + "controller_enrollment_joins_lost_worker_reply", + "controller_enrollment_requires_registered_initializer" + ] } ], "test_cases": [ @@ -1055,6 +1074,86 @@ "I03", "I12" ] + }, + { + "name": "controller_enrollment_survives_initializer_disconnect", + "oracle": "Drop the response while the registered initializer is held on a real SQL worker. Retain shared command/enrollment admission, settle the original creation record after release, and return it unchanged after a later mutation. Conflicting creation IDs/incarnations refuse without callback replay in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_seals_late_preparation", + "oracle": "Expire the response while initialization is held. Seal the eventual never-attempted preparation, join private cleanup and leave the cell absent. An explicit creation retry may initialize once; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_survives_accepted_disconnect", + "oracle": "Disconnect after commitment while manifest publication is held. Retain creation ownership until exact publication, then return the same creation record without reinitialization, including after mutation; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_resolves_after_deadline", + "oracle": "Expire the response after commitment while manifest sync is held. Continue bounded creation resolution independently of the waiter, and preserve the original creation record on retry in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_joins_cancelled_shutdown", + "oracle": "Cancel controller close during a held initializer. Preserve SQL admission and engine claims until physical worker joins, then release all invocation/command retention in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_releases_completed_initializer_errors", + "oracle": "A completed initializer error rolls back private schema changes and releases invocation ownership without closing the node. Explicit same-creation retry commits once in both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_joins_lost_worker_reply", + "oracle": "A physical capture panic loses the initialization reply. Keep ambiguous creation ownership through failed-controller cleanup and physical worker joins, releasing admission only afterward; both image modes.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] + }, + { + "name": "controller_enrollment_requires_registered_initializer", + "oracle": "Refuse enrollment before routing/SQL when the namespace-bound class has no registered initializer, and release invocation admission.", + "invariants": [ + "I01", + "I02", + "I03", + "I12" + ] } ], "invariants": [ From f77d0d939347ead7129b39e72f76fcff11b99635 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 06:15:42 -0700 Subject: [PATCH 093/568] style: remove redundant enrollment retry expression --- .../durability/raft/cell_groups/dispatch/enrollment.rs | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index b6708e028..40c51155b 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -347,10 +347,7 @@ impl CellEnrollmentInvocation { || matches!( error, CellDispatchError::Execution(ExecutionError::Uncertain) - ) => - { - () - } + ) => {} Err(error) => return Err(error), } progress.changed().await.map_err(|_| closed())?; From 0ab9ceebb0150921feaed28c4ec44a50439eacf4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 06:24:46 -0700 Subject: [PATCH 094/568] docs: record qualified shared enrollment checkpoint --- docs/design/multi-raft/cell-groups/README.md | 19 +++--- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 46 +++++++++---- .../design/multi-raft/cell-groups/progress.md | 64 +++++++++++++++++++ 4 files changed, 108 insertions(+), 25 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 42683ef5c..2f180e6ac 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -173,15 +173,16 @@ production controller ownership and reconciliation after caller disconnect remai required, followed by routed enrollment and SDK/shared-query RPC integration. Those fixtures still drive runtime events. See [progress](progress.md) for exact limits. -At `8c615ac9`, the shared controller owns fresh commands independently of -response disconnect/deadline. Exact receipt resolution, never-attempted preparation -sealing, bounded retained responses and joined physical shutdown pass seven new -native cases in both image modes plus three submission unit cases. The partial -gate passes 83 Rust cases and nine commands; transfer -`.cellar/cell-groups-evidence/CG-04-command-owner/` (248 source inputs, three -executables) with prior evidence. The next critical delivery is routed enrollment, -then remote shared command/query dispatch and public SDK integration. Fixed roster -and node-wide failure handling also remain gaps. See [progress](progress.md). +At `f77d0d93`, registered initialization and catalog-routed local enrollment +run on the shared controller. Commands and creations share bounded admission and +response-independent ownership. Exact creation records survive later mutations; +conflicts do not replay initialization. The partial gate passes 90 Rust cases +and ten commands; transfer +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/` (254 source inputs, +four executables) with prior evidence. Next connect public handles and remote +shared dispatch, including safe receipt routing. Lazy birth/dynamic roster, +per-group failure isolation and configuration/cache lifecycle remain gaps. +See [progress](progress.md); CG-04 is not complete. The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index f9a35af80..c035e2aaf 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "8c615ac91bc3c13e6c25da4f4ab584a73e609207", + "source_revision": "f77d0d939347ead7129b39e72f76fcff11b99635", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index cf295de84..a1be9db9d 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,21 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `8c615ac9`, adding bounded command +The newest reviewed source baseline is `f77d0d93`, adding registered, +catalog-routed local enrollment to the shared controller. Initializer digests bind +voting schema; old-host validation refuses this unsupported registration. Creation +identity and captured inputs remain owned independently from response lifetime. +Commands and enrollments share admission, response deadlines and physical shutdown. +Exact enrollment records survive subsequent mutations; conflicts refuse without +reinitialization. The partial record is +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/checkpoint.json`: 90 +selected Rust cases, ten commands, 254 source inputs and four executables. +Eight new native cases cover registration and lifecycle/fault boundaries, with +seven running both image modes. Transfer this directory with prior evidence. +CG-04 remains IN_PROGRESS; public SDK/remote RPC integration, lazy birth/dynamic +roster, per-group failure isolation and configuration/cache lifecycle are pending. +The public host has not cut over; shared-group RustFS/load remains unqualified. +The preceding reviewed source baseline is `8c615ac9`, adding bounded command ownership to the shared node controller. Response disconnect/deadline cannot drop entered SQL or ambiguous submission ownership. Only never-attempted preparations can be sealed; exact receipts or newer applied-term proof resolve submitted work. @@ -404,19 +418,23 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Complete catalog-routed enrollment with a stable creation identity on the - shared controller and SQL pool. Reuse the existing enrollment authority, - original producer and applied-term reconciliation; preserve cancellation and - ambiguous-result ownership just qualified for commands. Then connect remote - shared command/query dispatch and the public `RaftCellHandle` binding. Fresh - local commands now belong to `CellGroupController`, independently of response - lifetime. Preserve their original deadline and bounded resolution admission; - do not add per-cell tasks or an expiry that drops uncertainty. Missing origin - producers on followers still need safe leader receipt lookup; missing local - state is never proof of non-commit. Requested and namespace S3 durability must - continue to refuse until CG-06 supplies real archive proofs. The controller - still has a fixed roster and node-wide fatal-error handling; implement dynamic - registration and per-group failure isolation before runtime completion. +1. Connect public `RaftCellHandle` operations and remote shared command/query/ + enrollment dispatch to the existing controller and SQL pool. Local registered + creation now uses catalog routing and immutable creation records, preserving + response cancellation/deadline ownership and exact retries after mutation. + Reuse those owners and captured identities; do not add per-cell workers or + replay uncertain callbacks. Confirm route changes through the catalog and + provide safe leader receipt lookup when a follower has no original producer. + Missing local state never proves non-commit. Enrollment still requires an + attached group in a fixed roster: integrate lazy birth and dynamic registration. + Reuse the separate query connection in `query_transport.rs`, replacing its + old cell-scoped route/result binding with physical group and logical cell + identity. Keep existing namespace query-origin grants read-only; mutation RPC + needs explicit write authorization and must not inherit those grants. + The old host refuses registered initializers until its actual replacement; + do not remove that refusal without a functioning shared serving path. Request + and namespace S3 minima must continue refusing until CG-06 provides real archive + proofs. Add per-group failure isolation before claiming runtime completion. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1b910807c..b7d816f4c 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,70 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 controller-owned enrollment + +Qualified source: `f77d0d939347ead7129b39e72f76fcff11b99635`. + +`RaftClass::initializer` registers one compiled initialization callback with an +explicit contract digest. That digest participates in the voting schema identity; +changing or omitting it cannot claim the same namespace schema. Declaration order +with commands does not change the resulting identity. Existing old-host class +validation refuses this registration rather than silently ignoring the callback. +Shared initialization receives the logical cell history and captured host inputs +through its private transaction on the existing bounded SQL workers. + +`CellGroupDispatcher::enroll` freezes creation identity, schema and host inputs +before catalog discovery. `CellGroupController::enroll` takes fresh local +invocations into the same queue and byte/item budget used by commands. Both use +one response-deadline implementation and one physical shutdown owner. A dropped +or expired response cannot discard entered initialization or ambiguous submission. +Unconsumed creation replies remain charged. The SQL pool distinguishes completed +initialization errors from lost worker replies, preserving unresolved ownership +through controller failure and joined cleanup. + +The driver seals only never-attempted late preparations. Submitted creations +resolve through their original immutable enrollment record or the existing newer +applied-term absence/conflict proof, without automatic initializer replay. A +retry after later cell mutations returns the original creation record and never +restores the initial image. A different creation ID or logical incarnation is a +conflict. Request/namespace S3 minima continue to refuse without archive proof. + +Eight new native cases cover registration refusal plus lifecycle/fault behavior; +seven run both materialized and extent modes over three in-process TLS endpoints. +They hold initialization or committed manifest sync, disconnect/expire responses, +verify command/enrollment shared admission, preserve exact creation identity after +mutation, refuse conflicts without callbacks, and cancel shutdown while physical +work is held. Completed initializer errors permit an explicit retry; a physical +capture panic exercises a lost worker reply and joined cleanup. A unit case checks +initializer digest compatibility, declaration order, invalid/duplicate registration +and old-host validation refusal. The existing command-manifest compatibility case +also passes. The initial lifecycle fixture incorrectly began a new cell's client +sequence at 2; production correctly refused it. The fixture now starts at 1. +A subsequent compile caught a missing Box constructor in the common completion +envelope; that was corrected before qualification. The `bd3069d7` gate passed all +90 Rust cases, then Clippy rejected a redundant unit expression. That expression +was removed and the complete gate repeated at this successor. Both gates and +diagnostic logs are retained. + +All 90 selected Rust cases pass: 73 shared execution, 15 catalog, one initializer +unit case and one existing host compatibility case. All ten gate commands pass, +including all-target Clippy for Cellar/cellar-logic/cellar-ltx with warnings denied, +formatting, layout, dependency isolation, docs, source preflight and 37 validator +cases. The partial record at +`.cellar/cell-groups-evidence/CG-04-enrollment-controller-qualified/checkpoint.json` binds +254 source inputs and four copied executables. Transfer its complete directory +with prior evidence. The gate does not qualify shared-group RustFS, separate-process +crashes, power loss or load behavior. + +CG-04 remains IN_PROGRESS. The critical gap is now integration of public handles +and remote command/query/enrollment dispatch with these shared owners, including +catalog-confirmed retries and safe follower receipt lookup. Enrollment currently +requires a group already attached to the fixed controller roster; lazy group birth +and dynamic registration remain required. Per-group failure isolation and +configuration/cache lifecycle are also incomplete. Complete-group snapshots, +archive/reclamation, effects, topology, RustFS/load and legacy removal remain later +gates. The public host has not cut over. + ## CG-04 controller-owned commands Qualified source: `8c615ac91bc3c13e6c25da4f4ab584a73e609207`. From 3c297a06c9863fafb91e2cd8f428fae26915f2b4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 06:51:01 -0700 Subject: [PATCH 095/568] feat: route shared cell queries over bounded TLS lanes --- .../raft/cell_groups/dispatch/mod.rs | 101 ++++ .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/query_cell_transport.rs | 475 ++++++++++++++++++ .../cellar/durability/raft/query_cell_wire.rs | 187 +++++++ .../cellar/durability/raft/query_identity.rs | 110 +++- .../cellar/durability/raft/query_transport.rs | 346 +++++++------ crates/cellar/durability/raft/query_wire.rs | 55 +- .../durability/raft/query_wire_tests.rs | 278 ++++++++++ crates/cellar/durability/raft/transport.rs | 10 + .../durability/raft/transport_shared.rs | 2 +- .../tests/native_cell_groups_execution.rs | 15 + .../support/cell_group_remote_queries.rs | 441 ++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + 14 files changed, 1845 insertions(+), 183 deletions(-) create mode 100644 crates/cellar/durability/raft/query_cell_transport.rs create mode 100644 crates/cellar/durability/raft/query_cell_wire.rs create mode 100644 crates/cellar/tests/support/cell_group_remote_queries.rs diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index ecc2bfa5d..39e1d849d 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -40,6 +40,7 @@ pub enum CellQueryConsistency { Linearizable, } +#[derive(Clone)] pub struct CellQueryRequest { pub invocation: ProposalId, pub contract: OperationContract, @@ -55,6 +56,7 @@ pub enum CellDispatchError { Runtime(RuntimeError), Execution(ExecutionError), Version(VersionError), + Transport(crate::raft::transport::TransportError), } impl fmt::Display for CellDispatchError { @@ -78,6 +80,20 @@ pub struct CellGroupDispatcher { } impl CellGroupDispatcher { + pub(crate) fn uses_domain(&self, domain: &DomainToken) -> bool { + self.domain.same_owner(domain) + } + + pub(crate) fn namespace( + &self, + ) -> ( + cellar_logic::consensus::ClusterId, + cellar_logic::consensus::NamespaceId, + ) { + let config = self.directory.namespace_config(); + (config.cluster, config.namespace) + } + pub fn new( domain: &ExecutionDomain, directory: RangeDirectory, @@ -273,6 +289,91 @@ impl CellQueryInvocation { } } + /// One remote attempt over the node's existing bounded peer query lane. + /// The response owns its transport reservation until the caller drops it. + pub async fn execute_remote( + &mut self, + transport: &crate::raft::transport::RaftTransport, + member: cellar_logic::consensus::MemberId, + ) -> Result { + let attempt = self + .attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing route"))?; + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed route attempt")); + } + self.dispatched = true; + if !transport.uses_query_domain(&self.dispatcher.domain) { + return Err(CellDispatchError::Invalid("foreign query domain")); + } + let now = self.dispatcher.domain.mono_ms().map_err(|_| closed())?; + let timeout_ms = attempt + .deadline_ms() + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))?; + let call = crate::raft::query_wire::cell::CellQueryCall { + request: self.request.clone(), + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + let timer = self + .dispatcher + .domain + .sleep_until(attempt.deadline_ms()) + .map_err(|_| closed())?; + let response = crate::asyncrt::select! { + response = transport.query_cell(attempt.route(), &attempt.request().cell, member, call, self.dispatcher.keys.clone(), attempt.deadline_ms()) => response.map_err(CellDispatchError::Transport)?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + // Authenticate again at the caller boundary, then enforce the requested + // minimum independently from the destination's execution path. + let observation = self + .dispatcher + .keys + .verify_cell( + response.token(), + &attempt.request().cell, + attempt.route().schema_digest, + self.now().await?, + ) + .map_err(CellDispatchError::Version)?; + if observation != *response.observation() { + return Err(CellDispatchError::Invalid("query observation")); + } + if let CellQueryConsistency::AtLeast(token) = &self.request.consistency { + let minimum = self + .dispatcher + .keys + .verify_cell( + token, + &attempt.request().cell, + attempt.route().schema_digest, + self.now().await?, + ) + .map_err(CellDispatchError::Version)?; + if !observation + .version + .covers(&minimum.version) + .map_err(|_| CellDispatchError::Version(VersionError::WrongHistory))? + { + return Err(CellDispatchError::Runtime(RuntimeError::NotReady)); + } + } + self.session + .validate_attempt(attempt) + .map_err(CellDispatchError::Routing)?; + Ok(response) + } + async fn execute_at( &self, attempt: &CellRouteAttempt, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 0fd3ca577..edcb5abdd 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -9,6 +9,8 @@ pub use super::host_sql::cell_groups::{ CellSqlUsage, }; +pub use crate::raft::transport::{CellQueryGateway, CellRemoteQueryResponse}; + mod apply; mod controller; pub use controller::{ diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs new file mode 100644 index 000000000..ecbd1e831 --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -0,0 +1,475 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared cell reads reuse the peer query lane; namespace grants remain read-only. +use super::*; +use crate::application::consensus::{CommandFailure, RaftCells}; +use crate::raft::cell_groups::{ + CellDispatchError, CellGroupDispatcher, CellGroupRuntime, CellGroupSqlPool, CellQueryResponse, + CellRoutingLimits, CellSqlOptions, +}; +use crate::raft::query_identity::QueryRoute; +use crate::raft::query_wire::{ + self as wire, + cell::{CellQueryCall, CellQueryValue}, + QueryCall, +}; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{MemberId, NodeIdentity}, +}; +use queries::{QueryConnection, SendQuery}; +use std::time::Duration; + +/// Host-owned registration. The transport retains only a weak reference, so it +/// cannot extend the host's lifetime or create a runtime/transport ownership cycle. +#[derive(Clone)] +pub struct CellQueryGateway { + inner: Arc, +} + +pub(super) struct Gateway { + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + dispatcher: CellGroupDispatcher, + options: CellSqlOptions, +} + +/// Retains the same node/peer query admission through consumption of the result. +pub struct CellRemoteQueryResponse { + value: CellQueryValue, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteQueryResponse { + pub fn result(&self) -> &[u8] { + &self.value.result + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.served_by + } +} + +pub(super) struct SendCellQuery { + route: Arc>, + peer: RaftPeer, + cell: CellHistory, + call: CellQueryCall, + keys: VersionKeys, + sequence: u64, + deadline: u64, + cancel: watch::Receiver, + reply: oneshot::Sender>, + node: Permit, + peer_bytes: Permit, +} + +struct Cancel(watch::Sender); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl RaftTransport { + /// Register one namespace's compiled reads on this shared runtime and SQL + /// pool. Options provide the namespace's immutable-query services and cache; + /// cell images are selected only by fresh runtime authority. This grants no + /// origin permission. Retain the returned owner while serving requests. + pub fn install_cell_query_gateway( + &self, + dispatcher: CellGroupDispatcher, + sql: CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !dispatcher.uses_domain(&state.domain) + || !sql.uses_domain(&state.domain) + || !sql.belongs_to_engine(&runtime.log().owner()) + { + return Err(TransportError::Identity); + } + let key = dispatcher.namespace(); + let mut gateways = state + .cell_query_gateways + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + gateways.retain(|_, value| value.strong_count() > 0); + if gateways.contains_key(&key) { + return Err(TransportError::Refused); + } + if gateways.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let inner = Arc::new(Gateway { + runtime: runtime.clone(), + sql, + dispatcher, + options, + }); + gateways.insert(key, Arc::downgrade(&inner)); + Ok(CellQueryGateway { inner }) + } + + pub(crate) fn uses_query_domain(&self, domain: &DomainToken) -> bool { + self.handle.0.domain.same_owner(domain) + } + + /// Called only by a consumed catalog attempt. No forwarding or retry occurs + /// here; the caller keeps its original deadline and attempt budget. + pub(in crate::raft) async fn query_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellQueryCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let state = &self.handle.0; + // This checkpoint uses attached groups. Read-only ingress discovery is + // deliberately not implemented by fabricating a local voting identity. + state.runtime.shared()?; + let (view, peer, pool) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let view = registry + .physical_groups + .get(&route.history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if view.schema_digest != route.schema_digest || !route.history.same_namespace(cell) { + return Err(TransportError::Identity); + } + let peer = view + .peers + .iter() + .find(|p| { + p.replica.member == member + && p.replica.node != state.runtime.log().limits().node + }) + .ok_or(TransportError::Identity)? + .clone(); + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + (view, peer, pool) + }; + if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { + return Err(TransportError::Timeout); + } + let charge = queries::reservation( + call.request + .input + .len() + .checked_add(8192) + .ok_or(TransportError::Limit)?, + &state.options, + )?; + let node = state + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_bytes = pool + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + drop(call.encode(state.options.max_query_request_bytes)?); + if !state.current_physical(&view, &peer) { + return Err(TransportError::StalePeer); + } + let sequence = state + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + + 1; + let (reply, result) = oneshot::channel(); + let (cancel, cancelled) = watch::channel(false); + let _cancel = Cancel(cancel); + pool.query + .try_send(SendQuery::Shared(Box::new(SendCellQuery { + route: view, + peer, + cell: cell.clone(), + call, + keys, + sequence, + deadline, + cancel: cancelled, + reply, + node, + peer_bytes, + }))) + .map_err(|e| match e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + })?; + result.await.map_err(|_| TransportError::Closed)? + } +} + +impl CellQueryGateway { + /// The cluster and namespace bound to this host-owned registration. + pub fn namespace(&self) -> (ClusterId, NamespaceId) { + self.inner.dispatcher.namespace() + } +} + +impl State { + pub(super) fn authorize_cell_query( + &self, + wire: &QueryRoute, + ) -> Result<(Arc, Option>), TransportError> { + let runtime = self.runtime.shared()?; + let (source, target, _) = wire.decode_shared()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + shared::permitted(&group, &[&target])?; + let implicit = route.peers.iter().find(|p| { + p.replica.node_identity() == source + && p.label == wire.source_label() + && shared::permitted(&group, &[&p.replica]).is_ok() + }); + self.authorize_query_origin( + ®istry, + target.scope.cluster, + target.scope.namespace, + source, + wire.source_label(), + implicit, + ) + } + + pub(super) async fn send_cell_query( + &self, + pool: &Pool, + mut request: SendCellQuery, + connection: &mut Option, + ) -> bool { + let result = self + .query_until( + pool, + request.deadline, + request.cancel.clone(), + None, + async { + if !self.current_physical(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let remaining = request + .deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|n| *n > 0) + .ok_or(TransportError::Timeout)?; + request.call.timeout_ms = request.call.timeout_ms.min(remaining); + let body = request.call.encode(self.options.max_query_request_bytes)?; + let local = self.runtime.log().limits(); + let route = QueryRoute::new_shared( + NodeIdentity { + node: local.node, + disk: local.disk, + process: local.process, + }, + self.auth.source(), + &request.peer.replica, + &request.peer.label, + &request.cell, + request.route.schema_digest, + )?; + let header = wire::RequestHead::new(route, request.sequence, &body); + let body = self + .exchange_query( + connection, + &RaftPeerNode::from(&request.peer), + &header, + body, + ) + .await?; + if !self.current_physical(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::cell::decode_cell_result( + &body, + &request.cell, + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map_err(TransportError::Query) + }, + ) + .await; + // A fully authenticated application refusal consumed a complete reply; + // retain the peer connection just as the legacy query lane does. + let failed = matches!(&result, Err(error) if !matches!(error, TransportError::Query(_))); + let _ = request + .reply + .send(result.map(|value| CellRemoteQueryResponse { + value, + _node: request.node, + _peer: request.peer_bytes, + })); + failed + } +} + +pub(super) enum IncomingCall { + Legacy(QueryCall), + Shared(CellQueryCall), +} + +impl IncomingCall { + pub(super) fn decode( + route: &QueryRoute, + bytes: &[u8], + limit: usize, + ) -> Result { + if route.is_shared() { + Ok(Self::Shared(CellQueryCall::decode(bytes, limit)?)) + } else { + Ok(Self::Legacy(QueryCall::decode(bytes, limit)?)) + } + } + + pub(super) fn timeout_ms(&self) -> u64 { + match self { + Self::Legacy(call) => call.options.timeout.as_millis() as u64, + Self::Shared(call) => call.timeout_ms, + } + } + + pub(super) async fn execute( + self, + state: &State, + route: &QueryRoute, + ) -> Result, TransportError> { + match self { + Self::Legacy(call) => { + let (_, target) = route.decode()?; + let handle = state + .query_gateway + .lock() + .map_err(|_| TransportError::Closed)? + .as_ref() + .and_then(|h| h.upgrade()) + .ok_or(TransportError::Refused)?; + let cell = RaftCells { handle } + .cell(target.scope.namespace, &target.scope.cell) + .map_err(|_| TransportError::Refused)?; + if cell.scope() != &target.scope + || cell.binding.class.schema_digest() != route.schema() + { + return Err(TransportError::StalePeer); + } + let result = cell.query(call.contract, call.input, call.options).await; + if result.as_ref().is_ok_and(|v| v.served_by != target.member) { + return Err(TransportError::Identity); + } + wire::encode_result(&result, state.options.max_query_response_bytes) + } + Self::Shared(call) => { + let (_, target, cell) = route.decode_shared()?; + let gateway = state + .cell_query_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|h| h.upgrade()) + .ok_or(TransportError::Refused)?; + let result = gateway + .execute(call, target.scope, cell, route.schema()) + .await + .map_err(query_failure); + wire::cell::encode_cell_result(&result, state.options.max_query_response_bytes) + } + } + } +} + +impl Gateway { + async fn execute( + &self, + call: CellQueryCall, + group: GroupHistory, + cell: CellHistory, + schema: [u8; 32], + ) -> Result { + let epoch = call.route_epoch; + let birth = call.birth_digest; + let mut query = self.dispatcher.query( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + )?; + query.begin().await?; + let selected = query + .attempt() + .ok_or(CellDispatchError::Invalid("missing query route"))? + .route(); + if selected.history != group + || selected.schema_digest != schema + || selected.epoch.get() != epoch + || selected.birth_digest != birth + { + return Err(CellDispatchError::Invalid("remote query route")); + } + query + .execute(&self.runtime, &self.sql, self.options.clone()) + .await + } +} + +fn query_failure(error: CellDispatchError) -> CommandFailure { + use crate::raft::{execution::ExecutionError, version::VersionError}; + match error { + CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure, + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded | RuntimeError::NotReady) => { + CommandFailure::VersionUnavailable + } + CellDispatchError::Version(VersionError::WrongHistory) => CommandFailure::WrongIncarnation, + CellDispatchError::Version(_) => CommandFailure::VersionUnavailable, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/query_cell_wire.rs b/crates/cellar/durability/raft/query_cell_wire.rs new file mode 100644 index 000000000..78dde0a0a --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_wire.rs @@ -0,0 +1,187 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Logical cell query payloads on the existing authenticated query framing. +use super::*; +use crate::raft::cell_groups::{CellQueryConsistency, CellQueryRequest, CellQueryResponse}; +use crate::raft::version::{CellVersionObservation, CellVersionToken}; +use cellar_logic::{cell_group::CellHistory, consensus::ProposalId}; + +pub(in crate::raft) struct CellQueryCall { + pub request: CellQueryRequest, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +pub(in crate::raft) struct CellQueryValue { + pub result: Vec, + pub observation: CellVersionObservation, + pub token: CellVersionToken, + pub served_by: MemberId, +} + +impl CellQueryCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let request = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if request.contract.kind != OperationKind::Query + || request.contract.operation.is_empty() + || request.contract.operation.len() > 128 + || request.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCQQ01".to_vec(); + body.extend_from_slice(&request.invocation.bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &request.contract.operation)?; + body.extend_from_slice(&request.contract.digest); + match &request.consistency { + CellQueryConsistency::Stale => body.push(0), + CellQueryConsistency::Linearizable => body.push(1), + CellQueryConsistency::AtLeast(token) => { + body.push(2); + string(&mut body, token.as_str())?; + } + } + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(request.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(request.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&request.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCQQ01" { + return Err(TransportError::Protocol); + } + let invocation = ProposalId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Query, + digest: body.array()?, + }; + let consistency = match body.byte()? { + 0 => CellQueryConsistency::Stale, + 1 => CellQueryConsistency::Linearizable, + 2 => CellQueryConsistency::AtLeast( + CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?, + ), + _ => return Err(TransportError::Protocol), + }; + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellQueryRequest { + invocation, + contract, + input, + consistency, + }, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +pub(in crate::raft) fn encode_cell_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCQS01".to_vec(); + match result { + Ok(value) => { + body.push(0); + string(&mut body, value.token().as_str())?; + let length = u32::try_from(value.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(value.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(value.result()); + } + Err(error) => body.push(failure_code(error)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_cell_result( + bytes: &[u8], + cell: &CellHistory, + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCQS01" { + return Err(TransportError::Protocol); + } + let status = body.byte()?; + let result = if status == 0 { + let token = CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, cell, schema, now) + .map_err(|_| TransportError::Authentication)?; + let length = u32::from_be_bytes(body.array()?) as usize; + Ok(CellQueryValue { + result: body.take(length)?.to_vec(), + observation, + token, + served_by, + }) + } else { + Err(decode_failure(status)?) + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index 4baf8d7fb..07fc411d7 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -5,8 +5,10 @@ use super::transport::TransportError; use crate::application::consensus::protocol::WireScope; +use cellar_logic::cell_group::{CellHistory, CellId, GroupHistory}; use cellar_logic::consensus::{ - DiskLineage, MemberId, NodeId, NodeIdentity, ProcessGeneration, ReplicaIdentity, + ClusterId, DiskLineage, GroupId, IncarnationId, MemberId, NamespaceId, NodeId, NodeIdentity, + ProcessGeneration, ReplicaIdentity, }; use serde::{Deserialize, Serialize}; @@ -41,10 +43,22 @@ impl Endpoint { } } +// Marker-1 hard cutover: no reader for the former untagged query history. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(tag = "kind", content = "identity", deny_unknown_fields)] +enum History { + Cell(WireScope), + Shared { + group: [[u8; 16]; 4], + cell: String, + incarnation: [u8; 16], + }, +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct QueryRoute { - scope: WireScope, + scope: History, schema: [u8; 32], source: Endpoint, target: Endpoint, @@ -59,7 +73,7 @@ impl QueryRoute { schema: [u8; 32], ) -> Result { let route = Self { - scope: WireScope::from(&target.scope), + scope: History::Cell(WireScope::from(&target.scope)), schema, source: Endpoint::new(source, source_label, None), target: Endpoint::new(target.node_identity(), target_label, Some(target.member)), @@ -71,15 +85,21 @@ impl QueryRoute { /// Request orientation only. A reversed response route is compared with /// the exact outstanding request; it cannot decode as another request. pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { - if self.source.member.is_some() { + let History::Cell(scope) = &self.scope else { return Err(TransportError::Identity); - } - let scope = self - .scope + }; + let scope = scope .clone() .decode() .map_err(|_| TransportError::Identity)?; scope.validate().map_err(|_| TransportError::Identity)?; + self.endpoints(scope) + } + + fn endpoints(&self, scope: S) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { + if self.source.member.is_some() || self.schema == [0; 32] { + return Err(TransportError::Identity); + } let member = self .target .member @@ -104,6 +124,82 @@ impl QueryRoute { )) } + pub fn new_shared( + source: NodeIdentity, + source_label: &str, + target: &ReplicaIdentity, + target_label: &str, + cell: &CellHistory, + schema: [u8; 32], + ) -> Result { + if !target.scope.same_namespace(cell) { + return Err(TransportError::Identity); + } + let g = target.scope; + let route = Self { + scope: History::Shared { + group: [ + g.cluster.bytes(), + g.namespace.bytes(), + g.group.bytes(), + g.incarnation.bytes(), + ], + cell: cell.cell.as_str().into(), + incarnation: cell.incarnation.bytes(), + }, + schema, + source: Endpoint::new(source, source_label, None), + target: Endpoint::new(target.node_identity(), target_label, Some(target.member)), + }; + route.decode_shared()?; + Ok(route) + } + + pub fn decode_shared( + &self, + ) -> Result<(NodeIdentity, ReplicaIdentity, CellHistory), TransportError> { + let History::Shared { + group, + cell, + incarnation, + } = &self.scope + else { + return Err(TransportError::Identity); + }; + let scope = GroupHistory { + cluster: ClusterId::new(group[0]).map_err(|_| TransportError::Identity)?, + namespace: NamespaceId::new(group[1]).map_err(|_| TransportError::Identity)?, + group: GroupId::new(group[2]).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(group[3]).map_err(|_| TransportError::Identity)?, + }; + let cell = CellHistory { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new(cell).map_err(|_| TransportError::Identity)?, + incarnation: IncarnationId::new(*incarnation).map_err(|_| TransportError::Identity)?, + }; + let (source, target) = self.endpoints(scope)?; + Ok((source, target, cell)) + } + + pub fn is_shared(&self) -> bool { + matches!(self.scope, History::Shared { .. }) + } + + pub fn validate(&self) -> Result<(), TransportError> { + if self.is_shared() { + self.decode_shared()?; + } else { + self.decode()?; + } + Ok(()) + } + + pub fn source(&self) -> Result { + self.validate()?; + self.source.identity() + } + pub fn schema(&self) -> [u8; 32] { self.schema } diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 31d9aa28a..9601b7538 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -4,7 +4,7 @@ //! occupy the control or LTX connection required to establish its barrier. use super::*; -use crate::application::consensus::{client, CommandFailure, QueryResult, RaftCells, VersionKeys}; +use crate::application::consensus::{client, CommandFailure, QueryResult, VersionKeys}; use crate::raft::query_identity::QueryRoute; use crate::raft::query_wire::{self as wire, QueryCall}; use cellar_logic::consensus::{MemberId, NodeIdentity}; @@ -37,7 +37,12 @@ impl Drop for QueryDelivery { } } -pub(super) struct SendQuery { +pub(super) enum SendQuery { + Legacy(Box), + Shared(Box), +} + +pub(super) struct LegacySendQuery { scope: GroupScope, route: Arc, peer: RaftPeer, @@ -342,7 +347,7 @@ impl RaftTransport { let (reply, result) = oneshot::channel(); let (cancel, cancelled) = watch::channel(false); pool.query - .try_send(SendQuery { + .try_send(SendQuery::Legacy(Box::new(LegacySendQuery { scope: scope.clone(), route, peer, @@ -354,7 +359,7 @@ impl RaftTransport { reply, node, peer_bytes, - }) + }))) .map_err(|e| match e { mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, mpsc::error::TrySendError::Closed(_) => TransportError::Closed, @@ -366,7 +371,10 @@ impl RaftTransport { } } -fn reservation(request: usize, options: &TransportOptions) -> Result { +pub(super) fn reservation( + request: usize, + options: &TransportOptions, +) -> Result { request .checked_mul(2) .and_then(|n| { @@ -397,6 +405,9 @@ impl State { &self, wire: &QueryRoute, ) -> Result<(Arc, Option>), TransportError> { + if wire.is_shared() { + return self.authorize_cell_query(wire); + } let (source, target) = wire.decode()?; let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; self.admit(®istry)?; @@ -439,16 +450,37 @@ impl State { let implicit = member_peer.filter(|peer| { present(peer.replica.member.get()) && cache.membership.permits(&peer.replica) }); + self.authorize_query_origin( + ®istry, + target.scope.cluster, + target.scope.namespace, + source, + wire.source_label(), + implicit, + ) + } + + /// Shared and temporary legacy queries have exactly the same read-only + /// origin grant, process binding, revocation and peer-pool permission path. + pub(super) fn authorize_query_origin( + &self, + registry: &Registry, + cluster: ClusterId, + namespace: NamespaceId, + source: NodeIdentity, + source_label: &str, + implicit: Option<&RaftPeer>, + ) -> Result<(Arc, Option>), TransportError> { let grant = registry .origins - .get(&(target.scope.cluster, target.scope.namespace, source.node)) + .get(&(cluster, namespace, source.node)) .filter(|grant| !*grant.stop.borrow()) .filter(|grant| { grant.peer.as_ref().is_some_and(|peer| { peer.node == source.node && peer.disk == source.disk && peer.process == source.process - && peer.label == wire.source_label() + && peer.label == source_label }) }); let permission = if implicit.is_some() { @@ -578,7 +610,7 @@ impl State { .bounded(None, read_query_head(&mut stream, &self.options, false)) .await?; let header = wire::read_request(&head, &self.auth, &exporter, &self.options)?; - let (source, target) = header.route.decode()?; + let source = header.route.source()?; let host = ( source.node, source.disk, @@ -621,50 +653,30 @@ impl State { }) .await?; header.verify_body(&body)?; - let call = QueryCall::decode(&body, self.options.max_query_request_bytes)?; + let call = super::cell_queries::IncomingCall::decode( + &header.route, + &body, + self.options.max_query_request_bytes, + )?; drop(body); self.authorize_query_route(&header.route)?; - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|handle| handle.upgrade()) - .ok_or(TransportError::Refused)?; - let gateway = RaftCells { handle }; - let cell = gateway - .cell(target.scope.namespace, &target.scope.cell) - .map_err(|_| TransportError::Refused)?; - if cell.scope() != &target.scope - || cell.binding.class.schema_digest() != header.route.schema() - { - return Err(TransportError::StalePeer); - } - let timeout = u64::try_from(call.options.timeout.as_millis()) - .map_err(|_| TransportError::Limit)?; let until = self .domain .mono_ms() .map_err(|_| TransportError::Closed)? - .saturating_add(timeout); - let query = cell.query(call.contract, call.input, call.options); + .checked_add(call.timeout_ms()) + .ok_or(TransportError::Limit)?; + let query = call.execute(&self, &header.route); let mut extra = [0; 1]; - // Query connections do not pipeline. EOF or unexpected input while - // waiting cancels the SDK waiter and its bounded SQL work. - let result = self.query_until(&pool, until, cancel.clone(), permission.clone(), async { + // No pipelining: EOF/unexpected input cancels the SDK waiter while + // the SQL owner retains any entered physical work through cleanup. + let body = self.query_until(&pool, until, cancel.clone(), permission.clone(), async { crate::asyncrt::select! { - value = query => Ok(value), + value = query => value, read = stream.read(&mut extra) => { let _ = read?; Err(TransportError::Closed) }, } }).await?; - if result - .as_ref() - .is_ok_and(|value| value.served_by != target.member) - { - return Err(TransportError::Identity); - } self.authorize_query_route(&header.route)?; - let body = wire::encode_result(&result, self.options.max_query_response_bytes)?; let response = wire::response_head( &self.auth, &exporter, @@ -696,128 +708,154 @@ impl State { if *stop.borrow() || *peer_stop.borrow() { return; } - let mut request = crate::asyncrt::select! { _ = stop.changed() => return, _ = peer_stop.changed() => return, request = requests.recv() => match request { Some(request) => request, None => return } }; - let result = self - .query_until( - &pool, - request.deadline, - request.cancel.clone(), - None, - async { - if !self.current_query_route(&request.scope, &request.route, &request.peer) - { - return Err(TransportError::StalePeer); - } - if connection.is_none() { - let owner = self.outgoing_connections.query(); - let tcp = TcpStream::connect(request.peer.query_address).await?; - tcp.set_nodelay(true)?; - let tcp = self.observed_socket(tcp, metrics::Lane::Query); - let name = rustls::pki_types::ServerName::try_from( - request.peer.server_name.clone(), - ) - .map_err(|_| TransportError::Identity)?; - let tls = tokio_rustls::TlsConnector::from(self.tls.client.clone()) - .connect(name, tcp) - .await - .map_err(|e| TransportError::Tls(e.to_string()))?; - let exporter = tls - .get_ref() - .1 - .export_keying_material([0; 32], wire::EXPORT_LABEL, None) - .map_err(|e| TransportError::Tls(e.to_string()))?; - connection = Some((tls, exporter, owner)); - } - let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; - let remaining = request - .deadline - .checked_sub(now) - .filter(|ms| *ms > 0) - .ok_or(TransportError::Timeout)?; - request.call.options.timeout = request - .call - .options - .timeout - .min(std::time::Duration::from_millis(remaining)); - let body = request.call.encode(self.options.max_query_request_bytes)?; - let local = self.runtime.log().limits(); - let route = QueryRoute::new( - NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }, - self.auth.source(), - &request.peer.replica, - &request.peer.label, - request.route.schema_digest, - )?; - let header = wire::RequestHead::new(route, request.sequence, &body); - let (stream, exporter, _owner) = - connection.as_mut().ok_or(TransportError::Closed)?; - let head = wire::request_head( - &self.auth, - exporter, - &header, - &body, - &self.options, - )?; - write_head(stream, &head).await?; - stream.write_all(&body).await?; - stream.flush().await?; - drop(body); - let response = read_query_head(stream, &self.options, true).await?; - let digest = wire::read_response( - &response, - &self.auth, - exporter, - &header, - &head.tag, - &self.options, - )?; - let mut body = vec![0; response.payload_len]; - stream.read_exact(&mut body).await?; - if digest != <[u8; 32]>::from(Sha256::digest(&body)) { - return Err(TransportError::Authentication); - } - if !self.current_query_route(&request.scope, &request.route, &request.peer) - { - return Err(TransportError::StalePeer); - } - let now = u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| TransportError::Authentication)?; - wire::decode_result( - &body, - &request.scope, - request.route.schema_digest, - &request.keys, - now, - self.options.max_query_response_bytes, - request.peer.replica.member, + let request = crate::asyncrt::select! { _ = stop.changed() => return, _ = peer_stop.changed() => return, request = requests.recv() => match request { Some(request) => request, None => return } }; + let failed = match request { + SendQuery::Legacy(mut request) => { + let result = self + .query_until( + &pool, + request.deadline, + request.cancel.clone(), + None, + self.send_legacy_query(&mut request, &mut connection), ) - }, - ) - .await; - if result.is_err() { + .await; + let failed = result.is_err(); + let LegacySendQuery { + reply, + node, + peer_bytes, + .. + } = *request; + let _ = reply.send(result.map(|value| QueryResponse { + value, + _node: node, + _peer: peer_bytes, + })); + failed + } + SendQuery::Shared(request) => { + self.send_cell_query(&pool, *request, &mut connection).await + } + }; + if failed { connection = None; self.failed.fetch_add(1, Ordering::Relaxed); } - let SendQuery { - reply, - node, - peer_bytes, - .. - } = request; - let result = result.map(|value| QueryResponse { - value, - _node: node, - _peer: peer_bytes, - }); - let _ = reply.send(result); } } + + async fn send_legacy_query( + &self, + request: &mut LegacySendQuery, + connection: &mut Option, + ) -> Result, TransportError> { + if !self.current_query_route(&request.scope, &request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let remaining = request + .deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; + request.call.options.timeout = request + .call + .options + .timeout + .min(std::time::Duration::from_millis(remaining)); + let body = request.call.encode(self.options.max_query_request_bytes)?; + let local = self.runtime.log().limits(); + let route = QueryRoute::new( + NodeIdentity { + node: local.node, + disk: local.disk, + process: local.process, + }, + self.auth.source(), + &request.peer.replica, + &request.peer.label, + request.route.schema_digest, + )?; + let header = wire::RequestHead::new(route, request.sequence, &body); + let body = self + .exchange_query( + connection, + &RaftPeerNode::from(&request.peer), + &header, + body, + ) + .await?; + if !self.current_query_route(&request.scope, &request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let now = + u64::try_from(crate::asyncrt::wall_ms()).map_err(|_| TransportError::Authentication)?; + wire::decode_result( + &body, + &request.scope, + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + ) + } + + pub(super) async fn exchange_query( + &self, + connection: &mut Option, + peer: &RaftPeerNode, + header: &wire::RequestHead, + body: Vec, + ) -> Result, TransportError> { + if connection.is_none() { + let owner = self.outgoing_connections.query(); + let tcp = TcpStream::connect(peer.query_address).await?; + tcp.set_nodelay(true)?; + let tcp = self.observed_socket(tcp, metrics::Lane::Query); + let name = rustls::pki_types::ServerName::try_from(peer.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + let tls = tokio_rustls::TlsConnector::from(self.tls.client.clone()) + .connect(name, tcp) + .await + .map_err(|e| TransportError::Tls(e.to_string()))?; + let exporter = tls + .get_ref() + .1 + .export_keying_material([0; 32], wire::EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + *connection = Some((tls, exporter, owner)); + } + let (stream, exporter, _owner) = connection.as_mut().ok_or(TransportError::Closed)?; + let head = wire::request_head(&self.auth, exporter, header, &body, &self.options)?; + write_head(stream, &head).await?; + stream.write_all(&body).await?; + stream.flush().await?; + drop(body); + let response = read_query_head(stream, &self.options, true).await?; + let digest = wire::read_response( + &response, + &self.auth, + exporter, + header, + &head.tag, + &self.options, + )?; + let mut body = vec![0; response.payload_len]; + stream.read_exact(&mut body).await?; + if digest != <[u8; 32]>::from(Sha256::digest(&body)) { + return Err(TransportError::Authentication); + } + Ok(body) + } } +pub(super) type QueryConnection = ( + tokio_rustls::client::TlsStream>, + [u8; 32], + resources::ConnectionOwner, +); + async fn read_query_head( stream: &mut S, options: &TransportOptions, diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index 2e47a84d3..31509fcb5 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -17,6 +17,9 @@ use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::time::Duration; +#[path = "query_cell_wire.rs"] +pub(super) mod cell; + pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; const RESPONSE: &[u8] = b"cellar-raft-query-response-1"; @@ -144,16 +147,7 @@ pub(super) fn encode_result( body.extend_from_slice(&length.to_be_bytes()); body.extend_from_slice(&result.result); } - Err(failure) => body.push(match failure { - CommandFailure::VersionUnavailable => 1, - CommandFailure::WrongIncarnation => 2, - CommandFailure::UnsupportedCapability => 3, - CommandFailure::UnsupportedConsistency => 4, - CommandFailure::Overloaded => 5, - CommandFailure::CorruptState => 6, - CommandFailure::StorageFull => 8, - _ => 7, - }), + Err(failure) => body.push(failure_code(failure)), } if body.len() > limit { return Err(TransportError::Limit); @@ -192,15 +186,7 @@ pub(super) fn decode_result( served_by, }) } - 1 => Err(CommandFailure::VersionUnavailable), - 2 => Err(CommandFailure::WrongIncarnation), - 3 => Err(CommandFailure::UnsupportedCapability), - 4 => Err(CommandFailure::UnsupportedConsistency), - 5 => Err(CommandFailure::Overloaded), - 6 => Err(CommandFailure::CorruptState), - 7 => Err(CommandFailure::Unavailable), - 8 => Err(CommandFailure::StorageFull), - _ => return Err(TransportError::Protocol), + status => Err(decode_failure(status)?), }; if !body.0.is_empty() { return Err(TransportError::Protocol); @@ -208,6 +194,33 @@ pub(super) fn decode_result( Ok(result) } +fn failure_code(failure: &CommandFailure) -> u8 { + match failure { + CommandFailure::VersionUnavailable => 1, + CommandFailure::WrongIncarnation => 2, + CommandFailure::UnsupportedCapability => 3, + CommandFailure::UnsupportedConsistency => 4, + CommandFailure::Overloaded => 5, + CommandFailure::CorruptState => 6, + CommandFailure::StorageFull => 8, + _ => 7, + } +} + +fn decode_failure(status: u8) -> Result { + Ok(match status { + 1 => CommandFailure::VersionUnavailable, + 2 => CommandFailure::WrongIncarnation, + 3 => CommandFailure::UnsupportedCapability, + 4 => CommandFailure::UnsupportedConsistency, + 5 => CommandFailure::Overloaded, + 6 => CommandFailure::CorruptState, + 7 => CommandFailure::Unavailable, + 8 => CommandFailure::StorageFull, + _ => return Err(TransportError::Protocol), + }) +} + #[derive(Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct RequestHead { @@ -261,7 +274,7 @@ pub(super) fn request_head( options: &TransportOptions, ) -> Result { request.sequence()?; - request.route.decode()?; + request.route.validate()?; request.verify_body(body)?; make_head( auth, @@ -284,7 +297,7 @@ pub(super) fn read_request( let request: RequestHead = serde_json::from_slice(&head.header).map_err(|_| TransportError::Protocol)?; request.sequence()?; - request.route.decode()?; + request.route.validate()?; if serde_json::to_vec(&request).map_err(|_| TransportError::Protocol)? != head.header { return Err(TransportError::Protocol); } diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index 0a97ea159..e3aecb045 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -353,3 +353,281 @@ fn query_wire_failures_remain_typed() { .is_err()); } } + +fn cell_observation() -> crate::raft::version::CellVersionObservation { + use cellar_logic::cell_group::*; + crate::raft::version::CellVersionObservation { + version: CellVersion { + history: CellHistory::try_from(&identity(1).scope).unwrap(), + sequence: MutationSequence::new(7).unwrap(), + image: CellImage { + epoch: 1, + txid: 10, + checksum: 17, + commitment: CellImageCommitment::Materialized([8; 32]), + }, + }, + witness: CommitWitness { + group: GroupHistory::from(&identity(1).scope), + position: LogPosition { + index: LogIndex::new(100), + term: Term::new(9), + }, + proposal: ProposalId::new([6; 16]).unwrap(), + }, + } +} + +fn cell_call(consistency: crate::raft::cell_groups::CellQueryConsistency) -> cell::CellQueryCall { + cell::CellQueryCall { + request: crate::raft::cell_groups::CellQueryRequest { + invocation: ProposalId::new([91; 16]).unwrap(), + contract: call(QueryConsistency::Stale).contract, + input: vec![1, 2, 3], + consistency, + }, + timeout_ms: 700, + route_epoch: 3, + birth_digest: [92; 32], + } +} + +#[test] +fn shared_query_wire_preserves_cell_constraints_and_refuses_other_payloads() { + use crate::raft::cell_groups::CellQueryConsistency; + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); + for consistency in [ + CellQueryConsistency::Stale, + CellQueryConsistency::Linearizable, + CellQueryConsistency::AtLeast(token), + ] { + let call = cell_call(consistency); + let bytes = call.encode(4096).unwrap(); + let decoded = cell::CellQueryCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.invocation, call.request.invocation); + assert_eq!(decoded.request.contract, call.request.contract); + assert_eq!(decoded.request.input, call.request.input); + assert_eq!(decoded.timeout_ms, 700); + assert_eq!(decoded.route_epoch, 3); + assert_eq!(decoded.birth_digest, [92; 32]); + for end in 0..bytes.len() { + assert!(cell::CellQueryCall::decode(&bytes[..end], 4096).is_err()); + } + assert!(QueryCall::decode(&bytes, 4096).is_err()); + let mut trailing = bytes.clone(); + trailing.push(0); + assert!(cell::CellQueryCall::decode(&trailing, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, bytes.len() - 1).is_err()); + } + assert!(cell::CellQueryCall::decode( + &call(QueryConsistency::Stale).encode(4096).unwrap(), + 4096 + ) + .is_err()); + let mut call = cell_call(CellQueryConsistency::Stale); + call.request.contract.kind = OperationKind::Command; + assert!(call.encode(4096).is_err()); + call.request.contract.kind = OperationKind::Query; + call.timeout_ms = 0; + assert!(call.encode(4096).is_err()); + call.timeout_ms = 700; + call.route_epoch = 0; + assert!(call.encode(4096).is_err()); + call.route_epoch = 3; + call.birth_digest = [0; 32]; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_query_route_binds_physical_group_logical_cell_and_process() { + use crate::raft::cell_groups::CellQueryConsistency; + use cellar_logic::cell_group::{CellId, GroupHistory}; + let old = identity(2); + let target = ReplicaIdentity { + scope: GroupHistory::from(&old.scope), + node: old.node, + disk: old.disk, + process: old.process, + member: old.member, + }; + let cell = cell_observation().version.history; + let source = identity(1).node_identity(); + let route = Route::new_shared(source, "node-1", &target, "node-2", &cell, [9; 32]).unwrap(); + assert_eq!( + route.decode_shared().unwrap(), + (source, target.clone(), cell.clone()) + ); + assert!(route.decode().is_err()); + assert!(route.reversed().decode_shared().is_err()); + assert!(Route::new(source, "node-1", &old, "node-2", [9; 32]) + .unwrap() + .decode_shared() + .is_err()); + let value = serde_json::to_value(&route).unwrap(); + assert!(value["source"]["member"].is_null()); + assert!(serde_json::from_value::(value.clone()).is_err()); + for path in [ + ["source", "node"], + ["source", "disk"], + ["source", "process"], + ["target", "node"], + ["target", "disk"], + ["target", "process"], + ] { + let mut changed = value.clone(); + changed[path[0]][path[1]] = serde_json::to_value([0u8; 16]).unwrap(); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + } + let mut changed = value.clone(); + changed["source"]["member"] = serde_json::json!("1"); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + let mut changed = value.clone(); + changed["scope"]["identity"]["cell"] = serde_json::json!("bad\ncell"); + assert!(serde_json::from_value::(changed) + .unwrap() + .decode_shared() + .is_err()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); + let request = RequestHead::new(route.clone(), 17, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + assert!(read_request(&head, &auth, &[3; 32], &options).is_ok()); + assert!(read_request(&head, &auth, &[4; 32], &options).is_err()); + let reply = cell::encode_cell_result(&Err(CommandFailure::Overloaded), 4096).unwrap(); + let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); + let mut other_cell = cell.clone(); + other_cell.cell = CellId::new("neighbor").unwrap(); + let mut other_group = target.clone(); + other_group.scope.group = GroupId::new([99; 16]).unwrap(); + let mut other_process = target.clone(); + other_process.process = ProcessGeneration::new([99; 16]).unwrap(); + for (target, cell) in [ + (&target, &other_cell), + (&other_group, &cell), + (&other_process, &cell), + ] { + let wrong = RequestHead::new( + Route::new_shared(source, "node-1", target, "node-2", cell, [9; 32]).unwrap(), + 17, + &body, + ); + assert!(read_response(&response, &auth, &[3; 32], &wrong, &head.tag, &options).is_err()); + } + for i in 0..body.len() { + let mut altered = body.clone(); + altered[i] ^= 1; + assert!(request.verify_body(&altered).is_err()); + } + let mut wrong_namespace = cell; + wrong_namespace.namespace = NamespaceId::new([99; 16]).unwrap(); + assert!(Route::new_shared( + source, + "node-1", + &target, + "node-2", + &wrong_namespace, + [9; 32] + ) + .is_err()); +} + +#[test] +fn shared_query_result_authenticates_cell_progress_not_group_log_index() { + let observation = cell_observation(); + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let mut bytes = b"CLGCQS01\0".to_vec(); + string(&mut bytes, token.as_str()).unwrap(); + bytes.extend_from_slice(&3u32.to_be_bytes()); + bytes.extend_from_slice(b"yes"); + let member = MemberId::new(2).unwrap(); + let decoded = cell::decode_cell_result( + &bytes, + &observation.version.history, + [9; 32], + &keys(), + 1000, + 4096, + member, + ) + .unwrap() + .unwrap(); + assert_eq!(decoded.observation, observation); + assert_eq!(decoded.observation.version.sequence.get(), 7); + assert_eq!(decoded.observation.witness.position.index.get(), 100); + assert_eq!(decoded.result, b"yes"); + assert_eq!(decoded.served_by, member); + let mut wrong = observation.version.history.clone(); + wrong.incarnation = IncarnationId::new([99; 16]).unwrap(); + assert!( + cell::decode_cell_result(&bytes, &wrong, [9; 32], &keys(), 1000, 4096, member).is_err() + ); + assert!(cell::decode_cell_result( + &bytes, + &observation.version.history, + [8; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(cell::decode_cell_result( + &bytes, + &observation.version.history, + [9; 32], + &keys(), + 3000, + 4096, + member + ) + .is_err()); + assert!(decode_result( + &bytes, + &identity(1).scope, + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + for end in 0..bytes.len() { + assert!(cell::decode_cell_result( + &bytes[..end], + &observation.version.history, + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + } + for error in [ + CommandFailure::WrongIncarnation, + CommandFailure::Overloaded, + CommandFailure::VersionUnavailable, + CommandFailure::Unavailable, + CommandFailure::StorageFull, + CommandFailure::CorruptState, + CommandFailure::UnsupportedCapability, + CommandFailure::UnsupportedConsistency, + ] { + let bytes = cell::encode_cell_result(&Err(error.clone()), 4096).unwrap(); + assert!( + matches!(cell::decode_cell_result(&bytes, &observation.version.history, [9; 32], &keys(), 1000, 4096, member).unwrap(), Err(actual) if actual == error) + ); + } +} diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index db0f89078..db78d4dde 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -35,6 +35,8 @@ use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::{mpsc, oneshot, watch, Notify, Semaphore}; +#[path = "query_cell_transport.rs"] +mod cell_queries; #[path = "transport_control_send.rs"] mod control_sender; #[path = "transport_directory.rs"] @@ -55,6 +57,7 @@ mod metrics; pub(crate) mod node_inspections; #[path = "query_transport.rs"] mod queries; +pub use cell_queries::{CellQueryGateway, CellRemoteQueryResponse}; #[path = "transport_resources.rs"] mod resources; #[path = "transport_send.rs"] @@ -78,6 +81,7 @@ pub enum TransportError { Tls(String), Runtime(RuntimeError), Authentication, + Query(cellar_logic::consensus::CommandFailure), Protocol, Identity, Replay, @@ -328,6 +332,8 @@ struct State { local_control: SocketAddr, local_data: SocketAddr, local_query: SocketAddr, + cell_query_gateways: + Mutex>>, query_gateway: Mutex>>, registry: Mutex, outgoing: Banks, @@ -487,6 +493,7 @@ impl RaftTransport { local_data, local_query, query_gateway: Mutex::new(None), + cell_query_gateways: Mutex::new(BTreeMap::new()), registry: Mutex::new(Registry { closed: false, tasks: 0, @@ -1025,6 +1032,9 @@ impl State { if let Ok(mut gateway) = self.query_gateway.lock() { gateway.take(); } + if let Ok(mut gateways) = self.cell_query_gateways.lock() { + gateways.clear(); + } if finished { if let Ok(mut claim) = self.claim.lock() { claim.take(); diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 8166a682d..9fb7ecfc9 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -20,7 +20,7 @@ pub(super) enum SendRoute { }, } -fn permitted( +pub(super) fn permitted( group: &GroupLog, peers: &[&ReplicaIdentity], ) -> Result<(), TransportError> { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index bd2e76811..1279966e3 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1460,3 +1460,18 @@ async fn controller_enrollment_joins_lost_worker_reply() -> Result<()> { async fn controller_enrollment_requires_registered_initializer() -> Result<()> { Box::pin(runtime_fixture::enrollment_requires_registration()).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_bind_cell_versions_and_gateway_owners() -> Result<()> { + Box::pin(runtime_fixture::remote_queries()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_retain_cancelled_sql_until_join() -> Result<()> { + Box::pin(runtime_fixture::remote_query_lifecycle(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_queries_retain_expired_sql_until_join() -> Result<()> { + Box::pin(runtime_fixture::remote_query_lifecycle(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs new file mode 100644 index 000000000..1a00d738f --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -0,0 +1,441 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared catalog dispatch over the production query TLS lane and SQL owners. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn dispatcher( + fixture: &RuntimeCluster, + class: Arc, + keys: VersionKeys, +) -> Result { + Ok(CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class, + keys, + CellDispatchLimits { + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +fn keys(tag: u8) -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([tag; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?) +} + +async fn invocation( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + consistency: CellQueryConsistency, + timeout: Duration, + input: &[u8], +) -> Result { + let mut query = dispatcher.query( + cell.clone(), + CellQueryRequest { + invocation: ProposalId::new([197; 16])?, + contract: class.contract("Get").context("Get")?, + input: input.to_vec(), + consistency, + }, + CellRoutingLimits { + timeout, + max_attempts: 2, + }, + None, + )?; + query.begin().await?; + Ok(query) +} + +async fn gateways( + fixture: &RuntimeCluster, + class: Arc, +) -> Result> { + let mut gateways = Vec::new(); + let group = fixture.cluster.birth.route().history; + let cell = history(fixture.cluster.birth.route(), "query-options")?; + for n in 0..3 { + gateways.push(fixture.transports[n].install_cell_query_gateway( + dispatcher(fixture, class.clone(), keys(190)?)?, + fixture.appliers[&(n, group)].0.clone(), + fixture.cell_options_at(&cell, n).await?, + )?); + } + Ok(gateways) +} + +async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { + tokio::time::timeout(Duration::from_secs(10), async { + while !predicate().await? { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await? +} + +pub(crate) async fn remote_queries() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, _| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("physical group cut"); + }; + assert_eq!(context.identity(), &observation.version.history); + assert_eq!( + context.query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))?, + 0 + ); + Ok(context.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let mut gateways = gateways(&fixture, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + assert_eq!(gateways[0].namespace(), (a.cluster, a.namespace)); + let options = fixture.cell_options(&a).await?; + assert!(matches!( + fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + fixture.appliers[&(0, route.history)].0.clone(), + options.clone() + ), + Err(TransportError::Refused) + )); + assert!(matches!( + fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + fixture.appliers[&(1, route.history)].0.clone(), + options.clone() + ), + Err(TransportError::Identity) + )); + let source = &fixture.transports[1]; + let leader = MemberId::new(1)?; + let mut first = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let attempt = first.attempt().context("attempt")?.clone(); + let response = first.execute_remote(source, leader).await?; + assert_eq!(response.result(), a.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, a); + assert_eq!(response.observation().version.sequence.get(), 1); + assert_eq!(response.served_by(), leader); + let token = response.token().clone(); + let expected = response.observation().clone(); + assert_eq!(source.usage().query_items, 1); + assert!(source.usage().query_bytes > response.result().len()); + assert!(matches!( + first.execute_remote(source, leader).await, + Err(CellDispatchError::Invalid("consumed route attempt")) + )); + drop(response); + assert_eq!(source.usage().query_items, 0); + first.retry().await?; + assert_eq!( + first.attempt().context("retry")?.deadline_ms(), + attempt.deadline_ms() + ); + assert_eq!(first.attempt().context("retry")?.ordinal(), 2); + assert_eq!( + *first.execute_remote(source, leader).await?.observation(), + expected + ); + assert!(matches!( + first.retry().await, + Err(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::AttemptsExhausted, + .. + })) + )); + drop(first); + for (cell, member, consistency) in [ + (&a, leader, CellQueryConsistency::AtLeast(token.clone())), + (&b, MemberId::new(3)?, CellQueryConsistency::Stale), + ] { + let mut query = invocation( + &dispatch, + &class, + cell, + consistency, + Duration::from_secs(5), + b"", + ) + .await?; + let response = query.execute_remote(source, member).await?; + assert_eq!(response.result(), cell.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, *cell); + assert_eq!(response.observation().version.sequence.get(), 1); + } + let before = calls.load(Ordering::SeqCst); + let mut wrong = invocation( + &dispatch, + &class, + &b, + CellQueryConsistency::AtLeast(token), + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + wrong.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::WrongIncarnation + ))) + )); + drop(wrong); + let mut wrong_cell = a.clone(); + wrong_cell.incarnation = IncarnationId::new([199; 16])?; + let mut wrong = invocation( + &dispatch, + &class, + &wrong_cell, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(wrong.execute_remote(source, leader).await.is_err()); + drop(wrong); + assert_eq!(calls.load(Ordering::SeqCst), before); + // A signature under a different namespace key cannot become a read result. + let other_keys = dispatcher(&fixture, class.clone(), keys(191)?)?; + let mut wrong = invocation( + &other_keys, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + wrong.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Authentication)) + )); + drop(wrong); + // Dropping the host registration refuses new reads; transport does not + // secretly retain a public handle or keep an abandoned gateway alive. + drop(gateways.remove(0)); + let before = calls.load(Ordering::SeqCst); + let mut absent = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(absent.execute_remote(source, leader).await.is_err()); + assert_eq!(calls.load(Ordering::SeqCst), before); + let replacement = fixture.transports[0].install_cell_query_gateway( + dispatch.clone(), + fixture.appliers[&(0, route.history)].0.clone(), + options, + )?; + absent.retry().await?; + assert_eq!( + absent.execute_remote(source, leader).await?.result(), + a.cell.as_str().as_bytes() + ); + drop(absent); + assert_eq!(dispatch.usage(), (0, 0)); + // Change only the catalog, not physical state: the old destination must + // refuse the captured route. This is not a live-movement qualification. + let mut stale = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + let original_deadline = stale.attempt().context("original")?.deadline_ms(); + let directory = fixture.cluster.directory.clone(); + let next = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([203; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([204; 16])?, + voters: fixture.cluster.birth.voters().to_vec(), + learners: vec![], + })?; + fixture + .cluster + .domain + .run(directory.reserve_group_birth(&next)) + .await??; + let view = fixture.cluster.domain.run(directory.observe()).await??; + let publish = directory.prepare_replacement( + &view, + ProposalId::new([205; 16])?, + std::slice::from_ref(&route), + &[next.route().clone()], + )?; + fixture + .cluster + .domain + .run(directory.publish(&publish)) + .await??; + let before = calls.load(Ordering::SeqCst); + assert!(matches!( + stale.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::Unavailable + ))) + )); + assert_eq!(calls.load(Ordering::SeqCst), before); + stale.redirect(next.route()).await?; + assert_eq!( + stale.attempt().context("redirect")?.deadline_ms(), + original_deadline + ); + assert_eq!(stale.attempt().context("redirect")?.ordinal(), 2); + assert!(matches!( + stale.execute_remote(source, leader).await, + Err(CellDispatchError::Transport(TransportError::UnknownGroup)) + )); + assert_eq!(calls.load(Ordering::SeqCst), before); + drop(stale); + assert_eq!(dispatch.usage(), (0, 0)); + for transport in &fixture.transports { + assert_eq!(transport.usage().groups, 1); + assert_eq!(transport.usage().peers, 2); + } + for controller in controllers { + controller.close().await?; + } + drop(replacement); + drop(gateways); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { + for rooted in [false, true] { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, _| { + entered + .lock() + .unwrap() + .take() + .context("once")? + .send(()) + .map_err(|_| anyhow::anyhow!("entered"))?; + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + Ok(context + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let gateways = gateways(&fixture, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let timeout = if expire { + Duration::from_millis(600) + } else { + Duration::from_secs(10) + }; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + timeout, + b"", + ) + .await?; + let source = fixture.transports[1].clone(); + let waiter = fixture.cluster.domain.spawn(async move { + query + .execute_remote(&source, MemberId::new(1).unwrap()) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + assert_eq!(pool.usage()?.retained_operations, 1); + // The blocked query does not occupy control connections or Raft work. + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 1000 }) + .await?; + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + drop(authority); + if expire { + assert!(waiter.await?.is_err()); + } else { + waiter.abort(); + assert!(waiter.await.is_err()); + } + wait_for(async || { + Ok(fixture.transports[1].usage().query_items == 0 + && fixture.transports[0].usage().query_items == 0) + }) + .await?; + assert_eq!(pool.usage()?.retained_operations, 1); + assert_eq!(dispatch.usage(), (0, 0)); + // Cancelling the close waiter cannot release the entered SQL worker. + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert_eq!(pool.usage()?.running_workers, 1); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), controllers[0].close()).await??; + assert_eq!(pool.usage()?.retained_operations, 0); + assert_eq!(pool.usage()?.running_workers, 0); + for controller in &controllers[1..] { + controller.close().await?; + } + drop(gateways); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 2704a6968..3e9d3e830 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -705,3 +705,5 @@ pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { pub(super) use sql::{ enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, }; + +pub(super) use sql::{remote_queries, remote_query_lifecycle}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 5709b9dc3..3e915eedd 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -544,3 +544,7 @@ mod controller_enrollment; pub(crate) use controller_enrollment::{ enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, }; + +#[path = "cell_group_remote_queries.rs"] +mod remote_queries; +pub(crate) use remote_queries::{remote_queries, remote_query_lifecycle}; From 201ffaf0465546485f20d26d6e10c23d4867cb09 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 07:02:07 -0700 Subject: [PATCH 096/568] fix: fence query admission when host registration drops --- .../durability/raft/query_cell_transport.rs | 27 ++++++++++++++++--- .../support/cell_group_remote_queries.rs | 25 ++++++++++++++++- 2 files changed, 48 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index ecbd1e831..e1d3b27d5 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -25,10 +25,23 @@ use std::time::Duration; /// cannot extend the host's lifetime or create a runtime/transport ownership cycle. #[derive(Clone)] pub struct CellQueryGateway { - inner: Arc, + inner: Arc, +} + +// Accepted requests retain Gateway, never this admission owner. In-flight work +// cannot keep accepting new reads after the final host registration is dropped. +struct Registration { + gateway: Arc, +} + +impl Drop for Registration { + fn drop(&mut self) { + self.gateway.accepting.store(false, Ordering::Release); + } } pub(super) struct Gateway { + accepting: AtomicBool, runtime: CellGroupRuntime, sql: CellGroupSqlPool, dispatcher: CellGroupDispatcher, @@ -87,6 +100,8 @@ impl RaftTransport { /// pool. Options provide the namespace's immutable-query services and cache; /// cell images are selected only by fresh runtime authority. This grants no /// origin permission. Retain the returned owner while serving requests. + /// Dropping its final clone refuses new reads; already admitted reads retain + /// their physical owners until completion or joined shutdown. pub fn install_cell_query_gateway( &self, dispatcher: CellGroupDispatcher, @@ -116,13 +131,16 @@ impl RaftTransport { return Err(TransportError::Overloaded); } let inner = Arc::new(Gateway { + accepting: AtomicBool::new(true), runtime: runtime.clone(), sql, dispatcher, options, }); gateways.insert(key, Arc::downgrade(&inner)); - Ok(CellQueryGateway { inner }) + Ok(CellQueryGateway { + inner: Arc::new(Registration { gateway: inner }), + }) } pub(crate) fn uses_query_domain(&self, domain: &DomainToken) -> bool { @@ -229,7 +247,7 @@ impl RaftTransport { impl CellQueryGateway { /// The cluster and namespace bound to this host-owned registration. pub fn namespace(&self) -> (ClusterId, NamespaceId) { - self.inner.dispatcher.namespace() + self.inner.gateway.dispatcher.namespace() } } @@ -412,6 +430,9 @@ impl IncomingCall { .get(&(cell.cluster, cell.namespace)) .and_then(|h| h.upgrade()) .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } let result = gateway .execute(call, target.scope, cell, route.schema()) .await diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs index 1a00d738f..08e0b5faf 100644 --- a/crates/cellar/tests/support/cell_group_remote_queries.rs +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -347,10 +347,13 @@ pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { let entered = Mutex::new(Some(entered)); let (release, held) = std::sync::mpsc::sync_channel(1); let held = Mutex::new(held); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( "Get", [82; 32], move |context, _| { + count.fetch_add(1, Ordering::SeqCst); entered .lock() .unwrap() @@ -367,7 +370,7 @@ pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { )?); let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let gateways = gateways(&fixture, class.clone()).await?; + let mut gateways = gateways(&fixture, class.clone()).await?; let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; let timeout = if expire { Duration::from_millis(600) @@ -392,6 +395,26 @@ pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { tokio::time::timeout(Duration::from_secs(5), started).await??; let pool = fixture.appliers[&(0, route.history)].0.clone(); assert_eq!(pool.usage()?.retained_operations, 1); + assert_eq!(calls.load(Ordering::SeqCst), 1); + // The in-flight read retains Gateway data, but cannot retain its public + // admission owner and perpetuate serving after the host drops it. + drop(gateways.remove(0)); + let mut late = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(late + .execute_remote(&fixture.transports[2], MemberId::new(1)?) + .await + .is_err()); + drop(late); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(pool.usage()?.retained_operations, 1); // The blocked query does not occupy control connections or Raft work. let deadline = fixture .cluster From da79a69604d58be1c7d9663ebb69c3f6ded173f6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 07:11:22 -0700 Subject: [PATCH 097/568] fix: assign query sequences in peer send order --- .../durability/raft/query_cell_transport.rs | 15 +-------- .../cellar/durability/raft/query_transport.rs | 30 ++++++++--------- .../support/cell_group_remote_queries.rs | 33 +++++++++++++++++++ 3 files changed, 47 insertions(+), 31 deletions(-) diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index e1d3b27d5..c78734937 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -79,7 +79,6 @@ pub(super) struct SendCellQuery { cell: CellHistory, call: CellQueryCall, keys: VersionKeys, - sequence: u64, deadline: u64, cancel: watch::Receiver, reply: oneshot::Sender>, @@ -214,11 +213,6 @@ impl RaftTransport { if !state.current_physical(&view, &peer) { return Err(TransportError::StalePeer); } - let sequence = state - .sequence - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) - .map_err(|_| TransportError::Closed)? - + 1; let (reply, result) = oneshot::channel(); let (cancel, cancelled) = watch::channel(false); let _cancel = Cancel(cancel); @@ -229,7 +223,6 @@ impl RaftTransport { cell: cell.clone(), call, keys, - sequence, deadline, cancel: cancelled, reply, @@ -326,14 +319,8 @@ impl State { &request.cell, request.route.schema_digest, )?; - let header = wire::RequestHead::new(route, request.sequence, &body); let body = self - .exchange_query( - connection, - &RaftPeerNode::from(&request.peer), - &header, - body, - ) + .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) .await?; if !self.current_physical(&request.route, &request.peer) { return Err(TransportError::StalePeer); diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 9601b7538..f439f7f81 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -47,7 +47,6 @@ pub(super) struct LegacySendQuery { route: Arc, peer: RaftPeer, call: QueryCall, - sequence: u64, deadline: u64, keys: VersionKeys, cancel: watch::Receiver, @@ -339,11 +338,6 @@ impl RaftTransport { { return Err(TransportError::StalePeer); } - let sequence = state - .sequence - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) - .map_err(|_| TransportError::Closed)? - + 1; let (reply, result) = oneshot::channel(); let (cancel, cancelled) = watch::channel(false); pool.query @@ -352,7 +346,6 @@ impl RaftTransport { route, peer, call, - sequence, deadline, keys, cancel: cancelled, @@ -776,14 +769,8 @@ impl State { &request.peer.label, request.route.schema_digest, )?; - let header = wire::RequestHead::new(route, request.sequence, &body); let body = self - .exchange_query( - connection, - &RaftPeerNode::from(&request.peer), - &header, - body, - ) + .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) .await?; if !self.current_query_route(&request.scope, &request.route, &request.peer) { return Err(TransportError::StalePeer); @@ -805,7 +792,7 @@ impl State { &self, connection: &mut Option, peer: &RaftPeerNode, - header: &wire::RequestHead, + route: QueryRoute, body: Vec, ) -> Result, TransportError> { if connection.is_none() { @@ -827,7 +814,16 @@ impl State { *connection = Some((tls, exporter, owner)); } let (stream, exporter, _owner) = connection.as_mut().ok_or(TransportError::Closed)?; - let head = wire::request_head(&self.auth, exporter, header, &body, &self.options)?; + // Assign in this peer's single sender, after dequeueing. Allocating at + // ingress can enqueue concurrent callers in reverse sequence order and + // make the receiver correctly refuse an otherwise valid read as replay. + let sequence = self + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + + 1; + let header = wire::RequestHead::new(route, sequence, &body); + let head = wire::request_head(&self.auth, exporter, &header, &body, &self.options)?; write_head(stream, &head).await?; stream.write_all(&body).await?; stream.flush().await?; @@ -837,7 +833,7 @@ impl State { &response, &self.auth, exporter, - header, + &header, &head.tag, &self.options, )?; diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs index 08e0b5faf..077b1964c 100644 --- a/crates/cellar/tests/support/cell_group_remote_queries.rs +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -191,6 +191,39 @@ pub(crate) async fn remote_queries() -> Result<()> { assert_eq!(response.observation().version.history, *cell); assert_eq!(response.observation().version.sequence.get(), 1); } + // Concurrent ingress shares one FIFO peer lane. Wire sequences must be + // assigned by its sender, not racing callers before channel admission. + for _ in 0..8 { + let barrier = Arc::new(tokio::sync::Barrier::new(2)); + let mut waiters = Vec::new(); + for cell in [&a, &b] { + let cell = cell.clone(); + let source = source.clone(); + let dispatch = dispatch.clone(); + let class = class.clone(); + let barrier = barrier.clone(); + waiters.push(fixture.cluster.domain.spawn(async move { + let mut query = invocation( + &dispatch, + &class, + &cell, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + barrier.wait().await; + let response = query.execute_remote(&source, leader).await?; + assert_eq!(response.result(), cell.cell.as_str().as_bytes()); + assert_eq!(response.observation().version.history, cell); + Ok::<_, anyhow::Error>(()) + })?); + } + for waiter in waiters { + waiter.await??; + } + } + assert_eq!(source.usage().query_items, 0); let before = calls.load(Ordering::SeqCst); let mut wrong = invocation( &dispatch, From 02f7291a44f055bf79e9308c85276bb46acf9bb5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 07:40:43 -0700 Subject: [PATCH 098/568] docs: record qualified shared query RPC checkpoint --- docs/design/multi-raft/cell-groups/README.md | 12 +++- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 28 ++++++-- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 72 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 30 ++++++++ docs/design/multi-raft/cell-groups/tasks.json | 40 ++++++++++- 7 files changed, 178 insertions(+), 10 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 2f180e6ac..78b4bfb4b 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -184,6 +184,16 @@ shared dispatch, including safe receipt routing. Lazy birth/dynamic roster, per-group failure isolation and configuration/cache lifecycle remain gaps. See [progress](progress.md); CG-04 is not complete. +At `da79a696`, attached-peer shared query RPC uses the existing bounded TLS lane +with physical group/logical cell identity and actual cell tokens. Host-owned +registration, original deadlines/retry budgets and physical SQL cleanup remain +bound. The partial gate passes 195 Rust cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/` (260 source inputs, +4 executables). Nonmember shared discovery, bounded destination route caching +(the gateway currently resolves the catalog per RPC), write-authorized mutation +RPC and public SDK cutover remain gaps. See [progress](progress.md); CG-04 remains +IN_PROGRESS and no shared-group RustFS/load qualification is claimed. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent @@ -195,7 +205,7 @@ deliverable, not those implementation or deployment gates. | Artifact | Responsibility | | --- | --- | | `design.md` | Identities, routing, storage, APIs, compatibility and bounded ownership | -| `protocols.md` | Enrollment, apply, snapshot/archive, transfer, hibernation and disaster state machines | +| `protocols.md` | Enrollment, read RPC, apply, snapshot/archive, transfer, hibernation and disaster contracts | | `tasks.json` | Ordered implementation tasks, file ownership, steps, tests and invariant mapping | | `implementation-plan.md` | Human index generated from the same tasks; `show` supplies complete briefs | | `qualification.md`, `profiles.json` | Fault oracles, fixed workload contracts, numerical candidate gates and evidence format | diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index c035e2aaf..85fc2907b 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "f77d0d939347ead7129b39e72f76fcff11b99635", + "source_revision": "da79a69604d58be1c7d9663ebb69c3f6ded173f6", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index a1be9db9d..f1a8c9cb2 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,22 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `f77d0d93`, adding registered, +The newest reviewed source baseline is `da79a696`, adding attached-peer shared +query RPC on the existing TLS lane. Physical group and logical cell identity, +route epoch/birth, actual read authority and signed cell observations remain bound. +Host-owned weak gateway registration checks engine/domain owners; replies retain +admission and cancelled/expired reads retain physical SQL until joined shutdown. +Shared and old-host queries use one read-only grant/revocation guard. The record is +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/checkpoint.json`: 195 Rust cases, +eleven commands, 260 source inputs and 4 executables. Three native cases run +both image modes; three new wire cases and existing transport/host regressions +also pass. Transfer this directory with previous evidence. CG-04 remains +IN_PROGRESS. Nonmember shared discovery, bounded destination route caching and +write-authorized command/enrollment RPC are next integration gaps; the gateway +currently resolves the catalog per RPC. Public SDK cutover, dynamic roster, +per-group failure isolation, configuration/cache lifecycle and RustFS/load remain +unqualified. +The preceding reviewed source baseline is `f77d0d93`, adding registered, catalog-routed local enrollment to the shared controller. Initializer digests bind voting schema; old-host validation refuses this unsupported registration. Creation identity and captured inputs remain owned independently from response lifetime. @@ -427,10 +442,13 @@ Do not let plan maintenance replace implementation. provide safe leader receipt lookup when a follower has no original producer. Missing local state never proves non-commit. Enrollment still requires an attached group in a fixed roster: integrate lazy birth and dynamic registration. - Reuse the separate query connection in `query_transport.rs`, replacing its - old cell-scoped route/result binding with physical group and logical cell - identity. Keep existing namespace query-origin grants read-only; mutation RPC - needs explicit write authorization and must not inherit those grants. + Attached-peer shared reads now reuse the separate query connection with + physical group/logical cell binding and host-owned gateway registration. + Complete nonmember shared discovery and real-TLS namespace-grant revocation + qualification. Replace the gateway's per-RPC catalog lookup with bounded, + correctly invalidated routing state before claiming steady-state performance. + Keep query-origin grants read-only; command/enrollment RPC needs explicit write + authorization, response-independent producers and safe remote receipt lookup. The old host refuses registered initializers until its actual replacement; do not remove that refusal without a functioning shared serving path. Request and namespace S3 minima must continue refusing until CG-06 provides real archive diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 2f35e67eb..787d0bbb9 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 55 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 58 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index b7d816f4c..286b0de94 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,78 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 shared query RPC + +Qualified source: `da79a69604d58be1c7d9663ebb69c3f6ded173f6`. + +Catalog-routed `CellQueryInvocation::execute_remote` now uses the existing +node-owned TLS query lane. The authenticated marker-1 route separates physical +`GroupHistory` from logical `CellHistory`; its payload also binds the catalog +route epoch and birth. There is no reader for the former untagged query route. +The temporary old-host query arm remains explicit until CG-16. Request/reply +channel binding, sequence checks, node/disk/process/member identity and bounded +framing remain shared with the existing transport. + +`RaftTransport::install_cell_query_gateway` registers a namespace dispatcher on +the exact runtime/SQL engine and execution domain. The host owns the registration; +the transport stores a weak reference. A separate admission owner refuses new +reads after its final host handle is dropped, even while an accepted query retains +the gateway data. Destination catalog and coherent runtime +read authority are revalidated before registered SQL runs. Replies carry the +actual selected cell observation and signed cell token. Foreign histories, +incarnations and signing keys cannot produce a successful result. The caller +also enforces its requested AtLeast minimum. + +Both query paths use the same read-only origin grant/revocation guard, peer pools, +connection exchange and byte/item banks. These permissions cannot authorize a +mutation or consensus message. A fully authenticated query refusal keeps the +connection reusable; transport/protocol failures discard it. The single peer +sender assigns wire sequence numbers after dequeueing, so concurrent ingress +cannot invert them and cause false replay refusals. Each attempt is +consumed once; retries and catalog-confirmed redirects retain the original caller +deadline and total attempt budget. Returned replies keep transport admission until +dropped. Disconnect and expiry cancel read waiters without releasing physically +entered SQL or image ownership; controller shutdown still joins those workers. + +Three new native cases each run both materialized and extent images on three +real TLS endpoints in one process. They cover two cells sharing a group, strong/ +stale/AtLeast reads, exact cell observations, wrong tokens/keys/incarnations, +foreign SQL ownership, duplicate/dropped gateway registrations, concurrent-cell +requests, retained responses, +retry budgets and cancellation/deadline shutdown. A fresh linearizable barrier +still completes while a query callback is held. A catalog replacement after route +capture refuses at the old destination before callback execution; a confirmed +redirect cannot invent its new physical owner. That fixture changes metadata only +and does not qualify live movement. Three new wire cases cover bounded parsing, +physical/logical identity and channel binding, token authentication, typed failures +and payload-domain separation. + +All 195 selected Rust cases and all eleven gate commands pass. This includes the +wire tests, full transport-module regressions, shared execution/catalog suites and +existing host suite, plus all-target Clippy for Cellar/cellar-logic/cellar-ltx with +warnings denied, formatting, layout, native dependency isolation, docs, source +preflight and 37 validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-remote-queries-final/checkpoint.json` binds 260 +source inputs and 4 copied executables. Transfer the complete evidence directory +with prior checkpoints. The first diagnostic compile caught a cleanup hook in the +wrong owner; it was fixed before tests. The first committed gate passed Clippy, +eight wire cases and 63 transport cases, then was stopped during native testing +after review found the registration lifetime gap. The separate admission owner +and held-callback assertions fix that gap; this successor repeats the full gate. +A second candidate was stopped during Clippy to move sequence assignment into +send order and add concurrent-cell coverage. All diagnostic and interrupted gates +remain separate from this final qualification. + +CG-04 remains IN_PROGRESS. This checkpoint routes between attached group peers; +nonmember shared discovery and its real-TLS revocation qualification remain open. +The destination currently resolves the catalog per RPC, so bounded route-cache +integration is a critical cost/latency opportunity. Public SDK binding and explicit +write-authorized command/enrollment RPC, including safe follower receipt lookup, +are still missing. Lazy birth/dynamic roster, per-group failure isolation and +configuration/cache lifecycle also remain. Shared-group RustFS recovery, separate +process failure, power loss and load are unqualified; the public host has not cut +over. Continue with these concrete integration gaps before claiming CG-04 DONE. + ## CG-04 controller-owned enrollment Qualified source: `f77d0d939347ead7129b39e72f76fcff11b99635`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index dc7afcdb7..6c72bbcc1 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -84,6 +84,36 @@ authority nor permission to restore the initial image over newer mutations. | Concurrent creators | One committed enrollment wins; mismatch refuses rather than changing class/incarnation. | | Split seals during preparation | Revalidate before proposal; an accepted-before-seal entry belongs to the frozen prefix. | +## Routed read RPC + +Use the existing node-owned query lane, with independent control/data progress, +bounded peer/node admission and joined physical shutdown. Authenticate both the +physical group and logical cell histories, schema, route epoch, birth identity, +destination member and node/disk/process identities. Query origins identify a +node process; a namespace read grant never authorizes a mutation or Raft message. +Grant revocation cancels its in-flight read waiters. A destination still requires +live membership and the actual cell's read authority after routing checks. +Separate host registration admission from data retained by accepted queries. +Dropping the final host registration must refuse new reads even while an earlier +request still owns its gateway data and physical SQL work. + +Allocate authenticated wire sequence numbers in the single peer sender after +dequeueing. Concurrent callers can enqueue in a different order from admission; +that must not cause a valid request to be refused as a replay. +Consume each catalog attempt once. Carry its remaining duration across hosts; +never compare unrelated monotonic clocks or replenish the original caller's +deadline/attempt budget. Confirm relocation hints through the trusted catalog. +Return the selected cell observation and authenticated cell token, never the +group's newer log cut. The caller independently enforces an AtLeast minimum. +Retain result admission until consumption, and retain entered SQL/image owners +through physical cleanup after disconnect, timeout or cancelled shutdown. + +The current shared gateway resolves the destination catalog route for each RPC. +Bounded routing-cache integration remains a CG-04 performance requirement; this +intermediate behavior does not establish steady-state request cost or latency. +Attached-group peer reads are implemented; nonmember shared discovery and +write-authorized command/enrollment RPC remain separate integration work. + ## Apply, checkpoint and archive For entry N, verify the cell base/sequence, stage the result, sync its immutable diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index b0c0b52d2..ef6269c3f 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -503,7 +503,10 @@ "crates/cellar/durability/raft/cell_groups/controller/calls.rs", "crates/cellar/durability/raft/cell_groups/controller/requests.rs", "crates/cellar/application/consensus/class_initializer_tests.rs", - "crates/cellar/tests/support/cell_group_controller_enrollment.rs" + "crates/cellar/tests/support/cell_group_controller_enrollment.rs", + "crates/cellar/durability/raft/query_cell_transport.rs", + "crates/cellar/durability/raft/query_cell_wire.rs", + "crates/cellar/tests/support/cell_group_remote_queries.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -626,6 +629,14 @@ "controller_enrollment_joins_lost_worker_reply", "controller_enrollment_requires_registered_initializer" ] + }, + { + "action": "Route consumed shared query attempts through the existing physical peer query lane. Bind physical history, logical cell, schema, route epoch and birth to the authenticated request; use the same read-only namespace grant/revocation guard as existing queries. Destination catalog and runtime authority remain mandatory. Return signed cell observations while retaining bounded result admission, and preserve original deadlines and attempts across retries/redirects. Cancellation must leave entered SQL physically owned until joined cleanup. Complete nonmember shared discovery and public SDK binding separately; mutation RPC must use explicit write authorization, never query-origin grants. Remove temporary legacy query framing and serving with CG-16. Separate public registration admission lifetime from data retained by accepted queries; the last host registration drop must refuse new requests even while an earlier callback is held. Assign query wire sequence numbers in the single peer sender after dequeueing so concurrent ingress cannot cause a false replay refusal.", + "tests": [ + "shared_remote_queries_bind_cell_versions_and_gateway_owners", + "shared_remote_queries_retain_cancelled_sql_until_join", + "shared_remote_queries_retain_expired_sql_until_join" + ] } ], "test_cases": [ @@ -1154,6 +1165,33 @@ "I03", "I12" ] + }, + { + "name": "shared_remote_queries_bind_cell_versions_and_gateway_owners", + "oracle": "Two cells share one physical group and production query TLS lane in both image modes. Linearizable, stale and AtLeast reads preserve actual cell observations and signed tokens. Retries retain deadlines/attempt limits; unconsumed replies retain transport admission. Foreign tokens, logical incarnations, signing keys, SQL owners and duplicate/missing gateway registrations refuse. A catalog replacement after route capture refuses at the old destination without callbacks; a confirmed redirect cannot invent the new physical owner. Concurrent callers query different cells through the same peer lane without replay refusals.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_queries_retain_cancelled_sql_until_join", + "oracle": "Disconnect a remote query while its compiled SQL callback is physically held, in both image modes. Raft still establishes another linearizable read barrier. Routing/transport reservations release after cancellation, but SQL input/image/work ownership remains charged. Cancelling controller close cannot release the worker before its physical join. Dropping the final host gateway registration while the original callback is held refuses new reads before any additional callback; in-flight data retention cannot extend admission.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_queries_retain_expired_sql_until_join", + "oracle": "Expire the original remote-query deadline while SQL is held, in both image modes. Node/peer query admission drains while the SQL owner remains retained. Raft control progress continues and cancelled shutdown waits for physical cleanup. Dropping the final host gateway registration while the original callback is held refuses new reads before any additional callback; in-flight data retention cannot extend admission.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ From c25f0d97d5c98c6111ea2165b38a11a08613c0ca Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 07:56:02 -0700 Subject: [PATCH 099/568] feat: route nonmember cell queries with revocable read grants --- .../durability/raft/query_cell_discovery.rs | 147 ++++++ .../durability/raft/query_cell_transport.rs | 10 +- .../cellar/durability/raft/query_transport.rs | 6 +- crates/cellar/durability/raft/transport.rs | 18 +- .../durability/raft/transport_directory.rs | 3 + .../durability/raft/transport_resources.rs | 2 +- .../durability/raft/transport_shared.rs | 11 +- .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_query_origins.rs | 462 ++++++++++++++++++ .../support/cell_group_remote_queries.rs | 10 +- .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + crates/cellar/tests/support/cell_group_tcp.rs | 4 +- 13 files changed, 666 insertions(+), 28 deletions(-) create mode 100644 crates/cellar/durability/raft/query_cell_discovery.rs create mode 100644 crates/cellar/tests/support/cell_group_query_origins.rs diff --git a/crates/cellar/durability/raft/query_cell_discovery.rs b/crates/cellar/durability/raft/query_cell_discovery.rs new file mode 100644 index 000000000..dfb2cd721 --- /dev/null +++ b/crates/cellar/durability/raft/query_cell_discovery.rs @@ -0,0 +1,147 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read-only physical discovery; no local RawNode, membership or write authority. +use super::*; +use crate::raft::storage::LogError; + +impl RaftTransport { + /// Install trusted read destinations for a physical group absent from this + /// node. The caller supplies discovery; each destination independently checks + /// namespace permission, live membership and the requested cell read proof. + /// Shares node route/peer bounds with consensus and all other query routes. + pub fn install_cell_query_group( + &self, + route: GroupRoute, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !matches!( + runtime.log().cell_group(&route.scope), + Err(LogError::NotFound) + ) { + return Err(TransportError::Identity); + } + if route.revision == 0 + || route.schema_digest == [0; 32] + || route.peers.is_empty() + || route.peers.len() > 128 + { + return Err(TransportError::Identity); + } + let mut members = BTreeSet::new(); + let mut nodes = BTreeSet::new(); + let physical: Vec<_> = route.peers.iter().map(RaftPeerNode::from).collect(); + for (peer, endpoint) in route.peers.iter().zip(&physical) { + endpoint.validate_endpoint()?; + if peer.replica.scope != route.scope + || peer.replica.node == runtime.log().limits().node + || !members.insert(peer.replica.member) + || !nodes.insert(peer.replica.node) + { + return Err(TransportError::Identity); + } + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if registry.physical_groups.contains_key(&route.scope) { + return Err(TransportError::Identity); + } + for peer in &physical { + registry.check_node_view(peer)?; + } + if let Some(old) = registry.cell_query_groups.get(&route.scope) { + if **old == route { + return Ok(()); + } + if old.revision >= route.revision || old.schema_digest != route.schema_digest { + return Err(TransportError::StalePeer); + } + if route.peers.iter().any(|peer| { + old.peers.iter().any(|previous| { + previous.replica.member == peer.replica.member + && (previous.replica.node != peer.replica.node + || previous.replica.disk != peer.replica.disk) + }) + }) { + return Err(TransportError::Identity); + } + } else if registry.route_count() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + for peer in &physical { + if registry + .physical_groups + .values() + .chain( + registry + .cell_query_groups + .iter() + .filter(|(scope, _)| **scope != route.scope) + .map(|(_, view)| view), + ) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|view| &view.peers) + .any(|other| other.replica.node == peer.node && !peer.same_host(other)) + || registry + .origins + .values() + .chain(registry.inspection_origins.values()) + .filter_map(|grant| grant.peer.as_ref()) + .any(|other| other.node == peer.node && other != peer) + { + return Err(TransportError::StalePeer); + } + } + let starts = state.prepare_pools(&mut registry, &physical)?; + registry + .cell_query_groups + .insert(route.scope, Arc::new(route)); + registry.prune_pools(); + drop(registry); + state.start_pools(starts) + } + + /// Remove read-only discovery, invalidating queued/in-flight route identity + /// and retiring peer pools that no other route or grant retains. + pub fn remove_cell_query_group(&self, history: &GroupHistory) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry.cell_query_groups.remove(history); + registry.prune_pools(); + Ok(()) + } +} + +impl State { + pub(super) fn current_cell_query_route( + &self, + route: &Arc>, + peer: &RaftPeer, + ) -> bool { + let Ok(runtime) = self.runtime.shared() else { + return false; + }; + match runtime.log().cell_group(&route.scope) { + Ok(_) => self.current_physical(route, peer), + Err(LogError::NotFound) => self.registry.lock().is_ok_and(|registry| { + self.admit(®istry).is_ok() + && registry + .cell_query_groups + .get(&route.scope) + .is_some_and(|current| Arc::ptr_eq(current, route)) + && registry + .pools + .get(&peer.replica.node) + .is_some_and(|pool| pool.peer.same_host(peer)) + }), + Err(_) => false, + } + } +} diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index c78734937..ae297981f 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -158,8 +158,7 @@ impl RaftTransport { deadline: u64, ) -> Result { let state = &self.handle.0; - // This checkpoint uses attached groups. Read-only ingress discovery is - // deliberately not implemented by fabricating a local voting identity. + // Discovery never creates a local replica or grants voting authority. state.runtime.shared()?; let (view, peer, pool) = { let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; @@ -167,6 +166,7 @@ impl RaftTransport { let view = registry .physical_groups .get(&route.history) + .or_else(|| registry.cell_query_groups.get(&route.history)) .ok_or(TransportError::UnknownGroup)? .clone(); if view.schema_digest != route.schema_digest || !route.history.same_namespace(cell) { @@ -210,7 +210,7 @@ impl RaftTransport { .acquire(charge) .ok_or(TransportError::Overloaded)?; drop(call.encode(state.options.max_query_request_bytes)?); - if !state.current_physical(&view, &peer) { + if !state.current_cell_query_route(&view, &peer) { return Err(TransportError::StalePeer); } let (reply, result) = oneshot::channel(); @@ -296,7 +296,7 @@ impl State { request.cancel.clone(), None, async { - if !self.current_physical(&request.route, &request.peer) { + if !self.current_cell_query_route(&request.route, &request.peer) { return Err(TransportError::StalePeer); } let remaining = request @@ -322,7 +322,7 @@ impl State { let body = self .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) .await?; - if !self.current_physical(&request.route, &request.peer) { + if !self.current_cell_query_route(&request.route, &request.peer) { return Err(TransportError::StalePeer); } let now = u64::try_from(crate::asyncrt::wall_ms()) diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index f439f7f81..aaffb0d8f 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -212,11 +212,7 @@ impl RaftTransport { if old.revision >= route.revision || old.schema_digest != route.schema_digest { return Err(TransportError::StalePeer); } - } else if registry.query_groups.len() - + registry.physical_groups.len() - + registry.groups.len() - >= state.options.max_groups - { + } else if registry.route_count() >= state.options.max_groups { return Err(TransportError::Overloaded); } for peer in &physical { diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index db78d4dde..d120679be 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -37,6 +37,8 @@ use tokio::sync::{mpsc, oneshot, watch, Notify, Semaphore}; #[path = "query_cell_transport.rs"] mod cell_queries; +#[path = "query_cell_discovery.rs"] +mod cell_query_discovery; #[path = "transport_control_send.rs"] mod control_sender; #[path = "transport_directory.rs"] @@ -284,6 +286,7 @@ struct Registry { groups: BTreeMap>, physical_groups: BTreeMap>>, query_groups: BTreeMap>, + cell_query_groups: BTreeMap>>, origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_peers: BTreeMap>, @@ -294,6 +297,13 @@ struct Registry { } impl Registry { + fn route_count(&self) -> usize { + self.groups.len() + + self.physical_groups.len() + + self.query_groups.len() + + self.cell_query_groups.len() + } + fn prune_pools(&mut self) { let used: BTreeSet<_> = self .groups @@ -303,6 +313,7 @@ impl Registry { .chain( self.physical_groups .values() + .chain(self.cell_query_groups.values()) .flat_map(|route| route.peers.iter().map(|peer| peer.replica.node)), ) .chain( @@ -500,6 +511,7 @@ impl RaftTransport { groups: BTreeMap::new(), physical_groups: BTreeMap::new(), query_groups: BTreeMap::new(), + cell_query_groups: BTreeMap::new(), origins: BTreeMap::new(), inspection_origins: BTreeMap::new(), inspection_peers: BTreeMap::new(), @@ -657,11 +669,7 @@ impl RaftTransport { } } } - } else if registry.groups.len() - + registry.physical_groups.len() - + registry.query_groups.len() - >= state.options.max_groups - { + } else if registry.route_count() >= state.options.max_groups { return Err(TransportError::Overloaded); } // Existing peers' physical addresses/process must agree across groups. diff --git a/crates/cellar/durability/raft/transport_directory.rs b/crates/cellar/durability/raft/transport_directory.rs index dab5a62d1..b744f1668 100644 --- a/crates/cellar/durability/raft/transport_directory.rs +++ b/crates/cellar/durability/raft/transport_directory.rs @@ -105,6 +105,7 @@ impl RaftTransport { for old in registry .physical_groups .values() + .chain(registry.cell_query_groups.values()) .flat_map(|route| &route.peers) .filter(|old| old.replica.node == peer.node) { @@ -142,6 +143,7 @@ impl RaftTransport { || registry .physical_groups .values() + .chain(registry.cell_query_groups.values()) .any(|route| route.peers.iter().any(|old| old.replica.node == peer.node)) || registry .origins @@ -157,6 +159,7 @@ impl RaftTransport { // budget refuses the refresh without partially changing routing. let starts = state.prepare_pools(&mut registry, physical)?; refresh_routes(&mut registry.physical_groups, peer); + refresh_routes(&mut registry.cell_query_groups, peer); refresh_routes(&mut registry.groups, peer); refresh_routes(&mut registry.query_groups, peer); let Registry { diff --git a/crates/cellar/durability/raft/transport_resources.rs b/crates/cellar/durability/raft/transport_resources.rs index 17fa0f033..05c039f86 100644 --- a/crates/cellar/durability/raft/transport_resources.rs +++ b/crates/cellar/durability/raft/transport_resources.rs @@ -101,7 +101,7 @@ impl ResourceObserver { registry.tasks, registry.pools.len(), registry.groups.len() + registry.physical_groups.len(), - registry.query_groups.len(), + registry.query_groups.len() + registry.cell_query_groups.len(), connections, limits, ) diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 9fb7ecfc9..23ec46285 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -106,6 +106,9 @@ impl RaftTransport { for peer in &route.peers { registry.check_node_view(&RaftPeerNode::from(peer))?; } + if registry.cell_query_groups.contains_key(&route.scope) { + return Err(TransportError::Identity); + } if let Some(old) = registry.physical_groups.get(&route.scope) { if **old == route { return Ok(()); @@ -122,11 +125,7 @@ impl RaftTransport { return Err(TransportError::Identity); } } - } else if registry.groups.len() - + registry.query_groups.len() - + registry.physical_groups.len() - >= state.options.max_groups - { + } else if registry.route_count() >= state.options.max_groups { return Err(TransportError::Overloaded); } let physical: Vec<_> = route @@ -139,6 +138,7 @@ impl RaftTransport { if registry .physical_groups .values() + .chain(registry.cell_query_groups.values()) .filter(|view| view.scope != route.scope) .flat_map(|view| &view.peers) .any(|other| other.replica.node == peer.node && !peer.same_host(other)) @@ -356,6 +356,7 @@ impl Registry { if self .physical_groups .values() + .chain(self.cell_query_groups.values()) .flat_map(|route| &route.peers) .any(|other| other.replica.node == peer.node && !peer.same_host(other)) { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 1279966e3..e9399d064 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1475,3 +1475,18 @@ async fn shared_remote_queries_retain_cancelled_sql_until_join() -> Result<()> { async fn shared_remote_queries_retain_expired_sql_until_join() -> Result<()> { Box::pin(runtime_fixture::remote_query_lifecycle(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_queries_require_grants_without_local_replicas() -> Result<()> { + Box::pin(runtime_fixture::nonmember_queries()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_query_revocation_retains_physical_sql() -> Result<()> { + Box::pin(runtime_fixture::nonmember_query_revocation(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_nonmember_query_route_replacement_refuses_stale_reply() -> Result<()> { + Box::pin(runtime_fixture::nonmember_query_revocation(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_query_origins.rs b/crates/cellar/tests/support/cell_group_query_origins.rs new file mode 100644 index 000000000..2b70d1b1c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_query_origins.rs @@ -0,0 +1,462 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A fourth TLS process identity routes reads without any local group or voter. +use super::remote_queries::{dispatcher, gateways, invocation, keys, wait_for}; +use super::*; +use cellar::peer_auth::PeerAuth; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; +use tokio::net::TcpListener; + +struct Router { + log: ReplicaLog, + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + transport: RaftTransport, + peer: RaftPeerNode, +} + +impl Router { + async fn start(fixture: &RuntimeCluster) -> Result { + let node = NodeId::new([4; 16])?; + let disk = DiskLineage::new([14; 16])?; + let process = ProcessGeneration::new([20; 16])?; + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("router-log"), + node, + disk, + process, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([17; 32], "node-4".to_owned())?), + super::super::tcp::tls()?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TransportOptions { + max_groups: 2, + ..Default::default() + }, + ) + .await?; + let peer = RaftPeerNode { + node, + disk, + process, + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }; + Ok(Self { + log, + runtime, + events, + transport, + peer, + }) + } + + fn grant(&self, fixture: &RuntimeCluster, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + for transport in &fixture.transports { + transport.grant_query_origin(NamespaceQueryOrigin { + cluster: history.cluster, + namespace: history.namespace, + peer: self.peer.clone(), + revision, + })?; + } + Ok(()) + } + + async fn close(self) -> Result<()> { + self.transport.close().await; + self.runtime.close().await?; + drop(self.events); + self.log.close().await?; + Ok(()) + } +} + +pub(crate) async fn nonmember_queries() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, _| { + count.fetch_add(1, Ordering::SeqCst); + assert_eq!( + ctx.query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))?, + 0 + ); + Ok(ctx.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, class.clone()).await?; + let router = Router::start(&fixture).await?; + let discovery = fixture.tcp_route(route.history)?; + router + .transport + .install_cell_query_group(discovery.clone())?; + router + .transport + .install_cell_query_group(discovery.clone())?; + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(router.transport.usage().groups, 0); + assert_eq!(router.transport.usage().peers, 3); + assert!(router.runtime.status(route.history).await.is_err()); + let mut heartbeat = Message { + from: 1, + to: 2, + ..Default::default() + }; + heartbeat.set_msg_type(MessageType::MsgHeartbeat); + assert!(matches!( + router.transport.submit_physical(&route.history, heartbeat), + Err(TransportError::UnknownGroup) + )); + assert!(matches!( + fixture.transports[0].install_cell_query_group(discovery.clone()), + Err(TransportError::Identity) + )); + assert!(matches!( + router.transport.install_physical_group(discovery.clone()), + Err(TransportError::UnknownGroup) + )); + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + // Discovery and a grant in another namespace cannot authorize this read. + fixture.transports[0].grant_query_origin(NamespaceQueryOrigin { + cluster: a.cluster, + namespace: NamespaceId::new([199; 16])?, + revision: 1, + peer: router.peer.clone(), + })?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut wrong_process = router.peer.clone(); + wrong_process.process = ProcessGeneration::new([195; 16])?; + // Use a different destination: its namespace grant must bind the actual + // authenticated source process, not merely the router's stable node ID. + fixture.transports[1].grant_query_origin(NamespaceQueryOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: wrong_process, + })?; + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 0); + router.grant(&fixture, 2)?; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let first_attempt = query.attempt().context("first strong attempt")?.clone(); + assert!(matches!( + query + .execute_remote(&router.transport, MemberId::new(2)?) + .await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::VersionUnavailable + ))) + )); + assert_eq!(calls.load(Ordering::SeqCst), 0); + query.retry().await?; + assert_eq!( + query.attempt().context("leader retry")?.deadline_ms(), + first_attempt.deadline_ms() + ); + assert_eq!(query.attempt().context("leader retry")?.ordinal(), 2); + let reply = query + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(reply.result(), a.cell.as_str().as_bytes()); + assert_eq!(reply.observation().version.history, a); + let token = reply.token().clone(); + assert_eq!(router.transport.usage().query_items, 1); + drop(reply); + drop(query); + for (cell, consistency) in [ + (&a, CellQueryConsistency::AtLeast(token)), + (&b, CellQueryConsistency::Stale), + ] { + let mut query = invocation( + &dispatch, + &class, + cell, + consistency, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + query + .execute_remote(&router.transport, MemberId::new(3)?) + .await? + .result(), + cell.cell.as_str().as_bytes() + ); + } + assert_eq!(calls.load(Ordering::SeqCst), 3); + // All discovery routes share the same three physical peer pools and + // bounded route admission. They never allocate local group storage. + let mut second = discovery.clone(); + second.scope.group = GroupId::new([198; 16])?; + for peer in &mut second.peers { + peer.replica.scope = second.scope; + } + router.transport.install_cell_query_group(second.clone())?; + assert_eq!(router.transport.usage().peers, 3); + let mut third = second.clone(); + third.scope.group = GroupId::new([197; 16])?; + for peer in &mut third.peers { + peer.replica.scope = third.scope; + } + assert!(matches!( + router.transport.install_cell_query_group(third), + Err(TransportError::Overloaded) + )); + let mut changed = discovery.clone(); + changed.revision = 2; + changed.peers[0].replica.process = ProcessGeneration::new([196; 16])?; + assert!(matches!( + router.transport.install_cell_query_group(changed), + Err(TransportError::StalePeer) + )); + let mut changed = discovery.clone(); + changed.revision = 2; + router.transport.install_cell_query_group(changed)?; + assert!(matches!( + router.transport.install_cell_query_group(discovery), + Err(TransportError::StalePeer) + )); + router.transport.remove_cell_query_group(&second.scope)?; + assert_eq!(router.transport.usage().peers, 3); + router.transport.remove_cell_query_group(&route.history)?; + assert_eq!(router.transport.usage().peers, 0); + let mut removed = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + removed + .execute_remote(&router.transport, MemberId::new(1)?) + .await, + Err(CellDispatchError::Transport(TransportError::UnknownGroup)) + )); + drop(removed); + assert_eq!(dispatch.usage(), (0, 0)); + assert!(router.log.cell_groups()?.is_empty()); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + drop(owners); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn nonmember_query_revocation(replace_route: bool) -> Result<()> { + for rooted in [false, true] { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |ctx, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 { + entered + .lock() + .unwrap() + .take() + .context("once")? + .send(()) + .map_err(|_| anyhow::anyhow!("observer"))?; + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + Ok(ctx.identity().cell.as_str().as_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let owners = gateways(&fixture, class.clone()).await?; + let router = Router::start(&fixture).await?; + let mut discovery = fixture.tcp_route(route.history)?; + router + .transport + .install_cell_query_group(discovery.clone())?; + router.grant(&fixture, 1)?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut query = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(10), + b"", + ) + .await?; + let source = router.transport.clone(); + let waiter = fixture.cluster.domain.spawn(async move { + query + .execute_remote(&source, MemberId::new(1).unwrap()) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.retained_operations, 1); + if replace_route { + discovery.revision += 1; + router.transport.install_cell_query_group(discovery)?; + // Even a complete authenticated reply cannot escape with a captured + // discovery view superseded while the remote SQL callback ran. + release.send(())?; + assert!(matches!( + waiter.await?, + Err(CellDispatchError::Transport(TransportError::StalePeer)) + )); + } else { + fixture.transports[0].revoke_query_origin( + a.cluster, + a.namespace, + router.peer.node, + 2, + )?; + assert!(tokio::time::timeout(Duration::from_secs(5), waiter) + .await?? + .is_err()); + wait_for(async || { + Ok(router.transport.usage().query_items == 0 + && fixture.transports[0].usage().query_items == 0) + }) + .await?; + assert_eq!(pool.usage()?.retained_operations, 1); + assert!(router.grant(&fixture, 1).is_err()); + let mut denied = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(denied); + assert_eq!(calls.load(Ordering::SeqCst), 1); + release.send(())?; + router.grant(&fixture, 3)?; + } + wait_for(async || Ok(pool.usage()?.retained_operations == 0)).await?; + let mut fresh = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + fresh + .execute_remote(&router.transport, MemberId::new(1)?) + .await? + .result(), + a.cell.as_str().as_bytes() + ); + drop(fresh); + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(dispatch.usage(), (0, 0)); + assert!(router.log.cell_groups()?.is_empty()); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + drop(owners); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs index 077b1964c..06a9c75a8 100644 --- a/crates/cellar/tests/support/cell_group_remote_queries.rs +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -7,7 +7,7 @@ use std::sync::{ Mutex, }; -fn dispatcher( +pub(super) fn dispatcher( fixture: &RuntimeCluster, class: Arc, keys: VersionKeys, @@ -25,7 +25,7 @@ fn dispatcher( )?) } -fn keys(tag: u8) -> Result { +pub(super) fn keys(tag: u8) -> Result { Ok(VersionKeys::new( VersionSigningKey::new([tag; 32], u64::MAX)?, vec![], @@ -33,7 +33,7 @@ fn keys(tag: u8) -> Result { )?) } -async fn invocation( +pub(super) async fn invocation( dispatcher: &CellGroupDispatcher, class: &RaftClass, cell: &CellHistory, @@ -59,7 +59,7 @@ async fn invocation( Ok(query) } -async fn gateways( +pub(super) async fn gateways( fixture: &RuntimeCluster, class: Arc, ) -> Result> { @@ -76,7 +76,7 @@ async fn gateways( Ok(gateways) } -async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { +pub(super) async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { tokio::time::timeout(Duration::from_secs(10), async { while !predicate().await? { tokio::time::sleep(Duration::from_millis(2)).await; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 3e9d3e830..9a283e691 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -707,3 +707,5 @@ pub(super) use sql::{ }; pub(super) use sql::{remote_queries, remote_query_lifecycle}; + +pub(super) use sql::{nonmember_queries, nonmember_query_revocation}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 3e915eedd..2397d91e7 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -548,3 +548,7 @@ pub(crate) use controller_enrollment::{ #[path = "cell_group_remote_queries.rs"] mod remote_queries; pub(crate) use remote_queries::{remote_queries, remote_query_lifecycle}; + +#[path = "cell_group_query_origins.rs"] +mod query_origins; +pub(crate) use query_origins::{nonmember_queries, nonmember_query_revocation}; diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs index a56458156..565265d74 100644 --- a/crates/cellar/tests/support/cell_group_tcp.rs +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -6,7 +6,7 @@ use cellar::application::consensus::verification::GroupRoute as TransportRoute; use cellar::peer_auth::PeerAuth; use tokio::net::TcpListener; -fn tls() -> Result { +pub(super) fn tls() -> Result { Ok(RaftTls::new( vec![include_bytes!("../../transport/tcp/fixtures/ca.der") .to_vec() @@ -53,7 +53,7 @@ impl RuntimeCluster { Ok(()) } - fn tcp_route(&self, history: GroupHistory) -> Result> { + pub(super) fn tcp_route(&self, history: GroupHistory) -> Result> { let marker = self.cluster.voters[0] .log .cell_group(&history)? From e0be9f58b2bf25aed44178ab58c89c11b7dc5321 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 08:15:32 -0700 Subject: [PATCH 100/568] docs: record qualified nonmember query checkpoint --- docs/design/multi-raft/cell-groups/README.md | 9 ++++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 19 +++++-- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 52 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 12 ++++- docs/design/multi-raft/cell-groups/tasks.json | 39 +++++++++++++- 7 files changed, 128 insertions(+), 9 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 78b4bfb4b..a0fab0782 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -194,6 +194,15 @@ bound. The partial gate passes 195 Rust cases and eleven commands; transfer RPC and public SDK cutover remain gaps. See [progress](progress.md); CG-04 remains IN_PROGRESS and no shared-group RustFS/load qualification is claimed. +At `c25f0d97`, explicit nonmember read discovery uses that same query lane without +local group storage or voting identity. Four-endpoint TLS fixtures cover read +permission/process binding, revocation during held SQL, stale discovery replies, +route bounds and cleanup in both image modes. The partial gate passes 198 Rust +cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/` (262 source inputs, +4 executables). Bounded destination caching, write-authorized RPC and public-host +automatic discovery/SDK binding remain open. CG-04 remains IN_PROGRESS. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 85fc2907b..35b59a422 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "da79a69604d58be1c7d9663ebb69c3f6ded173f6", + "source_revision": "c25f0d97d5c98c6111ea2165b38a11a08613c0ca", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index f1a8c9cb2..1eaa1edf6 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,19 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `da79a696`, adding attached-peer shared +The newest reviewed source baseline is `c25f0d97`, adding explicit nonmember +read discovery on the shared query lane. Discovery allocates no local replica or +voting identity; namespace/process grants remain independent. New TLS fixtures +cover both image modes, foreign/missing grants, original-budget leader retries, +held-SQL revocation, stale grant revisions, late replies under replaced discovery, +route/peer bounds and cleanup. The record is +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/checkpoint.json`: 198 Rust cases, +eleven commands, 262 source inputs and 4 executables. Transfer this directory +with prior evidence. CG-04 remains IN_PROGRESS. Next address bounded destination +routing-cache integration, write-authorized mutation/enrollment RPC and safe +receipt lookup. Public-host automatic discovery and SDK binding, dynamic roster, +per-group failure isolation and RustFS/load qualification remain incomplete. +The preceding reviewed source baseline is `da79a696`, adding attached-peer shared query RPC on the existing TLS lane. Physical group and logical cell identity, route epoch/birth, actual read authority and signed cell observations remain bound. Host-owned weak gateway registration checks engine/domain owners; replies retain @@ -444,8 +456,9 @@ Do not let plan maintenance replace implementation. attached group in a fixed roster: integrate lazy birth and dynamic registration. Attached-peer shared reads now reuse the separate query connection with physical group/logical cell binding and host-owned gateway registration. - Complete nonmember shared discovery and real-TLS namespace-grant revocation - qualification. Replace the gateway's per-RPC catalog lookup with bounded, + Explicit nonmember shared discovery and real-TLS grant revocation now pass + scoped tests; public-host automatic discovery/SDK binding remain pending. + Replace the gateway's per-RPC catalog lookup with bounded, correctly invalidated routing state before claiming steady-state performance. Keep query-origin grants read-only; command/enrollment RPC needs explicit write authorization, response-independent producers and safe remote receipt lookup. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 787d0bbb9..69c7e1467 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 58 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 61 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 286b0de94..6ebacabd0 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 nonmember read discovery + +Qualified source: `c25f0d97d5c98c6111ea2165b38a11a08613c0ca`. + +`RaftTransport::install_cell_query_group` installs explicit trusted discovery for +physical groups absent from the router. It creates no local group storage, +RawNode, voting identity or mutation permission. The shared query path uses that +view or an attached group's view; destination namespace grants, live membership, +catalog binding and actual read authority are still required. Removing discovery +invalidates captured routes and retires unused peer pools; it does not revoke the +destination's independently managed read grant. + +Read-only routes share the node's route cap, endpoint consistency checks and peer +pools with consensus and other queries. They participate in process-view refresh +and router-route resource accounting. Route revisions cannot move backwards or +change schema, and existing member identities cannot silently change node/disk. +The sender revalidates captured route identity before sending and after receiving; +a late authenticated reply under a replaced route cannot become a success. + +Three new native cases run both materialized and extent images over four real TLS +endpoints in one OS process. The fourth node has no stored cell groups or attached +Raft owner before or after reads. Discovery alone, a foreign-namespace grant and a +wrong-process grant refuse before SQL. Explicit permission permits leader strong +reads and follower stale/AtLeast reads. The current strong-read contract refuses +a follower; the test verifies that retrying the leader preserves the original +invocation deadline and attempt budget. Discovery shares three physical pools +across multiple routes and enforces its configured capacity. + +Revocation while a callback is held ends the remote read waiter and drains +transport reservations while physical SQL remains owned. A stale grant revision +cannot reopen admission; a newer explicit grant permits a fresh read after cleanup. +Replacing discovery while SQL is held refuses the old reply and permits a fresh +attempt. Neither mechanism grants consensus authority or creates a router replica. + +All 198 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The source-bound record at +`.cellar/cell-groups-evidence/CG-04-nonmember-queries/checkpoint.json` contains 262 +source inputs and 4 copied executables. Transfer that complete directory with +prior checkpoints. Diagnostic targeted runs are retained separately: the first +used a local-only default Raft message before the intended absent-group assertion; +the fixture now sends a valid heartbeat. The next assumed follower strong reads; +the corrected fixture asserts the refusal and preserved-budget leader retry. + +CG-04 remains IN_PROGRESS. Discovery is explicitly installed by a trusted caller; +public-host automatic discovery and SDK binding are still missing. The destination +still resolves the catalog per RPC: bounded, correctly invalidated routing state +is the next cost/latency opportunity. Write-authorized command/enrollment RPC and +safe receipt lookup, lazy birth/dynamic roster, per-group failure isolation and +configuration/cache lifecycle remain. Complete-group snapshots, shared RustFS +recovery, independent-process failures, power loss and load remain unqualified. + ## CG-04 shared query RPC Qualified source: `da79a69604d58be1c7d9663ebb69c3f6ded173f6`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 6c72bbcc1..4fedb57e0 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -111,8 +111,16 @@ through physical cleanup after disconnect, timeout or cancelled shutdown. The current shared gateway resolves the destination catalog route for each RPC. Bounded routing-cache integration remains a CG-04 performance requirement; this intermediate behavior does not establish steady-state request cost or latency. -Attached-group peer reads are implemented; nonmember shared discovery and -write-authorized command/enrollment RPC remain separate integration work. +Attached-peer and explicitly discovered nonmember reads share the same query +lane. Nonmember discovery creates no local replica and grants no voting or write +authority. It consumes the same bounded route/peer registries, participates in +node-process refresh and endpoint consistency, and releases unused pools when +removed. A replaced discovery view refuses captured requests and late replies; +removal does not revoke the destination's independently managed namespace grant. +Grant revocation cancels read waiters without releasing physically entered SQL. +The public host must supply trusted discovery and explicit permissions; automatic +discovery/SDK binding and write-authorized command/enrollment RPC remain separate +integration work. ## Apply, checkpoint and archive diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index ef6269c3f..77215fb24 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -506,7 +506,9 @@ "crates/cellar/tests/support/cell_group_controller_enrollment.rs", "crates/cellar/durability/raft/query_cell_transport.rs", "crates/cellar/durability/raft/query_cell_wire.rs", - "crates/cellar/tests/support/cell_group_remote_queries.rs" + "crates/cellar/tests/support/cell_group_remote_queries.rs", + "crates/cellar/durability/raft/query_cell_discovery.rs", + "crates/cellar/tests/support/cell_group_query_origins.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -637,6 +639,14 @@ "shared_remote_queries_retain_cancelled_sql_until_join", "shared_remote_queries_retain_expired_sql_until_join" ] + }, + { + "action": "Install bounded physical read discovery for nonmember routers without creating a local group or borrowing voting authority. Include it in shared peer-pool lifetime, endpoint consistency, node-process refresh and route admission/observability. Revalidate captured route identity before sending and after receiving. Destination namespace grants remain independently authenticated, revocable and strictly read-only. Qualify absent, foreign-namespace and wrong-process grants and held-callback revocation over real TLS, preserving physical SQL ownership. Integrate trusted discovery publication and selection into the public host in CG-07; this API does not authorize automatic membership or mutation RPC.", + "tests": [ + "shared_nonmember_queries_require_grants_without_local_replicas", + "shared_nonmember_query_revocation_retains_physical_sql", + "shared_nonmember_query_route_replacement_refuses_stale_reply" + ] } ], "test_cases": [ @@ -1192,6 +1202,33 @@ "I05", "I12" ] + }, + { + "name": "shared_nonmember_queries_require_grants_without_local_replicas", + "oracle": "A fourth real TLS endpoint queries two cells in both image modes without local group storage or a RawNode. Discovery alone and wrong-namespace/process grants refuse before SQL. Explicit grants permit leader strong reads and follower stale/AtLeast reads with authenticated logical observations. A strong attempt at a follower refuses before SQL; retrying the leader preserves the original deadline and total attempt budget. Consensus submission remains refused. Read discovery shares route/peer limits, rejects conflicting endpoints and stale revisions, and releases unused pools on removal.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_nonmember_query_revocation_retains_physical_sql", + "oracle": "Revoke the nonmember namespace grant while its remote SQL callback is physically held, in both image modes. The waiter refuses and transport reservations drain while physical SQL remains owned. Stale grant revision cannot reopen admission and new requests cannot enter SQL. A newer explicit grant permits reads after physical cleanup; no local replica appears.", + "invariants": [ + "I01", + "I05", + "I12" + ] + }, + { + "name": "shared_nonmember_query_route_replacement_refuses_stale_reply", + "oracle": "Replace the router discovery revision while remote SQL is held, in both image modes. The complete authenticated reply under the captured old route is refused as stale. A new attempt under the replacement succeeds, without leaking invocation/transport ownership or creating a router replica.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ From c41d85657dc2f29040f885702fbf71b013e77042 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 08:34:19 -0700 Subject: [PATCH 101/568] feat: share bounded cell-group route hints across dispatch --- crates/cellar/durability/raft/catalog.rs | 5 +- .../raft/cell_groups/directory/mod.rs | 3 + .../raft/cell_groups/directory/route_cache.rs | 145 ++++++++++++ .../raft/cell_groups/directory/routing.rs | 42 +++- .../raft/cell_groups/dispatch/command/mod.rs | 3 +- .../raft/cell_groups/dispatch/enrollment.rs | 3 +- .../raft/cell_groups/dispatch/mod.rs | 56 ++++- .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../durability/raft/query_cell_transport.rs | 2 +- .../tests/native_cell_groups_catalog.rs | 15 ++ .../tests/native_cell_groups_execution.rs | 13 +- .../support/cell_group_controller_commands.rs | 4 + .../cell_group_controller_enrollment.rs | 2 + .../tests/support/cell_group_directory.rs | 15 +- .../cell_group_dispatch_cache_execution.rs | 174 +++++++++++++++ .../support/cell_group_remote_queries.rs | 2 + .../tests/support/cell_group_routing_cache.rs | 208 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../cell_group_sql_command_dispatch.rs | 2 + .../tests/support/cell_group_sql_dispatch.rs | 2 + 21 files changed, 690 insertions(+), 16 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/route_cache.rs create mode 100644 crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs create mode 100644 crates/cellar/tests/support/cell_group_routing_cache.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index cbd6123f1..4b0846445 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -24,10 +24,11 @@ pub use births::BirthDisposition; #[path = "cell_groups/directory/mod.rs"] mod range_directory; +pub(crate) use range_directory::RouteCache; pub use range_directory::{ CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, - CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, }; diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index efb57a512..5ae9c87b1 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -19,7 +19,10 @@ use std::{ mod birth; mod bootstrap; +mod route_cache; mod routing; +pub use route_cache::CellRouteCacheUsage; +pub(crate) use route_cache::RouteCache; mod store; mod tree; mod wire; diff --git a/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs b/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs new file mode 100644 index 000000000..a385ec7d9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/route_cache.rs @@ -0,0 +1,145 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded namespace range hints. Cached observations never grant authority. +use super::*; +use std::collections::VecDeque; + +// Two bounded routing-key strings plus the route, deque slot and bookkeeping. +// Reserve the full capacity up front; idle/evicted slots cannot hide memory. +const ENTRY_BYTES: usize = 8192; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellRouteCacheUsage { + pub entries: usize, + pub capacity: usize, + pub reserved_bytes: usize, + pub revision: Option, +} + +struct State { + head: Option<(u64, [u8; 32])>, + routes: VecDeque, +} + +pub(crate) struct RouteCache { + directory: RangeDirectory, + capacity: usize, + state: Mutex, +} + +impl RouteCache { + pub(crate) fn new( + directory: RangeDirectory, + entries: usize, + bytes: usize, + ) -> Result { + if !(1..=4096).contains(&entries) || !(ENTRY_BYTES..=64 * 1024 * 1024).contains(&bytes) { + return Err(CatalogError::Limit); + } + let capacity = entries.min(bytes / ENTRY_BYTES); + Ok(Self { + directory, + capacity, + state: Mutex::new(State { + head: None, + routes: VecDeque::with_capacity(capacity), + }), + }) + } + + pub(crate) fn usage(&self) -> Result { + let state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Ok(CellRouteCacheUsage { + entries: state.routes.len(), + capacity: self.capacity, + reserved_bytes: self.capacity * ENTRY_BYTES, + revision: state.head.map(|h| h.0), + }) + } + + pub(super) fn get( + &self, + cell: &CellHistory, + ) -> Result, CatalogError> { + let config = self.directory.namespace_config(); + if cell.cluster != config.cluster || cell.namespace != config.namespace { + return Err(CatalogError::Invalid("routing cache namespace")); + } + let key = cell_routing_key(cell.namespace, &cell.cell); + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + let Some((revision, _)) = state.head else { + return Ok(None); + }; + let Some(index) = state + .routes + .iter() + .position(|route| route.range.contains(&key)) + else { + return Ok(None); + }; + let route = state + .routes + .remove(index) + .ok_or(CatalogError::Corrupt("routing cache entry"))?; + let result = route.clone(); + state.routes.push_back(route); + Ok(Some((result, revision))) + } + + pub(crate) fn observe(&self, view: &DirectoryView) -> Result<(), CatalogError> { + self.directory.check_view(view)?; + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Self::advance(&mut state, view) + } + + fn advance(state: &mut State, view: &DirectoryView) -> Result<(), CatalogError> { + if let Some((revision, digest)) = state.head { + if view.revision() < revision { + return Err(CatalogError::Conflict("superseded routing observation")); + } + if view.revision() == revision { + return if view.digest() == digest { + Ok(()) + } else { + Err(CatalogError::Corrupt("routing revision changed digest")) + }; + } + } + state.routes.clear(); + state.head = Some((view.revision(), view.digest())); + Ok(()) + } + + // Only the routing session calls this after lookup in this exact pinned view. + // An older in-flight lookup cannot repopulate an observed successor root. + pub(super) fn remember( + &self, + view: &DirectoryView, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + self.directory.check_view(view)?; + wire::Route::encode(route, &self.directory)?; + let mut state = self + .state + .lock() + .map_err(|_| CatalogError::Corrupt("routing cache"))?; + Self::advance(&mut state, view)?; + if let Some(index) = state.routes.iter().position(|old| old == route) { + state.routes.remove(index); + } + if state.routes.len() == self.capacity { + state.routes.pop_front(); + } + state.routes.push_back(route.clone()); + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs index ed04eaa01..c8f29473c 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/routing.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -121,6 +121,7 @@ pub struct CellRoutingSession { active: bool, submitted: bool, cached: Option, + route_cache: Option>, owner: Arc<()>, } @@ -193,12 +194,26 @@ impl RangeDirectory { active: false, submitted: false, cached, + route_cache: None, owner: Arc::new(()), }) } } impl CellRoutingSession { + pub(crate) fn using_cache(mut self, cache: Arc) -> Self { + self.route_cache = Some(cache); + self + } + + pub(crate) async fn begin_current(&mut self) -> Result { + if self.attempts != 0 { + return Err(self.error(CellRoutingFailure::StaleAttempt)); + } + self.cached = None; + self.next(None, false).await + } + pub(crate) fn deadline_ms(&self) -> u64 { self.deadline } @@ -255,7 +270,7 @@ impl CellRoutingSession { if self.attempts != 0 { return Err(self.error(CellRoutingFailure::StaleAttempt)); } - self.next(None).await + self.next(None, true).await } /// Same-group retry, including a different member, under the same budget. @@ -264,7 +279,7 @@ impl CellRoutingSession { previous: &CellRouteAttempt, ) -> Result { self.check_attempt(previous)?; - self.next(None).await + self.next(None, true).await } /// A relocation hint is untrusted until an exact lookup in the configured @@ -276,12 +291,13 @@ impl CellRoutingSession { hint: &GroupRoute, ) -> Result { self.check_attempt(previous)?; - self.next(Some(hint)).await + self.next(Some(hint), true).await } async fn next( &mut self, hint: Option<&GroupRoute>, + use_cache: bool, ) -> Result { // Consume the previous ticket and attempt before any await. Dropping a // discovery future cannot refund attempts or authorize a late response. @@ -306,6 +322,21 @@ impl CellRoutingSession { return Err(self.error(CellRoutingFailure::InvalidHint)); } } + if use_cache && hint.is_none() && self.cached.is_none() { + if let Some(cache) = &self.route_cache { + if let Some((route, revision)) = cache + .get(&self.request.cell) + .map_err(|e| self.error(CellRoutingFailure::Catalog(e)))? + { + self.cached = Some(ResolvedCellRoute { + cell: self.request.cell.clone(), + route, + revision, + owner: self.directory.namespace.inner.owner.clone(), + }); + } + } + } if self.cached.is_none() || hint.is_some() { let minimum = self.cached.as_ref().map_or(0, |route| route.revision); let work = async { @@ -328,6 +359,11 @@ impl CellRoutingSession { if view.revision() < minimum || hint.is_some_and(|hint| &route != hint) { return Err(self.error(CellRoutingFailure::InvalidHint)); } + if let Some(cache) = &self.route_cache { + cache + .remember(&view, &route) + .map_err(|e| self.error(CellRoutingFailure::Catalog(e)))?; + } self.cached = Some(ResolvedCellRoute { cell: self.request.cell.clone(), route, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index ff9e059f5..0191f7500 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -176,7 +176,8 @@ impl CellGroupDispatcher { cached, started, ) - .map_err(|e| error(CellDispatchError::Routing(e)))?; + .map_err(|e| error(CellDispatchError::Routing(e)))? + .using_cache(self.routes.clone()); Ok(CellCommandInvocation { dispatcher: self.clone(), cell, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index 40c51155b..fee282622 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -127,7 +127,8 @@ impl CellGroupDispatcher { cached, started, ) - .map_err(|e| error(CellDispatchError::Routing(e)))?; + .map_err(|e| error(CellDispatchError::Routing(e)))? + .using_cache(self.routes.clone()); Ok(CellEnrollmentInvocation { dispatcher: self.clone(), cell, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 39e1d849d..8106d17a5 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -28,6 +28,10 @@ use std::{fmt, sync::Arc}; #[derive(Clone, Copy)] pub struct CellDispatchLimits { + /// Maximum retained group-range hints shared by this registration's clones. + pub max_cached_routes: usize, + /// Conservative reserved hint capacity; at least 8 KiB, at most 64 MiB. + pub route_cache_bytes: usize, pub max_invocations: usize, pub retained_bytes: usize, pub max_input_bytes: usize, @@ -77,6 +81,7 @@ pub struct CellGroupDispatcher { domain: DomainToken, limits: CellDispatchLimits, budget: Arc, + routes: Arc, } impl CellGroupDispatcher { @@ -117,7 +122,16 @@ impl CellGroupDispatcher { return Err(CellDispatchError::Invalid("invocation limits")); } let domain = domain.services().domain_token().ok_or_else(closed)?; + let routes = Arc::new( + crate::raft::catalog::RouteCache::new( + directory.clone(), + limits.max_cached_routes, + limits.route_cache_bytes, + ) + .map_err(|_| CellDispatchError::Invalid("route cache limits"))?, + ); Ok(Self { + routes, directory, class, keys, @@ -127,6 +141,24 @@ impl CellGroupDispatcher { }) } + /// Retained namespace range hints, never per-cell execution or authority. + pub fn route_cache_usage(&self) -> Result { + self.routes.usage().map_err(catalog_failure) + } + + /// Observe a verified current directory head and retire all older range hints. + /// This is host-driven metadata refresh, not a serving lease or a timer task. + pub async fn refresh_routes(&self) -> Result { + let view = self + .domain + .run(self.directory.observe()) + .await + .map_err(|_| closed())? + .map_err(catalog_failure)?; + self.routes.observe(&view).map_err(catalog_failure)?; + Ok(view.revision()) + } + pub fn usage(&self) -> (usize, usize) { self.budget.usage() } @@ -180,7 +212,8 @@ impl CellGroupDispatcher { cached, started_at, ) - .map_err(CellDispatchError::Routing)?; + .map_err(CellDispatchError::Routing)? + .using_cache(self.routes.clone()); Ok(CellQueryInvocation { dispatcher: self.clone(), request, @@ -220,6 +253,20 @@ impl CellQueryInvocation { Ok(()) } + // The destination still needs fresh catalog binding until replicated transfer + // seals are integrated. Caller-side hints cannot replace that admission check. + pub(crate) async fn begin_current(&mut self) -> Result<(), CellDispatchError> { + self.dispatched = true; + let attempt = self + .session + .begin_current() + .await + .map_err(CellDispatchError::Routing)?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + pub async fn retry(&mut self) -> Result<(), CellDispatchError> { let previous = self .attempt @@ -483,3 +530,10 @@ fn fingerprint( } hash.finalize().into() } + +fn catalog_failure(error: crate::raft::catalog::CatalogError) -> CellDispatchError { + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Catalog(error), + ambiguous_command: None, + }) +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index edcb5abdd..e4278fad0 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -59,8 +59,8 @@ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsag pub use crate::raft::catalog::{ CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - CellRouteAttempt, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, - CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index ae297981f..fdffe8756 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -449,7 +449,7 @@ impl Gateway { }, None, )?; - query.begin().await?; + query.begin_current().await?; let selected = query .attempt() .ok_or(CellDispatchError::Invalid("missing query route"))? diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index c23654785..35528e81d 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -339,3 +339,18 @@ async fn routing_cached_requests_preserve_identity_without_catalog_reads() -> an async fn routing_attempt_deadline_survives_redirect_and_cancellation() -> anyhow::Result<()> { directory::routing_deadline_and_cancellation().await } + +#[tokio::test] +async fn dispatcher_route_cache_shares_ranges_across_operations() -> anyhow::Result<()> { + directory::cached_operation_discovery().await +} + +#[tokio::test] +async fn dispatcher_route_cache_bounds_and_refreshes_hints() -> anyhow::Result<()> { + directory::bounded_route_hints().await +} + +#[tokio::test] +async fn dispatcher_route_cache_refuses_superseded_inflight_refill() -> anyhow::Result<()> { + directory::stale_cache_refill().await +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index e9399d064..c13bf7bb5 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -230,6 +230,7 @@ struct Cluster { _root: tempfile::TempDir, domain: ExecutionDomain, directory: RangeDirectory, + store: Arc, birth: CellGroupBirth, voters: Vec, network: VecDeque, @@ -275,10 +276,8 @@ impl Cluster { group: GroupId::new([3; 16])?, incarnation: IncarnationId::new([4; 16])?, }; - let catalog = Catalog::new( - Arc::new(object_store::memory::InMemory::new()), - CatalogOptions::default(), - )?; + let store = Arc::new(object_store::memory::InMemory::new()); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; let namespace = domain .run(catalog.open_namespace(NamespaceConfig::new( history.cluster, @@ -372,6 +371,7 @@ impl Cluster { _root: root, domain, directory, + store, birth, voters, network: VecDeque::new(), @@ -1490,3 +1490,8 @@ async fn shared_nonmember_query_revocation_retains_physical_sql() -> Result<()> async fn shared_nonmember_query_route_replacement_refuses_stale_reply() -> Result<()> { Box::pin(runtime_fixture::nonmember_query_revocation(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_cached_dispatch_preserves_authority_during_catalog_outage() -> Result<()> { + Box::pin(runtime_fixture::cached_dispatch_execution()).await +} diff --git a/crates/cellar/tests/support/cell_group_controller_commands.rs b/crates/cellar/tests/support/cell_group_controller_commands.rs index 04d2daa80..ca86dcdfc 100644 --- a/crates/cellar/tests/support/cell_group_controller_commands.rs +++ b/crates/cellar/tests/support/cell_group_controller_commands.rs @@ -100,6 +100,8 @@ pub(crate) async fn owned_commands(phase: u8, mode: u8) -> Result<()> { class.clone(), keys.clone(), CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, max_invocations: 8, retained_bytes: 1024 * 1024, max_input_bytes: 1024, @@ -397,6 +399,8 @@ pub(crate) async fn owned_command_failure(lost_reply: bool) -> Result<()> { Duration::from_secs(30), )?, CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, max_invocations: 8, retained_bytes: 1024 * 1024, max_input_bytes: 1024, diff --git a/crates/cellar/tests/support/cell_group_controller_enrollment.rs b/crates/cellar/tests/support/cell_group_controller_enrollment.rs index 834769ee3..1ab20272f 100644 --- a/crates/cellar/tests/support/cell_group_controller_enrollment.rs +++ b/crates/cellar/tests/support/cell_group_controller_enrollment.rs @@ -51,6 +51,8 @@ fn dispatcher(fixture: &RuntimeCluster, class: Arc) -> Result Result { + Self::with_schema(limits, route().schema_digest).await + } + + async fn with_schema(limits: DirectoryLimits, schema: [u8; 32]) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let store = Arc::new(fault::Store::default()); store.reject_listing.store(true, Ordering::Release); @@ -34,7 +38,7 @@ impl Fixture { route.history.cluster, route.history.namespace, cellar_logic::NamespaceMode::Raft, - route.schema_digest, + schema, ); let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; let namespace = domain.run(catalog.open_namespace(config.clone())).await??; @@ -964,6 +968,9 @@ async fn publish_route_split( ) -> Result { let view = f.run(f.directory.observe()).await?; let mut children = split(parent, 700); + for child in &mut children { + child.schema_digest = parent.schema_digest; + } f.reserve(&mut children).await?; let replace = f.directory.prepare_replacement( &view, @@ -1225,3 +1232,9 @@ pub async fn routing_deadline_and_cancellation() -> Result<()> { )); Ok(()) } + +#[path = "cell_group_routing_cache.rs"] +mod routing_cache; +pub(crate) use routing_cache::{ + bounded_route_hints, cached_operation_discovery, stale_cache_refill, +}; diff --git a/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs new file mode 100644 index 000000000..694acce7e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs @@ -0,0 +1,174 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Known local execution still needs real quorum; hints cannot authorize RPC. +use super::remote_queries::{dispatcher, gateways, invocation, keys}; +use super::*; +use object_store::{path::Path, ObjectStore}; + +pub(crate) async fn cached_dispatch_execution() -> Result<()> { + for rooted in [false, true] { + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([90; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"added".to_vec()) + })? + .query("Get", [83; 32], |context, _| { + Ok(context + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_le_bytes() + .to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let gateways = gateways(&fixture, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let mut warm = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!( + warm.execute(&fixture.runtimes[0], pool, fixture.cell_options(&a).await?) + .await? + .result(), + &0_i64.to_le_bytes() + ); + drop(warm); + assert_eq!(dispatch.route_cache_usage()?.entries, 1); + let head = Path::from(format!( + "namespaces/{}/cell-groups/head", + hex::encode(a.namespace.bytes()) + )); + let saved = fixture.cluster.store.get(&head).await?.bytes().await?; + fixture.cluster.store.delete(&head).await?; + assert!(dispatch.refresh_routes().await.is_err()); + // The range covers B and an as-yet unenrolled C; no per-cell cache entry + // or fresh metadata read is needed for these captured invocations. + let mut other = invocation( + &dispatch, + &class, + &b, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + other + .execute(&fixture.runtimes[0], pool, fixture.cell_options(&b).await?) + .await? + .result(), + &0_i64.to_le_bytes() + ); + drop(other); + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let command = dispatch + .command( + a.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([191; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }, + None, + ) + .await?; + let result = controllers[0] + .command(command, fixture.cell_options(&a).await?)? + .response() + .await?; + assert_eq!(result.receipt().receipt.result(), b"added"); + drop(result); + let c = history(&route, "cache-enrolled")?; + let enrollment = dispatch + .enroll( + c.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([192; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }, + None, + ) + .await?; + let enrolled = controllers[0] + .enroll(enrollment, fixture.cell_options(&c).await?)? + .response() + .await?; + drop(enrolled); + for (cell, expected) in [(&a, 1_i64), (&c, 7_i64)] { + let mut read = invocation( + &dispatch, + &class, + cell, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + assert_eq!( + read.execute( + &fixture.runtimes[0], + pool, + fixture.cell_options(cell).await? + ) + .await? + .result(), + &expected.to_le_bytes() + ); + } + // The RPC destination deliberately keeps fresh catalog admission until + // transfer-seal authority is integrated. Sender hints cannot bypass it. + let mut remote = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Stale, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(matches!( + remote + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await, + Err(CellDispatchError::Transport(TransportError::Query( + CommandFailure::Unavailable + ))) + )); + drop(remote); + assert_eq!(dispatch.route_cache_usage()?.entries, 1); + assert_eq!(dispatch.usage(), (0, 0)); + fixture.cluster.store.put(&head, saved.into()).await?; + for controller in controllers { + controller.close().await?; + } + drop(gateways); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs index 06a9c75a8..7c6b7a6a3 100644 --- a/crates/cellar/tests/support/cell_group_remote_queries.rs +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -18,6 +18,8 @@ pub(super) fn dispatcher( class, keys, CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, max_invocations: 8, retained_bytes: 1024 * 1024, max_input_bytes: 1024, diff --git a/crates/cellar/tests/support/cell_group_routing_cache.rs b/crates/cellar/tests/support/cell_group_routing_cache.rs new file mode 100644 index 000000000..c28d5644c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_routing_cache.rs @@ -0,0 +1,208 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real catalog calls qualify bounded hints, not permission to serve a cell. +use super::*; +use cellar::application::consensus::{RaftClass, VersionKeys, VersionSigningKey}; +use cellar_logic::consensus::{ClientId, CommandId, Durability}; + +async fn setup( + count: usize, + capacity: usize, + bytes: usize, +) -> Result<( + Fixture, + Arc, + CellGroupDispatcher, + Vec, +)> { + let class = Arc::new( + RaftClass::new("Cache", 1, [161; 32])? + .query("Get", [162; 32], |_, _| Ok(vec![]))? + .command("Add", [163; 32], |_, _| Ok(vec![]))?, + ); + let f = Fixture::with_schema( + DirectoryLimits { + max_cached_pages: 0, + max_cache_bytes: 0, + ..Default::default() + }, + class.schema_digest(), + ) + .await?; + let mut routes = ranges(count); + for route in &mut routes { + route.schema_digest = class.schema_digest(); + } + f.reserve(&mut routes).await?; + let mutation = f + .directory + .prepare_initialization(operation(990), &routes)?; + f.run(f.directory.publish(&mutation)).await?; + let dispatcher = CellGroupDispatcher::new( + &f.domain, + f.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([164; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: capacity, + route_cache_bytes: bytes, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + Ok((f, class, dispatcher, routes)) +} + +fn cell(route: &GroupRoute, tag: usize) -> CellHistory { + let cell = (0..100_000) + .map(|i| CellId::new(format!("cache-{tag}-{i}")).unwrap()) + .find(|cell| { + route + .range + .contains(&cell_routing_key(route.history.namespace, cell)) + }) + .unwrap(); + CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell, + incarnation: IncarnationId::new([165; 16]).unwrap(), + } +} + +fn query( + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: CellHistory, +) -> Result { + Ok(dispatcher.query( + cell, + CellQueryRequest { + invocation: operation(991), + contract: class.contract("Get").context("Get")?, + input: vec![], + consistency: CellQueryConsistency::Stale, + }, + routing_limits(4), + None, + )?) +} + +pub(crate) async fn cached_operation_discovery() -> Result<()> { + let (f, class, dispatcher, routes) = setup(1, 2, 16 * 1024).await?; + let first = cell(&routes[0], 0); + let mut warm = query(&dispatcher, &class, first.clone())?; + let (result, reads) = fault::count_reads(warm.begin()).await; + result?; + assert!(!reads.is_empty()); + drop(warm); + f.store.fail_read.store(true, Ordering::Release); + for n in 1..65 { + let mut query = query(&dispatcher.clone(), &class, cell(&routes[0], n))?; + let (result, reads) = fault::count_reads(query.begin()).await; + result?; + assert!(reads.is_empty()); + assert_eq!(query.attempt().context("cached")?.route(), &routes[0]); + } + let mut command = dispatcher + .command( + first, + CellCommandRequest { + id: CommandId::new(ClientId::new([166; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: u64::MAX, + }, + routing_limits(2), + None, + ) + .await?; + let (result, reads) = fault::count_reads(command.begin()).await; + result?; + assert!(reads.is_empty()); + assert_eq!(command.attempt().context("command")?.route(), &routes[0]); + drop(command); + let usage = dispatcher.route_cache_usage()?; + assert_eq!(usage.entries, 1); + assert!(usage.reserved_bytes <= 16 * 1024); + assert!(dispatcher.refresh_routes().await.is_err()); + assert_eq!(dispatcher.route_cache_usage()?, usage); + assert_eq!(dispatcher.usage(), (0, 0)); + f.domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn bounded_route_hints() -> Result<()> { + let (f, class, dispatcher, routes) = setup(4, 64, 16 * 1024).await?; + assert_eq!(dispatcher.route_cache_usage()?.capacity, 2); + for (index, hit) in [(0, false), (1, false), (0, true), (2, false), (1, false)] { + let mut query = query(&dispatcher, &class, cell(&routes[index], index))?; + let (result, reads) = fault::count_reads(query.begin()).await; + result?; + assert_eq!(reads.is_empty(), hit); + assert!(dispatcher.route_cache_usage()?.entries <= 2); + assert!(dispatcher.route_cache_usage()?.reserved_bytes <= 16 * 1024); + } + let selected = cell(&routes[0], 0); + let next = publish_route_split(&f, &routes[0], &selected.cell).await?; + assert_eq!(dispatcher.refresh_routes().await?, 2); + assert_eq!(dispatcher.route_cache_usage()?.entries, 0); + let mut moved = query(&dispatcher, &class, selected)?; + let (result, reads) = fault::count_reads(moved.begin()).await; + result?; + assert!(!reads.is_empty()); + assert_eq!(moved.attempt().context("successor")?.route(), &next); + assert_eq!(dispatcher.route_cache_usage()?.revision, Some(2)); + drop(moved); + f.domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn stale_cache_refill() -> Result<()> { + let (f, class, dispatcher, routes) = setup(1, 2, 16 * 1024).await?; + let selected = cell(&routes[0], 0); + let view = f.run(f.directory.observe()).await?; + let (result, reads) = + fault::count_reads(f.run(f.directory.lookup(&view, &selected.cell))).await; + result?; + let page = reads + .keys() + .find(|path| path.contains("/ranges/")) + .context("range page")? + .clone(); + let gate = Arc::new(fault::RegistryGate::default()); + gate.armed.store(true, Ordering::Release); + let wait_gate = gate.clone(); + let mut old = query(&dispatcher, &class, selected.clone())?; + let waiter = f + .domain + .spawn(async move { fault::gate_read(page, wait_gate, old.begin()).await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let successor = publish_route_split(&f, &routes[0], &selected.cell).await?; + assert_eq!(dispatcher.refresh_routes().await?, 2); + gate.resume.notify_one(); + let error = waiter.await?.err().context("old lookup must refuse")?; + assert!(matches!( + error, + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Catalog(CatalogError::Conflict( + "superseded routing observation" + )), + .. + }) + )); + assert_eq!(dispatcher.route_cache_usage()?.entries, 0); + let mut fresh = query(&dispatcher, &class, selected)?; + fresh.begin().await?; + assert_eq!(fresh.attempt().context("current")?.route(), &successor); + drop(fresh); + assert_eq!(dispatcher.usage(), (0, 0)); + f.domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 9a283e691..45bc77e2f 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -709,3 +709,5 @@ pub(super) use sql::{ pub(super) use sql::{remote_queries, remote_query_lifecycle}; pub(super) use sql::{nonmember_queries, nonmember_query_revocation}; + +pub(super) use sql::cached_dispatch_execution; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 2397d91e7..f7038c969 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -552,3 +552,7 @@ pub(crate) use remote_queries::{remote_queries, remote_query_lifecycle}; #[path = "cell_group_query_origins.rs"] mod query_origins; pub(crate) use query_origins::{nonmember_queries, nonmember_query_revocation}; + +#[path = "cell_group_dispatch_cache_execution.rs"] +mod dispatch_cache_execution; +pub(crate) use dispatch_cache_execution::cached_dispatch_execution; diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs index e4b2f47dd..482c7c8f8 100644 --- a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -93,6 +93,8 @@ pub(crate) async fn commands_with_transport(rooted: bool, tcp: bool) -> Result<( Duration::from_secs(30), )?; let bounds = CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, max_invocations: 3, retained_bytes: 1024 * 1024, max_input_bytes: 1024, diff --git a/crates/cellar/tests/support/cell_group_sql_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_dispatch.rs index 5191c8732..1168aefe1 100644 --- a/crates/cellar/tests/support/cell_group_sql_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_dispatch.rs @@ -88,6 +88,8 @@ pub(crate) async fn routed(rooted: bool) -> Result<()> { Duration::from_secs(10), )?; let bounds = CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, max_invocations: 1, retained_bytes: 1024 * 1024, max_input_bytes: 256 * 1024, From 06652e36869144624e0ca4d860782ef512018791 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 08:37:52 -0700 Subject: [PATCH 102/568] test: use execution-domain timeout for route cache fixture --- crates/cellar/tests/support/cell_group_routing_cache.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_routing_cache.rs b/crates/cellar/tests/support/cell_group_routing_cache.rs index c28d5644c..b5fb034e3 100644 --- a/crates/cellar/tests/support/cell_group_routing_cache.rs +++ b/crates/cellar/tests/support/cell_group_routing_cache.rs @@ -183,7 +183,11 @@ pub(crate) async fn stale_cache_refill() -> Result<()> { let waiter = f .domain .spawn(async move { fault::gate_read(page, wait_gate, old.begin()).await })?; - tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + f.run(cellar::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await?; let successor = publish_route_split(&f, &routes[0], &selected.cell).await?; assert_eq!(dispatcher.refresh_routes().await?, 2); gate.resume.notify_one(); From 40aa18b0b3e36f33741778f4e6e4a98f721f3881 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 08:51:10 -0700 Subject: [PATCH 103/568] docs: record qualified bounded route-cache checkpoint --- docs/design/multi-raft/cell-groups/README.md | 9 ++++ .../multi-raft/cell-groups/baseline.json | 12 ++--- docs/design/multi-raft/cell-groups/handoff.md | 22 +++++++-- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 46 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 15 +++++- docs/design/multi-raft/cell-groups/tasks.json | 24 +++++++++- 7 files changed, 116 insertions(+), 14 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index a0fab0782..c906ba683 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -203,6 +203,15 @@ cases and eleven commands; transfer 4 executables). Bounded destination caching, write-authorized RPC and public-host automatic discovery/SDK binding remain open. CG-04 remains IN_PROGRESS. +At `06652e36`, dispatcher clones share bounded range hints across reads, commands +and enrollment. Verified head changes invalidate hints; superseded lookups cannot +refill them. Three catalog cases and a both-image runtime case qualify eviction, +metadata-read reduction and catalog-outage behavior without weakening destination +admission. The partial gate passes 202 Rust cases and eleven commands; transfer +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/` (265 source inputs, +4 executables). Write-authorized RPC and safe receipt lookup are next; removing +destination catalog reads requires replicated transfer seals. CG-04 stays IN_PROGRESS. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 35b59a422..915be190c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "c25f0d97d5c98c6111ea2165b38a11a08613c0ca", + "source_revision": "06652e36869144624e0ca4d860782ef512018791", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "ee51dcf3b884e9edf39364046054d8aac7769005a2637fef1dd7cb1cb7c3a4d8", - "line": 113, + "sha256": "433421a645cba08f49e8e2332e1ed152034c6752eeb759b2e29e72890b877759", + "line": 114, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "ee51dcf3b884e9edf39364046054d8aac7769005a2637fef1dd7cb1cb7c3a4d8", - "line": 182, + "sha256": "433421a645cba08f49e8e2332e1ed152034c6752eeb759b2e29e72890b877759", + "line": 183, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 1eaa1edf6..e50b96e83 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,21 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `c25f0d97`, adding explicit nonmember +The newest reviewed source baseline is `06652e36`, adding bounded range hints +shared across query, command and enrollment invocations. Entry/byte reservations, +LRU eviction and verified head invalidation bound retention; a superseded in-flight +lookup cannot refill the cache. Three catalog cases measure real metadata reads; +a native case runs both image modes during a catalog outage, retaining local +quorum authority and fresh remote admission. The record is +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/checkpoint.json`: 202 Rust +cases, eleven commands, 265 source inputs and 4 executables. Transfer this +directory with prior evidence. CG-04 remains IN_PROGRESS. Next implement explicit +write-authorized command/enrollment RPC and safe receipt lookup, reusing existing +controller ownership. Destination cache removal depends on replicated transfer +seals/admission invalidation; stale hints must not grant serving authority. +Public-host discovery/SDK binding, dynamic roster, per-group failure isolation, +configuration lifecycle and shared RustFS/load qualification remain incomplete. +The preceding reviewed source baseline is `c25f0d97`, adding explicit nonmember read discovery on the shared query lane. Discovery allocates no local replica or voting identity; namespace/process grants remain independent. New TLS fixtures cover both image modes, foreign/missing grants, original-budget leader retries, @@ -458,8 +472,10 @@ Do not let plan maintenance replace implementation. physical group/logical cell binding and host-owned gateway registration. Explicit nonmember shared discovery and real-TLS grant revocation now pass scoped tests; public-host automatic discovery/SDK binding remain pending. - Replace the gateway's per-RPC catalog lookup with bounded, - correctly invalidated routing state before claiming steady-state performance. + Bounded invocation range caching now passes metadata-read and real-runtime + cases. The destination retains per-RPC catalog admission: removing that read + requires replicated transfer seals and serving-admission invalidation, linked + to CG-09. Do not substitute a TTL or cached hint for that authority contract. Keep query-origin grants read-only; command/enrollment RPC needs explicit write authorization, response-independent producers and safe remote receipt lookup. The old host refuses registered initializers until its actual replacement; diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 69c7e1467..39416f429 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 61 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 62 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 6ebacabd0..8b00cda15 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,52 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 bounded invocation route cache + +Qualified source: `06652e36869144624e0ca4d860782ef512018791`. + +`CellGroupDispatcher` clones now share bounded namespace range hints across query, +command and enrollment invocations. One cached range serves many cells; no +per-cell cache, worker or Raft owner is allocated by routing. Explicit entry and +reserved-byte limits bound retention, with LRU eviction. The conservative 8 KiB +reservation per slot remains charged even while the slot is empty. + +Hints come only from a verified directory lookup. `refresh_routes` observes the +catalog in the owning execution domain; a newer head invalidates older hints. +An older in-flight lookup cannot refill the cache after that observation, and a +same-revision/different-digest observation refuses. Failed refresh retains only +historical hints. Redirect confirmation, original deadlines, attempt budgets, +exact cell identities and submitted-command ambiguity remain enforced. + +Three catalog cases disable page caching and measure actual object-store calls. +After warming one range, 64 other cells and a command route with zero metadata +reads. Byte-limited LRU eviction, successor-head invalidation and a deliberately +held stale lookup pass. A fourth native case runs both physical image modes over +real TLS quorum owners: after removing the catalog head, local reads, a command +and enrollment succeed using the hint plus actual runtime authority. Remote +admission refuses because the destination still requires a fresh catalog check. +The fixture restores metadata and joins all physical owners. + +All 202 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The source-bound record at +`.cellar/cell-groups-evidence/CG-04-dispatch-route-cache-final/checkpoint.json` binds +265 source inputs and 4 copied executables. Transfer the complete directory +with previous checkpoints, including the initial `CG-04-dispatch-route-cache` +Clippy failure. That fixture called Tokio timeout directly; the successor uses +the execution-domain timeout without changing assertions or production behavior. +Targeted diagnostic logs are retained separately. + +CG-04 remains IN_PROGRESS. This removes repeat caller-side mapping reads; it does +not remove destination catalog admission or establish end-to-end latency gains. +Destination caching needs replicated transfer sealing and admission invalidation, +not a timeout that treats a stale hint as permission. The next critical delivery +is write-authorized command/enrollment RPC with response-independent producer +ownership and safe receipt lookup. Public-host discovery/SDK binding, lazy birth, +dynamic roster, per-group failure isolation and configuration lifecycle remain. +Complete-group snapshots/archive, shared RustFS recovery, independent-process +failures, power loss and load remain unqualified. + ## CG-04 nonmember read discovery Qualified source: `c25f0d97d5c98c6111ea2165b38a11a08613c0ca`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 4fedb57e0..24969074a 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -108,9 +108,20 @@ group's newer log cut. The caller independently enforces an AtLeast minimum. Retain result admission until consumption, and retain entered SQL/image owners through physical cleanup after disconnect, timeout or cancelled shutdown. +Invocation routing shares a bounded namespace range-hint cache across query, +command and enrollment dispatch. Configure both entry and reserved-byte bounds; +clones share the same cache, and cell count does not allocate one entry per cell. +Hints are learned only from verified directory lookups. An observed successor +head clears older hints; a lookup completing from an older head cannot refill +that cache. Same-revision/different-digest observations refuse. Explicit refresh +runs in the owning execution domain; failed refresh preserves only historical +hints, never a freshness or serving claim. Redirects still require exact catalog +confirmation under the original deadline and attempt budget. + The current shared gateway resolves the destination catalog route for each RPC. -Bounded routing-cache integration remains a CG-04 performance requirement; this -intermediate behavior does not establish steady-state request cost or latency. +Removing that destination read depends on replicated transfer sealing and +admission invalidation. The invocation cache does not relax the current check; +this intermediate behavior does not establish steady-state request latency. Attached-peer and explicitly discovered nonmember reads share the same query lane. Nonmember discovery creates no local replica and grants no voting or write authority. It consumes the same bounded route/peer registries, participates in diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 77215fb24..a801ef626 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -441,7 +441,9 @@ "crates/cellar/application/consensus/", "crates/cellar-logic/", "crates/cellar-ltx/src/generation.rs", - "crates/cellar-ltx/tests/private_clone.rs" + "crates/cellar-ltx/tests/private_clone.rs", + "crates/cellar/tests/native_cell_groups_catalog.rs", + "crates/cellar/tests/support/cell_group_directory.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/owner.rs", @@ -508,7 +510,10 @@ "crates/cellar/durability/raft/query_cell_wire.rs", "crates/cellar/tests/support/cell_group_remote_queries.rs", "crates/cellar/durability/raft/query_cell_discovery.rs", - "crates/cellar/tests/support/cell_group_query_origins.rs" + "crates/cellar/tests/support/cell_group_query_origins.rs", + "crates/cellar/durability/raft/cell_groups/directory/route_cache.rs", + "crates/cellar/tests/support/cell_group_routing_cache.rs", + "crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -647,6 +652,12 @@ "shared_nonmember_query_revocation_retains_physical_sql", "shared_nonmember_query_route_replacement_refuses_stale_reply" ] + }, + { + "action": "Share bounded namespace range hints across query, command and enrollment invocations, with explicit entry/byte reservations and LRU eviction. Cache ranges rather than cells; a failed metadata refresh does not invent a new serving proof. Verified successor heads invalidate prior hints and older in-flight observations cannot refill them. Preserve fresh destination RPC admission until replicated transfer-seal authority is integrated. Run the full native_cell_groups_catalog suite, including dispatcher_route_cache_shares_ranges_across_operations, dispatcher_route_cache_bounds_and_refreshes_hints and dispatcher_route_cache_refuses_superseded_inflight_refill, alongside the real runtime case below. Cached mappings never replace runtime ownership, exact cell/version checks, current-term quorum or producer ownership.", + "tests": [ + "shared_cached_dispatch_preserves_authority_during_catalog_outage" + ] } ], "test_cases": [ @@ -1229,6 +1240,15 @@ "I05", "I12" ] + }, + { + "name": "shared_cached_dispatch_preserves_authority_during_catalog_outage", + "oracle": "Warm one range hint, remove the actual catalog head, then query another cell, commit a command and enroll a new cell through real controller/SQL/TLS owners in both image modes without fresh mapping reads. Cache occupancy stays one range. Strong local reads still use real quorum. Explicit refresh fails; remote gateway admission remains fresh and refuses while the catalog head is absent. Restore metadata and join every owner.", + "invariants": [ + "I01", + "I05", + "I12" + ] } ], "invariants": [ From 5fd4443796a6e990fbe70c16acb7fa5578b7e763 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 09:24:10 -0700 Subject: [PATCH 104/568] feat: route shared commands with explicit write authority --- crates/cellar/application/consensus/mod.rs | 6 +- .../raft/cell_groups/controller/mod.rs | 6 + .../raft/cell_groups/dispatch/command/mod.rs | 68 ++- .../cell_groups/dispatch/command/remote.rs | 85 ++++ .../raft/cell_groups/dispatch/mod.rs | 17 + .../cellar/durability/raft/cell_groups/mod.rs | 6 +- .../durability/raft/cell_groups/owner.rs | 5 + .../durability/raft/command_cell_transport.rs | 438 ++++++++++++++++++ .../durability/raft/command_cell_wire.rs | 233 ++++++++++ .../cellar/durability/raft/command_origins.rs | 120 +++++ .../durability/raft/query_cell_transport.rs | 8 + .../cellar/durability/raft/query_identity.rs | 17 + .../cellar/durability/raft/query_transport.rs | 8 + crates/cellar/durability/raft/query_wire.rs | 2 + .../durability/raft/query_wire_tests.rs | 189 ++++++++ crates/cellar/durability/raft/transport.rs | 15 + .../durability/raft/transport_directory.rs | 4 + .../cellar/durability/raft/transport_fleet.rs | 7 +- .../durability/raft/transport_options.rs | 14 + .../durability/raft/transport_shared.rs | 8 + .../cellar/durability/raft/transport_tests.rs | 88 ++++ .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_query_origins.rs | 14 +- .../support/cell_group_remote_commands.rs | 319 +++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 6 +- 26 files changed, 1669 insertions(+), 31 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs create mode 100644 crates/cellar/durability/raft/command_cell_transport.rs create mode 100644 crates/cellar/durability/raft/command_cell_wire.rs create mode 100644 crates/cellar/durability/raft/command_origins.rs create mode 100644 crates/cellar/tests/support/cell_group_remote_commands.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 23903fdcc..3904cdba4 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -107,7 +107,8 @@ pub use crate::raft::operations::{ }; pub use crate::raft::options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}; pub use crate::raft::transport_options::{ - NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, TransportOptions, + NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, + TransportOptions, }; pub use crate::raft::version::{ VersionError, VersionKeys, VersionSigningKey, VersionToken, VersionVerificationKey, @@ -213,7 +214,8 @@ pub mod verification { Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, }; pub use crate::raft::transport_options::{ - NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, TransportOptions, + NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, + TransportOptions, }; /// Contracts under integration into the replacement group runtime. pub mod cell_groups { diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 0cd4f13fc..790331145 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -79,6 +79,7 @@ pub struct CellGroupControllerUsage { } struct Shared { + runtime: CellGroupRuntime, groups: BTreeSet, limits: CellGroupControllerLimits, domain: crate::asyncrt::DomainToken, @@ -146,6 +147,10 @@ pub struct CellGroupController { } impl CellGroupController { + pub(crate) fn belongs_to(&self, runtime: &CellGroupRuntime) -> bool { + self.handle.0.runtime.same_owner(runtime) + } + /// Consume the exclusive event receiver after validating exact engine and /// runtime ownership. Every listed group must already be attached. Later /// attachment requires a future roster API; an unknown event fails closed. @@ -192,6 +197,7 @@ impl CellGroupController { let domain = runtime.log().domain(); let (requests, receiver) = tokio::sync::mpsc::channel(limits.max_commands); let shared = Arc::new(Shared { + runtime: runtime.clone(), groups: roster.keys().copied().collect(), limits, domain: domain.clone(), diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 0191f7500..b54d74997 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -3,6 +3,7 @@ //! Routed commands retain their originating producer until exact resolution. //! The node controller drives publication; acceptance is only provisional. mod owned; +mod remote; use super::*; use crate::application::consensus::command_request_digest; @@ -11,6 +12,7 @@ use crate::storage::CommandInputs; use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; use cellar_ltx::generation::ImageId; +#[derive(Clone)] pub struct CellCommandRequest { pub id: CommandId, pub contract: OperationContract, @@ -99,16 +101,29 @@ pub struct CellCommandInvocation { dispatched: bool, ambiguous: bool, origin: Option, + remote_route: Option, _admission: Permit, } impl CellGroupDispatcher { pub async fn command( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + self.command_with_inputs(cell, request, limits, cached, None) + .await + } + + pub(crate) async fn command_with_inputs( &self, cell: CellHistory, mut request: CellCommandRequest, limits: CellRoutingLimits, cached: Option, + captured: Option, ) -> Result { let error = |cause| CellCommandError { command: request.id, @@ -150,19 +165,22 @@ impl CellGroupDispatcher { .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; request.input = request.input.into_boxed_slice().into_vec(); request.contract.operation = request.contract.operation.into_boxed_str().into_string(); - let inputs = self - .domain - .run(async { - use rand::RngCore as _; - let mut random = [0; 32]; - crate::asyncrt::rng("cell-group-command-inputs").fill_bytes(&mut random); - CommandInputs { - now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, - random, - } - }) - .await - .map_err(|_| error(closed()))?; + let inputs = if let Some(inputs) = captured { + inputs + } else { + self.domain + .run(async { + use rand::RngCore as _; + let mut random = [0; 32]; + crate::asyncrt::rng("cell-group-command-inputs").fill_bytes(&mut random); + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + } + }) + .await + .map_err(|_| error(closed()))? + }; let session = self .directory .routing_session_in( @@ -188,6 +206,7 @@ impl CellGroupDispatcher { dispatched: false, ambiguous: false, origin: None, + remote_route: None, _admission: admission, }) } @@ -226,12 +245,23 @@ impl CellCommandInvocation { pub async fn begin(&mut self) -> Result<(), CellCommandError> { self.dispatched = true; - let result = self - .session - .begin() - .await - .map_err(CellDispatchError::Routing); - self.attempt = Some(result.map_err(|e| self.error(e))?); + let result = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing); + let attempt = result.map_err(|e| self.error(e))?; + if self + .remote_route + .as_ref() + .is_some_and(|expected| !expected.matches(attempt.route())) + { + return Err(self.error(CellDispatchError::Invalid( + "remote command destination route", + ))); + } + self.attempt = Some(attempt); self.dispatched = false; Ok(()) } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs new file mode 100644 index 000000000..f5581e8ff --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One remote attempt consumes the original route budget; loss stays ambiguous. +use super::*; +use crate::raft::query_wire::command::CellCommandCall; +use crate::raft::transport::{CellRemoteCommandResponse, RaftTransport, TransportError}; +use cellar_logic::consensus::MemberId; + +impl CellCommandInvocation { + pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteCommandRoute) { + self.remote_route = Some(route); + } + + /// Execute this catalog attempt on an explicitly selected remote member. + /// Any possible transmission makes failure ambiguous. Retry this same + /// invocation/identity; a transport response is never a non-commit proof. + pub async fn execute_remote( + &mut self, + transport: &RaftTransport, + member: MemberId, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched || self.origin.is_some() || self.remote_route.is_some() { + return Err(self.error(CellDispatchError::Invalid( + "consumed or local command attempt", + ))); + } + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(self.error(CellDispatchError::Invalid("command transport domain"))); + } + let now = self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))?; + let timeout_ms = self + .session + .deadline_ms() + .checked_sub(now) + .filter(|n| *n > 0) + .ok_or_else(|| { + self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + })?; + let call = CellCommandCall { + request: self.request.clone(), + inputs: self.inputs, + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + self.session + .mark_submitted(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.dispatched = true; + self.ambiguous = true; + let result = self + .dispatcher + .domain + .run(transport.command_cell( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + self.session.deadline_ms(), + )) + .await + .map_err(|_| self.error(closed()))?; + let response = result.map_err(|error| { + self.error(match error { + TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + }) + })?; + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 8106d17a5..150093477 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -537,3 +537,20 @@ fn catalog_failure(error: crate::raft::catalog::CatalogError) -> CellDispatchErr ambiguous_command: None, }) } + +/// Receiver-side destination constraint, checked against a fresh catalog lookup. +pub(crate) struct RemoteCommandRoute { + pub group: cellar_logic::cell_group::GroupHistory, + pub schema: [u8; 32], + pub epoch: u64, + pub birth: [u8; 32], +} + +impl RemoteCommandRoute { + fn matches(&self, route: &GroupRoute) -> bool { + self.group == route.history + && self.schema == route.schema_digest + && self.epoch == route.epoch.get() + && self.birth == route.birth_digest + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e4278fad0..eb82aff02 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -9,7 +9,9 @@ pub use super::host_sql::cell_groups::{ CellSqlUsage, }; -pub use crate::raft::transport::{CellQueryGateway, CellRemoteQueryResponse}; +pub use crate::raft::transport::{ + CellCommandGateway, CellQueryGateway, CellRemoteCommandResponse, CellRemoteQueryResponse, +}; mod apply; mod controller; @@ -75,3 +77,5 @@ pub use owner::{ CellGroupApply, CellGroupEffect, CellGroupOwner, CellGroupPersist, CellGroupPersisted, CellGroupProposal, CellGroupStatus, CellGroupTicket, }; + +pub(crate) use dispatch::RemoteCommandRoute; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index e53a34205..a61d7165f 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -275,6 +275,11 @@ impl CellGroupOwner { cell: &CellHistory, ) -> Result { let status = self.status()?; + // Role refusal is distinct from a leader still awaiting durable apply. + // Remote callers must not retry a follower as generic unavailability. + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } if !status.ready || status.applied.index != status.committed { return Err(RuntimeError::NotReady); } diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs new file mode 100644 index 000000000..212f57a55 --- /dev/null +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -0,0 +1,438 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Commands share the bounded RPC lane but require independent write grants. +use super::*; +use crate::application::consensus::CommandFailure; +use crate::raft::cell_groups::{ + CellDispatchError, CellGroupController, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, +}; +use crate::raft::query_identity::QueryRoute; +use crate::raft::query_wire::{ + self as wire, + command::{CellCommandCall, CellCommandValue}, +}; +use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{MemberId, NodeIdentity}, +}; +use queries::{QueryConnection, SendQuery}; +use std::time::Duration; + +/// Host-owned registration. The transport retains only a weak reference, so it +/// cannot extend the host's lifetime or create a runtime/transport ownership cycle. +#[derive(Clone)] +pub struct CellCommandGateway { + inner: Arc, +} + +// Accepted requests retain Gateway, never this admission owner. In-flight work +// cannot keep accepting new commands after the final host registration is dropped. +struct Registration { + gateway: Arc, +} + +impl Drop for Registration { + fn drop(&mut self) { + self.gateway.accepting.store(false, Ordering::Release); + } +} + +pub(super) struct Gateway { + accepting: AtomicBool, + controller: CellGroupController, + dispatcher: CellGroupDispatcher, + options: CellSqlOptions, +} + +/// Retains the same node/peer query admission through consumption of the result. +pub struct CellRemoteCommandResponse { + value: CellCommandValue, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteCommandResponse { + pub fn command(&self) -> cellar_logic::consensus::CommandId { + self.value.command + } + + pub fn retry_until_ms(&self) -> u64 { + self.value.retry_until_ms + } + + pub fn durability(&self) -> cellar_logic::consensus::Durability { + cellar_logic::consensus::Durability::Quorum + } + + pub fn result(&self) -> &[u8] { + &self.value.value.result + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.value.served_by + } +} + +pub(super) struct SendCellCommand { + route: Arc>, + peer: RaftPeer, + cell: CellHistory, + call: CellCommandCall, + keys: VersionKeys, + deadline: u64, + cancel: watch::Receiver, + reply: oneshot::Sender>, + node: Permit, + peer_bytes: Permit, +} + +struct Cancel(watch::Sender); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl RaftTransport { + /// Register compiled commands on the exact controller/runtime owner. Retain + /// this host-owned handle while accepting commands. Dropping it refuses new + /// admission; commands already transferred to the controller remain owned. + /// Namespace write grants are independently required even for voter peers. + pub fn install_cell_command_gateway( + &self, + dispatcher: CellGroupDispatcher, + controller: CellGroupController, + options: CellSqlOptions, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !dispatcher.uses_domain(&state.domain) || !controller.belongs_to(runtime) { + return Err(TransportError::Identity); + } + let key = dispatcher.namespace(); + let mut gateways = state + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + gateways.retain(|_, value| value.strong_count() > 0); + if gateways.contains_key(&key) { + return Err(TransportError::Refused); + } + if gateways.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let inner = Arc::new(Gateway { + accepting: AtomicBool::new(true), + controller, + dispatcher, + options, + }); + gateways.insert(key, Arc::downgrade(&inner)); + Ok(CellCommandGateway { + inner: Arc::new(Registration { gateway: inner }), + }) + } + + pub(crate) fn uses_command_domain(&self, domain: &DomainToken) -> bool { + self.handle.0.domain.same_owner(domain) + } + + /// Called only by a consumed catalog attempt. No forwarding or retry occurs + /// here; the caller keeps its original deadline and attempt budget. + pub(in crate::raft) async fn command_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellCommandCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let state = &self.handle.0; + // Discovery never creates a local replica or grants voting authority. + state.runtime.shared()?; + let (view, peer, pool) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let view = registry + .physical_groups + .get(&route.history) + .or_else(|| registry.cell_query_groups.get(&route.history)) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if view.schema_digest != route.schema_digest || !route.history.same_namespace(cell) { + return Err(TransportError::Identity); + } + let peer = view + .peers + .iter() + .find(|p| { + p.replica.member == member + && p.replica.node != state.runtime.log().limits().node + }) + .ok_or(TransportError::Identity)? + .clone(); + let pool = registry + .pools + .get(&peer.replica.node) + .ok_or(TransportError::StalePeer)? + .clone(); + (view, peer, pool) + }; + if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { + return Err(TransportError::Timeout); + } + let charge = queries::reservation( + call.request + .input + .len() + .checked_add(8192) + .ok_or(TransportError::Limit)?, + &state.options, + )?; + let node = state + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + let peer_bytes = pool + .outgoing + .query + .acquire(charge) + .ok_or(TransportError::Overloaded)?; + drop(call.encode(state.options.max_query_request_bytes)?); + if !state.current_cell_query_route(&view, &peer) { + return Err(TransportError::StalePeer); + } + let (reply, result) = oneshot::channel(); + let (cancel, cancelled) = watch::channel(false); + let _cancel = Cancel(cancel); + pool.query + .try_send(SendQuery::Command(Box::new(SendCellCommand { + route: view, + peer, + cell: cell.clone(), + call, + keys, + deadline, + cancel: cancelled, + reply, + node, + peer_bytes, + }))) + .map_err(|e| match e { + mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, + mpsc::error::TrySendError::Closed(_) => TransportError::Closed, + })?; + result.await.map_err(|_| TransportError::Closed)? + } +} + +impl CellCommandGateway { + /// The cluster and namespace bound to this host-owned registration. + pub fn namespace(&self) -> (ClusterId, NamespaceId) { + self.inner.gateway.dispatcher.namespace() + } +} + +impl State { + pub(super) fn authorize_cell_command( + &self, + wire: &QueryRoute, + ) -> Result<(Arc, Option>), TransportError> { + let runtime = self.runtime.shared()?; + let (source, target, _) = wire.decode_shared()?; + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let route = registry + .physical_groups + .get(&target.scope) + .ok_or(TransportError::UnknownGroup)?; + let group = runtime + .log() + .cell_group(&target.scope) + .map_err(|_| TransportError::UnknownGroup)?; + if *group.identity() != target + || wire.target_label() != self.auth.source() + || wire.schema() != route.schema_digest + { + return Err(TransportError::StalePeer); + } + shared::permitted(&group, &[&target])?; + let grant = registry + .command_origins + .get(&(target.scope.cluster, target.scope.namespace, source.node)) + .filter(|grant| !*grant.stop.borrow()) + .filter(|grant| { + grant.peer.as_ref().is_some_and(|peer| { + peer.node == source.node + && peer.disk == source.disk + && peer.process == source.process + && peer.label == wire.source_label() + }) + }) + .ok_or(TransportError::Refused)?; + let pool = registry + .pools + .get(&source.node) + .filter(|pool| grant.peer.as_ref() == Some(&pool.peer)) + .cloned() + .ok_or(TransportError::Refused)?; + Ok((pool, Some(grant.stop.subscribe()))) + } + + pub(super) async fn send_cell_command( + &self, + pool: &Pool, + mut request: SendCellCommand, + connection: &mut Option, + ) -> bool { + let result = self + .query_until( + pool, + request.deadline, + request.cancel.clone(), + None, + async { + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let remaining = request + .deadline + .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|n| *n > 0) + .ok_or(TransportError::Timeout)?; + request.call.timeout_ms = request.call.timeout_ms.min(remaining); + let body = request.call.encode(self.options.max_query_request_bytes)?; + let local = self.runtime.log().limits(); + let route = QueryRoute::new_shared( + NodeIdentity { + node: local.node, + disk: local.disk, + process: local.process, + }, + self.auth.source(), + &request.peer.replica, + &request.peer.label, + &request.cell, + request.route.schema_digest, + )? + .for_command()?; + let body = self + .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) + .await?; + if !self.current_cell_query_route(&request.route, &request.peer) { + return Err(TransportError::StalePeer); + } + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::command::decode_result( + &body, + (request.call.request.id, &request.cell), + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map_err(TransportError::Command) + }, + ) + .await; + // A fully authenticated application refusal consumed a complete reply; + // retain the peer connection just as the legacy query lane does. + let failed = matches!(&result, Err(error) if !matches!(error, TransportError::Command(_))); + let _ = request + .reply + .send(result.map(|value| CellRemoteCommandResponse { + value, + _node: request.node, + _peer: request.peer_bytes, + })); + failed + } +} + +impl State { + pub(super) async fn execute_cell_command( + &self, + route: &QueryRoute, + call: CellCommandCall, + ) -> Result, TransportError> { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let binding = crate::raft::cell_groups::RemoteCommandRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .command_with_inputs( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + Some(call.inputs), + ) + .await + .map_err(|error| command_failure(&error.cause))?; + invocation.bind_remote_route(binding); + if !gateway.accepting.load(Ordering::Acquire) { + return Err(CommandFailure::Unavailable); + } + // Recheck after asynchronous request construction, before ownership + // transfers. Later revocation cannot undo an admitted mutation. + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + let call = gateway + .controller + .command(invocation, gateway.options.clone()) + .map_err(|error| command_failure(&error.error.cause))?; + call.response() + .await + .map_err(|error| command_failure(&error.cause)) + } + .await; + wire::command::encode_result(&result, self.options.max_query_response_bytes) + } +} + +fn command_failure(error: &CellDispatchError) -> CommandFailure { + use crate::raft::execution::ExecutionError; + match error { + CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure.clone(), + CellDispatchError::Runtime(RuntimeError::NotLeader) => { + CommandFailure::NotLeader { leader: None } + } + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/command_cell_wire.rs b/crates/cellar/durability/raft/command_cell_wire.rs new file mode 100644 index 000000000..e4455897b --- /dev/null +++ b/crates/cellar/durability/raft/command_cell_wire.rs @@ -0,0 +1,233 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Command payloads are accepted only under an authenticated command-purpose route. +use super::*; +use crate::raft::cell_groups::{CellCommandRequest, CellCommandResponse}; +use crate::storage::CommandInputs; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{ClientId, CommandId, Durability}, +}; + +pub(in crate::raft) struct CellCommandCall { + pub request: CellCommandRequest, + pub inputs: CommandInputs, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +pub(in crate::raft) struct CellCommandValue { + pub command: CommandId, + pub retry_until_ms: u64, + pub value: cell::CellQueryValue, +} + +impl CellCommandCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let r = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if r.contract.kind != OperationKind::Command + || r.contract.operation.is_empty() + || r.contract.operation.len() > 128 + || r.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + || r.durability != Durability::Quorum + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCCQ01".to_vec(); + body.extend_from_slice(&r.id.client.bytes()); + body.extend_from_slice(&r.id.sequence().to_be_bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &r.contract.operation)?; + body.extend_from_slice(&r.contract.digest); + body.push(1); // Quorum; never silently lower an S3 request. + body.extend_from_slice(&r.retry_until_ms.to_be_bytes()); + body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); + body.extend_from_slice(&self.inputs.random); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(r.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(r.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&r.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCCQ01" { + return Err(TransportError::Protocol); + } + let id = command_id(&mut body)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Command, + digest: body.array()?, + }; + if body.byte()? != 1 { + return Err(TransportError::Protocol); + } + let retry_until_ms = u64::from_be_bytes(body.array()?); + let inputs = CommandInputs { + now_unix_millis: u64::from_be_bytes(body.array()?), + random: body.array()?, + }; + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellCommandRequest { + id, + contract, + input, + durability: Durability::Quorum, + retry_until_ms, + }, + inputs, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +fn command_id(body: &mut Reader<'_>) -> Result { + let client = ClientId::new(body.array()?).map_err(|_| TransportError::Identity)?; + CommandId::new(client, u64::from_be_bytes(body.array()?)).map_err(|_| TransportError::Identity) +} + +pub(in crate::raft) fn encode_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCCS01".to_vec(); + match result { + Ok(response) => { + let receipt = &response.receipt().receipt; + body.push(0); + body.extend_from_slice(&receipt.command().client.bytes()); + body.extend_from_slice(&receipt.command().sequence().to_be_bytes()); + body.extend_from_slice(&receipt.retry_until_ms().to_be_bytes()); + string(&mut body, response.token().as_str())?; + let length = + u32::try_from(receipt.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(receipt.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(receipt.result()); + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::CommandExpired) => body.push(10), + Err(CommandFailure::SequenceGap { next }) => { + body.push(11); + body.extend_from_slice(&next.to_be_bytes()); + } + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(error) => body.push(failure_code(error)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_result( + bytes: &[u8], + expected: (CommandId, &CellHistory), + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCCS01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + 0 => { + let command = command_id(&mut body)?; + if command != expected.0 { + return Err(TransportError::Authentication); + } + let retry_until_ms = u64::from_be_bytes(body.array()?); + let token = crate::raft::version::CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, expected.1, schema, now) + .map_err(|_| TransportError::Authentication)?; + let length = u32::from_be_bytes(body.array()?) as usize; + Ok(CellCommandValue { + command, + retry_until_ms, + value: cell::CellQueryValue { + result: body.take(length)?.to_vec(), + observation, + token, + served_by, + }, + }) + } + 9 => Err(CommandFailure::CommandConflict), + 10 => Err(CommandFailure::CommandExpired), + 11 => Err(CommandFailure::SequenceGap { + next: u64::from_be_bytes(body.array()?), + }), + 12 => { + let id = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if id == 0 { + None + } else { + Some(MemberId::new(id).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/command_origins.rs b/crates/cellar/durability/raft/command_origins.rs new file mode 100644 index 000000000..cd157183c --- /dev/null +++ b/crates/cellar/durability/raft/command_origins.rs @@ -0,0 +1,120 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit write grants remain independent from query and voting permission. +use super::*; +use crate::raft::transport_options::NamespaceCommandOrigin; + +impl RaftTransport { + /// Install trusted namespace command permission. It cannot create a group or + /// authorize a consensus message. Revisions include revocation tombstones. + pub fn grant_command_origin( + &self, + grant: NamespaceCommandOrigin, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + grant.peer.validate_endpoint()?; + if grant.revision == 0 || grant.peer.node == state.runtime.log().limits().node { + return Err(TransportError::Identity); + } + let key = (grant.cluster, grant.namespace, grant.peer.node); + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry.check_node_view(&grant.peer)?; + registry.check_physical_endpoint(&grant.peer)?; + if let Some(old) = registry.command_origins.get(&key) { + if old.revision == grant.revision && old.peer.as_ref() == Some(&grant.peer) { + return Ok(()); + } + if old.revision >= grant.revision { + return Err(TransportError::StalePeer); + } + } else if registry.command_origins.len() >= state.options.max_command_origins { + return Err(TransportError::Overloaded); + } + if registry + .groups + .values() + .chain(registry.query_groups.values()) + .flat_map(|route| &route.peers) + .any(|peer| peer.replica.node == grant.peer.node && !grant.peer.same_host(peer)) + || registry + .command_origins + .iter() + .filter(|(other, _)| **other != key) + .filter_map(|(_, old)| old.peer.as_ref()) + .any(|peer| peer.node == grant.peer.node && *peer != grant.peer) + { + return Err(TransportError::StalePeer); + } + if registry + .origins + .values() + .filter_map(|grant| grant.peer.as_ref()) + .any(|old| old.node == grant.peer.node && old != &grant.peer) + { + return Err(TransportError::StalePeer); + } + if registry + .inspection_origins + .values() + .filter_map(|origin| origin.peer.as_ref()) + .any(|peer| peer.node == grant.peer.node && peer != &grant.peer) + { + return Err(TransportError::StalePeer); + } + let starts = state.prepare_pools(&mut registry, std::slice::from_ref(&grant.peer))?; + let (stop, _) = watch::channel(false); + if let Some(old) = registry.command_origins.insert( + key, + Arc::new(queries::OriginGrant { + revision: grant.revision, + peer: Some(grant.peer), + stop, + }), + ) { + old.stop.send_replace(true); + } + registry.prune_pools(); + drop(registry); + state.start_pools(starts) + } + + pub fn revoke_command_origin( + &self, + cluster: ClusterId, + namespace: NamespaceId, + node: NodeId, + revision: u64, + ) -> Result<(), TransportError> { + if revision == 0 { + return Err(TransportError::Identity); + } + let state = &self.handle.0; + let key = (cluster, namespace, node); + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if let Some(old) = registry.command_origins.get(&key) { + if old.revision == revision && old.peer.is_none() { + return Ok(()); + } + if old.revision >= revision { + return Err(TransportError::StalePeer); + } + } else if registry.command_origins.len() >= state.options.max_command_origins { + return Err(TransportError::Overloaded); + } + let (stop, _) = watch::channel(true); + if let Some(old) = registry.command_origins.insert( + key, + Arc::new(queries::OriginGrant { + revision, + peer: None, + stop, + }), + ) { + old.stop.send_replace(true); + } + registry.prune_pools(); + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index fdffe8756..f73d76a3e 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -357,6 +357,7 @@ impl State { pub(super) enum IncomingCall { Legacy(QueryCall), Shared(CellQueryCall), + Command(crate::raft::query_wire::command::CellCommandCall), } impl IncomingCall { @@ -365,6 +366,11 @@ impl IncomingCall { bytes: &[u8], limit: usize, ) -> Result { + if route.is_command() { + return Ok(Self::Command( + crate::raft::query_wire::command::CellCommandCall::decode(bytes, limit)?, + )); + } if route.is_shared() { Ok(Self::Shared(CellQueryCall::decode(bytes, limit)?)) } else { @@ -376,6 +382,7 @@ impl IncomingCall { match self { Self::Legacy(call) => call.options.timeout.as_millis() as u64, Self::Shared(call) => call.timeout_ms, + Self::Command(call) => call.timeout_ms, } } @@ -385,6 +392,7 @@ impl IncomingCall { route: &QueryRoute, ) -> Result, TransportError> { match self { + Self::Command(call) => state.execute_cell_command(route, call).await, Self::Legacy(call) => { let (_, target) = route.decode()?; let handle = state diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index 07fc411d7..9f058206a 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -58,6 +58,7 @@ enum History { #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct QueryRoute { + command: bool, scope: History, schema: [u8; 32], source: Endpoint, @@ -73,6 +74,7 @@ impl QueryRoute { schema: [u8; 32], ) -> Result { let route = Self { + command: false, scope: History::Cell(WireScope::from(&target.scope)), schema, source: Endpoint::new(source, source_label, None), @@ -85,6 +87,9 @@ impl QueryRoute { /// Request orientation only. A reversed response route is compared with /// the exact outstanding request; it cannot decode as another request. pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { + if self.command { + return Err(TransportError::Identity); + } let History::Cell(scope) = &self.scope else { return Err(TransportError::Identity); }; @@ -137,6 +142,7 @@ impl QueryRoute { } let g = target.scope; let route = Self { + command: false, scope: History::Shared { group: [ g.cluster.bytes(), @@ -182,6 +188,16 @@ impl QueryRoute { Ok((source, target, cell)) } + pub fn for_command(mut self) -> Result { + self.decode_shared()?; + self.command = true; + Ok(self) + } + + pub fn is_command(&self) -> bool { + self.command + } + pub fn is_shared(&self) -> bool { matches!(self.scope, History::Shared { .. }) } @@ -214,6 +230,7 @@ impl QueryRoute { pub fn reversed(&self) -> Self { Self { + command: self.command, scope: self.scope.clone(), schema: self.schema, source: self.target.clone(), diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index aaffb0d8f..0458dbbf1 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -40,6 +40,7 @@ impl Drop for QueryDelivery { pub(super) enum SendQuery { Legacy(Box), Shared(Box), + Command(Box), } pub(super) struct LegacySendQuery { @@ -394,6 +395,9 @@ impl State { &self, wire: &QueryRoute, ) -> Result<(Arc, Option>), TransportError> { + if wire.is_command() { + return self.authorize_cell_command(wire); + } if wire.is_shared() { return self.authorize_cell_query(wire); } @@ -723,6 +727,10 @@ impl State { })); failed } + SendQuery::Command(request) => { + self.send_cell_command(&pool, *request, &mut connection) + .await + } SendQuery::Shared(request) => { self.send_cell_query(&pool, *request, &mut connection).await } diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index 31509fcb5..1f002c4c6 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -19,6 +19,8 @@ use std::time::Duration; #[path = "query_cell_wire.rs"] pub(super) mod cell; +#[path = "command_cell_wire.rs"] +pub(super) mod command; pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index e3aecb045..b70a74e3d 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -631,3 +631,192 @@ fn shared_query_result_authenticates_cell_progress_not_group_log_index() { ); } } + +fn command_call() -> command::CellCommandCall { + command::CellCommandCall { + request: crate::raft::cell_groups::CellCommandRequest { + id: CommandId::new(ClientId::new([71; 16]).unwrap(), 3).unwrap(), + contract: OperationContract { + operation: "Add".into(), + digest: [72; 32], + kind: OperationKind::Command, + }, + input: vec![1, 2, 3], + durability: Durability::Quorum, + retry_until_ms: 9000, + }, + inputs: crate::storage::CommandInputs { + now_unix_millis: 1000, + random: [73; 32], + }, + timeout_ms: 1000, + route_epoch: 3, + birth_digest: [74; 32], + } +} + +#[test] +fn shared_command_wire_preserves_inputs_and_refuses_query_payloads() { + let call = command_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = command::CellCommandCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.inputs.random, call.inputs.random); + assert_eq!(decoded.inputs.now_unix_millis, call.inputs.now_unix_millis); + assert_eq!(decoded.request.id, call.request.id); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + assert!(command::CellCommandCall::decode( + &cell_call(crate::raft::cell_groups::CellQueryConsistency::Stale) + .encode(4096) + .unwrap(), + 4096 + ) + .is_err()); + for end in 0..bytes.len() { + assert!(command::CellCommandCall::decode(&bytes[..end], 4096).is_err()); + } + let mut trailing = bytes.clone(); + trailing.push(0); + assert!(command::CellCommandCall::decode(&trailing, 4096).is_err()); + assert!(command::CellCommandCall::decode(&bytes, bytes.len() - 1).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply() { + use cellar_logic::cell_group::GroupHistory; + let old = identity(2); + let target = ReplicaIdentity { + scope: GroupHistory::from(&old.scope), + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let cell = cell_observation().version.history; + let read = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell, + [9; 32], + ) + .unwrap(); + let write = read.clone().for_command().unwrap(); + assert!(write.is_command()); + assert!(!read.is_command()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = command_call().encode(4096).unwrap(); + let request = RequestHead::new(write.clone(), 3, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["command"] = serde_json::json!(false); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + let response_body = + command::encode_result(&Err(CommandFailure::CommandConflict), 4096).unwrap(); + let response = response_head( + &auth, + &[3; 32], + &request, + &head.tag, + &response_body, + &options, + ) + .unwrap(); + let read_request = RequestHead::new(read, 3, &body); + assert!(read_response( + &response, + &auth, + &[3; 32], + &read_request, + &head.tag, + &options + ) + .is_err()); + let mut missing = serde_json::to_value(write).unwrap(); + missing.as_object_mut().unwrap().remove("command"); + assert!(serde_json::from_value::(missing).is_err()); +} + +#[test] +fn shared_command_results_bind_identity_history_and_typed_refusals() { + let call = command_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let mut body = b"CLGCCS01".to_vec(); + body.push(0); + body.extend_from_slice(&call.request.id.client.bytes()); + body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + body.extend_from_slice(&9000u64.to_be_bytes()); + string(&mut body, token.as_str()).unwrap(); + body.extend_from_slice(&2u32.to_be_bytes()); + body.extend_from_slice(b"ok"); + let decoded = command::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ) + .unwrap() + .unwrap(); + assert_eq!(decoded.value.observation, observation); + assert_eq!(decoded.value.result, b"ok"); + let wrong = CommandId::new(call.request.id.client, 4).unwrap(); + assert!(command::decode_result( + &body, + (wrong, cell), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + let mut other = cell.clone(); + other.incarnation = IncarnationId::new([78; 16]).unwrap(); + assert!(command::decode_result( + &body, + (call.request.id, &other), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + for failure in [ + CommandFailure::CommandConflict, + CommandFailure::CommandExpired, + CommandFailure::SequenceGap { next: 9 }, + CommandFailure::NotLeader { + leader: Some(MemberId::new(2).unwrap()), + }, + ] { + let body = command::encode_result(&Err(failure.clone()), 4096).unwrap(); + let decoded = command::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ) + .unwrap(); + assert_eq!(decoded.err(), Some(failure)); + } +} diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index d120679be..72bfbdcd5 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -35,8 +35,13 @@ use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt}; use tokio::net::{TcpListener, TcpStream}; use tokio::sync::{mpsc, oneshot, watch, Notify, Semaphore}; +#[path = "command_cell_transport.rs"] +mod cell_commands; #[path = "query_cell_transport.rs"] mod cell_queries; +#[path = "command_origins.rs"] +mod command_origins; +pub use cell_commands::{CellCommandGateway, CellRemoteCommandResponse}; #[path = "query_cell_discovery.rs"] mod cell_query_discovery; #[path = "transport_control_send.rs"] @@ -84,6 +89,7 @@ pub enum TransportError { Runtime(RuntimeError), Authentication, Query(cellar_logic::consensus::CommandFailure), + Command(cellar_logic::consensus::CommandFailure), Protocol, Identity, Replay, @@ -287,6 +293,7 @@ struct Registry { physical_groups: BTreeMap>>, query_groups: BTreeMap>, cell_query_groups: BTreeMap>>, + command_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, inspection_peers: BTreeMap>, @@ -319,6 +326,7 @@ impl Registry { .chain( self.origins .values() + .chain(self.command_origins.values()) .filter_map(|grant| grant.peer.as_ref().map(|peer| peer.node)), ) .collect(); @@ -343,6 +351,8 @@ struct State { local_control: SocketAddr, local_data: SocketAddr, local_query: SocketAddr, + cell_command_gateways: + Mutex>>, cell_query_gateways: Mutex>>, query_gateway: Mutex>>, @@ -504,6 +514,7 @@ impl RaftTransport { local_data, local_query, query_gateway: Mutex::new(None), + cell_command_gateways: Mutex::new(BTreeMap::new()), cell_query_gateways: Mutex::new(BTreeMap::new()), registry: Mutex::new(Registry { closed: false, @@ -512,6 +523,7 @@ impl RaftTransport { physical_groups: BTreeMap::new(), query_groups: BTreeMap::new(), cell_query_groups: BTreeMap::new(), + command_origins: BTreeMap::new(), origins: BTreeMap::new(), inspection_origins: BTreeMap::new(), inspection_peers: BTreeMap::new(), @@ -1040,6 +1052,9 @@ impl State { if let Ok(mut gateway) = self.query_gateway.lock() { gateway.take(); } + if let Ok(mut gateways) = self.cell_command_gateways.lock() { + gateways.clear(); + } if let Ok(mut gateways) = self.cell_query_gateways.lock() { gateways.clear(); } diff --git a/crates/cellar/durability/raft/transport_directory.rs b/crates/cellar/durability/raft/transport_directory.rs index b744f1668..f4fe81654 100644 --- a/crates/cellar/durability/raft/transport_directory.rs +++ b/crates/cellar/durability/raft/transport_directory.rs @@ -117,6 +117,7 @@ impl RaftTransport { .origins .values() .chain(registry.inspection_origins.values()) + .chain(registry.command_origins.values()) .filter_map(|grant| grant.peer.as_ref()) .filter(|old| old.node == peer.node) { @@ -149,6 +150,7 @@ impl RaftTransport { .origins .values() .chain(registry.inspection_origins.values()) + .chain(registry.command_origins.values()) .any(|grant| grant.peer.as_ref().is_some_and(|old| old.node == peer.node))); let physical = if registry.pools.contains_key(&peer.node) || routed { std::slice::from_ref(peer) @@ -165,11 +167,13 @@ impl RaftTransport { let Registry { origins, inspection_origins, + command_origins, .. } = &mut *registry; for grant in origins .values_mut() .chain(inspection_origins.values_mut()) + .chain(command_origins.values_mut()) .filter(|grant| { grant .peer diff --git a/crates/cellar/durability/raft/transport_fleet.rs b/crates/cellar/durability/raft/transport_fleet.rs index 70721e849..b284238a6 100644 --- a/crates/cellar/durability/raft/transport_fleet.rs +++ b/crates/cellar/durability/raft/transport_fleet.rs @@ -163,9 +163,14 @@ impl RaftTransport { let Registry { origins, inspection_origins, + command_origins, .. } = &mut *registry; - for grant in origins.values_mut().chain(inspection_origins.values_mut()) { + for grant in origins + .values_mut() + .chain(inspection_origins.values_mut()) + .chain(command_origins.values_mut()) + { if grant.peer.as_ref().is_some_and(|peer| !allowed(peer)) { grant.stop.send_replace(true); // A boot supersession cancels this physical owner, but the diff --git a/crates/cellar/durability/raft/transport_options.rs b/crates/cellar/durability/raft/transport_options.rs index 29e1f5f77..47cf0dee4 100644 --- a/crates/cellar/durability/raft/transport_options.rs +++ b/crates/cellar/durability/raft/transport_options.rs @@ -31,6 +31,16 @@ pub struct NamespaceQueryOrigin { pub peer: RaftPeerNode, } +/// Explicit command permission for an exact namespace and physical process. +/// Read grants, discovery and Raft membership never create this permission. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct NamespaceCommandOrigin { + pub cluster: ClusterId, + pub namespace: NamespaceId, + pub revision: u64, + pub peer: RaftPeerNode, +} + /// Explicit namespace inspection permission for a physical node process. /// Query grants and directory observations cannot create this permission. #[derive(Clone, Debug, PartialEq, Eq)] @@ -46,6 +56,7 @@ pub struct TransportOptions { pub max_peers: usize, pub max_groups: usize, pub max_query_origins: usize, + pub max_command_origins: usize, pub max_connections_per_lane: usize, pub max_header_bytes: usize, pub max_message_bytes: usize, @@ -77,6 +88,7 @@ impl Default for TransportOptions { max_peers: 128, max_groups: 4096, max_query_origins: 4096, + max_command_origins: 4096, max_connections_per_lane: 128, max_header_bytes: 8192, max_message_bytes: 20 * 1024 * 1024, @@ -129,6 +141,8 @@ impl TransportOptions { || self.max_peers > 4096 || self.max_groups == 0 || self.max_groups > 32_768 + || self.max_command_origins == 0 + || self.max_command_origins > 65_536 || self.max_query_origins == 0 || self.max_query_origins > 65_536 || self.max_connections_per_lane == 0 diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 23ec46285..2026a0aed 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -353,6 +353,14 @@ impl Registry { &self, peer: &RaftPeerNode, ) -> Result<(), TransportError> { + if self + .command_origins + .values() + .filter_map(|grant| grant.peer.as_ref()) + .any(|old| old.node == peer.node && old != peer) + { + return Err(TransportError::StalePeer); + } if self .physical_groups .values() diff --git a/crates/cellar/durability/raft/transport_tests.rs b/crates/cellar/durability/raft/transport_tests.rs index 3f0e20c25..43ce4aa33 100644 --- a/crates/cellar/durability/raft/transport_tests.rs +++ b/crates/cellar/durability/raft/transport_tests.rs @@ -544,3 +544,91 @@ async fn query_grants_are_scoped_revocable_and_never_voting_authority() -> anyho .await??; fixture.close().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn command_grants_bound_tombstones_without_changing_read_permission() -> anyhow::Result<()> { + use crate::raft::transport_options::NamespaceCommandOrigin; + let fixture = Fixture::start( + TransportOptions { + max_command_origins: 2, + ..Default::default() + }, + vec![], + ) + .await?; + fixture + .domain + .run(async { + let peer = RaftPeerNode { + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([5; 16])?, + process: ProcessGeneration::new([6; 16])?, + label: "node-4".into(), + server_name: "localhost".into(), + control_address: "127.0.0.1:41000".parse()?, + data_address: "127.0.0.1:41001".parse()?, + query_address: "127.0.0.1:41002".parse()?, + }; + let grant = NamespaceCommandOrigin { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + revision: 1, + peer: peer.clone(), + }; + let key = (grant.cluster, grant.namespace, peer.node); + let transport = &fixture.transport; + transport.grant_command_origin(grant.clone())?; + transport.grant_command_origin(grant.clone())?; + let old = { + let registry = transport.handle.0.registry.lock().unwrap(); + assert!(registry.origins.is_empty()); + assert_eq!(registry.command_origins.len(), 1); + registry.command_origins[&key].clone() + }; + assert_eq!(transport.usage().peers, 1); + let mut foreign = grant.clone(); + foreign.namespace = NamespaceId::new([3; 16])?; + transport.grant_command_origin(foreign.clone())?; + let mut full = grant.clone(); + full.namespace = NamespaceId::new([7; 16])?; + assert_eq!( + transport.grant_command_origin(full.clone()), + Err(TransportError::Overloaded) + ); + transport.revoke_command_origin(grant.cluster, grant.namespace, peer.node, 2)?; + assert!(*old.stop.borrow()); + assert_eq!( + transport.grant_command_origin(grant.clone()), + Err(TransportError::StalePeer) + ); + assert_eq!( + transport.revoke_command_origin(full.cluster, full.namespace, peer.node, 2), + Err(TransportError::Overloaded) + ); + // A read grant neither removes the write tombstone nor grants writes. + transport.grant_query_origin(NamespaceQueryOrigin { + cluster: grant.cluster, + namespace: grant.namespace, + revision: 1, + peer: peer.clone(), + })?; + { + let registry = transport.handle.0.registry.lock().unwrap(); + assert_eq!(registry.command_origins.len(), 2); + assert!(registry.command_origins[&key].peer.is_none()); + assert!(registry.origins[&key].peer.is_some()); + } + let mut restored = grant.clone(); + restored.revision = 3; + transport.grant_command_origin(restored)?; + transport.revoke_command_origin(grant.cluster, grant.namespace, peer.node, 4)?; + transport.revoke_command_origin(foreign.cluster, foreign.namespace, peer.node, 2)?; + assert_eq!(transport.usage().peers, 1); // Independent read grant retains its pool. + transport.revoke_query_origin(grant.cluster, grant.namespace, peer.node, 2)?; + assert_eq!(transport.usage().peers, 0); + assert!(fixture.log.groups()?.is_empty()); + Ok::<_, anyhow::Error>(()) + }) + .await??; + fixture.close().await +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index c13bf7bb5..c33559440 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1495,3 +1495,18 @@ async fn shared_nonmember_query_route_replacement_refuses_stale_reply() -> Resul async fn shared_cached_dispatch_preserves_authority_during_catalog_outage() -> Result<()> { Box::pin(runtime_fixture::cached_dispatch_execution()).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_commands_require_write_grants_and_exact_receipts() -> Result<()> { + Box::pin(runtime_fixture::remote_commands()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_commands_survive_disconnected_sql_waiters() -> Result<()> { + Box::pin(runtime_fixture::remote_command_lifecycle(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_command_revocation_preserves_admitted_ownership() -> Result<()> { + Box::pin(runtime_fixture::remote_command_lifecycle(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_query_origins.rs b/crates/cellar/tests/support/cell_group_query_origins.rs index 2b70d1b1c..baa2ddd3b 100644 --- a/crates/cellar/tests/support/cell_group_query_origins.rs +++ b/crates/cellar/tests/support/cell_group_query_origins.rs @@ -10,16 +10,16 @@ use std::sync::{ }; use tokio::net::TcpListener; -struct Router { - log: ReplicaLog, +pub(super) struct Router { + pub(super) log: ReplicaLog, runtime: CellGroupRuntime, events: CellGroupRuntimeEvents, - transport: RaftTransport, - peer: RaftPeerNode, + pub(super) transport: RaftTransport, + pub(super) peer: RaftPeerNode, } impl Router { - async fn start(fixture: &RuntimeCluster) -> Result { + pub(super) async fn start(fixture: &RuntimeCluster) -> Result { let node = NodeId::new([4; 16])?; let disk = DiskLineage::new([14; 16])?; let process = ProcessGeneration::new([20; 16])?; @@ -77,7 +77,7 @@ impl Router { }) } - fn grant(&self, fixture: &RuntimeCluster, revision: u64) -> Result<()> { + pub(super) fn grant(&self, fixture: &RuntimeCluster, revision: u64) -> Result<()> { let history = fixture.cluster.birth.route().history; for transport in &fixture.transports { transport.grant_query_origin(NamespaceQueryOrigin { @@ -90,7 +90,7 @@ impl Router { Ok(()) } - async fn close(self) -> Result<()> { + pub(super) async fn close(self) -> Result<()> { self.transport.close().await; self.runtime.close().await?; drop(self.events); diff --git a/crates/cellar/tests/support/cell_group_remote_commands.rs b/crates/cellar/tests/support/cell_group_remote_commands.rs new file mode 100644 index 000000000..d59e6de45 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_commands.rs @@ -0,0 +1,319 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real TLS ingress never inherits write authority from read grants or membership. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn invocation( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + retry_until: u64, + input: &[u8], +) -> Result { + let mut command = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([211; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 8, + }, + None, + ) + .await?; + command.begin().await?; + Ok(command) +} + +fn grant(fixture: &RuntimeCluster, router: &Router, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: history.cluster, + namespace: history.namespace, + revision, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn remote_commands() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + let mut result = ctx.inputs().random.to_vec(); + result.extend_from_slice(&ctx.inputs().now_unix_millis.to_be_bytes()); + result.extend_from_slice(input); + Ok(result) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let options = fixture.cell_options(&a).await?; + assert!(matches!( + fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + options.clone() + ), + Err(TransportError::Identity) + )); + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + options, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let deadline = command.attempt().unwrap().deadline_ms(); + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut wrong_process = router.peer.clone(); + wrong_process.process = ProcessGeneration::new([213; 16])?; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: wrong_process, + })?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + fixture.transports[0].revoke_command_origin(a.cluster, a.namespace, router.peer.node, 2)?; + router.grant(&fixture, 1)?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + // Voter membership also does not confer write permission. + let mut voter = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + assert!(voter + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await + .is_err()); + drop(voter); + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: NamespaceId::new([212; 16])?, + revision: 1, + peer: router.peer.clone(), + })?; + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + grant(&fixture, &router, 3)?; + command.retry().await?; + assert_eq!(command.attempt().unwrap().deadline_ms(), deadline); + fixture.transports[1].grant_command_origin(NamespaceCommandOrigin { + cluster: a.cluster, + namespace: a.namespace, + revision: 1, + peer: router.peer.clone(), + })?; + let follower_gateway = fixture.transports[1].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + fixture.cell_options_at(&a, 1).await?, + )?; + let mut follower = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let refusal = follower + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .err() + .context("follower refuses")?; + assert!( + matches!( + refusal.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::NotLeader { .. } + )) + ), + "{refusal:?}" + ); + assert_eq!(calls.load(Ordering::SeqCst), 0); + drop(follower); + drop(follower_gateway); + let result = command + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + let bytes = result.result().to_vec(); + assert_eq!( + result.command(), + CommandId::new(ClientId::new([211; 16])?, 1)? + ); + assert_eq!(result.retry_until_ms(), retry_until); + assert_eq!(result.observation().version.history, a); + assert_eq!(result.observation().version.sequence.get(), 2); + assert_eq!(result.durability(), Durability::Quorum); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(router.transport.usage().query_items, 1); + assert!(router.transport.usage().query_bytes > bytes.len()); + drop(result); + command.retry().await?; + assert_eq!( + command + .execute_remote(&router.transport, MemberId::new(1)?) + .await? + .result(), + bytes + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let mut conflict = invocation(&dispatch, &class, &a, retry_until, b"other").await?; + let error = conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err() + .context("conflicting receipt")?; + assert!(error.ambiguous); + assert!(matches!( + *error.cause, + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + drop(conflict); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(gateway); + command.retry().await?; + assert!(command + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(command); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn remote_command_lifecycle(revoke: bool) -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, _| { + if count.fetch_add(1, Ordering::SeqCst) == 0 { + let _ = entered.lock().unwrap().take().context("signal")?.send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + fixture.cell_options(&a).await?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + // No read-origin grant is installed: write permission is independent. + grant(&fixture, &router, 1)?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + command + .execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + assert_eq!(controllers[0].usage().retained_commands, 1); + if revoke { + fixture.transports[0].revoke_command_origin( + a.cluster, + a.namespace, + router.peer.node, + 2, + )?; + let error = tokio::time::timeout(Duration::from_secs(5), task) + .await?? + .err() + .context("revoked reply")?; + assert!(error.ambiguous); + assert!(grant(&fixture, &router, 1).is_err()); + } else { + task.abort(); + assert!(task.await.err().context("cancelled task")?.is_cancelled()); + } + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + release.send(())?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + if revoke { + grant(&fixture, &router, 3)?; + } + let mut retry = invocation(&dispatch, &class, &a, retry_until, b"once").await?; + let result = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(result.result(), b"committed"); + assert_eq!(result.observation().version.sequence.get(), 2); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(result); + drop(retry); + drop(gateway); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 45bc77e2f..7c1f58a54 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -711,3 +711,5 @@ pub(super) use sql::{remote_queries, remote_query_lifecycle}; pub(super) use sql::{nonmember_queries, nonmember_query_revocation}; pub(super) use sql::cached_dispatch_execution; + +pub(super) use sql::{remote_command_lifecycle, remote_commands}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f7038c969..cdb37e2de 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -223,7 +223,7 @@ pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { fixture.runtimes[1] .execution_authority(route.history, a.clone()) .await, - Err(RuntimeError::NotReady) + Err(RuntimeError::NotLeader) )); let mut wrong = a.clone(); wrong.incarnation = IncarnationId::new([99; 16])?; @@ -556,3 +556,7 @@ pub(crate) use query_origins::{nonmember_queries, nonmember_query_revocation}; #[path = "cell_group_dispatch_cache_execution.rs"] mod dispatch_cache_execution; pub(crate) use dispatch_cache_execution::cached_dispatch_execution; + +#[path = "cell_group_remote_commands.rs"] +mod remote_commands; +pub(crate) use remote_commands::{remote_command_lifecycle, remote_commands}; From 8852b705d80bdf0cb181b671897f9a33be77d016 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 09:37:16 -0700 Subject: [PATCH 105/568] docs: record qualified remote command checkpoint --- docs/design/multi-raft/cell-groups/README.md | 8 +++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 23 +++++++- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 59 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 22 ++++++- docs/design/multi-raft/cell-groups/tasks.json | 45 +++++++++++++- 7 files changed, 155 insertions(+), 8 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index c906ba683..a4f928bab 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -212,6 +212,14 @@ admission. The partial gate passes 202 Rust cases and eleven commands; transfer 4 executables). Write-authorized RPC and safe receipt lookup are next; removing destination catalog reads requires replicated transfer seals. CG-04 stays IN_PROGRESS. +At `5fd44437`, remote commands use explicit namespace/process write grants and +an authenticated command purpose on the bounded TLS RPC lane. Existing controller +owners retain admitted work after disconnect/revocation, and committed duplicates +return the same receipt without rerunning SQL. The partial gate passes 209 Rust +cases and eleven commands; transfer `.cellar/cell-groups-evidence/CG-04-command-rpc/` +(273 source inputs, 4 executables). Remote enrollment, standalone receipt lookup +and the public SDK cutover remain open; CG-04 remains IN_PROGRESS. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 915be190c..eabf0df96 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "06652e36869144624e0ca4d860782ef512018791", + "source_revision": "5fd4443796a6e990fbe70c16acb7fa5578b7e763", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index e50b96e83..a8306a554 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,22 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `06652e36`, adding bounded range hints +The newest reviewed source baseline is `5fd44437`, adding remote commands over +the bounded TLS application RPC lane. Mandatory authenticated command purpose and +independent namespace/process write grants prevent read/voter authority from +becoming write permission. The exact controller owns accepted work through +response loss or revocation, verifies fresh catalog binding and real leader +execution authority, and returns committed receipts with logical cell tokens. +Three both-image native cases cover nonmember ingress, grant/identity/follower +refusals, duplicates/conflicts, held-SQL disconnect/revocation and cleanup. Wire +and transport tests cover bounded independent grants and authenticated payloads. +The record is `.cellar/cell-groups-evidence/CG-04-command-rpc/checkpoint.json`: +209 Rust cases, eleven commands, 273 source inputs and 4 executables. Transfer +this directory with prior evidence. CG-04 remains IN_PROGRESS. Next implement +remote enrollment and standalone safe receipt lookup, then public-host discovery/ +SDK binding, dynamic roster and per-group isolation. Destination caching still +requires replicated transfer seals; shared RustFS/load remains unqualified. +The preceding reviewed source baseline is `06652e36`, adding bounded range hints shared across query, command and enrollment invocations. Entry/byte reservations, LRU eviction and verified head invalidation bound retention; a superseded in-flight lookup cannot refill the cache. Three catalog cases measure real metadata reads; @@ -476,8 +491,10 @@ Do not let plan maintenance replace implementation. cases. The destination retains per-RPC catalog admission: removing that read requires replicated transfer seals and serving-admission invalidation, linked to CG-09. Do not substitute a TTL or cached hint for that authority contract. - Keep query-origin grants read-only; command/enrollment RPC needs explicit write - authorization, response-independent producers and safe remote receipt lookup. + Remote commands now use explicit write grants and response-independent + controller producers. Keep read grants independent. Add remote enrollment and + standalone safe receipt lookup without treating missing local state as proof + of non-commit or replaying unresolved callbacks. The old host refuses registered initializers until its actual replacement; do not remove that refusal without a functioning shared serving path. Request and namespace S3 minima must continue refusing until CG-06 provides real archive diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 39416f429..5c7e48627 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 62 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 65 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 8b00cda15..8ee2e1acf 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,65 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 write-authorized remote commands + +Qualified source: `5fd4443796a6e990fbe70c16acb7fa5578b7e763`. + +`CellCommandInvocation::execute_remote` sends one consumed catalog attempt over +the bounded application RPC lane. The authenticated route has a mandatory command +purpose; command payloads/results have distinct marker-1 domains. Read grants and +voter membership never create write permission. `NamespaceCommandOrigin` grants +one namespace to an exact physical process, with independently bounded revision +tombstones. Endpoint checks, rooted process refresh, fleet revocation and peer-pool +retention include command grants. Discovery still allocates no local replica. + +A host-owned `CellCommandGateway` binds the exact controller/runtime owner. The +request carries the caller's captured time/random inputs, immutable command +identity, operation contract, retry horizon and remaining deadline. The receiver +controller checks the current catalog group/schema/epoch/birth and actual leader +authority before SQL. It retains admitted work independently from the connection; +disconnect, timeout or write-grant revocation cannot undo an admitted mutation or +release its physical producer. Final registration drop refuses new commands. + +Success returns the exact committed command identity/result and authenticated +logical cell observation/token. Reply admission remains charged until consumption. +Original deadlines/attempt budgets survive retries. Conflict, expiry, sequence-gap +and not-leader refusals remain typed. Once remote transmission may occur, errors +remain conservatively ambiguous; neither admission nor response loss proves +non-commit. Committed duplicate retries return the stored result without another +application callback. + +Three new native cases run both image modes over four TLS endpoints in one OS +process. A nonmember router with no group storage tests missing/read-only, +wrong-process and foreign-namespace write permission, voter non-authority, follower +refusal, exact controller binding, committed duplicate/conflict handling, reply +reservations and registration drop. Held physical SQL tests disconnect and grant +revocation: the controller keeps ownership, then resolves the command and returns +its committed receipt on retry without a second callback. Three wire cases cover +captured inputs, payload/purpose separation, exact command/history/token binding +and typed failures. A transport case checks bounded tombstones, stale revision +refusal, independent read grants and pool cleanup. + +All 209 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight and +37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-command-rpc/checkpoint.json` binds 273 source +inputs and 4 copied executables. Transfer that entire directory with prior +checkpoints. Targeted compile/test logs are retained as diagnostic evidence. +The initial follower assertion ran without a follower gateway and failed before +leadership admission. After adding the gateway, diagnostics exposed follower +authority requests returning NotReady and becoming generic Unavailable remotely. +The owner now reports NotLeader for that role, with leader readiness and stale +authority checks preserved. Direct and remote tests assert the precise refusal. + +CG-04 remains IN_PROGRESS. Remote enrollment and standalone safe receipt lookup +are the next request-path gaps; public-host automatic discovery/SDK binding, lazy +birth/dynamic roster, per-group isolation and configuration lifecycle remain. +The RPC lane serializes exchanges per peer; no load or latency gain is claimed. +Destination catalog reads remain until replicated transfer sealing and serving +invalidation are integrated. Complete-group snapshots/archive, shared RustFS +recovery, independent-process failure, power loss and load remain unqualified. + ## CG-04 bounded invocation route cache Qualified source: `06652e36869144624e0ca4d860782ef512018791`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 24969074a..a4849113a 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -130,9 +130,29 @@ removed. A replaced discovery view refuses captured requests and late replies; removal does not revoke the destination's independently managed namespace grant. Grant revocation cancels read waiters without releasing physically entered SQL. The public host must supply trusted discovery and explicit permissions; automatic -discovery/SDK binding and write-authorized command/enrollment RPC remain separate +discovery/SDK binding, remote enrollment and standalone receipt lookup remain integration work. +Remote commands use the same bounded application RPC lane with a mandatory, +MAC-bound command purpose and distinct payload/result markers. A read grant or +voter membership never grants write admission. `NamespaceCommandOrigin` supplies +an explicit namespace/process permission with bounded revision tombstones; +endpoint checks, rooted process refresh and fleet revocation include these grants. +A host-owned command gateway binds the exact node controller. Dropping its final +registration closes new admission without rolling back accepted work. + +Carry the caller's captured time/random inputs, command identity, request contract, +retry horizon and remaining deadline to the destination. The controller verifies +a fresh catalog route against the transmitted group, schema, epoch and birth, +then checks actual leader authority before entering SQL. It owns accepted work +independently from the connection. Disconnect, revocation and timeout can suppress +a reply but cannot prove non-commit or release a producer. Success contains the +exact committed command identity/result and an authenticated logical cell token; +conflict, expiration and sequence-gap refusals remain typed. Retain caller reply +byte admission through consumption. Retry the same identity and input under the +original budget; a committed duplicate must not re-execute application code. +The current lane serializes RPC exchanges per peer; this is not load qualification. + ## Apply, checkpoint and archive For entry N, verify the cell base/sequence, stage the result, sync its immutable diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index a801ef626..7ee7e85c7 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -513,7 +513,12 @@ "crates/cellar/tests/support/cell_group_query_origins.rs", "crates/cellar/durability/raft/cell_groups/directory/route_cache.rs", "crates/cellar/tests/support/cell_group_routing_cache.rs", - "crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs" + "crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs", + "crates/cellar/durability/raft/command_origins.rs", + "crates/cellar/durability/raft/command_cell_wire.rs", + "crates/cellar/durability/raft/command_cell_transport.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs", + "crates/cellar/tests/support/cell_group_remote_commands.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -658,6 +663,14 @@ "tests": [ "shared_cached_dispatch_preserves_authority_during_catalog_outage" ] + }, + { + "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Remote enrollment and standalone safe receipt lookup remain separate required integration. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", + "tests": [ + "shared_remote_commands_require_write_grants_and_exact_receipts", + "shared_remote_commands_survive_disconnected_sql_waiters", + "shared_remote_command_revocation_preserves_admitted_ownership" + ] } ], "test_cases": [ @@ -1249,6 +1262,36 @@ "I05", "I12" ] + }, + { + "name": "shared_remote_commands_require_write_grants_and_exact_receipts", + "oracle": "Over real TLS and both image modes, a router without local group storage cannot invoke commands using discovery, a read grant, a wrong-process or foreign-namespace write grant, or voter membership. A fully registered follower with write permission still refuses before SQL. Explicit namespace/process write permission at the leader permits a committed receipt with authenticated cell observation. Same-identity retries return that receipt without another callback; conflicting input refuses. Original attempt deadlines, exact controller ownership, response byte admission and host-registration drop remain enforced.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_commands_survive_disconnected_sql_waiters", + "oracle": "Hold entered command SQL behind a physical callback gate, disconnect the remote waiter, and verify the destination controller retains the producer. Release SQL, observe ownership resolve, then retry the exact command and obtain one committed effect in both image modes. No fixture event pumping or local router replica is used.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_command_revocation_preserves_admitted_ownership", + "oracle": "During a held admitted command, revoke the explicit write grant. The remote response terminates ambiguously while controller ownership remains. Stale grant revisions refuse; after physical completion a newer grant allows the original committed receipt to be returned without another callback. Run both image modes with no read grant.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From 0fe665d0267f1b831b172aad5e8d435f5bf0cd39 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 09:47:43 -0700 Subject: [PATCH 106/568] feat: route enrollment through shared write ownership --- .../raft/cell_groups/dispatch/command/mod.rs | 2 +- .../cell_groups/dispatch/command/remote.rs | 2 +- .../raft/cell_groups/dispatch/enrollment.rs | 88 +++-- .../cell_groups/dispatch/enrollment_remote.rs | 142 +++++++ .../raft/cell_groups/dispatch/mod.rs | 4 +- .../cellar/durability/raft/cell_groups/mod.rs | 5 +- .../durability/raft/cell_groups/owner.rs | 3 + .../durability/raft/command_cell_transport.rs | 261 +++++++++++-- .../durability/raft/enrollment_cell_wire.rs | 154 ++++++++ .../durability/raft/query_cell_transport.rs | 8 + .../cellar/durability/raft/query_identity.rs | 35 +- .../cellar/durability/raft/query_transport.rs | 9 +- crates/cellar/durability/raft/query_wire.rs | 2 + .../durability/raft/query_wire_tests.rs | 166 ++++++++- crates/cellar/durability/raft/transport.rs | 4 +- .../durability/raft/transport_options.rs | 2 +- .../tests/native_cell_groups_execution.rs | 20 + .../support/cell_group_remote_enrollment.rs | 346 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + 20 files changed, 1178 insertions(+), 81 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs create mode 100644 crates/cellar/durability/raft/enrollment_cell_wire.rs create mode 100644 crates/cellar/tests/support/cell_group_remote_enrollment.rs diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index b54d74997..d0b22cc2e 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -101,7 +101,7 @@ pub struct CellCommandInvocation { dispatched: bool, ambiguous: bool, origin: Option, - remote_route: Option, + remote_route: Option, _admission: Permit, } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs index f5581e8ff..640090b88 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -7,7 +7,7 @@ use crate::raft::transport::{CellRemoteCommandResponse, RaftTransport, Transport use cellar_logic::consensus::MemberId; impl CellCommandInvocation { - pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteCommandRoute) { + pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteMutationRoute) { self.remote_route = Some(route); } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index fee282622..d21949ed7 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -1,6 +1,9 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Routed creation identity and controller-owned initialization, never a cell task. +#[path = "enrollment_remote.rs"] +mod remote; + use super::*; use crate::raft::execution::CellEnrollmentAttempt; use crate::storage::CommandInputs; @@ -51,6 +54,9 @@ pub struct CellEnrollmentInvocation { inputs: CommandInputs, session: CellRoutingSession, started: bool, + attempt: Option, + dispatched: bool, + remote_route: Option, ambiguous: bool, origin: Option, _admission: Permit, @@ -63,6 +69,18 @@ impl CellGroupDispatcher { request: CellEnrollmentRequest, limits: CellRoutingLimits, cached: Option, + ) -> Result { + self.enroll_with_inputs(cell, request, limits, cached, None) + .await + } + + pub(crate) async fn enroll_with_inputs( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option<(CommandInputs, ImageId)>, ) -> Result { let error = |cause| CellEnrollmentError { creation: request.creation, @@ -84,27 +102,32 @@ impl CellGroupDispatcher { .budget .acquire(8192) .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; - let (inputs, preview) = self - .domain - .run(async { - use rand::RngCore as _; - let mut rng = crate::asyncrt::rng("cell-group-enrollment-inputs"); - let mut random = [0; 32]; - let mut preview = [0; 16]; - rng.fill_bytes(&mut random); - rng.fill_bytes(&mut preview); - ( - CommandInputs { - now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, - random, - }, - ImageId::new(preview), - ) - }) - .await - .map_err(|_| error(closed()))?; - let preview = - preview.map_err(|_| error(CellDispatchError::Invalid("enrollment preview")))?; + let (inputs, preview) = if let Some(captured) = captured { + captured + } else { + let (inputs, preview) = self + .domain + .run(async { + use rand::RngCore as _; + let mut rng = crate::asyncrt::rng("cell-group-enrollment-inputs"); + let mut random = [0; 32]; + let mut preview = [0; 16]; + rng.fill_bytes(&mut random); + rng.fill_bytes(&mut preview); + ( + CommandInputs { + now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, + random, + }, + ImageId::new(preview), + ) + }) + .await + .map_err(|_| error(closed()))?; + let preview = + preview.map_err(|_| error(CellDispatchError::Invalid("enrollment preview")))?; + (inputs, preview) + }; let mut hash = Sha256::new(); hash.update(b"cellar-cell-enrollment-invocation-1\0"); hash.update(cell_image_history_digest(&cell)); @@ -137,6 +160,9 @@ impl CellGroupDispatcher { inputs, session, started: false, + attempt: None, + dispatched: false, + remote_route: None, ambiguous: false, origin: None, _admission: admission, @@ -195,11 +221,21 @@ impl CellEnrollmentInvocation { roster: &BTreeSet, progress: &mut watch::Receiver<()>, ) -> Result { - let attempt = self - .session - .begin() - .await - .map_err(CellDispatchError::Routing)?; + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|expected| !expected.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid( + "remote enrollment destination route", + )); + } self.session .validate_attempt(&attempt) .map_err(CellDispatchError::Routing)?; diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs new file mode 100644 index 000000000..b959e9b48 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs @@ -0,0 +1,142 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Remote creation retries preserve the captured identity, inputs and route budget. +use super::*; +use crate::raft::query_wire::enrollment::CellEnrollmentCall; +use crate::raft::transport::{CellRemoteEnrollmentResponse, RaftTransport, TransportError}; +use cellar_logic::consensus::MemberId; + +impl CellEnrollmentInvocation { + pub(crate) fn bind_remote_route(&mut self, route: super::super::RemoteMutationRoute) { + self.remote_route = Some(route); + } + + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + /// Select a remote attempt. Once selected, this invocation cannot be handed + /// to a local controller as a fresh creation, even after transport failure. + pub async fn begin(&mut self) -> Result<(), CellEnrollmentError> { + if self.origin.is_some() || self.remote_route.is_some() { + return Err(self.error(CellDispatchError::Invalid("local enrollment ownership"))); + } + self.started = true; + self.dispatched = true; + let attempt = self + .session + .begin() + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellEnrollmentError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let next = self + .session + .retry(&previous) + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(next); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellEnrollmentError> { + let previous = self.current()?.clone(); + self.dispatched = true; + let next = self + .session + .redirect(&previous, hint) + .await + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.attempt = Some(next); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellEnrollmentError> { + self.attempt + .as_ref() + .ok_or_else(|| self.error(CellDispatchError::Invalid("missing enrollment route"))) + } + + /// One consumed remote attempt. Possible transmission makes errors sticky + /// ambiguous; neither a lost reply nor admission refusal proves non-commit. + pub async fn execute_remote( + &mut self, + transport: &RaftTransport, + member: MemberId, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + if self.dispatched || self.origin.is_some() || self.remote_route.is_some() { + return Err(self.error(CellDispatchError::Invalid( + "consumed or local enrollment attempt", + ))); + } + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(self.error(CellDispatchError::Invalid("enrollment transport domain"))); + } + let now = self + .dispatcher + .domain + .mono_ms() + .map_err(|_| self.error(closed()))?; + let timeout_ms = self + .session + .deadline_ms() + .checked_sub(now) + .filter(|n| *n > 0) + .ok_or_else(|| { + self.error(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + })?; + let call = CellEnrollmentCall { + request: CellEnrollmentRequest { + creation: self.creation, + durability: Durability::Quorum, + }, + inputs: self.inputs, + preview: self.preview, + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + self.session + .mark_submitted(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + self.dispatched = true; + self.ambiguous = true; + let result = self + .dispatcher + .domain + .run(transport.enroll_cell( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + self.session.deadline_ms(), + )) + .await + .map_err(|_| self.error(closed()))?; + let response = result.map_err(|error| { + self.error(match error { + TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + }) + })?; + self.session + .validate_attempt(&attempt) + .map_err(|e| self.error(CellDispatchError::Routing(e)))?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 150093477..67721e85d 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -539,14 +539,14 @@ fn catalog_failure(error: crate::raft::catalog::CatalogError) -> CellDispatchErr } /// Receiver-side destination constraint, checked against a fresh catalog lookup. -pub(crate) struct RemoteCommandRoute { +pub(crate) struct RemoteMutationRoute { pub group: cellar_logic::cell_group::GroupHistory, pub schema: [u8; 32], pub epoch: u64, pub birth: [u8; 32], } -impl RemoteCommandRoute { +impl RemoteMutationRoute { fn matches(&self, route: &GroupRoute) -> bool { self.group == route.history && self.schema == route.schema_digest diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index eb82aff02..808798540 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -10,7 +10,8 @@ pub use super::host_sql::cell_groups::{ }; pub use crate::raft::transport::{ - CellCommandGateway, CellQueryGateway, CellRemoteCommandResponse, CellRemoteQueryResponse, + CellCommandGateway, CellQueryGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, + CellRemoteQueryResponse, }; mod apply; @@ -78,4 +79,4 @@ pub use owner::{ CellGroupProposal, CellGroupStatus, CellGroupTicket, }; -pub(crate) use dispatch::RemoteCommandRoute; +pub(crate) use dispatch::RemoteMutationRoute; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index a61d7165f..c9838b4dc 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -359,6 +359,9 @@ impl CellGroupOwner { cell: CellHistory, ) -> Result { let status = self.status()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } if !status.ready || status.applied.index != status.committed { return Err(RuntimeError::NotReady); } diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs index 212f57a55..8408b6056 100644 --- a/crates/cellar/durability/raft/command_cell_transport.rs +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -1,15 +1,17 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Commands share the bounded RPC lane but require independent write grants. +//! Commands and enrollment share bounded RPC admission and explicit write grants. use super::*; use crate::application::consensus::CommandFailure; use crate::raft::cell_groups::{ CellDispatchError, CellGroupController, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, + EnrollmentRecord, }; use crate::raft::query_identity::QueryRoute; use crate::raft::query_wire::{ self as wire, command::{CellCommandCall, CellCommandValue}, + enrollment::CellEnrollmentCall, }; use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; use cellar_logic::{ @@ -82,15 +84,79 @@ impl CellRemoteCommandResponse { } } -pub(super) struct SendCellCommand { +/// An authenticated creation receipt, retaining transport admission until dropped. +/// The original enrollment image is not a claim about the latest cell version. +pub struct CellRemoteEnrollmentResponse { + record: EnrollmentRecord, + served_by: MemberId, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteEnrollmentResponse { + pub fn record(&self) -> &EnrollmentRecord { + &self.record + } + + pub fn served_by(&self) -> MemberId { + self.served_by + } + + pub fn durability(&self) -> cellar_logic::consensus::Durability { + cellar_logic::consensus::Durability::Quorum + } +} + +enum WriteCall { + Command(CellCommandCall), + Enrollment(CellEnrollmentCall), +} + +impl WriteCall { + fn encode(&self, limit: usize) -> Result, TransportError> { + match self { + Self::Command(c) => c.encode(limit), + Self::Enrollment(c) => c.encode(limit), + } + } + + fn shorten_timeout(&mut self, remaining: u64) { + let timeout = match self { + Self::Command(c) => &mut c.timeout_ms, + Self::Enrollment(c) => &mut c.timeout_ms, + }; + *timeout = (*timeout).min(remaining); + } + + fn input_bytes(&self) -> usize { + match self { + Self::Command(c) => c.request.input.len(), + Self::Enrollment(_) => 0, + } + } +} + +enum WriteValue { + Command(CellCommandValue), + Enrollment(EnrollmentRecord), +} + +struct WriteResponse { + value: WriteValue, + served_by: MemberId, + node: Permit, + peer: Permit, +} + +pub(super) struct SendCellWrite { route: Arc>, peer: RaftPeer, cell: CellHistory, - call: CellCommandCall, + call: WriteCall, keys: VersionKeys, deadline: u64, cancel: watch::Receiver, - reply: oneshot::Sender>, + reply: oneshot::Sender>, node: Permit, peer_bytes: Permit, } @@ -104,7 +170,7 @@ impl Drop for Cancel { } impl RaftTransport { - /// Register compiled commands on the exact controller/runtime owner. Retain + /// Register compiled commands and initialization on the exact controller/runtime owner. Retain /// this host-owned handle while accepting commands. Dropping it refuses new /// admission; commands already transferred to the controller remain owned. /// Namespace write grants are independently required even for voter peers. @@ -160,6 +226,65 @@ impl RaftTransport { keys: VersionKeys, deadline: u64, ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Command(call), + keys, + deadline, + ) + .await?; + let WriteValue::Command(value) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteCommandResponse { + value, + _node: response.node, + _peer: response.peer, + }) + } + + pub(in crate::raft) async fn enroll_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellEnrollmentCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Enrollment(call), + keys, + deadline, + ) + .await?; + let WriteValue::Enrollment(record) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteEnrollmentResponse { + record, + served_by: response.served_by, + _node: response.node, + _peer: response.peer, + }) + } + + async fn write_cell( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: WriteCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { let state = &self.handle.0; // Discovery never creates a local replica or grants voting authority. state.runtime.shared()?; @@ -195,9 +320,7 @@ impl RaftTransport { return Err(TransportError::Timeout); } let charge = queries::reservation( - call.request - .input - .len() + call.input_bytes() .checked_add(8192) .ok_or(TransportError::Limit)?, &state.options, @@ -220,7 +343,7 @@ impl RaftTransport { let (cancel, cancelled) = watch::channel(false); let _cancel = Cancel(cancel); pool.query - .try_send(SendQuery::Command(Box::new(SendCellCommand { + .try_send(SendQuery::Write(Box::new(SendCellWrite { route: view, peer, cell: cell.clone(), @@ -293,10 +416,10 @@ impl State { Ok((pool, Some(grant.stop.subscribe()))) } - pub(super) async fn send_cell_command( + pub(super) async fn send_cell_write( &self, pool: &Pool, - mut request: SendCellCommand, + mut request: SendCellWrite, connection: &mut Option, ) -> bool { let result = self @@ -314,7 +437,7 @@ impl State { .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) .filter(|n| *n > 0) .ok_or(TransportError::Timeout)?; - request.call.timeout_ms = request.call.timeout_ms.min(remaining); + request.call.shorten_timeout(remaining); let body = request.call.encode(self.options.max_query_request_bytes)?; let local = self.runtime.log().limits(); let route = QueryRoute::new_shared( @@ -328,25 +451,41 @@ impl State { &request.peer.label, &request.cell, request.route.schema_digest, - )? - .for_command()?; + )?; + let route = match &request.call { + WriteCall::Command(_) => route.for_command()?, + WriteCall::Enrollment(_) => route.for_enrollment()?, + }; let body = self .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) .await?; if !self.current_cell_query_route(&request.route, &request.peer) { return Err(TransportError::StalePeer); } - let now = u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| TransportError::Authentication)?; - wire::command::decode_result( - &body, - (request.call.request.id, &request.cell), - request.route.schema_digest, - &request.keys, - now, - self.options.max_query_response_bytes, - request.peer.replica.member, - )? + match &request.call { + WriteCall::Command(call) => { + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::command::decode_result( + &body, + (call.request.id, &request.cell), + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map(WriteValue::Command) + } + WriteCall::Enrollment(call) => wire::enrollment::decode_result( + &body, + call.request.creation, + &request.cell, + request.route.schema_digest, + self.options.max_query_response_bytes, + )? + .map(WriteValue::Enrollment), + } .map_err(TransportError::Command) }, ) @@ -354,18 +493,74 @@ impl State { // A fully authenticated application refusal consumed a complete reply; // retain the peer connection just as the legacy query lane does. let failed = matches!(&result, Err(error) if !matches!(error, TransportError::Command(_))); - let _ = request - .reply - .send(result.map(|value| CellRemoteCommandResponse { - value, - _node: request.node, - _peer: request.peer_bytes, - })); + let _ = request.reply.send(result.map(|value| WriteResponse { + value, + served_by: request.peer.replica.member, + node: request.node, + peer: request.peer_bytes, + })); failed } } impl State { + pub(super) async fn execute_cell_enrollment( + &self, + route: &QueryRoute, + call: CellEnrollmentCall, + ) -> Result, TransportError> { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let binding = crate::raft::cell_groups::RemoteMutationRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .enroll_with_inputs( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + Some((call.inputs, call.preview)), + ) + .await + .map_err(|error| command_failure(&error.cause))?; + invocation.bind_remote_route(binding); + if !gateway.accepting.load(Ordering::Acquire) { + return Err(CommandFailure::Unavailable); + } + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + let call = gateway + .controller + .enroll(invocation, gateway.options.clone()) + .map_err(|error| command_failure(&error.error.cause))?; + let response = call + .response() + .await + .map_err(|error| command_failure(&error.cause))?; + Ok(response.record().clone()) + } + .await; + wire::enrollment::encode_result(&result, self.options.max_query_response_bytes) + } + pub(super) async fn execute_cell_command( &self, route: &QueryRoute, @@ -382,7 +577,7 @@ impl State { if !gateway.accepting.load(Ordering::Acquire) { return Err(TransportError::Refused); } - let binding = crate::raft::cell_groups::RemoteCommandRoute { + let binding = crate::raft::cell_groups::RemoteMutationRoute { group: target.scope, schema: route.schema(), epoch: call.route_epoch, diff --git a/crates/cellar/durability/raft/enrollment_cell_wire.rs b/crates/cellar/durability/raft/enrollment_cell_wire.rs new file mode 100644 index 000000000..8ad707819 --- /dev/null +++ b/crates/cellar/durability/raft/enrollment_cell_wire.rs @@ -0,0 +1,154 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Creation payloads use a distinct authenticated RPC purpose and exact receipt. +use super::*; +use crate::raft::cell_groups::{CellEnrollmentRequest, EnrollmentRecord}; +use crate::storage::CommandInputs; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{Durability, ProposalId}, +}; +use cellar_ltx::generation::ImageId; + +pub(in crate::raft) struct CellEnrollmentCall { + pub request: CellEnrollmentRequest, + pub inputs: CommandInputs, + pub preview: ImageId, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +impl CellEnrollmentCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if !(1..=3_600_000).contains(&self.timeout_ms) + || self.request.durability != Durability::Quorum + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCEQ01".to_vec(); + body.extend_from_slice(&self.request.creation.bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + body.push(1); + body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); + body.extend_from_slice(&self.inputs.random); + body.extend_from_slice(&self.preview.bytes()); + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCEQ01" { + return Err(TransportError::Protocol); + } + let creation = ProposalId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if body.byte()? != 1 { + return Err(TransportError::Protocol); + } + let inputs = CommandInputs { + now_unix_millis: u64::from_be_bytes(body.array()?), + random: body.array()?, + }; + let preview = ImageId::new(body.array()?).map_err(|_| TransportError::Identity)?; + let call = Self { + request: CellEnrollmentRequest { + creation, + durability: Durability::Quorum, + }, + inputs, + preview, + route_epoch, + birth_digest, + timeout_ms: u64::from_be_bytes(body.array()?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + // Fixed-size encoding validates the same domain constraints in both directions. + call.encode(limit)?; + Ok(call) + } +} + +pub(in crate::raft) fn encode_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCES01".to_vec(); + match result { + Ok(record) => { + let record = record.encode().map_err(|_| TransportError::Protocol)?; + body.push(0); + body.extend_from_slice(&(record.len() as u32).to_be_bytes()); + body.extend_from_slice(&record); + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(failure) => body.push(failure_code(failure)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +/// Only the authenticated exchange may call this decoder. A record alone is +/// creation data, never a signed version token or proof of the current cell image. +pub(in crate::raft) fn decode_result( + bytes: &[u8], + creation: ProposalId, + cell: &CellHistory, + schema: [u8; 32], + limit: usize, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCES01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + 0 => { + let length = u32::from_be_bytes(body.array()?) as usize; + let record = EnrollmentRecord::decode(body.take(length)?) + .map_err(|_| TransportError::Protocol)?; + if record.creation() != creation || record.cell() != cell || record.schema() != schema { + return Err(TransportError::Authentication); + } + Ok(record) + } + 9 => Err(CommandFailure::CommandConflict), + 12 => { + let member = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if member == 0 { + None + } else { + Some(MemberId::new(member).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index f73d76a3e..0ad67dc10 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -358,6 +358,7 @@ pub(super) enum IncomingCall { Legacy(QueryCall), Shared(CellQueryCall), Command(crate::raft::query_wire::command::CellCommandCall), + Enrollment(crate::raft::query_wire::enrollment::CellEnrollmentCall), } impl IncomingCall { @@ -366,6 +367,11 @@ impl IncomingCall { bytes: &[u8], limit: usize, ) -> Result { + if route.is_enrollment() { + return Ok(Self::Enrollment( + crate::raft::query_wire::enrollment::CellEnrollmentCall::decode(bytes, limit)?, + )); + } if route.is_command() { return Ok(Self::Command( crate::raft::query_wire::command::CellCommandCall::decode(bytes, limit)?, @@ -383,6 +389,7 @@ impl IncomingCall { Self::Legacy(call) => call.options.timeout.as_millis() as u64, Self::Shared(call) => call.timeout_ms, Self::Command(call) => call.timeout_ms, + Self::Enrollment(call) => call.timeout_ms, } } @@ -393,6 +400,7 @@ impl IncomingCall { ) -> Result, TransportError> { match self { Self::Command(call) => state.execute_cell_command(route, call).await, + Self::Enrollment(call) => state.execute_cell_enrollment(route, call).await, Self::Legacy(call) => { let (_, target) = route.decode()?; let handle = state diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index 9f058206a..adaf11493 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -55,10 +55,17 @@ enum History { }, } +#[derive(Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +enum Purpose { + Query, + Command, + Enrollment, +} + #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct QueryRoute { - command: bool, + purpose: Purpose, scope: History, schema: [u8; 32], source: Endpoint, @@ -74,7 +81,7 @@ impl QueryRoute { schema: [u8; 32], ) -> Result { let route = Self { - command: false, + purpose: Purpose::Query, scope: History::Cell(WireScope::from(&target.scope)), schema, source: Endpoint::new(source, source_label, None), @@ -87,7 +94,7 @@ impl QueryRoute { /// Request orientation only. A reversed response route is compared with /// the exact outstanding request; it cannot decode as another request. pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { - if self.command { + if self.is_write() { return Err(TransportError::Identity); } let History::Cell(scope) = &self.scope else { @@ -142,7 +149,7 @@ impl QueryRoute { } let g = target.scope; let route = Self { - command: false, + purpose: Purpose::Query, scope: History::Shared { group: [ g.cluster.bytes(), @@ -190,12 +197,26 @@ impl QueryRoute { pub fn for_command(mut self) -> Result { self.decode_shared()?; - self.command = true; + self.purpose = Purpose::Command; Ok(self) } pub fn is_command(&self) -> bool { - self.command + self.purpose == Purpose::Command + } + + pub fn for_enrollment(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::Enrollment; + Ok(self) + } + + pub fn is_enrollment(&self) -> bool { + self.purpose == Purpose::Enrollment + } + + pub fn is_write(&self) -> bool { + self.is_command() || self.is_enrollment() } pub fn is_shared(&self) -> bool { @@ -230,7 +251,7 @@ impl QueryRoute { pub fn reversed(&self) -> Self { Self { - command: self.command, + purpose: self.purpose, scope: self.scope.clone(), schema: self.schema, source: self.target.clone(), diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 0458dbbf1..636dd8169 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -40,7 +40,7 @@ impl Drop for QueryDelivery { pub(super) enum SendQuery { Legacy(Box), Shared(Box), - Command(Box), + Write(Box), } pub(super) struct LegacySendQuery { @@ -395,7 +395,7 @@ impl State { &self, wire: &QueryRoute, ) -> Result<(Arc, Option>), TransportError> { - if wire.is_command() { + if wire.is_write() { return self.authorize_cell_command(wire); } if wire.is_shared() { @@ -727,9 +727,8 @@ impl State { })); failed } - SendQuery::Command(request) => { - self.send_cell_command(&pool, *request, &mut connection) - .await + SendQuery::Write(request) => { + self.send_cell_write(&pool, *request, &mut connection).await } SendQuery::Shared(request) => { self.send_cell_query(&pool, *request, &mut connection).await diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index 1f002c4c6..fa7c62732 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -21,6 +21,8 @@ use std::time::Duration; pub(super) mod cell; #[path = "command_cell_wire.rs"] pub(super) mod command; +#[path = "enrollment_cell_wire.rs"] +pub(super) mod enrollment; pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index b70a74e3d..d3a091cc0 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -714,7 +714,7 @@ fn shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply() { let request = RequestHead::new(write.clone(), 3, &body); let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); let mut value = serde_json::to_value(&request).unwrap(); - value["route"]["command"] = serde_json::json!(false); + value["route"]["purpose"] = serde_json::json!("Query"); let changed = FrameHead { prefix: head.prefix, header: serde_json::to_vec(&value).unwrap(), @@ -744,7 +744,7 @@ fn shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply() { ) .is_err()); let mut missing = serde_json::to_value(write).unwrap(); - missing.as_object_mut().unwrap().remove("command"); + missing.as_object_mut().unwrap().remove("purpose"); assert!(serde_json::from_value::(missing).is_err()); } @@ -820,3 +820,165 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { assert_eq!(decoded.err(), Some(failure)); } } + +fn enrollment_call() -> enrollment::CellEnrollmentCall { + enrollment::CellEnrollmentCall { + request: crate::raft::cell_groups::CellEnrollmentRequest { + creation: cellar_logic::consensus::ProposalId::new([81; 16]).unwrap(), + durability: Durability::Quorum, + }, + inputs: crate::storage::CommandInputs { + now_unix_millis: 1234, + random: [82; 32], + }, + preview: cellar_ltx::generation::ImageId::new([83; 16]).unwrap(), + timeout_ms: 1000, + route_epoch: 3, + birth_digest: [84; 32], + } +} + +#[test] +fn shared_enrollment_wire_preserves_captured_identity_and_inputs() { + let call = enrollment_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = enrollment::CellEnrollmentCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.creation, call.request.creation); + assert_eq!(decoded.preview, call.preview); + assert_eq!(decoded.inputs.random, call.inputs.random); + assert_eq!(decoded.inputs.now_unix_millis, call.inputs.now_unix_millis); + for end in 0..bytes.len() { + assert!(enrollment::CellEnrollmentCall::decode(&bytes[..end], 4096).is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(enrollment::CellEnrollmentCall::decode(&extra, 4096).is_err()); + assert!(enrollment::CellEnrollmentCall::decode(&bytes, bytes.len() - 1).is_err()); + assert!( + enrollment::CellEnrollmentCall::decode(&command_call().encode(4096).unwrap(), 4096) + .is_err() + ); + assert!(command::CellCommandCall::decode(&bytes, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + assert!(call.encode(4096).is_err()); + call.request.durability = Durability::Quorum; + call.timeout_ms = 0; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read() { + let old = identity(2); + let target = ReplicaIdentity { + scope: cellar_logic::cell_group::GroupHistory::from(&old.scope), + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let read = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell_observation().version.history, + [9; 32], + ) + .unwrap(); + let enrollment = read.clone().for_enrollment().unwrap(); + assert!(enrollment.is_write()); + assert!(enrollment.is_enrollment()); + assert!(!enrollment.is_command()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = enrollment_call().encode(4096).unwrap(); + let request = RequestHead::new(enrollment, 4, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + for purpose in ["Query", "Command"] { + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["purpose"] = serde_json::json!(purpose); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + } + let bytes = enrollment::encode_result(&Err(CommandFailure::CommandConflict), 4096).unwrap(); + let response = response_head(&auth, &[3; 32], &request, &head.tag, &bytes, &options).unwrap(); + for route in [read.clone(), read.for_command().unwrap()] { + let other = RequestHead::new(route, 4, &body); + assert!(read_response(&response, &auth, &[3; 32], &other, &head.tag, &options).is_err()); + } +} + +#[test] +fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { + let call = enrollment_call(); + let cell = cell_observation().version.history; + let image = cell_observation().version.image; + let record = crate::raft::cell_groups::EnrollmentRecord::new( + cell.clone(), + call.request.creation, + [9; 32], + image, + ) + .unwrap(); + let bytes = enrollment::encode_result(&Ok(record.clone()), 4096).unwrap(); + assert_eq!( + enrollment::decode_result(&bytes, call.request.creation, &cell, [9; 32], 4096) + .unwrap() + .unwrap(), + record + ); + let mut other = cell.clone(); + other.incarnation = cellar_logic::consensus::IncarnationId::new([85; 16]).unwrap(); + assert!( + enrollment::decode_result(&bytes, call.request.creation, &other, [9; 32], 4096).is_err() + ); + assert!( + enrollment::decode_result(&bytes, call.request.creation, &cell, [8; 32], 4096).is_err() + ); + assert!(enrollment::decode_result( + &bytes, + cellar_logic::consensus::ProposalId::new([86; 16]).unwrap(), + &cell, + [9; 32], + 4096 + ) + .is_err()); + for end in 0..bytes.len() { + assert!(enrollment::decode_result( + &bytes[..end], + call.request.creation, + &cell, + [9; 32], + 4096 + ) + .is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!( + enrollment::decode_result(&extra, call.request.creation, &cell, [9; 32], 4096).is_err() + ); + for failure in [ + CommandFailure::CommandConflict, + CommandFailure::NotLeader { + leader: Some(MemberId::new(2).unwrap()), + }, + CommandFailure::Overloaded, + ] { + let bytes = enrollment::encode_result(&Err(failure.clone()), 4096).unwrap(); + assert_eq!( + enrollment::decode_result(&bytes, call.request.creation, &cell, [9; 32], 4096) + .unwrap() + .unwrap_err(), + failure + ); + } +} diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index 72bfbdcd5..b993a6e5a 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -41,7 +41,9 @@ mod cell_commands; mod cell_queries; #[path = "command_origins.rs"] mod command_origins; -pub use cell_commands::{CellCommandGateway, CellRemoteCommandResponse}; +pub use cell_commands::{ + CellCommandGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, +}; #[path = "query_cell_discovery.rs"] mod cell_query_discovery; #[path = "transport_control_send.rs"] diff --git a/crates/cellar/durability/raft/transport_options.rs b/crates/cellar/durability/raft/transport_options.rs index 47cf0dee4..566176353 100644 --- a/crates/cellar/durability/raft/transport_options.rs +++ b/crates/cellar/durability/raft/transport_options.rs @@ -31,7 +31,7 @@ pub struct NamespaceQueryOrigin { pub peer: RaftPeerNode, } -/// Explicit command permission for an exact namespace and physical process. +/// Explicit command and enrollment permission for an exact namespace/process. /// Read grants, discovery and Raft membership never create this permission. #[derive(Clone, Debug, PartialEq, Eq)] pub struct NamespaceCommandOrigin { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index c33559440..ea6a12b4c 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1510,3 +1510,23 @@ async fn shared_remote_commands_survive_disconnected_sql_waiters() -> Result<()> async fn shared_remote_command_revocation_preserves_admitted_ownership() -> Result<()> { Box::pin(runtime_fixture::remote_command_lifecycle(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_requires_write_grants_and_exact_creation() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_survives_disconnected_initializer() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(0)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_revocation_retains_admitted_creation() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(1)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_enrollment_resolves_after_publication_deadline() -> Result<()> { + Box::pin(runtime_fixture::remote_enrollment_lifecycle(2)).await +} diff --git a/crates/cellar/tests/support/cell_group_remote_enrollment.rs b/crates/cellar/tests/support/cell_group_remote_enrollment.rs new file mode 100644 index 000000000..a90ea3693 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_remote_enrollment.rs @@ -0,0 +1,346 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Enrollment traverses real TLS and the shared controller without router replicas. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn invocation( + dispatch: &CellGroupDispatcher, + cell: &CellHistory, + tag: u8, + timeout: Duration, +) -> Result { + let mut call = dispatch + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([tag; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout, + max_attempts: 8, + }, + None, + ) + .await?; + call.begin().await?; + Ok(call) +} + +fn grant(fixture: &RuntimeCluster, router: &Router, index: usize, revision: u64) -> Result<()> { + let history = fixture.cluster.birth.route().history; + fixture.transports[index].grant_command_origin(NamespaceCommandOrigin { + cluster: history.cluster, + namespace: history.namespace, + revision, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn remote_enrollment() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([90; 32], move |_, tx| { + count.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, _| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"updated".to_vec()) + })?, + ); + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let cell = history(&route, "remote-enrollment")?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + fixture.cell_options(&cell).await?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let mut call = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + let deadline = call.attempt().unwrap().deadline_ms(); + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + router.grant(&fixture, 1)?; + call.retry().await?; + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut voter = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + assert!(voter + .execute_remote(&fixture.transports[1], MemberId::new(1)?) + .await + .is_err()); + drop(voter); + grant(&fixture, &router, 0, 1)?; + grant(&fixture, &router, 1, 1)?; + let follower_gateway = fixture.transports[1].install_cell_command_gateway( + dispatch.clone(), + controllers[1].clone(), + fixture.cell_options_at(&cell, 1).await?, + )?; + let mut follower = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + let error = follower + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .err() + .context("follower refuses")?; + assert!( + matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::NotLeader { .. } + )) + ), + "{error:?}" + ); + assert_eq!(calls.load(Ordering::SeqCst), 0); + drop(follower); + drop(follower_gateway); + call.retry().await?; + assert_eq!(call.attempt().unwrap().deadline_ms(), deadline); + let response = call + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + let original = response.record().clone(); + assert_eq!(original.cell(), &cell); + assert_eq!(original.creation(), ProposalId::new([215; 16])?); + assert_eq!(original.schema(), class.schema_digest()); + assert_eq!(response.served_by(), MemberId::new(1)?); + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!(router.transport.usage().query_items, 1); + assert!(router.transport.usage().query_bytes >= 8192); + assert!(call + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(response); + // Mutate through the same remote write gateway before retrying creation. + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + let mut command = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([216; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![], + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + command.begin().await?; + let updated = command + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(updated.observation().version.sequence.get(), 2); + drop(updated); + drop(command); + call.retry().await?; + assert_eq!( + call.execute_remote(&router.transport, MemberId::new(1)?) + .await? + .record(), + &original + ); + assert_eq!(calls.load(Ordering::SeqCst), 1); + for new_incarnation in [false, true] { + let mut other = cell.clone(); + if new_incarnation { + other.incarnation = IncarnationId::new([217; 16])?; + } + let mut conflict = invocation(&dispatch, &other, 218, Duration::from_secs(15)).await?; + let error = conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err() + .context("creation conflict")?; + assert!(error.ambiguous); + assert!( + matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + )) + ), + "{error:?}" + ); + } + assert_eq!(calls.load(Ordering::SeqCst), 1); + // A routed/possibly transmitted invocation is never fresh local work. + assert!(controllers[0] + .enroll(call, fixture.cell_options(&cell).await?) + .is_err()); + let mut late = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; + drop(gateway); + assert!(late + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(late); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +// Disconnect/revocation hold entered initialization. Timeout holds committed +// manifest publication, so a response deadline cannot seal a committed creation. +pub(crate) async fn remote_enrollment_lifecycle(mode: u8) -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = + super::apply_faults::physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(super::apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Arc::new(Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(Mutex::new(held)); + let init_entered = entered.clone(); + let init_held = held.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.initializer( + [90; 32], + move |_, tx| { + if count.fetch_add(1, Ordering::SeqCst) == 0 && mode != 2 { + let _ = init_entered + .lock() + .unwrap() + .take() + .context("signal")? + .send(()); + init_held + .lock() + .unwrap() + .recv_timeout(Duration::from_secs(15))?; + } + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + }, + )?); + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class, keys(190)?)?; + let cell = history(&route, "remote-enrollment-lifecycle")?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + fixture.cell_options(&cell).await?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + grant(&fixture, &router, 0, 1)?; + if mode == 2 { + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered + .lock() + .unwrap() + .take() + .expect("apply entered") + .send(()); + held.lock() + .unwrap() + .recv_timeout(Duration::from_secs(15)) + .expect("release apply"); + })); + } + let timeout = if mode == 2 { + Duration::from_millis(800) + } else { + Duration::from_secs(15) + }; + let mut call = invocation(&dispatch, &cell, 219, timeout).await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + call.execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + assert_eq!(controllers[0].usage().retained_commands, 1); + if mode == 0 { + task.abort(); + assert!(task.await.err().context("cancelled task")?.is_cancelled()); + } else { + if mode == 1 { + fixture.transports[0].revoke_command_origin( + cell.cluster, + cell.namespace, + router.peer.node, + 2, + )?; + } + let error = tokio::time::timeout(Duration::from_secs(5), task) + .await?? + .err() + .context("lost response")?; + assert!(error.ambiguous); + } + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + release.send(())?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + if mode == 1 { + grant(&fixture, &router, 0, 3)?; + } + let state = fixture.runtimes[0] + .enrollment_state(route.history, cell.clone()) + .await?; + let original = state.record().context("committed enrollment")?.clone(); + let mut retry = invocation(&dispatch, &cell, 219, Duration::from_secs(15)).await?; + let result = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(result.record(), &original); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(result); + drop(retry); + drop(gateway); + router.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 7c1f58a54..14941bc28 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -713,3 +713,5 @@ pub(super) use sql::{nonmember_queries, nonmember_query_revocation}; pub(super) use sql::cached_dispatch_execution; pub(super) use sql::{remote_command_lifecycle, remote_commands}; + +pub(super) use sql::{remote_enrollment, remote_enrollment_lifecycle}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index cdb37e2de..c887e48ae 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -560,3 +560,7 @@ pub(crate) use dispatch_cache_execution::cached_dispatch_execution; #[path = "cell_group_remote_commands.rs"] mod remote_commands; pub(crate) use remote_commands::{remote_command_lifecycle, remote_commands}; + +#[path = "cell_group_remote_enrollment.rs"] +mod remote_enrollment; +pub(crate) use remote_enrollment::{remote_enrollment, remote_enrollment_lifecycle}; From 3056c4bbb6575d9569d05c4e3b70d6f806f3b03b Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 10:05:21 -0700 Subject: [PATCH 107/568] docs: record qualified remote enrollment checkpoint --- docs/design/multi-raft/cell-groups/README.md | 8 +++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 23 +++++++- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 51 +++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 13 ++++- docs/design/multi-raft/cell-groups/tasks.json | 56 ++++++++++++++++++- 7 files changed, 148 insertions(+), 9 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index a4f928bab..40373fab2 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -220,6 +220,14 @@ cases and eleven commands; transfer `.cellar/cell-groups-evidence/CG-04-command- (273 source inputs, 4 executables). Remote enrollment, standalone receipt lookup and the public SDK cutover remain open; CG-04 remains IN_PROGRESS. +At `0fe665d0`, remote enrollment shares the bounded command exchange, explicit +write grants and exact controller registration. Creation receipts survive later +mutations and retries; admitted initialization survives disconnect/revocation and +publication deadline. The partial gate passes 216 Rust cases and eleven commands; +transfer `.cellar/cell-groups-evidence/CG-04-enrollment-rpc/` (276 source inputs, +4 executables). Safe standalone receipt lookup and public SDK cutover remain; +CG-04 remains IN_PROGRESS. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index eabf0df96..7d2f0c8cb 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "5fd4443796a6e990fbe70c16acb7fa5578b7e763", + "source_revision": "0fe665d0267f1b831b172aad5e8d435f5bf0cd39", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index a8306a554..5ba21994c 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,24 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `5fd44437`, adding remote commands over +The newest reviewed source baseline is `0fe665d0`, adding remote enrollment over +the same bounded write exchange and exact controller registration as commands. +Query, Command and Enrollment have distinct authenticated mandatory purposes. +Explicit namespace/process write grants gate initialization; discovery/read grants +and voter membership do not. Captured creation identity, inputs, preview and +original routing budget survive transport. Receiver fresh catalog/leader checks +precede controller ownership, which survives disconnect, revocation and expiry. +Exact committed creation records survive later mutations and duplicate retries. +Four both-image native cases plus three wire cases qualify these boundaries. +The record at `.cellar/cell-groups-evidence/CG-04-enrollment-rpc/checkpoint.json` +binds 216 Rust cases, eleven commands, 276 source inputs and 4 executables. +Transfer this directory with previous evidence. CG-04 remains IN_PROGRESS. Next +implement standalone safe receipt lookup: validate the registered contract and +request identity, inspect the committed cell image without invoking application +code, and keep absence ambiguous unless the existing reconciliation proof excludes +the original proposal. Public SDK/discovery, dynamic roster and per-group failure +isolation follow. Shared RustFS recovery and load remain unqualified. +The preceding reviewed source baseline is `5fd44437`, adding remote commands over the bounded TLS application RPC lane. Mandatory authenticated command purpose and independent namespace/process write grants prevent read/voter authority from becoming write permission. The exact controller owns accepted work through @@ -492,8 +509,8 @@ Do not let plan maintenance replace implementation. requires replicated transfer seals and serving-admission invalidation, linked to CG-09. Do not substitute a TTL or cached hint for that authority contract. Remote commands now use explicit write grants and response-independent - controller producers. Keep read grants independent. Add remote enrollment and - standalone safe receipt lookup without treating missing local state as proof + controller producers; enrollment now shares that write path. Keep read grants + independent. Add standalone safe receipt lookup without treating missing state as proof of non-commit or replaying unresolved callbacks. The old host refuses registered initializers until its actual replacement; do not remove that refusal without a functioning shared serving path. Request diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 5c7e48627..604f42665 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 65 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 69 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 8ee2e1acf..ed73a286d 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,57 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 remote enrollment through the shared write owner + +Qualified source: `0fe665d0267f1b831b172aad5e8d435f5bf0cd39`. + +`CellEnrollmentInvocation` now consumes catalog attempts over the same bounded +write exchange as commands. An authenticated mandatory purpose distinguishes +Query, Command and Enrollment; marker 1 has no reader for the old boolean purpose. +The existing exact controller registration and namespace/process write grants +admit compiled initializers. Read permission, discovery and voter membership +cannot authorize creation. The router still allocates no local group replica. + +Creation identity, captured clock/random inputs and preview identity are frozen +before discovery and retained through retries. Original attempt/deadline budgets +remain in force. The receiver checks fresh catalog group/schema/epoch/birth and +actual leader authority before handing initialization to the controller. Followers +return NotLeader; a leader awaiting apply still returns NotReady internally. +Disconnect, permission revocation and response expiry cannot discard accepted +controller ownership. A possibly transmitted invocation cannot be reused as fresh +local enrollment. + +Success returns the exact committed creation record with bounded reply admission. +The original enrollment image survives later mutations; duplicate retries return +that record without initialization or reverting current cell state. The record +is not a current-version token. Creation/incarnation conflicts remain typed. +Command and enrollment requests share physical TLS exchange and reservation code, +so there is no second per-peer write connection or independent budget. + +Four new native cases run four TLS endpoints in one OS process in both image +modes. They cover missing/read-only write permission, voter/follower refusal, +exact creation receipts, duplicate creation after a later remote command, +creation/incarnation conflicts, consumed attempts, original deadlines, reply +reservations, gateway drop and refusal to treat routed work as fresh local work. +Held initialization tests disconnect and revocation. A held committed manifest +publication tests response expiry followed by exact creation recovery. Three +wire cases verify captured inputs/preview, bounded payloads, authenticated purpose +separation and creation/cell/schema/image reply binding. + +All 216 selected Rust cases and eleven gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight +and 37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-enrollment-rpc/checkpoint.json` binds 276 +source inputs and 4 copied executables. Transfer this entire directory with +prior checkpoints. Targeted runs remain diagnostic evidence. + +CG-04 remains IN_PROGRESS. The next request-path gap is standalone safe receipt +lookup after an ambiguous response. Public SDK/discovery, dynamic roster and +lazy group birth, per-group failure isolation and configuration lifecycle remain. +The shared lane still serializes exchanges per peer. Complete-group snapshots, +archive/reclamation, shared RustFS recovery, independent-process faults and load +remain unqualified; no throughput or latency improvement is claimed. + ## CG-04 write-authorized remote commands Qualified source: `5fd4443796a6e990fbe70c16acb7fa5578b7e763`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index a4849113a..016044a2f 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -130,7 +130,7 @@ removed. A replaced discovery view refuses captured requests and late replies; removal does not revoke the destination's independently managed namespace grant. Grant revocation cancels read waiters without releasing physically entered SQL. The public host must supply trusted discovery and explicit permissions; automatic -discovery/SDK binding, remote enrollment and standalone receipt lookup remain +discovery/SDK binding and standalone receipt lookup remain integration work. Remote commands use the same bounded application RPC lane with a mandatory, @@ -151,6 +151,17 @@ exact committed command identity/result and an authenticated logical cell token; conflict, expiration and sequence-gap refusals remain typed. Retain caller reply byte admission through consumption. Retry the same identity and input under the original budget; a committed duplicate must not re-execute application code. +Remote enrollment uses the same bounded write exchange, namespace/process grants +and exact controller registration. The mandatory authenticated purpose distinguishes +Query, Command and Enrollment under marker 1. A command payload cannot be decoded +as an initializer call. Capture creation identity, host inputs and preview identity +before discovery; retries retain them and the original route/deadline budget. +Receiver enrollment verifies a fresh catalog binding and actual leader authority +before initialization. Its admitted controller owner survives disconnect, +revocation and response deadline. Return only the exact committed creation record, +including the original enrollment image after later mutations. That record is not +a current-version token. Conflicting creation/incarnation refuses without replay. + The current lane serializes RPC exchanges per peer; this is not load qualification. ## Apply, checkpoint and archive diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 7ee7e85c7..24c8d18fd 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -518,7 +518,10 @@ "crates/cellar/durability/raft/command_cell_wire.rs", "crates/cellar/durability/raft/command_cell_transport.rs", "crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs", - "crates/cellar/tests/support/cell_group_remote_commands.rs" + "crates/cellar/tests/support/cell_group_remote_commands.rs", + "crates/cellar/durability/raft/enrollment_cell_wire.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs", + "crates/cellar/tests/support/cell_group_remote_enrollment.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -665,12 +668,21 @@ ] }, { - "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Remote enrollment and standalone safe receipt lookup remain separate required integration. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", + "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Standalone safe receipt lookup remains required integration; remote enrollment is specified in the following step. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", "tests": [ "shared_remote_commands_require_write_grants_and_exact_receipts", "shared_remote_commands_survive_disconnected_sql_waiters", "shared_remote_command_revocation_preserves_admitted_ownership" ] + }, + { + "action": "Route enrollment through the same bounded write RPC exchange, explicit namespace/process grants and exact controller registration as commands. Authenticate Query, Command and Enrollment as distinct mandatory purposes under marker 1, with no legacy reader. Preserve creation identity, captured inputs, preview identity and original routing budget. Recheck fresh destination catalog and leader authority before transferring initialization to the controller; return only the exact committed creation record, never treat the initial image as a current version token. Reuse shared reply budgets and preserve ownership after response disconnect, revocation and publication deadline. Run all query_wire tests including shared_enrollment_wire_preserves_captured_identity_and_inputs, shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read and shared_enrollment_reply_binds_creation_cell_schema_and_initial_image, plus all existing command/query and controller enrollment regressions. Standalone safe receipt lookup, dynamic roster and public SDK cutover remain open.", + "tests": [ + "shared_remote_enrollment_requires_write_grants_and_exact_creation", + "shared_remote_enrollment_survives_disconnected_initializer", + "shared_remote_enrollment_revocation_retains_admitted_creation", + "shared_remote_enrollment_resolves_after_publication_deadline" + ] } ], "test_cases": [ @@ -1292,6 +1304,46 @@ "I05", "I12" ] + }, + { + "name": "shared_remote_enrollment_requires_write_grants_and_exact_creation", + "oracle": "Over real TLS in both image modes, nonmember ingress without write grants, read-only ingress and voter membership cannot enroll. A fully registered follower with a write grant returns NotLeader before initialization. Leader admission returns the exact creation identity and original image; duplicate enrollment after a later remote command preserves the original record without rerunning initialization. Creation/incarnation conflicts refuse. Consumed attempts, original deadlines, response admission and gateway drop remain enforced; routed work cannot masquerade as fresh local work.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_survives_disconnected_initializer", + "oracle": "Hold entered initialization, disconnect the remote waiter, and verify controller ownership survives while network admission drains. Release SQL and retry the exact creation identity; one initialized cell and the committed creation record survive in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_revocation_retains_admitted_creation", + "oracle": "Revoke explicit write permission while initialization is physically held. The remote error remains ambiguous and controller ownership remains until the creation resolves. Renew permission and recover the exact record without rerunning initialization, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_remote_enrollment_resolves_after_publication_deadline", + "oracle": "Hold committed manifest publication until the remote deadline expires. The caller receives an ambiguous error while the controller retains accepted creation. Release publication and retry the same identity; the exact original record returns with one initialization in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From fbc15053f8dcfd3b46078abad3cbbae7b68da8d6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 10:36:58 -0700 Subject: [PATCH 108/568] feat: inspect shared-cell receipts without replaying commands --- .../raft/cell_groups/controller/mod.rs | 27 ++ .../raft/cell_groups/directory/routing.rs | 1 + .../raft/cell_groups/dispatch/mod.rs | 3 + .../raft/cell_groups/dispatch/receipt.rs | 323 +++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 8 +- .../durability/raft/command_cell_transport.rs | 136 ++++++ .../durability/raft/command_cell_wire.rs | 2 +- .../cellar/durability/raft/execution/cell.rs | 2 +- .../cellar/durability/raft/execution/mod.rs | 3 +- .../durability/raft/execution/registered.rs | 20 +- .../raft/host_sql/cell_groups/mod.rs | 17 +- .../raft/host_sql/cell_groups/query.rs | 182 +++++-- .../durability/raft/query_cell_transport.rs | 8 + .../cellar/durability/raft/query_identity.rs | 17 +- .../cellar/durability/raft/query_transport.rs | 2 +- crates/cellar/durability/raft/query_wire.rs | 2 + .../durability/raft/query_wire_tests.rs | 203 +++++++- .../durability/raft/receipt_cell_wire.rs | 225 +++++++++ crates/cellar/durability/raft/receipts.rs | 28 ++ .../cellar/durability/raft/receipts_tests.rs | 55 +++ crates/cellar/durability/raft/transport.rs | 1 + .../durability/raft/transport_options.rs | 2 +- .../tests/native_cell_groups_execution.rs | 15 + .../support/cell_group_receipt_lookup.rs | 455 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 6 + 26 files changed, 1688 insertions(+), 57 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs create mode 100644 crates/cellar/durability/raft/receipt_cell_wire.rs create mode 100644 crates/cellar/tests/support/cell_group_receipt_lookup.rs diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 790331145..19511772b 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -80,6 +80,7 @@ pub struct CellGroupControllerUsage { struct Shared { runtime: CellGroupRuntime, + sql: crate::raft::host_sql::cell_groups::WeakCellGroupSqlPool, groups: BTreeSet, limits: CellGroupControllerLimits, domain: crate::asyncrt::DomainToken, @@ -198,6 +199,7 @@ impl CellGroupController { let (requests, receiver) = tokio::sync::mpsc::channel(limits.max_commands); let shared = Arc::new(Shared { runtime: runtime.clone(), + sql: sql.downgrade(), groups: roster.keys().copied().collect(), limits, domain: domain.clone(), @@ -274,3 +276,28 @@ impl CellGroupController { self.wait().await } } + +impl CellGroupController { + pub(crate) async fn lookup_receipt( + &self, + mut invocation: super::CellReceiptInvocation, + options: super::CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + if shared.closing.load(Ordering::Acquire) { + return Err(super::CellDispatchError::Runtime(RuntimeError::Closed)); + } + invocation.begin().await?; + if !invocation + .attempt() + .is_some_and(|a| shared.groups.contains(&a.route().history)) + { + return Err(super::CellDispatchError::Runtime(RuntimeError::NotFound)); + } + let sql = shared + .sql + .upgrade() + .ok_or(super::CellDispatchError::Runtime(RuntimeError::Closed))?; + invocation.execute(&shared.runtime, &sql, options).await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs index c8f29473c..1d9ab7046 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/routing.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -11,6 +11,7 @@ use std::fmt; #[derive(Clone, Debug, PartialEq, Eq)] pub enum CellRouteOperation { Command(CommandId), + Receipt(CommandId), Enrollment(ProposalId), Query(ProposalId), } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 67721e85d..1742c4875 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -3,6 +3,9 @@ //! Catalog-bound invocations over the existing consensus and SQL owners. //! Routing observations never replace destination authority or ingress grants. +mod receipt; +pub use receipt::{CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse}; + mod enrollment; pub use enrollment::{CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest}; diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs new file mode 100644 index 000000000..90387284e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -0,0 +1,323 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt-only dispatch never stages SQL or turns absence into non-commit. +use super::*; +use crate::application::consensus::command_request_digest; +use crate::raft::execution::{CommandRequest, StoredReceipt}; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; + +#[derive(Clone)] +pub struct CellReceiptRequest { + pub id: CommandId, + pub contract: OperationContract, + pub input: Vec, + pub durability: Durability, +} + +pub struct CellReceiptInvocation { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + request: CellReceiptRequest, + session: CellRoutingSession, + attempt: Option, + dispatched: bool, + remote_route: Option, + _admission: Permit, +} + +pub struct CellReceiptResponse { + command: CommandId, + selected: CellSqlReceiptLookup, + token: CellVersionToken, +} + +impl CellReceiptResponse { + pub fn command(&self) -> CommandId { + self.command + } + + /// None is ambiguous, even after a quorum read: an unresolved or delayed + /// original command may still commit. This response releases no producer. + pub fn receipt(&self) -> Option<&StoredReceipt> { + self.selected.receipt() + } + + pub fn observation(&self) -> &CellVersionObservation { + self.selected.observation() + } + + pub fn token(&self) -> &CellVersionToken { + &self.token + } +} + +impl CellGroupDispatcher { + pub fn lookup_receipt( + &self, + cell: CellHistory, + mut request: CellReceiptRequest, + limits: CellRoutingLimits, + cached: Option, + ) -> Result { + if request.durability != Durability::Quorum + || self.directory.namespace_config().minimum_durability != Durability::Quorum + { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + if request.contract.kind != OperationKind::Command { + return Err(CellDispatchError::Registration( + RegistrationError::KindMismatch, + )); + } + self.class + .validate_operation(&request.contract) + .map_err(CellDispatchError::Registration)?; + if request.contract.operation.len() > 128 + || request.input.len() > self.limits.max_input_bytes + { + return Err(CellDispatchError::Runtime(RuntimeError::Overloaded)); + } + let bytes = request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let permit = self + .budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + request.input = request.input.into_boxed_slice().into_vec(); + let started = self.domain.mono_ms().map_err(|_| closed())?; + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-receipt-invocation-1\0"); + hash.update(cell_image_history_digest(&cell)); + hash.update(request.id.client.bytes()); + hash.update(request.id.sequence().to_be_bytes()); + hash.update((request.contract.operation.len() as u32).to_be_bytes()); + hash.update(request.contract.operation.as_bytes()); + hash.update(request.contract.digest); + hash.update(command_request_digest(&request.input)); + hash.update([1, limits.max_attempts]); + hash.update(limits.timeout.as_nanos().to_be_bytes()); + let session = self + .directory + .routing_session_in( + self.domain.clone(), + CellRouteRequest { + cell: cell.clone(), + operation: CellRouteOperation::Receipt(request.id), + fingerprint: hash.finalize().into(), + }, + limits, + cached, + started, + ) + .map_err(CellDispatchError::Routing)? + .using_cache(self.routes.clone()); + Ok(CellReceiptInvocation { + dispatcher: self.clone(), + cell, + request, + session, + attempt: None, + dispatched: false, + remote_route: None, + _admission: permit, + }) + } +} + +impl CellReceiptInvocation { + pub fn attempt(&self) -> Option<&CellRouteAttempt> { + self.attempt.as_ref() + } + + pub(crate) fn bind_remote_route(&mut self, route: RemoteMutationRoute) { + self.remote_route = Some(route); + } + + pub async fn begin(&mut self) -> Result<(), CellDispatchError> { + self.dispatched = true; + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|r| !r.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid("receipt destination route")); + } + self.attempt = Some(attempt); + self.dispatched = false; + Ok(()) + } + + pub async fn retry(&mut self) -> Result<(), CellDispatchError> { + let previous = self.current()?.clone(); + self.dispatched = true; + self.attempt = Some( + self.session + .retry(&previous) + .await + .map_err(CellDispatchError::Routing)?, + ); + self.dispatched = false; + Ok(()) + } + + pub async fn redirect(&mut self, hint: &GroupRoute) -> Result<(), CellDispatchError> { + let previous = self.current()?.clone(); + self.dispatched = true; + self.attempt = Some( + self.session + .redirect(&previous, hint) + .await + .map_err(CellDispatchError::Routing)?, + ); + self.dispatched = false; + Ok(()) + } + + fn current(&self) -> Result<&CellRouteAttempt, CellDispatchError> { + self.attempt + .as_ref() + .ok_or(CellDispatchError::Invalid("missing receipt route")) + } + + pub async fn execute( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched { + return Err(CellDispatchError::Invalid("consumed receipt attempt")); + } + self.dispatched = true; + if !runtime.uses_domain(&self.dispatcher.domain) + || !sql.uses_domain(&self.dispatcher.domain) + { + return Err(CellDispatchError::Invalid("receipt execution domain")); + } + let authority = runtime + .read_authority( + attempt.route().history, + self.cell.clone(), + CellReadConsistency::Linearizable, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() || authority.cell().version.history != self.cell { + return Err(CellDispatchError::Invalid("receipt destination route")); + } + let selected = sql + .lookup_receipt_registered( + options, + authority, + self.dispatcher.class.clone(), + CommandRequest { + id: self.request.id, + operation: self.request.contract.operation.clone(), + contract_digest: self.request.contract.digest, + request_digest: command_request_digest(&self.request.input), + durability: self.request.durability, + retry_until_ms: 0, + }, + self.request.input.clone(), + ) + .await + .map_err(CellDispatchError::Execution)?; + if selected + .receipt() + .is_some_and(|r| r.minimum_durability() != Durability::Quorum) + { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + let now = self + .dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed())?; + let token = self + .dispatcher + .keys + .issue_cell(selected.observation(), attempt.route().schema_digest, now) + .map_err(CellDispatchError::Version)?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + Ok(CellReceiptResponse { + command: self.request.id, + selected, + token, + }) + } + + pub async fn execute_remote( + &mut self, + transport: &crate::raft::transport::RaftTransport, + member: cellar_logic::consensus::MemberId, + ) -> Result { + let attempt = self.current()?.clone(); + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + if self.dispatched || self.remote_route.is_some() { + return Err(CellDispatchError::Invalid("consumed receipt attempt")); + } + self.dispatched = true; + if !transport.uses_command_domain(&self.dispatcher.domain) { + return Err(CellDispatchError::Invalid("receipt transport domain")); + } + let timeout_ms = attempt + .deadline_ms() + .checked_sub(self.dispatcher.domain.mono_ms().map_err(|_| closed())?) + .filter(|n| *n > 0) + .ok_or(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded))?; + let call = crate::raft::query_wire::receipt::CellReceiptCall { + request: self.request.clone(), + timeout_ms, + route_epoch: attempt.route().epoch.get(), + birth_digest: attempt.route().birth_digest, + }; + let response = self + .dispatcher + .domain + .run(transport.lookup_cell_receipt( + attempt.route(), + &self.cell, + member, + call, + self.dispatcher.keys.clone(), + attempt.deadline_ms(), + )) + .await + .map_err(|_| closed())? + .map_err(|error| match error { + crate::raft::transport::TransportError::Command(failure) => { + CellDispatchError::Execution(ExecutionError::Refused(failure)) + } + other => CellDispatchError::Transport(other), + })?; + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + Ok(response) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 808798540..e5ca6cf6a 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -5,13 +5,13 @@ pub use super::host_sql::cell_groups::{ CellGroupPreparedApply, CellGroupSqlPool, CellSqlEnrollment, CellSqlEnrollmentResolution, CellSqlLimits, CellSqlOptions, CellSqlOutcome, CellSqlPrepared, CellSqlQueryResult, - CellSqlReceipt, CellSqlReconciliation, CellSqlRejection, CellSqlRootOptions, CellSqlSubmission, - CellSqlUsage, + CellSqlReceipt, CellSqlReceiptLookup, CellSqlReconciliation, CellSqlRejection, + CellSqlRootOptions, CellSqlSubmission, CellSqlUsage, }; pub use crate::raft::transport::{ CellCommandGateway, CellQueryGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, - CellRemoteQueryResponse, + CellRemoteQueryResponse, CellRemoteReceiptResponse, }; mod apply; @@ -28,7 +28,7 @@ pub use dispatch::{ CellCommandResponse, CellCommandSettlement, CellCommandSubmission, CellDispatchError, CellDispatchLimits, CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest, CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, CellQueryRequest, - CellQueryResponse, + CellQueryResponse, CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse, }; mod execution_authority; diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs index 8408b6056..7972c4fcd 100644 --- a/crates/cellar/durability/raft/command_cell_transport.rs +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -12,6 +12,7 @@ use crate::raft::query_wire::{ self as wire, command::{CellCommandCall, CellCommandValue}, enrollment::CellEnrollmentCall, + receipt::{CellReceiptCall, CellReceiptValue}, }; use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; use cellar_logic::{ @@ -107,9 +108,48 @@ impl CellRemoteEnrollmentResponse { } } +/// Receipt inspection retains reply admission but never owns a producer. +pub struct CellRemoteReceiptResponse { + value: CellReceiptValue, + _node: Permit, + _peer: Permit, +} + +impl CellRemoteReceiptResponse { + pub fn command(&self) -> cellar_logic::consensus::CommandId { + self.value.value.command + } + + /// None remains ambiguous; no command or initializer was invoked by lookup. + pub fn result(&self) -> Option<&[u8]> { + self.value + .retained + .then_some(self.value.value.value.result.as_slice()) + } + + pub fn retry_until_ms(&self) -> Option { + self.value + .retained + .then_some(self.value.value.retry_until_ms) + } + + pub fn observation(&self) -> &CellVersionObservation { + &self.value.value.value.observation + } + + pub fn token(&self) -> &CellVersionToken { + &self.value.value.value.token + } + + pub fn served_by(&self) -> MemberId { + self.value.value.value.served_by + } +} + enum WriteCall { Command(CellCommandCall), Enrollment(CellEnrollmentCall), + Receipt(CellReceiptCall), } impl WriteCall { @@ -117,6 +157,7 @@ impl WriteCall { match self { Self::Command(c) => c.encode(limit), Self::Enrollment(c) => c.encode(limit), + Self::Receipt(c) => c.encode(limit), } } @@ -124,6 +165,7 @@ impl WriteCall { let timeout = match self { Self::Command(c) => &mut c.timeout_ms, Self::Enrollment(c) => &mut c.timeout_ms, + Self::Receipt(c) => &mut c.timeout_ms, }; *timeout = (*timeout).min(remaining); } @@ -132,6 +174,7 @@ impl WriteCall { match self { Self::Command(c) => c.request.input.len(), Self::Enrollment(_) => 0, + Self::Receipt(c) => c.request.input.len(), } } } @@ -139,6 +182,7 @@ impl WriteCall { enum WriteValue { Command(CellCommandValue), Enrollment(EnrollmentRecord), + Receipt(CellReceiptValue), } struct WriteResponse { @@ -276,6 +320,35 @@ impl RaftTransport { }) } + pub(in crate::raft) async fn lookup_cell_receipt( + &self, + route: &cellar_logic::cell_group::GroupRoute, + cell: &CellHistory, + member: MemberId, + call: CellReceiptCall, + keys: VersionKeys, + deadline: u64, + ) -> Result { + let response = self + .write_cell( + route, + cell, + member, + WriteCall::Receipt(call), + keys, + deadline, + ) + .await?; + let WriteValue::Receipt(value) = response.value else { + return Err(TransportError::Protocol); + }; + Ok(CellRemoteReceiptResponse { + value, + _node: response.node, + _peer: response.peer, + }) + } + async fn write_cell( &self, route: &cellar_logic::cell_group::GroupRoute, @@ -455,6 +528,7 @@ impl State { let route = match &request.call { WriteCall::Command(_) => route.for_command()?, WriteCall::Enrollment(_) => route.for_enrollment()?, + WriteCall::Receipt(_) => route.for_receipt()?, }; let body = self .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) @@ -477,6 +551,20 @@ impl State { )? .map(WriteValue::Command) } + WriteCall::Receipt(call) => { + let now = u64::try_from(crate::asyncrt::wall_ms()) + .map_err(|_| TransportError::Authentication)?; + wire::receipt::decode_result( + &body, + (call.request.id, &request.cell), + request.route.schema_digest, + &request.keys, + now, + self.options.max_query_response_bytes, + request.peer.replica.member, + )? + .map(WriteValue::Receipt) + } WriteCall::Enrollment(call) => wire::enrollment::decode_result( &body, call.request.creation, @@ -504,6 +592,54 @@ impl State { } impl State { + pub(super) async fn execute_cell_receipt( + &self, + route: &QueryRoute, + call: CellReceiptCall, + ) -> Result, TransportError> { + let (_, target, cell) = route.decode_shared()?; + let gateway = self + .cell_command_gateways + .lock() + .map_err(|_| TransportError::Closed)? + .get(&(cell.cluster, cell.namespace)) + .and_then(|g| g.upgrade()) + .ok_or(TransportError::Refused)?; + if !gateway.accepting.load(Ordering::Acquire) { + return Err(TransportError::Refused); + } + let binding = crate::raft::cell_groups::RemoteMutationRoute { + group: target.scope, + schema: route.schema(), + epoch: call.route_epoch, + birth: call.birth_digest, + }; + let result = async { + let mut invocation = gateway + .dispatcher + .lookup_receipt( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + ) + .map_err(|e| command_failure(&e))?; + invocation.bind_remote_route(binding); + self.authorize_cell_command(route) + .map_err(|_| CommandFailure::Unavailable)?; + gateway + .controller + .lookup_receipt(invocation, gateway.options.clone()) + .await + .map_err(|e| command_failure(&e)) + } + .await; + wire::receipt::encode_result(&result, self.options.max_query_response_bytes) + } + pub(super) async fn execute_cell_enrollment( &self, route: &QueryRoute, diff --git a/crates/cellar/durability/raft/command_cell_wire.rs b/crates/cellar/durability/raft/command_cell_wire.rs index e4455897b..0bbbfcb09 100644 --- a/crates/cellar/durability/raft/command_cell_wire.rs +++ b/crates/cellar/durability/raft/command_cell_wire.rs @@ -120,7 +120,7 @@ impl CellCommandCall { } } -fn command_id(body: &mut Reader<'_>) -> Result { +pub(super) fn command_id(body: &mut Reader<'_>) -> Result { let client = ClientId::new(body.array()?).map_err(|_| TransportError::Identity)?; CommandId::new(client, u64::from_be_bytes(body.array()?)).map_err(|_| TransportError::Identity) } diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 66b3b6d0e..e83e55864 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -132,7 +132,7 @@ impl CellPrivateExecutor { self.options.limits, self.options.application_digest, &history, - &attempt, + &attempt.request, class, input, false, diff --git a/crates/cellar/durability/raft/execution/mod.rs b/crates/cellar/durability/raft/execution/mod.rs index a7efcf215..731ca7042 100644 --- a/crates/cellar/durability/raft/execution/mod.rs +++ b/crates/cellar/durability/raft/execution/mod.rs @@ -15,6 +15,7 @@ mod command; mod enrollment; mod registered; pub use enrollment::{CellEnrollmentAttempt, CellEnrollmentResolution, CellPrivateEnrollment}; +pub(crate) use registered::validate as validate_registered_command; mod images; pub use cell::{CellCommittedReceipt, CellPrivateExecutor, CellReconciliation, CellStageOutcome}; mod rooted_working; @@ -640,7 +641,7 @@ impl PrivateExecutor { self.options.limits, self.options.application_digest, &scope, - &attempt, + &attempt.request, class, input, system, diff --git a/crates/cellar/durability/raft/execution/registered.rs b/crates/cellar/durability/raft/execution/registered.rs index 479d4131e..1f2067599 100644 --- a/crates/cellar/durability/raft/execution/registered.rs +++ b/crates/cellar/durability/raft/execution/registered.rs @@ -4,19 +4,19 @@ use super::*; use cellar_logic::cell_group::CellHistory; -pub(super) fn validate( +pub(crate) fn validate( limits: ExecutionLimits, schema: [u8; 32], history: &CellHistory, - attempt: &CommandAttempt, + request: &CommandRequest, class: &RaftClass, input: &[u8], system: bool, ) -> Result { let contract = OperationContract { - operation: attempt.request.operation.clone(), + operation: request.operation.clone(), kind: OperationKind::Command, - digest: attempt.request.contract_digest, + digest: request.contract_digest, }; if system { class.validate_system_operation(&contract) @@ -32,18 +32,12 @@ pub(super) fn validate( if input.len() > limits.envelope.max_payload_bytes { return Err(ExecutionError::Refused(CommandFailure::Overloaded)); } - if command_request_digest(input) != attempt.request.request_digest { + if command_request_digest(input) != request.request_digest { return Err(ExecutionError::Refused(CommandFailure::CommandConflict)); } if !system { class - .validate_command_identity( - history, - &contract, - attempt.request.id, - attempt.request.durability, - input, - ) + .validate_command_identity(history, &contract, request.id, request.durability, input) .map_err(|error| { ExecutionError::Refused( error @@ -52,7 +46,7 @@ pub(super) fn validate( .unwrap_or(CommandFailure::UnsupportedCapability), ) })?; - } else if attempt.request.id.client + } else if request.id.client != crate::raft::effects::system_client(history).map_err(ExecutionError::Refused)? { return Err(ExecutionError::Refused( diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 815df9db1..2f68b9931 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -8,7 +8,7 @@ pub(in crate::raft::host_sql) use apply::ApplyRequest; pub use apply::CellGroupPreparedApply; mod query; mod worker; -pub use query::CellSqlQueryResult; +pub use query::{CellSqlQueryResult, CellSqlReceiptLookup}; use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; use crate::raft::cell_groups::{ @@ -275,6 +275,7 @@ pub enum CellSqlOutcome { EnrollmentResolved(CellSqlEnrollmentResolution), Rejected, Queried(CellSqlQueryResult), + ReceiptLookedUp(CellSqlReceiptLookup), } pub enum CellSqlReconciliation { @@ -301,6 +302,20 @@ pub struct CellGroupSqlPool { inner: Arc, } +pub(crate) struct WeakCellGroupSqlPool(std::sync::Weak); + +impl WeakCellGroupSqlPool { + pub(crate) fn upgrade(&self) -> Option { + self.0.upgrade().map(|inner| CellGroupSqlPool { inner }) + } +} + +impl CellGroupSqlPool { + pub(crate) fn downgrade(&self) -> WeakCellGroupSqlPool { + WeakCellGroupSqlPool(Arc::downgrade(&self.inner)) + } +} + struct Handle { engine_owner: Arc<()>, workers: Vec, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index 85c65f3c7..883a85a34 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -12,6 +12,11 @@ type VerificationCallback = Box< >; enum Callback { + Receipt { + class: Arc, + request: crate::raft::execution::CommandRequest, + input: Box<[u8]>, + }, Verification(VerificationCallback), Registered { class: Arc, @@ -49,6 +54,37 @@ impl CellSqlQueryResult { } } +/// None means the selected image has no retained receipt, never non-commit. +pub struct CellSqlReceiptLookup { + receipt: Option, + observation: crate::raft::version::CellVersionObservation, + _bytes: Arc, +} + +impl CellSqlReceiptLookup { + pub fn receipt(&self) -> Option<&crate::raft::execution::StoredReceipt> { + self.receipt.as_ref() + } + + pub fn observation(&self) -> &crate::raft::version::CellVersionObservation { + &self.observation + } +} + +enum Output { + Query(Vec), + Receipt(Option), +} + +impl Output { + fn retained_bytes(&self) -> usize { + match self { + Self::Query(bytes) => bytes.len(), + Self::Receipt(receipt) => receipt.as_ref().map_or(0, |r| r.result().len()), + } + } +} + struct Cancel(Arc); impl Drop for Cancel { @@ -69,12 +105,17 @@ impl CellGroupSqlPool { + Send + 'static, ) -> Result { - self.query_callback( - options, - authority, - Callback::Verification(Box::new(callback)), - ) - .await + match self + .query_callback( + options, + authority, + Callback::Verification(Box::new(callback)), + ) + .await? + { + CellSqlOutcome::Queried(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell query reply")), + } } /// Invoke the namespace's compiled query against the exact selected cell @@ -97,16 +138,60 @@ impl CellGroupSqlPool { if input.len() > options.limits().envelope.max_payload_bytes { return Err(overloaded()); } - self.query_callback( - options, - authority, - Callback::Registered { - class, - contract, - input: input.into_boxed_slice(), - }, - ) - .await + match self + .query_callback( + options, + authority, + Callback::Registered { + class, + contract, + input: input.into_boxed_slice(), + }, + ) + .await? + { + CellSqlOutcome::Queried(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell query reply")), + } + } + + /// Inspect a committed image without a private writer or command callback. + /// Contract, payload digest and authenticated command identity are checked + /// before returning even a duplicate receipt. Missing state is not a proof. + pub async fn lookup_receipt_registered( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + mut request: crate::raft::execution::CommandRequest, + input: Vec, + ) -> Result { + if authority.barrier().is_none() { + return Err(ExecutionError::Invalid( + "receipt lookup requires quorum read", + )); + } + if request.operation.len() > 128 + || input.len() > options.limits().envelope.max_payload_bytes + { + return Err(overloaded()); + } + request.operation = request.operation.into_boxed_str().into_string(); + match self + .query_callback( + options, + authority, + Callback::Receipt { + class, + request, + input: input.into_boxed_slice(), + }, + ) + .await? + { + CellSqlOutcome::ReceiptLookedUp(output) => Ok(output), + _ => Err(ExecutionError::Invalid("cell receipt lookup reply")), + } } async fn query_callback( @@ -114,7 +199,7 @@ impl CellGroupSqlPool { options: CellSqlOptions, authority: CellReadAuthority, callback: Callback, - ) -> Result { + ) -> Result { self.admit()?; let domain = &self.inner.workers[0].domain; if !authority.belongs_to_engine(&self.inner.engine_owner) @@ -128,7 +213,7 @@ impl CellGroupSqlPool { options.limits().validate()?; let input_bytes = match &callback { Callback::Verification(_) => 0, - Callback::Registered { input, .. } => input.len(), + Callback::Registered { input, .. } | Callback::Receipt { input, .. } => input.len(), }; let bytes = self.inner.admission.reserve_read( authority.route().history, @@ -155,7 +240,7 @@ impl CellGroupSqlPool { crate::asyncrt::select! { result = self.call(cell, worker::Action::Query(Box::new(query))) => { match result? { - CellSqlOutcome::Queried(output) => { + output @ (CellSqlOutcome::Queried(_) | CellSqlOutcome::ReceiptLookedUp(_)) => { if domain.mono_ms().map_err(|_| unavailable())? >= deadline { return Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)); } @@ -256,15 +341,45 @@ pub(super) fn run( check(&query)?; let observation = query.authority.cell().observation(); let result = std::panic::catch_unwind(AssertUnwindSafe(|| match query.callback { - Callback::Verification(callback) => callback(&database, &observation.version), + Callback::Verification(callback) => callback(&database, &observation.version) + .map(Output::Query) + .map_err(crate::raft::execution::storage), Callback::Registered { class, contract, input, - } => class.invoke_cell_query(&contract, &observation, &database, &input), + } => class + .invoke_cell_query(&contract, &observation, &database, &input) + .map(Output::Query) + .map_err(crate::raft::execution::storage), + Callback::Receipt { + class, + request, + input, + } => { + crate::raft::execution::validate_registered_command( + query.options.limits(), + query.authority.route().schema_digest, + &observation.version.history, + &request, + &class, + &input, + false, + )?; + let receipt = crate::raft::receipts::lookup_cell_identity( + database.connection(), + &request, + &query.options.limits(), + observation.version.sequence.get(), + )?; + Ok(Output::Receipt(receipt)) + } })) - .unwrap_or_else(|panic| Err(crate::native_worker::panic_error(panic))) - .map_err(crate::raft::execution::storage)?; + .unwrap_or_else(|panic| { + Err(crate::raft::execution::storage( + crate::native_worker::panic_error(panic), + )) + })?; // The callback has been consumed, so check the remaining owners explicitly. if query.cancelled.load(Ordering::Acquire) || query.closing.load(Ordering::Acquire) { return Err(unavailable()); @@ -273,15 +388,22 @@ pub(super) fn run( .authority .check(domain.mono_ms().map_err(|_| unavailable())?) .map_err(crate::raft::execution::authority_failure)?; - if result.len() > query.options.limits().max_result_bytes { + if result.retained_bytes() > query.options.limits().max_result_bytes { return Err(overloaded()); } drop(database); drop(query.authority); - Bytes::shrink(&mut query.bytes, result.len() + 4096)?; - Ok(CellSqlOutcome::Queried(CellSqlQueryResult { - result, - observation, - _bytes: query.bytes, - })) + Bytes::shrink(&mut query.bytes, result.retained_bytes() + 4096)?; + Ok(match result { + Output::Query(result) => CellSqlOutcome::Queried(CellSqlQueryResult { + result, + observation, + _bytes: query.bytes, + }), + Output::Receipt(receipt) => CellSqlOutcome::ReceiptLookedUp(CellSqlReceiptLookup { + receipt, + observation, + _bytes: query.bytes, + }), + }) } diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index 0ad67dc10..ab8e344a1 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -359,6 +359,7 @@ pub(super) enum IncomingCall { Shared(CellQueryCall), Command(crate::raft::query_wire::command::CellCommandCall), Enrollment(crate::raft::query_wire::enrollment::CellEnrollmentCall), + Receipt(crate::raft::query_wire::receipt::CellReceiptCall), } impl IncomingCall { @@ -367,6 +368,11 @@ impl IncomingCall { bytes: &[u8], limit: usize, ) -> Result { + if route.is_receipt() { + return Ok(Self::Receipt( + crate::raft::query_wire::receipt::CellReceiptCall::decode(bytes, limit)?, + )); + } if route.is_enrollment() { return Ok(Self::Enrollment( crate::raft::query_wire::enrollment::CellEnrollmentCall::decode(bytes, limit)?, @@ -390,6 +396,7 @@ impl IncomingCall { Self::Shared(call) => call.timeout_ms, Self::Command(call) => call.timeout_ms, Self::Enrollment(call) => call.timeout_ms, + Self::Receipt(call) => call.timeout_ms, } } @@ -401,6 +408,7 @@ impl IncomingCall { match self { Self::Command(call) => state.execute_cell_command(route, call).await, Self::Enrollment(call) => state.execute_cell_enrollment(route, call).await, + Self::Receipt(call) => state.execute_cell_receipt(route, call).await, Self::Legacy(call) => { let (_, target) = route.decode()?; let handle = state diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index adaf11493..d454b2c99 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -60,6 +60,7 @@ enum Purpose { Query, Command, Enrollment, + Receipt, } #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] @@ -94,7 +95,7 @@ impl QueryRoute { /// Request orientation only. A reversed response route is compared with /// the exact outstanding request; it cannot decode as another request. pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { - if self.is_write() { + if self.requires_command_grant() { return Err(TransportError::Identity); } let History::Cell(scope) = &self.scope else { @@ -215,8 +216,18 @@ impl QueryRoute { self.purpose == Purpose::Enrollment } - pub fn is_write(&self) -> bool { - self.is_command() || self.is_enrollment() + pub fn for_receipt(mut self) -> Result { + self.decode_shared()?; + self.purpose = Purpose::Receipt; + Ok(self) + } + + pub fn is_receipt(&self) -> bool { + self.purpose == Purpose::Receipt + } + + pub fn requires_command_grant(&self) -> bool { + self.is_command() || self.is_enrollment() || self.is_receipt() } pub fn is_shared(&self) -> bool { diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index 636dd8169..b5ed91a88 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -395,7 +395,7 @@ impl State { &self, wire: &QueryRoute, ) -> Result<(Arc, Option>), TransportError> { - if wire.is_write() { + if wire.requires_command_grant() { return self.authorize_cell_command(wire); } if wire.is_shared() { diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index fa7c62732..dff07bee8 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -23,6 +23,8 @@ pub(super) mod cell; pub(super) mod command; #[path = "enrollment_cell_wire.rs"] pub(super) mod enrollment; +#[path = "receipt_cell_wire.rs"] +pub(super) mod receipt; pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index d3a091cc0..930563974 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -889,7 +889,7 @@ fn shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read() { ) .unwrap(); let enrollment = read.clone().for_enrollment().unwrap(); - assert!(enrollment.is_write()); + assert!(enrollment.requires_command_grant()); assert!(enrollment.is_enrollment()); assert!(!enrollment.is_command()); let auth = PeerAuth::new([17; 32], "node-1").unwrap(); @@ -982,3 +982,204 @@ fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { ); } } + +fn receipt_call() -> receipt::CellReceiptCall { + let command = command_call(); + receipt::CellReceiptCall { + request: crate::raft::cell_groups::CellReceiptRequest { + id: command.request.id, + contract: command.request.contract, + input: command.request.input, + durability: command.request.durability, + }, + timeout_ms: command.timeout_ms, + route_epoch: command.route_epoch, + birth_digest: command.birth_digest, + } +} + +#[test] +fn shared_receipt_wire_refuses_command_payloads_and_preserves_identity() { + let call = receipt_call(); + let bytes = call.encode(4096).unwrap(); + let decoded = receipt::CellReceiptCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.id, call.request.id); + for end in 0..bytes.len() { + assert!(receipt::CellReceiptCall::decode(&bytes[..end], 4096).is_err()); + } + let mut extra = bytes.clone(); + extra.push(0); + assert!(receipt::CellReceiptCall::decode(&extra, 4096).is_err()); + assert!(receipt::CellReceiptCall::decode(&bytes, bytes.len() - 1).is_err()); + assert!(command::CellCommandCall::decode(&bytes, 4096).is_err()); + assert!(enrollment::CellEnrollmentCall::decode(&bytes, 4096).is_err()); + assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); + assert!(receipt::CellReceiptCall::decode(&command_call().encode(4096).unwrap(), 4096).is_err()); + let mut call = call; + call.request.durability = Durability::S3; + assert!(call.encode(4096).is_err()); +} + +#[test] +fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { + let call = receipt_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let member = MemberId::new(2).unwrap(); + for retained in [false, true] { + let mut body = b"CLGCRS01".to_vec(); + body.push(if retained { 0 } else { 13 }); + body.extend_from_slice(&call.request.id.client.bytes()); + body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + string(&mut body, token.as_str()).unwrap(); + if retained { + body.extend_from_slice(&9000_u64.to_be_bytes()); + body.extend_from_slice(&0_u32.to_be_bytes()); + } + let value = receipt::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member, + ) + .unwrap() + .unwrap(); + assert_eq!(value.retained, retained); + assert!(value.value.value.result.is_empty()); + assert_eq!(value.value.value.observation, observation); + assert_eq!(value.value.retry_until_ms, if retained { 9000 } else { 0 }); + for end in 0..body.len() { + assert!(receipt::decode_result( + &body[..end], + (call.request.id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + } + let other_id = CommandId::new(call.request.id.client, 99).unwrap(); + assert!(receipt::decode_result( + &body, + (other_id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(receipt::decode_result( + &body, + (call.request.id, cell), + [8; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + let mut other = cell.clone(); + other.incarnation = IncarnationId::new([99; 16]).unwrap(); + assert!(receipt::decode_result( + &body, + (call.request.id, &other), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + assert!(command::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + body.push(0); + assert!(receipt::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .is_err()); + } + for error in [ + CommandFailure::CommandExpired, + CommandFailure::CommandConflict, + CommandFailure::SequenceGap { next: 4 }, + ] { + let body = receipt::encode_result(&Err(error.clone()), 4096).unwrap(); + assert_eq!( + receipt::decode_result( + &body, + (call.request.id, cell), + [9; 32], + &keys(), + 1000, + 4096, + member + ) + .unwrap() + .err(), + Some(error) + ); + } +} + +#[test] +fn shared_receipt_purpose_cannot_authorize_mutation() { + let old = identity(2); + let target = ReplicaIdentity { + scope: cellar_logic::cell_group::GroupHistory::from(&old.scope), + member: old.member, + node: old.node, + disk: old.disk, + process: old.process, + }; + let base = Route::new_shared( + identity(1).node_identity(), + "node-1", + &target, + "node-2", + &cell_observation().version.history, + [9; 32], + ) + .unwrap(); + let lookup = base.clone().for_receipt().unwrap(); + assert!(lookup.requires_command_grant()); + assert!(!lookup.is_command()); + assert!(!lookup.is_enrollment()); + let auth = PeerAuth::new([17; 32], "node-1").unwrap(); + let options = TransportOptions::default(); + let body = receipt_call().encode(4096).unwrap(); + let request = RequestHead::new(lookup, 8, &body); + let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); + for purpose in ["Query", "Command", "Enrollment"] { + let mut value = serde_json::to_value(&request).unwrap(); + value["route"]["purpose"] = serde_json::json!(purpose); + let changed = FrameHead { + prefix: head.prefix, + header: serde_json::to_vec(&value).unwrap(), + tag: head.tag, + payload_len: head.payload_len, + }; + assert!(read_request(&changed, &auth, &[3; 32], &options).is_err()); + } +} diff --git a/crates/cellar/durability/raft/receipt_cell_wire.rs b/crates/cellar/durability/raft/receipt_cell_wire.rs new file mode 100644 index 000000000..ddb5be98d --- /dev/null +++ b/crates/cellar/durability/raft/receipt_cell_wire.rs @@ -0,0 +1,225 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt inspection has a distinct authenticated purpose and never stages a command. +use super::*; +use crate::raft::cell_groups::{CellReceiptRequest, CellReceiptResponse}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{CommandId, Durability}, +}; + +pub(in crate::raft) struct CellReceiptCall { + pub request: CellReceiptRequest, + pub timeout_ms: u64, + pub route_epoch: u64, + pub birth_digest: [u8; 32], +} + +pub(in crate::raft) struct CellReceiptValue { + pub retained: bool, + pub value: command::CellCommandValue, +} + +impl CellReceiptCall { + pub fn encode(&self, limit: usize) -> Result, TransportError> { + let r = &self.request; + if self.route_epoch == 0 || self.birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + if r.contract.kind != OperationKind::Command + || r.contract.operation.is_empty() + || r.contract.operation.len() > 128 + || r.contract.operation.chars().any(char::is_control) + || !(1..=3_600_000).contains(&self.timeout_ms) + || r.durability != Durability::Quorum + { + return Err(TransportError::Protocol); + } + let mut body = b"CLGCRQ01".to_vec(); + body.extend_from_slice(&r.id.client.bytes()); + body.extend_from_slice(&r.id.sequence().to_be_bytes()); + body.extend_from_slice(&self.route_epoch.to_be_bytes()); + body.extend_from_slice(&self.birth_digest); + string(&mut body, &r.contract.operation)?; + body.extend_from_slice(&r.contract.digest); + body.push(1); // Quorum; never silently lower an S3 request. + body.extend_from_slice(&self.timeout_ms.to_be_bytes()); + let length = u32::try_from(r.input.len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(r.input.len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&length.to_be_bytes()); + body.extend_from_slice(&r.input); + Ok(body) + } + + pub fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCRQ01" { + return Err(TransportError::Protocol); + } + let id = command::command_id(&mut body)?; + let route_epoch = u64::from_be_bytes(body.array()?); + let birth_digest = body.array()?; + if route_epoch == 0 || birth_digest == [0; 32] { + return Err(TransportError::Identity); + } + let operation = body.string(128)?; + if operation.is_empty() || operation.chars().any(char::is_control) { + return Err(TransportError::Protocol); + } + let contract = OperationContract { + operation, + kind: OperationKind::Command, + digest: body.array()?, + }; + if body.byte()? != 1 { + return Err(TransportError::Protocol); + } + let timeout_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&timeout_ms) { + return Err(TransportError::Protocol); + } + let length = u32::from_be_bytes(body.array()?) as usize; + let input = body.take(length)?.to_vec(); + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(Self { + request: CellReceiptRequest { + id, + contract, + input, + durability: Durability::Quorum, + }, + timeout_ms, + route_epoch, + birth_digest, + }) + } +} + +pub(in crate::raft) fn encode_result( + result: &Result, + limit: usize, +) -> Result, TransportError> { + let mut body = b"CLGCRS01".to_vec(); + match result { + Ok(response) => { + body.push(if response.receipt().is_some() { 0 } else { 13 }); + body.extend_from_slice(&response.command().client.bytes()); + body.extend_from_slice(&response.command().sequence().to_be_bytes()); + string(&mut body, response.token().as_str())?; + if let Some(receipt) = response.receipt() { + body.extend_from_slice(&receipt.retry_until_ms().to_be_bytes()); + let len = + u32::try_from(receipt.result().len()).map_err(|_| TransportError::Limit)?; + if body + .len() + .checked_add(4) + .and_then(|n| n.checked_add(receipt.result().len())) + .is_none_or(|n| n > limit) + { + return Err(TransportError::Limit); + } + body.extend_from_slice(&len.to_be_bytes()); + body.extend_from_slice(receipt.result()); + } + } + Err(CommandFailure::CommandConflict) => body.push(9), + Err(CommandFailure::CommandExpired) => body.push(10), + Err(CommandFailure::SequenceGap { next }) => { + body.push(11); + body.extend_from_slice(&next.to_be_bytes()); + } + Err(CommandFailure::NotLeader { leader }) => { + body.push(12); + body.extend_from_slice(&leader.map_or(0, MemberId::get).to_be_bytes()); + } + Err(error) => body.push(failure_code(error)), + } + if body.len() > limit { + return Err(TransportError::Limit); + } + Ok(body) +} + +pub(in crate::raft) fn decode_result( + bytes: &[u8], + expected: (CommandId, &CellHistory), + schema: [u8; 32], + keys: &VersionKeys, + now: u64, + limit: usize, + served_by: MemberId, +) -> Result, TransportError> { + if bytes.len() > limit { + return Err(TransportError::Limit); + } + let mut body = Reader(bytes); + if body.take(8)? != b"CLGCRS01" { + return Err(TransportError::Protocol); + } + let result = match body.byte()? { + status @ (0 | 13) => { + let command = command::command_id(&mut body)?; + if command != expected.0 { + return Err(TransportError::Authentication); + } + let token = crate::raft::version::CellVersionToken::from_encoded(&body.string(4096)?) + .map_err(|_| TransportError::Authentication)?; + let observation = keys + .verify_cell(&token, expected.1, schema, now) + .map_err(|_| TransportError::Authentication)?; + let retained = status == 0; + let (retry_until_ms, result) = if retained { + let retry = u64::from_be_bytes(body.array()?); + let len = u32::from_be_bytes(body.array()?) as usize; + (retry, body.take(len)?.to_vec()) + } else { + (0, vec![]) + }; + Ok(CellReceiptValue { + retained, + value: command::CellCommandValue { + command, + retry_until_ms, + value: cell::CellQueryValue { + result, + observation, + token, + served_by, + }, + }, + }) + } + 9 => Err(CommandFailure::CommandConflict), + 10 => Err(CommandFailure::CommandExpired), + 11 => Err(CommandFailure::SequenceGap { + next: u64::from_be_bytes(body.array()?), + }), + 12 => { + let id = u64::from_be_bytes(body.array()?); + Err(CommandFailure::NotLeader { + leader: if id == 0 { + None + } else { + Some(MemberId::new(id).map_err(|_| TransportError::Identity)?) + }, + }) + } + status => Err(decode_failure(status)?), + }; + if !body.0.is_empty() { + return Err(TransportError::Protocol); + } + Ok(result) +} diff --git a/crates/cellar/durability/raft/receipts.rs b/crates/cellar/durability/raft/receipts.rs index 86376f150..6a2df6c97 100644 --- a/crates/cellar/durability/raft/receipts.rs +++ b/crates/cellar/durability/raft/receipts.rs @@ -120,6 +120,34 @@ pub(super) fn lookup_identity( } } +/// An initial cell image may precede receipt schema installation. Losing schema +/// after a mutation, or retaining only half of it, is corruption, not absence. +/// A missing receipt never proves that a submitted command cannot still commit. +pub(super) fn lookup_cell_identity( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + cell_sequence: u64, +) -> Result, ExecutionError> { + let tables: u64 = connection.query_row( + "SELECT count(*) FROM sqlite_schema WHERE type='table' AND name IN ('__sys_raft_clients','__sys_raft_receipts')", + [], |row| row.get(0), + ).map_err(storage)?; + match tables { + 0 if cell_sequence == 1 => { + if request.id.sequence() == 1 { + Ok(None) + } else { + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 1, + })) + } + } + 2 => lookup_identity(connection, request, limits), + _ => Err(corrupt()), + } +} + enum Identity { Retained(StoredReceipt), Vacant { new_client: bool }, diff --git a/crates/cellar/durability/raft/receipts_tests.rs b/crates/cellar/durability/raft/receipts_tests.rs index f5e91fa4c..f4e0fa570 100644 --- a/crates/cellar/durability/raft/receipts_tests.rs +++ b/crates/cellar/durability/raft/receipts_tests.rs @@ -135,3 +135,58 @@ fn committed_lookup_preserves_identity_floor_and_allocation_checks() -> anyhow:: assert_eq!(client(&connection, original.id.client)?, Some((1, 1))); Ok(()) } + +#[test] +fn cell_lookup_distinguishes_initial_absence_from_lost_schema() -> anyhow::Result<()> { + let connection = Connection::open_in_memory()?; + let command = request(1, 1000); + let limits = ExecutionLimits::default(); + assert!(lookup_cell_identity(&connection, &command, &limits, 1)?.is_none()); + assert!(matches!( + lookup_cell_identity(&connection, &request(2, 1000), &limits, 1), + Err(ExecutionError::Refused(CommandFailure::SequenceGap { + next: 1 + })) + )); + for sequence in [0, 2] { + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, sequence), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + } + install(&connection)?; + // Inspection does not consume write capacity or invoke receipt maintenance. + let saturated = ExecutionLimits { + max_retained_receipts: 1, + max_client_identities: 1, + ..limits + }; + record(&connection, &command, vec![7])?; + assert_eq!( + lookup_cell_identity(&connection, &command, &saturated, 2)? + .unwrap() + .result(), + &[7] + ); + let mut other = command.clone(); + other.id = CommandId::new(ClientId::new([9; 16])?, 1)?; + assert!(lookup_cell_identity(&connection, &other, &saturated, 2)?.is_none()); + prune(&connection, command.id.client, 1, 1001)?; + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, 3), + Err(ExecutionError::Refused(CommandFailure::CommandExpired)) + )); + connection.execute_batch("DROP TABLE __sys_raft_receipts")?; + for sequence in [1, 3] { + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, sequence), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + } + connection.execute_batch("DROP TABLE __sys_raft_clients")?; + assert!(matches!( + lookup_cell_identity(&connection, &command, &limits, 3), + Err(ExecutionError::Refused(CommandFailure::CorruptState)) + )); + Ok(()) +} diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index b993a6e5a..b7d6bf340 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -43,6 +43,7 @@ mod cell_queries; mod command_origins; pub use cell_commands::{ CellCommandGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, + CellRemoteReceiptResponse, }; #[path = "query_cell_discovery.rs"] mod cell_query_discovery; diff --git a/crates/cellar/durability/raft/transport_options.rs b/crates/cellar/durability/raft/transport_options.rs index 566176353..7f35ec5b3 100644 --- a/crates/cellar/durability/raft/transport_options.rs +++ b/crates/cellar/durability/raft/transport_options.rs @@ -31,7 +31,7 @@ pub struct NamespaceQueryOrigin { pub peer: RaftPeerNode, } -/// Explicit command and enrollment permission for an exact namespace/process. +/// Explicit command, enrollment and receipt-inspection permission for an exact namespace/process. /// Read grants, discovery and Raft membership never create this permission. #[derive(Clone, Debug, PartialEq, Eq)] pub struct NamespaceCommandOrigin { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index ea6a12b4c..da2c6468f 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1530,3 +1530,18 @@ async fn shared_remote_enrollment_revocation_retains_admitted_creation() -> Resu async fn shared_remote_enrollment_resolves_after_publication_deadline() -> Result<()> { Box::pin(runtime_fixture::remote_enrollment_lifecycle(2)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_requires_grants_and_never_executes_commands() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_ingress()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_keeps_unresolved_producers_at_capacity() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_pending()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_receipt_lookup_cancellation_preserves_active_writer() -> Result<()> { + Box::pin(runtime_fixture::receipt_lookup_cancelled()).await +} diff --git a/crates/cellar/tests/support/cell_group_receipt_lookup.rs b/crates/cellar/tests/support/cell_group_receipt_lookup.rs new file mode 100644 index 000000000..a21761f41 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_receipt_lookup.rs @@ -0,0 +1,455 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receipt-only inspection across real TLS and immutable SQLite images. +use super::query_origins::Router; +use super::remote_queries::{dispatcher, keys, wait_for}; +use super::*; +use cellar::application::consensus::NamespaceCommandOrigin; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +fn request(class: &RaftClass, input: &[u8]) -> Result { + Ok(CellReceiptRequest { + id: CommandId::new(ClientId::new([221; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + }) +} + +fn limits() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 8, + } +} + +async fn lookup( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + input: &[u8], +) -> Result { + let mut lookup = + dispatch.lookup_receipt(cell.clone(), request(class, input)?, limits(), None)?; + lookup.begin().await?; + Ok(lookup) +} + +async fn command( + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + now: u64, +) -> Result { + let r = request(class, b"once")?; + Ok(dispatch + .command( + cell.clone(), + CellCommandRequest { + id: r.id, + contract: r.contract, + input: r.input, + durability: r.durability, + retry_until_ms: now + 60_000, + }, + limits(), + None, + ) + .await?) +} + +fn grant(fixture: &RuntimeCluster, router: &Router) -> Result<()> { + let g = fixture.cluster.birth.route().history; + fixture.transports[0].grant_command_origin(NamespaceCommandOrigin { + cluster: g.cluster, + namespace: g.namespace, + revision: 1, + peer: router.peer.clone(), + })?; + Ok(()) +} + +pub(crate) async fn receipt_lookup_ingress() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + fixture.cell_options(&a).await?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let deadline = read.attempt().unwrap().deadline_ms(); + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + router.grant(&fixture, 1)?; + read.retry().await?; + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + assert_eq!(calls.load(Ordering::SeqCst), 0); + grant(&fixture, &router)?; + read.retry().await?; + assert_eq!(read.attempt().unwrap().deadline_ms(), deadline); + let unknown = read + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert!(unknown.result().is_none()); + assert!(unknown.retry_until_ms().is_none()); + assert_eq!(unknown.observation().version.sequence.get(), 1); + assert_eq!(unknown.command(), request(&class, b"once")?.id); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(router.transport.usage().query_items, 1); + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(unknown); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let committed = controllers[0] + .command( + command(&dispatch, &class, &a, now).await?, + fixture.cell_options(&a).await?, + )? + .response() + .await?; + assert_eq!(committed.receipt().receipt.result(), b"once"); + drop(committed); + read.retry().await?; + let found = read + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(found.result(), Some(b"once".as_slice())); + assert_eq!(found.retry_until_ms(), Some(now + 60_000)); + assert_eq!(found.observation().version.sequence.get(), 2); + assert_eq!(found.served_by(), MemberId::new(1)?); + let observed = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + assert_eq!( + keys(190)?.verify_cell(found.token(), &a, class.schema_digest(), observed)?, + *found.observation() + ); + drop(found); + let mut conflict = lookup(&dispatch, &class, &a, b"different").await?; + assert!(matches!( + conflict + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err(), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + ))) + )); + drop(conflict); + let mut gap = request(&class, b"once")?; + gap.id = CommandId::new(gap.id.client, 3)?; + let mut gap = dispatch.lookup_receipt(a.clone(), gap, limits(), None)?; + gap.begin().await?; + assert!(matches!( + gap.execute_remote(&router.transport, MemberId::new(1)?) + .await + .err(), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::SequenceGap { next: 2 } + ))) + )); + drop(gap); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(gateway); + read.retry().await?; + assert!(read + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .is_err()); + drop(read); + assert!(router.log.cell_groups()?.is_empty()); + assert_eq!(dispatch.usage(), (0, 0)); + router.close().await?; + for c in controllers { + c.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +async fn allow_two_sql_reservations( + fixture: &mut RuntimeCluster, + group: GroupHistory, +) -> Result<()> { + let (old_pool, applier) = fixture.appliers.remove(&(0, group)).context("old pool")?; + old_pool.close().await?; + drop(old_pool); + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((0, group), (pool, applier)); + Ok(()) +} + +pub(crate) async fn receipt_lookup_pending() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + allow_two_sql_reservations(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let mut mutation = command(&dispatch, &class, &a, now).await?; + mutation.begin().await?; + assert!(matches!( + mutation + .prepare(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?, + CellCommandPreparation::Prepared + )); + // Inspection can coexist with a pending producer, but it shares bounded + // SQL admission and cannot expose or retire the uncommitted receipt. + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(calls.load(Ordering::SeqCst), 1); + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let empty = read + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?; + assert!(empty.receipt().is_none()); + assert_eq!(empty.observation().version.sequence.get(), 1); + assert_eq!(pool.usage()?.active_cells, 1); + let mut saturated = lookup(&dispatch, &class, &a, b"once").await?; + assert!(matches!( + saturated + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await, + Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded + ))) + )); + assert_eq!(pool.usage()?.active_cells, 1); + drop(saturated); + drop(empty); + let r = request(&class, b"once")?; + for invalid in [0, 1, 2] { + let authority = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + if invalid == 2 { + CellReadConsistency::Stale + } else { + CellReadConsistency::Linearizable + }, + read.attempt().unwrap().deadline_ms(), + ) + .await?; + let error = pool + .lookup_receipt_registered( + fixture.cell_options(&a).await?, + authority, + class.clone(), + CommandRequest { + id: r.id, + operation: r.contract.operation.clone(), + contract_digest: if invalid == 0 { + [0; 32] + } else { + r.contract.digest + }, + request_digest: if invalid == 1 { + [0; 32] + } else { + command_request_digest(&r.input) + }, + durability: Durability::Quorum, + retry_until_ms: 0, + }, + r.input.clone(), + ) + .await + .err() + .context("invalid lookup refuses")?; + match invalid { + 0 => assert!(matches!(error, ExecutionError::Registration(_))), + 1 => assert!(matches!( + error, + ExecutionError::Refused(CommandFailure::CommandConflict) + )), + _ => assert!(matches!( + error, + ExecutionError::Invalid("receipt lookup requires quorum read") + )), + } + } + assert_eq!(pool.usage()?.active_cells, 1); + assert!(matches!( + mutation.submit().await?, + CellCommandSubmission::Accepted(_) + )); + wait_for(async || { + match fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await + { + Ok(authority) => Ok(authority.cell().version.sequence.get() == 2), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + read.retry().await?; + let found = read + .execute(&fixture.runtimes[0], &pool, fixture.cell_options(&a).await?) + .await?; + assert_eq!(found.receipt().context("receipt")?.result(), b"once"); + assert_eq!(pool.usage()?.active_cells, 1); // Lookup cannot finish another owner. + drop(found); + assert!(matches!( + mutation.settle().await?, + CellCommandSettlement::Committed(_) + )); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(read); + drop(mutation); + for c in &controllers { + c.close().await?; + } + // Closed controller handles cannot keep the SQL engine claim alive. + drop(fixture.appliers.remove(&(0, route.history))); + drop(pool); + let replacement = apply::pool(&fixture, 0)?; + replacement.close().await?; + drop(replacement); + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn receipt_lookup_cancelled() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + let _ = entered.lock().unwrap().take().context("entered")?.send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + allow_two_sql_reservations(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + let gateway = fixture.transports[0].install_cell_command_gateway( + dispatch.clone(), + controllers[0].clone(), + fixture.cell_options(&a).await?, + )?; + let router = Router::start(&fixture).await?; + router + .transport + .install_cell_query_group(fixture.tcp_route(route.history)?)?; + grant(&fixture, &router)?; + let now = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 }) + .await?; + let work = controllers[0].command( + command(&dispatch, &class, &a, now).await?, + fixture.cell_options(&a).await?, + )?; + tokio::time::timeout(Duration::from_secs(5), entered_rx).await??; + let mut read = lookup(&dispatch, &class, &a, b"once").await?; + let transport = router.transport.clone(); + let task = tokio::spawn(async move { + read.execute_remote(&transport, MemberId::new(1).unwrap()) + .await + }); + let pool = fixture.appliers[&(0, route.history)].0.clone(); + wait_for(async || Ok(pool.usage()?.queued_or_running >= 2)).await?; + task.abort(); + assert!(task.await.err().context("cancelled lookup")?.is_cancelled()); + wait_for(async || Ok(router.transport.usage().query_items == 0)).await?; + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + let committed = work.response().await?; + drop(committed); + wait_for(async || Ok(pool.usage()?.queued_or_running == 0)).await?; + let mut retry = lookup(&dispatch, &class, &a, b"once").await?; + let found = retry + .execute_remote(&router.transport, MemberId::new(1)?) + .await?; + assert_eq!(found.result(), Some(b"once".as_slice())); + assert_eq!(calls.load(Ordering::SeqCst), 1); + drop(found); + drop(retry); + drop(gateway); + router.close().await?; + for c in controllers { + c.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 14941bc28..def00f237 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -715,3 +715,5 @@ pub(super) use sql::cached_dispatch_execution; pub(super) use sql::{remote_command_lifecycle, remote_commands}; pub(super) use sql::{remote_enrollment, remote_enrollment_lifecycle}; + +pub(super) use sql::{receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c887e48ae..9d9158bc6 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -564,3 +564,9 @@ pub(crate) use remote_commands::{remote_command_lifecycle, remote_commands}; #[path = "cell_group_remote_enrollment.rs"] mod remote_enrollment; pub(crate) use remote_enrollment::{remote_enrollment, remote_enrollment_lifecycle}; + +#[path = "cell_group_receipt_lookup.rs"] +mod receipt_lookup; +pub(crate) use receipt_lookup::{ + receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending, +}; From 7e09e792ad36c4e825d47cd5dd58471ab241310c Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 10:55:28 -0700 Subject: [PATCH 109/568] docs: record qualified shared receipt lookup and next CG gaps --- docs/design/multi-raft/cell-groups/README.md | 8 +++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 62 +++++++++++-------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 48 ++++++++++++++ .../multi-raft/cell-groups/protocols.md | 19 +++++- docs/design/multi-raft/cell-groups/tasks.json | 47 +++++++++++++- 7 files changed, 157 insertions(+), 33 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 40373fab2..843374cdb 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -228,6 +228,14 @@ transfer `.cellar/cell-groups-evidence/CG-04-enrollment-rpc/` (276 source inputs 4 executables). Safe standalone receipt lookup and public SDK cutover remain; CG-04 remains IN_PROGRESS. +At `fbc15053`, standalone receipt lookup uses a quorum image without invoking +callbacks or releasing unresolved producers. It requires explicit command grants, +returns signed observations and keeps absence ambiguous. The partial gate passes +227 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-receipt-rpc/` (279 source inputs, +4 executables). Dynamic group lifecycle/lazy birth and per-group failure +isolation are next; CG-04 remains IN_PROGRESS and public cutover is still pending. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 7d2f0c8cb..f3560faf1 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "0fe665d0267f1b831b172aad5e8d435f5bf0cd39", + "source_revision": "fbc15053f8dcfd3b46078abad3cbbae7b68da8d6", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 5ba21994c..1e99e92ba 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,24 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `0fe665d0`, adding remote enrollment over +The newest reviewed source baseline is `fbc15053`, adding standalone local and +remote receipt inspection with distinct authenticated purpose and explicit command +permission. Registered contract/digest/identity validation precedes immutable SQL +on a fresh catalog-bound quorum image. Exact retained results and ambiguous +absence carry signed cell observations; lookup never invokes callbacks or retires +another producer. Corrupt schema, expiry and conflict refuse. SQL admission stays +bounded; cancelled queued reads retain physical ownership through completion. +Controller read service references are weak, so closed handles cannot retain the +SQL engine claim. Three native cases cover both image modes; remote cases use +four TLS endpoints in one OS process. Three wire cases and one added SQLite case +cover identity, empty results, absence, capacity and schema loss. +The record at `.cellar/cell-groups-evidence/CG-04-receipt-rpc/checkpoint.json` +binds 227 Rust cases, twelve commands, 279 source inputs and 4 executables. +Transfer the ignored evidence directory with previous checkpoints. CG-04 remains +IN_PROGRESS. Next implement bounded dynamic controller registration/lazy birth, +then isolate per-group failures. Public SDK/discovery and configuration lifecycle +remain open; shared-group RustFS recovery and load remain unqualified. +The preceding reviewed source baseline is `0fe665d0`, adding remote enrollment over the same bounded write exchange and exact controller registration as commands. Query, Command and Enrollment have distinct authenticated mandatory purposes. Explicit namespace/process write grants gate initialization; discovery/read grants @@ -491,31 +508,24 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Connect public `RaftCellHandle` operations and remote shared command/query/ - enrollment dispatch to the existing controller and SQL pool. Local registered - creation now uses catalog routing and immutable creation records, preserving - response cancellation/deadline ownership and exact retries after mutation. - Reuse those owners and captured identities; do not add per-cell workers or - replay uncertain callbacks. Confirm route changes through the catalog and - provide safe leader receipt lookup when a follower has no original producer. - Missing local state never proves non-commit. Enrollment still requires an - attached group in a fixed roster: integrate lazy birth and dynamic registration. - Attached-peer shared reads now reuse the separate query connection with - physical group/logical cell binding and host-owned gateway registration. - Explicit nonmember shared discovery and real-TLS grant revocation now pass - scoped tests; public-host automatic discovery/SDK binding remain pending. - Bounded invocation range caching now passes metadata-read and real-runtime - cases. The destination retains per-RPC catalog admission: removing that read - requires replicated transfer seals and serving-admission invalidation, linked - to CG-09. Do not substitute a TTL or cached hint for that authority contract. - Remote commands now use explicit write grants and response-independent - controller producers; enrollment now shares that write path. Keep read grants - independent. Add standalone safe receipt lookup without treating missing state as proof - of non-commit or replaying unresolved callbacks. - The old host refuses registered initializers until its actual replacement; - do not remove that refusal without a functioning shared serving path. Request - and namespace S3 minima must continue refusing until CG-06 provides real archive - proofs. Add per-group failure isolation before claiming runtime completion. +1. Deliver bounded dynamic controller registration and lazy group birth. The + current controller has a fixed roster, although command, enrollment, query and + receipt dispatch now reuse its shared owners. Define one registration lifecycle + spanning runtime, transport and applier; revalidate catalog admission before + serving and retain physical ownership through cancellation or close. Test + duplicate/foreign registration, capacity, attach/close races, failed birth and + selected-root restart while another group continues serving. Pair this with + per-group failure isolation: quarantine a failed group without closing healthy + groups or discarding retained SQL/apply work. These are the critical gaps for + continuing CG-04, before exposing the public SDK replacement. + Keep explicit write grants independent from read grants. Standalone lookup now + preserves ambiguous absence and never replays or retires unresolved callbacks. + Destination routing still requires fresh catalog admission: removing that read + depends on replicated transfer seals and serving-admission invalidation in + CG-09. A TTL or cached hint cannot substitute for that authority. + Connect public `RaftCellHandle` and automatic discovery only through the actual + shared host. Keep the old host's initializer refusal until replacement. + Request and namespace S3 minima must refuse until CG-06 provides archive proofs. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 604f42665..af0dfeed8 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 69 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 72 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index ed73a286d..d000d3a09 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,54 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 standalone receipt inspection + +Qualified source: `fbc15053f8dcfd3b46078abad3cbbae7b68da8d6`. + +Local and remote receipt inspection now select an immutable committed cell image +through a correlated quorum read. The registered command contract, payload digest +and authenticated identity are validated before lookup. Inspection neither stages +a producer nor invokes the command or initializer; it does not install receipt +schema or run receipt maintenance. Initial images may lack receipt tables; +partial schema or schema loss after mutation refuses as corruption. + +The existing bounded RPC exchange carries a distinct authenticated Receipt purpose. +Explicit namespace/process command permission and the exact controller/SQL owner +are required; read permission alone cannot inspect command results. Fresh receiver +catalog binding checks group, schema, epoch and birth. Success binds the original +command identity and signed cell observation, distinguishing a retained empty +result from absence. Absence remains ambiguous even after a quorum read: the +original command may still arrive or commit. Lookup cannot release its producer. +Replicated expiry, conflicts and sequence gaps remain typed refusals. + +Three native cases run both image modes. Real TLS ingress exercises permission, +identity, retry/deadline, consumed-attempt, reply-admission and gateway-drop checks. +A prepared writer remains owned while lookup sees the old image, while the shared +SQL reservation budget is saturated, and after its receipt becomes committed. +Cancellation of a queued remote lookup preserves an entered controller writer. +A closed controller uses a weak SQL handle, allowing a replacement pool after +joined shutdown even when the closed controller handle remains retained. Three +wire cases and a real SQLite schema-loss case qualify serialization and inspection +boundaries. The initial fixture failures and their corrected bounded admission +setup are retained as diagnostic logs; production limits were not loosened. + +All 227 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, native dependency isolation, docs, source preflight +and 37 plan-validator cases. The record at +`.cellar/cell-groups-evidence/CG-04-receipt-rpc/checkpoint.json` binds 279 +source inputs and 4 copied executables. Transfer the entire directory with +prior checkpoints; local ignored evidence is not included in the PR. + +CG-04 remains IN_PROGRESS. The critical next gap is dynamic group lifecycle: +replace the fixed controller roster with bounded registration and lazy birth, +while preserving exact consensus, transport, apply and SQL ownership. Qualify +attach/close races, cancellation and recovery before publishing serving authority. +Per-group failure isolation is the next companion gap: a failed group must stop +serving while unrelated groups continue, and retained physical work must join. +Public SDK/discovery cutover, configuration handling, complete-group snapshots, +archive/reclamation and shared RustFS/load qualification remain open. No latency, +throughput, independent-process failure or production-cutover claim is made. + ## CG-04 remote enrollment through the shared write owner Qualified source: `0fe665d0267f1b831b172aad5e8d435f5bf0cd39`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 016044a2f..c0aecdcc5 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -130,7 +130,7 @@ removed. A replaced discovery view refuses captured requests and late replies; removal does not revoke the destination's independently managed namespace grant. Grant revocation cancels read waiters without releasing physically entered SQL. The public host must supply trusted discovery and explicit permissions; automatic -discovery/SDK binding and standalone receipt lookup remain +discovery/SDK binding and dynamic group lifecycle remain integration work. Remote commands use the same bounded application RPC lane with a mandatory, @@ -162,6 +162,23 @@ revocation and response deadline. Return only the exact committed creation recor including the original enrollment image after later mutations. That record is not a current-version token. Conflicting creation/incarnation refuses without replay. +Standalone receipt lookup has its own authenticated purpose and requires the +explicit command-origin grant. Read permission alone cannot inspect command +results. The exact registered controller supplies the runtime and SQL pool; +receiver admission checks fresh catalog binding and selects a correlated quorum +read. Registered contract, payload digest and authenticated command identity +validation precede receipt inspection. Immutable reads use the existing worker, +read admission, cancellation and deadline owners without a private writer, +initializer, command callback or receipt maintenance. + +A retained receipt returns its exact result/retry horizon with a signed cell +observation. An absent receipt remains ambiguous, even after a quorum read: +original delayed or unresolved work may still commit. Lookup never releases a +producer or produces a NotCommitted proof. Preserve replicated expiry/conflict +and sequence-gap failures. Only a sequence-1 image can lack both receipt tables; +partial schema or later schema loss refuses as corruption. Missing and an empty +committed result have distinct wire encodings and API results. + The current lane serializes RPC exchanges per peer; this is not load qualification. ## Apply, checkpoint and archive diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 24c8d18fd..d05c5f080 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -521,7 +521,10 @@ "crates/cellar/tests/support/cell_group_remote_commands.rs", "crates/cellar/durability/raft/enrollment_cell_wire.rs", "crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs", - "crates/cellar/tests/support/cell_group_remote_enrollment.rs" + "crates/cellar/tests/support/cell_group_remote_enrollment.rs", + "crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs", + "crates/cellar/durability/raft/receipt_cell_wire.rs", + "crates/cellar/tests/support/cell_group_receipt_lookup.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -668,7 +671,7 @@ ] }, { - "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Standalone safe receipt lookup remains required integration; remote enrollment is specified in the following step. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", + "action": "Connect remote shared commands to the existing bounded RPC lane and controller. Authenticate command purpose separately from read payloads, require independent bounded namespace/process write grants even for voters, and include those grants in endpoint consistency, process refresh, fleet revocation and pool lifecycle. Preserve captured inputs, immutable command identity, remaining timeout and attempt budget. Destination fresh catalog binding precedes actual leader/SQL authority; only committed exact receipts and signed cell observations return success. Disconnect/revocation cannot cancel already admitted ownership. Also run all query_wire tests, including shared_command_wire_preserves_inputs_and_refuses_query_payloads, shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply and shared_command_results_bind_identity_history_and_typed_refusals. Remote enrollment and standalone receipt lookup are specified in later steps. Run command_grants_bound_tombstones_without_changing_read_permission in the full transport suite to qualify independent read grants, write tombstone caps and shared-pool cleanup. Distinguish follower execution-authority requests as NotLeader from leaders awaiting durable apply (NotReady); preserve stale captured-authority checks.", "tests": [ "shared_remote_commands_require_write_grants_and_exact_receipts", "shared_remote_commands_survive_disconnected_sql_waiters", @@ -676,13 +679,21 @@ ] }, { - "action": "Route enrollment through the same bounded write RPC exchange, explicit namespace/process grants and exact controller registration as commands. Authenticate Query, Command and Enrollment as distinct mandatory purposes under marker 1, with no legacy reader. Preserve creation identity, captured inputs, preview identity and original routing budget. Recheck fresh destination catalog and leader authority before transferring initialization to the controller; return only the exact committed creation record, never treat the initial image as a current version token. Reuse shared reply budgets and preserve ownership after response disconnect, revocation and publication deadline. Run all query_wire tests including shared_enrollment_wire_preserves_captured_identity_and_inputs, shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read and shared_enrollment_reply_binds_creation_cell_schema_and_initial_image, plus all existing command/query and controller enrollment regressions. Standalone safe receipt lookup, dynamic roster and public SDK cutover remain open.", + "action": "Route enrollment through the same bounded write RPC exchange, explicit namespace/process grants and exact controller registration as commands. Authenticate Query, Command and Enrollment as distinct mandatory purposes under marker 1, with no legacy reader. Preserve creation identity, captured inputs, preview identity and original routing budget. Recheck fresh destination catalog and leader authority before transferring initialization to the controller; return only the exact committed creation record, never treat the initial image as a current version token. Reuse shared reply budgets and preserve ownership after response disconnect, revocation and publication deadline. Run all query_wire tests including shared_enrollment_wire_preserves_captured_identity_and_inputs, shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read and shared_enrollment_reply_binds_creation_cell_schema_and_initial_image, plus all existing command/query and controller enrollment regressions. Standalone receipt lookup follows below; dynamic roster and public SDK cutover remain open.", "tests": [ "shared_remote_enrollment_requires_write_grants_and_exact_creation", "shared_remote_enrollment_survives_disconnected_initializer", "shared_remote_enrollment_revocation_retains_admitted_creation", "shared_remote_enrollment_resolves_after_publication_deadline" ] + }, + { + "action": "Add standalone local and remote receipt inspection with a distinct authenticated purpose on the bounded existing RPC exchange. Require explicit command-origin permission and exact controller/SQL ownership; validate command contract, payload digest and authenticated identity before reading receipts. Select a fresh catalog-bound quorum image and use immutable SQL without staging a producer, installing schema, maintaining receipts, or invoking a command/initializer. Return exact retained result/identity and signed cell observation, or ambiguous absence; absence must never retire an unresolved producer. Reuse physical read cancellation, deadline, byte accounting and joined cleanup. Run full query_wire and receipts suites, including shared_receipt_wire_refuses_command_payloads_and_preserves_identity, shared_receipt_reply_distinguishes_unknown_from_empty_committed_result, shared_receipt_purpose_cannot_authorize_mutation and cell_lookup_distinguishes_initial_absence_from_lost_schema. Keep corrupt/partial schema and replicated expiry distinct from absence. Dynamic roster/lazy group birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete.", + "tests": [ + "shared_receipt_lookup_requires_grants_and_never_executes_commands", + "shared_receipt_lookup_keeps_unresolved_producers_at_capacity", + "shared_receipt_lookup_cancellation_preserves_active_writer" + ] } ], "test_cases": [ @@ -1344,6 +1355,36 @@ "I05", "I12" ] + }, + { + "name": "shared_receipt_lookup_requires_grants_and_never_executes_commands", + "oracle": "Over four real TLS endpoints in one process and both image modes, receipt inspection requires explicit command-origin permission; discovery and read grants refuse. Initial receipt absence returns an authenticated cell observation without invoking the command. After a normal commit, lookup returns the exact retained result and retry horizon without a second callback. Conflicting payload and sequence gaps remain typed. Original deadlines, one-use attempts, result admission and gateway drop remain enforced; the router has no local group replica.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_receipt_lookup_keeps_unresolved_producers_at_capacity", + "oracle": "With a prepared command retaining its producer, quorum lookup observes only the committed image and reports ambiguous absence. Holding that lookup result fills the two-reservation SQL budget; another lookup refuses overload without releasing the producer. It neither exposes the uncommitted receipt nor releases the producer. Contract/digest mismatch and non-quorum read authority refuse. After the original command publishes, lookup returns its receipt while leaving producer cleanup to its actual owner; no callback reruns. Closed controller handles cannot retain the engine claim after the SQL pool closes; a replacement pool starts. Run both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_receipt_lookup_cancellation_preserves_active_writer", + "oracle": "Hold a local controller command inside SQL, queue a remote lookup on a bounded two-slot fixture worker, then disconnect the lookup. Network admission drains while the active writer remains controller-owned. Release SQL, join the cancelled read, and inspect the committed receipt without a second command callback in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 11:17:59 -0700 Subject: [PATCH 110/568] feat: register shared Raft groups under controller ownership --- .../durability/raft/cell_groups/README.md | 31 +- .../durability/raft/cell_groups/apply/mod.rs | 19 + .../raft/cell_groups/controller/mod.rs | 26 +- .../cell_groups/controller/registration.rs | 348 +++++++++ .../raft/cell_groups/controller/requests.rs | 2 + .../raft/cell_groups/controller/worker.rs | 34 +- .../cell_groups/dispatch/command/owned.rs | 10 +- .../raft/cell_groups/dispatch/enrollment.rs | 10 +- .../cellar/durability/raft/cell_groups/mod.rs | 3 +- crates/cellar/durability/raft/transport.rs | 1 + .../durability/raft/transport_shared.rs | 64 +- .../tests/native_cell_groups_execution.rs | 20 + .../tests/support/cell_group_registration.rs | 713 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 6 + crates/cellar/tests/support/cell_group_tcp.rs | 5 +- .../tests/support/raft_catalog_fault.rs | 23 + 17 files changed, 1289 insertions(+), 30 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/registration.rs create mode 100644 crates/cellar/tests/support/cell_group_registration.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 5bc731b9a..97153a5ae 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -1,6 +1,6 @@ -# Cell-group durable state +# Cell-group state and execution CG-02's storage boundary is qualified. The component supplies immutable manifest pages, their filesystem owner, complete-root preparation and atomic Raft Engine root publication. Production host admission @@ -323,6 +323,29 @@ makes subsequent routing failures carry its original ambiguous command ID; a redirect never proves the earlier proposal uncommitted. Physical command producers remain the dispatcher's responsibility and must outlive cancelled waiters. -This component prepares routing for the shared host. Native tests exercise real -conditional directory changes and controlled reads, not synthetic serving proofs. -It does not dispatch commands or qualify the production routing integration. +The directory supplies routing observations, not serving authority. The shared +dispatcher and controller combine those observations with actual quorum, SQL and +transport ownership. The public host cutover remains unfinished. + + +## Dynamic controller registration + +`CellGroupController::register` adds a prepared group to a running node, including +one whose initial roster is empty. The caller supplies a bootstrap permit, +verified selected root, the applier owning that root's manifest page store, and +an authenticated physical peer route. Pending and active groups share the group +limit; attachment and unconsumed replies also use the shared item/byte budget. +Admission failure returns the prepared owners. After admission, dropping the +response does not cancel attachment. + +The worker registers the applier before polling runtime attachment, revalidates +the catalog permit through the existing runtime, and activates roster membership +only after success. Current-term quorum is still required for serving. Physical +route registration refuses adopting an existing route; cleanup checks exact route +identity. Failed attachment releases its group slot before publishing a refusal. +Shutdown joins attachment before closing transport, SQL and consensus ownership. + +A fatal group apply/command currently closes the controller. Per-group failure +isolation, request-driven group provisioning, configuration lifecycle and public +SDK integration remain unfinished. See the delivery package for qualification +status; dynamic registration does not establish those later capabilities. diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 7ef0bde98..65455e920 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -75,6 +75,25 @@ impl CellGroupApplier { Arc::ptr_eq(&self.group.owner, engine) } + pub(crate) fn check_startup(&self, root: &PreparedGroupManifest) -> Result<(), ExecutionError> { + let owner = self.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => ExecutionError::Busy, + std::sync::TryLockError::Poisoned(_) => { + ExecutionError::Invalid("poisoned registration applier") + } + })?; + if !self.belongs_to_engine(&root.engine_owner) + || self.history() != root.marker().route().history + || !Arc::ptr_eq(&owner.pages.owner, &root.page_owner) + || self.group.group_manifest().map_err(storage)?.as_ref() != Some(root.marker()) + { + return Err(ExecutionError::Invalid( + "registration selected root/page owner", + )); + } + Ok(()) + } + pub(crate) fn reservation_bytes(&self, work: &CellGroupApply) -> Result { self.check(work)?; let owner = self.storage.try_lock().map_err(|error| match error { diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 19511772b..753225a45 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -1,13 +1,18 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Owns a fixed roster's consensus events, transport dispatch and physical apply. +//! Owns registered groups' consensus events, transport dispatch and physical apply. //! Admitted command ownership survives response cancellation and expiry. mod calls; mod commands; mod enrollments; +mod registration; mod requests; pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; +pub(crate) use registration::GroupRoster; +pub use registration::{ + CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, +}; mod worker; pub use commands::{CellCommandAdmissionError, CellCommandCall}; @@ -18,7 +23,7 @@ use crate::raft::{ transport::{RaftTransport, TransportError}, }; use futures_util::FutureExt; -use std::collections::{BTreeMap, BTreeSet}; +use std::collections::BTreeMap; use std::sync::{ atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, Arc, Mutex, @@ -31,7 +36,7 @@ pub struct CellGroupControllerLimits { pub max_parallel_apply: usize, /// One node timer backs off transient SQL admission, never one timer per cell. pub retry_interval_ms: u64, - /// Shared by commands and enrollments, including queued work, resolution + /// Shared by commands, enrollments and registrations, including queued work, resolution /// and unconsumed responses. pub max_commands: usize, /// Invocation/input/options retention. SQL capture, apply and result bytes @@ -67,6 +72,7 @@ impl From for CellGroupControllerError { #[derive(Clone, Copy, Debug)] pub struct CellGroupControllerUsage { pub groups: usize, + pub pending_registrations: usize, pub active_apply: usize, pub pending_apply: usize, pub applied: u64, @@ -81,7 +87,7 @@ pub struct CellGroupControllerUsage { struct Shared { runtime: CellGroupRuntime, sql: crate::raft::host_sql::cell_groups::WeakCellGroupSqlPool, - groups: BTreeSet, + groups: Arc, limits: CellGroupControllerLimits, domain: crate::asyncrt::DomainToken, commands: Arc, @@ -138,7 +144,7 @@ impl Drop for Handle { } } -/// One worker per node, with a bounded fixed roster and no cell-specific tasks. +/// One worker per node, with a bounded dynamic roster and no cell-specific tasks. /// Dropping the final public handle requests joined shutdown. An explicit close /// is required to observe its result. Runtime/transport/pool clones remain /// usable during operation; closing this owner closes all three services. @@ -153,8 +159,8 @@ impl CellGroupController { } /// Consume the exclusive event receiver after validating exact engine and - /// runtime ownership. Every listed group must already be attached. Later - /// attachment requires a future roster API; an unknown event fails closed. + /// runtime ownership. Initial groups must already be attached. Later groups + /// enter through register(); unknown events still fail closed. pub async fn start( runtime: CellGroupRuntime, events: CellGroupRuntimeEvents, @@ -168,7 +174,6 @@ impl CellGroupController { || limits.max_parallel_apply == 0 || limits.max_parallel_apply > limits.max_groups.min(256) || !(1..=60_000).contains(&limits.retry_interval_ms) - || appliers.is_empty() || appliers.len() > limits.max_groups || !(1..=65_536).contains(&limits.max_commands) || limits.max_command_bytes < 8192 @@ -200,7 +205,7 @@ impl CellGroupController { let shared = Arc::new(Shared { runtime: runtime.clone(), sql: sql.downgrade(), - groups: roster.keys().copied().collect(), + groups: GroupRoster::new(roster.keys().copied(), limits.max_groups), limits, domain: domain.clone(), commands: commands::budget(limits), @@ -254,6 +259,7 @@ impl CellGroupController { let (command_bytes, retained_commands) = shared.commands.usage(); CellGroupControllerUsage { groups: shared.groups.len(), + pending_registrations: shared.groups.pending(), command_bytes, retained_commands, active_apply: shared.active_apply.load(Ordering::Acquire), @@ -290,7 +296,7 @@ impl CellGroupController { invocation.begin().await?; if !invocation .attempt() - .is_some_and(|a| shared.groups.contains(&a.route().history)) + .is_some_and(|a| shared.groups.contains(&a.route().history).unwrap_or(false)) { return Err(super::CellDispatchError::Runtime(RuntimeError::NotFound)); } diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs new file mode 100644 index 000000000..b306178f3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -0,0 +1,348 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded attachment ownership survives response loss and joins before shutdown. +use super::calls::Delivery; +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{CellGroupBootstrapPermit, PreparedGroupManifest}; +use crate::raft::transport::{GroupRoute, PhysicalRegistration}; +use cellar_logic::cell_group::GroupHistory; +use tokio::sync::{mpsc, oneshot}; + +pub(crate) struct GroupRoster { + groups: Mutex>, + limit: usize, +} + +impl GroupRoster { + pub(super) fn new(groups: impl Iterator, limit: usize) -> Arc { + Arc::new(Self { + groups: Mutex::new(groups.map(|g| (g, true)).collect()), + limit, + }) + } + + pub(crate) fn contains(&self, group: &GroupHistory) -> Result { + Ok(self + .groups + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(group) + == Some(&true)) + } + + pub(super) fn len(&self) -> usize { + self.groups + .lock() + .map(|g| g.values().filter(|active| **active).count()) + .unwrap_or(0) + } + + pub(super) fn pending(&self) -> usize { + self.groups + .lock() + .map(|g| g.values().filter(|active| !**active).count()) + .unwrap_or(0) + } + + fn reserve(self: &Arc, group: GroupHistory) -> Result { + let mut groups = self.groups.lock().map_err(|_| RuntimeError::Closed)?; + if groups.contains_key(&group) { + return Err(RuntimeError::AlreadyAttached); + } + if groups.len() >= self.limit { + return Err(RuntimeError::Overloaded); + } + groups.insert(group, false); + Ok(Reservation { + roster: self.clone(), + group, + active: false, + }) + } +} + +struct Reservation { + roster: Arc, + group: GroupHistory, + active: bool, +} + +impl Reservation { + fn activate(&mut self) -> Result<(), RuntimeError> { + let mut groups = self + .roster + .groups + .lock() + .map_err(|_| RuntimeError::Closed)?; + let entry = groups.get_mut(&self.group).ok_or(RuntimeError::NotFound)?; + *entry = true; + self.active = true; + Ok(()) + } +} + +impl Drop for Reservation { + fn drop(&mut self) { + if !self.active { + if let Ok(mut groups) = self.roster.groups.lock() { + groups.remove(&self.group); + } + } + } +} + +/// The applier must own the exact page store used to verify the selected root. +/// Bootstrap/recovery grants no serving authority; attachment revalidates the +/// catalog and application admission still requires a current-term quorum. +pub struct CellGroupRegistration { + pub permit: CellGroupBootstrapPermit, + pub current: PreparedGroupManifest, + pub applier: CellGroupApplier, + pub transport: GroupRoute, +} + +pub struct CellGroupRegistrationError { + pub error: CellGroupControllerError, + pub registration: Box, +} + +impl std::fmt::Debug for CellGroupRegistrationError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("CellGroupRegistrationError") + .field("error", &self.error) + .finish_non_exhaustive() + } +} + +impl std::fmt::Display for CellGroupRegistrationError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + std::fmt::Display::fmt(&self.error, f) + } +} + +impl std::error::Error for CellGroupRegistrationError {} + +/// Dropping this waiter does not cancel attachment or release its group slot. +pub struct CellGroupRegistrationCall { + receiver: oneshot::Receiver>, +} + +impl CellGroupRegistrationCall { + pub async fn response(self) -> Result<(), Arc> { + self.receiver + .await + .map_err(|_| Arc::new(CellGroupControllerError::Closed))? + .result + } +} + +pub(super) struct Request { + pub registration: CellGroupRegistration, + reservation: Reservation, + reply: oneshot::Sender>, + permit: Arc, +} + +pub(super) struct Completed { + pub group: GroupHistory, + result: Result, + reservation: Reservation, + reply: oneshot::Sender>, + permit: Arc, +} + +impl CellGroupController { + /// Atomically reserve the controller slot and transfer attachment ownership. + /// Admission failure returns all prepared owners without starting attachment. + pub fn register( + &self, + mut registration: CellGroupRegistration, + ) -> Result { + let shared = &self.handle.0; + let admitted = (|| { + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + let route = registration.permit.route(); + if route != registration.current.marker().route() + || route.history != registration.applier.history() + || route.history != registration.transport.scope + || route.schema_digest != registration.transport.schema_digest + || !Arc::ptr_eq( + ®istration.current.engine_owner, + &shared.runtime.log().owner(), + ) + || !Arc::ptr_eq( + ®istration.permit.group().owner, + &shared.runtime.log().owner(), + ) + { + return Err(CellGroupControllerError::Invalid( + "registration identity/engine", + )); + } + registration + .applier + .check_startup(®istration.current) + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + if registration + .permit + .group() + .consensus_claimed + .load(Ordering::Acquire) + { + return Err(RuntimeError::AlreadyAttached.into()); + } + if registration.transport.peers.is_empty() || registration.transport.peers.len() > 128 { + return Err(CellGroupControllerError::Invalid( + "registration peer bounds", + )); + } + let mut bytes = registration + .current + .reserved_bytes() + .checked_add(16384) + .ok_or(RuntimeError::Overloaded)?; + for peer in &mut registration.transport.peers { + if peer.label.len() > 4096 || peer.server_name.len() > 4096 { + return Err(RuntimeError::Overloaded.into()); + } + bytes = bytes + .checked_add(peer.label.len() + peer.server_name.len() + 512) + .ok_or(RuntimeError::Overloaded)?; + peer.label = std::mem::take(&mut peer.label) + .into_boxed_str() + .into_string(); + peer.server_name = std::mem::take(&mut peer.server_name) + .into_boxed_str() + .into_string(); + } + registration.transport.peers = std::mem::take(&mut registration.transport.peers) + .into_boxed_slice() + .into_vec(); + let permit = shared + .commands + .acquire(bytes) + .ok_or(RuntimeError::Overloaded)?; + let reservation = shared.groups.reserve(route.history)?; + Ok((permit, reservation)) + })(); + let (permit, reservation) = match admitted { + Ok(pair) => pair, + Err(error) => { + return Err(CellGroupRegistrationError { + error, + registration: Box::new(registration), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Registration(Box::new(Request { + registration, + reservation, + reply, + permit: Arc::new(permit), + }))) + .map_err(|error| { + let (request, error) = match error { + mpsc::error::TrySendError::Closed(request) => { + (request, CellGroupControllerError::Closed) + } + mpsc::error::TrySendError::Full(request) => { + (request, RuntimeError::Overloaded.into()) + } + }; + let requests::Request::Registration(request) = request else { + unreachable!("registration envelope") + }; + CellGroupRegistrationError { + error, + registration: Box::new(request.registration), + } + })?; + Ok(CellGroupRegistrationCall { receiver }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + transport: RaftTransport, +) -> Completed { + let Request { + registration, + reservation, + reply, + permit, + } = request; + let group = registration.permit.route().history; + let result = async { + let guard = transport + .register_physical_group(registration.transport) + .map_err(CellGroupControllerError::Transport)?; + runtime + .attach(registration.permit, registration.current) + .await?; + Ok(guard) + } + .await; + Completed { + group, + result, + reservation, + reply, + permit, + } +} + +impl Completed { + /// A runtime-wide close or competing external owner is not a safely failed + /// registration. Preserve the conservative joined node shutdown boundary. + pub fn uncertain(&self) -> bool { + matches!( + &self.result, + Err(CellGroupControllerError::Runtime( + RuntimeError::Closed | RuntimeError::AlreadyAttached + )) + ) + } + + pub fn finish( + self, + closing: bool, + ) -> Result, CellGroupControllerError> { + let Self { + result, + mut reservation, + reply, + permit, + .. + } = self; + let outcome = if closing { + Err(CellGroupControllerError::Closed) + } else { + result.and_then(|guard| { + reservation.activate()?; + Ok(guard) + }) + }; + // A failed attachment's group capacity is free before publishing its + // refusal; success keeps the active roster entry after this guard drops. + drop(reservation); + let _ = reply.send(Delivery { + result: outcome + .as_ref() + .map(|_| ()) + .map_err(|e| Arc::new(e.clone())), + _permit: permit, + }); + match outcome { + Ok(guard) => Ok(Some(guard)), + Err(_) if closing => Ok(None), + Err(error) => Err(error), + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index e6fa4ceae..d988701c8 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -4,6 +4,7 @@ use super::*; pub(super) enum Request { + Registration(Box), Command(Box), Enrollment(Box), } @@ -39,6 +40,7 @@ pub(super) async fn drive( progress: tokio::sync::watch::Receiver<()>, ) -> Completed { match request { + Request::Registration(_) => unreachable!("registration has its own physical join owner"), Request::Command(request) => Completed::Command(Box::new( commands::drive(*request, runtime, sql, shared, progress).await, )), diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 23d6fdf14..150cce2b8 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -34,6 +34,8 @@ pub(super) struct Worker { requests: tokio::sync::mpsc::Receiver, commands: FuturesUnordered>, failed_command: Option, + registrations: FuturesUnordered>, + routes: BTreeMap, progress: tokio::sync::watch::Sender<()>, } @@ -62,6 +64,8 @@ impl Worker { requests, commands: FuturesUnordered::new(), failed_command: None, + registrations: FuturesUnordered::new(), + routes: BTreeMap::new(), progress: tokio::sync::watch::channel(()).0, } } @@ -125,7 +129,28 @@ impl Worker { .saturating_add(self.limits.retry_interval_ms); }, Some(request) = self.requests.recv() => { - self.commands.push(Box::pin(requests::drive(request, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))); + match request { + requests::Request::Registration(request) => { + let group = request.registration.applier.history(); + if self.roster.insert(group, request.registration.applier.clone()).is_some() { + return Err(CellGroupControllerError::Invalid("duplicate registered applier")); + } + self.registrations.push(Box::pin(registration::drive(*request, self.runtime.clone(), self.transport.clone()))); + } + other => self.commands.push(Box::pin(requests::drive(other, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))), + } + } + Some(completed) = self.registrations.next(), if !self.registrations.is_empty() => { + let group = completed.group; + let uncertain = completed.uncertain(); + match completed.finish(false) { + Ok(Some(guard)) => { self.routes.insert(group, guard); } + Ok(None) => unreachable!("active registration completion"), + Err(error) => { + self.roster.remove(&group); + if uncertain { return Err(error); } + } + } } Some(completed) = self.commands.next(), if !self.commands.is_empty() => { if let Some(error) = completed.failure() { @@ -190,6 +215,13 @@ impl Worker { } pub(super) async fn finish(&mut self) -> Result<(), CellGroupControllerError> { + self.requests.close(); + while self.requests.try_recv().is_ok() {} + // Attachment may already own a RawNode/claim or be inside catalog I/O. + // Poll it through completion before closing its physical dependencies. + while let Some(completed) = self.registrations.next().await { + let _ = completed.finish(true); + } self.transport.close().await; let sql = self .sql diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs index 6fa39fdbe..cb1ee1db7 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs @@ -2,7 +2,6 @@ //! Controller-only execution: response expiry never cancels a physical phase. use super::*; -use std::collections::BTreeSet; use tokio::sync::watch; impl CellCommandInvocation { @@ -76,7 +75,7 @@ impl CellCommandInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, - roster: &BTreeSet, + roster: &crate::raft::cell_groups::controller::GroupRoster, progress: &mut watch::Receiver<()>, ) -> Result { self.begin().await?; @@ -89,7 +88,7 @@ impl CellCommandInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, - roster: &BTreeSet, + roster: &crate::raft::cell_groups::controller::GroupRoster, progress: &mut watch::Receiver<()>, ) -> Result { let attempt = self @@ -100,7 +99,10 @@ impl CellCommandInvocation { self.session .validate_attempt(&attempt) .map_err(CellDispatchError::Routing)?; - if !roster.contains(&attempt.route().history) { + if !roster + .contains(&attempt.route().history) + .map_err(CellDispatchError::Runtime)? + { return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); } self.dispatched = true; diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index d21949ed7..e0425c1bc 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -9,7 +9,6 @@ use crate::raft::execution::CellEnrollmentAttempt; use crate::storage::CommandInputs; use cellar_logic::consensus::{CommandFailure, Durability}; use cellar_ltx::generation::ImageId; -use std::collections::BTreeSet; use tokio::sync::watch; #[derive(Clone, Copy)] @@ -204,7 +203,7 @@ impl CellEnrollmentInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, - roster: &BTreeSet, + roster: &crate::raft::cell_groups::controller::GroupRoster, progress: &mut watch::Receiver<()>, ) -> Result { self.started = true; @@ -218,7 +217,7 @@ impl CellEnrollmentInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, - roster: &BTreeSet, + roster: &crate::raft::cell_groups::controller::GroupRoster, progress: &mut watch::Receiver<()>, ) -> Result { let attempt = if self.remote_route.is_some() { @@ -239,7 +238,10 @@ impl CellEnrollmentInvocation { self.session .validate_attempt(&attempt) .map_err(CellDispatchError::Routing)?; - if !roster.contains(&attempt.route().history) { + if !roster + .contains(&attempt.route().history) + .map_err(CellDispatchError::Runtime)? + { return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); } let authority = runtime diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e5ca6cf6a..5f7484c8b 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -19,7 +19,8 @@ mod controller; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, CellGroupController, CellGroupControllerError, CellGroupControllerLimits, - CellGroupControllerUsage, + CellGroupControllerUsage, CellGroupRegistration, CellGroupRegistrationCall, + CellGroupRegistrationError, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index b7d6bf340..011bb89c4 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -22,6 +22,7 @@ use ::raft::prelude::{Message, MessageType}; use cellar_logic::consensus::{ClusterId, GroupScope, NamespaceId, NodeId, ReplicaIdentity}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; +pub(crate) use shared::PhysicalRegistration; use shared::SendRoute; use std::collections::{BTreeMap, BTreeSet}; use std::future::Future; diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 2026a0aed..37b7a74cc 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -56,6 +56,25 @@ impl RaftTransport { /// Install already-authorized physical membership. This creates no group, /// grants no serving authority and changes no committed configuration. pub fn install_physical_group(&self, route: PhysicalRoute) -> Result<(), TransportError> { + self.install_physical(route, false).map(|_| ()) + } + + pub(crate) fn register_physical_group( + &self, + route: PhysicalRoute, + ) -> Result { + let route = self.install_physical(route, true)?; + Ok(PhysicalRegistration { + state: Arc::downgrade(&self.handle.0), + route, + }) + } + + fn install_physical( + &self, + route: PhysicalRoute, + exclusive: bool, + ) -> Result, TransportError> { let state = &self.handle.0; let runtime = state.runtime.shared()?; let group = runtime @@ -110,8 +129,11 @@ impl RaftTransport { return Err(TransportError::Identity); } if let Some(old) = registry.physical_groups.get(&route.scope) { + if exclusive { + return Err(TransportError::Identity); + } if **old == route { - return Ok(()); + return Ok(old.clone()); } if old.revision >= route.revision || old.schema_digest != route.schema_digest { return Err(TransportError::StalePeer); @@ -159,12 +181,20 @@ impl RaftTransport { } } let starts = state.prepare_pools(&mut registry, &physical)?; - registry - .physical_groups - .insert(route.scope, Arc::new(route)); + let route = Arc::new(route); + registry.physical_groups.insert(route.scope, route.clone()); registry.prune_pools(); drop(registry); - state.start_pools(starts) + if let Err(error) = state.start_pools(starts) { + if exclusive { + drop(PhysicalRegistration { + state: Arc::downgrade(state), + route, + }); + } + return Err(error); + } + Ok(route) } pub fn remove_physical_group(&self, history: &GroupHistory) -> Result<(), TransportError> { @@ -373,3 +403,27 @@ impl Registry { Ok(()) } } + +/// Removing an old registration must never remove a replacement route. +pub(crate) struct PhysicalRegistration { + state: std::sync::Weak, + route: Arc, +} + +impl Drop for PhysicalRegistration { + fn drop(&mut self) { + let Some(state) = self.state.upgrade() else { + return; + }; + if let Ok(mut registry) = state.registry.lock() { + if registry + .physical_groups + .get(&self.route.scope) + .is_some_and(|r| Arc::ptr_eq(r, &self.route)) + { + registry.physical_groups.remove(&self.route.scope); + registry.prune_pools(); + } + }; + } +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index da2c6468f..6798e0ce9 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1545,3 +1545,23 @@ async fn shared_receipt_lookup_keeps_unresolved_producers_at_capacity() -> Resul async fn shared_receipt_lookup_cancellation_preserves_active_writer() -> Result<()> { Box::pin(runtime_fixture::receipt_lookup_cancelled()).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_preserves_live_groups_and_disconnected_waiters() -> Result<()> { + runtime_fixture::registration_live().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog() -> Result<()> { + runtime_fixture::registration_refusals().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_joins_attachment_before_cancelled_shutdown() -> Result<()> { + runtime_fixture::registration_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn dynamic_registration_recovers_selected_roots_into_an_empty_controller() -> Result<()> { + runtime_fixture::registration_recovery().await +} diff --git a/crates/cellar/tests/support/cell_group_registration.rs b/crates/cellar/tests/support/cell_group_registration.rs new file mode 100644 index 000000000..3dde8c796 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_registration.rs @@ -0,0 +1,713 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Runtime registration uses real engines, catalog reads, TLS and shared SQL. +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::Ordering; +#[path = "raft_catalog_fault.rs"] +#[allow(dead_code)] +mod catalog_fault; + +struct Prepared { + directory: RangeDirectory, + birth: CellGroupBirth, + store: Arc, + inputs: Vec, + number: u8, +} + +fn class() -> Result> { + Ok(Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, input| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + )) +} + +async fn prepare(f: &RuntimeCluster, number: u8, a: &CellHistory) -> Result { + let store = Arc::new(catalog_fault::Store::default()); + let old = f.cluster.birth.route(); + let scope = GroupHistory { + cluster: old.history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = f + .cluster + .domain + .run(catalog.open_namespace(NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + old.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + f.cluster + .domain + .run(directory.reserve_group_birth(&birth)) + .await??; + let initialize = directory + .prepare_initialization(ProposalId::new([number + 1; 16])?, &[birth.route().clone()])?; + f.cluster + .domain + .run(directory.publish(&initialize)) + .await??; + let birth = f + .cluster + .domain + .run(directory.materialize_group_birth(&birth)) + .await??; + let mut prepared = Prepared { + directory, + birth, + store, + inputs: vec![], + number, + }; + prepared.inputs = recover(f, &prepared, a).await?; + Ok(prepared) +} + +async fn recover( + f: &RuntimeCluster, + prepared: &Prepared, + a: &CellHistory, +) -> Result> { + let mut parts = vec![]; + for n in 0..3 { + let permit = f + .cluster + .domain + .run( + prepared + .directory + .bootstrap_group_member(&prepared.birth, &f.cluster.voters[n].log), + ) + .await??; + let pages = ManifestPageStore::open( + &f.cluster + ._root + .path() + .join(format!("registered-{}-{n}", prepared.number)), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = match f.cell_options_at(a, n).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let (applier, current) = CellGroupApplier::recover( + permit.group().clone(), + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + parts.push((permit, current, applier)); + } + let route = f.tcp_route(prepared.birth.route().history)?; + Ok(parts + .into_iter() + .map(|(permit, current, applier)| CellGroupRegistration { + permit, + current, + applier, + transport: route.clone(), + }) + .collect()) +} + +async fn duplicate( + f: &RuntimeCluster, + prepared: &Prepared, + input: &CellGroupRegistration, + n: usize, +) -> Result { + Ok(CellGroupRegistration { + permit: f + .cluster + .domain + .run( + prepared + .directory + .bootstrap_group_member(&prepared.birth, &f.cluster.voters[n].log), + ) + .await??, + current: input.current.clone(), + applier: input.applier.clone(), + transport: input.transport.clone(), + }) +} + +fn dispatcher( + f: &RuntimeCluster, + prepared: &Prepared, + class: Arc, +) -> Result { + Ok(CellGroupDispatcher::new( + &f.cluster.domain, + prepared.directory.clone(), + class, + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?) +} + +async fn enroll( + f: &RuntimeCluster, + controller: &CellGroupController, + dispatch: &CellGroupDispatcher, + cell: &CellHistory, +) -> Result<()> { + let invocation = dispatch + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([219; 16])?, + durability: Durability::Quorum, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 2, + }, + None, + ) + .await?; + let enrolled = controller + .enroll(invocation, f.cell_options(cell).await?)? + .response() + .await?; + assert_eq!(enrolled.record().cell(), cell); + drop(enrolled); + Ok(()) +} + +async fn replicate( + f: &RuntimeCluster, + route: &GroupRoute, + cell: &CellHistory, + rooted: bool, +) -> Result<()> { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + let accepted = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(route, cell, &bytes, rooted)?, + ) + .await?; + wait_for(async || { + for runtime in &f.runtimes { + if runtime.status(route.history).await?.applied.index < accepted.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +pub(crate) async fn registration_live() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + let duplicate = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + let mut excess = prepare(&f, 41, &a).await?; + prepared + .store + .hold_group_head_reads + .store(true, Ordering::Release); + let calls = prepared + .inputs + .drain(..) + .enumerate() + .map(|(n, input)| controllers[n].register(input)) + .collect::, _>>()?; + wait_for(async || { + Ok(prepared + .store + .active_group_head_reads + .load(Ordering::Acquire) + == 3) + }) + .await?; + assert_eq!(controllers[0].usage().groups, 1); + assert_eq!(controllers[0].usage().pending_registrations, 1); + let rejected = controllers[0] + .register(duplicate) + .err() + .context("duplicate")?; + assert!(matches!( + rejected.error, + CellGroupControllerError::Runtime(RuntimeError::AlreadyAttached) + )); + drop(rejected); + let rejected = controllers[0] + .register(excess.inputs.remove(0)) + .err() + .context("pending capacity")?; + assert!(matches!( + rejected.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + replicate( + &f, + &route, + &history(&route, "live-during-register")?, + rooted, + ) + .await?; + // Lost waiters cannot abandon the attachment or free its reserved slot. + drop(calls); + prepared + .store + .hold_group_head_reads + .store(false, Ordering::Release); + prepared.store.group_head_reads_resume.notify_waiters(); + wait_for(async || { + Ok(controllers.iter().all(|c| { + c.usage().groups == 2 + && c.usage().pending_registrations == 0 + && c.usage().retained_commands == 0 + })) + }) + .await?; + assert!(matches!( + controllers[0] + .register(*rejected.registration) + .err() + .context("active capacity")? + .error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + let group = prepared.birth.route().history; + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + let cell = history(prepared.birth.route(), "dynamic-enrollment")?; + let dispatch = dispatcher(&f, &prepared, class.clone())?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(group, cell.clone()) + .await + .is_ok()) + }) + .await?; + assert_eq!( + runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get(), + 1 + ); + } + for transport in &f.transports { + assert_eq!(transport.usage().peers, 2); + } + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn registration_refusals() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + wrong.applier = prepared.inputs[1].applier.clone(); + assert!(controllers[0].register(wrong).is_err()); + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + wrong.current = prepared.inputs[1].current.clone(); + assert!(controllers[0].register(wrong).is_err()); + let mut wrong = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + let pages = ManifestPageStore::open( + &f.cluster._root.path().join("foreign-page-owner"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = match f.cell_options(&a).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + wrong.applier = CellGroupApplier::new( + wrong.permit.group().clone(), + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + assert!(matches!( + controllers[0] + .register(wrong) + .err() + .context("foreign pages")? + .error, + CellGroupControllerError::Apply(_) + )); + // An already installed external route cannot be adopted or removed. + f.transports[0].install_physical_group(prepared.inputs[0].transport.clone())?; + let input = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + assert!( + matches!(controllers[0].register(input)?.response().await, Err(e) if matches!(e.as_ref(),CellGroupControllerError::Transport(TransportError::Identity))) + ); + f.transports[0].remove_physical_group(&prepared.birth.route().history)?; + // Bootstrap revalidation fails after real local genesis; healthy groups + // remain usable and the failed registration's slot/route is released. + let input = duplicate(&f, &prepared, &prepared.inputs[0], 0).await?; + prepared + .store + .fail_directory_reads + .store(true, Ordering::Release); + assert!(controllers[0].register(input)?.response().await.is_err()); + prepared + .store + .fail_directory_reads + .store(false, Ordering::Release); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().groups, 1); + assert_eq!(controllers[0].usage().pending_registrations, 0); + replicate( + &f, + &route, + &history(&route, "after-registration-failure")?, + rooted, + ) + .await?; + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + f.runtimes[0] + .campaign(prepared.birth.route().history) + .await?; + wait_for(async || { + Ok(f.runtimes[0] + .status(prepared.birth.route().history) + .await? + .ready) + }) + .await?; + let dispatch = dispatcher(&f, &prepared, class)?; + enroll( + &f, + &controllers[0], + &dispatch, + &history(prepared.birth.route(), "after-retry")?, + ) + .await?; + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn registration_close() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + prepared + .store + .hold_group_head_reads + .store(true, Ordering::Release); + let call = controllers[0].register(prepared.inputs.remove(0))?; + wait_for(async || { + Ok(prepared + .store + .active_group_head_reads + .load(Ordering::Acquire) + == 1) + }) + .await?; + drop(call); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + prepared + .store + .hold_group_head_reads + .store(false, Ordering::Release); + prepared.store.group_head_reads_resume.notify_waiters(); + tokio::time::timeout(Duration::from_secs(5), controllers[0].wait()).await??; + assert_eq!(controllers[0].usage().pending_registrations, 0); + let (replacement, events) = CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +async fn mutate( + f: &RuntimeCluster, + controller: &CellGroupController, + dispatch: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + retry: u64, +) -> Result<()> { + let invocation = dispatch + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([223; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"persisted".to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry, + }, + CellRoutingLimits { + timeout: Duration::from_secs(10), + max_attempts: 2, + }, + None, + ) + .await?; + let response = controller + .command(invocation, f.cell_options(cell).await?)? + .response() + .await?; + assert_eq!(response.receipt().receipt.result(), b"persisted"); + assert_eq!(response.receipt().version.sequence.get(), 2); + drop(response); + Ok(()) +} + +pub(crate) async fn registration_recovery() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut prepared = prepare(&f, 31, &a).await?; + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + let group = prepared.birth.route().history; + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + let cell = history(prepared.birth.route(), "recovered-registration")?; + let dispatch = dispatcher(&f, &prepared, class.clone())?; + enroll(&f, &controllers[0], &dispatch, &cell).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || { + for runtime in &f.runtimes { + if runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != 2 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + for c in &controllers { + c.close().await?; + } + drop(controllers); + f.appliers.clear(); + f.transports.clear(); + f.runtimes.clear(); + for n in 0..3 { + f.cluster.voters[n].log.close().await?; + let number = n as u8 + 1; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + f.cluster._root.path().join(format!("node-{number}/log")), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: f.cluster.voters[n].gate.clone(), + }), + )) + .await??; + f.cluster.voters[n].log = log; + } + + // Reopen only the engine-selected complete closure, then register with + // empty controllers; registration alone still grants no quorum authority. + for n in 0..3 { + let (runtime, events) = CellGroupRuntime::start( + f.cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + f.transports + .push(super::super::tcp::start(&runtime, n).await?); + f.runtimes.push(runtime); + f.events.push(events); + } + prepared.inputs = recover(&f, &prepared, &a).await?; + let mut controllers = vec![]; + for (n, events) in f.events.drain(..).enumerate() { + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + f.appliers.insert( + (n, group), + (pool.clone(), prepared.inputs[n].applier.clone()), + ); + let c = CellGroupController::start( + f.runtimes[n].clone(), + events, + f.transports[n].clone(), + pool, + vec![], + controller::limits(), + ) + .await?; + assert_eq!(c.usage().groups, 0); + controllers.push(c); + } + for (n, input) in prepared.inputs.drain(..).enumerate() { + controllers[n].register(input)?.response().await?; + } + assert!(matches!( + f.runtimes[0].execution_authority(group, cell.clone()).await, + Err(RuntimeError::NotLeader) + )); + f.runtimes[0].campaign(group).await?; + wait_for(async || Ok(f.runtimes[0].status(group).await?.ready)).await?; + mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5_000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + group, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let output = f.appliers[&(0, group)] + .0 + .query(f.cell_options(&cell).await?, authority, |db, version| { + assert_eq!(version.sequence.get(), 2); + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await?; + assert_eq!(output.result(), &8_i64.to_le_bytes()); + drop(output); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index def00f237..255105827 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -717,3 +717,7 @@ pub(super) use sql::{remote_command_lifecycle, remote_commands}; pub(super) use sql::{remote_enrollment, remote_enrollment_lifecycle}; pub(super) use sql::{receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending}; + +pub(super) use sql::{ + registration_close, registration_live, registration_recovery, registration_refusals, +}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 9d9158bc6..d3cce2830 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -570,3 +570,9 @@ mod receipt_lookup; pub(crate) use receipt_lookup::{ receipt_lookup_cancelled, receipt_lookup_ingress, receipt_lookup_pending, }; + +#[path = "cell_group_registration.rs"] +mod registration; +pub(crate) use registration::{ + registration_close, registration_live, registration_recovery, registration_refusals, +}; diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs index 565265d74..d7d2a57b3 100644 --- a/crates/cellar/tests/support/cell_group_tcp.rs +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -21,7 +21,10 @@ pub(super) fn tls() -> Result { )?) } -async fn start(runtime: &CellGroupRuntime, n: usize) -> Result { +pub(super) async fn start( + runtime: &CellGroupRuntime, + n: usize, +) -> Result { RaftTransport::start_shared( runtime.clone(), Arc::new( diff --git a/crates/cellar/tests/support/raft_catalog_fault.rs b/crates/cellar/tests/support/raft_catalog_fault.rs index 32c352860..1ab2869dd 100644 --- a/crates/cellar/tests/support/raft_catalog_fault.rs +++ b/crates/cellar/tests/support/raft_catalog_fault.rs @@ -180,6 +180,9 @@ pub struct Store { pub view_waiting: AtomicBool, pub view_resume: tokio::sync::Notify, pub fail_directory_reads: AtomicBool, + pub hold_group_head_reads: AtomicBool, + pub active_group_head_reads: AtomicUsize, + pub group_head_reads_resume: tokio::sync::Notify, pub hold_view_reads: AtomicBool, pub view_reads: AtomicUsize, pub active_view_reads: AtomicUsize, @@ -582,6 +585,26 @@ impl ObjectStore for Store { { return Err(failure()); } + if path.filename() == Some("head") { + struct Reading<'a>(&'a AtomicUsize); + + impl Drop for Reading<'_> { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::AcqRel); + } + } + self.active_group_head_reads.fetch_add(1, Ordering::AcqRel); + let _reading = Reading(&self.active_group_head_reads); + loop { + let resumed = self.group_head_reads_resume.notified(); + tokio::pin!(resumed); + resumed.as_mut().enable(); + if !self.hold_group_head_reads.load(Ordering::Acquire) { + break; + } + resumed.await; + } + } if path.filename() == Some("view") { self.view_reads.fetch_add(1, Ordering::AcqRel); From d847a7bc84d9b46556370e8fb9704bdaa7073727 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 11:33:08 -0700 Subject: [PATCH 111/568] docs: record qualified dynamic group registration and next CG gaps --- docs/design/multi-raft/cell-groups/README.md | 7 +++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 57 ++++++++++++------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 52 +++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 18 ++++++ docs/design/multi-raft/cell-groups/tasks.json | 53 ++++++++++++++++- 7 files changed, 170 insertions(+), 23 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 843374cdb..62e485bc6 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -236,6 +236,13 @@ returns signed observations and keeps absence ambiguous. The partial gate passes 4 executables). Dynamic group lifecycle/lazy birth and per-group failure isolation are next; CG-04 remains IN_PROGRESS and public cutover is still pending. +At `4326421a`, controllers admit bounded dynamic registrations and may start empty. +Catalog/root/page ownership, fresh quorum, response-independent attachment and +joined shutdown remain enforced. The partial gate passes 231 Rust cases and twelve +commands; transfer `.cellar/cell-groups-evidence/CG-04-dynamic-registration/` +(281 source inputs, 4 executables). Per-group failure isolation and +request-driven lazy birth are next; CG-04 and public cutover remain incomplete. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index f3560faf1..f053ec09b 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "fbc15053f8dcfd3b46078abad3cbbae7b68da8d6", + "source_revision": "4326421a94c4f9a257bf6ceb8e0f7e518eacf3de", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 1e99e92ba..30effd3d7 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,26 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `fbc15053`, adding standalone local and +The newest reviewed source baseline is `4326421a`, adding bounded dynamic group +registration and empty-controller startup. Exact engine/root/applier page ownership +and exclusive transport registration precede existing bootstrap revalidation. +The applier is registered before attachment can emit events; active membership +still requires a successful attachment and serving requires current-term quorum. +Pending groups share capacity with active groups, and queued/unconsumed work uses +the existing item/byte budget. Lost waiters cannot abandon attachment; shutdown +joins it before closing physical services. Failed slots release before response. +Four native cases cover both image modes, live TLS progress during a catalog +stall, ownership/capacity refusals, failed registration retry, cancelled close, +and all-voter joined engine/page-store restart into empty controllers with exact +receipt replay. Image-store handles remain live in one process. +The record `.cellar/cell-groups-evidence/CG-04-dynamic-registration/checkpoint.json` +binds 231 Rust cases, twelve commands, 281 source inputs and 4 executables. +Transfer this ignored evidence directory with previous checkpoints. CG-04 remains +IN_PROGRESS. Next isolate group-local failures without closing healthy groups +(shared engine sync failures still quarantine the engine), then wire +request-driven provisioning/lazy birth into registration. Configuration lifecycle, +public SDK cutover and shared RustFS/load remain unfinished. +The preceding reviewed source baseline is `fbc15053`, adding standalone local and remote receipt inspection with distinct authenticated purpose and explicit command permission. Registered contract/digest/identity validation precedes immutable SQL on a fresh catalog-bound quorum image. Exact retained results and ambiguous @@ -508,24 +527,24 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Deliver bounded dynamic controller registration and lazy group birth. The - current controller has a fixed roster, although command, enrollment, query and - receipt dispatch now reuse its shared owners. Define one registration lifecycle - spanning runtime, transport and applier; revalidate catalog admission before - serving and retain physical ownership through cancellation or close. Test - duplicate/foreign registration, capacity, attach/close races, failed birth and - selected-root restart while another group continues serving. Pair this with - per-group failure isolation: quarantine a failed group without closing healthy - groups or discarding retained SQL/apply work. These are the critical gaps for - continuing CG-04, before exposing the public SDK replacement. - Keep explicit write grants independent from read grants. Standalone lookup now - preserves ambiguous absence and never replays or retires unresolved callbacks. - Destination routing still requires fresh catalog admission: removing that read - depends on replicated transfer seals and serving-admission invalidation in - CG-09. A TTL or cached hint cannot substitute for that authority. - Connect public `RaftCellHandle` and automatic discovery only through the actual - shared host. Keep the old host's initializer refusal until replacement. - Request and namespace S3 minima must refuse until CG-06 provides archive proofs. +1. Deliver per-group failure isolation. Dynamic registration now adds groups + without rebuilding the controller, but a group-local fatal apply/command/event + still closes the node. Add an explicit quarantined group lifecycle that revokes + serving, keeps ambiguous producers and physical apply ownership, and permits + unrelated groups to progress. Qualify group-local apply failure and unresolved command + ownership while another group serves; do not discard work to make it continue. + Separately verify that a shared log-engine sync failure still quarantines all + groups on that engine, as required by design.md. + Then connect request-driven group provisioning/lazy birth to register(). The + caller currently prepares the exact bootstrap permit and selected root; a host + must bound catalog selection, physical preparation and concurrent birth before + transferring them. Reuse the existing admission and cancellation owners. + Keep read and command grants independent. Receipt absence remains ambiguous. + Destination catalog caching still depends on replicated transfer seals and + serving-admission invalidation in CG-09. Do not substitute a TTL for authority. + Connect public RaftCellHandle/discovery only through the actual replacement + host, preserving the old initializer refusal until cutover. Quorum/S3 minima + must refuse unsupported archive guarantees until CG-06 supplies real proofs. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index af0dfeed8..e1662ef1a 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 72 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 76 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index d000d3a09..0d444b5c6 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 dynamic controller registration + +Qualified source: `4326421a94c4f9a257bf6ceb8e0f7e518eacf3de`. + +`CellGroupController::register` now adds a group without replacing the controller +or creating per-cell workers. Controllers may start empty. Pending and active +groups share a capacity bound; queued attachment and unconsumed replies retain +the existing item/byte budget. Command, enrollment and receipt dispatch consult +the live roster under short locks rather than copying it for each invocation. + +Admission binds the bootstrap permit, selected root, applier and physical route +to the same engine/history/schema. The applier must own the exact manifest page +store that verified the root. Its event registration precedes polling attachment, +so early consensus and apply events remain owned. Runtime attachment revalidates +the catalog permit before active membership is published. Registration alone +never grants application authority; current-term quorum remains required. + +The transport registration refuses adoption of an existing route. Its cleanup +compares exact route ownership so it cannot erase a replacement. A disconnected +waiter cannot abandon attachment or release its group slot. Definite failures +release group capacity before responding. Shutdown joins entered catalog/runtime +attachment before closing physical dependencies. Unexpected competing runtime +owners or runtime-wide closure keep the existing conservative node shutdown. + +Four new native cases run both image modes over real engines and TLS. Held +catalog reads leave an already active group serving. Duplicate and pending/active +capacity checks, foreign engines/roots/page owners, failed catalog retry and +cancelled close exercise ownership boundaries. Recovery joins all controllers, +closes and reopens all voter engines under a new process generation, reopens page +stores and registers selected roots into empty controllers. Fresh quorum returns +the prior exact receipt without another SQL mutation. Image-store handles remain +in the same process; this does not qualify independent-process or power loss. +The first diagnostic run exposed an obsolete catalog stall key and an attempted +page-store reopen while the engines correctly retained page pins. The corrected +fixtures preserve those safeguards and are retained with the passing evidence. + +All 231 selected Rust cases and twelve commands pass, including all-target Clippy, +formatting, layout, dependency isolation, docs, source preflight and 37 validator +cases. `.cellar/cell-groups-evidence/CG-04-dynamic-registration/checkpoint.json` +binds 281 source inputs and 4 copied executables. Transfer this ignored +local directory with prior checkpoints; it is not included in the PR. + +CG-04 remains IN_PROGRESS. The critical next gap is per-group failure isolation: +a group-local apply/command failure currently closes the node controller. +Quarantine that group, revoke serving authority and retain its physical owners +through cleanup, while proving that a healthy group continues. Shared log-engine +sync failure must still quarantine every group on the affected engine. +Request-driven provisioning/lazy birth must still connect catalog selection and +bounded local preparation to this registration API. Configuration lifecycle and public SDK discovery/cutover remain +open, followed by the CG-05–16 dependencies. Shared-group RustFS recovery and load +remain unqualified; no throughput or latency gain is claimed. + ## CG-04 standalone receipt inspection Qualified source: `fbc15053f8dcfd3b46078abad3cbbae7b68da8d6`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index c0aecdcc5..5de19cd15 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -179,6 +179,24 @@ and sequence-gap failures. Only a sequence-1 image can lack both receipt tables; partial schema or later schema loss refuses as corruption. Missing and an empty committed result have distinct wire encodings and API results. +The controller accepts bounded dynamic group registrations, including startup +with an empty roster. A registration supplies the exact bootstrap permit, verified +selected root, applier owning the same manifest page store, and physical peer +route. Pending and active groups share the group limit; pending work and unconsumed +replies also retain the existing item/byte budget. A pre-existing physical route +cannot be silently adopted. Cleanup removes only the exact registered route and +cannot erase a replacement view. + +The worker installs the applier before polling runtime attachment so early +consensus events cannot escape ownership. Attachment revalidates catalog claims; +only successful completion publishes application roster membership, and serving +still requires fresh current-term quorum. Dropping a registration waiter cannot +cancel this work. Definite failures release their slot before responding. Shutdown +joins entered catalog/attachment futures before closing transport, SQL and the +runtime. A competing external runtime owner or runtime-wide failure still closes +the controller conservatively. Request-driven provisioning/lazy birth, per-group +failure isolation and configuration lifecycle remain separate unfinished work. + The current lane serializes RPC exchanges per peer; this is not load qualification. ## Apply, checkpoint and archive diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index d05c5f080..2870b32d7 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -524,7 +524,9 @@ "crates/cellar/tests/support/cell_group_remote_enrollment.rs", "crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs", "crates/cellar/durability/raft/receipt_cell_wire.rs", - "crates/cellar/tests/support/cell_group_receipt_lookup.rs" + "crates/cellar/tests/support/cell_group_receipt_lookup.rs", + "crates/cellar/durability/raft/cell_groups/controller/registration.rs", + "crates/cellar/tests/support/cell_group_registration.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -694,6 +696,15 @@ "shared_receipt_lookup_keeps_unresolved_producers_at_capacity", "shared_receipt_lookup_cancellation_preserves_active_writer" ] + }, + { + "action": "Replace the fixed controller roster with bounded dynamic registration, including empty-controller startup. Reserve pending plus active group slots and shared command item/byte admission before transfer. Bind exact engine/history/schema, selected root and applier page owner; install an exclusively owned physical route without adopting a prior registration. Register the applier before polling attachment so early consensus/apply events remain owned. Revalidate the bootstrap catalog through the existing runtime attach path; expose application roster membership only after success, with fresh quorum still required. Retain attachment through response cancellation and join catalog/runtime attachment before closing physical dependencies. Release definite failures before responding; unexpected competing runtime owners or runtime closure keep the conservative node shutdown boundary. Configuration changes, per-group failure isolation and request-driven lazy birth/public SDK binding remain incomplete. Run the full transport and shared runtime suites.", + "tests": [ + "dynamic_registration_preserves_live_groups_and_disconnected_waiters", + "dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog", + "dynamic_registration_joins_attachment_before_cancelled_shutdown", + "dynamic_registration_recovers_selected_roots_into_an_empty_controller" + ] } ], "test_cases": [ @@ -1385,6 +1396,46 @@ "I05", "I12" ] + }, + { + "name": "dynamic_registration_preserves_live_groups_and_disconnected_waiters", + "oracle": "Three real voter controllers register a second group while its catalog reads are held; the first group continues replicated SQL apply. Pending and active slots enforce the same capacity, duplicate attachment refuses, and disconnected registration waiters cannot abandon ownership. After revalidation the new group establishes quorum and runs registered enrollment and exact creation retry through the shared SQL workers and TLS pools, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog", + "oracle": "Reject foreign engines, roots and a different page owner on the same engine. Refuse adoption of an existing transport route. A real bootstrap catalog-read failure releases only the failed registration slot/route; an already serving group continues. Retry the same prepared history successfully and enroll after real quorum, in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_joins_attachment_before_cancelled_shutdown", + "oracle": "Hold real catalog revalidation after registration admission, drop the response and cancel the close waiter. The controller retains the pending slot and runtime engine claim until attachment joins. After release, close finishes and a replacement runtime may start; run both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "dynamic_registration_recovers_selected_roots_into_an_empty_controller", + "oracle": "After registering and mutating a group, join all controllers, close and reopen all engines from the existing voter disks under a new process generation, reopen manifest page owners and recover selected roots into fresh empty controllers. Attachment alone refuses leader authority; fresh quorum then returns the prior exact receipt without another SQL mutation. Verify the persisted value and cell sequence in both image modes. Image-store handles remain in the same process. This is joined engine/runtime/page-store restart, not independent-process or power-loss qualification.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 11:52:00 -0700 Subject: [PATCH 112/568] feat: quarantine failed cell groups without stopping healthy groups --- .../durability/raft/cell_groups/README.md | 18 +- .../raft/cell_groups/controller/failure.rs | 49 +++ .../raft/cell_groups/controller/mod.rs | 21 +- .../cell_groups/controller/registration.rs | 82 ++++- .../raft/cell_groups/controller/requests.rs | 7 + .../raft/cell_groups/controller/worker.rs | 81 ++++- .../cell_groups/dispatch/command/owned.rs | 4 + .../raft/cell_groups/dispatch/enrollment.rs | 4 + .../raft/cell_groups/runtime/mod.rs | 13 +- .../raft/cell_groups/runtime/worker.rs | 17 + crates/cellar/durability/raft/completion.rs | 2 + .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_controller.rs | 60 ++-- .../support/cell_group_controller_commands.rs | 2 +- .../tests/support/cell_group_quarantine.rs | 320 ++++++++++++++++++ .../tests/support/cell_group_registration.rs | 14 +- .../tests/support/cell_group_runtime.rs | 5 + crates/cellar/tests/support/cell_group_sql.rs | 7 + 18 files changed, 656 insertions(+), 60 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/failure.rs create mode 100644 crates/cellar/tests/support/cell_group_quarantine.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 97153a5ae..1d0684b1d 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -333,7 +333,7 @@ transport ownership. The public host cutover remains unfinished. `CellGroupController::register` adds a prepared group to a running node, including one whose initial roster is empty. The caller supplies a bootstrap permit, verified selected root, the applier owning that root's manifest page store, and -an authenticated physical peer route. Pending and active groups share the group +an authenticated physical peer route. Pending, active and quarantined groups share the group limit; attachment and unconsumed replies also use the shared item/byte budget. Admission failure returns the prepared owners. After admission, dropping the response does not cancel attachment. @@ -345,7 +345,15 @@ route registration refuses adopting an existing route; cleanup checks exact rout identity. Failed attachment releases its group slot before publishing a refusal. Shutdown joins attachment before closing transport, SQL and consensus ownership. -A fatal group apply/command currently closes the controller. Per-group failure -isolation, request-driven group provisioning, configuration lifecycle and public -SDK integration remain unfinished. See the delivery package for qualification -status; dynamic registration does not establish those later capabilities. +Group-local apply/runtime failure or an unresolved command failure quarantines +its group. `group_failure` preserves the original cause and `usage` reports active +and quarantined groups separately. A reserved runtime control request revokes +leader authority even while apply is waiting. New group work refuses; pending +apply and ambiguous command owners remain charged through physical shutdown. +Quarantine cannot be cleared by registration. Recovery requires joined restart; +a late persistence completion cannot restore the revoked serving fence. + +Shared log-engine I/O/corruption/closure and lost shared SQL worker replies still +close the node. Unknown events and controller ownership violations fail closed. +Request-driven group provisioning, configuration lifecycle and public SDK +integration remain unfinished. See the delivery package for qualification status. diff --git a/crates/cellar/durability/raft/cell_groups/controller/failure.rs b/crates/cellar/durability/raft/cell_groups/controller/failure.rs new file mode 100644 index 000000000..88dc09aa9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/failure.rs @@ -0,0 +1,49 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Group isolation never converts shared-engine uncertainty into availability. +use super::*; +use crate::raft::{cell_groups::CellDispatchError, storage::LogError}; + +fn runtime(error: &RuntimeError) -> bool { + matches!(error, RuntimeError::Closed) + || matches!(error, RuntimeError::Storage(error) if storage(error)) +} + +fn storage(error: &LogError) -> bool { + matches!( + error, + LogError::Io(_) | LogError::Corrupt(_) | LogError::LostStableState | LogError::Closed + ) +} + +fn execution(error: &ExecutionError) -> bool { + match error { + // A lost shared SQL worker reply cannot prove that the pool remains + // usable. Preserve its existing joined node shutdown boundary. + ExecutionError::Refused(cellar_logic::consensus::CommandFailure::Unavailable) => true, + ExecutionError::Storage(error) => { + error.downcast_ref::().is_some_and(storage) + || error.downcast_ref::().is_some_and(runtime) + } + _ => false, + } +} + +fn dispatch(error: &CellDispatchError) -> bool { + match error { + CellDispatchError::Runtime(error) => runtime(error), + CellDispatchError::Execution(error) => execution(error), + _ => false, + } +} + +pub(super) fn node_wide(error: &CellGroupControllerError) -> bool { + match error { + CellGroupControllerError::Runtime(error) => runtime(error), + CellGroupControllerError::Apply(error) => execution(error), + CellGroupControllerError::Command(error) => dispatch(&error.cause), + CellGroupControllerError::Enrollment(error) => dispatch(&error.cause), + CellGroupControllerError::Closed | CellGroupControllerError::Transport(_) => true, + CellGroupControllerError::Invalid(_) => false, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 753225a45..0ead568b3 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -6,6 +6,7 @@ mod calls; mod commands; mod enrollments; +mod failure; mod registration; mod requests; pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; @@ -73,6 +74,8 @@ impl From for CellGroupControllerError { pub struct CellGroupControllerUsage { pub groups: usize, pub pending_registrations: usize, + /// Retained failed groups still consume the registration capacity. + pub quarantined_groups: usize, pub active_apply: usize, pub pending_apply: usize, pub applied: u64, @@ -254,12 +257,22 @@ impl CellGroupController { Ok(handle) } + /// Original failure retained for a quarantined group. Recovery requires a + /// joined restart; registration cannot replace the failed owner in place. + pub fn group_failure( + &self, + group: cellar_logic::cell_group::GroupHistory, + ) -> Result, RuntimeError> { + self.handle.0.groups.failure(&group) + } + pub fn usage(&self) -> CellGroupControllerUsage { let shared = &self.handle.0; let (command_bytes, retained_commands) = shared.commands.usage(); CellGroupControllerUsage { groups: shared.groups.len(), pending_registrations: shared.groups.pending(), + quarantined_groups: shared.groups.quarantined(), command_bytes, retained_commands, active_apply: shared.active_apply.load(Ordering::Acquire), @@ -294,9 +307,13 @@ impl CellGroupController { return Err(super::CellDispatchError::Runtime(RuntimeError::Closed)); } invocation.begin().await?; - if !invocation + let attempt = invocation .attempt() - .is_some_and(|a| shared.groups.contains(&a.route().history).unwrap_or(false)) + .ok_or(super::CellDispatchError::Runtime(RuntimeError::NotFound))?; + if !shared + .groups + .contains(&attempt.route().history) + .map_err(super::CellDispatchError::Runtime)? { return Err(super::CellDispatchError::Runtime(RuntimeError::NotFound)); } diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs index b306178f3..3809a4a1c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/registration.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -9,39 +9,98 @@ use crate::raft::transport::{GroupRoute, PhysicalRegistration}; use cellar_logic::cell_group::GroupHistory; use tokio::sync::{mpsc, oneshot}; +enum GroupState { + Pending, + Active, + Quarantined(CellGroupControllerError), +} + pub(crate) struct GroupRoster { - groups: Mutex>, + groups: Mutex>, limit: usize, } impl GroupRoster { pub(super) fn new(groups: impl Iterator, limit: usize) -> Arc { Arc::new(Self { - groups: Mutex::new(groups.map(|g| (g, true)).collect()), + groups: Mutex::new(groups.map(|g| (g, GroupState::Active)).collect()), limit, }) } pub(crate) fn contains(&self, group: &GroupHistory) -> Result { - Ok(self + match self .groups .lock() .map_err(|_| RuntimeError::Closed)? .get(group) - == Some(&true)) + { + Some(GroupState::Active) => Ok(true), + Some(GroupState::Quarantined(_)) => Err(RuntimeError::GroupQuarantined), + _ => Ok(false), + } + } + + pub(super) fn quarantine( + &self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), RuntimeError> { + let mut groups = self.groups.lock().map_err(|_| RuntimeError::Closed)?; + let state = groups.get_mut(&group).ok_or(RuntimeError::NotFound)?; + if !matches!(state, GroupState::Quarantined(_)) { + *state = GroupState::Quarantined(error); + } + Ok(()) + } + + pub(super) fn failure( + &self, + group: &GroupHistory, + ) -> Result, RuntimeError> { + Ok( + match self + .groups + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(group) + { + Some(GroupState::Quarantined(error)) => Some(error.clone()), + _ => None, + }, + ) } pub(super) fn len(&self) -> usize { self.groups .lock() - .map(|g| g.values().filter(|active| **active).count()) + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Active)) + .count() + }) .unwrap_or(0) } pub(super) fn pending(&self) -> usize { self.groups .lock() - .map(|g| g.values().filter(|active| !**active).count()) + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Pending)) + .count() + }) + .unwrap_or(0) + } + + pub(super) fn quarantined(&self) -> usize { + self.groups + .lock() + .map(|g| { + g.values() + .filter(|s| matches!(s, GroupState::Quarantined(_))) + .count() + }) .unwrap_or(0) } @@ -53,7 +112,7 @@ impl GroupRoster { if groups.len() >= self.limit { return Err(RuntimeError::Overloaded); } - groups.insert(group, false); + groups.insert(group, GroupState::Pending); Ok(Reservation { roster: self.clone(), group, @@ -76,7 +135,10 @@ impl Reservation { .lock() .map_err(|_| RuntimeError::Closed)?; let entry = groups.get_mut(&self.group).ok_or(RuntimeError::NotFound)?; - *entry = true; + if !matches!(entry, GroupState::Pending) { + return Err(RuntimeError::GroupQuarantined); + } + *entry = GroupState::Active; self.active = true; Ok(()) } @@ -86,7 +148,9 @@ impl Drop for Reservation { fn drop(&mut self) { if !self.active { if let Ok(mut groups) = self.roster.groups.lock() { - groups.remove(&self.group); + if matches!(groups.get(&self.group), Some(GroupState::Pending)) { + groups.remove(&self.group); + } } } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index d988701c8..b72341532 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -15,6 +15,13 @@ pub(super) enum Completed { } impl Completed { + pub fn group(&self) -> Option { + match self { + Self::Command(done) => done.invocation.unresolved_group(), + Self::Enrollment(done) => done.invocation.unresolved_group(), + } + } + pub fn failure(&self) -> Option { match self { Self::Command(done) if done.invocation.unresolved() => Some(done.error.clone().map_or( diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 150cce2b8..6a0216aa9 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -6,6 +6,7 @@ use cellar_logic::{cell_group::GroupHistory, consensus::CommandFailure}; use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; use std::collections::{BTreeSet, VecDeque}; +#[derive(Clone)] enum Work { Prepare(Box), Publish(Box), @@ -17,7 +18,11 @@ struct Pending { retry_at: u64, } -type Completed = (GroupHistory, Result, CellGroupControllerError>); +type Completed = ( + GroupHistory, + Result, CellGroupControllerError>, + Work, +); pub(super) struct Worker { runtime: CellGroupRuntime, @@ -33,7 +38,9 @@ pub(super) struct Worker { hints: FuturesUnordered>, requests: tokio::sync::mpsc::Receiver, commands: FuturesUnordered>, - failed_command: Option, + failed_commands: Vec, + quarantined: BTreeSet, + retained_apply: Vec, registrations: FuturesUnordered>, routes: BTreeMap, progress: tokio::sync::watch::Sender<()>, @@ -63,7 +70,9 @@ impl Worker { hints: FuturesUnordered::new(), requests, commands: FuturesUnordered::new(), - failed_command: None, + failed_commands: Vec::new(), + quarantined: BTreeSet::new(), + retained_apply: Vec::new(), registrations: FuturesUnordered::new(), routes: BTreeMap::new(), progress: tokio::sync::watch::channel(()).0, @@ -80,6 +89,10 @@ impl Worker { let Some(job) = self.pending.pop_front() else { break; }; + if self.quarantined.contains(&job.group) { + self.retained_apply.push(job.work); + continue; + } if job.retry_at > now || self.active.contains(&job.group) { self.pending.push_back(job); continue; @@ -89,8 +102,9 @@ impl Worker { let applier = self.roster[&job.group].clone(); let runtime = self.runtime.clone(); self.jobs.push(Box::pin(async move { + let retained = job.work.clone(); let result = apply(sql, applier, runtime, job.work).await; - (job.group, result) + (job.group, result, retained) })); } self.shared @@ -101,6 +115,25 @@ impl Worker { .store(self.pending.len(), Ordering::Release); } + async fn quarantine( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), CellGroupControllerError> { + if failure::node_wide(&error) { + return Err(error); + } + if self.quarantined.insert(group) { + // Stop application admission first. Runtime revocation then bypasses + // any waiting apply, fences captured leader authority and refuses + // new direct/transport work without dropping the RawNode claim. + self.shared.groups.quarantine(group, error)?; + self.runtime.quarantine(group).await?; + self.progress.send_replace(()); + } + Ok(()) + } + pub(super) async fn run( &mut self, domain: &crate::asyncrt::DomainToken, @@ -147,7 +180,9 @@ impl Worker { Ok(Some(guard)) => { self.routes.insert(group, guard); } Ok(None) => unreachable!("active registration completion"), Err(error) => { - self.roster.remove(&group); + if !self.quarantined.contains(&group) { + self.roster.remove(&group); + } if uncertain { return Err(error); } } } @@ -156,15 +191,24 @@ impl Worker { if let Some(error) = completed.failure() { // Retain the invocation and its permit through physical // shutdown even after delivering the error to its caller. - self.failed_command = Some(completed); - return Err(error); + let group = completed.group().ok_or(CellGroupControllerError::Invalid("unresolved command without group"))?; + self.failed_commands.push(completed); + self.quarantine(group, error).await?; } } - Some((group, result)) = self.jobs.next(), if !self.jobs.is_empty() => { + Some((group, result, retained)) = self.jobs.next(), if !self.jobs.is_empty() => { self.active.remove(&group); - if let Some(work) = result? { - self.shared.apply_retries.fetch_add(1, Ordering::Relaxed); - self.pending.push_back(Pending { group, work, retry_at: now.saturating_add(self.limits.retry_interval_ms) }); + match result { + Err(error) => { + self.retained_apply.push(retained); + self.quarantine(group, error).await?; + } + Ok(Some(work)) if self.quarantined.contains(&group) => self.retained_apply.push(work), + Ok(Some(work)) => { + self.shared.apply_retries.fetch_add(1, Ordering::Relaxed); + self.pending.push_back(Pending { group, work, retry_at: now.saturating_add(self.limits.retry_interval_ms) }); + } + Ok(None) => (), } } Some(()) = self.hints.next(), if !self.hints.is_empty() => (), @@ -174,7 +218,20 @@ impl Worker { if !self.roster.contains_key(&group) { return Err(CellGroupControllerError::Invalid("unregistered group event")); } - match event.effect().map_err(Clone::clone)? { + if self.quarantined.contains(&group) { + if let Some(work) = event.retain_apply() { + self.retained_apply.push(Work::Prepare(Box::new(work))); + } + continue; + } + let effect = match event.effect() { + Ok(effect) => effect, + Err(error) => { + self.quarantine(group, error.clone().into()).await?; + continue; + } + }; + match effect { CellGroupEffect::Message(message) => { if let Err(error) = self.transport.submit_physical(&group, message.clone()) { if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer) { diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs index cb1ee1db7..047b03b38 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs @@ -32,6 +32,10 @@ impl CellCommandInvocation { Ok((self.request.id, deadline, bytes)) } + pub(crate) fn unresolved_group(&self) -> Option { + self.origin.as_ref().map(|origin| origin.route.history) + } + pub(crate) fn unresolved(&self) -> bool { self.origin.is_some() } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index e0425c1bc..6be3134da 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -194,6 +194,10 @@ impl CellEnrollmentInvocation { Ok((self.creation, deadline)) } + pub(crate) fn unresolved_group(&self) -> Option { + self.origin.as_ref().map(|origin| origin.route.history) + } + pub(crate) fn unresolved(&self) -> bool { self.origin.is_some() } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 9489085e6..83750d190 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -91,6 +91,7 @@ struct Handle { enum Command { Attach(Box), + Quarantine, Campaign, Execution(cellar_logic::cell_group::CellHistory), Enrollment(cellar_logic::cell_group::CellHistory), @@ -273,7 +274,7 @@ impl CellGroupRuntime { bytes: usize, ) -> Result { self.admit()?; - let completion = matches!(command, Command::Apply(..)); + let completion = matches!(command, Command::Apply(..) | Command::Quarantine); let budget = if completion { &self.inner.completions } else { @@ -329,6 +330,16 @@ impl CellGroupRuntime { result } + /// Revoke this group's authority without retiring its physical owners. This + /// control request bypasses the waiting apply/ordinary queues and shares the + /// reserved completion lane. Only joined node shutdown releases the claim. + pub(crate) async fn quarantine(&self, group: GroupHistory) -> Result<(), RuntimeError> { + match self.call(Some(group), Command::Quarantine, 4096).await? { + Response::Unit => Ok(()), + _ => Err(RuntimeError::Protocol("quarantine reply".into())), + } + } + pub async fn attach( &self, permit: CellGroupBootstrapPermit, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 17cb888f5..ecbc4044e 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -211,6 +211,15 @@ impl Worker { reply(request, Err(RuntimeError::NotFound)); return; }; + if matches!(request.command, Command::Quarantine) { + if group.failed.is_none() { + group.fail(RuntimeError::GroupQuarantined); + group.wake(history, &mut self.order); + } + reply(request, Ok(Response::Unit)); + self.refresh(history); + return; + } if let Some(error) = &group.failed { reply(request, Err(error.clone())); return; @@ -558,6 +567,11 @@ impl Worker { .groups .get_mut(&job.history) .ok_or(RuntimeError::NotFound)?; + // The physical batch must join even after quarantine. Its + // successful sync cannot restore the revoked execution fence. + if group.failed.is_some() { + continue; + } group.owner.complete_persist(CellGroupPersisted { ticket: job.work.ticket, sequence: receipt.sequence, @@ -577,6 +591,9 @@ impl Worker { .groups .get_mut(&job.history) .ok_or(RuntimeError::NotFound)?; + if group.failed.is_some() { + continue; + } group.write = Some(job.work); group.write_bytes = job.bytes; self.writes.push_front(job.history); diff --git a/crates/cellar/durability/raft/completion.rs b/crates/cellar/durability/raft/completion.rs index 1a62ceb00..9f9793de8 100644 --- a/crates/cellar/durability/raft/completion.rs +++ b/crates/cellar/durability/raft/completion.rs @@ -26,6 +26,8 @@ pub enum RuntimeError { AlreadyAttached, Overloaded, StaleCompletion, + /// This process retains the failed group; serving requires recovery on restart. + GroupQuarantined, Closed, } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 6798e0ce9..f67961eaf 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1565,3 +1565,13 @@ async fn dynamic_registration_joins_attachment_before_cancelled_shutdown() -> Re async fn dynamic_registration_recovers_selected_roots_into_an_empty_controller() -> Result<()> { runtime_fixture::registration_recovery().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_quarantine_retains_producer_and_serves_healthy_group() -> Result<()> { + runtime_fixture::group_quarantine_retains_producer_and_serves_healthy_group().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_engine_sync_failure_still_stops_all_groups() -> Result<()> { + runtime_fixture::shared_engine_sync_failure_still_stops_all_groups().await +} diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs index a870efdac..951c2de7c 100644 --- a/crates/cellar/tests/support/cell_group_controller.rs +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -75,6 +75,33 @@ async fn applied(fixture: &RuntimeCluster, group: GroupHistory, index: LogIndex) .await } +pub(super) async fn add_second( + fixture: &mut RuntimeCluster, + primary: GroupHistory, +) -> Result { + let second = fixture.add_group(31).await?; + for node in 0..3 { + let pool = fixture.appliers[&(node, primary)].0.clone(); + let voter = &mut fixture.extra.get_mut(&second).context("extra")?[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&second)?, + voter.pages.take().context("pages")?, + CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + fixture.appliers.insert((node, second), (pool, applier)); + } + Ok(second) +} + pub(crate) async fn controller_replication(rooted: bool) -> Result<()> { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -99,26 +126,7 @@ pub(crate) async fn controller_replication(rooted: bool) -> Result<()> { }, )?); let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; - let second = fixture.add_group(31).await?; - for node in 0..3 { - let pool = fixture.appliers[&(node, route.history)].0.clone(); - let voter = &mut fixture.extra.get_mut(&second).context("extra")?[node]; - let applier = CellGroupApplier::new( - voter.log.cell_group(&second)?, - voter.pages.take().context("pages")?, - CellGroupApplyImages::Materialized { - images: voter.images.clone(), - services: cellar_ltx::generation::ReadServices { - now_unix_millis: Arc::new(|| 10), - fill_random: Arc::new(|bytes| bytes.fill(17)), - }, - }, - ManifestLimits::default(), - GroupEntryLimits::default(), - 100, - )?; - fixture.appliers.insert((node, second), (pool, applier)); - } + let second = add_second(&mut fixture, route.history).await?; let controllers = start(&mut fixture).await?; fixture.runtimes[0].campaign(second).await?; wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; @@ -349,11 +357,17 @@ pub(crate) async fn controller_apply_faults(rooted: bool) -> Result<()> { ) .await?; if mode == 0 { - let result = - tokio::time::timeout(Duration::from_secs(10), controllers[0].wait()).await?; + wait_for(async || Ok(controllers[0].group_failure(route.history)?.is_some())).await?; assert!( - matches!(result, Err(CellGroupControllerError::Apply(ref error)) if matches!(error.as_ref(), ExecutionError::Storage(_))) + matches!(controllers[0].group_failure(route.history)?, Some(CellGroupControllerError::Apply(ref error)) if matches!(error.as_ref(), ExecutionError::Storage(_))) ); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().quarantined_groups, 1); + assert!(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::GroupQuarantined) + )); + controllers[0].close().await?; assert_eq!( faults.mode.load(Ordering::Acquire), 0, diff --git a/crates/cellar/tests/support/cell_group_controller_commands.rs b/crates/cellar/tests/support/cell_group_controller_commands.rs index ca86dcdfc..c5cb0b52d 100644 --- a/crates/cellar/tests/support/cell_group_controller_commands.rs +++ b/crates/cellar/tests/support/cell_group_controller_commands.rs @@ -18,7 +18,7 @@ async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<() .await? } -async fn invocation( +pub(super) async fn invocation( dispatcher: &CellGroupDispatcher, class: &RaftClass, cell: &CellHistory, diff --git a/crates/cellar/tests/support/cell_group_quarantine.rs b/crates/cellar/tests/support/cell_group_quarantine.rs new file mode 100644 index 000000000..3256dcdb8 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_quarantine.rs @@ -0,0 +1,320 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Faults cross real manifest/engine barriers; healthy groups share the same node. +use super::apply_faults::physical; +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn group_quarantine_retains_producer_and_serves_healthy_group() -> Result<()> { + for rooted in [false, true] { + let root = tempfile::tempdir()?; + let faults = physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], move |ctx, input| { + count.fetch_add(1, Ordering::SeqCst); + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + ); + let (mut f, route, a, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + // One retained producer must consume its group quota, not every node + // slot. Keep the same one physical SQL worker and bounded queue. + for n in 0..3 { + let (old, applier) = f.appliers.remove(&(n, route.history)).context("old pool")?; + old.close().await?; + drop(old); + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 4, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + f.appliers.insert((n, route.history), (pool, applier)); + } + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let mut second = registration::prepare(&f, 31, &a).await?; + let other = second.birth.route().history; + for (n, input) in std::mem::take(&mut second.inputs).into_iter().enumerate() { + controllers[n].register(input)?.response().await?; + } + let healthy_dispatch = registration::dispatcher(&f, &second, class.clone())?; + let healthy_cell = history(second.birth.route(), "healthy-after-quarantine")?; + f.runtimes[0].campaign(other).await?; + wait_for(async || Ok(f.runtimes[0].status(other).await?.ready)).await?; + let before = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("before")?; + let captured = f.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let dispatcher = CellGroupDispatcher::new( + &f.cluster.domain, + f.cluster.directory.clone(), + class.clone(), + keys(89)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + faults + .mode + .store(physical::FAIL_DIRECTORY_SYNC, Ordering::Release); + let call = controllers[0].command( + controller_commands::invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry, + ) + .await?, + f.cell_options(&a).await?, + )?; + let error = call.response().await.err().context("quarantined command")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::GroupQuarantined) + )); + wait_for(async || { + Ok(controllers[0].usage().quarantined_groups == 1 + && controllers[0].usage().retained_commands == 1) + }) + .await?; + assert_eq!(faults.mode.load(Ordering::Acquire), 0); + assert!(matches!( + controllers[0].group_failure(route.history)?, + Some(CellGroupControllerError::Apply(_)) + )); + assert!(!controllers[0].usage().closing); + assert_eq!(controllers[0].usage().groups, 1); + assert!( + captured.check().is_err(), + "captured leader authority revoked" + ); + assert!(matches!( + f.runtimes[0].campaign(route.history).await, + Err(RuntimeError::GroupQuarantined) + )); + assert!(matches!( + f.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + f.cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await? + ) + .await, + Err(RuntimeError::GroupQuarantined) + )); + let pool = f.appliers[&(0, route.history)].0.clone(); + assert_eq!(pool.usage()?.active_cells, 1, "ambiguous producer retained"); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + // A refused retry must not replay the callback or settle the original producer. + let refused = controllers[0] + .command( + controller_commands::invocation( + &dispatcher, + &class, + &a, + Duration::from_secs(15), + retry, + ) + .await?, + f.cell_options(&a).await?, + )? + .response() + .await + .err() + .context("quarantine refusal")?; + assert!(!refused.ambiguous); + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::GroupQuarantined) + )); + assert_eq!(calls.load(Ordering::SeqCst), 1); + registration::enroll(&f, &controllers[0], &healthy_dispatch, &healthy_cell).await?; + registration::mutate( + &f, + &controllers[0], + &healthy_dispatch, + &class, + &healthy_cell, + retry, + ) + .await?; + registration::mutate( + &f, + &controllers[0], + &healthy_dispatch, + &class, + &healthy_cell, + retry, + ) + .await?; + assert_eq!( + calls.load(Ordering::SeqCst), + 2, + "healthy retry returns its receipt without replay" + ); + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + other, + healthy_cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let result = pool + .query( + f.cell_options(&healthy_cell).await?, + authority, + |db, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_le_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), &8_i64.to_le_bytes()); + drop(result); + let mut third = registration::prepare(&f, 35, &a).await?; + let refused = controllers[0] + .register(third.inputs.remove(0)) + .err() + .context("quarantine retains group capacity")?; + assert!(matches!( + refused.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + drop(third); + assert_eq!( + f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&before) + ); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(f.transports[0].usage().peers, 2); + for controller in &controllers { + controller.close().await?; + } + assert_eq!(controllers[0].usage().retained_commands, 0); + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(dispatcher.usage().1, 0); + drop(captured); + drop(controllers); + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn shared_engine_sync_failure_still_stops_all_groups() -> Result<()> { + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let other = controller::add_second(&mut f, route.history).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + f.runtimes[0].campaign(other).await?; + wait_for(async || Ok(f.runtimes[0].status(other).await?.ready)).await?; + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::FAIL_SYNC); + let cell = history(&route, "engine-failure")?; + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let _ = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &image, rooted)?, + ) + .await; + assert!( + tokio::time::timeout(Duration::from_secs(10), controllers[0].wait()) + .await? + .is_err() + ); + assert_eq!( + gate.mode.load(Ordering::Acquire), + fault::OFF, + "real log sync fault consumed" + ); + assert!(controllers[0].usage().closing); + for group in [route.history, other] { + assert!(matches!( + f.runtimes[0].status(group).await, + Err(RuntimeError::Closed) + )); + assert!(f.runtimes[0].campaign(group).await.is_err()); + } + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + drop(release); + for controller in &controllers[1..] { + controller.close().await?; + } + for (pool, _) in f.appliers.values() { + pool.close().await?; + } + f.appliers.clear(); + assert!(f.runtimes[0].close().await.is_err()); + for runtime in &f.runtimes[1..] { + runtime.close().await?; + } + f.runtimes.clear(); + drop(controllers); + assert!(f.cluster.voters[0].log.close().await.is_err()); + for voter in &f.cluster.voters[1..] { + voter.log.close().await?; + } + f.cluster.domain.shutdown().await; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_registration.rs b/crates/cellar/tests/support/cell_group_registration.rs index 3dde8c796..c9d182366 100644 --- a/crates/cellar/tests/support/cell_group_registration.rs +++ b/crates/cellar/tests/support/cell_group_registration.rs @@ -8,11 +8,11 @@ use std::sync::atomic::Ordering; #[allow(dead_code)] mod catalog_fault; -struct Prepared { +pub(super) struct Prepared { directory: RangeDirectory, - birth: CellGroupBirth, + pub(super) birth: CellGroupBirth, store: Arc, - inputs: Vec, + pub(super) inputs: Vec, number: u8, } @@ -30,7 +30,7 @@ fn class() -> Result> { )) } -async fn prepare(f: &RuntimeCluster, number: u8, a: &CellHistory) -> Result { +pub(super) async fn prepare(f: &RuntimeCluster, number: u8, a: &CellHistory) -> Result { let store = Arc::new(catalog_fault::Store::default()); let old = f.cluster.birth.route(); let scope = GroupHistory { @@ -174,7 +174,7 @@ async fn duplicate( }) } -fn dispatcher( +pub(super) fn dispatcher( f: &RuntimeCluster, prepared: &Prepared, class: Arc, @@ -194,7 +194,7 @@ fn dispatcher( )?) } -async fn enroll( +pub(super) async fn enroll( f: &RuntimeCluster, controller: &CellGroupController, dispatch: &CellGroupDispatcher, @@ -522,7 +522,7 @@ pub(crate) async fn registration_close() -> Result<()> { Ok(()) } -async fn mutate( +pub(super) async fn mutate( f: &RuntimeCluster, controller: &CellGroupController, dispatch: &CellGroupDispatcher, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 255105827..bf0eedd00 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -721,3 +721,8 @@ pub(super) use sql::{receipt_lookup_cancelled, receipt_lookup_ingress, receipt_l pub(super) use sql::{ registration_close, registration_live, registration_recovery, registration_refusals, }; + +pub(super) use sql::{ + group_quarantine_retains_producer_and_serves_healthy_group, + shared_engine_sync_failure_still_stops_all_groups, +}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index d3cce2830..b9ef5a66e 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -576,3 +576,10 @@ mod registration; pub(crate) use registration::{ registration_close, registration_live, registration_recovery, registration_refusals, }; + +#[path = "cell_group_quarantine.rs"] +mod quarantine; +pub(crate) use quarantine::{ + group_quarantine_retains_producer_and_serves_healthy_group, + shared_engine_sync_failure_still_stops_all_groups, +}; From 41121a7d08f283db6a83982ebfcba2feced036ca Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 12:06:51 -0700 Subject: [PATCH 113/568] test: await group quarantine at the runtime authority boundary --- crates/cellar/tests/support/cell_group_controller.rs | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs index 951c2de7c..bc1b6c58f 100644 --- a/crates/cellar/tests/support/cell_group_controller.rs +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -357,7 +357,15 @@ pub(crate) async fn controller_apply_faults(rooted: bool) -> Result<()> { ) .await?; if mode == 0 { - wait_for(async || Ok(controllers[0].group_failure(route.history)?.is_some())).await?; + // The diagnostic roster changes before the runtime control request + // completes; wait for the actual authority fence, not scheduler timing. + wait_for(async || { + Ok(matches!( + fixture.runtimes[0].status(route.history).await, + Err(RuntimeError::GroupQuarantined) + )) + }) + .await?; assert!( matches!(controllers[0].group_failure(route.history)?, Some(CellGroupControllerError::Apply(ref error)) if matches!(error.as_ref(), ExecutionError::Storage(_))) ); From acb3ee457a6577942926d3f95d70b8e22ef1b301 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 12:11:36 -0700 Subject: [PATCH 114/568] docs: qualify group quarantine and prioritize bounded provisioning --- docs/design/multi-raft/cell-groups/README.md | 6 +++ .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 54 ++++++++++++------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 44 +++++++++++++++ .../multi-raft/cell-groups/protocols.md | 9 +++- docs/design/multi-raft/cell-groups/tasks.json | 35 ++++++++++-- 7 files changed, 127 insertions(+), 27 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 62e485bc6..f419638d4 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -243,6 +243,12 @@ commands; transfer `.cellar/cell-groups-evidence/CG-04-dynamic-registration/` (281 source inputs, 4 executables). Per-group failure isolation and request-driven lazy birth are next; CG-04 and public cutover remain incomplete. +At `41121a7d`, group-local failures quarantine only the owning group while retaining +its capacity and physical work. Real shared-engine faults still stop the node. +The partial gate passes 233 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-group-quarantine-final/` (283 source inputs, +4 executables). Bounded request-driven provisioning is next; CG-04 remains open. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index f053ec09b..6839898f5 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "4326421a94c4f9a257bf6ceb8e0f7e518eacf3de", + "source_revision": "41121a7d08f283db6a83982ebfcba2feced036ca", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 30effd3d7..1af7da135 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,22 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `4326421a`, adding bounded dynamic group +The newest reviewed source baseline is `41121a7d`, adding retained group quarantine. +Group-local failure revokes admission and runtime execution while retaining group +capacity, apply tickets and ambiguous producers. Captured leader authority refuses; +late persistence cannot restore it. Shared engine failures and lost shared SQL +worker replies remain node-wide. The original group cause and quarantine count +are observable; recovery requires joined restart. Two native cases cover both +image modes with actual manifest/engine sync faults. The healthy group completes +controller enrollment/command/exact retry and a linearizable SQL read while the +failed producer remains retained. Capacity refusal and joined cleanup are checked. +The record `.cellar/cell-groups-evidence/CG-04-group-quarantine-final/checkpoint.json` +binds 233 Rust cases, twelve commands, 283 source inputs and 4 executables. +Transfer this ignored directory and its predecessors. CG-04 remains IN_PROGRESS. +Next wire bounded request-driven provisioning into registration, then complete +configuration lifecycle and public host/SDK integration under the plan dependencies. +Shared RustFS disaster recovery and load remain unqualified. +The preceding reviewed source baseline is `4326421a`, adding bounded dynamic group registration and empty-controller startup. Exact engine/root/applier page ownership and exclusive transport registration precede existing bootstrap revalidation. The applier is registered before attachment can emit events; active membership @@ -527,24 +542,25 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Deliver per-group failure isolation. Dynamic registration now adds groups - without rebuilding the controller, but a group-local fatal apply/command/event - still closes the node. Add an explicit quarantined group lifecycle that revokes - serving, keeps ambiguous producers and physical apply ownership, and permits - unrelated groups to progress. Qualify group-local apply failure and unresolved command - ownership while another group serves; do not discard work to make it continue. - Separately verify that a shared log-engine sync failure still quarantines all - groups on that engine, as required by design.md. - Then connect request-driven group provisioning/lazy birth to register(). The - caller currently prepares the exact bootstrap permit and selected root; a host - must bound catalog selection, physical preparation and concurrent birth before - transferring them. Reuse the existing admission and cancellation owners. - Keep read and command grants independent. Receipt absence remains ambiguous. - Destination catalog caching still depends on replicated transfer seals and - serving-admission invalidation in CG-09. Do not substitute a TTL for authority. - Connect public RaftCellHandle/discovery only through the actual replacement - host, preserving the old initializer refusal until cutover. Quorum/S3 minima - must refuse unsupported archive guarantees until CG-06 supplies real proofs. +1. Deliver bounded host provisioning and lazy group birth. Registration and + group-local quarantine now have concrete ownership boundaries. The caller + still supplies the bootstrap permit, selected root, applier and physical route. + Connect catalog selection and initial namespace/range birth to bounded local + preparation, coalescing concurrent requests by physical GroupHistory rather + than allocating a group per cell. Transfer prepared owners through register(). + Use the explicitly provisioned non-overlapping ranges in design.md. Managed + fleet bootstrap currently refuses in directory/bootstrap.rs; preserve that + refusal until the rooted fleet authorization required by CG-08 exists. + Keep catalog authorization and current-term quorum separate from routing hints; + retain preparation/attachment after the initiating request disconnects. + Quarantined groups remain charged until joined restart; never bypass them with + an in-place replacement or discard unresolved producers to recover capacity. + Shared-engine faults must remain node-wide. Keep read/command grants distinct + and receipt absence ambiguous. Destination catalog caching still depends on + replicated transfer seals/admission invalidation in CG-09, not a TTL. + Public RaftCellHandle/discovery must use the actual replacement host, preserving + old initializer refusal until cutover. Quorum/S3 minima must refuse unsupported + archive guarantees until CG-06 supplies real proofs. 2. Complete CG-04's production retry/cache lifecycle and configuration-entry handling, then its full gate before DONE. Use selected-root recovery at startup. Whole-process kill/power-loss remains distinct from the qualified directory-sync diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index e1662ef1a..928d3e671 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 76 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 78 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 0d444b5c6..ace48575e 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,50 @@ CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 retained group quarantine + +Qualified source: `41121a7d08f283db6a83982ebfcba2feced036ca`. + +A group-local apply/runtime failure now quarantines the group instead of closing +healthy groups on the node. The controller first refuses application admission, +then revokes runtime execution through reserved control capacity that bypasses a +waiting apply. Captured leader authority becomes invalid. Pending apply tickets, +unresolved command/enrollment producers and the group slot remain retained until +joined shutdown. Quarantine cannot be cleared by re-registration; a late physical +persistence completion cannot restore the revoked execution fence. + +`group_failure` reports the original cause; usage distinguishes active, pending +and quarantined groups. Shared log-engine I/O/corruption/closure and lost shared +SQL worker replies still close the node. Unknown group events and controller +ownership violations also retain the fail-closed boundary. Quarantine does not +promise capacity beyond the configured node/group budgets; retained work remains +charged rather than discarded to make another request fit. + +Both image modes pass real manifest directory-sync faults while an ambiguous +producer remains owned. A dynamically registered healthy group then completes +controller enrollment, command, exact retry and a linearizable SQLite query on +the same engine, single SQL worker and TLS peers. The failed group's selected root +stays unchanged, captured authority and retries refuse, callback counts exclude +replay, and a third registration cannot reuse its occupied capacity. Joined close +releases the producer. A separate real Raft-engine sync fault still stops both +groups and refuses replacement runtime admission. Existing physical cancellation +and lost-worker-reply tests remain in the gate. + +All 233 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, dependency isolation, docs, source preflight and 37 +validator cases. `.cellar/cell-groups-evidence/CG-04-group-quarantine-final/checkpoint.json` +binds 283 source inputs and 4 copied executables. Initial fixture compilation +errors and diagnostic reruns are retained with this ignored local evidence; +transfer it with the predecessor checkpoints. + +CG-04 remains IN_PROGRESS. The next critical gap is bounded request-driven group +provisioning: catalog selection/birth and physical root preparation must transfer +into the existing registration owner, coalesced by physical group rather than +cell. Configuration lifecycle and public host/SDK integration remain unfinished. +Later CG tasks still own snapshots, archive/reclamation, effects, topology, +hibernation, fenced recovery, load qualification and removal of the legacy host. +This checkpoint is not independent-process/power-loss, RustFS or load qualification. + ## CG-04 dynamic controller registration Qualified source: `4326421a94c4f9a257bf6ceb8e0f7e518eacf3de`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 5de19cd15..d5644eb01 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -194,8 +194,13 @@ still requires fresh current-term quorum. Dropping a registration waiter cannot cancel this work. Definite failures release their slot before responding. Shutdown joins entered catalog/attachment futures before closing transport, SQL and the runtime. A competing external runtime owner or runtime-wide failure still closes -the controller conservatively. Request-driven provisioning/lazy birth, per-group -failure isolation and configuration lifecycle remain separate unfinished work. +the controller conservatively. A group-local apply/runtime failure quarantines +that group: admission refuses, execution authority is revoked through reserved +runtime control capacity, and unresolved producers/apply tickets remain owned. +Quarantined groups retain their roster capacity until joined restart. Late shared +persistence completion cannot restore their serving fence. Shared log-engine +I/O/corruption/closure and lost shared SQL worker replies still close the node. +Request-driven provisioning/lazy birth and configuration lifecycle remain open. The current lane serializes RPC exchanges per peer; this is not load qualification. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 2870b32d7..f3e989003 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -526,7 +526,9 @@ "crates/cellar/durability/raft/receipt_cell_wire.rs", "crates/cellar/tests/support/cell_group_receipt_lookup.rs", "crates/cellar/durability/raft/cell_groups/controller/registration.rs", - "crates/cellar/tests/support/cell_group_registration.rs" + "crates/cellar/tests/support/cell_group_registration.rs", + "crates/cellar/durability/raft/cell_groups/controller/failure.rs", + "crates/cellar/tests/support/cell_group_quarantine.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -705,6 +707,13 @@ "dynamic_registration_joins_attachment_before_cancelled_shutdown", "dynamic_registration_recovers_selected_roots_into_an_empty_controller" ] + }, + { + "action": "Isolate group-local controller apply/runtime/ambiguous command failures through a bounded retained quarantine lifecycle. Reserve a control lane to revoke execution while apply is waiting, refuse new group admission, preserve captured producer/apply owners until joined shutdown, and prevent late persistence completion from restoring authority. Keep failed group capacity charged. Shared engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Expose original group failure and quarantine counts; no in-place replacement or automatic callback replay. Qualify healthy-group SQL progress and node-wide storage failure separately. Request-driven provisioning, configuration lifecycle and public cutover remain unfinished.", + "tests": [ + "group_quarantine_retains_producer_and_serves_healthy_group", + "shared_engine_sync_failure_still_stops_all_groups" + ] } ], "test_cases": [ @@ -1066,7 +1075,7 @@ }, { "name": "group_controller_apply_failure_and_shutdown_preserve_selected_root", - "oracle": "A real manifest sync failure closes the controller without advancing the selected root. Interrupted physical preparation cannot publish or allow premature replacement.", + "oracle": "A real manifest sync failure quarantines the group without advancing the selected root. Captured authority refuses; explicit shutdown still joins physical ownership. Interrupted preparation cannot publish or allow premature replacement.", "invariants": [ "I01", "I02", @@ -1076,7 +1085,7 @@ }, { "name": "rooted_group_controller_apply_failure_and_shutdown_preserve_selected_root", - "oracle": "Repeat manifest sync failure and joined controller shutdown with extent images; preserve the engine-selected root.", + "oracle": "Repeat group quarantine after manifest sync failure and joined controller shutdown with extent images; preserve the engine-selected root and physical ownership.", "invariants": [ "I01", "I02", @@ -1436,6 +1445,26 @@ "I05", "I12" ] + }, + { + "name": "group_quarantine_retains_producer_and_serves_healthy_group", + "oracle": "A real manifest directory-sync failure quarantines one group in both image modes. Preserve its selected root, revoke captured leader authority and new reads/writes, refuse callback replay, and retain its ambiguous producer and group capacity. Dynamically register another group on the same engine/SQL/TLS resources and prove controller enrollment, command, exact retry and linearizable SQLite query remain usable with per-group producer admission. Joined close releases retained command admission.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "shared_engine_sync_failure_still_stops_all_groups", + "oracle": "A real shared Raft-engine sync failure remains node-wide in both image modes: both groups refuse serving, controller/physical services join, and replacement runtime admission refuses the uncertain engine.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From 9d8cb9b25e77a97268233f78bd98a0497e5afa42 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 12:39:02 -0700 Subject: [PATCH 115/568] feat: provision assigned cell groups under joined bounded ownership --- .../raft/cell_groups/apply/images.rs | 4 +- .../raft/cell_groups/controller/failure.rs | 3 + .../raft/cell_groups/controller/mod.rs | 27 + .../controller/provision/config.rs | 129 ++++ .../controller/provision/manager.rs | 260 +++++++ .../cell_groups/controller/provision/mod.rs | 133 ++++ .../controller/provision/prepare.rs | 113 +++ .../cell_groups/controller/registration.rs | 38 +- .../raft/cell_groups/controller/requests.rs | 5 +- .../raft/cell_groups/controller/worker.rs | 7 + .../raft/cell_groups/directory/birth.rs | 19 + .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../tests/native_cell_groups_execution.rs | 25 + .../tests/support/cell_group_provision.rs | 657 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 6 + crates/cellar/tests/support/cell_group_sql.rs | 8 + .../cell-groups/implementation-plan.md | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 67 +- 18 files changed, 1487 insertions(+), 20 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/provision/config.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs create mode 100644 crates/cellar/tests/support/cell_group_provision.rs diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs index aa235d5bb..d31dd2ad1 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/images.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -43,7 +43,7 @@ pub(super) struct ImageApply<'a> { } impl CellGroupApplyImages { - pub(super) fn memory_bound(&self) -> usize { + pub(crate) fn memory_bound(&self) -> usize { let limits = match self { Self::Materialized { images, .. } => images.apply_limits(), Self::Extents { limits, .. } => *limits, @@ -53,7 +53,7 @@ impl CellGroupApplyImages { limits.max_total_decoded_bytes as usize + EXTENT_BYTES as usize } - pub(super) fn validate(&self) -> Result<(), ExecutionError> { + pub(crate) fn validate(&self) -> Result<(), ExecutionError> { match self { Self::Materialized { images, .. } => images.apply_limits().validate().map_err(storage), Self::Extents { diff --git a/crates/cellar/durability/raft/cell_groups/controller/failure.rs b/crates/cellar/durability/raft/cell_groups/controller/failure.rs index 88dc09aa9..195645946 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/failure.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/failure.rs @@ -39,6 +39,9 @@ fn dispatch(error: &CellDispatchError) -> bool { pub(super) fn node_wide(error: &CellGroupControllerError) -> bool { match error { + CellGroupControllerError::Catalog(error) => { + matches!(error.as_ref(), crate::raft::catalog::CatalogError::Log(error) if storage(error)) + } CellGroupControllerError::Runtime(error) => runtime(error), CellGroupControllerError::Apply(error) => execution(error), CellGroupControllerError::Command(error) => dispatch(&error.cause), diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 0ead568b3..fddf0fb6e 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -7,7 +7,9 @@ mod calls; mod commands; mod enrollments; mod failure; +mod provision; mod registration; +pub use provision::{CellGroupProvisionCall, CellGroupProvisioning}; mod requests; pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; pub(crate) use registration::GroupRoster; @@ -51,6 +53,7 @@ pub enum CellGroupControllerError { Runtime(RuntimeError), Apply(Arc), Transport(TransportError), + Catalog(Arc), Command(Arc), Enrollment(Arc), Closed, @@ -85,6 +88,9 @@ pub struct CellGroupControllerUsage { pub closing: bool, pub retained_commands: usize, pub command_bytes: usize, + pub retained_preparations: usize, + pub preparation_bytes: usize, + pub provision_waiters: usize, } struct Shared { @@ -94,6 +100,7 @@ struct Shared { limits: CellGroupControllerLimits, domain: crate::asyncrt::DomainToken, commands: Arc, + provisioning: Mutex>>, requests: tokio::sync::mpsc::Sender, active_apply: AtomicUsize, pending_apply: AtomicUsize, @@ -212,6 +219,7 @@ impl CellGroupController { limits, domain: domain.clone(), commands: commands::budget(limits), + provisioning: Mutex::new(None), requests, active_apply: AtomicUsize::new(0), pending_apply: AtomicUsize::new(0), @@ -249,6 +257,11 @@ impl CellGroupController { shared.stop(); let joined = worker.finish().await; drop(worker); + shared + .provisioning + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(); shared.active_apply.store(0, Ordering::Release); shared.pending_apply.store(0, Ordering::Release); shared.complete(result.and(joined)); @@ -269,7 +282,21 @@ impl CellGroupController { pub fn usage(&self) -> CellGroupControllerUsage { let shared = &self.handle.0; let (command_bytes, retained_commands) = shared.commands.usage(); + let (preparation_bytes, retained_preparations, provision_waiters) = shared + .provisioning + .lock() + .ok() + .and_then(|slot| { + slot.as_ref().map(|config| { + let (bytes, items) = config.preparations.usage(); + (bytes, items, config.waiters.usage().1) + }) + }) + .unwrap_or_default(); CellGroupControllerUsage { + preparation_bytes, + retained_preparations, + provision_waiters, groups: shared.groups.len(), pending_registrations: shared.groups.pending(), quarantined_groups: shared.groups.quarantined(), diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs new file mode 100644 index 000000000..f92529f6e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs @@ -0,0 +1,129 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit node resources; configuration itself performs no physical work. +use super::*; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::cell_groups::{CellGroupApplyImages, ManifestLimits, ManifestPins}; +use crate::raft::{budget::Budget, transport_options::RaftPeerNode}; +use std::path::PathBuf; + +/// Resources for initial members of already assigned ranges. Managed fleet +/// admission is not inferred from this configuration. Image stores retain their +/// own quotas; page inventory and recovery work have additional bounds here. +pub struct CellGroupProvisioning { + pub directory: PathBuf, + pub filesystem: Arc, + pub images: CellGroupApplyImages, + pub manifest: ManifestLimits, + pub entries: GroupEntryLimits, + pub max_effect_rows: u32, + pub max_pages: usize, + pub max_page_bytes: u64, + pub max_retained_roots: usize, + pub peers: Vec, + pub peer_revision: u64, + /// Includes recovered owners retained after a failed attachment. + pub max_preparations: usize, + pub max_preparation_bytes: usize, + pub max_parallel_preparations: usize, + /// Includes queued calls and unconsumed replies, independently of commands. + pub max_waiters: usize, +} + +pub(in super::super) struct Config { + pub settings: CellGroupProvisioning, + pub preparations: Arc, + pub waiters: Arc, + pub parallel: Arc, + pub bytes: usize, +} + +impl Config { + pub fn new( + mut settings: CellGroupProvisioning, + shared: &Shared, + ) -> Result { + let invalid = || CellGroupControllerError::Invalid("group provisioning configuration"); + settings.manifest.validate().map_err(|_| invalid())?; + settings.entries.validate().map_err(|_| invalid())?; + settings.images.validate().map_err(apply_error)?; + if !settings.directory.is_absolute() + || settings.directory.as_os_str().len() > 4096 + || settings + .directory + .components() + .any(|p| matches!(p, std::path::Component::ParentDir)) + || !(1..=1_000_000).contains(&settings.max_effect_rows) + || !(1..=4_000_000).contains(&settings.max_pages) + || !(1..=68_719_476_736).contains(&settings.max_page_bytes) + || !(2..=4096).contains(&settings.max_retained_roots) + || !(1..=128).contains(&settings.peers.len()) + || settings.peer_revision == 0 + || settings.max_preparations == 0 + || settings.max_preparations > shared.limits.max_groups + || settings.max_parallel_preparations == 0 + || settings.max_parallel_preparations > settings.max_preparations.min(256) + || !(1..=65_536).contains(&settings.max_waiters) + { + return Err(invalid()); + } + let local = shared.runtime.log().limits(); + let mut nodes = std::collections::BTreeSet::new(); + let mut addresses = std::collections::BTreeSet::new(); + let mut labels = std::collections::BTreeSet::new(); + for peer in &mut settings.peers { + if peer.label.len() > 4096 + || peer.server_name.len() > 4096 + || !nodes.insert(peer.node) + || !labels.insert(peer.label.clone()) + || ![peer.control_address, peer.data_address, peer.query_address] + .into_iter() + .all(|a| addresses.insert(a)) + { + return Err(invalid()); + } + peer.validate_endpoint().map_err(|_| invalid())?; + peer.label = std::mem::take(&mut peer.label) + .into_boxed_str() + .into_string(); + peer.server_name = std::mem::take(&mut peer.server_name) + .into_boxed_str() + .into_string(); + } + if !settings + .peers + .iter() + .any(|p| p.node == local.node && p.disk == local.disk && p.process == local.process) + { + return Err(invalid()); + } + // Bound recovery pins, BTree inventory/leases, recursive page decoding, + // image scratch and all bounded catalog/peer/request metadata. The active + // roster retains at most max_groups stores of these configured sizes. + let bytes = ManifestPins::maximum_bytes(settings.manifest.max_cells) + .checked_add(settings.max_pages.checked_mul(256).ok_or_else(invalid)?) + .and_then(|n| { + n.checked_add( + settings.manifest.max_depth as usize + * crate::raft::cell_groups::page_wire::MAX_PAGE_BYTES, + ) + }) + .and_then(|n| n.checked_add(settings.images.memory_bound())) + .and_then(|n| n.checked_add(2 * 1024 * 1024)) + .ok_or_else(invalid)?; + if settings.max_preparation_bytes < bytes { + return Err(invalid()); + } + settings.directory = settings.directory.into_boxed_path().into_path_buf(); + settings.peers = settings.peers.into_boxed_slice().into_vec(); + Ok(Self { + preparations: Budget::new(settings.max_preparation_bytes, settings.max_preparations), + waiters: Budget::new(settings.max_waiters * WAITER_BYTES, settings.max_waiters), + parallel: Arc::new(tokio::sync::Semaphore::new( + settings.max_parallel_preparations, + )), + bytes, + settings, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs new file mode 100644 index 000000000..1584925bd --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs @@ -0,0 +1,260 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Retained preparations are retryable without reopening an engine-pinned store. +use super::*; +use crate::raft::transport::PhysicalRegistration; +use cellar_logic::cell_group::GroupHistory; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; + +struct Entry { + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + reservation: registration::Reservation, + permit: Option, + prepared: Option, + waiters: Vec, + running: bool, + active: bool, +} + +pub(in super::super) enum Completed { + Prepared( + GroupHistory, + Result, CellGroupControllerError>, + ), + Attached( + GroupHistory, + Result, + ), +} + +#[derive(Default)] +pub(in super::super) struct Manager { + entries: BTreeMap, + jobs: FuturesUnordered>, +} + +impl Manager { + pub fn is_running(&self) -> bool { + !self.jobs.is_empty() + } + + pub async fn next(&mut self) -> Option { + self.jobs.next().await + } + + pub fn request(&mut self, request: Request, shared: &Shared) { + let Request { + directory, + route, + config, + waiter, + } = request; + let group = route.history; + let ready = match shared.groups.contains(&group) { + Ok(ready) => ready, + Err(error) => { + waiter.finish(Err(error.into())); + return; + } + }; + if let Some(entry) = self.entries.get_mut(&group) { + if !entry.directory.same_owner(&directory) || entry.route != route { + waiter.finish(Err(CellGroupControllerError::Invalid( + "conflicting provision binding", + ))); + return; + } + if entry.active { + // This owner already attached the exact assignment. No network + // read or fresh quorum claim is implied by this idempotent reply. + waiter.finish(if ready { + Ok(()) + } else { + Err(RuntimeError::NotFound.into()) + }); + return; + } + entry.waiters.push(waiter); + if !entry.running { + self.launch(group, shared.runtime.clone()); + } + return; + } + let admitted = (|| { + // Manual registrations cannot be adopted by a different lifecycle. + if ready { + return Err(RuntimeError::AlreadyAttached); + } + let permit = config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded)?; + let reservation = shared.groups.reserve(group)?; + Ok((permit, reservation)) + })(); + let (permit, reservation) = match admitted { + Ok(value) => value, + Err(error) => { + waiter.finish(Err(error.into())); + return; + } + }; + self.entries.insert( + group, + Entry { + directory, + route, + config, + reservation, + permit: Some(permit), + prepared: None, + waiters: vec![waiter], + running: false, + active: false, + }, + ); + self.launch(group, shared.runtime.clone()); + } + + fn launch(&mut self, group: GroupHistory, runtime: CellGroupRuntime) { + let entry = self.entries.get_mut(&group).expect("provision entry"); + entry.running = true; + let directory = entry.directory.clone(); + let route = entry.route.clone(); + let config = entry.config.clone(); + let prepared = entry.prepared.clone(); + self.jobs.push(Box::pin(async move { + Completed::Prepared( + group, + prepare::recover(directory, route, config, runtime, prepared) + .await + .map(Box::new), + ) + })); + } + + pub fn complete( + &mut self, + done: Completed, + shared: &Shared, + transport: &RaftTransport, + roster: &mut BTreeMap, + routes: &mut BTreeMap, + ) -> Result<(), CellGroupControllerError> { + match done { + Completed::Prepared(group, Ok(input)) => { + let entry = self.entries.get_mut(&group).expect("prepared entry"); + // Save the owners before polling attachment: early Raft events + // must see the applier, and a catalog refusal must retain pins. + entry.prepared = Some(prepare::Prepared { + applier: input.applier.clone(), + current: input.current.clone(), + }); + roster.insert(group, input.applier.clone()); + let runtime = shared.runtime.clone(); + let transport = transport.clone(); + self.jobs.push(Box::pin(async move { + Completed::Attached( + group, + registration::attach(*input, runtime, transport).await, + ) + })); + Ok(()) + } + Completed::Prepared(group, Err(error)) => self.failed(group, error), + Completed::Attached(group, result) => match result { + Ok(guard) => { + let entry = self.entries.get_mut(&group).expect("attached entry"); + if let Err(error) = entry.reservation.activate() { + // Quarantine may have raced an early runtime event. Keep + // the route and all owners until the joined shutdown. + routes.insert(group, guard); + return self.failed(group, error.into()); + } + routes.insert(group, guard); + entry.active = true; + entry.running = false; + entry.prepared = None; + entry.permit = None; + for waiter in std::mem::take(&mut entry.waiters) { + waiter.finish(Ok(())); + } + Ok(()) + } + Err(error) => self.failed(group, error), + }, + } + } + + fn failed( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + ) -> Result<(), CellGroupControllerError> { + let entry = self.entries.get_mut(&group).expect("failed entry"); + entry.running = false; + let waiters = std::mem::take(&mut entry.waiters); + // Before recovery installs a physical owner, a failed attempt can give + // back its preparation/roster capacity. Immutable birth claims persist. + if entry.prepared.is_none() { + self.entries.remove(&group); + } + for waiter in waiters { + waiter.finish(Err(error.clone())); + } + // Registration refusal happens before any transport submission. An + // identity/capacity refusal is retryable without stopping healthy groups; + // an actual closed service or uncertain engine/runtime still is fatal. + let uncertain = match &error { + CellGroupControllerError::Transport(TransportError::Runtime(error)) => { + failure::node_wide(&CellGroupControllerError::Runtime(error.clone())) + || matches!(error, RuntimeError::AlreadyAttached) + } + CellGroupControllerError::Transport(error) => matches!(error, TransportError::Closed), + // Recovery checks the engine marker again after installing its pins. + // A competing selection cannot safely reopen the same locked files. + CellGroupControllerError::Apply(error) + if matches!( + error.as_ref(), + ExecutionError::Invalid("recovery selection changed") + ) => + { + true + } + _ => { + failure::node_wide(&error) + || matches!( + error, + CellGroupControllerError::Runtime(RuntimeError::AlreadyAttached) + ) + } + }; + if uncertain { + return Err(error); + } + Ok(()) + } + + pub async fn join(&mut self) { + // Do not launch attachment after close, but do join every entered + // blocking recovery and attachment before releasing dependencies. + while let Some(done) = self.jobs.next().await { + if let Completed::Prepared(group, Ok(input)) = done { + self.entries + .get_mut(&group) + .expect("closing entry") + .prepared = Some(prepare::Prepared { + applier: input.applier, + current: input.current, + }); + } + } + for entry in self.entries.values_mut() { + for waiter in std::mem::take(&mut entry.waiters) { + waiter.finish(Err(CellGroupControllerError::Closed)); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs new file mode 100644 index 000000000..27a236c01 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs @@ -0,0 +1,133 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One preparation per physical group, owned and joined by the node controller. +mod config; +mod manager; +mod prepare; +pub use config::CellGroupProvisioning; +pub(super) use config::Config; +pub(super) use manager::Manager; + +use super::{calls::Delivery, *}; +use crate::raft::{budget::Permit, catalog::CatalogError, cell_groups::RangeDirectory}; +use cellar_logic::cell_group::GroupRoute; +use tokio::sync::oneshot; + +// Logical routes have fixed-width bounded keys. Directory handles share their +// separately bounded cache; no tree or cell list is copied into a request. +const WAITER_BYTES: usize = 4096; + +fn apply_error(error: ExecutionError) -> CellGroupControllerError { + CellGroupControllerError::Apply(Arc::new(error)) +} + +fn catalog_error(error: CatalogError) -> CellGroupControllerError { + CellGroupControllerError::Catalog(Arc::new(error)) +} + +/// Response loss never cancels physical recovery or attachment. Success means +/// locally attached, not elected, caught up, or authorized to serve a request. +pub struct CellGroupProvisionCall { + receiver: oneshot::Receiver>, +} + +impl CellGroupProvisionCall { + pub async fn response(self) -> Result<(), Arc> { + self.receiver + .await + .map_err(|_| Arc::new(CellGroupControllerError::Closed))? + .result + } +} + +struct Waiter { + reply: oneshot::Sender>, + permit: Arc, +} + +impl Waiter { + fn finish(self, result: Result<(), CellGroupControllerError>) { + let _ = self.reply.send(Delivery { + result: result.map_err(Arc::new), + _permit: self.permit, + }); + } +} + +pub(super) struct Request { + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + waiter: Waiter, +} + +impl CellGroupController { + /// Install immutable node resources once. Reconfiguration requires a joined + /// restart so pending recovery cannot change filesystem or image owners. + pub fn configure_provisioning( + &self, + settings: CellGroupProvisioning, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + let config = Arc::new(Config::new(settings, shared)?); + let mut slot = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + if slot.is_some() { + return Err(CellGroupControllerError::Invalid( + "provisioning already configured", + )); + } + *slot = Some(config); + Ok(()) + } + + /// Materialize and recover an already selected initial-member assignment. + /// Concurrent requests coalesce only for the same directory owner and exact + /// route. This never allocates a range or a Raft group for an individual cell. + pub fn provision( + &self, + directory: RangeDirectory, + route: GroupRoute, + ) -> Result { + let shared = &self.handle.0; + if shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Closed); + } + let config = shared + .provisioning + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + .ok_or(CellGroupControllerError::Invalid( + "provisioning not configured", + ))?; + let permit = config + .waiters + .acquire(WAITER_BYTES) + .ok_or(RuntimeError::Overloaded)?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Provision(Box::new(Request { + directory, + route, + config, + waiter: Waiter { + reply, + permit: Arc::new(permit), + }, + }))) + .map_err(|e| match e { + tokio::sync::mpsc::error::TrySendError::Closed(_) => { + CellGroupControllerError::Closed + } + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded.into(), + })?; + Ok(CellGroupProvisionCall { receiver }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs new file mode 100644 index 000000000..7ba87057c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs @@ -0,0 +1,113 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::cell_groups::{ManifestPageStore, PreparedGroupManifest}; +use crate::raft::transport::{GroupRoute as PhysicalRoute, RaftPeer}; +use cellar_logic::consensus::ReplicaIdentity; + +#[derive(Clone)] +pub(super) struct Prepared { + pub applier: CellGroupApplier, + pub current: PreparedGroupManifest, +} + +pub(super) async fn recover( + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + runtime: CellGroupRuntime, + existing: Option, +) -> Result { + let _parallel = config + .parallel + .acquire() + .await + .map_err(|_| CellGroupControllerError::Closed)?; + let birth = directory + .materialize_selected_group(&route) + .await + .map_err(catalog_error)?; + // Check complete trusted placement before creating genesis or page files. + let mut peers = Vec::new(); + for member in birth.voters().iter().chain(birth.learners()) { + let node = config + .settings + .peers + .iter() + .find(|p| p.node == member.node) + .ok_or(CellGroupControllerError::Invalid( + "unconfigured birth member", + ))?; + peers.push(RaftPeer { + replica: ReplicaIdentity { + scope: route.history, + member: member.member, + node: node.node, + disk: node.disk, + process: node.process, + }, + label: node.label.clone(), + server_name: node.server_name.clone(), + control_address: node.control_address, + data_address: node.data_address, + query_address: node.query_address, + }); + } + let permit = directory + .bootstrap_group_member(&birth, runtime.log()) + .await + .map_err(catalog_error)?; + let prepared = match existing { + Some(prepared) => prepared, + None => { + let config = config.clone(); + let group = permit.group().clone(); + runtime + .log() + .domain() + .blocking(move || { + let settings = &config.settings; + let history = group.identity().scope; + let path = settings.directory.join(format!( + "{}-{}-{}-{}", + hex::encode(history.cluster.bytes()), + hex::encode(history.namespace.bytes()), + hex::encode(history.group.bytes()), + hex::encode(history.incarnation.bytes()) + )); + let pages = ManifestPageStore::open( + &path, + settings.filesystem.clone(), + settings.max_pages, + settings.max_page_bytes, + settings.max_retained_roots, + ) + .map_err(crate::raft::execution::storage)?; + let (applier, current) = CellGroupApplier::recover( + group, + pages, + settings.images.clone(), + settings.manifest, + settings.entries, + settings.max_effect_rows, + )?; + Ok::<_, ExecutionError>(Prepared { applier, current }) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(apply_error)? + } + }; + Ok(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: PhysicalRoute { + scope: route.history, + schema_digest: route.schema_digest, + revision: config.settings.peer_revision, + peers, + }, + }) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs index 3809a4a1c..06d43247c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/registration.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -104,7 +104,10 @@ impl GroupRoster { .unwrap_or(0) } - fn reserve(self: &Arc, group: GroupHistory) -> Result { + pub(super) fn reserve( + self: &Arc, + group: GroupHistory, + ) -> Result { let mut groups = self.groups.lock().map_err(|_| RuntimeError::Closed)?; if groups.contains_key(&group) { return Err(RuntimeError::AlreadyAttached); @@ -121,14 +124,14 @@ impl GroupRoster { } } -struct Reservation { +pub(super) struct Reservation { roster: Arc, group: GroupHistory, active: bool, } impl Reservation { - fn activate(&mut self) -> Result<(), RuntimeError> { + pub(super) fn activate(&mut self) -> Result<(), RuntimeError> { let mut groups = self .roster .groups @@ -343,16 +346,7 @@ pub(super) async fn drive( permit, } = request; let group = registration.permit.route().history; - let result = async { - let guard = transport - .register_physical_group(registration.transport) - .map_err(CellGroupControllerError::Transport)?; - runtime - .attach(registration.permit, registration.current) - .await?; - Ok(guard) - } - .await; + let result = attach(registration, runtime, transport).await; Completed { group, result, @@ -362,6 +356,24 @@ pub(super) async fn drive( } } +pub(super) async fn attach( + registration: CellGroupRegistration, + runtime: CellGroupRuntime, + transport: RaftTransport, +) -> Result { + registration + .applier + .check_startup(®istration.current) + .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; + let guard = transport + .register_physical_group(registration.transport) + .map_err(CellGroupControllerError::Transport)?; + runtime + .attach(registration.permit, registration.current) + .await?; + Ok(guard) +} + impl Completed { /// A runtime-wide close or competing external owner is not a safely failed /// registration. Preserve the conservative joined node shutdown boundary. diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index b72341532..ae06071de 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -5,6 +5,7 @@ use super::*; pub(super) enum Request { Registration(Box), + Provision(Box), Command(Box), Enrollment(Box), } @@ -47,7 +48,9 @@ pub(super) async fn drive( progress: tokio::sync::watch::Receiver<()>, ) -> Completed { match request { - Request::Registration(_) => unreachable!("registration has its own physical join owner"), + Request::Registration(_) | Request::Provision(_) => { + unreachable!("registration has its own physical join owner") + } Request::Command(request) => Completed::Command(Box::new( commands::drive(*request, runtime, sql, shared, progress).await, )), diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 6a0216aa9..7b8319007 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -41,6 +41,7 @@ pub(super) struct Worker { failed_commands: Vec, quarantined: BTreeSet, retained_apply: Vec, + provisioning: provision::Manager, registrations: FuturesUnordered>, routes: BTreeMap, progress: tokio::sync::watch::Sender<()>, @@ -74,6 +75,7 @@ impl Worker { quarantined: BTreeSet::new(), retained_apply: Vec::new(), registrations: FuturesUnordered::new(), + provisioning: provision::Manager::default(), routes: BTreeMap::new(), progress: tokio::sync::watch::channel(()).0, } @@ -163,6 +165,7 @@ impl Worker { }, Some(request) = self.requests.recv() => { match request { + requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared), requests::Request::Registration(request) => { let group = request.registration.applier.history(); if self.roster.insert(group, request.registration.applier.clone()).is_some() { @@ -173,6 +176,9 @@ impl Worker { other => self.commands.push(Box::pin(requests::drive(other, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))), } } + Some(done) = self.provisioning.next(), if self.provisioning.is_running() => { + self.provisioning.complete(done, &self.shared, &self.transport, &mut self.roster, &mut self.routes)?; + } Some(completed) = self.registrations.next(), if !self.registrations.is_empty() => { let group = completed.group; let uncertain = completed.uncertain(); @@ -279,6 +285,7 @@ impl Worker { while let Some(completed) = self.registrations.next().await { let _ = completed.finish(true); } + self.provisioning.join().await; self.transport.close().await; let sql = self .sql diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs index 4a295d3d4..5cd02aa4b 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/birth.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -75,6 +75,25 @@ impl CellGroupBirth { } impl RangeDirectory { + /// Recover the immutable reservation for an already selected range. This + /// never chooses placement or allocates another group in response to a cell. + pub async fn materialize_selected_group( + &self, + route: &GroupRoute, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.selected_birth_route(route).await?; + let request = self.verify_birth_reservation(route).await?; + self.materialize_group_birth(&request).await + }) + .await + } + + pub(crate) fn same_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.cache, &other.cache) + } + pub(super) fn unmanaged_birth(&self) -> Result<(), CatalogError> { if self.namespace.config.managed_fleet.is_some() { return Err(CatalogError::ManagedAdmissionRequired); diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 5f7484c8b..fafeb0c70 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -19,8 +19,8 @@ mod controller; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, CellGroupController, CellGroupControllerError, CellGroupControllerLimits, - CellGroupControllerUsage, CellGroupRegistration, CellGroupRegistrationCall, - CellGroupRegistrationError, + CellGroupControllerUsage, CellGroupProvisionCall, CellGroupProvisioning, CellGroupRegistration, + CellGroupRegistrationCall, CellGroupRegistrationError, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index f67961eaf..9d9609fc4 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1575,3 +1575,28 @@ async fn group_quarantine_retains_producer_and_serves_healthy_group() -> Result< async fn shared_engine_sync_failure_still_stops_all_groups() -> Result<()> { runtime_fixture::shared_engine_sync_failure_still_stops_all_groups().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_coalesces_and_serves() -> Result<()> { + runtime_fixture::provisioning_coalesces_and_serves().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_retries_retained_owner() -> Result<()> { + runtime_fixture::provisioning_retries_retained_owner().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_joins_cancelled_close() -> Result<()> { + runtime_fixture::provisioning_joins_cancelled_close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_recovers_selected_roots() -> Result<()> { + runtime_fixture::provisioning_recovers_selected_roots().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn provisioning_refusals_preserve_healthy_group() -> Result<()> { + runtime_fixture::provisioning_refusals_preserve_healthy_group().await +} diff --git a/crates/cellar/tests/support/cell_group_provision.rs b/crates/cellar/tests/support/cell_group_provision.rs new file mode 100644 index 000000000..efb218e9a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_provision.rs @@ -0,0 +1,657 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit assigned-range provisioning over real engines, files and TLS. +use super::remote_queries::{keys, wait_for}; +use super::*; +use std::sync::atomic::{AtomicBool, Ordering}; +#[path = "raft_catalog_fault.rs"] +#[allow(dead_code)] +mod catalog_fault; +use super::apply_faults::physical; + +async fn assignment( + f: &RuntimeCluster, + number: u8, +) -> Result<(RangeDirectory, GroupRoute, Arc)> { + let store = Arc::new(catalog_fault::Store::default()); + let old = f.cluster.birth.route(); + let scope = GroupHistory { + cluster: old.history.cluster, + namespace: NamespaceId::new([number; 16])?, + group: GroupId::new([number; 16])?, + incarnation: IncarnationId::new([number; 16])?, + }; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = f + .cluster + .domain + .run(catalog.open_namespace(NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + old.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = directory.prepare_group_birth(CellGroupBirthSpec { + history: scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([number; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + f.cluster + .domain + .run(directory.reserve_group_birth(&birth)) + .await??; + let initialize = directory + .prepare_initialization(ProposalId::new([number + 1; 16])?, &[birth.route().clone()])?; + f.cluster + .domain + .run(directory.publish(&initialize)) + .await??; + // No materialized birth, bootstrap permit, page store or selected root has + // been prepared by the fixture. Production provisioning owns all of them. + for voter in &f.cluster.voters { + assert!(matches!( + voter.log.cell_group(&scope), + Err(LogError::NotFound) + )); + } + Ok((directory, birth.route().clone(), store)) +} + +fn class() -> Result> { + Ok(Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |ctx, input| { + ctx.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + )) +} + +async fn settings( + f: &RuntimeCluster, + a: &CellHistory, + node: usize, +) -> Result { + let images = match f.cell_options_at(a, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }; + let peers = f + .tcp_route(f.cluster.birth.route().history)? + .peers + .iter() + .map(RaftPeerNode::from) + .collect(); + Ok(CellGroupProvisioning { + directory: f.cluster._root.path().join(format!("provision-{node}")), + filesystem: Arc::new(DirectFileSystem), + images, + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: GroupEntryLimits::default(), + max_effect_rows: 100, + max_pages: 1024, + max_page_bytes: 16 * 1024 * 1024, + max_retained_roots: 16, + peers, + peer_revision: 1, + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + }) +} + +fn pages_path(base: &std::path::Path, h: GroupHistory) -> std::path::PathBuf { + base.join(format!( + "{}-{}-{}-{}", + hex::encode(h.cluster.bytes()), + hex::encode(h.namespace.bytes()), + hex::encode(h.group.bytes()), + hex::encode(h.incarnation.bytes()) + )) +} + +pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let mut limits = controller::limits(); + limits.max_commands = 1; + let controllers = controller::start_with_limits(&mut f, limits).await?; + let (directory, route, store) = assignment(&f, 31).await?; + let (foreign, _, _) = assignment(&f, 31).await?; + let (other, other_route, _) = assignment(&f, 41).await?; + for (node, c) in controllers.iter().enumerate() { + c.configure_provisioning(settings(&f, &a, node).await?)?; + } + assert!(controllers[0] + .configure_provisioning(settings(&f, &a, 0).await?) + .is_err()); + store.hold_group_head_reads.store(true, Ordering::Release); + let first = controllers[0].provision(directory.clone(), route.clone())?; + wait_for(async || Ok(store.active_group_head_reads.load(Ordering::Acquire) == 1)).await?; + let second = controllers[0].provision(directory.clone(), route.clone())?; + wait_for(async || Ok(controllers[0].usage().provision_waiters == 2)).await?; + assert!(matches!( + controllers[0].provision(directory.clone(), route.clone()), + Err(CellGroupControllerError::Runtime(RuntimeError::Overloaded)) + )); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert_eq!(store.active_group_head_reads.load(Ordering::Acquire), 1); + let mut second = Box::pin(second.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), second.as_mut()) + .await + .is_err() + ); + drop(first); + // Dropping waiters cannot release a still-running physical preparation. + assert_eq!(controllers[0].usage().retained_preparations, 1); + store.hold_group_head_reads.store(false, Ordering::Release); + store.group_head_reads_resume.notify_waiters(); + second.await?; + wait_for(async || { + Ok(controllers[0].usage().groups == 2 && controllers[0].usage().provision_waiters == 0) + }) + .await?; + assert_eq!(controllers[0].usage().preparation_bytes, 0); + let rejected = controllers[0] + .provision(other, other_route)? + .response() + .await + .unwrap_err(); + assert!(matches!( + rejected.as_ref(), + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + assert!(controllers[0] + .provision(foreign, route.clone())? + .response() + .await + .is_err()); + let mut wrong = route.clone(); + wrong.schema_digest = [99; 32]; + assert!(controllers[0] + .provision(directory.clone(), wrong)? + .response() + .await + .is_err()); + for c in &controllers[1..] { + c.provision(directory.clone(), route.clone())? + .response() + .await?; + } + controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await?; + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let cell = history(&route, "provisioned-cell")?; + registration::enroll(&f, &controllers[0], &dispatch, &cell).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + wait_for(async || Ok(controllers[0].usage().retained_commands == 0)).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(route.history, cell.clone()) + .await + .is_ok_and(|a| a.cell().version.sequence.get() == 2)) + }) + .await?; + } + for c in &controllers { + assert_eq!(c.usage().groups, 2); + c.close().await?; + assert_eq!(c.usage().provision_waiters, 0); + } + drop(dispatch); + drop(controllers); + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_retries_retained_owner() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, store) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0).await?; + let path = pages_path(&options.directory, route.history); + let faults = physical::Faults::new(path.clone(), None); + options.filesystem = Arc::new(physical::FaultFileSystem { + faults: faults.clone(), + }); + controllers[0].configure_provisioning(options)?; + store.fail_read.store(true, Ordering::Release); + assert!(controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(controllers[0].usage().pending_registrations, 0); + assert_eq!(controllers[0].usage().preparation_bytes, 0); + let fault_store = store.clone(); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + fault_store.fail_read.store(true, Ordering::Release); + })); + // Catalog refusal occurs only after physical recovery. The selected page + // owner remains held and must not be reopened on the next attempt. + assert!(controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert_eq!(controllers[0].usage().pending_registrations, 1); + assert!(cellar_ltx::FileSystem::lock_file(&DirectFileSystem, &path.join("LOCK")).is_err()); + let syncs = faults.synced_paths.lock().unwrap().len(); + controllers[0] + .provision(directory.clone(), route.clone())? + .response() + .await?; + assert_eq!( + faults.synced_paths.lock().unwrap().len(), + syncs, + "retry must reuse recovered page owner" + ); + assert_eq!(controllers[0].usage().retained_preparations, 0); + for c in controllers { + c.close().await?; + } + f.close().await?; + } + Ok(()) +} + +struct Release(Option>); + +impl Drop for Release { + fn drop(&mut self) { + if let Some(sender) = self.0.take() { + let _ = sender.send(()); + } + } +} + +pub(crate) async fn provisioning_joins_cancelled_close() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0).await?; + let faults = physical::Faults::new(pages_path(&options.directory, route.history), None); + let entered = Arc::new(AtomicBool::new(false)); + let reached = entered.clone(); + let (send, receive) = std::sync::mpsc::sync_channel(1); + let release = Release(Some(send)); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + reached.store(true, Ordering::Release); + receive + .recv_timeout(Duration::from_secs(15)) + .expect("physical join release"); + })); + options.filesystem = Arc::new(physical::FaultFileSystem { faults }); + controllers[0].configure_provisioning(options)?; + let response = controllers[0].provision(directory, route)?; + wait_for(async || Ok(entered.load(Ordering::Acquire))).await?; + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].wait()) + .await + .is_err() + ); + assert_eq!(controllers[0].usage().retained_preparations, 1); + assert!(matches!( + CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + drop(response); + drop(release); + controllers[0].close().await?; + let (replacement, events) = CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for c in controllers { + c.close().await?; + assert_eq!(c.usage().retained_preparations, 0); + } + f.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_recovers_selected_roots() -> Result<()> { + for rooted in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + for (n, c) in controllers.iter().enumerate() { + c.configure_provisioning(settings(&f, &a, n).await?)?; + c.provision(directory.clone(), route.clone())? + .response() + .await?; + } + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let cell = history(&route, "recovered-provision")?; + registration::enroll(&f, &controllers[0], &dispatch, &cell).await?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &cell, retry).await?; + for runtime in &f.runtimes { + wait_for(async || { + Ok(runtime + .reconciliation_authority(route.history, cell.clone()) + .await + .is_ok_and(|a| a.cell().version.sequence.get() == 2)) + }) + .await?; + } + for c in controllers { + c.close().await?; + } + reopen_and_verify(&mut f, directory, &route, &cell, &dispatch, &class, retry).await?; + drop(dispatch); + f.close().await?; + } + Ok(()) +} + +// Image-store handles remain in this process; the engine/page owners and runtime +// are fresh. This is deliberately not independent-process/power-loss evidence. +async fn reopen_and_verify( + f: &mut RuntimeCluster, + directory: RangeDirectory, + route: &GroupRoute, + cell: &CellHistory, + dispatch: &CellGroupDispatcher, + class: &RaftClass, + retry: u64, +) -> Result<()> { + f.appliers.clear(); + f.transports.clear(); + f.runtimes.clear(); + for n in 0..3 { + f.cluster.voters[n].log.close().await?; + let number = n as u8 + 1; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + f.cluster._root.path().join(format!("node-{number}/log")), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: f.cluster.voters[n].gate.clone(), + }), + )) + .await??; + f.cluster.voters[n].log = log; + let (runtime, events) = CellGroupRuntime::start( + f.cluster.voters[n].log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([71; 32]), + ..Default::default() + }, + ) + .await?; + f.transports + .push(super::super::tcp::start(&runtime, n).await?); + f.runtimes.push(runtime); + f.events.push(events); + } + let mut controllers = vec![]; + let mut pools = vec![]; + let events = std::mem::take(&mut f.events); + for (n, events) in events.into_iter().enumerate() { + let pool = CellGroupSqlPool::start( + &f.cluster.voters[n].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let c = CellGroupController::start( + f.runtimes[n].clone(), + events, + f.transports[n].clone(), + pool.clone(), + vec![], + controller::limits(), + ) + .await?; + assert_eq!(c.usage().groups, 0); + let options = settings(f, cell, n).await?; + let marker = f.cluster.voters[n] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected marker")?; + let page = pages_path(&options.directory, route.history).join(format!( + "{}.page", + hex::encode(marker.root().digest().context("nonempty root")?) + )); + let saved = if n == 0 { + Some(std::fs::read(&page)?) + } else { + None + }; + c.configure_provisioning(options)?; + if let Some(bytes) = saved { + std::fs::remove_file(&page)?; + assert!(c + .provision(directory.clone(), route.clone())? + .response() + .await + .is_err()); + assert_eq!(c.usage().groups, 0); + assert_eq!( + f.cluster.voters[n] + .log + .cell_group(&route.history)? + .group_manifest()? + .as_ref(), + Some(&marker) + ); + std::fs::write(&page, bytes)?; + } + c.provision(directory.clone(), route.clone())? + .response() + .await?; + controllers.push(c); + pools.push(pool); + } + assert!(matches!( + f.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await, + Err(RuntimeError::NotLeader) + )); + f.runtimes[0].campaign(route.history).await?; + wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + registration::mutate(f, &controllers[0], dispatch, class, cell, retry).await?; + let deadline = f + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5_000 }) + .await?; + let authority = f.runtimes[0] + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + let output = pools[0] + .query(f.cell_options(cell).await?, authority, |db, version| { + assert_eq!(version.sequence.get(), 2); + let n: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(n.to_le_bytes().to_vec()) + }) + .await?; + assert_eq!(output.result(), &8_i64.to_le_bytes()); + drop(output); + for c in controllers { + c.close().await?; + } + Ok(()) +} + +pub(crate) async fn provisioning_refusals_preserve_healthy_group() -> Result<()> { + for rooted in [false, true] { + for missing_member in [false, true] { + let class = class()?; + let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + let (directory, route, _) = assignment(&f, 31).await?; + let mut options = settings(&f, &a, 0).await?; + let incorrect_endpoint = std::net::TcpListener::bind("127.0.0.1:0")?; + if missing_member { + options.peers.pop(); + } else { + options.peers[0].control_address = incorrect_endpoint.local_addr()?; + } + controllers[0].configure_provisioning(options)?; + let error = controllers[0] + .provision(directory, route.clone())? + .response() + .await + .unwrap_err(); + if missing_member { + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Invalid("unconfigured birth member") + )); + assert!(matches!( + f.cluster.voters[0].log.cell_group(&route.history), + Err(LogError::NotFound) + )); + assert_eq!(controllers[0].usage().retained_preparations, 0); + } else { + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Transport(TransportError::Identity) + )); + assert_eq!(controllers[0].usage().retained_preparations, 1); + } + assert!(!controllers[0].usage().closing); + let dispatch = CellGroupDispatcher::new( + &f.cluster.domain, + f.cluster.directory.clone(), + class.clone(), + keys(190)?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + registration::mutate(&f, &controllers[0], &dispatch, &class, &a, retry).await?; + for c in controllers { + c.close().await?; + } + drop(dispatch); + f.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index bf0eedd00..20d50d85a 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -726,3 +726,9 @@ pub(super) use sql::{ group_quarantine_retains_producer_and_serves_healthy_group, shared_engine_sync_failure_still_stops_all_groups, }; + +pub(super) use sql::{ + provisioning_coalesces_and_serves, provisioning_joins_cancelled_close, + provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, + provisioning_retries_retained_owner, +}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index b9ef5a66e..1018b1626 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -583,3 +583,11 @@ pub(crate) use quarantine::{ group_quarantine_retains_producer_and_serves_healthy_group, shared_engine_sync_failure_still_stops_all_groups, }; + +#[path = "cell_group_provision.rs"] +mod provision; +pub(crate) use provision::{ + provisioning_coalesces_and_serves, provisioning_joins_cancelled_close, + provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, + provisioning_retries_retained_owner, +}; diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 928d3e671..1f1018215 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 78 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index f3e989003..1d9d52a39 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -528,7 +528,12 @@ "crates/cellar/durability/raft/cell_groups/controller/registration.rs", "crates/cellar/tests/support/cell_group_registration.rs", "crates/cellar/durability/raft/cell_groups/controller/failure.rs", - "crates/cellar/tests/support/cell_group_quarantine.rs" + "crates/cellar/tests/support/cell_group_quarantine.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/config.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs", + "crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs", + "crates/cellar/tests/support/cell_group_provision.rs" ], "test_target": "native_cell_groups_execution", "steps": [ @@ -714,6 +719,16 @@ "group_quarantine_retains_producer_and_serves_healthy_group", "shared_engine_sync_failure_still_stops_all_groups" ] + }, + { + "action": "Provision already selected initial-member ranges through the controller with immutable explicit filesystem/image/peer settings. Bound preparation bytes, parallelism, resident failed preparations and response waiters independently from commands. Coalesce by exact physical group and directory owner; preserve recovered page owners across attachment retries. Register appliers before polling attachment and join entered catalog/filesystem work before physical shutdown. This explicit local API does not allocate ranges, orchestrate followers, authorize managed fleet admission or wire automatic request-driven/public SDK startup. Complete those integrations separately.", + "tests": [ + "provisioning_coalesces_and_serves", + "provisioning_retries_retained_owner", + "provisioning_joins_cancelled_close", + "provisioning_recovers_selected_roots", + "provisioning_refusals_preserve_healthy_group" + ] } ], "test_cases": [ @@ -1465,6 +1480,56 @@ "I05", "I12" ] + }, + { + "name": "provisioning_coalesces_and_serves", + "oracle": "Starting from selected reserved ranges with no genesis or page owners, production provisioning materializes birth, verifies configured members, bootstraps all three engines, recovers roots and attaches real TLS peers. Same-owner concurrent calls share one preparation; dropped waiters cannot cancel it. Waiter, pending and active group limits refuse excess requests; conflicting assignments refuse. A single command slot remains usable. Both image modes then enroll, mutate and return the exact retry without another cell mutation.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_retries_retained_owner", + "oracle": "A catalog failure before recovery releases preparation and group capacity. A real filesystem hook injects a catalog failure after physical recovery but before attachment; the exact page owner remains locked and charged. Retrying the same assignment succeeds without reopening or syncing the page store again. Both image modes retain the same selected-root owner.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_joins_cancelled_close", + "oracle": "Hold a real manifest directory sync inside execution-domain recovery, cancel close and drop the response. Preparation admission and the runtime engine claim remain held; a replacement runtime refuses. Release the physical operation, then observe joined close and released preparation capacity in both image modes.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_recovers_selected_roots", + "oracle": "After actual controller enrollment and mutation through provisioned groups, join all controllers and reopen all three voter engines under a new process generation. Empty controllers recover using the same production provisioner. Removing the engine-selected manifest page refuses recovery and leaves the selected marker unchanged; restoring its exact bytes permits recovery. Attachment alone refuses leader authority; fresh quorum then returns the prior exact receipt and a linearizable query verifies value 8 and cell sequence 2. Run both image modes. Image-store handles remain in process; this is not independent-process or power-loss qualification.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] + }, + { + "name": "provisioning_refusals_preserve_healthy_group", + "oracle": "In both image modes, configured peers missing an immutable birth member refuse before genesis and free preparation capacity. A configured but wrong local endpoint fails physical registration after recovery, retains the prepared owner and its budget, and leaves healthy groups serving actual controller commands. Neither refusal grants membership or authority. Joined close releases all owners.", + "invariants": [ + "I01", + "I03", + "I05", + "I12" + ] } ], "invariants": [ From 69e41e865c267dff1dec5bce6f2b3d0966be09db Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 12:46:22 -0700 Subject: [PATCH 116/568] fix: bound provisioning cache paths and reuse fault fixtures --- .../raft/cell_groups/controller/mod.rs | 2 +- .../controller/provision/config.rs | 11 +++++++++ .../raft/cell_groups/controller/requests.rs | 3 ++- .../tests/support/cell_group_provision.rs | 24 +++++++++++++++---- .../tests/support/cell_group_registration.rs | 2 +- 5 files changed, 35 insertions(+), 7 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index fddf0fb6e..507edcbf6 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -170,7 +170,7 @@ impl CellGroupController { /// Consume the exclusive event receiver after validating exact engine and /// runtime ownership. Initial groups must already be attached. Later groups - /// enter through register(); unknown events still fail closed. + /// enter through register() or provision(); unknown events still fail closed. pub async fn start( runtime: CellGroupRuntime, events: CellGroupRuntimeEvents, diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs index f92529f6e..7c87666af 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs @@ -47,6 +47,17 @@ impl Config { settings.manifest.validate().map_err(|_| invalid())?; settings.entries.validate().map_err(|_| invalid())?; settings.images.validate().map_err(apply_error)?; + if let CellGroupApplyImages::Extents { + cache_directory, .. + } = &mut settings.images + { + if !cache_directory.is_absolute() || cache_directory.as_os_str().len() > 4096 { + return Err(invalid()); + } + *cache_directory = std::mem::take(cache_directory) + .into_boxed_path() + .into_path_buf(); + } if !settings.directory.is_absolute() || settings.directory.as_os_str().len() > 4096 || settings diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index ae06071de..ea8fb60b0 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -1,6 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! All mutations share the same node queue, admission and physical shutdown owner. +//! One node queue and physical shutdown owner. Provisioning has independent +//! admission so command ownership need not recursively acquire its own budget. use super::*; pub(super) enum Request { diff --git a/crates/cellar/tests/support/cell_group_provision.rs b/crates/cellar/tests/support/cell_group_provision.rs index efb218e9a..dca961c76 100644 --- a/crates/cellar/tests/support/cell_group_provision.rs +++ b/crates/cellar/tests/support/cell_group_provision.rs @@ -1,13 +1,11 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Explicit assigned-range provisioning over real engines, files and TLS. +use super::apply_faults::physical; +use super::registration::catalog_fault; use super::remote_queries::{keys, wait_for}; use super::*; use std::sync::atomic::{AtomicBool, Ordering}; -#[path = "raft_catalog_fault.rs"] -#[allow(dead_code)] -mod catalog_fault; -use super::apply_faults::physical; async fn assignment( f: &RuntimeCluster, @@ -148,6 +146,24 @@ pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { let (directory, route, store) = assignment(&f, 31).await?; let (foreign, _, _) = assignment(&f, 31).await?; let (other, other_route, _) = assignment(&f, 41).await?; + let mut invalid = settings(&f, &a, 0).await?; + let directory_before = invalid.directory.clone(); + match &mut invalid.images { + CellGroupApplyImages::Extents { + cache_directory, .. + } => *cache_directory = "relative-cache".into(), + CellGroupApplyImages::Materialized { .. } => { + invalid.directory = "relative-pages".into() + } + } + assert!(matches!( + controllers[0].configure_provisioning(invalid), + Err(CellGroupControllerError::Invalid(_)) + )); + assert!( + !directory_before.exists(), + "invalid configuration performs no filesystem work" + ); for (node, c) in controllers.iter().enumerate() { c.configure_provisioning(settings(&f, &a, node).await?)?; } diff --git a/crates/cellar/tests/support/cell_group_registration.rs b/crates/cellar/tests/support/cell_group_registration.rs index c9d182366..9e0cda5d9 100644 --- a/crates/cellar/tests/support/cell_group_registration.rs +++ b/crates/cellar/tests/support/cell_group_registration.rs @@ -6,7 +6,7 @@ use super::*; use std::sync::atomic::Ordering; #[path = "raft_catalog_fault.rs"] #[allow(dead_code)] -mod catalog_fault; +pub(super) mod catalog_fault; pub(super) struct Prepared { directory: RangeDirectory, From 2e8dd8e89d3de2337585389e26602157e31f3c84 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 13:23:21 -0700 Subject: [PATCH 117/568] docs: complete CG-04 scoped execution qualification --- docs/design/multi-raft/cell-groups/README.md | 31 +++++-- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 48 +++++++++-- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 86 ++++++++++++++++++- docs/design/multi-raft/cell-groups/tasks.json | 12 +-- 6 files changed, 159 insertions(+), 24 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index f419638d4..c66dbe140 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -3,8 +3,8 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 contracts, CG-02 storage and CG-03 routing/admission preparation have -passed their local evidence gates. CG-04 shared execution is in progress. +CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and +CG-04 shared execution have passed their scoped local evidence gates. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -30,7 +30,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-04` emits a complete executor brief, source excerpts, + eligibility; `show CG-05` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -42,16 +42,18 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package This checkout retains evidence under the ignored `.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/`, `.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` and -`.cellar/cell-groups-evidence/CG-03-routing-final/` directories. A fresh checkout must -receive all three retained directories at the same relative paths before validating -the DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. +`.cellar/cell-groups-evidence/CG-03-routing-final/` directories. CG-04 adds +`.cellar/cell-groups-evidence/CG-04-qualified/` and its supporting +`.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` directory. A fresh +checkout must receive these retained directories at the same relative paths +before validating the four DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. The design, task briefs and validator unit tests are committed in this package. ```sh python3 docs/design/multi-raft/cell-groups/plan.py validate python3 docs/design/multi-raft/cell-groups/plan.py preflight python3 docs/design/multi-raft/cell-groups/plan.py ready -python3 docs/design/multi-raft/cell-groups/plan.py show CG-04 +python3 docs/design/multi-raft/cell-groups/plan.py show CG-05 python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' ``` @@ -249,6 +251,21 @@ The partial gate passes 233 Rust cases and twelve commands; transfer `.cellar/cell-groups-evidence/CG-04-group-quarantine-final/` (283 source inputs, 4 executables). Bounded request-driven provisioning is next; CG-04 remains open. +At `69e41e86`, the controller provisions selected initial-member ranges with bounded +coalesced recovery, retained prepared owners and joined shutdown. Actual commands +and all-voter joined engine/page-store recovery pass in both image modes. The +partial gate passes 238 Rust cases and twelve commands; transfer +`.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` (288 source inputs, +4 executables). The full scoped CG-04 gate below supplements this incremental +checkpoint. Production integration and shared RustFS/load remain unqualified. + +CG-04 is now DONE at `69e41e86` after all 83 declared cases pass individually and +the plan runner validates every required command. Transfer +`.cellar/cell-groups-evidence/CG-04-qualified/` and its supporting evidence +(98 distinct command records, 288 source inputs). CG-05 complete-group +snapshots and membership recovery is next. Public SDK/fleet cutover belongs to +CG-07; complete release qualification still requires CG-16. + The implementation executor should deliver code, updated component contracts and the required retained evidence for each task on the active PR. The final handoff requires CG-16 and the release gate, including the real RustFS and independent diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 6839898f5..cb7021b6f 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "41121a7d08f283db6a83982ebfcba2feced036ca", + "source_revision": "69e41e865c267dff1dec5bce6f2b3d0966be09db", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 1af7da135..6465fda4e 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,41 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The newest reviewed source baseline is `41121a7d`, adding retained group quarantine. +CG-04 is now DONE at `69e41e86` after its complete scoped acceptance gate: +83 individual exact-case passes, all required commands, and the supporting 238-case +regression gate. `.cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json` +retains 98 distinct command records and 288 source hashes. The plan runner +validates the evidence and now selects CG-05 as the next eligible task. Preserve +all acceptance cases; the task-boundary audit removed no requirements or checks. + +Start CG-05 with complete-group snapshots and verified atomic installation, then +learner/membership recovery. CG-06 follows with archival reclamation. Automatic +production request/voter startup and public SDK/fleet binding belong to CG-07; +effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track +these remaining capabilities in their owning tasks. +Shared RustFS, independent-process/power-loss and load remain unqualified. + +The following records describe the preceding incremental checkpoints. + +The newest reviewed source baseline is `69e41e86`, adding bounded explicit local +provisioning from already selected initial-member assignments. The controller owns +birth/bootstrap, blocking root recovery and attachment. Exact directory/group +requests coalesce; waiter admission is independent from commands. Failed recovered +owners remain charged and retry without reopening pinned page stores. Joined close +waits for physical work. Registration refusals preserve healthy groups, while +shared engine uncertainty remains node-wide. Five new native cases cover both +image modes, actual application receipts, ownership/capacity refusals, cancelled +physical close and all-voter joined engine/page-store recovery. A missing selected +page refuses without reset; restored bytes plus fresh quorum recover the receipt. +The record `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/checkpoint.json` +binds 238 Rust cases, twelve commands, 288 source inputs and 4 executables. +Transfer this ignored evidence with its predecessors. This incremental checkpoint +preceded the complete CG-04 gate above. Automatic production dispatch/voter +startup and managed fleet authorization remain in CG-07. Membership recovery, +public cutover, shared RustFS, independent-process/power-loss and load remain +unqualified under their later task gates. + +The preceding reviewed source baseline is `41121a7d`, adding retained group quarantine. Group-local failure revokes admission and runtime execution while retaining group capacity, apply tickets and ambiguous producers. Captured leader authority refuses; late persistence cannot restore it. Shared engine failures and lost shared SQL @@ -463,23 +497,23 @@ export PATH="$HOME/.cargo/bin:$PATH" export CARGO_TARGET_DIR="$HOME/Workspace/crabbuild-target/cellar-3617-cell-groups" python3 docs/design/multi-raft/cell-groups/plan.py validate python3 docs/design/multi-raft/cell-groups/plan.py preflight -python3 docs/design/multi-raft/cell-groups/plan.py show CG-04 +python3 docs/design/multi-raft/cell-groups/plan.py show CG-05 python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' ``` Choose a different private target directory for a different checkout. The plan runner validates and prints commands; it does not execute Rust tests or provision -infrastructure. `ready` currently returns no TODO task because CG-04 is already -IN_PROGRESS and its successors depend on it. Resume CG-04 rather than resetting -statuses. Global preflight rejects any further unreviewed source drift; record +infrastructure. `ready` now returns CG-05 because CG-01 through CG-04 have passed +their scoped gates. Start CG-05 under its declared requirements. Global preflight +rejects any further unreviewed source drift; record a reviewed successor source after reconciling each implementation change. -The three DONE records require their retained evidence directories listed in +The four DONE records require their retained evidence directories listed in [README](README.md). Transfer those artifacts with the source when validating on another machine. Missing evidence must refuse; do not replace it with a pass flag or reset completed tasks merely to bypass validation. -The latest fully passing partial CG-04 record is +An earlier passing partial CG-04 record is `.cellar/cell-groups-evidence/CG-04-apply-final/checkpoint.json` at `c9100ec7`: 254 selected Rust cases, fifteen successful commands, 202 source inputs and 12 copied executables. Transfer the entire directory and its failed predecessor diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 1f1018215..f67588c4c 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -15,7 +15,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-01 | Separate logical cell history from physical group consensus | None | DONE | 4 | | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | -| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | IN_PROGRESS | 83 | +| CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index ace48575e..5e20b1aa2 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -5,11 +5,93 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). -CG-01 through CG-03 have passed their scoped gates; -CG-04 is IN_PROGRESS and CG-05 through CG-16 remain TODO. Intermediate contracts +CG-01 through CG-04 have passed their scoped gates; +CG-05 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-04 scoped execution gate complete + +CG-04 is DONE at `69e41e865c267dff1dec5bce6f2b3d0966be09db`. All 83 declared acceptance cases pass +individually, with the required test inventory, all-target check, formatting, +layout and all-target Clippy commands. The evidence record +`.cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json` validates through the +plan runner and retains 98 distinct command records, 288 source hashes +and 4 copied executables, including the supporting 238-case regression gate. +Transfer it together with the local-provisioning and predecessor evidence. + +The completed scope is native shared-group execution: one Raft owner per group +replica, independent cell SQLite histories, bounded node services, durable apply, +cell receipts and reads, authenticated dispatch, response-independent ownership, +group failure isolation and explicit local provisioning/recovery. All acceptance +cases remain intact. Completion does not claim production SDK/fleet cutover or +independent-process, power-loss, shared RustFS or performance qualification. + +The critical next task is CG-05: complete-group snapshots, atomic installation, +learner catch-up and membership recovery. It unlocks CG-06 archive/reclamation and +then effect/fleet integration. Automatic production request/voter orchestration +and public SDK binding belong to CG-07; final legacy removal belongs to CG-16. +Keep those dependencies explicit in subsequent execution. + +## CG-04 bounded local provisioning + +Qualified source: `69e41e865c267dff1dec5bce6f2b3d0966be09db`. + +`CellGroupController::configure_provisioning` installs explicit immutable node +filesystem, image-store, manifest, peer and preparation limits. `provision` +accepts an already selected range assignment; it materializes the immutable birth, +checks configured initial members, acquires permanent member/engine claims, +recovers the complete selected root on the owning blocking execution domain and +attaches through the existing runtime and exclusive transport registration. It +neither assigns ranges nor creates a Raft group for an individual cell. Managed +fleet bootstrap still refuses pending CG-07 authorization. + +Exact directory-owner/group requests share one preparation. Pending and active +groups share the controller capacity; recovery items, bytes and parallelism have +explicit bounds. Response waiters have separate admission, including queued and +unconsumed replies, so preparation does not recursively consume command permits. +Active stores remain bounded by the configured per-group limits and maximum +roster size; image stores keep their independent quotas. Settings reject invalid +physical identities, duplicate endpoints and unbounded paths before I/O. + +Failure before recovery frees its preparation and group reservations. After +recovery, an attachment refusal retains the exact page/root/applier owners and +their preparation budget. A retry revalidates catalog/bootstrap and reuses these +owners rather than reopening an engine-pinned directory. The applier enters the +event roster before attachment can emit events. Registration identity/capacity +refusals preserve healthy groups; closed services or uncertain shared-engine +failures remain node-wide. Shutdown joins admitted catalog/filesystem/attachment +work before transport, SQL and runtime shutdown, even when waiters disconnect. +Success means locally attached; serving still requires fresh current-term quorum. + +Five native cases cover both image modes with real engines, filesystem operations +and TLS. They exercise same-group coalescing, dropped waiters, bounds and foreign +bindings, single-command-slot operation, actual enrollment/mutation/exact retry, +pre-recovery catalog failure and post-recovery attachment refusal with page-owner +reuse. A blocked physical directory sync retains engine ownership across cancelled +close. Missing initial peers refuse before genesis; a wrong configured local +endpoint retains preparation while the existing group still executes a command. +All voter engines and manifest stores reopen into empty controllers; removing a +selected page refuses without resetting the marker, restoration permits recovery, +and fresh quorum returns the original receipt and SQLite value. Image-store +handles stay in the process, so this is not independent-process/power-loss evidence. + +All 238 selected Rust cases and twelve gate commands pass, including all-target +Clippy, formatting, layout, dependency isolation, docs, source preflight and 37 +validator cases. `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/checkpoint.json` +binds 288 source inputs and 4 copied executables. Diagnostic failures and +reruns are retained separately; transfer this ignored directory with predecessors. + +The provisioning checkpoint below the full gate is scoped to explicit local +startup. CG-07 integrates automatic request-driven provisioning and assigned-voter +startup, preserving original operation identity, +deadline and captured inputs while using this bounded owner from local and remote +dispatch. Read/receipt requests must never implicitly enroll missing cells. +Configuration/membership lifecycle, managed fleet authorization and public +host/SDK integration remain unfinished. Later CG tasks still own snapshots, +archive/reclamation, effects, topology, hibernation, fenced recovery, load and +legacy removal. Shared-group RustFS and load remain unqualified. + ## CG-04 retained group quarantine Qualified source: `41121a7d08f283db6a83982ebfcba2feced036ca`. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 1d9d52a39..7ef7ab30a 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -420,7 +420,7 @@ { "id": "CG-04", "title": "Execute many independent SQLite cells through one real Raft group", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -705,7 +705,7 @@ ] }, { - "action": "Replace the fixed controller roster with bounded dynamic registration, including empty-controller startup. Reserve pending plus active group slots and shared command item/byte admission before transfer. Bind exact engine/history/schema, selected root and applier page owner; install an exclusively owned physical route without adopting a prior registration. Register the applier before polling attachment so early consensus/apply events remain owned. Revalidate the bootstrap catalog through the existing runtime attach path; expose application roster membership only after success, with fresh quorum still required. Retain attachment through response cancellation and join catalog/runtime attachment before closing physical dependencies. Release definite failures before responding; unexpected competing runtime owners or runtime closure keep the conservative node shutdown boundary. Configuration changes, per-group failure isolation and request-driven lazy birth/public SDK binding remain incomplete. Run the full transport and shared runtime suites.", + "action": "Replace the fixed controller roster with bounded dynamic registration, including empty-controller startup. Reserve pending plus active group slots and shared command item/byte admission before transfer. Bind exact engine/history/schema, selected root and applier page owner; install an exclusively owned physical route without adopting a prior registration. Register the applier before polling attachment so early consensus/apply events remain owned. Revalidate the bootstrap catalog through the existing runtime attach path; expose application roster membership only after success, with fresh quorum still required. Retain attachment through response cancellation and join catalog/runtime attachment before closing physical dependencies. Release definite failures before responding; unexpected competing runtime owners or runtime closure keep the conservative node shutdown boundary. Complete-group membership changes follow CG-05; production request/voter orchestration and SDK/fleet binding follow CG-07. Run the full transport and shared runtime suites.", "tests": [ "dynamic_registration_preserves_live_groups_and_disconnected_waiters", "dynamic_registration_refuses_foreign_owners_and_retries_failed_catalog", @@ -714,14 +714,14 @@ ] }, { - "action": "Isolate group-local controller apply/runtime/ambiguous command failures through a bounded retained quarantine lifecycle. Reserve a control lane to revoke execution while apply is waiting, refuse new group admission, preserve captured producer/apply owners until joined shutdown, and prevent late persistence completion from restoring authority. Keep failed group capacity charged. Shared engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Expose original group failure and quarantine counts; no in-place replacement or automatic callback replay. Qualify healthy-group SQL progress and node-wide storage failure separately. Request-driven provisioning, configuration lifecycle and public cutover remain unfinished.", + "action": "Isolate group-local controller apply/runtime/ambiguous command failures through a bounded retained quarantine lifecycle. Reserve a control lane to revoke execution while apply is waiting, refuse new group admission, preserve captured producer/apply owners until joined shutdown, and prevent late persistence completion from restoring authority. Keep failed group capacity charged. Shared engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Expose original group failure and quarantine counts; no in-place replacement or automatic callback replay. Qualify healthy-group SQL progress and node-wide storage failure separately. Explicit local provisioning uses the lifecycle below; production configuration and public cutover are integrated in CG-07 and CG-16.", "tests": [ "group_quarantine_retains_producer_and_serves_healthy_group", "shared_engine_sync_failure_still_stops_all_groups" ] }, { - "action": "Provision already selected initial-member ranges through the controller with immutable explicit filesystem/image/peer settings. Bound preparation bytes, parallelism, resident failed preparations and response waiters independently from commands. Coalesce by exact physical group and directory owner; preserve recovered page owners across attachment retries. Register appliers before polling attachment and join entered catalog/filesystem work before physical shutdown. This explicit local API does not allocate ranges, orchestrate followers, authorize managed fleet admission or wire automatic request-driven/public SDK startup. Complete those integrations separately.", + "action": "Provision already selected initial-member ranges through the controller with immutable explicit filesystem/image/peer settings. Bound preparation bytes, parallelism, resident failed preparations and response waiters independently from commands. Coalesce by exact physical group and directory owner; preserve recovered page owners across attachment retries. Register appliers before polling attachment and join entered catalog/filesystem work before physical shutdown. This explicit local API does not allocate ranges, orchestrate followers, authorize managed fleet admission or wire automatic request-driven/public SDK startup. CG-07 integrates those production paths after its snapshot, archive and effect dependencies.", "tests": [ "provisioning_coalesces_and_serves", "provisioning_retries_retained_owner", @@ -1540,7 +1540,9 @@ "I05", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From b15da415c9c9c0663274d0aedd520feb37d90d2d Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 13:45:11 -0700 Subject: [PATCH 118/568] feat: capture and stream complete cell-group checkpoint sources --- .../durability/raft/cell_groups/checkpoint.rs | 412 ++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 6 + .../durability/raft/cell_groups/pins.rs | 26 + .../tests/native_cell_groups_execution.rs | 34 ++ .../tests/support/cell_group_checkpoint.rs | 523 ++++++++++++++++++ docs/design/multi-raft/cell-groups/handoff.md | 8 +- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 34 +- .../multi-raft/cell-groups/protocols.md | 17 + docs/design/multi-raft/cell-groups/tasks.json | 6 +- 10 files changed, 1062 insertions(+), 6 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint.rs create mode 100644 crates/cellar/tests/support/cell_group_checkpoint.rs diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/checkpoint.rs new file mode 100644 index 000000000..a48481b00 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint.rs @@ -0,0 +1,412 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete immutable source cut, independent of snapshot installation and +//! voter admission. Run capture and streaming on an admitted blocking owner. + +use super::{page_wire, CellImagePin, GroupManifest, ManifestLimits, PreparedGroupManifest}; +use crate::raft::{ + budget::{Budget, Permit}, + cache::GroupLog, + member_state::GroupMembership, +}; +use cellar_logic::cell_group::{CellId, GroupHistory}; +use cellar_ltx::internal::{ExtentId, EXTENT_BYTES}; +use serde::Serialize; +use sha2::{Digest, Sha256}; +use std::sync::Arc; + +const HEADER_LIMIT: usize = 64 * 1024; + +// Covers one verified extent, page/extent metadata encoding, serialization +// temporaries, one flat-image read, membership, traversal stack and fixed owners. +const STREAM_WORKSPACE: usize = EXTENT_BYTES as usize + 1024 * 1024; + +#[derive(Debug)] +pub enum CellGroupCheckpointError { + Invalid(&'static str), + Limit, + Busy, + Manifest(super::ManifestError), + Log(crate::raft::storage::LogError), + Image(cellar_ltx::generation::GenerationError), + Extent(cellar_ltx::internal::StoreError), + Sink(std::io::Error), +} + +impl std::fmt::Display for CellGroupCheckpointError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group checkpoint: {self:?}") + } +} + +impl std::error::Error for CellGroupCheckpointError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Manifest(error) => Some(error), + Self::Log(error) => Some(error), + Self::Image(error) => Some(error), + Self::Extent(error) => Some(error), + Self::Sink(error) => Some(error), + _ => None, + } + } +} + +impl From for CellGroupCheckpointError { + fn from(error: super::ManifestError) -> Self { + Self::Manifest(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: crate::raft::storage::LogError) -> Self { + Self::Log(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: cellar_ltx::generation::GenerationError) -> Self { + Self::Image(error) + } +} + +impl From for CellGroupCheckpointError { + fn from(error: cellar_ltx::internal::StoreError) -> Self { + Self::Extent(error) + } +} + +type Result = std::result::Result; + +/// One pool must be shared by the node's checkpoint producers. Byte accounting +/// conservatively charges pinned metadata to each source as well as its buffer; +/// physical stores separately bound disk retention. No full-image spool is made. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupCheckpointLimits { + pub max_sources: usize, + pub max_retained_bytes: usize, + pub max_cells: u32, + pub max_transfer_bytes: u64, + pub max_chunk_bytes: usize, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupCheckpointUsage { + pub sources: usize, + pub retained_bytes: usize, +} + +#[derive(Clone)] +pub struct CellGroupCheckpointPool { + budget: Arc, + limits: CellGroupCheckpointLimits, +} + +impl CellGroupCheckpointPool { + pub fn new(limits: CellGroupCheckpointLimits) -> Result { + if !(1..=4096).contains(&limits.max_sources) + || limits.max_retained_bytes < STREAM_WORKSPACE + || !(1..=1_000_000).contains(&limits.max_cells) + || limits.max_transfer_bytes < HEADER_LIMIT as u64 + || !(1..=64 * 1024).contains(&limits.max_chunk_bytes) + { + return Err(CellGroupCheckpointError::Limit); + } + Ok(Self { + budget: Budget::new(limits.max_retained_bytes, limits.max_sources), + limits, + }) + } + + pub fn usage(&self) -> CellGroupCheckpointUsage { + let (retained_bytes, sources) = self.budget.usage(); + CellGroupCheckpointUsage { + sources, + retained_bytes, + } + } + + /// Select only the exact engine-published complete root. Membership and + /// configuration are copied under the same cache read lock as that check. + /// A prospective, stale or foreign preparation is not a checkpoint source. + /// O(cells) metadata admission runs off the consensus loop, without image I/O. + pub fn capture( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + ) -> Result { + if !Arc::ptr_eq(&root.engine_owner, &group.owner) + || !root + .marker() + .matches_group(&group.identity().scope, group.birth_digest) + || root.pinned_images() != root.marker().root().cells() as usize + || root.pinned_images() > self.limits.max_cells as usize + { + return Err(CellGroupCheckpointError::Invalid("source root/owner")); + } + let mut retained = STREAM_WORKSPACE + .checked_add(root.reserved_bytes()) + .ok_or(CellGroupCheckpointError::Limit)?; + let mut transfer = HEADER_LIMIT as u64; + root.images.visit(&mut |_, leaf, pin, _, _| -> Result<()> { + // Account even shared extent trees and inventory identities at a + // conservative per-extent cost. Payloads remain in their stores. + let extra = match pin { + CellImagePin::Extents(pin) => { + let count = pin.manifest().geometry().extent_count() as usize; + let encoded = pin.reference().encoded_length() as usize; + transfer = transfer + .checked_add(encoded as u64) + .ok_or(CellGroupCheckpointError::Limit)?; + count + .checked_mul(1024) + .and_then(|n| n.checked_add(encoded)) + .ok_or(CellGroupCheckpointError::Limit)? + } + _ => 1024, + }; + retained = retained + .checked_add(extra) + .ok_or(CellGroupCheckpointError::Limit)?; + transfer = transfer + .checked_add(page_wire::MAX_PAGE_BYTES as u64) + .and_then(|n| n.checked_add(leaf.geometry.length)) + .ok_or(CellGroupCheckpointError::Limit)?; + Ok(()) + })?; + if transfer > self.limits.max_transfer_bytes { + return Err(CellGroupCheckpointError::Limit); + } + let reservation = self + .budget + .acquire(retained) + .ok_or(CellGroupCheckpointError::Busy)?; + let header = { + let cache = group.read()?; + let marker = root.marker(); + if cache.group_manifest.as_ref() != Some(marker) + || cache.applied.position != marker.position() + || cache.applied.image.is_some() + || marker.position().index.get() > cache.hard_state.commit + || cache.term(marker.position().index.get()) != Some(marker.position().term.get()) + || marker.membership_revision() != cache.membership.revision().get() + || cache + .manifest_pins + .as_ref() + .is_some_and(|pins| pins.root() != marker.root()) + { + return Err(CellGroupCheckpointError::Invalid("source durable cut")); + } + cache + .membership + .validate(&cache.conf_state, marker.position().index.get())?; + Header { + domain: "cellar-cell-group-checkpoint-1", + root: marker.clone(), + conf: crate::raft::manifest::encode_conf(&cache.conf_state) + .map_err(|_| CellGroupCheckpointError::Invalid("configuration encoding"))?, + membership: cache.membership.clone(), + // Shared owners currently refuse all topology/park entries. + // Their owning tasks must extend the durable root and this + // binding together before admitting such transitions. + transition: "active", + } + }; + let encoded = serde_json::to_vec(&header) + .map_err(|_| CellGroupCheckpointError::Invalid("header encoding"))?; + if encoded.len() > HEADER_LIMIT { + return Err(CellGroupCheckpointError::Limit); + } + Ok(CellGroupCheckpoint { + group: group.clone(), + root: root.clone(), + header: encoded, + limits: self.limits, + started: false, + _reservation: reservation, + }) + } +} + +#[derive(Serialize)] +struct Header { + domain: &'static str, + root: GroupManifest, + conf: Vec, + membership: GroupMembership, + transition: &'static str, +} + +/// Borrowed part identity. Each manifest page carries the full cell leaf, +/// including empty enrollment, version, receipt-image commitment and effects. +/// Extents are emitted in logical order, including repeated content identities; +/// a receiver may reuse verified objects but must check every manifest binding. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellGroupCheckpointPart<'a> { + Header, + Page([u8; 32]), + Materialized(&'a CellId), + ExtentManifest(&'a CellId), + Extent { + cell: &'a CellId, + ordinal: u32, + id: ExtentId, + }, +} + +pub struct CellGroupCheckpointChunk<'a> { + pub part: CellGroupCheckpointPart<'a>, + pub offset: u64, + pub length: u64, + pub bytes: &'a [u8], +} + +/// Non-cloneable admitted source. A dropped waiter must leave this value on its +/// joined physical worker until streaming finishes. Borrows prevent parallel +/// streams or retaining a chunk without copying it into separately charged work. +/// Neither this source nor a successful stream grants install/vote authority. +pub struct CellGroupCheckpoint { + group: GroupLog, + root: PreparedGroupManifest, + header: Vec, + limits: CellGroupCheckpointLimits, + started: bool, + _reservation: Permit, +} + +impl CellGroupCheckpoint { + pub fn marker(&self) -> &GroupManifest { + self.root.marker() + } + + /// A one-shot stream. On error, previously emitted bytes are incomplete and + /// must remain unselected on the receiver. Flat-image digest verification + /// completes only at EOF; extent reads verify their full content first. + /// No engine/storage mutex is held while invoking the sink. + pub fn stream( + &mut self, + mut sink: impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<()>, + ) -> Result<()> { + if self.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + self.started = true; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Header, + &self.header, + &mut sink, + )?; + // Reconstruct canonical pages from the immutable verified closure. + // Exact page hashes must match; current mutable page-store state is + // never consulted and cannot substitute a later cell at this cut. + self.root + .images + .visit(&mut |reference, leaf, pin, left, right| -> Result<()> { + let page = page_wire::Page { + leaf: leaf.clone(), + left, + right, + }; + let bytes = page_wire::encode( + &page, + self.root.marker().route(), + ManifestLimits::default(), + )?; + if page_wire::digest(&bytes) != reference.digest { + return Err(CellGroupCheckpointError::Invalid( + "captured page commitment", + )); + } + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Page(reference.digest), + &bytes, + &mut sink, + )?; + drop(bytes); + let cell = &leaf.version.history.cell; + match pin { + CellImagePin::Empty => {} + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut offset = 0; + let mut digest = Sha256::new(); + while offset < reader.length() { + self.group.cache_revision()?; + let size = (reader.length() - offset) + .min(self.limits.max_chunk_bytes as u64) + as usize; + let bytes = reader.read_exact_at(offset, size)?; + digest.update(&bytes); + sink(CellGroupCheckpointChunk { + part: CellGroupCheckpointPart::Materialized(cell), + offset, + length: reader.length(), + bytes: &bytes, + })?; + offset += size as u64; + } + if <[u8; 32]>::from(digest.finalize()) != pin.state().digest { + return Err(CellGroupCheckpointError::Invalid( + "materialized image digest", + )); + } + } + CellImagePin::Extents(pin) => { + let manifest = pin + .manifest() + .encode() + .map_err(cellar_ltx::internal::StoreError::from)?; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::ExtentManifest(cell), + &manifest, + &mut sink, + )?; + drop(manifest); + for ordinal in 0..pin.manifest().geometry().extent_count() { + self.group.cache_revision()?; + let id = pin + .manifest() + .extent(ordinal) + .ok_or(CellGroupCheckpointError::Invalid("extent coverage"))?; + let bytes = pin.view().read_extent(id)?; + stream_bytes( + &self.group, + self.limits.max_chunk_bytes, + CellGroupCheckpointPart::Extent { cell, ordinal, id }, + &bytes, + &mut sink, + )?; + } + } + } + Ok(()) + })?; + self.group.cache_revision()?; + Ok(()) + } +} + +fn stream_bytes( + group: &GroupLog, + chunk_bytes: usize, + part: CellGroupCheckpointPart<'_>, + bytes: &[u8], + sink: &mut impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<()>, +) -> Result<()> { + for (n, chunk) in bytes.chunks(chunk_bytes).enumerate() { + group.cache_revision()?; + sink(CellGroupCheckpointChunk { + part, + offset: (n * chunk_bytes) as u64, + length: bytes.len() as u64, + bytes: chunk, + })?; + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index fafeb0c70..2e0226f7d 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -15,6 +15,12 @@ pub use crate::raft::transport::{ }; mod apply; +mod checkpoint; +pub use checkpoint::{ + CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, + CellGroupCheckpointLimits, CellGroupCheckpointPart, CellGroupCheckpointPool, + CellGroupCheckpointUsage, +}; mod controller; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs index a3941c6ca..5697a3a47 100644 --- a/crates/cellar/durability/raft/cell_groups/pins.rs +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -26,6 +26,32 @@ struct PinnedPage { } impl ManifestPins { + /// Walk the immutable closure without cloning its cells or allocating a + /// second inventory. Verified AVL height bounds the recursive stack to 32. + pub(super) fn visit( + &self, + visitor: &mut impl FnMut( + PageRef, + &CellManifestLeaf, + &CellImagePin, + Option, + Option, + ) -> Result<(), E>, + ) -> Result<(), E> { + if let Some(page) = &self.0 { + page.left.visit(visitor)?; + visitor( + page.reference, + &page.leaf, + &page.image, + page.left.root().0, + page.right.root().0, + )?; + page.right.visit(visitor)?; + } + Ok(()) + } + pub(crate) fn len(&self) -> usize { self.0.as_ref().map_or(0, |p| p.reference.cells as usize) } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 9d9609fc4..b0b5c6606 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -11,6 +11,29 @@ mod fault; #[path = "support/cell_group_runtime.rs"] mod runtime_fixture; +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_streams_one_cut_during_writes() -> Result<()> { + checkpoint_fixture::coherent().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_rejects_stale_foreign_and_unpublished_roots() -> Result<()> { + checkpoint_fixture::refusals().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_capture_retains_bounds_until_physical_release() -> Result<()> { + checkpoint_fixture::ownership().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_stream_detects_corruption_and_closed_engine() -> Result<()> { + checkpoint_fixture::corruption().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn controller_owned_commands_survive_sql_disconnect() -> Result<()> { runtime_fixture::owned_commands(0, 0).await @@ -625,6 +648,17 @@ fn prepare_cell_image( use cellar_ltx::internal::{ ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, }; + if image.txid == 0 { + anyhow::ensure!(base.is_none() && files.is_empty()); + return Ok(EnrollmentImage { + pin: CellImagePin::Empty, + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + effects: EffectSummary::default(), + }); + } let (pin, geometry) = match image.commitment { ImageCommitment::Sha256(_) => { let base = match base { diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs new file mode 100644 index 000000000..ca65f79a1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -0,0 +1,523 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Source-side checkpoint qualification over the existing real three-voter +//! fixture. This does not qualify a receiver, learner promotion or snapshot RPC. + +use super::*; +use cellar_ltx::internal::{ + ExtentId, ExtentManifest, ExtentRootReference, Geometry, ImageBinding, NodeExtentStore, + EXTENT_BYTES, +}; +use std::collections::BTreeMap; + +fn limits() -> CellGroupCheckpointLimits { + CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + } +} + +async fn populated(rooted: bool) -> Result<(Cluster, BTreeMap>)> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + let route = cluster.birth.route().clone(); + if rooted { + for n in 0..3 { + cluster.voters[n].extents = Some(NodeExtentStore::open( + cluster._root.path().join(format!("extents-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?); + } + } + let mut expected = BTreeMap::new(); + for name in ["alpha", "beta", "empty"] { + let cell = history(&route, name)?; + let bytes = if name == "empty" { + Vec::new() + } else { + // More than one verified extent; the two images also exceed one + // source workspace together. Source transfer never aggregates them. + images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(1); CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob(4202496));")? + }; + let encoded = if bytes.is_empty() { + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([99; 16])?, + }, + command: GroupCommand::Enroll { + cell: cell.clone(), + image: LtxImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: ImageCommitment::Sha256(Sha256::digest([]).into()), + }, + files: vec![], + }, + } + .encode(&route, GroupEntryLimits::default())? + } else { + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let encoded = request(&route, cell.clone(), 1, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + if rooted { + let commitment = extent_commitment(&route, &cell, cut.result, &bytes)?; + let GroupCommand::Enroll { image, .. } = &mut entry.command else { + unreachable!() + }; + image.commitment = commitment; + } + entry.encode(&route, GroupEntryLimits::default())? + }; + cluster.voters[0].owner.as_mut().unwrap().propose(encoded)?; + cluster.drive(false).await?; + expected.insert(name.to_owned(), bytes); + } + Ok((cluster, expected)) +} + +fn extent_commitment( + route: &GroupRoute, + cell: &CellHistory, + state: ImageState, + bytes: &[u8], +) -> Result { + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(cell), + schema: route.schema_digest, + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }, + Geometry::new(state.page_size, state.length)?, + &bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?, + )?; + Ok(ImageCommitment::Extents(ExtentRootReference::new( + manifest.root(), + manifest.encode()?.len() as u32, + )?)) +} + +async fn advance(cluster: &mut Cluster, name: &str, before: &[u8], rooted: bool) -> Result<()> { + let route = cluster.birth.route().clone(); + let cell = history(&route, name)?; + let after = images::image(Some(before), "UPDATE value SET n=n+1")?; + let cut = images::Cut::new(before, &after, 1, true, false); + let encoded = request(&route, cell.clone(), 2, &cut)?; + let mut entry = CellGroupEntry::decode(&encoded, &route, GroupEntryLimits::default())?; + if rooted { + let GroupCommand::Mutation(mutation) = &mut entry.command else { + unreachable!() + }; + mutation.base.commitment = extent_commitment(&route, &cell, cut.base, before)?; + mutation.result.commitment = extent_commitment(&route, &cell, cut.result, &after)?; + } + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(entry.encode(&route, GroupEntryLimits::default())?)?; + cluster.drive(false).await?; + Ok(()) +} + +#[derive(Default)] +struct Received { + header: Vec, + pages: BTreeMap<[u8; 32], Vec>, + images: BTreeMap>, + manifests: BTreeMap>, + extents: BTreeMap<(String, u32), (ExtentId, Vec)>, + largest_chunk: usize, +} + +impl Received { + fn append( + &mut self, + chunk: CellGroupCheckpointChunk<'_>, + ) -> std::result::Result<(), CellGroupCheckpointError> { + self.largest_chunk = self.largest_chunk.max(chunk.bytes.len()); + let output = match chunk.part { + CellGroupCheckpointPart::Header => &mut self.header, + CellGroupCheckpointPart::Page(key) => self.pages.entry(key).or_default(), + CellGroupCheckpointPart::Materialized(cell) => { + self.images.entry(cell.as_str().to_owned()).or_default() + } + CellGroupCheckpointPart::ExtentManifest(cell) => { + self.manifests.entry(cell.as_str().to_owned()).or_default() + } + CellGroupCheckpointPart::Extent { cell, ordinal, id } => { + let output = self + .extents + .entry((cell.as_str().to_owned(), ordinal)) + .or_insert_with(|| (id, vec![])); + assert_eq!(output.0, id); + &mut output.1 + } + }; + assert_eq!(chunk.offset, output.len() as u64); + assert!(chunk.offset + chunk.bytes.len() as u64 <= chunk.length); + output.extend_from_slice(chunk.bytes); + Ok(()) + } + + fn verify( + mut self, + marker: &GroupManifest, + leaves: &BTreeMap, + expected: &BTreeMap>, + rooted: bool, + ) -> Result<()> { + assert!(self.largest_chunk <= limits().max_chunk_bytes); + let header: serde_json::Value = serde_json::from_slice(&self.header)?; + assert_eq!(header["domain"], "cellar-cell-group-checkpoint-1"); + assert_eq!(header["transition"], "active"); + assert_eq!( + header["membership"]["revision"], + marker.membership_revision() + ); + assert_eq!(header["root"], serde_json::to_value(marker)?); + let conf = header["conf"] + .as_array() + .context("configuration")? + .iter() + .map(|n| n.as_u64().unwrap() as u8) + .collect::>(); + let conf: raft::prelude::ConfState = protobuf::Message::parse_from_bytes(&conf)?; + assert_eq!(conf.voters, vec![1, 2, 3]); + assert_eq!(self.pages.len(), expected.len()); + assert!(self.pages.contains_key(&marker.root().digest().unwrap())); + for (digest, bytes) in &self.pages { + let mut hash = Sha256::new(); + hash.update(b"cellar-cell-group-manifest-page-1\0"); + hash.update(bytes); + assert_eq!(*digest, <[u8; 32]>::from(hash.finalize())); + } + for (name, wanted) in expected { + let leaf = &leaves[name]; + assert_eq!(leaf.version.sequence.get(), 1); + if wanted.is_empty() { + assert_eq!(leaf.version.image.txid, 0); + assert!(!self.images.contains_key(name)); + assert!(!self.manifests.contains_key(name)); + continue; + } + if rooted { + let CellImageCommitment::Extents { + root, + manifest_bytes, + } = leaf.version.image.commitment + else { + unreachable!() + }; + let bytes = self.manifests.remove(name).context("manifest")?; + assert_eq!(bytes.len(), manifest_bytes as usize); + let manifest = ExtentManifest::decode( + &bytes, + cellar_ltx::internal::ImageRoot::from_bytes(root), + )?; + let mut assembled = Vec::new(); + for n in 0..manifest.geometry().extent_count() { + let (id, bytes) = self.extents.remove(&(name.clone(), n)).context("extent")?; + assert_eq!(Some(id), manifest.extent(n)); + id.verify(&bytes)?; + assembled.extend(bytes); + } + assert_eq!(&assembled, wanted); + } else { + assert_eq!(self.images.remove(name).as_ref(), Some(wanted)); + } + } + assert!(self.images.is_empty() && self.extents.is_empty() && self.manifests.is_empty()); + Ok(()) + } +} + +pub(super) async fn coherent() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, expected) = populated(rooted).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let old_page = cluster._root.path().join("node-1/pages").join(format!( + "{}.page", + hex::encode(root.marker().root().digest().unwrap()) + )); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, &root)?; + let marker = root.marker().clone(); + let leaves = expected + .keys() + .map(|name| { + let cell = CellId::new(name.clone()).unwrap(); + (name.clone(), root.pin_cell(&cell).unwrap().0.clone()) + }) + .collect::>(); + drop(root); // Only the stream retains the old physical closure after apply. + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let worker = cluster.domain.blocking(move || -> Result<_> { + let mut entered = Some(entered); + let mut received = Received::default(); + source.stream(|chunk| { + if matches!(chunk.part, CellGroupCheckpointPart::Page(_)) { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + release_rx.recv().map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("released")) + })?; + } + } + received.append(chunk) + })?; + Ok((source, received)) + })?; + tokio::time::timeout(Duration::from_secs(10), entered_rx).await??; + // Real quorum publication continues while the transfer owner is held. + advance(&mut cluster, "alpha", &expected["alpha"], rooted).await?; + advance(&mut cluster, "beta", &expected["beta"], rooted).await?; + for voter in &mut cluster.voters { + let current = voter.owner.as_ref().unwrap().manifest(); + assert!(current.marker().position().index > marker.position().index); + assert_eq!( + current + .pin_cell(&CellId::new("alpha")?) + .unwrap() + .0 + .version + .sequence + .get(), + 2 + ); + voter.pages.as_mut().unwrap().reconcile_retained(&[])?; + voter.pages.as_mut().unwrap().reclaim(1024)?; + voter.images.retain_only(&[])?; + voter.images.reclaim(1024)?; + if let Some(store) = &voter.extents { + store.retain_only(&[])?; + store.reclaim(1024)?; + } + } + assert!(old_page.exists(), "live source must pin its manifest page"); + release.send(())?; + let (source, received) = worker.await??; + received.verify(&marker, &leaves, &expected, rooted)?; + assert_eq!(source.marker(), &marker); + assert_eq!(pool.usage().sources, 1); + drop(source); + assert_eq!(pool.usage().sources, 0); + cluster.voters[0].pages.as_mut().unwrap().reclaim(1024)?; + cluster.voters[0].pages.as_mut().unwrap().reclaim(1024)?; + assert!( + !old_page.exists(), + "released obsolete source page can be reclaimed" + ); + cluster.close().await?; + } + Ok(()) +} + +pub(super) async fn refusals() -> Result<()> { + let mut cluster = Cluster::new().await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let original = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let pool = CellGroupCheckpointPool::new(limits())?; + // A recovered empty genesis is a complete closure, with no cache image pins yet. + let mut empty = pool.capture(&group, &original)?; + let mut parts = 0; + empty.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + parts += 1; + Ok(()) + })?; + assert!(parts > 0); + drop(empty); + let foreign = cluster.voters[1].owner.as_ref().unwrap().manifest(); + assert!(matches!( + pool.capture(&group, foreign), + Err(CellGroupCheckpointError::Invalid(_)) + )); + let prospective = cluster.voters[0].tree.prepare_unchanged( + cluster.voters[0].pages.as_ref().unwrap(), + &original, + LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + )?; + assert!(matches!( + pool.capture(&group, &prospective), + Err(CellGroupCheckpointError::Invalid(_)) + )); + assert_eq!(pool.usage().retained_bytes, 0); + cluster.elect().await?; + assert!(matches!( + pool.capture(&group, &original), + Err(CellGroupCheckpointError::Invalid(_)) + )); + let current = cluster.voters[0].owner.as_ref().unwrap().manifest(); + let source = pool.capture(&group, current)?; + let bytes = pool.usage().retained_bytes; + // Two slots, but only enough bytes for one source: exercise byte admission + // independently of the item ceiling, with no refund on a failed acquire. + let small = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: bytes, + ..limits() + })?; + let first = small.capture(&group, current)?; + assert!(matches!( + small.capture(&group, current), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(small.usage().retained_bytes, bytes); + drop(first); + assert_eq!(small.usage().retained_bytes, 0); + drop(source); + drop(prospective); + drop(original); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn ownership() -> Result<()> { + let (cluster, _) = populated(false).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest(); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, root)?; + let usage = pool.usage(); + let mut limited = limits(); + limited.max_cells = 2; + assert!(matches!( + CellGroupCheckpointPool::new(limited)?.capture(&group, root), + Err(CellGroupCheckpointError::Invalid(_)) + )); + limited = limits(); + limited.max_transfer_bytes = 64 * 1024; + assert!(matches!( + CellGroupCheckpointPool::new(limited)?.capture(&group, root), + Err(CellGroupCheckpointError::Limit) + )); + assert!(matches!( + pool.capture(&group, root), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pool.usage(), usage); + let (entered, rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let worker = cluster.domain.blocking(move || { + let mut entered = Some(entered); + let result = source.stream(|_| { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + let _ = release_rx.recv(); + } + Err(CellGroupCheckpointError::Sink(std::io::Error::other( + "cancelled sink", + ))) + }); + assert!(matches!(result, Err(CellGroupCheckpointError::Sink(_)))); + assert!(matches!( + source.stream(|_| Ok(())), + Err(CellGroupCheckpointError::Invalid(_)) + )); + })?; + tokio::time::timeout(Duration::from_secs(10), rx).await??; + drop(worker); // Losing the result waiter must not refund the physical source. + assert_eq!(pool.usage(), usage); + assert!(matches!( + pool.capture(&group, root), + Err(CellGroupCheckpointError::Busy) + )); + release.send(())?; + tokio::time::timeout(Duration::from_secs(10), async { + while pool.usage().sources != 0 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + }) + .await?; + assert_eq!(pool.usage().retained_bytes, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn corruption() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = populated(rooted).await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let pool = CellGroupCheckpointPool::new(limits())?; + let mut source = pool.capture(&group, &root)?; + let (_, pin) = root.pin_cell(&CellId::new("alpha")?).unwrap(); + let path = match pin { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("node-1/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).unwrap(); + cluster._root.path().join("extents-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )) + } + CellImagePin::Empty => unreachable!(), + }; + let mut bytes = std::fs::read(&path)?; + bytes[100] ^= 0x80; + std::fs::write(path, bytes)?; + let result = cluster + .domain + .blocking(move || { + let result = source.stream(|_| Ok(())); + (source, result) + })? + .await?; + assert!(matches!( + result.1, + Err(CellGroupCheckpointError::Invalid( + "materialized image digest" + )) | Err(CellGroupCheckpointError::Extent(_)) + )); + assert_eq!(pool.usage().sources, 1); + drop(result); + let mut closed = pool.capture(&group, &root)?; + drop(cluster.voters[0].owner.take()); + cluster.voters[0].log.close().await?; + let mut emissions = 0; + assert!(matches!( + closed.stream(|_| { + emissions += 1; + Ok(()) + }), + Err(CellGroupCheckpointError::Log(_)) + )); + assert_eq!(emissions, 0); + drop(closed); + drop(root); + cluster.close().await?; + } + Ok(()) +} diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 6465fda4e..77acac556 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -22,8 +22,12 @@ retains 98 distinct command records and 288 source hashes. The plan runner validates the evidence and now selects CG-05 as the next eligible task. Preserve all acceptance cases; the task-boundary audit removed no requirements or checks. -Start CG-05 with complete-group snapshots and verified atomic installation, then -learner/membership recovery. CG-06 follows with archival reclamation. Automatic +CG-05 is now IN_PROGRESS. Its native source captures an exact selected complete +root, configuration and membership, and retains an admitted immutable closure +through bounded streaming and cancellation. Both image forms and empty enrollment +are represented. See the latest [progress](progress.md) for source evidence and +limits. Continue with bounded receiver staging and verified atomic engine +installation, then whole-group transport and learner/membership recovery. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index f67588c4c..e5fda9c48 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -16,7 +16,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | -| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | TODO | 5 | +| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | IN_PROGRESS | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | | CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 5e20b1aa2..94a06551e 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -6,10 +6,42 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). CG-01 through CG-04 have passed their scoped gates; -CG-05 through CG-16 remain TODO. Intermediate contracts +CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 complete-root capture and bounded streaming + +The first source-side snapshot slice adds a shared checkpoint pool and a +non-cloneable physical source. Capture checks the exact selected group root, +engine owner, committed/applied cut and membership revision. It copies membership +and configuration under that same cache lock, and retains the complete immutable +closure. The source streams canonical manifest pages, including empty enrollment, +plus materialized image bytes or extent manifests and verified objects. It does +not aggregate group images or consult a later mutable manifest during transfer. + +Admission accounts for source slots, retained metadata, bounded stream workspace, +cell count and total transfer size. Borrowed chunks are at most 64 KiB, one extent +buffer is at most 4 MiB, and the source keeps its charge until its physical worker +releases it. Missing/corrupt data, a closed engine or a sink error fails the stream; +failed sources cannot be reused. Streaming grants no receiver or voting authority. + +The CG-05 test scope now includes the existing three-voter execution fixture and +`support/cell_group_checkpoint.rs` so this slice exercises real Raft commits, +SQLite/LTX, both image stores and engine-selected roots without duplicating that +fixture. The five full snapshot acceptance cases remain unchanged and outstanding. +The shared fixture additionally handles canonical empty enrollment; existing +execution regression checks cover that helper change. + +Critical gaps for the next turn: build the receiver's bounded staged closure, +verify/sync every dependency, and publish snapshot metadata plus the group root +atomically through the engine. Then connect whole-group transport, learner +catch-up, joint membership and lost-disk replacement. The current header denotes +active groups only; topology/park state must become part of the durable root before +those operations are enabled. CG-05 remains IN_PROGRESS. No RustFS/load, receiver +install, membership-change, independent-process or power-loss qualification is +claimed by this capture slice. + ## CG-04 scoped execution gate complete CG-04 is DONE at `69e41e865c267dff1dec5bce6f2b3d0966be09db`. All 83 declared acceptance cases pass diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index d5644eb01..874b21081 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -226,6 +226,23 @@ and verify the receiver's own synced closure before atomic install. A successful transfer is not proof that another replica has installed it. A new voter receives fresh member/disk identity and joins only through committed membership changes. +The native checkpoint source now captures only an exact engine-selected complete +root. It snapshots configuration and membership under the same cache lock, retains +all immutable manifest/image pins, and streams canonical pages plus materialized +bytes or extent manifests/objects. Empty enrollment remains an authenticated leaf. +The source pool charges both retained metadata and a bounded streaming workspace; +its non-cloneable source and borrowed chunks prevent overlapping streams on one +reservation. Run capture and streaming on a joined blocking owner. A cancelled +waiter does not release the source's physical pins or quota. Flat-image integrity +is final only at EOF, so partial output must never be selected by a receiver. + +This source contract currently represents active groups only: the shared owner +still refuses topology and park entries. Those owning tasks must extend the durable +root and checkpoint transition binding together. Receiver decoding/staging, atomic +engine installation, retained-checkpoint selection, transport admission and +membership catch-up are not implemented by this source API. Successful streaming +supplies no receiver-install, voter, serving, archive-coverage or reclamation proof. + An archive publishes a complete checkpoint plus a contiguous committed tail to K. The tail addresses cells and validates each base chain; administrative entries also replay. Do not rerun commands. Publish coverage only after all dependencies diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 7ef7ab30a..b83dfe175 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1558,7 +1558,7 @@ { "id": "CG-05", "title": "Snapshot and rebuild complete groups through membership changes", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -1573,7 +1573,9 @@ ], "scope": [ "crates/cellar/durability/raft/", - "crates/cellar-ltx/src/" + "crates/cellar-ltx/src/", + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/tests/support/cell_group_checkpoint.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint.rs", From b675f01b9c0a332f979ccdeeadbcc2e9d93a8b31 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 13:56:05 -0700 Subject: [PATCH 119/568] docs: record CG-05 checkpoint source qualification --- docs/design/multi-raft/cell-groups/README.md | 6 +++++- docs/design/multi-raft/cell-groups/handoff.md | 12 +++++++++++- docs/design/multi-raft/cell-groups/progress.md | 9 +++++++++ 3 files changed, 25 insertions(+), 2 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index c66dbe140..914625f79 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -5,7 +5,9 @@ Status: executable design and implementation plan delivered on 2026-10-29. CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and CG-04 shared execution have passed their scoped local evidence gates. -The production host has not yet cut over to shared groups. +CG-05 is in progress: bounded complete-group checkpoint capture and streaming +pass their source-side regression gate. Receiver install and membership recovery +remain open. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -47,6 +49,8 @@ This checkout retains evidence under the ignored `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` directory. A fresh checkout must receive these retained directories at the same relative paths before validating the four DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. +CG-05's partial source checkpoint is retained in +`.cellar/cell-groups-evidence/CG-05-checkpoint-source/`; it does not complete CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 77acac556..fae6a7c06 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -33,9 +33,19 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. +The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands +and 37 validator tests. Transfer +`.cellar/cell-groups-evidence/CG-05-checkpoint-source/` with its predecessors; +its record binds 290 source hashes and three copied executables. The four new +cases cover concurrent real quorum writes while streaming a held cut, both image +representations, empty enrollment, obsolete-page retention/release, bounds, +cancelled physical ownership, stale/foreign/unpublished roots, corruption and +closed engines. These are embedded source tests; receiver install, snapshot RPC, +membership recovery and new process/RustFS/load qualification remain open. + The following records describe the preceding incremental checkpoints. -The newest reviewed source baseline is `69e41e86`, adding bounded explicit local +The preceding reviewed source baseline is `69e41e86`, adding bounded explicit local provisioning from already selected initial-member assignments. The controller owns birth/bootstrap, blocking root recovery and attachment. Exact directory/group requests coalesce; waiter admission is independent from commands. Failed recovered diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 94a06551e..189d549ac 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -12,6 +12,15 @@ is being added. ## CG-05 complete-root capture and bounded streaming +Qualified source: `b15da415c9c9c0663274d0aedd520feb37d90d2d`. All 138 selected Rust cases pass across +shared execution, catalog and manifest storage, including the four new checkpoint +source cases. Eight gate commands pass: all-target Clippy, those regression suites, +formatting, layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. `.cellar/cell-groups-evidence/CG-05-checkpoint-source/checkpoint.json` binds +290 source hashes, three copied executables and retained command/diagnostic logs. +The plan inputs are frozen alongside the evidence. Transfer this ignored local +directory with its predecessor evidence; this is a partial CG-05 gate. + The first source-side snapshot slice adds a shared checkpoint pool and a non-cloneable physical source. Capture checks the exact selected group root, engine owner, committed/applied cut and membership revision. It copies membership From 394be03daf8f946a040d5f9c9130d98b907e8b80 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 14:33:00 -0700 Subject: [PATCH 120/568] feat: verify and atomically install cell-group checkpoints --- crates/cellar-ltx/src/generation.rs | 4 + crates/cellar-ltx/src/generation_receive.rs | 99 ++++ crates/cellar/durability/raft/cache.rs | 80 ++- .../durability/raft/cell_groups/README.md | 32 ++ .../raft/cell_groups/apply/images.rs | 2 +- .../durability/raft/cell_groups/apply/mod.rs | 15 +- .../raft/cell_groups/apply/recovery.rs | 25 + .../raft/cell_groups/checkpoint/header.rs | 85 +++ .../{checkpoint.rs => checkpoint/mod.rs} | 41 +- .../raft/cell_groups/checkpoint/prepared.rs | 107 ++++ .../raft/cell_groups/checkpoint/receive.rs | 507 ++++++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../durability/raft/cell_groups/owner.rs | 3 + .../durability/raft/cell_groups/retention.rs | 120 ++++- .../durability/raft/cell_groups/root.rs | 18 +- crates/cellar/durability/raft/records.rs | 15 +- .../cellar/durability/raft/snapshot_seed.rs | 1 + crates/cellar/durability/raft/storage.rs | 10 +- .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_checkpoint.rs | 16 +- .../support/cell_group_checkpoint_receive.rs | 451 ++++++++++++++++ .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 2 +- .../tests/support/cell_group_sql_apply.rs | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 8 +- 25 files changed, 1604 insertions(+), 62 deletions(-) create mode 100644 crates/cellar-ltx/src/generation_receive.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/header.rs rename crates/cellar/durability/raft/cell_groups/{checkpoint.rs => checkpoint/mod.rs} (91%) create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs create mode 100644 crates/cellar/tests/support/cell_group_checkpoint_receive.rs diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index 7635d5c63..b418dd784 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -11,6 +11,10 @@ use std::ops::{Deref, DerefMut}; use std::path::{Path, PathBuf}; use std::sync::{Arc, Mutex, MutexGuard}; +#[path = "generation_receive.rs"] +mod receive; +pub use receive::GenerationReceive; + pub use crate::generation_accounting::{GenerationAccounting, GenerationResources}; pub use crate::generation_vfs::{open_immutable, ImmutableDatabase, ReadServices}; diff --git a/crates/cellar-ltx/src/generation_receive.rs b/crates/cellar-ltx/src/generation_receive.rs new file mode 100644 index 000000000..5f9c4a591 --- /dev/null +++ b/crates/cellar-ltx/src/generation_receive.rs @@ -0,0 +1,99 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded raw-image receive material. A finished image is still private data; +//! only the owning host can publish its complete durable state marker. + +use super::*; +use crate::stream_snapshot::ScratchReservation; + +pub struct GenerationReceive { + store: GenerationStore, + file: HostFile, + id: ImageId, + expected: ImageState, + received: u64, + failed: bool, + _reservation: ScratchReservation, +} + +impl GenerationStore { + /// Reserve the complete anonymous spool before opening it. Filesystem work + /// belongs on the caller's joined blocking owner. Only one image is buffered + /// on disk; append never holds more than the supplied <=64 KiB chunk. + pub fn receive_image( + &self, + id: ImageId, + expected: ImageState, + scratch: &crate::stream_snapshot::ScratchPool, + directory: &Path, + ) -> Result { + expected.validate(&self.inner.limits.apply)?; + if expected.length == 0 || !directory.is_absolute() { + return Err(GenerationError::Identity); + } + let reservation = scratch.acquire(expected.length, 1)?; + let file = self.inner.filesystem.temporary_file(Some(directory))?; + Ok(GenerationReceive { + store: self.clone(), + file, + id, + expected, + received: 0, + failed: false, + _reservation: reservation, + }) + } +} + +impl GenerationReceive { + pub fn append(&mut self, offset: u64, bytes: &[u8]) -> Result<(), GenerationError> { + if self.failed { + return Err(GenerationError::Quarantined); + } + self.failed = true; + let next = offset + .checked_add(bytes.len() as u64) + .ok_or(GenerationError::Limit)?; + if offset != self.received + || bytes.is_empty() + || bytes.len() > 64 * 1024 + || next > self.expected.length + { + return Err(GenerationError::Identity); + } + self.file.write_all_at(offset, bytes)?; + self.received = next; + self.failed = false; + Ok(()) + } + + /// Independently inspect incoming bytes even if the inventory already has + /// this image. The final immutable destination is separately verified/synced. + /// New unselected images are reclaimable after the returned pin is released; + /// existing host retention is preserved. + pub fn finish(mut self) -> Result { + if self.failed || self.received != self.expected.length { + return Err(GenerationError::Identity); + } + apply::inspect(&mut self.file, &self.expected, None)?; + self.file.sync_all()?; + match self.store.pin_prepared(self.id, self.expected) { + Ok(pin) => return Ok(pin), + Err(GenerationError::Missing) => {} + Err(error) => return Err(error), + } + let pin = self.store.prepare_private( + self.id, + self.expected.length, + None, + None, + |destination, _| { + apply::inspect(&mut self.file, &self.expected, Some(destination))?; + apply::inspect(destination, &self.expected, None)?; + Ok(self.expected) + }, + )?; + self.store.retire(pin.id())?; + Ok(pin) + } +} diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index b742a6e56..5f73b1017 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -91,6 +91,9 @@ pub struct DurableApply { pub enum SnapshotChange { /// A verified local snapshot; retain entries beyond its included index. Checkpoint(Snapshot), + /// A complete receiver-local cell-group closure. Its producer supplies no + /// Raft authority; the consensus owner must bind this to its snapshot effect. + CellGroup(super::cell_groups::PreparedCellGroupCheckpoint), /// A verified incoming snapshot and its already durable state image. Install { snapshot: Snapshot, @@ -126,6 +129,15 @@ impl GroupWrite { } } + pub(crate) fn prepared_group_manifest(&self) -> Option<&PreparedGroupManifest> { + self.group_manifest + .as_ref() + .or_else(|| match &self.snapshot { + Some(SnapshotChange::CellGroup(prepared)) => Some(prepared.root()), + _ => None, + }) + } + pub(crate) fn estimated_bytes(&self) -> Result { let mut bytes = 1024_usize; for entry in &self.entries { @@ -136,6 +148,7 @@ impl GroupWrite { if let Some(snapshot) = &self.snapshot { let snapshot = match snapshot { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, + SnapshotChange::CellGroup(prepared) => prepared.snapshot(), }; if snapshot.compute_size() > 64 * 1024 { return Err(LogError::Overloaded); @@ -157,7 +170,7 @@ impl GroupWrite { .checked_add(membership.reserved_bytes()) .ok_or(LogError::Overloaded)?; } - if let Some(manifest) = &self.group_manifest { + if let Some(manifest) = self.prepared_group_manifest() { bytes = bytes .checked_add(16 * 1024 + manifest.reserved_bytes()) .ok_or(LogError::Overloaded)?; @@ -218,6 +231,12 @@ impl GroupLog { Ok(self.read()?.group_manifest.clone()) } + /// Metadata only; startup must recover this older closure as well as the + /// applied root before any collection or snapshot service can begin. + pub fn checkpoint_manifest(&self) -> Result, LogError> { + Ok(self.read()?.checkpoint_manifest.clone()) + } + /// Last synced bindings for this group history. This observation alone is /// neither a fresh-member seeding capability nor proof of live quorum. pub fn membership(&self) -> Result { @@ -425,6 +444,7 @@ pub(crate) struct CachedGroup { pub applied: DurableApply, pub group_manifest: Option, pub manifest_pins: Option, + pub checkpoint_manifest: Option, } impl CachedGroup { @@ -469,6 +489,8 @@ impl CachedGroup { ManifestPins::maximum_bytes(m.root().cells()), ManifestPins::reserved_bytes, ) + }) + self.checkpoint_manifest.as_ref().map_or(0, |m| { + 16 * 1024 + ManifestPins::maximum_bytes(m.root().cells()) }) + self.membership.reserved_bytes() + self.snapshot.compute_size() as usize + self.conf_state.compute_size() as usize @@ -519,6 +541,14 @@ impl CachedGroup { image.validate()?; } if let Some(manifest) = &self.group_manifest { + if super::cell_groups::checkpoint::persisted_marker( + &self.snapshot, + manifest, + &self.membership, + )? != self.checkpoint_manifest + { + return Err(LogError::Corrupt("group checkpoint selection".into())); + } manifest .check() .map_err(|e| LogError::Corrupt(e.to_string()))?; @@ -532,7 +562,7 @@ impl CachedGroup { { return Err(LogError::Corrupt("incoherent cell-group root".into())); } - } else if self.manifest_pins.is_some() { + } else if self.manifest_pins.is_some() || self.checkpoint_manifest.is_some() { return Err(LogError::Corrupt("image pins without group root".into())); } Ok(()) @@ -545,16 +575,27 @@ impl CachedGroup { history: GroupHistory, ) -> Result { let mut next = self.clone(); + let group_install = matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))); + if group_install + && (self.group_manifest.is_none() + || write.group_manifest.is_some() + || write.applied.is_some() + || write.conf_state.is_some() + || write.membership.is_some()) + { + return Err(LogError::Invalid("mixed cell-group snapshot write".into())); + } if (self.group_manifest.is_some() || write.group_manifest.is_some()) - && (write.applied.is_some() || write.snapshot.is_some()) + && (write.applied.is_some() || (write.snapshot.is_some() && !group_install)) { return Err(LogError::Invalid( - "mixed image/group apply or unsupported group snapshot".into(), + "mixed image/group apply or snapshot".into(), )); } if let Some(change) = &write.snapshot { let snapshot = match change { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, + SnapshotChange::CellGroup(prepared) => prepared.snapshot(), }; let metadata = snapshot.get_metadata(); if metadata.index < self.snapshot.get_metadata().index @@ -565,6 +606,34 @@ impl CachedGroup { return Err(LogError::Invalid("conflicting or stale snapshot".into())); } match change { + SnapshotChange::CellGroup(prepared) => { + let root = prepared.root(); + let marker = root.marker(); + if !Arc::ptr_eq(&root.engine_owner, &write.group.owner) + || !marker.matches_group(&history, write.group.birth_digest) + || marker.route() != self.group_manifest.as_ref().unwrap().route() + || (root.base.as_ref() != self.group_manifest.as_ref() + && Some(marker) != self.group_manifest.as_ref()) + || metadata.index < self.hard_state.commit + || marker.position().index.get() != metadata.index + || marker.position().term.get() != metadata.term + || (marker.position().index == self.applied.position.index + && Some(marker) != self.group_manifest.as_ref()) + { + return Err(LogError::Invalid("cell-group snapshot binding/base".into())); + } + prepared.membership().succeeds(&self.membership)?; + next.entries.clear(); + next.applied = DurableApply { + position: marker.position(), + image: None, + }; + next.group_manifest = Some(marker.clone()); + next.manifest_pins = Some(root.images.clone()); + next.checkpoint_manifest = Some(marker.clone()); + next.conf_state = metadata.get_conf_state().clone(); + next.membership = prepared.membership().clone(); + } SnapshotChange::Checkpoint(_) => { if metadata.index > self.applied.position.index.get() || self.term(metadata.index) != Some(metadata.term) @@ -725,7 +794,7 @@ impl CachedGroup { membership.succeeds(&self.membership)?; next.membership = membership.clone(); } - if (next.conf_state != self.conf_state && write.membership.is_none()) + if (next.conf_state != self.conf_state && write.membership.is_none() && !group_install) || (write.conf_state.is_some() && (write.membership.is_none() || next.membership.revision() != next.applied.position.index)) @@ -804,6 +873,7 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { entries: BTreeMap::new(), group_manifest: None, manifest_pins: None, + checkpoint_manifest: None, applied: DurableApply { position: LogPosition { index: LogIndex::new(1), diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 1d0684b1d..12f110309 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -129,6 +129,38 @@ preparation. These are storage-boundary checks, not quorum, SDK deduplication or RustFS qualification. The full CG-02 source-bound evidence gate is recorded in the delivery plan. +## Complete-group checkpoint storage + +`CellGroupCheckpointPool` admits both sources and receivers under one node-wide +item/byte budget. A source captures the exact engine-selected root, membership +and configuration. It retains immutable dependencies while later writes proceed, +and streams bounded chunks without aggregating the group images. + +A receiver binds the destination engine and its selected base. It validates +ordered page/image parts, route/history, cut, membership and complete tree closure. +Materialized images spool one at a time under a shared `ScratchPool`, then undergo +independent SQLite/checksum verification and destination sync. Extent manifests +and objects are verified against each leaf and independently sealed. Partial, +corrupt, foreign or over-budget transfers cannot publish. All physical work runs +on a joined blocking owner; reservations follow that owner through completion. + +`PreparedCellGroupCheckpoint` owns the sealed destination closure and receive +reservation. `SnapshotChange::CellGroup` atomically publishes its snapshot, +configuration, membership and applied root through the shared engine fence. +Mixed single-cell/group writes, conflicting bases and stale cuts refuse. The +capability establishes physical storage only: its future Raft consumer must bind +it to the exact accepted snapshot effect before submitting the engine write. + +The page owner retains the installed checkpoint separately from the advancing +current root. Checkpoint retention consumes a third root slot in addition to the +current/pending publication slots. Recovery reconstructs both complete closures; +a missing older checkpoint refuses startup and reclamation even when the current +applied root is complete. Physical recovery never renews quorum authority. + +These are verification APIs. The shared runtime still rejects incoming Raft +snapshots and configuration changes. Snapshot transport, owner completion tickets, +learner catch-up, joint membership and disk replacement remain CG-05 work. + ## Enrollment admission and ordered preparation `decide_enrollment` checks a bounded cell-addressed enrollment entry against the diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs index d31dd2ad1..ac084af6d 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/images.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -253,7 +253,7 @@ impl CellGroupApplyImages { // Names are physical lookup hints. The selected manifest and exact image state // still have to verify; a filename never supplies publication authority. -pub(super) fn image_id( +pub(in crate::raft::cell_groups) fn image_id( cell: &CellHistory, sequence: u64, image: ImageState, diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 65455e920..23e9ee108 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -2,7 +2,7 @@ //! Worker-owned physical preparation; only the Raft owner can publish its result. -mod images; +pub(super) mod images; mod recovery; mod summary; pub use images::CellGroupApplyImages; @@ -19,10 +19,10 @@ use crate::raft::{ use cellar_logic::cell_group::{CommitWitness, GroupHistory}; use std::sync::{Arc, Mutex}; -struct Storage { - tree: ManifestTree, - pages: ManifestPageStore, - images: CellGroupApplyImages, +pub(super) struct Storage { + pub(super) tree: ManifestTree, + pub(super) pages: ManifestPageStore, + pub(super) images: CellGroupApplyImages, limits: GroupEntryLimits, max_effect_rows: u32, } @@ -32,8 +32,8 @@ struct Storage { /// owning execution domain; transfer the same page owner used for startup. #[derive(Clone)] pub struct CellGroupApplier { - group: GroupLog, - storage: Arc>, + pub(super) group: GroupLog, + pub(super) storage: Arc>, } impl CellGroupApplier { @@ -91,6 +91,7 @@ impl CellGroupApplier { "registration selected root/page owner", )); } + root.require_checkpoint(&self.group).map_err(storage)?; Ok(()) } diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs index 6eb91882a..b693a908c 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -61,6 +61,31 @@ impl CellGroupApplier { ) .map_err(storage)? }; + if let Some(checkpoint) = applier.group.checkpoint_manifest().map_err(storage)? { + let mut owner = applier + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + let Storage { + tree, + pages, + images, + .. + } = &mut *owner; + tree.prepare_complete( + &applier.group, + pages, + checkpoint.root(), + checkpoint.position(), + checkpoint.membership_revision(), + |leaf| { + images + .recover_image(leaf, checkpoint.route().schema_digest) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }, + ) + .map_err(storage)?; + } if applier.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) { return Err(ExecutionError::Invalid("recovery selection changed")); } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs new file mode 100644 index 000000000..154c7c8a7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::{ConfState, Snapshot}; +use serde::{Deserialize, Serialize}; + +const DOMAIN: &str = "cellar-cell-group-checkpoint-1"; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub(super) enum Transition { + Active, +} + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Header { + pub domain: String, + pub root: GroupManifest, + pub conf: Vec, + pub membership: GroupMembership, + pub transition: Transition, +} + +impl Header { + pub(super) fn capture( + root: GroupManifest, + conf: &ConfState, + membership: GroupMembership, + ) -> Result { + Ok(Self { + domain: DOMAIN.into(), + root, + conf: crate::raft::manifest::encode_conf(conf) + .map_err(|_| CellGroupCheckpointError::Invalid("configuration encoding"))?, + membership, + transition: Transition::Active, + }) + } + + pub(super) fn decode(bytes: &[u8]) -> Result { + if bytes.is_empty() || bytes.len() > HEADER_LIMIT { + return Err(CellGroupCheckpointError::Limit); + } + let header: Self = serde_json::from_slice(bytes) + .map_err(|_| CellGroupCheckpointError::Invalid("checkpoint header"))?; + if header.domain != DOMAIN + || header.root.membership_revision() != header.membership.revision().get() + { + return Err(CellGroupCheckpointError::Invalid( + "checkpoint domain/membership", + )); + } + header.root.check()?; + header + .membership + .validate(&header.configuration()?, header.root.position().index.get())?; + Ok(header) + } + + pub(super) fn configuration(&self) -> Result { + crate::raft::manifest::decode_conf(&self.conf) + .map_err(|_| CellGroupCheckpointError::Invalid("checkpoint configuration")) + } + + pub(super) fn encode(&self) -> Result> { + let bytes = serde_json::to_vec(self) + .map_err(|_| CellGroupCheckpointError::Invalid("header encoding"))?; + if bytes.len() > HEADER_LIMIT { + return Err(CellGroupCheckpointError::Limit); + } + Ok(bytes) + } + + pub(super) fn snapshot(&self) -> Result { + let mut snapshot = Snapshot::default(); + snapshot.mut_metadata().index = self.root.position().index.get(); + snapshot.mut_metadata().term = self.root.position().term.get(); + snapshot + .mut_metadata() + .set_conf_state(self.configuration()?); + snapshot.data = self.encode()?.into(); + Ok(snapshot) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs similarity index 91% rename from crates/cellar/durability/raft/cell_groups/checkpoint.rs rename to crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index a48481b00..bbf241200 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -11,7 +11,13 @@ use crate::raft::{ }; use cellar_logic::cell_group::{CellId, GroupHistory}; use cellar_ltx::internal::{ExtentId, EXTENT_BYTES}; -use serde::Serialize; +mod header; +mod prepared; +mod receive; +use header::Header; +pub(crate) use prepared::persisted_marker; +pub use prepared::PreparedCellGroupCheckpoint; +pub use receive::CellGroupCheckpointReceiver; use sha2::{Digest, Sha256}; use std::sync::Arc; @@ -78,9 +84,9 @@ impl From for CellGroupCheckpointError { type Result = std::result::Result; -/// One pool must be shared by the node's checkpoint producers. Byte accounting -/// conservatively charges pinned metadata to each source as well as its buffer; -/// physical stores separately bound disk retention. No full-image spool is made. +/// One pool must be shared by the node's checkpoint sources and receivers. Byte +/// accounting charges retained metadata and workspace; physical stores bound +/// disk retention. Receivers additionally share a scratch pool for image spools. #[derive(Clone, Copy, Debug)] pub struct CellGroupCheckpointLimits { pub max_sources: usize, @@ -200,23 +206,9 @@ impl CellGroupCheckpointPool { cache .membership .validate(&cache.conf_state, marker.position().index.get())?; - Header { - domain: "cellar-cell-group-checkpoint-1", - root: marker.clone(), - conf: crate::raft::manifest::encode_conf(&cache.conf_state) - .map_err(|_| CellGroupCheckpointError::Invalid("configuration encoding"))?, - membership: cache.membership.clone(), - // Shared owners currently refuse all topology/park entries. - // Their owning tasks must extend the durable root and this - // binding together before admitting such transitions. - transition: "active", - } + Header::capture(marker.clone(), &cache.conf_state, cache.membership.clone())? }; - let encoded = serde_json::to_vec(&header) - .map_err(|_| CellGroupCheckpointError::Invalid("header encoding"))?; - if encoded.len() > HEADER_LIMIT { - return Err(CellGroupCheckpointError::Limit); - } + let encoded = header.encode()?; Ok(CellGroupCheckpoint { group: group.clone(), root: root.clone(), @@ -228,15 +220,6 @@ impl CellGroupCheckpointPool { } } -#[derive(Serialize)] -struct Header { - domain: &'static str, - root: GroupManifest, - conf: Vec, - membership: GroupMembership, - transition: &'static str, -} - /// Borrowed part identity. Each manifest page carries the full cell leaf, /// including empty enrollment, version, receipt-image commitment and effects. /// Extents are emitted in logical order, including repeated content identities; diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs new file mode 100644 index 000000000..d4d417a73 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/prepared.rs @@ -0,0 +1,107 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::CellGroupApplier; +use super::*; +use raft::prelude::Snapshot; + +struct Inner { + applier: CellGroupApplier, + root: PreparedGroupManifest, + header: Header, + snapshot: Snapshot, + _reservation: Permit, +} + +/// Receiver-local complete closure. Clones retain the same admission and physical +/// owners through queued engine work, including a lost publication response. +/// This capability proves storage, not that Raft has accepted the snapshot. +#[derive(Clone)] +pub struct PreparedCellGroupCheckpoint(Arc); + +impl std::fmt::Debug for PreparedCellGroupCheckpoint { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PreparedCellGroupCheckpoint") + .field("marker", self.0.root.marker()) + .finish() + } +} + +impl PreparedCellGroupCheckpoint { + pub(super) fn new( + applier: CellGroupApplier, + root: PreparedGroupManifest, + header: Header, + reservation: Permit, + ) -> Result { + let snapshot = header.snapshot()?; + Ok(Self(Arc::new(Inner { + applier, + root, + header, + snapshot, + _reservation: reservation, + }))) + } + + pub fn marker(&self) -> &GroupManifest { + self.0.root.marker() + } + + pub fn snapshot(&self) -> &Snapshot { + &self.0.snapshot + } + + pub fn applier(&self) -> &CellGroupApplier { + &self.0.applier + } + + /// Complete physical closure. Admission still requires the engine-selected + /// marker and the current catalog permit; preparation alone cannot serve. + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.0.root + } + + pub(crate) fn root(&self) -> &PreparedGroupManifest { + &self.0.root + } + + pub(crate) fn membership(&self) -> &GroupMembership { + &self.0.header.membership + } +} + +/// Decode persisted group snapshot metadata. This does not recover its physical +/// closure; startup must independently recover every referenced page and image. +pub(crate) fn persisted_marker( + snapshot: &Snapshot, + current: &GroupManifest, + membership: &GroupMembership, +) -> std::result::Result, crate::raft::storage::LogError> { + use crate::raft::storage::LogError; + if snapshot.data.is_empty() { + if snapshot.get_metadata().index != 1 || snapshot.get_metadata().term != 1 { + return Err(LogError::Corrupt( + "group snapshot lacks complete descriptor".into(), + )); + } + return Ok(None); + } + let header = Header::decode(&snapshot.data).map_err(|e| LogError::Corrupt(e.to_string()))?; + let metadata = snapshot.get_metadata(); + let conf = header + .configuration() + .map_err(|e| LogError::Corrupt(e.to_string()))?; + if header.root.route() != current.route() + || header.root.position().index.get() != metadata.index + || header.root.position().term.get() != metadata.term + || crate::raft::manifest::encode_conf(&conf) + .map_err(|e| LogError::Corrupt(e.to_string()))? + != crate::raft::manifest::encode_conf(metadata.get_conf_state()) + .map_err(|e| LogError::Corrupt(e.to_string()))? + || header.root.position().index > current.position().index + { + return Err(LogError::Corrupt("group snapshot metadata binding".into())); + } + membership.succeeds(&header.membership)?; + Ok(Some(header.root)) +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs new file mode 100644 index 000000000..ced4b1cae --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -0,0 +1,507 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::{ + apply::images::image_id, CellGroupApplier, CellImageGeometry, CellManifestLeaf, ManifestTree, +}; +use super::*; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellImageCommitment, CellKey}; +use cellar_ltx::{ + apply::ImageState, + generation::GenerationReceive, + internal::{ExtentManifest, ImageBinding, ImageRoot, RootView}, + stream_snapshot::ScratchPool, + TXID, +}; +use std::{collections::BTreeMap, path::PathBuf}; + +// Includes simultaneously received/verified extent bytes, receiver metadata +// decoding, header/page encoding, scratch verification and traversal stacks. +const RECEIVE_WORKSPACE: usize = 2 * EXTENT_BYTES as usize + 2 * 1024 * 1024; + +impl CellGroupCheckpointPool { + /// Share this node pool across both source and receiver owners. The existing + /// max_sources setting bounds their combined count. The scratch pool must + /// likewise be shared by all receivers using this volume. + pub fn receive( + &self, + applier: CellGroupApplier, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + if !directory.is_absolute() || directory.as_os_str().len() > 4096 { + return Err(CellGroupCheckpointError::Invalid( + "receive scratch directory", + )); + } + let cells = self.limits.max_cells as usize; + let extents = usize::try_from(self.limits.max_transfer_bytes.div_ceil(EXTENT_BYTES)) + .map_err(|_| CellGroupCheckpointError::Limit)?; + let bytes = super::super::ManifestPins::maximum_bytes(self.limits.max_cells) + .checked_mul(3) + .and_then(|n| n.checked_add(cells.checked_mul(4096)?)) + .and_then(|n| n.checked_add(extents.checked_mul(2048)?)) + .and_then(|n| n.checked_add(RECEIVE_WORKSPACE)) + .ok_or(CellGroupCheckpointError::Limit)?; + let reservation = self + .budget + .acquire(bytes) + .ok_or(CellGroupCheckpointError::Busy)?; + let base = applier + .group + .group_manifest()? + .ok_or(CellGroupCheckpointError::Invalid("receiver base"))?; + applier + .storage + .lock() + .map_err(|_| CellGroupCheckpointError::Invalid("receiver page owner"))? + .pages + .require_live()?; + Ok(CellGroupCheckpointReceiver { + applier, + base, + scratch, + directory, + limits: self.limits, + header: None, + part: None, + cell: None, + images: BTreeMap::new(), + pages: Vec::new(), + last_key: None, + transferred: 0, + failed: false, + reservation, + }) + } +} + +/// Owns all staged page/image pins and the full receive reservation. Appending +/// and finishing belong on the same joined blocking owner. No partial state is +/// published, and no source pin is accepted as a receiver durability proof. +pub struct CellGroupCheckpointReceiver { + applier: CellGroupApplier, + base: GroupManifest, + scratch: ScratchPool, + directory: PathBuf, + limits: CellGroupCheckpointLimits, + header: Option
, + part: Option, + cell: Option, + images: BTreeMap, + pages: Vec, + last_key: Option, + transferred: u64, + failed: bool, + reservation: Permit, +} + +struct ReceivingCell { + leaf: CellManifestLeaf, + view: Option, + next_extent: u32, +} + +#[derive(Clone, PartialEq, Eq)] +enum Kind { + Header, + Page([u8; 32]), + Materialized(CellId), + ExtentManifest(CellId), + Extent(CellId, u32, ExtentId), +} + +impl Kind { + fn of(part: CellGroupCheckpointPart<'_>) -> Self { + match part { + CellGroupCheckpointPart::Header => Self::Header, + CellGroupCheckpointPart::Page(id) => Self::Page(id), + CellGroupCheckpointPart::Materialized(cell) => Self::Materialized(cell.clone()), + CellGroupCheckpointPart::ExtentManifest(cell) => Self::ExtentManifest(cell.clone()), + CellGroupCheckpointPart::Extent { cell, ordinal, id } => { + Self::Extent(cell.clone(), ordinal, id) + } + } + } +} + +struct Part { + kind: Kind, + length: u64, + received: u64, + data: Data, +} + +enum Data { + Bytes(Vec), + Image(GenerationReceive), +} + +fn invalid(message: &'static str) -> CellGroupCheckpointError { + CellGroupCheckpointError::Invalid(message) +} + +impl CellGroupCheckpointReceiver { + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + if self.failed { + return Err(invalid("failed receiver")); + } + self.failed = true; + self.append_inner(chunk)?; + self.failed = false; + Ok(()) + } + + fn append_inner(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.applier.group.cache_revision()?; + if chunk.bytes.is_empty() || chunk.bytes.len() > self.limits.max_chunk_bytes { + return Err(CellGroupCheckpointError::Limit); + } + self.transferred = self + .transferred + .checked_add(chunk.bytes.len() as u64) + .filter(|n| *n <= self.limits.max_transfer_bytes) + .ok_or(CellGroupCheckpointError::Limit)?; + let kind = Kind::of(chunk.part); + if self.part.is_none() { + if chunk.offset != 0 { + return Err(invalid("receive part offset")); + } + self.part = Some(self.start_part(kind.clone(), chunk.length)?); + } + let part = self.part.as_mut().ok_or(invalid("receive part"))?; + let next = chunk + .offset + .checked_add(chunk.bytes.len() as u64) + .ok_or(CellGroupCheckpointError::Limit)?; + if part.kind != kind + || part.length != chunk.length + || part.received != chunk.offset + || next > part.length + { + return Err(invalid("mixed or noncontiguous receive part")); + } + match &mut part.data { + Data::Bytes(bytes) => bytes.extend_from_slice(chunk.bytes), + Data::Image(image) => image.append(chunk.offset, chunk.bytes)?, + } + part.received = next; + if next == part.length { + let completed = self.part.take().ok_or(invalid("receive completion"))?; + self.complete_part(completed)?; + } + Ok(()) + } + + fn start_part(&self, kind: Kind, length: u64) -> Result { + if length == 0 || length > self.limits.max_transfer_bytes { + return Err(CellGroupCheckpointError::Limit); + } + let limit = match &kind { + Kind::Header if self.header.is_none() && self.pages.is_empty() => HEADER_LIMIT as u64, + Kind::Page(_) if self.header.is_some() && self.cell.is_none() => { + if self.pages.len() >= self.header.as_ref().unwrap().root.root().cells() as usize { + return Err(invalid("extra checkpoint page")); + } + page_wire::MAX_PAGE_BYTES as u64 + } + Kind::Materialized(cell) => { + let pending = self.cell.as_ref().ok_or(invalid("image without leaf"))?; + if &pending.leaf.version.history.cell != cell + || !matches!( + pending.leaf.version.image.commitment, + CellImageCommitment::Materialized(_) + ) + || length != pending.leaf.geometry.length + { + return Err(invalid("materialized part binding")); + } + let CellImageCommitment::Materialized(digest) = + pending.leaf.version.image.commitment + else { + unreachable!() + }; + let expected = ImageState { + epoch: pending.leaf.version.image.epoch, + txid: TXID(pending.leaf.version.image.txid), + checksum: pending.leaf.version.image.checksum, + digest, + length, + page_size: pending.leaf.geometry.page_size, + }; + let id = image_id( + &pending.leaf.version.history, + pending.leaf.version.sequence.get(), + expected, + ) + .map_err(|_| invalid("received image identity"))?; + let storage = self + .applier + .storage + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Materialized { images, .. } = + &storage.images + else { + return Err(invalid("receiver image representation")); + }; + return Ok(Part { + kind, + length, + received: 0, + data: Data::Image(images.receive_image( + id, + expected, + &self.scratch, + &self.directory, + )?), + }); + } + Kind::ExtentManifest(cell) => { + let pending = self + .cell + .as_ref() + .ok_or(invalid("extent manifest without leaf"))?; + let CellImageCommitment::Extents { manifest_bytes, .. } = + pending.leaf.version.image.commitment + else { + return Err(invalid("extent manifest representation")); + }; + if &pending.leaf.version.history.cell != cell + || pending.view.is_some() + || length != u64::from(manifest_bytes) + { + return Err(invalid("extent manifest binding")); + } + // Includes the largest supported extent tree, while staying + // inside the admitted metadata workspace before allocation. + u64::from(manifest_bytes).min(1024 * 1024) + } + Kind::Extent(cell, ordinal, id) => { + let pending = self.cell.as_ref().ok_or(invalid("extent without leaf"))?; + let view = pending + .view + .as_ref() + .ok_or(invalid("extent without manifest"))?; + if &pending.leaf.version.history.cell != cell + || *ordinal != pending.next_extent + || view.manifest().extent(*ordinal) != Some(*id) + || length != u64::from(id.length()) + { + return Err(invalid("extent part binding")); + } + EXTENT_BYTES + } + _ => return Err(invalid("unexpected checkpoint part")), + }; + if length == 0 || length > limit { + return Err(CellGroupCheckpointError::Limit); + } + Ok(Part { + kind, + length, + received: 0, + data: Data::Bytes(Vec::with_capacity(length as usize)), + }) + } + + fn complete_part(&mut self, part: Part) -> Result<()> { + match (part.kind, part.data) { + (Kind::Header, Data::Bytes(bytes)) => { + let header = Header::decode(&bytes)?; + if header.root.route() != self.base.route() + || header.root.root().cells() > self.limits.max_cells + || header.root.position().index < self.base.position().index + { + return Err(invalid("receiver history/range/cut")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + self.header = Some(header); + } + (Kind::Page(digest), Data::Bytes(bytes)) => { + let header = self.header.as_ref().ok_or(invalid("page without header"))?; + if page_wire::digest(&bytes) != digest { + return Err(invalid("received page digest")); + } + let limits = ManifestLimits { + max_cells: self.limits.max_cells, + max_depth: 32, + }; + let page = page_wire::decode(&bytes, header.root.route(), limits)?; + page.reference(&bytes, limits)?; + let key = cell_routing_key( + header.root.route().history.namespace, + &page.leaf.version.history.cell, + ); + if self.last_key.as_ref().is_some_and(|last| last >= &key) + || page.leaf.last_mutation.index > header.root.position().index + || page.leaf.last_mutation.term > header.root.position().term + { + return Err(invalid("checkpoint leaf order/cut")); + } + let mut storage = self + .applier + .storage + .lock() + .map_err(|_| invalid("receiver storage"))?; + storage.pages.put(digest, &bytes)?; + self.pages.push(storage.pages.pin(digest)?); + self.last_key = Some(key.clone()); + if page.leaf.version.image.txid == 0 { + CellImagePin::Empty.check(&page.leaf, header.root.route().schema_digest)?; + self.images.insert(key, (page.leaf, CellImagePin::Empty)); + } else { + self.cell = Some(ReceivingCell { + leaf: page.leaf, + view: None, + next_extent: 0, + }); + } + } + (Kind::Materialized(_), Data::Image(image)) => { + self.finish_cell(CellImagePin::Materialized(image.finish()?))? + } + (Kind::ExtentManifest(_), Data::Bytes(bytes)) => { + let pending = self.cell.as_mut().ok_or(invalid("extent cell"))?; + let CellImageCommitment::Extents { root, .. } = + pending.leaf.version.image.commitment + else { + return Err(invalid("extent leaf")); + }; + let manifest = ExtentManifest::decode(&bytes, ImageRoot::from_bytes(root)) + .map_err(cellar_ltx::internal::StoreError::from)?; + let image = pending.leaf.version.image; + if manifest.binding() + != (ImageBinding { + history: super::super::cell_image_history_digest( + &pending.leaf.version.history, + ), + schema: self.base.route().schema_digest, + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }) + || (CellImageGeometry { + length: manifest.geometry().length(), + page_size: manifest.geometry().page_size(), + }) != pending.leaf.geometry + { + return Err(invalid("extent image binding")); + } + let storage = self + .applier + .storage + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Extents { images, .. } = &storage.images + else { + return Err(invalid("receiver image representation")); + }; + pending.view = Some(images.stage_transient_root(manifest)?); + } + (Kind::Extent(_, _, id), Data::Bytes(bytes)) => { + let pin = { + let pending = self.cell.as_mut().ok_or(invalid("extent cell"))?; + let view = pending.view.as_ref().ok_or(invalid("extent view"))?; + let storage = self + .applier + .storage + .lock() + .map_err(|_| invalid("receiver storage"))?; + let super::super::CellGroupApplyImages::Extents { images, .. } = + &storage.images + else { + return Err(invalid("receiver image representation")); + }; + images.put_extent_bytes(id, &bytes)?; + drop(bytes); + pending.next_extent += 1; + if pending.next_extent == view.manifest().geometry().extent_count() { + Some(CellImagePin::Extents(Arc::new(images.seal_image_root( + view.manifest().clone(), + &mut vec![0; EXTENT_BYTES as usize], + )?))) + } else { + None + } + }; + if let Some(pin) = pin { + self.finish_cell(pin)?; + } + } + _ => return Err(invalid("receive part type")), + } + Ok(()) + } + + fn finish_cell(&mut self, pin: CellImagePin) -> Result<()> { + let pending = self.cell.take().ok_or(invalid("missing receive cell"))?; + pin.check(&pending.leaf, self.base.route().schema_digest)?; + let key = cell_routing_key( + self.base.route().history.namespace, + &pending.leaf.version.history.cell, + ); + if self.images.insert(key, (pending.leaf, pin)).is_some() { + return Err(invalid("duplicate receive cell")); + } + Ok(()) + } + + /// Verify the complete authenticated tree against every independently sealed + /// destination image and sync its page closure. This still does not install + /// it: only the Raft owner's exact snapshot effect may publish the result. + pub fn finish(self) -> Result { + if self.failed || self.part.is_some() || self.cell.is_some() { + return Err(invalid("incomplete checkpoint")); + } + let header = self.header.ok_or(invalid("missing checkpoint header"))?; + if self.images.len() != header.root.root().cells() as usize + || self.pages.len() != self.images.len() + || self.applier.group.group_manifest()?.as_ref() != Some(&self.base) + { + return Err(invalid("incomplete closure or changed receiver base")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + let root = { + let mut storage = self + .applier + .storage + .lock() + .map_err(|_| invalid("receiver storage"))?; + let tree = ManifestTree::new( + header.root.route().clone(), + ManifestLimits { + max_cells: self.limits.max_cells, + max_depth: 32, + }, + )?; + let mut root = tree.prepare_complete( + &self.applier.group, + &mut storage.pages, + header.root.root(), + header.root.position(), + header.root.membership_revision(), + |leaf| { + let key = cell_routing_key( + header.root.route().history.namespace, + &leaf.version.history.cell, + ); + self.images + .get(&key) + .filter(|(expected, _)| expected == leaf) + .map(|(_, pin)| pin.clone()) + .ok_or(super::super::ManifestError::Invalid( + "missing received cell image", + )) + }, + )?; + root.base = Some(self.base); + root + }; + PreparedCellGroupCheckpoint::new(self.applier, root, header, self.reservation) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 2e0226f7d..ef6d92285 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -15,11 +15,11 @@ pub use crate::raft::transport::{ }; mod apply; -mod checkpoint; +pub(crate) mod checkpoint; pub use checkpoint::{ CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, CellGroupCheckpointLimits, CellGroupCheckpointPart, CellGroupCheckpointPool, - CellGroupCheckpointUsage, + CellGroupCheckpointReceiver, CellGroupCheckpointUsage, PreparedCellGroupCheckpoint, }; mod controller; pub use controller::{ diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index c9838b4dc..ee01b7963 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -203,6 +203,9 @@ impl CellGroupOwner { )); } drop(cache); + current + .require_checkpoint(&group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; let config = Config { id: group.identity().member.get(), applied: current.marker().position().index.get(), diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs index 210913d56..1855a037e 100644 --- a/crates/cellar/durability/raft/cell_groups/retention.rs +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -25,6 +25,9 @@ struct Group { closed: Arc, current: Option, pending: Option, + checkpoint: Option, + expected_checkpoint: Option, + pending_checkpoint: bool, } #[derive(Default)] @@ -66,11 +69,19 @@ impl RootRetentions { Ok(state) } + fn used_roots(state: &State) -> usize { + state.groups.len() * 2 + + state.retained.len() + + state + .groups + .values() + .filter(|g| g.checkpoint.is_some() || g.pending_checkpoint) + .count() + } + fn admit_group(&self, state: &State, key: GroupKey) -> Result<(), ManifestError> { // Reserve current + pending slots before a group can submit a write. - if !state.groups.contains_key(&key) - && (state.groups.len() + 1) * 2 + state.retained.len() > self.max_roots - { + if !state.groups.contains_key(&key) && Self::used_roots(state) + 2 > self.max_roots { return Err(ManifestError::Limit); } Ok(()) @@ -85,6 +96,9 @@ impl RootRetentions { if group.closed.load(Ordering::Acquire) { return Err(ManifestError::Quarantined); } + let expected_checkpoint = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; let mut state = self.state()?; let key = key(marker); self.admit_group(&state, key)?; @@ -93,6 +107,9 @@ impl RootRetentions { closed: group.closed.clone(), current: None, pending: None, + checkpoint: None, + expected_checkpoint, + pending_checkpoint: false, }); if !Arc::ptr_eq(&selected.engine, &group.owner) || selected.pending.is_some() @@ -112,11 +129,51 @@ impl RootRetentions { Ok(()) } + pub(super) fn recover_checkpoint( + &self, + group: &GroupLog, + marker: &GroupManifest, + pins: &ManifestPins, + ) -> Result<(), ManifestError> { + let mut state = self.state()?; + let key = key(marker); + let selected = state + .groups + .get(&key) + .ok_or(ManifestError::Invalid("recover applied root first"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.pending.is_some() + || selected.closed.load(Ordering::Acquire) + || selected.expected_checkpoint.as_ref() != Some(marker) + || selected + .checkpoint + .as_ref() + .is_some_and(|r| r.marker != *marker) + { + return Err(ManifestError::Conflict); + } + if selected.checkpoint.is_none() { + if Self::used_roots(&state) >= self.max_roots { + return Err(ManifestError::Limit); + } + state + .groups + .get_mut(&key) + .ok_or(ManifestError::Conflict)? + .checkpoint = Some(Root { + marker: marker.clone(), + _pins: pins.clone(), + }); + } + Ok(()) + } + pub(super) fn begin( self: &Arc, group: &GroupLog, prepared: &PreparedGroupManifest, previous: Option<&GroupManifest>, + checkpoint: bool, ) -> Result { if group.closed.load(Ordering::Acquire) { return Err(ManifestError::Quarantined); @@ -129,11 +186,24 @@ impl RootRetentions { "selected root has not been recovered", )); } + if checkpoint + && state + .groups + .get(&key) + .is_none_or(|g| g.checkpoint.is_none()) + && Self::used_roots(&state) + usize::from(!state.groups.contains_key(&key)) * 2 + >= self.max_roots + { + return Err(ManifestError::Limit); + } let selected = state.groups.entry(key).or_insert_with(|| Group { engine: group.owner.clone(), closed: group.closed.clone(), current: None, pending: None, + checkpoint: None, + expected_checkpoint: None, + pending_checkpoint: false, }); if !Arc::ptr_eq(&selected.engine, &group.owner) || selected.pending.is_some() @@ -141,6 +211,7 @@ impl RootRetentions { { return Err(ManifestError::Conflict); } + selected.pending_checkpoint = checkpoint; selected.pending = Some(Root { marker: prepared.marker().clone(), _pins: prepared.images.clone(), @@ -148,11 +219,32 @@ impl RootRetentions { Ok(PublicationPin { owner: self.clone(), key, + checkpoint, armed: false, finished: false, }) } + pub(super) fn require_checkpoint(&self, group: &GroupLog) -> Result<(), ManifestError> { + let expected = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + let Some(expected) = expected else { + return Ok(()); + }; + let state = self.state()?; + let selected = state + .groups + .get(&key(&expected)) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.checkpoint.as_ref().map(|r| &r.marker) != Some(&expected) + { + return Err(ManifestError::Invalid("checkpoint not recovered")); + } + Ok(()) + } + pub(super) fn reconcile(&self, retained: Vec) -> Result<(), ManifestError> { let mut state = self.state()?; if state @@ -165,7 +257,7 @@ impl RootRetentions { if state.groups.values().any(|g| g.pending.is_some()) { return Err(ManifestError::Conflict); } - if state.groups.len() * 2 + retained.len() > self.max_roots { + if Self::used_roots(&state) - state.retained.len() + retained.len() > self.max_roots { return Err(ManifestError::Limit); } state.retained = retained; @@ -182,6 +274,13 @@ impl RootRetentions { { return Err(ManifestError::Quarantined); } + if state + .groups + .values() + .any(|g| g.checkpoint.as_ref().map(|r| &r.marker) != g.expected_checkpoint.as_ref()) + { + return Err(ManifestError::Invalid("checkpoint not recovered")); + } if !state.reconciled { return Err(ManifestError::Invalid("root inventory not reconciled")); } @@ -195,6 +294,7 @@ impl RootRetentions { pub(crate) struct PublicationPin { owner: Arc, key: GroupKey, + checkpoint: bool, armed: bool, finished: bool, } @@ -211,7 +311,16 @@ impl PublicationPin { .groups .get_mut(&self.key) .ok_or(ManifestError::Conflict)?; - group.current = Some(group.pending.take().ok_or(ManifestError::Conflict)?); + let pending = group.pending.take().ok_or(ManifestError::Conflict)?; + if self.checkpoint { + group.expected_checkpoint = Some(pending.marker.clone()); + group.checkpoint = Some(Root { + marker: pending.marker.clone(), + _pins: pending._pins.clone(), + }); + } + group.pending_checkpoint = false; + group.current = Some(pending); } self.finished = true; Ok(()) @@ -230,6 +339,7 @@ impl Drop for PublicationPin { state.quarantined = true; } else if let Some(group) = state.groups.get_mut(&self.key) { group.pending = None; + group.pending_checkpoint = false; if group.current.is_none() { state.groups.remove(&self.key); } diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 897960306..a20dc9a62 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -109,6 +109,13 @@ impl PreparedGroupManifest { .get(&cell_routing_key(self.marker.route.history.namespace, cell)) } + pub(crate) fn require_checkpoint( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result<(), ManifestError> { + self.retention.require_checkpoint(group) + } + pub(crate) fn reserved_bytes(&self) -> usize { self.images.reserved_bytes() } @@ -117,8 +124,9 @@ impl PreparedGroupManifest { &self, group: &crate::raft::cache::GroupLog, previous: Option<&GroupManifest>, + checkpoint: bool, ) -> Result { - self.retention.begin(group, self, previous) + self.retention.begin(group, self, previous, checkpoint) } } @@ -189,6 +197,14 @@ impl ManifestTree { { store.retention.recover(group, &marker, &pins)?; } + if group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .as_ref() + == Some(&marker) + { + store.retention.recover_checkpoint(group, &marker, &pins)?; + } Ok(PreparedGroupManifest { marker, images: pins, diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index 692d24b2b..669565638 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -271,14 +271,27 @@ impl GroupRecord { (None, Some(marker.as_ref().clone())) } }; + let snapshot = Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?; + let checkpoint_manifest = group_manifest + .as_ref() + .map(|current| { + super::cell_groups::checkpoint::persisted_marker( + &snapshot, + current, + &self.membership, + ) + }) + .transpose()? + .flatten(); let cache = CachedGroup { hard_state: HardState::parse_from_bytes(&self.hard_state).map_err(corrupt)?, conf_state: ConfState::parse_from_bytes(&self.conf_state).map_err(corrupt)?, membership: self.membership.clone(), - snapshot: Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?, + snapshot, entries: Default::default(), group_manifest, manifest_pins: None, + checkpoint_manifest, applied: DurableApply { position: LogPosition { index: LogIndex::new(self.applied_index), diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index 262ce44d0..3e7af8791 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -295,6 +295,7 @@ impl PreparedSeed { entries: std::collections::BTreeMap::new(), group_manifest: None, manifest_pins: None, + checkpoint_manifest: None, applied: crate::raft::cache::DurableApply { position: LogPosition { index: LogIndex::new(self.descriptor.index), diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index fbd0969a2..af8c8fceb 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1366,7 +1366,7 @@ impl Worker { drop(old); if let Some(change) = &write.snapshot { match change { - SnapshotChange::Install { .. } => { + SnapshotChange::Install { .. } | SnapshotChange::CellGroup(_) => { batch.add_command(write.group.key, Command::Clean) } SnapshotChange::Checkpoint(snapshot) => batch.add_command( @@ -1394,11 +1394,15 @@ impl Worker { // and quarantines collection until recovery reconstructs the selection. let mut publications = Vec::new(); for write in &writes { - if let Some(prepared) = &write.group_manifest { + if let Some(prepared) = write.prepared_group_manifest() { let previous = write.group.group_manifest()?; publications.push( prepared - .begin_publication(&write.group, previous.as_ref()) + .begin_publication( + &write.group, + previous.as_ref(), + matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))), + ) .map_err(|e| LogError::Invalid(e.to_string()))?, ); } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index b0b5c6606..8d0509761 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,21 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_installs_and_recovers_materialized_closure() -> Result<()> { + runtime_fixture::install_checkpoint(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_installs_and_recovers_extent_closure() -> Result<()> { + runtime_fixture::install_checkpoint(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_receive_rejects_partial_corrupt_and_over_budget_streams() -> Result<()> { + runtime_fixture::refuse_checkpoint().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_capture_streams_one_cut_during_writes() -> Result<()> { checkpoint_fixture::coherent().await diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs index ca65f79a1..a0e578ae3 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -10,7 +10,7 @@ use cellar_ltx::internal::{ }; use std::collections::BTreeMap; -fn limits() -> CellGroupCheckpointLimits { +pub(super) fn limits() -> CellGroupCheckpointLimits { CellGroupCheckpointLimits { max_sources: 1, max_retained_bytes: 32 * 1024 * 1024, @@ -21,8 +21,16 @@ fn limits() -> CellGroupCheckpointLimits { } async fn populated(rooted: bool) -> Result<(Cluster, BTreeMap>)> { + populated_with_lagging(rooted, false).await +} + +pub(super) async fn populated_with_lagging( + rooted: bool, + lagging: bool, +) -> Result<(Cluster, BTreeMap>)> { let mut cluster = Cluster::new().await?; cluster.elect().await?; + cluster.voters[2].paused = lagging; let route = cluster.birth.route().clone(); if rooted { for n in 0..3 { @@ -134,7 +142,7 @@ async fn advance(cluster: &mut Cluster, name: &str, before: &[u8], rooted: bool) } #[derive(Default)] -struct Received { +pub(super) struct Received { header: Vec, pages: BTreeMap<[u8; 32], Vec>, images: BTreeMap>, @@ -144,7 +152,7 @@ struct Received { } impl Received { - fn append( + pub(super) fn append( &mut self, chunk: CellGroupCheckpointChunk<'_>, ) -> std::result::Result<(), CellGroupCheckpointError> { @@ -173,7 +181,7 @@ impl Received { Ok(()) } - fn verify( + pub(super) fn verify( mut self, marker: &GroupManifest, leaves: &BTreeMap, diff --git a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs new file mode 100644 index 000000000..ac260636d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs @@ -0,0 +1,451 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Receiver/storage qualification with a lagging member of the real quorum. +//! Snapshot transport and Raft Ready acceptance are deliberately separate gates. +use super::*; +use cellar_logic::consensus::OperationId; +use cellar_ltx::internal::{CacheLimits, NodeExtentStore, EXTENT_BYTES}; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; +use raft::Storage; +use std::path::Path; + +fn scratch() -> Result { + Ok(ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?) +} + +fn config( + images: GenerationStore, + extents: Option, + directory: &Path, +) -> Result { + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + Ok(match extents { + Some(images) => CellGroupApplyImages::Extents { + images, + services, + limits: Default::default(), + cache_directory: directory.join("cache"), + cache_pool: scratch()?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }, + None => CellGroupApplyImages::Materialized { images, services }, + }) +} + +fn applier(cluster: &mut Cluster) -> Result { + let route = cluster.birth.route(); + let voter = &mut cluster.voters[2]; + drop(voter.owner.take()); + drop(voter.held.take()); + voter.paused = false; + Ok(CellGroupApplier::new( + voter.log.cell_group(&route.history)?, + voter.pages.take().context("pages")?, + config( + voter.images.clone(), + voter.extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) +} + +pub(crate) async fn install_checkpoint(rooted: bool) -> Result<()> { + let (mut cluster, expected) = checkpoint_fixture::populated_with_lagging(rooted, true).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let cut = source.marker().clone(); + assert!(cut.position().index > before.position().index); + let applier = applier(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let scratch = scratch()?; + let mut receiver = pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into(), + )?; + let prepared = cluster + .domain + .blocking(move || { + source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + assert_eq!(scratch.usage()?.reserved_bytes, 0); + assert_eq!(prepared.marker(), &cut); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 1); + // A receiver proof cannot grant serving admission before atomic publication. + assert!(cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(2).await?, + prepared.manifest().clone(), + cluster.options.clone(), + [91; 32] + )) + .await? + .is_err()); + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroup(prepared.clone())); + let mut hard = group.initial_state()?.hard_state; + hard.commit = cut.position().index.get(); + write.hard_state = Some(hard); + // Reject mixed contracts without releasing the receiver's physical owners. + let mut mixed = write.clone(); + mixed.group_manifest = Some(prepared.manifest().clone()); + assert!(cluster.voters[2] + .log + .persist_cells(OperationId::new(910), vec![mixed]) + .await + .is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + let installed = prepared.manifest().clone(); + drop(prepared); + let gate = cluster.voters[2].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let writer = cluster.voters[2].log.clone(); + let mut pending = cluster.domain.spawn(async move { + writer + .persist_cells(OperationId::new(911), vec![write]) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + pool.usage().sources, + 1, + "queued physical work owns receive admission" + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + cluster.voters[2].images.retain_only(&[])?; + cluster.voters[2].images.reclaim(1024)?; + if let Some(extents) = &cluster.voters[2].extents { + extents.retain_only(&[])?; + extents.reclaim(1024)?; + } + drop(release); + pending.await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + // Stream the receiver's independently sealed closure and compare every byte + // against the SQL fixtures, including the canonical empty cell. + let mut received = checkpoint_fixture::Received::default(); + let mut local = source_pool.capture(&group, &installed)?; + local.stream(|chunk| received.append(chunk))?; + let leaves = expected + .keys() + .map(|name| { + Ok(( + name.clone(), + installed + .pin_cell(&CellId::new(name)?) + .context("received leaf")? + .0 + .clone(), + )) + }) + .collect::>>()?; + received.verify(&cut, &leaves, &expected, rooted)?; + drop(local); + cluster.voters[2].owner = Some( + cluster + .domain + .run(CellGroupOwner::start( + cluster.permit(2).await?, + installed.clone(), + cluster.options.clone(), + [92; 32], + )) + .await??, + ); + assert!(!cluster.voters[2].owner.as_ref().unwrap().status()?.ready); + drop(installed); + assert_eq!(pool.usage().sources, 0); + drop(group); + // Resume real quorum replication, now through the production apply pool on + // the receiver. A newer root must not erase the installed Raft checkpoint. + cluster.network.clear(); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let later = history(&route, "after-checkpoint")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(42);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + sql::apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + let current = group.group_manifest()?.context("later marker")?; + assert!(current.position().index > cut.position().index); + assert_ne!(current.root(), cut.root()); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + drop(group); + let directory = fixture.cluster.directory.clone(); + let birth = fixture.cluster.birth.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + fixture.close().await?; + // Every original engine, runtime, image/page pin and receiver is now gone. + // Remove the old checkpoint root page, while the later root stays on disk. + let checkpoint_path = disk.path().join("node-3/pages").join(format!( + "{}.page", + hex::encode(cut.root().digest().unwrap()) + )); + // Independently traverse the page references to prove this fault removes + // only the old checkpoint, not any dependency of the current applied tree. + let mut pending_pages = vec![current.root().digest().unwrap()]; + while let Some(digest) = pending_pages.pop() { + assert_ne!(digest, cut.root().digest().unwrap()); + let bytes = std::fs::read( + disk.path() + .join("node-3/pages") + .join(format!("{}.page", hex::encode(digest))), + )?; + let page: serde_json::Value = serde_json::from_slice(&bytes[8..])?; + for child in ["left", "right"] { + if !page[child].is_null() { + pending_pages.push(serde_json::from_value::<[u8; 32]>( + page[child]["digest"].clone(), + )?); + } + } + } + let checkpoint_bytes = std::fs::read(&checkpoint_path)?; + std::fs::remove_file(&checkpoint_path)?; + for missing in [true, false] { + if !missing { + std::fs::write(&checkpoint_path, &checkpoint_bytes)?; + } + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-3"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([3; 16])?, + DiskLineage::new([13; 16])?, + ProcessGeneration::new([if missing { 41 } else { 42 }; 16])?, + LogOpenMode::Resume, + ), + Arc::new(fault::FaultFileSystem { + gate: fault::Gate::new(None), + }), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(¤t)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = config( + GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + if rooted { + Some(NodeExtentStore::open( + disk.path().join("extents-2"), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }, + disk.path(), + )?; + let recovering = group.clone(); + let recovered = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await?; + if missing { + assert!( + recovered.is_err(), + "missing installed checkpoint was admitted" + ); + } else { + let (applier, root) = recovered?; + assert_eq!(root.marker(), ¤t); + assert_eq!(root.pinned_images(), 4); + assert!(root.pin_cell(&later.cell).is_some()); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [93; 32], + )) + .await??; + assert!( + !owner.status()?.ready, + "storage recovery cannot renew quorum authority" + ); + drop(owner); + drop(applier); + } + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + } + Ok(()) +} + +pub(crate) async fn refuse_checkpoint() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = checkpoint_fixture::populated_with_lagging(rooted, true).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let applier = applier(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for failure in 0..6 { + let scratch = if failure == 3 { + ScratchPool::new(ScratchLimits { + max_bytes: 1, + max_files: 1, + })? + } else { + scratch()? + }; + let mut receiver = pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into(), + )?; + assert!(matches!( + pool.receive( + applier.clone(), + scratch.clone(), + cluster._root.path().into() + ), + Err(CellGroupCheckpointError::Busy) + )); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut changed = false; + let result = source.stream(|chunk| { + if failure >= 4 && matches!(chunk.part, CellGroupCheckpointPart::Header) { + changed = true; + if failure == 5 { + return receiver.append(CellGroupCheckpointChunk { + length: 65537, + ..chunk + }); + } + assert_eq!(chunk.offset, 0); + assert_eq!(chunk.length, chunk.bytes.len() as u64); + let mut header: serde_json::Value = + serde_json::from_slice(chunk.bytes).unwrap(); + header["root"]["incarnation"] = serde_json::to_value([99u8; 16]).unwrap(); + let foreign = serde_json::to_vec(&header).unwrap(); + return receiver.append(CellGroupCheckpointChunk { + length: foreign.len() as u64, + bytes: &foreign, + ..chunk + }); + } + if !changed + && (matches!(chunk.part, CellGroupCheckpointPart::Materialized(_)) + || matches!(chunk.part, CellGroupCheckpointPart::Extent { .. })) + { + changed = true; + if failure == 0 { + return Err(CellGroupCheckpointError::Invalid("test truncation")); + } + if failure == 1 { + return receiver.append(CellGroupCheckpointChunk { + offset: chunk.offset + 1, + ..chunk + }); + } + if failure == 2 { + let mut corrupt = chunk.bytes.to_vec(); + corrupt[0] ^= 0xff; + return receiver.append(CellGroupCheckpointChunk { + bytes: &corrupt, + ..chunk + }); + } + } + receiver.append(chunk) + }); + if failure != 3 || !rooted { + assert!(changed); + assert!(result.is_err()); + assert!(receiver.finish().is_err()); + } else { + // Extent transfer consumes bounded memory, no flat-image spool. + result?; + drop(receiver.finish()?); + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(scratch.usage()?.reserved_files, 0); + assert_eq!(pool.usage().sources, 0); + } + drop(applier); + drop(group); + cluster.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 20d50d85a..25d1245c6 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -5,6 +5,10 @@ use super::*; use std::collections::BTreeMap; +#[path = "cell_group_checkpoint_receive.rs"] +mod checkpoint_receive; +pub(super) use checkpoint_receive::{install_checkpoint, refuse_checkpoint}; + #[path = "cell_group_tcp.rs"] mod tcp; pub(super) use tcp::physical_transport; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 1018b1626..f4e3630d5 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -499,7 +499,7 @@ mod reads; pub(crate) use reads::{coherent as coherent_reads, lifecycle as query_lifecycle}; #[path = "cell_group_sql_apply.rs"] -mod apply; +pub(super) mod apply; pub(crate) use apply::apply_images; #[path = "cell_group_sql_apply_faults.rs"] diff --git a/crates/cellar/tests/support/cell_group_sql_apply.rs b/crates/cellar/tests/support/cell_group_sql_apply.rs index d0aba067e..1c99b3440 100644 --- a/crates/cellar/tests/support/cell_group_sql_apply.rs +++ b/crates/cellar/tests/support/cell_group_sql_apply.rs @@ -53,7 +53,7 @@ pub(super) async fn install( Ok(()) } -pub(super) fn enrollment_bytes( +pub(in crate::runtime_fixture) fn enrollment_bytes( route: &GroupRoute, cell: &CellHistory, bytes: &[u8], diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index b83dfe175..cbf1d6127 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1575,10 +1575,14 @@ "crates/cellar/durability/raft/", "crates/cellar-ltx/src/", "crates/cellar/tests/native_cell_groups_execution.rs", - "crates/cellar/tests/support/cell_group_checkpoint.rs" + "crates/cellar/tests/support/cell_group_checkpoint.rs", + "crates/cellar/tests/support/cell_group_checkpoint_receive.rs", + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/tests/support/cell_group_sql_apply.rs" ], "creates": [ - "crates/cellar/durability/raft/cell_groups/checkpoint.rs", + "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", "crates/cellar/tests/native_cell_groups_snapshot.rs" ], "test_target": "native_cell_groups_snapshot", From cb164e025e129330f576450d214cef0e8fe92aa1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 14:39:52 -0700 Subject: [PATCH 121/568] fix: reverify cached checkpoint destination images --- crates/cellar-ltx/src/generation_receive.rs | 18 +++++++- .../support/cell_group_checkpoint_receive.rs | 45 +++++++++++++++++++ 2 files changed, 62 insertions(+), 1 deletion(-) diff --git a/crates/cellar-ltx/src/generation_receive.rs b/crates/cellar-ltx/src/generation_receive.rs index 5f9c4a591..85c1a2ab7 100644 --- a/crates/cellar-ltx/src/generation_receive.rs +++ b/crates/cellar-ltx/src/generation_receive.rs @@ -78,7 +78,23 @@ impl GenerationReceive { apply::inspect(&mut self.file, &self.expected, None)?; self.file.sync_all()?; match self.store.pin_prepared(self.id, self.expected) { - Ok(pin) => return Ok(pin), + Ok(pin) => { + // pin_prepared may reuse an in-memory seal. A transfer must + // independently inspect the existing destination as well as + // the incoming spool, while the pin prevents reclamation. + let mut destination = self + .store + .inner + .filesystem + .open(&self.store.inner.directory.join(final_name(self.id)))?; + apply::inspect(&mut destination, &self.expected, None)?; + destination.sync_all()?; + self.store + .inner + .filesystem + .sync_all(&self.store.inner.directory)?; + return Ok(pin); + } Err(GenerationError::Missing) => {} Err(error) => return Err(error), } diff --git a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs index ac260636d..53eb560f4 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs @@ -443,6 +443,51 @@ pub(crate) async fn refuse_checkpoint() -> Result<()> { assert_eq!(scratch.usage()?.reserved_files, 0); assert_eq!(pool.usage().sources, 0); } + if !rooted { + // Keep the existing image seal cached while corrupting only this + // disposable, unselected destination. Fresh incoming bytes must not + // make the stale cached seal an install proof. + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut receiver = + pool.receive(applier.clone(), scratch()?, cluster._root.path().into())?; + source.stream(|chunk| receiver.append(chunk))?; + drop(source); + let prepared = receiver.finish()?; + let (_, CellImagePin::Materialized(pin)) = prepared + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("alpha image")? + else { + anyhow::bail!("materialized image required"); + }; + drop(prepared); + let path = cluster + ._root + .path() + .join("node-3/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))); + let mut file = std::fs::OpenOptions::new().write(true).open(path)?; + std::io::Write::write_all(&mut file, b"corrupt")?; + file.sync_all()?; + drop(file); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?; + let mut receiver = + pool.receive(applier.clone(), scratch()?, cluster._root.path().into())?; + assert!( + source.stream(|chunk| receiver.append(chunk)).is_err(), + "cached destination corruption must refuse transfer" + ); + assert!(receiver.finish().is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(pool.usage().sources, 0); + drop(pin); + } drop(applier); drop(group); cluster.close().await?; From 7e4f54b20b0abb1deac02dae4f0c646c91223a98 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 15:00:36 -0700 Subject: [PATCH 122/568] docs: record CG-05 checkpoint receiver qualification --- docs/design/multi-raft/cell-groups/README.md | 9 ++-- .../multi-raft/cell-groups/baseline.json | 10 ++-- docs/design/multi-raft/cell-groups/handoff.md | 30 +++++++++-- .../design/multi-raft/cell-groups/progress.md | 54 ++++++++++++++++++- 4 files changed, 89 insertions(+), 14 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 914625f79..144038d71 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -5,9 +5,9 @@ Status: executable design and implementation plan delivered on 2026-10-29. CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and CG-04 shared execution have passed their scoped local evidence gates. -CG-05 is in progress: bounded complete-group checkpoint capture and streaming -pass their source-side regression gate. Receiver install and membership recovery -remain open. The production host has not yet cut over to shared groups. +CG-05 is in progress: bounded checkpoint capture, receiver verification and atomic +storage installation pass their scoped regression gates. Raft snapshot transport +and membership recovery remain open. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -50,7 +50,8 @@ This checkout retains evidence under the ignored checkout must receive these retained directories at the same relative paths before validating the four DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. CG-05's partial source checkpoint is retained in -`.cellar/cell-groups-evidence/CG-05-checkpoint-source/`; it does not complete CG-05. +`.cellar/cell-groups-evidence/CG-05-checkpoint-source/` and the receiver slice in +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`; neither completes CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index cb7021b6f..13aa2ea7e 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "69e41e865c267dff1dec5bce6f2b3d0966be09db", + "source_revision": "cb164e025e129330f576450d214cef0e8fe92aa1", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "ced64a028c84a88887a716b94ee3077382b6c4956ecf2fae39b1e2565c44e292", - "line": 419, - "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n}\n" + "sha256": "a4d5be18b6165bb60890dd35223d0bc062ee40577c68baf71d5048e46e6c63f9", + "line": 438, + "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index fae6a7c06..9031a72ed 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -26,13 +26,34 @@ CG-05 is now IN_PROGRESS. Its native source captures an exact selected complete root, configuration and membership, and retains an admitted immutable closure through bounded streaming and cancellation. Both image forms and empty enrollment are represented. See the latest [progress](progress.md) for source evidence and -limits. Continue with bounded receiver staging and verified atomic engine -installation, then whole-group transport and learner/membership recovery. CG-06 follows with archival reclamation. Automatic +limits. Receiver verification and atomic storage installation now pass their +partial gate, including cold recovery of the current and older checkpoint roots. +Continue with exact Raft snapshot tickets and bounded snapshot transport, then +whole-group learner/membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. +The receiver checkpoint at `cb164e02` passes 226 selected Rust cases, +9 gate commands and 37 validator tests. Its evidence binds 295 source hashes +and 6 copied executables under +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`. Three native cases cover +complete receive/install, both representations and empty cells, held engine sync, +transfer refusal/bounds, later real quorum writes and joined storage reopen. +A missing checkpoint-only page refuses recovery while the current tree remains +complete. A reproduced cached-destination corruption gap is fixed by independent +destination reread/sync. Preserve failed and interrupted diagnostic evidence. + +The most critical gap is still the Raft protocol boundary: `CellGroupOwner` rejects +`MsgSnapshot` and nonempty Ready snapshots. Bind a received complete closure to the +exact accepted snapshot effect before persistence, then acknowledge only after the +engine fence. Integrate this through bounded controller/transport ownership. +Qualify receiver crash/sync-error/cancellation and checkpoint root-slot exhaustion, +then learner catch-up, joint membership and lost-disk replacement. The storage +capability grants no voter or serving authority; do not bypass the existing +refusals or count these direct engine tests as full snapshot-RPC qualification. + The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands and 37 validator tests. Transfer `.cellar/cell-groups-evidence/CG-05-checkpoint-source/` with its predecessors; @@ -40,8 +61,9 @@ its record binds 290 source hashes and three copied executables. The four new cases cover concurrent real quorum writes while streaming a held cut, both image representations, empty enrollment, obsolete-page retention/release, bounds, cancelled physical ownership, stale/foreign/unpublished roots, corruption and -closed engines. These are embedded source tests; receiver install, snapshot RPC, -membership recovery and new process/RustFS/load qualification remain open. +closed engines. Those are the preceding embedded source tests; the receiver slice above extends +storage coverage. Snapshot RPC, membership recovery and new process/RustFS/load +qualification remain open. The following records describe the preceding incremental checkpoints. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 189d549ac..8c6a868be 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 verified receiver and atomic installation + +Qualified source: `cb164e025e129330f576450d214cef0e8fe92aa1`. All 226 selected Rust cases pass across shared +execution/catalog/storage, existing single-cell storage/snapshot suites and the +LTX library. All 9 gate commands pass, including all-target Clippy for +Cellar, its pure core and LTX, formatting/layout, dependency isolation, docs, +source preflight and all 37 plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/checkpoint.json` binds +295 source hashes, 6 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +A receiver now stages ordered bounded parts against the destination engine's +selected base. It checks route/history/cut/membership, every manifest page and +independently sealed image, then verifies and syncs the complete tree. Flat images +use one admitted scratch spool; extent objects use bounded buffers. Repeated flat +transfers reread and sync the existing destination even when its seal is cached. +A regression reproduced acceptance of a corrupt cached destination before this +fix; that failure and the interrupted candidate gate are retained separately. + +The prepared checkpoint owns the complete destination closure and receive +reservation. The engine's group snapshot variant atomically selects the snapshot, +configuration, membership and applied root behind the shared durability fence. +Mixed contracts, stale/conflicting bases and incomplete transfers refuse. A +separate retained checkpoint root survives later ordinary apply; recovery must +verify both it and the current applied tree before startup or reclamation. + +Three new cases use a lagging existing voter of the real embedded three-voter +fixture. Both image forms and canonical empty enrollment survive byte-for-byte +transfer, a held engine sync, subsequent quorum writes through the production +apply pool, joined shutdown of all prior owners, and cold reopen of the receiver. Before +publication, the selected root stays unchanged and owner startup refuses. The +receiver reservation remains charged after the caller drops its prepared handle +while engine sync is held. A checkpoint-only page is proved absent from the +current tree, then deleted: recovery refuses despite a complete current tree. +Restoring it permits storage startup without renewing quorum authority. Truncated, +noncontiguous, corrupt, foreign-history, oversized and scratch-limited transfers +leave the prior marker selected; existing cached destination corruption also +refuses. Extent receiving does not require a materialized-image scratch spool. + +The tests invoke the verified engine installation boundary directly. They do not +qualify Raft Ready snapshot acceptance or snapshot RPC, fresh learners, joint +configuration, disk replacement, receiver process-kill/sync-error/cancelled-waiter +cuts, checkpoint root-slot exhaustion, shared-group RustFS, load or zero-quorum DR. +No full CG-05 acceptance case has been replaced with these narrower cases. + +Critical next work is to bind the receiver result to the exact Raft snapshot +effect, integrate bounded snapshot transport and completion, and qualify peer +catch-up plus crash/cancellation/root-slot boundaries. Then deliver whole-group +learner/joint membership and lost-disk recovery and run all five declared CG-05 +acceptance cases. CG-06 archival reclamation follows that task; public host/SDK +cutover and the later topology, hibernation, DR and legacy-removal gates remain open. + ## CG-05 complete-root capture and bounded streaming Qualified source: `b15da415c9c9c0663274d0aedd520feb37d90d2d`. All 138 selected Rust cases pass across @@ -42,7 +94,7 @@ fixture. The five full snapshot acceptance cases remain unchanged and outstandin The shared fixture additionally handles canonical empty enrollment; existing execution regression checks cover that helper change. -Critical gaps for the next turn: build the receiver's bounded staged closure, +Gaps recorded at this source checkpoint: build the receiver's bounded staged closure, verify/sync every dependency, and publish snapshot metadata plus the group root atomically through the engine. Then connect whole-group transport, learner catch-up, joint membership and lost-disk replacement. The current header denotes From 096cfb132397d9588ba9138b1cf474210fc062f3 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 15:23:29 -0700 Subject: [PATCH 123/568] feat: bind group checkpoint restore to Raft Ready persistence --- .../durability/raft/cell_groups/README.md | 17 +- .../raft/cell_groups/controller/worker.rs | 2 +- .../cellar/durability/raft/cell_groups/mod.rs | 2 +- .../durability/raft/cell_groups/owner.rs | 182 ++++++++- .../raft/cell_groups/runtime/mod.rs | 32 ++ .../raft/cell_groups/runtime/worker.rs | 11 +- .../tests/native_cell_groups_execution.rs | 41 ++ .../tests/support/cell_group_checkpoint.rs | 18 +- .../support/cell_group_checkpoint_receive.rs | 4 +- .../tests/support/cell_group_runtime.rs | 9 +- .../support/cell_group_snapshot_owner.rs | 379 ++++++++++++++++++ 11 files changed, 671 insertions(+), 26 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_snapshot_owner.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 12f110309..a0e229039 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -157,9 +157,20 @@ current/pending publication slots. Recovery reconstructs both complete closures; a missing older checkpoint refuses startup and reclamation even when the current applied root is complete. Physical recovery never renews quorum authority. -These are verification APIs. The shared runtime still rejects incoming Raft -snapshots and configuration changes. Snapshot transport, owner completion tickets, -learner catch-up, joint membership and disk replacement remain CG-05 work. +These are verification APIs. Ordinary peer ingress still rejects snapshot +messages. `step_checkpoint` on the owner/runtime accepts an authenticated envelope +only with its exact receiver-local closure, engine/page owner, selected base and +membership binding. It requires settled prior apply and delegates restoration to +RawNode. Admission returns `Restoring` or `Ignored`; neither is a persistence +receipt. A matching log can advance commit and replay entries without installing +the received image. A restored snapshot is attached to its exact Ready write; +only the matching engine completion advances the application view and releases +Raft responses and `CheckpointInstalled`. Queued work retains the receiver budget +through engine sync and runtime shutdown. + +Source-side local checkpoint publication, snapshot RPC transport, learner catch-up, +joint membership and disk replacement remain CG-05 work. Configuration entries +remain closed. The explicit snapshot API does not authorize a new voter. ## Enrollment admission and ordered preparation diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 7b8319007..8b6a4eeb0 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -264,7 +264,7 @@ impl Worker { let work = event.retain_apply().ok_or(CellGroupControllerError::Invalid("apply ticket"))?; self.pending.push_back(Pending { group, work: Work::Prepare(Box::new(work)), retry_at: now }); } - CellGroupEffect::Applied { .. } => { + CellGroupEffect::Applied { .. } | CellGroupEffect::CheckpointInstalled { .. } => { // Only this post-sync event advances the published count. self.shared.applied.fetch_add(1, Ordering::Relaxed); self.progress.send_replace(()); diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index ef6d92285..363bd77a1 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -83,7 +83,7 @@ pub use root::{GroupManifest, PreparedGroupManifest}; pub use owner::{ CellGroupApply, CellGroupEffect, CellGroupOwner, CellGroupPersist, CellGroupPersisted, - CellGroupProposal, CellGroupStatus, CellGroupTicket, + CellGroupProposal, CellGroupSnapshotAdmission, CellGroupStatus, CellGroupTicket, }; pub(crate) use dispatch::RemoteMutationRoute; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index ee01b7963..88d1b85c1 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -3,12 +3,14 @@ //! One physical group owns one RawNode. The node scheduler executes the emitted //! effects; this owner performs no image I/O and starts no per-cell tasks. -use super::{CellGroupBootstrapPermit, GroupManifest, PreparedGroupManifest}; +use super::{ + CellGroupBootstrapPermit, GroupManifest, PreparedCellGroupCheckpoint, PreparedGroupManifest, +}; use crate::application::consensus::group_protocol::{ CellGroupEntry, GroupCommand, GroupEntryLimits, }; use crate::raft::{ - cache::{ConsensusStorage, GroupLog, GroupWrite}, + cache::{ConsensusStorage, GroupLog, GroupWrite, SnapshotChange}, completion::{entry_digest, entry_position, RuntimeError}, options::RaftRuntimeOptions, storage::ReplicaLog, @@ -95,8 +97,15 @@ pub struct CellGroupApply { pub base: PreparedGroupManifest, } -/// A ReadIndex result is a barrier to wait for, not permission to serve a cell. -/// The dispatcher must recheck owner/term and pin an applied coherent cell view. +/// Snapshot admission does not establish persistence or serving authority. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum CellGroupSnapshotAdmission { + /// Raft selected the verified snapshot; persistence is still outstanding. + Restoring, + /// Raft retained its log. Any resulting hard state/messages still need sync. + Ignored, +} + pub enum CellGroupEffect { Persist(Box), Message(Message), @@ -105,6 +114,11 @@ pub enum CellGroupEffect { ticket: CellGroupTicket, manifest: Box, }, + CheckpointInstalled { + manifest: Box, + }, + /// A barrier to wait for, not permission to serve a cell. The dispatcher + /// must recheck owner/term and pin an applied coherent cell view. ReadIndex { term: Term, index: LogIndex, @@ -119,7 +133,7 @@ impl CellGroupEffect { Self::Message(message) => message.compute_size() as usize, Self::Apply(work) => work.entry.compute_size() as usize + work.base.reserved_bytes(), Self::ReadIndex { context, .. } => context.len(), - Self::Applied { .. } => 1024, + Self::Applied { .. } | Self::CheckpointInstalled { .. } => 1024, } } } @@ -153,6 +167,7 @@ pub struct CellGroupOwner { group: GroupLog, node: RawNode, current: PreparedGroupManifest, + checkpoint: Option, options: RaftRuntimeOptions, phase: Option, waiting: Option, @@ -172,7 +187,8 @@ impl CellGroupOwner { /// Attach only a claimed initial voter with a completely verified selected /// root. The catalog permit is revalidated; serving additionally waits for /// the current term's committed/applied Raft barrier. Managed members and - /// incoming snapshots remain closed until their complete protocols exist. + /// ordinary snapshot messages remain closed; step_checkpoint requires the + /// exact independently verified destination closure. pub async fn start( permit: CellGroupBootstrapPermit, current: PreparedGroupManifest, @@ -238,6 +254,7 @@ impl CellGroupOwner { group, node, current, + checkpoint: None, proposal_budget: crate::raft::budget::Budget::new( options.max_request_bytes, options.max_requests, @@ -389,6 +406,7 @@ impl CellGroupOwner { committed: LogIndex::new(cache.hard_state.commit), ready: self.node.raft.state == StateRole::Leader && self.phase.is_none() + && self.checkpoint.is_none() && self.apply_bytes < self.options.max_group_apply_bytes && self.current.marker().position().term.get() == self.node.raft.term && cache.hard_state.term == self.node.raft.term @@ -398,7 +416,7 @@ impl CellGroupOwner { pub fn campaign(&mut self) -> Result<(), RuntimeError> { self.live()?; - if self.phase.is_some() { + if self.phase.is_some() || self.checkpoint.is_some() { return Err(RuntimeError::NotReady); } let result = self.node.campaign(); @@ -422,7 +440,7 @@ impl CellGroupOwner { pub(super) fn report_unreachable(&mut self, member: u64) -> Result<(), RuntimeError> { self.live()?; - if self.phase.is_some() { + if self.phase.is_some() || self.checkpoint.is_some() { return Err(RuntimeError::NotReady); } if member == 0 || self.node.raft.prs().get(member).is_none() { @@ -432,9 +450,70 @@ impl CellGroupOwner { Ok(()) } + /// Admit an authenticated snapshot envelope only with its exact complete + /// receiver-local closure. This returns protocol admission, never a durable + /// acknowledgement. Snapshot messages are emitted after the Ready fence. + pub fn step_checkpoint( + &mut self, + message: Message, + checkpoint: PreparedCellGroupCheckpoint, + ) -> Result { + self.live()?; + if !self.idle() || self.applying.is_some() || !self.apply.is_empty() { + return Err(RuntimeError::NotReady); + } + self.validate_peer_message(&message, true)?; + let root = checkpoint.manifest(); + if message.get_msg_type() != MessageType::MsgSnapshot + || message.get_snapshot() != checkpoint.snapshot() + || checkpoint.marker().position().index.get() >= u64::MAX - 2 + || checkpoint.marker().position().term.get() >= u64::MAX - 2 + || message.term < self.node.raft.term + || message.term < checkpoint.marker().position().term.get() + || root.base.as_ref() != Some(self.current.marker()) + || root.marker().route() != self.current.marker().route() + || !Arc::ptr_eq(&root.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&root.page_owner, &self.current.page_owner) + || !checkpoint.membership().permits_group(self.group.identity()) + || root.marker().position().index < self.current.marker().position().index + || root.marker().root().cells() > self.options.max_cells_per_group + || crate::raft::member_state::members( + checkpoint.snapshot().get_metadata().get_conf_state(), + ) + .len() + > self.options.max_members_per_group + { + return Err(RuntimeError::Invalid( + "checkpoint envelope/owner/base".into(), + )); + } + if root + .reserved_bytes() + .saturating_add(message.compute_size() as usize) + .saturating_add(4096) + > self.options.max_ready_bytes + { + return Err(RuntimeError::Overloaded); + } + let result = self.node.step(message); + self.refresh_execution_fence(); + result?; + if let Some(snapshot) = self.node.snap() { + if snapshot != checkpoint.snapshot() { + self.failed = Some(RuntimeError::StaleCompletion); + self.revoke_execution(); + return Err(RuntimeError::StaleCompletion); + } + self.checkpoint = Some(checkpoint); + Ok(CellGroupSnapshotAdmission::Restoring) + } else { + Ok(CellGroupSnapshotAdmission::Ignored) + } + } + pub fn step(&mut self, message: Message) -> Result<(), RuntimeError> { self.live()?; - if self.phase.is_some() { + if self.phase.is_some() || self.checkpoint.is_some() { return Err(RuntimeError::NotReady); } if self.apply_bytes >= self.options.max_group_apply_bytes { @@ -604,7 +683,7 @@ impl CellGroupOwner { next: PreparedGroupManifest, ) -> Result<(), RuntimeError> { self.live()?; - if self.phase.is_some() { + if self.phase.is_some() || self.checkpoint.is_some() { return Err(RuntimeError::NotReady); } if self.applying.as_ref().is_none_or(|t| !t.matches(&ticket)) @@ -644,6 +723,7 @@ impl CellGroupOwner { pub(super) fn idle(&self) -> bool { self.phase.is_none() + && self.checkpoint.is_none() && self.waiting.is_none() && self.effects.is_empty() && !self.node.has_ready() @@ -697,6 +777,9 @@ impl CellGroupOwner { }; ( ready + + self.checkpoint.as_ref().map_or(0, |c| { + c.manifest().reserved_bytes() + c.snapshot().compute_size() as usize + }) + self .effects .iter() @@ -821,6 +904,27 @@ impl CellGroupOwner { } match self.phase.take() { Some(Phase::Ready(mut ready, term)) => { + let installed = if !ready.snapshot().is_empty() { + let checkpoint = self + .checkpoint + .take() + .ok_or(RuntimeError::StaleCompletion)?; + if checkpoint.snapshot() != ready.snapshot() + || self.group.group_manifest()?.as_ref() != Some(checkpoint.marker()) + || self.group.checkpoint_manifest()?.as_ref() + != Some(checkpoint.marker()) + { + return Err(RuntimeError::StaleCompletion); + } + checkpoint + .manifest() + .require_checkpoint(&self.group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + self.current = checkpoint.manifest().clone(); + Some(self.current.marker().position().index.get()) + } else { + None + }; self.messages(ready.take_messages())?; self.messages(ready.take_persisted_messages())?; let reads = ready.take_read_states(); @@ -849,6 +953,25 @@ impl CellGroupOwner { } self.enqueue(ready.take_committed_entries())?; let light = self.node.advance_append(*ready); + if let Some(index) = installed { + self.node.advance_apply_to(index); + self.proposals.clear(); + if self.effects.len() >= self.options.max_events + || self + .effects + .iter() + .map(CellGroupEffect::bytes) + .sum::() + .saturating_add(5120) + > self.options.max_event_bytes + { + return Err(RuntimeError::Overloaded); + } + self.effects + .push_back(CellGroupEffect::CheckpointInstalled { + manifest: Box::new(self.current.marker().clone()), + }); + } let commit = light.commit_index(); self.phase = Some(Phase::Light(light)); if let Some(commit) = commit { @@ -906,7 +1029,12 @@ impl CellGroupOwner { && self.apply_bytes < self.options.max_group_apply_bytes { let ready = self.node.ready(); - if !ready.snapshot().is_empty() { + if !ready.snapshot().is_empty() + && self + .checkpoint + .as_ref() + .is_none_or(|c| c.snapshot() != ready.snapshot()) + { return Err(RuntimeError::Invalid( "group snapshot is not attached".into(), )); @@ -927,14 +1055,29 @@ impl CellGroupOwner { .read_states() .iter() .map(|r| r.request_ctx.len() + 128) - .sum::(); + .sum::() + + ready.snapshot().compute_size() as usize + + self + .checkpoint + .as_ref() + .map_or(0, |c| c.manifest().reserved_bytes()); if size > self.options.max_ready_bytes { return Err(RuntimeError::Overloaded); } - let changed = !ready.entries().is_empty() || ready.hs().is_some(); + let changed = !ready.entries().is_empty() + || ready.hs().is_some() + || !ready.snapshot().is_empty(); let mut write = GroupWrite::new(&self.group); write.entries = ready.entries().clone(); write.hard_state = ready.hs().cloned(); + if !ready.snapshot().is_empty() { + write.snapshot = Some(SnapshotChange::CellGroup( + self.checkpoint + .as_ref() + .ok_or(RuntimeError::StaleCompletion)? + .clone(), + )); + } self.phase = Some(Phase::Ready( Box::new(ready), Term::new(self.node.raft.term), @@ -1043,6 +1186,14 @@ impl CellGroupOwner { } pub(super) fn validate_message(&self, message: &Message) -> Result<(), RuntimeError> { + self.validate_peer_message(message, false) + } + + fn validate_peer_message( + &self, + message: &Message, + checkpoint: bool, + ) -> Result<(), RuntimeError> { let invalid = || RuntimeError::Invalid("cell-group peer message".into()); let kind = message.get_msg_type(); if message.to != self.group.identity().member.get() @@ -1052,9 +1203,10 @@ impl CellGroupOwner { || ::raft::raw_node::is_local_msg(kind) || matches!( kind, - MessageType::MsgPropose | MessageType::MsgTransferLeader | MessageType::MsgSnapshot + MessageType::MsgPropose | MessageType::MsgTransferLeader ) - || !message.get_snapshot().is_empty() + || (!checkpoint + && (kind == MessageType::MsgSnapshot || !message.get_snapshot().is_empty())) || message.entries.len() > 65_536 || message.compute_size() as usize > self.options.max_message_bytes || [ diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 83750d190..ec8d6ee9b 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -98,6 +98,7 @@ enum Command { EnrollmentState(cellar_logic::cell_group::CellHistory), Reconciliation(cellar_logic::cell_group::CellHistory), Step(Message), + Checkpoint(Message, PreparedCellGroupCheckpoint), AdmitStep(Message), Propose(Vec), ProposeAtTerm(cellar_logic::consensus::Term, Vec), @@ -115,6 +116,7 @@ enum Command { enum Response { Unit, + Checkpoint(CellGroupSnapshotAdmission), Proposal(CellGroupProposal), Submission(crate::raft::execution::CellSubmissionOutcome), Execution(Box), @@ -527,6 +529,36 @@ impl CellGroupRuntime { .map(|_| ()) } + /// Queue a verified snapshot on this group owner. Admission is not a + /// persistence receipt; only its post-fence Raft response acknowledges it. + pub async fn step_checkpoint( + &self, + history: GroupHistory, + message: Message, + checkpoint: PreparedCellGroupCheckpoint, + ) -> Result { + let size = message.compute_size() as usize; + if size > self.inner.options.max_message_bytes { + return Err(RuntimeError::Overloaded); + } + let bytes = checkpoint + .manifest() + .reserved_bytes() + .saturating_add(size) + .saturating_add(4096); + match self + .call( + Some(history), + Command::Checkpoint(message, checkpoint), + bytes, + ) + .await? + { + Response::Checkpoint(outcome) => Ok(outcome), + _ => Err(RuntimeError::Protocol("checkpoint admission reply".into())), + } + } + pub async fn propose( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index ecbc4044e..9909a5ef9 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -370,6 +370,10 @@ impl Worker { Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } + Command::Checkpoint(message, checkpoint) => group + .owner + .step_checkpoint(message, checkpoint) + .map(Response::Checkpoint), Command::Step(message) => { group.owner.step(message).map(|()| Response::Unit) } @@ -426,8 +430,11 @@ impl Worker { if group.pending.is_none() { group.pending = Some(Ok(effect)); } else { - if !matches!(effect, CellGroupEffect::Applied { .. }) - || group.settled.is_some() + if !matches!( + effect, + CellGroupEffect::Applied { .. } + | CellGroupEffect::CheckpointInstalled { .. } + ) || group.settled.is_some() { return Err(RuntimeError::Protocol( "reserved physical completion output".into(), diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 8d0509761..44cff6e64 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,31 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_rejects_overflow_cuts_before_raft() -> Result<()> { + runtime_fixture::checkpoint_bounds().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_replays_matching_log_without_install() -> Result<()> { + runtime_fixture::checkpoint_matching_log().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_restores_only_after_raft_sync() -> Result<()> { + runtime_fixture::checkpoint_owner(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_owner_restores_extents_only_after_raft_sync() -> Result<()> { + runtime_fixture::checkpoint_owner(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_runtime_joins_accepted_snapshot_shutdown() -> Result<()> { + runtime_fixture::checkpoint_shutdown().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_receive_installs_and_recovers_materialized_closure() -> Result<()> { runtime_fixture::install_checkpoint(false).await @@ -429,9 +454,16 @@ impl Cluster { } async fn drive(&mut self, isolated: bool) -> Result<()> { + self.drive_partition(isolated, None).await + } + + async fn drive_partition(&mut self, isolated: bool, excluded: Option) -> Result<()> { for _ in 0..2000 { let mut progress = false; for (n, voter) in self.voters.iter_mut().enumerate() { + if excluded == Some(n) { + continue; + } let event = voter.owner.as_mut().unwrap().poll()?; let Some(event) = event else { continue }; progress = true; @@ -464,6 +496,12 @@ impl Cluster { Some(manifest.as_ref()) ); } + CellGroupEffect::CheckpointInstalled { manifest } => { + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(manifest.as_ref()) + ); + } CellGroupEffect::ReadIndex { term, index, @@ -474,6 +512,9 @@ impl Cluster { for _ in 0..self.network.len() { let message = self.network.pop_front().unwrap(); let target = message.to as usize - 1; + if excluded == Some(target) || excluded == Some(message.from as usize - 1) { + continue; + } match self.voters[target] .owner .as_mut() diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs index a0e578ae3..2bac46cbf 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -27,6 +27,20 @@ async fn populated(rooted: bool) -> Result<(Cluster, BTreeMap>)> pub(super) async fn populated_with_lagging( rooted: bool, lagging: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate(rooted, lagging, false).await +} + +pub(super) async fn populated_partition( + rooted: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate(rooted, false, true).await +} + +async fn populate( + rooted: bool, + lagging: bool, + partition: bool, ) -> Result<(Cluster, BTreeMap>)> { let mut cluster = Cluster::new().await?; cluster.elect().await?; @@ -86,7 +100,9 @@ pub(super) async fn populated_with_lagging( entry.encode(&route, GroupEntryLimits::default())? }; cluster.voters[0].owner.as_mut().unwrap().propose(encoded)?; - cluster.drive(false).await?; + cluster + .drive_partition(false, partition.then_some(2)) + .await?; expected.insert(name.to_owned(), bytes); } Ok((cluster, expected)) diff --git a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs index 53eb560f4..59bb51fc8 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs @@ -9,14 +9,14 @@ use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; use raft::Storage; use std::path::Path; -fn scratch() -> Result { +pub(super) fn scratch() -> Result { Ok(ScratchPool::new(ScratchLimits { max_bytes: 32 * 1024 * 1024, max_files: 1, })?) } -fn config( +pub(super) fn config( images: GenerationStore, extents: Option, directory: &Path, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 25d1245c6..20b7fd56a 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -5,6 +5,12 @@ use super::*; use std::collections::BTreeMap; +#[path = "cell_group_snapshot_owner.rs"] +mod snapshot_owner; +pub(super) use snapshot_owner::{ + checkpoint_bounds, checkpoint_matching_log, checkpoint_owner, checkpoint_shutdown, +}; + #[path = "cell_group_checkpoint_receive.rs"] mod checkpoint_receive; pub(super) use checkpoint_receive::{install_checkpoint, refuse_checkpoint}; @@ -129,7 +135,8 @@ impl RuntimeCluster { self.runtimes[n].complete_apply(scope, work, next).await?; } } - CellGroupEffect::Applied { manifest, .. } => { + CellGroupEffect::Applied { manifest, .. } + | CellGroupEffect::CheckpointInstalled { manifest } => { self.applied.get_mut(&scope).context("applied")?[n] = manifest.position().index } diff --git a/crates/cellar/tests/support/cell_group_snapshot_owner.rs b/crates/cellar/tests/support/cell_group_snapshot_owner.rs new file mode 100644 index 000000000..db1dcf568 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_snapshot_owner.rs @@ -0,0 +1,379 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual RawNode restore/Ready acceptance on a partitioned existing voter. +//! Transfer uses the native streaming API, not yet the snapshot RPC transport. +use super::*; + +async fn prepared( + rooted: bool, +) -> Result<( + Cluster, + CellGroupApplier, + PreparedCellGroupCheckpoint, + CellGroupCheckpointPool, + Message, +)> { + prepared_at(rooted, None).await +} + +async fn prepared_at( + rooted: bool, + index: Option, +) -> Result<( + Cluster, + CellGroupApplier, + PreparedCellGroupCheckpoint, + CellGroupCheckpointPool, + Message, +)> { + let (mut cluster, _) = checkpoint_fixture::populated_partition(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old marker")?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source_owner = cluster.voters[0].owner.as_ref().unwrap(); + let mut source = source_pool.capture( + &cluster.voters[0].log.cell_group(&route.history)?, + source_owner.manifest(), + )?; + let term = source_owner.status()?.term.get(); + assert!(source.marker().position().index > before.position().index); + let config = checkpoint_receive::config( + cluster.voters[2].images.clone(), + cluster.voters[2].extents.clone(), + cluster._root.path(), + )?; + let applier = CellGroupApplier::new( + group, + cluster.voters[2].pages.take().context("pages")?, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut receiver = pool.receive( + applier.clone(), + checkpoint_receive::scratch()?, + cluster._root.path().into(), + )?; + let checkpoint = cluster + .domain + .blocking(move || { + source.stream(|chunk| { + if let Some(index) = + index.filter(|_| matches!(chunk.part, CellGroupCheckpointPart::Header)) + { + assert_eq!(chunk.offset, 0); + assert_eq!(chunk.length, chunk.bytes.len() as u64); + let mut header: serde_json::Value = + serde_json::from_slice(chunk.bytes).unwrap(); + header["root"]["index"] = serde_json::json!(index); + let bytes = serde_json::to_vec(&header).unwrap(); + receiver.append(CellGroupCheckpointChunk { + length: bytes.len() as u64, + bytes: &bytes, + ..chunk + }) + } else { + receiver.append(chunk) + } + })?; + receiver.finish() + })? + .await??; + let mut message = Message::default(); + message.set_msg_type(MessageType::MsgSnapshot); + message.from = 1; + message.to = 3; + message.term = term; + message.set_snapshot(checkpoint.snapshot().clone()); + Ok((cluster, applier, checkpoint, pool, message)) +} + +pub(crate) async fn checkpoint_owner(rooted: bool) -> Result<()> { + let (mut cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + assert!( + owner.step(message.clone()).is_err(), + "unverified peer ingress must stay closed" + ); + for variant in 0..4 { + let mut bad = message.clone(); + match variant { + 0 => bad.mut_snapshot().data = b"foreign descriptor".to_vec().into(), + 1 => bad.to = 2, + 2 => bad.from = 99, + _ => bad.term = 1, + } + assert!(owner.step_checkpoint(bad, checkpoint.clone()).is_err()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(owner.poll()?.is_none(), "refusal must not mutate Raft"); + } + assert_eq!( + owner.step_checkpoint(message.clone(), checkpoint.clone())?, + CellGroupSnapshotAdmission::Restoring + ); + drop(checkpoint); + assert!(matches!(owner.campaign(), Err(RuntimeError::NotReady))); + assert!(matches!(owner.step(message), Err(RuntimeError::NotReady))); + let Some(CellGroupEffect::Persist(work)) = owner.poll()? else { + anyhow::bail!("snapshot Ready persistence"); + }; + let gate = cluster.voters[2].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut pending = cluster.domain.spawn(work.run())?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!( + cluster.voters[2].owner.as_ref().unwrap().status()?.applied, + before.position() + ); + assert!( + cluster.voters[2].owner.as_mut().unwrap().poll()?.is_none(), + "snapshot acknowledgement escaped sync" + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + let completion = pending.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + assert_eq!( + owner.status()?.applied, + before.position(), + "engine receipt alone cannot advance RawNode" + ); + owner.complete_persist(completion)?; + let mut acknowledged = false; + let mut installed = false; + for _ in 0..20 { + match owner.poll()? { + Some(CellGroupEffect::Message(response)) => { + assert_eq!(response.get_msg_type(), MessageType::MsgAppendResponse); + assert_eq!( + (response.from, response.to, response.index), + (3, 1, cut.position().index.get()) + ); + acknowledged = true; + } + Some(CellGroupEffect::CheckpointInstalled { manifest }) => { + assert_eq!(manifest.as_ref(), &cut); + installed = true; + } + None => break, + _ => anyhow::bail!("unexpected snapshot completion"), + } + } + assert!(acknowledged && installed); + assert_eq!(owner.manifest().marker(), &cut); + assert_eq!(owner.manifest().pinned_images(), 3); + assert_eq!(pool.usage().sources, 0); + assert!(!owner.status()?.ready); + drop(group); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let later = history(&route, "after-raft-snapshot")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(17);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + sql::apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + assert_eq!( + group + .group_manifest()? + .context("later root")? + .root() + .cells(), + 4 + ); + drop(group); + fixture.close().await +} + +pub(crate) async fn checkpoint_shutdown() -> Result<()> { + for rooted in [false, true] { + let (cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + let gate = cluster.voters[2].gate.clone(); + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let release = gate.arm(fault::HOLD_SYNC); + assert_eq!( + fixture.runtimes[2] + .step_checkpoint(route.history, message, checkpoint) + .await?, + CellGroupSnapshotAdmission::Restoring + ); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[2].clone(); + let mut close = fixture + .cluster + .domain + .spawn(async move { runtime.close().await })?; + assert!(tokio::time::timeout(Duration::from_millis(30), &mut close) + .await + .is_err()); + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + drop(release); + close.await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + assert_eq!(pool.usage().sources, 0); + drop(applier); + drop(group); + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn checkpoint_bounds() -> Result<()> { + for index in [u64::MAX, u64::MAX - 1, u64::MAX - 2] { + let (mut cluster, applier, checkpoint, pool, message) = + prepared_at(false, Some(index)).await?; + let owner = cluster.voters[2].owner.as_mut().unwrap(); + let before = owner.manifest().marker().clone(); + assert_eq!(checkpoint.marker().position().index.get(), index); + assert!(matches!( + owner.step_checkpoint(message, checkpoint), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(owner.manifest().marker(), &before); + assert!(owner.poll()?.is_none()); + assert_eq!(pool.usage().sources, 0); + drop(applier); + cluster.close().await?; + } + Ok(()) +} + +pub(crate) async fn checkpoint_matching_log() -> Result<()> { + use raft::{GetEntriesContext, Storage}; + for rooted in [false, true] { + let (mut cluster, applier, checkpoint, pool, message) = prepared(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[2].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("old root")?; + let cut = checkpoint.marker().clone(); + // Store the leader's actual entries, but withhold its commit advance. + // Raft must recognize this log and replay it, not install the image. + let mut append = Message::default(); + append.set_msg_type(MessageType::MsgAppend); + append.from = 1; + append.to = 3; + append.term = message.term; + append.index = before.position().index.get(); + append.log_term = before.position().term.get(); + append.commit = before.position().index.get(); + append.entries = cluster.voters[0] + .log + .cell_group(&route.history)? + .entries( + append.index + 1, + cut.position().index.get() + 1, + None, + GetEntriesContext::empty(false), + )? + .into(); + let owner = cluster.voters[2].owner.as_mut().unwrap(); + owner.step(append)?; + loop { + match owner.poll()? { + Some(CellGroupEffect::Persist(work)) => { + owner.complete_persist(work.run().await?)? + } + Some(CellGroupEffect::Message(_)) => (), + None => break, + _ => anyhow::bail!("uncommitted entries must not apply"), + } + } + assert_eq!( + group.initial_state()?.hard_state.commit, + before.position().index.get() + ); + assert_eq!( + owner.step_checkpoint(message, checkpoint)?, + CellGroupSnapshotAdmission::Ignored + ); + assert_eq!(pool.usage().sources, 0); + let Some(CellGroupEffect::Persist(work)) = owner.poll()? else { + anyhow::bail!("commit must persist before acknowledgement"); + }; + assert!(owner.poll()?.is_none()); + owner.complete_persist(work.run().await?)?; + let Some(CellGroupEffect::Message(response)) = owner.poll()? else { + anyhow::bail!("ignored snapshot response"); + }; + assert_eq!(response.index, cut.position().index.get()); + assert_eq!(owner.manifest().marker(), &before); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + group.initial_state()?.hard_state.commit, + cut.position().index.get() + ); + // Restart RawNode ownership over the actual durable committed log and + // replay through the production image adapter on the shared workers. + drop(group); + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let group = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!( + group + .group_manifest()? + .context("replayed root")? + .root() + .cells(), + 3 + ); + assert_eq!( + group.group_manifest()?.context("replayed root")?.root(), + cut.root() + ); + drop(group); + fixture.close().await?; + } + Ok(()) +} From d174d4efcee638b9adfa251f6c94a1d49aea3fcf Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 15:37:00 -0700 Subject: [PATCH 124/568] docs: record CG-05 Raft snapshot owner qualification --- docs/design/multi-raft/cell-groups/README.md | 11 +- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 110 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 45 ++++++- docs/design/multi-raft/cell-groups/tasks.json | 3 +- 5 files changed, 113 insertions(+), 60 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 144038d71..7f0d9f221 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -5,9 +5,10 @@ Status: executable design and implementation plan delivered on 2026-10-29. CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and CG-04 shared execution have passed their scoped local evidence gates. -CG-05 is in progress: bounded checkpoint capture, receiver verification and atomic -storage installation pass their scoped regression gates. Raft snapshot transport -and membership recovery remain open. The production host has not yet cut over to shared groups. +CG-05 is in progress: checkpoint capture, receiver verification, atomic storage +installation and exact Raft Ready acceptance pass their scoped regression gates. +Source local checkpoint publication, snapshot transport and membership recovery +remain open. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -51,7 +52,9 @@ checkout must receive these retained directories at the same relative paths before validating the four DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. CG-05's partial source checkpoint is retained in `.cellar/cell-groups-evidence/CG-05-checkpoint-source/` and the receiver slice in -`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`; neither completes CG-05. +`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`, followed by owner/runtime +acceptance in `.cellar/cell-groups-evidence/CG-05-snapshot-owner/`. These partial +checkpoints do not complete CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 13aa2ea7e..7c8f6c94e 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "cb164e025e129330f576450d214cef0e8fe92aa1", + "source_revision": "096cfb132397d9588ba9138b1cf474210fc062f3", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 9031a72ed..b99d0961d 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -28,8 +28,9 @@ through bounded streaming and cancellation. Both image forms and empty enrollmen are represented. See the latest [progress](progress.md) for source evidence and limits. Receiver verification and atomic storage installation now pass their partial gate, including cold recovery of the current and older checkpoint roots. -Continue with exact Raft snapshot tickets and bounded snapshot transport, then -whole-group learner/membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic +Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate. +Continue with source local checkpoint publication and bounded snapshot transport, +then whole-group learner/membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. @@ -45,14 +46,24 @@ A missing checkpoint-only page refuses recovery while the current tree remains complete. A reproduced cached-destination corruption gap is fixed by independent destination reread/sync. Preserve failed and interrupted diagnostic evidence. -The most critical gap is still the Raft protocol boundary: `CellGroupOwner` rejects -`MsgSnapshot` and nonempty Ready snapshots. Bind a received complete closure to the -exact accepted snapshot effect before persistence, then acknowledge only after the -engine fence. Integrate this through bounded controller/transport ownership. -Qualify receiver crash/sync-error/cancellation and checkpoint root-slot exhaustion, +The latest owner/runtime checkpoint at `096cfb13` passes 146 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-snapshot-owner/` binds 296 source hashes +and 3 copied executables. Explicit `step_checkpoint` admits a verified +closure, binds the exact RawNode Ready to engine persistence, and advances applied +state and acknowledgement only after the matching completion. Matching retained +logs commit/replay without installing received roots. Both image forms, held +sync, invalid/overflow refusal, subsequent quorum writes and joined runtime close +are covered. The tests construct the envelope from a captured source; they do +not qualify Raft-generated outbound snapshots or snapshot RPC. + +The most critical gap is source-side local checkpoint publication followed by +bounded authenticated snapshot transfer and real peer catch-up. Preserve the +source log suffix: the receiver-install variant is not a local checkpoint +operation. Qualify receiver crash/sync-error/cancellation and root-slot exhaustion, then learner catch-up, joint membership and lost-disk replacement. The storage -capability grants no voter or serving authority; do not bypass the existing -refusals or count these direct engine tests as full snapshot-RPC qualification. +capability grants no new voter or serving authority. Ordinary peer snapshot +ingress stays closed until the verified receiver is wired through transport. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands and 37 validator tests. Transfer @@ -612,41 +623,40 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Deliver bounded host provisioning and lazy group birth. Registration and - group-local quarantine now have concrete ownership boundaries. The caller - still supplies the bootstrap permit, selected root, applier and physical route. - Connect catalog selection and initial namespace/range birth to bounded local - preparation, coalescing concurrent requests by physical GroupHistory rather - than allocating a group per cell. Transfer prepared owners through register(). - Use the explicitly provisioned non-overlapping ranges in design.md. Managed - fleet bootstrap currently refuses in directory/bootstrap.rs; preserve that - refusal until the rooted fleet authorization required by CG-08 exists. - Keep catalog authorization and current-term quorum separate from routing hints; - retain preparation/attachment after the initiating request disconnects. - Quarantined groups remain charged until joined restart; never bypass them with - an in-place replacement or discard unresolved producers to recover capacity. - Shared-engine faults must remain node-wide. Keep read/command grants distinct - and receipt absence ambiguous. Destination catalog caching still depends on - replicated transfer seals/admission invalidation in CG-09, not a TTL. - Public RaftCellHandle/discovery must use the actual replacement host, preserving - old initializer refusal until cutover. Quorum/S3 minima must refuse unsupported - archive guarantees until CG-06 supplies real proofs. -2. Complete CG-04's production retry/cache lifecycle and configuration-entry - handling, then its full gate before DONE. Use selected-root recovery at startup. - Whole-process kill/power-loss remains distinct from the qualified directory-sync - faults and joined cold restart. Full snapshots/membership follow in CG-05, - archive/reclamation in CG-06, and shared effect scheduling/cache lifecycle in - CG-08 before the CG-07 public cutover. -3. Remove `TransportRuntime::Legacy`, the typed cell-scoped frame arm and old - transport entry points in CG-16. Also remove `QueryVersion::GroupCut` and - `RaftClass::invoke_group_query` with the old host by CG-16. They explicitly preserve the old in-memory group observation - during integration; they must not become a permanent alternate serving mode. - Final `QueryContext` exposes the actual cell observation directly. Never derive - a cell sequence or witness from a group log position. - -The opportunity is to reuse the now-shared consensus, SQL, read and apply paths in -one dispatcher. Additional per-cell workers, a second production mode or a demo -that bypasses the registered API would increase integration work. +1. Complete CG-05 source publication and transport. Capture an exact applied cut, + retain its complete closure/configuration, publish a local checkpoint while + preserving every later entry, and make RawNode-generated snapshots stream that + same retained cut. Carry route/member authentication, budgets, cancellation, + retries and joined physical ownership through the controller/peer transport. + Admit the destination through `step_checkpoint`; use its post-persistence Raft + acknowledgement, never the admission return value, as completion. +2. Finish CG-05 membership and recovery. Qualify the outstanding receiver fault + cuts and root-slot pressure, then whole-group learner catch-up, joint membership + and lost-disk replacement. Run every exact acceptance case before DONE. +3. Deliver CG-06 archive/reclamation with verified complete checkpoints and + contiguous tails. Upload completion alone cannot authorize reclaiming local + recovery dependencies or an S3 durability acknowledgement. +4. Follow the declared dependency order through CG-08 and CG-07 for effect/cache + scheduling and production SDK/fleet cutover. Retain catalog and current-term + authorization, coalesced preparation, physical ownership after disconnect and + group-local quarantine. Shared storage uncertainty stays node-wide. A missing + receipt stays ambiguous; callbacks cannot be replayed automatically. +5. Continue topology, placement, hibernation and fresh-incarnation recovery gates. + Remove `TransportRuntime::Legacy`, cell-scoped transport, `QueryVersion::GroupCut` + and `RaftClass::invoke_group_query` with the old host in CG-16. These transitional + paths must not become permanent modes. Never derive cell sequence or witness + from a group log position. Final qualification reruns all gates at one revision. + +Start source publication in `checkpoint/mod.rs`, `cache.rs` and the group root +reservation path in `storage.rs` / `cell_groups/retention.rs`. The current capture +API accepts the current applied root; outbound transfer must also select the +retained checkpoint root after newer writes advance current state. Bind its +header/configuration to the snapshot returned by Raft Storage, not a fresh capture +of a later current root. Keep snapshot publication on the owner persistence lane. + +The opportunity is to reuse the existing shared consensus, SQL, read, apply and +provisioning paths. Avoid adding per-cell workers or another serving mode while +integrating the remaining protocols. ## Delivery sequence @@ -665,14 +675,10 @@ an old-format migration reader; preserve marker 1 with explicit domain separatio and refuse unsupported persisted layouts. Never delete existing user data as part of the hard cutover. -CG-04 needs its six core real-Raft cases, plus the admission and submission cases -listed in `tasks.json`: `three_cells_one_rawnode_per_voter`, -`concurrent_enrollment_one_cell`, `shared_group_fair_admission`, -`cell_receipt_not_group_progress`, `strong_read_at_group_barrier` and -`apply_prefix_never_skips_slow_cell`. Fixing a prototype's compilation or passing -a low-level owner test does not establish fair SQL execution, SDK dispatch, -receipt settlement or strong-read serving. Its complete brief prints the owning -paths and exact command/evidence gate. +CG-04's completed gate remains in its source-bound evidence. CG-05's complete +brief prints the five required acceptance cases, owning paths and exact commands. +The source, receiver and owner checkpoints above establish partial boundaries; +they cannot replace full snapshot, membership and lost-disk recovery qualification. ## Release boundary diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 8c6a868be..4ac5d9d59 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,49 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 Raft snapshot acceptance and persistence + +Qualified source: `096cfb132397d9588ba9138b1cf474210fc062f3`. All 146 selected Rust cases pass across shared +execution/catalog/storage. All 8 gate commands pass, including all-target +Clippy for Cellar/core/LTX, formatting/layout, dependency isolation, docs, +source preflight and 37 plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-snapshot-owner/checkpoint.json` binds +296 source hashes, 3 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +The explicit owner/runtime `step_checkpoint` API admits only a complete verified +receiver closure matching the snapshot envelope, route, selected base, engine, +page-store owner and permanent replica membership. Bounds and overflowing cuts +are checked before RawNode mutation. Ordinary peer ingress still rejects snapshot +messages. Admission reports whether Raft selected the snapshot or retained its +matching log; neither result is a persistence receipt or serving authority. + +A selected snapshot binds to the exact Ready and existing persistence ticket, +exclusive owner lease and shared engine fence. The old applied view remains until +the matching completion is consumed. Only then does the owner advance to the +verified cut and emit the acknowledgement and `CheckpointInstalled` event. +Controller publication observes that event. The prepared receiver and its budget +remain owned through queued sync and joined runtime shutdown. If Raft already has +the matching log, it persists the commit advance and replays entries without +selecting the received checkpoint. + +Five new native cases cover a partitioned existing voter, both image formats, +invalid envelopes and overflowing cuts, held sync with no early acknowledgement, +subsequent real quorum writes, matching-log replay and shutdown during accepted +snapshot persistence. The fixture builds the snapshot envelope from captured +source state and uses the native streaming callback. It does not exercise a +source RawNode-generated snapshot or authenticated snapshot RPC. + +The critical next gap is source-side local checkpoint publication that preserves +the unapplied log suffix, followed by bounded snapshot transport and actual Raft +peer catch-up. The receiver-install write must not be reused as a local leader +checkpoint: it replaces the received state and truncates prior entries. Then +qualify receiver process-kill/sync-error/cancellation and root-slot boundaries, +whole-group learners, joint membership and lost-disk recovery. All five full CG-05 +acceptance cases remain required. No new shared-group RustFS, load or zero-quorum +DR qualification is claimed. CG-06 archival/reclamation and the later public +cutover and legacy-removal tasks remain open. + ## CG-05 verified receiver and atomic installation Qualified source: `cb164e025e129330f576450d214cef0e8fe92aa1`. All 226 selected Rust cases pass across shared @@ -55,7 +98,7 @@ configuration, disk replacement, receiver process-kill/sync-error/cancelled-wait cuts, checkpoint root-slot exhaustion, shared-group RustFS, load or zero-quorum DR. No full CG-05 acceptance case has been replaced with these narrower cases. -Critical next work is to bind the receiver result to the exact Raft snapshot +Next work recorded at this receiver checkpoint was to bind its result to the exact Raft snapshot effect, integrate bounded snapshot transport and completion, and qualify peer catch-up plus crash/cancellation/root-slot boundaries. Then deliver whole-group learner/joint membership and lost-disk recovery and run all five declared CG-05 diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index cbf1d6127..2d92a6beb 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1579,7 +1579,8 @@ "crates/cellar/tests/support/cell_group_checkpoint_receive.rs", "crates/cellar/tests/support/cell_group_runtime.rs", "crates/cellar/tests/support/cell_group_sql.rs", - "crates/cellar/tests/support/cell_group_sql_apply.rs" + "crates/cellar/tests/support/cell_group_sql_apply.rs", + "crates/cellar/tests/support/cell_group_snapshot_owner.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 15:58:43 -0700 Subject: [PATCH 125/568] feat: publish retained group checkpoints without discarding log suffixes --- crates/cellar/durability/raft/cache.rs | 39 +- .../durability/raft/cell_groups/README.md | 20 +- .../raft/cell_groups/checkpoint/local.rs | 51 +++ .../raft/cell_groups/checkpoint/mod.rs | 89 +++- .../raft/cell_groups/controller/worker.rs | 4 + .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../durability/raft/cell_groups/owner.rs | 69 ++- .../durability/raft/cell_groups/retention.rs | 24 + .../durability/raft/cell_groups/root.rs | 20 + .../raft/cell_groups/runtime/mod.rs | 18 + .../raft/cell_groups/runtime/worker.rs | 5 + crates/cellar/durability/raft/storage.rs | 14 +- .../tests/native_cell_groups_execution.rs | 18 +- .../support/cell_group_checkpoint_local.rs | 429 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 22 +- 15 files changed, 797 insertions(+), 26 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/local.rs create mode 100644 crates/cellar/tests/support/cell_group_checkpoint_local.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 5f73b1017..934ef8690 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -94,6 +94,8 @@ pub enum SnapshotChange { /// A complete receiver-local cell-group closure. Its producer supplies no /// Raft authority; the consensus owner must bind this to its snapshot effect. CellGroup(super::cell_groups::PreparedCellGroupCheckpoint), + /// A captured exact local applied cut; retains all later entries and state. + CellGroupCheckpoint(super::cell_groups::PreparedLocalCellGroupCheckpoint), /// A verified incoming snapshot and its already durable state image. Install { snapshot: Snapshot, @@ -134,6 +136,7 @@ impl GroupWrite { .as_ref() .or_else(|| match &self.snapshot { Some(SnapshotChange::CellGroup(prepared)) => Some(prepared.root()), + Some(SnapshotChange::CellGroupCheckpoint(prepared)) => Some(prepared.root()), _ => None, }) } @@ -149,6 +152,7 @@ impl GroupWrite { let snapshot = match snapshot { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, SnapshotChange::CellGroup(prepared) => prepared.snapshot(), + SnapshotChange::CellGroupCheckpoint(prepared) => prepared.snapshot(), }; if snapshot.compute_size() > 64 * 1024 { return Err(LogError::Overloaded); @@ -576,7 +580,14 @@ impl CachedGroup { ) -> Result { let mut next = self.clone(); let group_install = matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))); - if group_install + let group_checkpoint = matches!( + &write.snapshot, + Some(SnapshotChange::CellGroupCheckpoint(_)) + ); + if group_checkpoint && (!write.entries.is_empty() || write.hard_state.is_some()) { + return Err(LogError::Invalid("mixed local checkpoint write".into())); + } + if (group_install || group_checkpoint) && (self.group_manifest.is_none() || write.group_manifest.is_some() || write.applied.is_some() @@ -586,7 +597,8 @@ impl CachedGroup { return Err(LogError::Invalid("mixed cell-group snapshot write".into())); } if (self.group_manifest.is_some() || write.group_manifest.is_some()) - && (write.applied.is_some() || (write.snapshot.is_some() && !group_install)) + && (write.applied.is_some() + || (write.snapshot.is_some() && !group_install && !group_checkpoint)) { return Err(LogError::Invalid( "mixed image/group apply or snapshot".into(), @@ -596,6 +608,7 @@ impl CachedGroup { let snapshot = match change { SnapshotChange::Checkpoint(s) | SnapshotChange::Install { snapshot: s, .. } => s, SnapshotChange::CellGroup(prepared) => prepared.snapshot(), + SnapshotChange::CellGroupCheckpoint(prepared) => prepared.snapshot(), }; let metadata = snapshot.get_metadata(); if metadata.index < self.snapshot.get_metadata().index @@ -634,6 +647,28 @@ impl CachedGroup { next.conf_state = metadata.get_conf_state().clone(); next.membership = prepared.membership().clone(); } + SnapshotChange::CellGroupCheckpoint(prepared) => { + let root = prepared.root(); + if !Arc::ptr_eq(&root.engine_owner, &write.group.owner) + || !root + .marker() + .matches_group(&history, write.group.birth_digest) + || self.group_manifest.as_ref() != Some(root.marker()) + || root.marker().position() != self.applied.position + || self.term(metadata.index) != Some(metadata.term) + || metadata.index != self.applied.position.index.get() + || metadata.index > self.hard_state.commit + || metadata.get_conf_state() != &self.conf_state + || prepared + .membership() + .map_err(|e| LogError::Invalid(e.to_string()))? + != self.membership + { + return Err(LogError::Invalid("local checkpoint binding/cut".into())); + } + next.entries = next.entries.split_off(&(metadata.index + 1)); + next.checkpoint_manifest = Some(root.marker().clone()); + } SnapshotChange::Checkpoint(_) => { if metadata.index > self.applied.position.index.get() || self.term(metadata.index) != Some(metadata.term) diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index a0e229039..53c736c2e 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -168,9 +168,23 @@ only the matching engine completion advances the application view and releases Raft responses and `CheckpointInstalled`. Queued work retains the receiver budget through engine sync and runtime shutdown. -Source-side local checkpoint publication, snapshot RPC transport, learner catch-up, -joint membership and disk replacement remain CG-05 work. Configuration entries -remain closed. The explicit snapshot API does not authorize a new voter. +An unstreamed captured source can become a `PreparedLocalCellGroupCheckpoint`. +`publish_checkpoint` admits only the owner's exact current applied cut and runs +publication on its persistence lane. The distinct `CellGroupCheckpoint` storage +operation preserves hard state, applied state and every later log entry; it +compacts only the prefix represented by the complete checkpoint. Mixed writes +and stale cuts refuse. `CheckpointPublished` is emitted after the matching engine +completion. Source admission and pins survive accepted publication and shutdown. + +`capture_snapshot` pins the retained checkpoint named by the exact Raft snapshot, +including its original membership/configuration, after ordinary apply has advanced +the current root. A concurrent checkpoint replacement may refuse capture; a +successful source keeps its immutable closure until streaming ends. Recovery +reconstructs this retained closure before it can become an outbound source. + +Snapshot RPC transport, learner catch-up, joint membership and disk replacement +remain CG-05 work. Configuration entries remain closed. Neither checkpoint API +authorizes a new voter or establishes S3 durability. ## Enrollment admission and ordered preparation diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs new file mode 100644 index 000000000..600efe51a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/local.rs @@ -0,0 +1,51 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::Snapshot; + +struct Inner { + source: CellGroupCheckpoint, + snapshot: Snapshot, +} + +/// A captured local cut. It retains source admission and physical pins through +/// publication; only the matching engine completion establishes a checkpoint. +#[derive(Clone)] +pub struct PreparedLocalCellGroupCheckpoint(Arc); + +impl std::fmt::Debug for PreparedLocalCellGroupCheckpoint { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("PreparedLocalCellGroupCheckpoint") + .field("marker", self.marker()) + .finish() + } +} + +pub(super) fn prepare(source: CellGroupCheckpoint) -> Result { + if source.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + let snapshot = source.snapshot()?; + Ok(PreparedLocalCellGroupCheckpoint(Arc::new(Inner { + source, + snapshot, + }))) +} + +impl PreparedLocalCellGroupCheckpoint { + pub fn marker(&self) -> &GroupManifest { + self.0.source.marker() + } + + pub fn snapshot(&self) -> &Snapshot { + &self.0.snapshot + } + + pub(crate) fn root(&self) -> &PreparedGroupManifest { + &self.0.source.root + } + + pub(crate) fn membership(&self) -> Result { + Ok(Header::decode(&self.0.source.header)?.membership) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index bbf241200..824ae18fb 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -12,9 +12,11 @@ use crate::raft::{ use cellar_logic::cell_group::{CellId, GroupHistory}; use cellar_ltx::internal::{ExtentId, EXTENT_BYTES}; mod header; +mod local; mod prepared; mod receive; use header::Header; +pub use local::PreparedLocalCellGroupCheckpoint; pub(crate) use prepared::persisted_marker; pub use prepared::PreparedCellGroupCheckpoint; pub use receive::CellGroupCheckpointReceiver; @@ -140,6 +142,34 @@ impl CellGroupCheckpointPool { &self, group: &GroupLog, root: &PreparedGroupManifest, + ) -> Result { + self.capture_root(group, root, false) + } + + /// Pin exactly the retained checkpoint selected by Raft Storage, even after + /// the current applied root advances. A concurrent replacement may refuse; + /// a successfully captured source owns its immutable cut through streaming. + pub fn capture_snapshot( + &self, + group: &GroupLog, + current: &PreparedGroupManifest, + snapshot: &raft::prelude::Snapshot, + ) -> Result { + let root = current.retained_checkpoint(group)?; + let source = self.capture_root(group, &root, true)?; + if source.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot changed", + )); + } + Ok(source) + } + + fn capture_root( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + is_retained: bool, ) -> Result { if !Arc::ptr_eq(&root.engine_owner, &group.owner) || !root @@ -190,23 +220,39 @@ impl CellGroupCheckpointPool { let header = { let cache = group.read()?; let marker = root.marker(); - if cache.group_manifest.as_ref() != Some(marker) - || cache.applied.position != marker.position() - || cache.applied.image.is_some() - || marker.position().index.get() > cache.hard_state.commit - || cache.term(marker.position().index.get()) != Some(marker.position().term.get()) - || marker.membership_revision() != cache.membership.revision().get() - || cache - .manifest_pins - .as_ref() - .is_some_and(|pins| pins.root() != marker.root()) - { - return Err(CellGroupCheckpointError::Invalid("source durable cut")); + if is_retained { + if cache.checkpoint_manifest.as_ref() != Some(marker) { + return Err(CellGroupCheckpointError::Invalid( + "retained checkpoint changed", + )); + } + let header = Header::decode(&cache.snapshot.data)?; + if header.snapshot()? != cache.snapshot || &header.root != marker { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot binding", + )); + } + header + } else { + if cache.group_manifest.as_ref() != Some(marker) + || cache.applied.position != marker.position() + || cache.applied.image.is_some() + || marker.position().index.get() > cache.hard_state.commit + || cache.term(marker.position().index.get()) + != Some(marker.position().term.get()) + || marker.membership_revision() != cache.membership.revision().get() + || cache + .manifest_pins + .as_ref() + .is_some_and(|pins| pins.root() != marker.root()) + { + return Err(CellGroupCheckpointError::Invalid("source durable cut")); + } + cache + .membership + .validate(&cache.conf_state, marker.position().index.get())?; + Header::capture(marker.clone(), &cache.conf_state, cache.membership.clone())? } - cache - .membership - .validate(&cache.conf_state, marker.position().index.get())?; - Header::capture(marker.clone(), &cache.conf_state, cache.membership.clone())? }; let encoded = header.encode()?; Ok(CellGroupCheckpoint { @@ -258,6 +304,17 @@ pub struct CellGroupCheckpoint { } impl CellGroupCheckpoint { + /// Metadata for this immutable source cut; it does not establish publication. + pub fn snapshot(&self) -> Result { + Header::decode(&self.header)?.snapshot() + } + + /// Consume an unstreamed captured cut into a bounded publication capability. + /// The owner and engine independently require that it is still current. + pub fn prepare_publication(self) -> Result { + local::prepare(self) + } + pub fn marker(&self) -> &GroupManifest { self.root.marker() } diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 8b6a4eeb0..8ab764125 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -269,6 +269,10 @@ impl Worker { self.shared.applied.fetch_add(1, Ordering::Relaxed); self.progress.send_replace(()); } + CellGroupEffect::CheckpointPublished { .. } => { + // Local compaction does not advance applied state. + self.progress.send_replace(()); + } CellGroupEffect::ReadIndex { .. } => (), // Runtime-owned read authority. CellGroupEffect::Persist(_) => return Err(CellGroupControllerError::Invalid("escaped persistence")), } diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 363bd77a1..dab91f3ae 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -20,6 +20,7 @@ pub use checkpoint::{ CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, CellGroupCheckpointLimits, CellGroupCheckpointPart, CellGroupCheckpointPool, CellGroupCheckpointReceiver, CellGroupCheckpointUsage, PreparedCellGroupCheckpoint, + PreparedLocalCellGroupCheckpoint, }; mod controller; pub use controller::{ diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 88d1b85c1..cecc1d676 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -5,6 +5,7 @@ use super::{ CellGroupBootstrapPermit, GroupManifest, PreparedCellGroupCheckpoint, PreparedGroupManifest, + PreparedLocalCellGroupCheckpoint, }; use crate::application::consensus::group_protocol::{ CellGroupEntry, GroupCommand, GroupEntryLimits, @@ -117,6 +118,9 @@ pub enum CellGroupEffect { CheckpointInstalled { manifest: Box, }, + CheckpointPublished { + manifest: Box, + }, /// A barrier to wait for, not permission to serve a cell. The dispatcher /// must recheck owner/term and pin an applied coherent cell view. ReadIndex { @@ -133,7 +137,9 @@ impl CellGroupEffect { Self::Message(message) => message.compute_size() as usize, Self::Apply(work) => work.entry.compute_size() as usize + work.base.reserved_bytes(), Self::ReadIndex { context, .. } => context.len(), - Self::Applied { .. } | Self::CheckpointInstalled { .. } => 1024, + Self::Applied { .. } + | Self::CheckpointInstalled { .. } + | Self::CheckpointPublished { .. } => 1024, } } } @@ -158,6 +164,7 @@ enum Phase { Ready(Box, Term), Light(LightReady), Apply(CellGroupTicket, Box, bool), + Checkpoint(PreparedLocalCellGroupCheckpoint, bool), } pub struct CellGroupOwner { @@ -450,6 +457,38 @@ impl CellGroupOwner { Ok(()) } + /// Queue publication of the exact applied cut without changing Raft progress + /// or the applied view. A later CheckpointPublished event proves persistence. + pub fn publish_checkpoint( + &mut self, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + self.live()?; + if !self.idle() || self.applying.is_some() { + return Err(RuntimeError::NotReady); + } + let root = checkpoint.root(); + if root.marker() != self.current.marker() + || !Arc::ptr_eq(&root.engine_owner, &self.group.owner) + || !Arc::ptr_eq(&root.page_owner, &self.current.page_owner) + || root.marker().position().index.get() <= 1 + || root.marker().position().index.get() >= u64::MAX - 2 + || root.marker().position().term.get() >= u64::MAX - 2 + { + return Err(RuntimeError::Invalid("local checkpoint owner/cut".into())); + } + if root + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096) + > self.options.max_ready_bytes + { + return Err(RuntimeError::Overloaded); + } + self.phase = Some(Phase::Checkpoint(checkpoint, false)); + Ok(()) + } + /// Admit an authenticated snapshot envelope only with its exact complete /// receiver-local closure. This returns protocol admission, never a durable /// acknowledgement. Snapshot messages are emitted after the Ready fence. @@ -773,6 +812,11 @@ impl CellGroupOwner { entries(light.committed_entries()) + messages(light.messages()) } Some(Phase::Apply(_, next, _)) => next.reserved_bytes() + 4096, + Some(Phase::Checkpoint(checkpoint, _)) => { + checkpoint.root().reserved_bytes() + + checkpoint.snapshot().compute_size() as usize + + 4096 + } None => 0, }; ( @@ -986,6 +1030,29 @@ impl CellGroupOwner { self.messages(light.take_messages())?; self.enqueue(light.take_committed_entries())?; } + Some(Phase::Checkpoint(checkpoint, submitted)) => { + if !submitted { + let mut write = GroupWrite::new(&self.group); + write.snapshot = + Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + self.phase = Some(Phase::Checkpoint(checkpoint, true)); + return self.persistence(write).map(Some); + } + let cache = self.group.read()?; + if cache.snapshot != *checkpoint.snapshot() + || cache.checkpoint_manifest.as_ref() != Some(checkpoint.marker()) + || cache.group_manifest.as_ref() != Some(self.current.marker()) + { + return Err(RuntimeError::StaleCompletion); + } + drop(cache); + self.current + .require_checkpoint(&self.group) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + return Ok(Some(CellGroupEffect::CheckpointPublished { + manifest: Box::new(checkpoint.marker().clone()), + })); + } Some(Phase::Apply(mut ticket, next, submitted)) => { if !submitted { let mut write = GroupWrite::new(&self.group); diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs index 1855a037e..e5a192bf9 100644 --- a/crates/cellar/durability/raft/cell_groups/retention.rs +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -245,6 +245,30 @@ impl RootRetentions { Ok(()) } + pub(super) fn checkpoint( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result { + let state = self.state()?; + let selected = state + .groups + .get(&key(marker)) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || selected.closed.load(Ordering::Acquire) + || selected.expected_checkpoint.as_ref() != Some(marker) + { + return Err(ManifestError::Conflict); + } + let root = selected + .checkpoint + .as_ref() + .filter(|root| &root.marker == marker) + .ok_or(ManifestError::Invalid("checkpoint not recovered"))?; + Ok(root._pins.clone()) + } + pub(super) fn reconcile(&self, retained: Vec) -> Result<(), ManifestError> { let mut state = self.state()?; if state diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index a20dc9a62..0c1b92b9a 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -116,6 +116,26 @@ impl PreparedGroupManifest { self.retention.require_checkpoint(group) } + pub(crate) fn retained_checkpoint( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result { + if !Arc::ptr_eq(&self.engine_owner, &group.owner) { + return Err(ManifestError::Conflict); + } + let marker = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .ok_or(ManifestError::Invalid("checkpoint not published"))?; + let images = self.retention.checkpoint(group, &marker)?; + Ok(Self { + marker, + images, + base: None, + ..self.clone() + }) + } + pub(crate) fn reserved_bytes(&self) -> usize { self.images.reserved_bytes() } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index ec8d6ee9b..9150a68ad 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -99,6 +99,7 @@ enum Command { Reconciliation(cellar_logic::cell_group::CellHistory), Step(Message), Checkpoint(Message, PreparedCellGroupCheckpoint), + PublishCheckpoint(PreparedLocalCellGroupCheckpoint), AdmitStep(Message), Propose(Vec), ProposeAtTerm(cellar_logic::consensus::Term, Vec), @@ -529,6 +530,23 @@ impl CellGroupRuntime { .map(|_| ()) } + /// Admit local checkpoint publication. Completion is delivered through the + /// CheckpointPublished event after engine persistence, not this return value. + pub async fn publish_checkpoint( + &self, + history: GroupHistory, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + let bytes = checkpoint + .root() + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096); + self.call(Some(history), Command::PublishCheckpoint(checkpoint), bytes) + .await + .map(|_| ()) + } + /// Queue a verified snapshot on this group owner. Admission is not a /// persistence receipt; only its post-fence Raft response acknowledges it. pub async fn step_checkpoint( diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 9909a5ef9..83eb5b2ba 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -370,6 +370,10 @@ impl Worker { Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } + Command::PublishCheckpoint(checkpoint) => group + .owner + .publish_checkpoint(checkpoint) + .map(|()| Response::Unit), Command::Checkpoint(message, checkpoint) => group .owner .step_checkpoint(message, checkpoint) @@ -434,6 +438,7 @@ impl Worker { effect, CellGroupEffect::Applied { .. } | CellGroupEffect::CheckpointInstalled { .. } + | CellGroupEffect::CheckpointPublished { .. } ) || group.settled.is_some() { return Err(RuntimeError::Protocol( diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index af8c8fceb..9a6763f19 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1369,6 +1369,12 @@ impl Worker { SnapshotChange::Install { .. } | SnapshotChange::CellGroup(_) => { batch.add_command(write.group.key, Command::Clean) } + SnapshotChange::CellGroupCheckpoint(prepared) => batch.add_command( + write.group.key, + Command::Compact { + index: prepared.snapshot().get_metadata().index + 1, + }, + ), SnapshotChange::Checkpoint(snapshot) => batch.add_command( write.group.key, Command::Compact { @@ -1401,7 +1407,13 @@ impl Worker { .begin_publication( &write.group, previous.as_ref(), - matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))), + matches!( + &write.snapshot, + Some( + SnapshotChange::CellGroup(_) + | SnapshotChange::CellGroupCheckpoint(_) + ) + ), ) .map_err(|e| LogError::Invalid(e.to_string()))?, ); diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 44cff6e64..c87a1478b 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,21 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_publication_preserves_suffix_and_catches_up() -> Result<()> { + runtime_fixture::local_publication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_extents_preserve_suffix_and_catch_up() -> Result<()> { + runtime_fixture::local_publication(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_local_runtime_joins_publication_shutdown() -> Result<()> { + runtime_fixture::local_shutdown().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_owner_rejects_overflow_cuts_before_raft() -> Result<()> { runtime_fixture::checkpoint_bounds().await @@ -496,7 +511,8 @@ impl Cluster { Some(manifest.as_ref()) ); } - CellGroupEffect::CheckpointInstalled { manifest } => { + CellGroupEffect::CheckpointInstalled { manifest } + | CellGroupEffect::CheckpointPublished { manifest } => { assert_eq!( voter.log.cell_groups()?[0].group_manifest()?.as_ref(), Some(manifest.as_ref()) diff --git a/crates/cellar/tests/support/cell_group_checkpoint_local.rs b/crates/cellar/tests/support/cell_group_checkpoint_local.rs new file mode 100644 index 000000000..beeeb0b7e --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_local.rs @@ -0,0 +1,429 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local publication and actual Raft-generated snapshot catch-up. Transfer uses +//! the native streaming callback; this does not qualify authenticated snapshot RPC. +use super::*; +use raft::{GetEntriesContext, Storage}; + +impl RuntimeCluster { + pub(super) async fn transfer_checkpoint( + &mut self, + history: GroupHistory, + message: Message, + ) -> Result<()> { + let from = message.from as usize - 1; + let to = message.to as usize - 1; + let root = self + .snapshot_sources + .get(&(from, history)) + .context("snapshot source not enabled")?; + let group = self.cluster.voters[from].log.cell_group(&history)?; + let mut limits = checkpoint_fixture::limits(); + limits.max_sources = 2; + limits.max_retained_bytes *= 2; + let pool = CellGroupCheckpointPool::new(limits)?; + let mut source = pool.capture_snapshot(&group, root, message.get_snapshot())?; + let (_, applier) = self + .appliers + .get(&(to, history)) + .context("snapshot destination not enabled")?; + let mut receiver = pool.receive( + applier.clone(), + checkpoint_receive::scratch()?, + self.cluster._root.path().into(), + )?; + let checkpoint = self + .cluster + .domain + .blocking(move || { + source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + match self.runtimes[to] + .step_checkpoint(history, message.clone(), checkpoint) + .await + { + Ok(CellGroupSnapshotAdmission::Restoring) => self.snapshot_transfers += 1, + Ok(CellGroupSnapshotAdmission::Ignored) => (), + Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { + self.network.push_back((history, message)) + } + Err(error) => return Err(error.into()), + } + assert!( + pool.usage().sources <= 1, + "only the accepted receiver may retain admission" + ); + Ok(()) + } +} + +pub(crate) async fn local_publication(rooted: bool) -> Result<()> { + let (mut cluster, _) = checkpoint_fixture::populated_partition(rooted).await?; + let route = cluster.birth.route().clone(); + let group = cluster.voters[0].log.cell_group(&route.history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let cut = root.marker().clone(); + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool.capture(&group, &root)?.prepare_publication()?; + let snapshot = checkpoint.snapshot().clone(); + assert!(pool.capture_snapshot(&group, &root, &snapshot).is_err()); + let cell = history(&route, "suffix")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(41);")?; + let proposal = cluster.voters[0] + .owner + .as_mut() + .unwrap() + .propose(sql::apply::enrollment_bytes(&route, &cell, &bytes, rooted)?)?; + // Persist a real suffix but withhold all peer replies, so the cut remains + // applied while the leader's next proposal remains uncommitted. + loop { + match cluster.voters[0].owner.as_mut().unwrap().poll()? { + Some(CellGroupEffect::Persist(work)) => { + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .complete_persist(work.run().await?)?; + } + Some(CellGroupEffect::Message(message)) => cluster.network.push_back(message), + None => break, + _ => anyhow::bail!("isolated suffix must not apply"), + } + } + let hard = group.initial_state()?.hard_state; + assert_eq!(hard.commit, cut.position().index.get()); + assert_eq!(group.last_index()?, proposal.position.index.get()); + let suffix = group.entries( + hard.commit + 1, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?; + // A local publication cannot bundle even an identical hard state, and a + // captured closure from another engine cannot publish on this replica. + let mut mixed = GroupWrite::new(&group); + mixed.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + mixed.hard_state = Some(hard.clone()); + assert!(cluster.voters[0] + .log + .persist_cells(cellar_logic::consensus::OperationId::new(973), vec![mixed]) + .await + .is_err()); + let foreign_group = cluster.voters[1].log.cell_group(&route.history)?; + let mut foreign = GroupWrite::new(&foreign_group); + foreign.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + assert!(cluster.voters[1] + .log + .persist_cells( + cellar_logic::consensus::OperationId::new(974), + vec![foreign] + ) + .await + .is_err()); + assert_eq!(group.checkpoint_manifest()?, None); + drop(foreign_group); + let stale = checkpoint.clone(); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(checkpoint)?; + let Some(CellGroupEffect::Persist(work)) = cluster.voters[0].owner.as_mut().unwrap().poll()? + else { + anyhow::bail!("local checkpoint persistence"); + }; + let release = cluster.voters[0].gate.arm(fault::HOLD_SYNC); + let mut pending = cluster.domain.spawn(work.run())?; + tokio::time::timeout( + Duration::from_secs(5), + cluster.voters[0].gate.entered.notified(), + ) + .await?; + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 1); + assert!(cluster.voters[0].owner.as_mut().unwrap().poll()?.is_none()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err() + ); + drop(release); + let completion = pending.await??; + assert_eq!(group.snapshot(0, 3)?, snapshot); + assert_eq!(group.first_index()?, cut.position().index.get() + 1); + assert_eq!(group.initial_state()?.hard_state, hard); + assert_eq!(group.group_manifest()?.as_ref(), Some(&cut)); + assert_eq!( + group.entries( + hard.commit + 1, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false) + )?, + suffix + ); + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .complete_persist(completion)?; + let Some(CellGroupEffect::CheckpointPublished { manifest }) = + cluster.voters[0].owner.as_mut().unwrap().poll()? + else { + anyhow::bail!("local publication completion"); + }; + assert_eq!(manifest.as_ref(), &cut); + cluster.drive_partition(false, Some(2)).await?; + assert_eq!( + cluster.voters[0].owner.as_ref().unwrap().status()?.applied, + proposal.position + ); + assert!(matches!( + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(stale), + Err(RuntimeError::Invalid(_)) + )); + assert_eq!(pool.usage().sources, 0); + assert!(pool.capture(&group, &root).is_err()); + let retained = pool.capture_snapshot( + &group, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + &snapshot, + )?; + assert_eq!(retained.marker(), &cut); + drop(retained); + let anchor = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let destination = cluster.voters[2].log.cell_group(&route.history)?; + let applier = CellGroupApplier::new( + destination, + cluster.voters[2] + .pages + .take() + .context("destination pages")?, + checkpoint_receive::config( + cluster.voters[2].images.clone(), + cluster.voters[2].extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let sql = CellGroupSqlPool::start( + &fixture.cluster.voters[2].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + fixture.appliers.insert((2, route.history), (sql, applier)); + fixture.snapshot_sources.insert((0, route.history), anchor); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + assert!( + fixture.snapshot_transfers >= 1, + "Raft must generate a snapshot after compaction" + ); + let restored = fixture.cluster.voters[2].log.cell_group(&route.history)?; + assert_eq!(restored.checkpoint_manifest()?.as_ref(), Some(&cut)); + let source = group.group_manifest()?.context("source root")?; + assert_eq!(restored.group_manifest()?.as_ref(), Some(&source)); + assert_eq!(restored.group_manifest()?.unwrap().root().cells(), 4); + let tail = group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?; + assert_eq!(tail.first(), suffix.first()); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + drop(restored); + drop(root); + drop(group); + fixture.close().await?; + // Reopen the actual leader engine and physical closures. Cache-only suffix + // preservation cannot pass this check if publication cleaned durable entries. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let base = disk.path().join("node-1"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([67; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.snapshot(0, 3)?, snapshot); + assert_eq!(group.group_manifest()?.as_ref(), Some(&source)); + assert_eq!( + group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false) + )?, + tail + ); + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = checkpoint_receive::config( + GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + if rooted { + Some(cellar_ltx::internal::NodeExtentStore::open( + disk.path().join("extents-0"), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }, + disk.path(), + )?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || recover_source(recovering, pages, config, rooted))? + .await??; + assert_eq!(recovered.marker(), &source); + let mut retained = pool.capture_snapshot(&group, &recovered, &snapshot)?; + assert_eq!(retained.marker(), &cut); + domain + .blocking(move || retained.stream(|_| Ok(())))? + .await??; + assert_eq!(pool.usage().sources, 0); + drop(recovered); + drop(applier); + drop(group); + log.close().await?; + drop(log); + domain.shutdown().await; + Ok(()) +} + +pub(crate) async fn local_shutdown() -> Result<()> { + for rooted in [false, true] { + let (cluster, _) = checkpoint_fixture::populated_with_lagging(rooted, false).await?; + let history = cluster.birth.route().history; + let group = cluster.voters[0].log.cell_group(&history)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool + .capture(&group, cluster.voters[0].owner.as_ref().unwrap().manifest())? + .prepare_publication()?; + let cut = checkpoint.marker().clone(); + let gate = cluster.voters[0].gate.clone(); + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; + let release = gate.arm(fault::HOLD_SYNC); + fixture.runtimes[0] + .publish_checkpoint(history, checkpoint) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let runtime = fixture.runtimes[0].clone(); + let mut close = fixture + .cluster + .domain + .spawn(async move { runtime.close().await })?; + assert!(tokio::time::timeout(Duration::from_millis(30), &mut close) + .await + .is_err()); + assert_eq!(pool.usage().sources, 1); + assert_eq!(group.checkpoint_manifest()?, None); + drop(release); + close.await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&cut)); + drop(group); + fixture.close().await?; + } + Ok(()) +} + +// The source fixture deliberately uses local generation IDs distinct from the +// production SQL adapter. Restore its materialized pins with that same explicit +// resolver; logical roots exclude local filenames. Extents use the production +// recovery adapter. Both paths verify current and retained checkpoint closures. +fn recover_source( + group: GroupLog, + mut pages: ManifestPageStore, + config: CellGroupApplyImages, + rooted: bool, +) -> Result<(Option, PreparedGroupManifest)> { + if rooted { + let (applier, root) = CellGroupApplier::recover( + group, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + return Ok((Some(applier), root)); + } + let CellGroupApplyImages::Materialized { images, .. } = config else { + anyhow::bail!("materialized fixture resolver"); + }; + let current = group.group_manifest()?.context("current marker")?; + let checkpoint = group.checkpoint_manifest()?.context("checkpoint marker")?; + let tree = ManifestTree::new(current.route().clone(), ManifestLimits::default())?; + let mut recovered = None; + for marker in [¤t, &checkpoint] { + let root = tree.prepare_complete( + &group, + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |leaf| { + if leaf.version.image.txid == 0 { + return Ok(CellImagePin::Empty); + } + let CellImageCommitment::Materialized(digest) = leaf.version.image.commitment + else { + return Err(ManifestError::Invalid("fixture materialized commitment")); + }; + let id = image_id(&leaf.version.history, leaf.version.sequence.get()) + .map_err(|error| ManifestError::Io(std::io::Error::other(error)))?; + images + .adopt( + id, + ImageState { + epoch: leaf.version.image.epoch, + txid: TXID(leaf.version.image.txid), + checksum: leaf.version.image.checksum, + digest, + length: leaf.geometry.length, + page_size: leaf.geometry.page_size, + }, + ) + .map(CellImagePin::Materialized) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }, + )?; + if marker == ¤t { + recovered = Some(root); + } + } + Ok((None, recovered.context("current closure")?)) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 20b7fd56a..a83d2adad 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -5,6 +5,10 @@ use super::*; use std::collections::BTreeMap; +#[path = "cell_group_checkpoint_local.rs"] +mod checkpoint_local; +pub(super) use checkpoint_local::{local_publication, local_shutdown}; + #[path = "cell_group_snapshot_owner.rs"] mod snapshot_owner; pub(super) use snapshot_owner::{ @@ -20,6 +24,8 @@ mod tcp; pub(super) use tcp::physical_transport; struct RuntimeCluster { + snapshot_sources: BTreeMap<(usize, GroupHistory), PreparedGroupManifest>, + snapshot_transfers: usize, transports: Vec, appliers: BTreeMap<(usize, GroupHistory), (CellGroupSqlPool, CellGroupApplier)>, cluster: Cluster, @@ -81,6 +87,8 @@ impl RuntimeCluster { } let applied = BTreeMap::from([(cluster.birth.route().history, applied)]); Ok(Self { + snapshot_sources: BTreeMap::new(), + snapshot_transfers: 0, transports: Vec::new(), appliers: BTreeMap::new(), cluster, @@ -136,7 +144,8 @@ impl RuntimeCluster { } } CellGroupEffect::Applied { manifest, .. } - | CellGroupEffect::CheckpointInstalled { manifest } => { + | CellGroupEffect::CheckpointInstalled { manifest } + | CellGroupEffect::CheckpointPublished { manifest } => { self.applied.get_mut(&scope).context("applied")?[n] = manifest.position().index } @@ -156,8 +165,16 @@ impl RuntimeCluster { } return Ok(()); } + let mut ordinary = VecDeque::new(); + for (scope, message) in messages { + if message.get_msg_type() == MessageType::MsgSnapshot { + self.transfer_checkpoint(scope, message).await?; + } else { + ordinary.push_back((scope, message)); + } + } let runtimes = &self.runtimes; - for (scope, message, result) in futures_util::future::join_all(messages.into_iter().map( + for (scope, message, result) in futures_util::future::join_all(ordinary.into_iter().map( |(scope, message)| async move { let result = runtimes[message.to as usize - 1] .admit_step(scope, message.clone()) @@ -241,6 +258,7 @@ impl RuntimeCluster { drop(self.events); drop(self.runtimes); drop(self.extra); + drop(self.snapshot_sources); self.cluster.close().await } } From 1c05a189d773dc6758dd4328ee23714dc2d2dc6d Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 16:19:13 -0700 Subject: [PATCH 126/568] docs: record CG-05 local checkpoint and native catch-up qualification --- docs/design/multi-raft/cell-groups/README.md | 9 +-- .../multi-raft/cell-groups/baseline.json | 8 +-- docs/design/multi-raft/cell-groups/handoff.md | 59 ++++++++++++------- .../design/multi-raft/cell-groups/progress.md | 47 ++++++++++++++- docs/design/multi-raft/cell-groups/tasks.json | 4 +- 5 files changed, 96 insertions(+), 31 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 7f0d9f221..7e9ebd247 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -6,9 +6,9 @@ Status: executable design and implementation plan delivered on 2026-10-29. CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and CG-04 shared execution have passed their scoped local evidence gates. CG-05 is in progress: checkpoint capture, receiver verification, atomic storage -installation and exact Raft Ready acceptance pass their scoped regression gates. -Source local checkpoint publication, snapshot transport and membership recovery -remain open. The production host has not yet cut over to shared groups. +installation, exact Raft Ready acceptance, local publication and native Raft-generated +catch-up pass their scoped regression gates. Authenticated snapshot RPC, automated +controller lifecycle and membership recovery remain open. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -54,7 +54,8 @@ CG-05's partial source checkpoint is retained in `.cellar/cell-groups-evidence/CG-05-checkpoint-source/` and the receiver slice in `.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`, followed by owner/runtime acceptance in `.cellar/cell-groups-evidence/CG-05-snapshot-owner/`. These partial -checkpoints do not complete CG-05. +checkpoints and local publication in +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/` do not complete CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 7c8f6c94e..c444059c9 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "096cfb132397d9588ba9138b1cf474210fc062f3", + "source_revision": "742b1e6d4e7155e16fb1b4eba0cce3b9056944ae", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "a4d5be18b6165bb60890dd35223d0bc062ee40577c68baf71d5048e46e6c63f9", - "line": 438, + "sha256": "a961606bd7b2af30028b5956117e6f3c2e38f42b6caf485e334433c4dea4dff2", + "line": 442, "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n}\n" }, "apply": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index b99d0961d..75169cd04 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -29,8 +29,9 @@ are represented. See the latest [progress](progress.md) for source evidence and limits. Receiver verification and atomic storage installation now pass their partial gate, including cold recovery of the current and older checkpoint roots. Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate. -Continue with source local checkpoint publication and bounded snapshot transport, -then whole-group learner/membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic +Local checkpoint publication and native Raft-generated catch-up now pass their +partial gate. Continue with bounded authenticated snapshot transport and its +controller lifecycle, then whole-group membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. @@ -57,13 +58,26 @@ sync, invalid/overflow refusal, subsequent quorum writes and joined runtime clos are covered. The tests construct the envelope from a captured source; they do not qualify Raft-generated outbound snapshots or snapshot RPC. -The most critical gap is source-side local checkpoint publication followed by -bounded authenticated snapshot transfer and real peer catch-up. Preserve the -source log suffix: the receiver-install variant is not a local checkpoint -operation. Qualify receiver crash/sync-error/cancellation and root-slot exhaustion, -then learner catch-up, joint membership and lost-disk replacement. The storage -capability grants no new voter or serving authority. Ordinary peer snapshot -ingress stays closed until the verified receiver is wired through transport. +The latest local-publication checkpoint at `742b1e6d` passes 222 selected +Rust cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/` binds 298 source hashes +and 5 copied executables. The owner admits an exact captured current cut, +preserves every later log entry and applied/hard state through publication, and +emits `CheckpointPublished` after its persistence ticket completes. Outbound capture +pins the retained checkpoint named by Raft's exact snapshot, after current apply +has advanced. Tests prove actual RawNode-generated catch-up, suffix replay, cold +leader engine/root recovery and re-stream, plus held publication shutdown in both +image formats. Materialized source recovery uses its fixture filename resolver; +extent recovery and destination replay use production adapters. + +The most critical gap is bounded authenticated snapshot RPC and controller-owned +source/publication lifecycle. Native streaming callbacks do not qualify network +framing/authentication, retries/cancellation, failed snapshot reporting or automatic +checkpoint policy. Ordinary peer snapshot ingress remains closed. Preserve the +post-persistence receiver acknowledgement boundary and exact retained cut. Then +qualify receiver crash/sync-error/cancellation and root-slot pressure, learner +catch-up, joint membership and lost-disk replacement. Snapshots grant no new voter +or serving authority. All five declared CG-05 acceptance cases remain required. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands and 37 validator tests. Transfer @@ -623,13 +637,14 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Complete CG-05 source publication and transport. Capture an exact applied cut, - retain its complete closure/configuration, publish a local checkpoint while - preserving every later entry, and make RawNode-generated snapshots stream that - same retained cut. Carry route/member authentication, budgets, cancellation, - retries and joined physical ownership through the controller/peer transport. +1. Complete CG-05 bounded authenticated snapshot transport. Connect exact retained + checkpoint capture, source leasing, framing and destination verification to the + controller/peer transport. Carry route/member authentication, byte/item budgets, + cancellation, retries and joined physical ownership through the full transfer. + Add the source publication policy and report actual snapshot failures to Raft. Admit the destination through `step_checkpoint`; use its post-persistence Raft - acknowledgement, never the admission return value, as completion. + acknowledgement, never the admission return value, as completion. Exercise real + physical TLS catch-up while unrelated groups continue making progress. 2. Finish CG-05 membership and recovery. Qualify the outstanding receiver fault cuts and root-slot pressure, then whole-group learner catch-up, joint membership and lost-disk replacement. Run every exact acceptance case before DONE. @@ -647,12 +662,14 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Start source publication in `checkpoint/mod.rs`, `cache.rs` and the group root -reservation path in `storage.rs` / `cell_groups/retention.rs`. The current capture -API accepts the current applied root; outbound transfer must also select the -retained checkpoint root after newer writes advance current state. Bind its -header/configuration to the snapshot returned by Raft Storage, not a fresh capture -of a later current root. Keep snapshot publication on the owner persistence lane. +Start transport integration in `transport_frame.rs`, `transport_runtime.rs`, +`snapshot_transport.rs` and the shared controller. Existing snapshot pulls use +cell-scoped identities; preserve physical group-history, permanent member and +incarnation checks for shared groups. Use `checkpoint/mod.rs::capture_snapshot` +for the exact retained cut. Local publication already has its own owner persistence +phase and `SnapshotChange::CellGroupCheckpoint`; never reuse receiver installation +to publish a local leader checkpoint. Capture/publication refusal is retryable only +under an explicit bounded policy; a missing receiver response remains ambiguous. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 4ac5d9d59..25617f9df 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,51 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 local publication and native Raft snapshot catch-up + +Qualified source: `742b1e6d4e7155e16fb1b4eba0cce3b9056944ae`. All 222 selected Rust cases pass across shared +execution/catalog/storage and the existing single-cell storage/snapshot suites. +All 8 gate commands pass, including all-target Clippy for Cellar/core/LTX, +formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/checkpoint.json` binds +298 source hashes, 5 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +An unstreamed source becomes a bounded local publication capability. The owner +requires its exact applied cut, engine and page owner, and runs publication on the +existing persistence lane. The distinct local engine operation preserves hard +state, applied state and every later log entry while compacting the represented +prefix. Mixed, foreign and stale preparations refuse. `CheckpointPublished` +arrives only after the matching engine completion; local compaction does not +advance the controller's applied-state counter. Admission and physical pins remain +owned through queued sync and joined runtime shutdown. + +`capture_snapshot` selects the retained complete closure and original metadata +named by the exact snapshot from Raft Storage. Later current-root updates cannot +substitute a newer cut. A concurrent replacement may refuse capture; a captured +source owns the old immutable closure until streaming ends. + +Three new cases cover both image formats, a real uncommitted suffix surviving +held publication and later quorum commit, and a partitioned existing voter restored +by an actual RawNode-generated snapshot followed by suffix replay. The leader's +engine is then closed and reopened: the retained snapshot and exact tail must +match, both physical roots are recovered, and the older checkpoint streams again. +Materialized source recovery uses the fixture's explicit generation-ID resolver; +extent recovery and destination suffix replay use production adapters. The initial +materialized cold-recovery failure from using the wrong resolver is retained with +the compile diagnostics. No assertion was weakened. Joined runtime shutdown during +accepted local publication is covered in both representations. + +The critical gap is bounded authenticated snapshot RPC and its controller-owned +source/publication lifecycle. These cases use native streaming callbacks; they do +not qualify network framing/authentication, transfer cancellation/retries, failed +snapshot reporting, automated checkpoint policy, learners/joint membership or disk +replacement. Receiver process-kill/sync-error/cancelled-waiter cuts and root-slot +pressure also remain open. All five full CG-05 acceptance cases remain required. +There is no new shared-group RustFS, load or zero-quorum DR qualification. CG-06 +archival/reclamation and the later public cutover and legacy removal remain open. + ## CG-05 Raft snapshot acceptance and persistence Qualified source: `096cfb132397d9588ba9138b1cf474210fc062f3`. All 146 selected Rust cases pass across shared @@ -43,7 +88,7 @@ snapshot persistence. The fixture builds the snapshot envelope from captured source state and uses the native streaming callback. It does not exercise a source RawNode-generated snapshot or authenticated snapshot RPC. -The critical next gap is source-side local checkpoint publication that preserves +The gap recorded at this owner checkpoint was source-side local publication that preserves the unapplied log suffix, followed by bounded snapshot transport and actual Raft peer catch-up. The receiver-install write must not be reused as a local leader checkpoint: it replaces the received state and truncates prior entries. Then diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 2d92a6beb..325a1f6d4 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1580,7 +1580,9 @@ "crates/cellar/tests/support/cell_group_runtime.rs", "crates/cellar/tests/support/cell_group_sql.rs", "crates/cellar/tests/support/cell_group_sql_apply.rs", - "crates/cellar/tests/support/cell_group_snapshot_owner.rs" + "crates/cellar/tests/support/cell_group_snapshot_owner.rs", + "crates/cellar/durability/raft/cell_groups/checkpoint/local.rs", + "crates/cellar/tests/support/cell_group_checkpoint_local.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From ceb7ed3149202660855d32df1ec5feed7775e068 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 16:55:40 -0700 Subject: [PATCH 127/568] feat: stream authenticated cell-group checkpoints over TLS --- crates/cellar/application/consensus/mod.rs | 3 +- .../durability/raft/cell_groups/apply/mod.rs | 23 ++ .../raft/cell_groups/checkpoint/mod.rs | 3 +- .../raft/cell_groups/checkpoint/receive.rs | 23 ++ .../raft/cell_groups/checkpoint/wire.rs | 93 ++++++ .../cell_groups/controller/registration.rs | 7 +- .../raft/cell_groups/controller/worker.rs | 17 +- .../durability/raft/cell_groups/owner.rs | 28 ++ .../durability/raft/cell_groups/root.rs | 20 ++ .../raft/cell_groups/runtime/mod.rs | 29 +- .../raft/cell_groups/runtime/worker.rs | 33 ++ .../raft/group_snapshot_transport/codec.rs | 61 ++++ .../raft/group_snapshot_transport/mod.rs | 180 +++++++++++ .../raft/group_snapshot_transport/receive.rs | 167 +++++++++++ .../raft/group_snapshot_transport/send.rs | 234 +++++++++++++++ .../raft/group_snapshot_transport/workers.rs | 95 ++++++ .../durability/raft/snapshot_transport.rs | 2 +- crates/cellar/durability/raft/transport.rs | 15 + .../durability/raft/transport_runtime.rs | 7 +- .../durability/raft/transport_shared.rs | 10 +- .../tests/native_cell_groups_execution.rs | 25 ++ .../support/cell_group_checkpoint_local.rs | 122 +++++++- .../support/cell_group_network_probes.rs | 281 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 56 +++- .../multi-raft/cell-groups/protocols.md | 36 +++ docs/design/multi-raft/cell-groups/tasks.json | 4 +- 26 files changed, 1552 insertions(+), 22 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/codec.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/mod.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/receive.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/send.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/workers.rs create mode 100644 crates/cellar/tests/support/cell_group_network_probes.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 3904cdba4..3710843d5 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -211,7 +211,8 @@ pub mod verification { CellGroupBootstrap, GroupBootstrap, LogError, ReplicaLog, SyncReceipt, }; pub use crate::raft::transport::{ - Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, + CellGroupCheckpointRegistration, Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, + TransportError, TransportUsage, }; pub use crate::raft::transport_options::{ NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 23e9ee108..5a5cd5edf 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -67,6 +67,29 @@ impl CellGroupApplier { }) } + /// Capture on the blocking domain: only the exact retained snapshot from + /// this applier's engine and page owner may become a network source. + pub(crate) fn capture_checkpoint( + &self, + pool: &CellGroupCheckpointPool, + snapshot: &::raft::prelude::Snapshot, + ) -> Result { + let root = { + let storage = self + .storage + .lock() + .map_err(|_| CellGroupCheckpointError::Invalid("source page owner"))?; + PreparedGroupManifest::checkpoint_from_store(&self.group, &storage.pages)? + }; + let source = pool.capture_root(&self.group, &root, true)?; + if source.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid( + "retained snapshot changed", + )); + } + Ok(source) + } + pub fn history(&self) -> GroupHistory { self.group.identity().scope } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 824ae18fb..66571e9ec 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -15,6 +15,7 @@ mod header; mod local; mod prepared; mod receive; +pub(crate) mod wire; use header::Header; pub use local::PreparedLocalCellGroupCheckpoint; pub(crate) use prepared::persisted_marker; @@ -165,7 +166,7 @@ impl CellGroupCheckpointPool { Ok(source) } - fn capture_root( + pub(super) fn capture_root( &self, group: &GroupLog, root: &PreparedGroupManifest, diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs index ced4b1cae..c752b70df 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -142,6 +142,29 @@ fn invalid(message: &'static str) -> CellGroupCheckpointError { } impl CellGroupCheckpointReceiver { + pub(super) fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + let pending = self.cell.as_ref().ok_or(invalid("extent without leaf"))?; + let id = pending + .view + .as_ref() + .and_then(|view| view.manifest().extent(ordinal)) + .ok_or(invalid("extent without manifest"))?; + if &pending.leaf.version.history.cell != cell + || ordinal != pending.next_extent + || id.bytes() != digest + || u64::from(id.length()) != length + { + return Err(invalid("wire extent binding")); + } + Ok(id) + } + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { if self.failed { return Err(invalid("failed receiver")); diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs new file mode 100644 index 000000000..70693953b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs @@ -0,0 +1,93 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned bounded chunk identifiers. Extent identities are resolved from the +//! receiver's already verified manifest, never reconstructed from wire claims. +use super::*; +use serde::{Deserialize, Serialize}; + +#[derive(Serialize, Deserialize)] +#[serde(tag = "kind", deny_unknown_fields)] +pub(crate) enum Part { + Header, + Page { + digest: [u8; 32], + }, + Materialized { + cell: String, + }, + ExtentManifest { + cell: String, + }, + Extent { + cell: String, + ordinal: u32, + digest: [u8; 32], + }, +} + +pub(crate) struct Chunk { + pub part: Part, + pub offset: u64, + pub length: u64, + pub bytes: Vec, +} + +impl Chunk { + pub fn copy(chunk: CellGroupCheckpointChunk<'_>) -> Self { + let part = match chunk.part { + CellGroupCheckpointPart::Header => Part::Header, + CellGroupCheckpointPart::Page(digest) => Part::Page { digest }, + CellGroupCheckpointPart::Materialized(cell) => Part::Materialized { + cell: cell.as_str().to_owned(), + }, + CellGroupCheckpointPart::ExtentManifest(cell) => Part::ExtentManifest { + cell: cell.as_str().to_owned(), + }, + CellGroupCheckpointPart::Extent { cell, ordinal, id } => Part::Extent { + cell: cell.as_str().to_owned(), + ordinal, + digest: id.bytes(), + }, + }; + Self { + part, + offset: chunk.offset, + length: chunk.length, + bytes: chunk.bytes.to_vec(), + } + } + + pub fn append(self, receiver: &mut CellGroupCheckpointReceiver) -> Result<()> { + let cell = match &self.part { + Part::Materialized { cell } + | Part::ExtentManifest { cell } + | Part::Extent { cell, .. } => Some( + CellId::new(cell).map_err(|_| CellGroupCheckpointError::Invalid("wire cell"))?, + ), + _ => None, + }; + let part = match &self.part { + Part::Header => CellGroupCheckpointPart::Header, + Part::Page { digest } => CellGroupCheckpointPart::Page(*digest), + Part::Materialized { .. } => { + CellGroupCheckpointPart::Materialized(cell.as_ref().unwrap()) + } + Part::ExtentManifest { .. } => { + CellGroupCheckpointPart::ExtentManifest(cell.as_ref().unwrap()) + } + Part::Extent { + ordinal, digest, .. + } => CellGroupCheckpointPart::Extent { + cell: cell.as_ref().unwrap(), + ordinal: *ordinal, + id: receiver.wire_extent(cell.as_ref().unwrap(), *ordinal, *digest, self.length)?, + }, + }; + receiver.append(CellGroupCheckpointChunk { + part, + offset: self.offset, + length: self.length, + bytes: &self.bytes, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs index 06d43247c..30fa9cb38 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/registration.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -365,9 +365,14 @@ pub(super) async fn attach( .applier .check_startup(®istration.current) .map_err(|e| CellGroupControllerError::Apply(Arc::new(e)))?; - let guard = transport + let mut guard = transport .register_physical_group(registration.transport) .map_err(CellGroupControllerError::Transport)?; + guard.checkpoint = Some( + transport + .register_group_checkpoint(registration.applier) + .map_err(CellGroupControllerError::Transport)?, + ); runtime .attach(registration.permit, registration.current) .await?; diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 8ab764125..a98f7dfaa 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -240,17 +240,26 @@ impl Worker { match effect { CellGroupEffect::Message(message) => { if let Err(error) = self.transport.submit_physical(&group, message.clone()) { - if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer) { + if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer | TransportError::Refused) { return Err(CellGroupControllerError::Transport(error)); } self.shared.send_refusals.fetch_add(1, Ordering::Relaxed); // A hint is advisory, never delivery or commit evidence. - // Drop excess hints; Raft's own ticks retry lost traffic. - if self.hints.len() < self.limits.max_groups { + // Excess ordinary unreachable hints may be dropped. + let snapshot = message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot; + // Snapshot failure cannot be dropped: Raft pauses + // that peer until this feedback. At most one can + // be pending per known peer; ordinary hints keep + // the stricter one-per-group aggregate cap. + if snapshot || self.hints.len() < self.limits.max_groups { let runtime = self.runtime.clone(); let member = message.to; self.hints.push(Box::pin(async move { - let _ = runtime.report_unreachable(group, member).await; + if snapshot { + let _ = runtime.report_snapshot(group, member, false).await; + } else { + let _ = runtime.report_unreachable(group, member).await; + } })); } } diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index cecc1d676..a3fd1bf59 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -445,6 +445,34 @@ impl CellGroupOwner { Ok(()) } + pub(super) fn validate_snapshot_member(&self, member: u64) -> Result<(), RuntimeError> { + self.live()?; + if member == 0 || self.node.raft.prs().get(member).is_none() { + return Err(RuntimeError::Protocol("unknown snapshot member".into())); + } + Ok(()) + } + + pub(super) fn report_snapshot( + &mut self, + member: u64, + delivered: bool, + ) -> Result<(), RuntimeError> { + self.validate_snapshot_member(member)?; + if self.phase.is_some() || self.checkpoint.is_some() { + return Err(RuntimeError::NotReady); + } + self.node.report_snapshot( + member, + if delivered { + ::raft::SnapshotStatus::Finish + } else { + ::raft::SnapshotStatus::Failure + }, + ); + Ok(()) + } + pub(super) fn report_unreachable(&mut self, member: u64) -> Result<(), RuntimeError> { self.live()?; if self.phase.is_some() || self.checkpoint.is_some() { diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 0c1b92b9a..ae4808c0b 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -136,6 +136,26 @@ impl PreparedGroupManifest { }) } + pub(super) fn checkpoint_from_store( + group: &crate::raft::cache::GroupLog, + store: &ManifestPageStore, + ) -> Result { + store.require_live()?; + let marker = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))? + .ok_or(ManifestError::Invalid("checkpoint not published"))?; + let images = store.retention.checkpoint(group, &marker)?; + Ok(Self { + marker, + images, + engine_owner: group.owner.clone(), + base: None, + page_owner: store.owner.clone(), + retention: store.retention.clone(), + }) + } + pub(crate) fn reserved_bytes(&self) -> usize { self.images.reserved_bytes() } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 9150a68ad..a62a850f7 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -25,6 +25,7 @@ use std::sync::{ use tokio::sync::{mpsc, oneshot, Notify}; const COMPLETION_SLOTS: usize = 8; +const SNAPSHOT_HINT_SLOTS: usize = 256; pub struct CellGroupRuntimeEvent { group: GroupHistory, @@ -82,6 +83,7 @@ struct Handle { requests: mailbox::Sender, budget: Arc, completions: Arc, + snapshot_hints: Arc, read_pins: Arc, transport_claimed: AtomicBool, closing: Arc, @@ -109,6 +111,7 @@ enum Command { ), ReadIndex(Vec), Unreachable(u64), + SnapshotStatus(u64, bool), Read(Box), Apply(CellGroupTicket, Box), Status, @@ -208,7 +211,8 @@ impl CellGroupRuntime { log.cell_groups()?; let lease = log.claim_runtime()?; let domain = log.domain(); - let (sender, requests) = mailbox::channel(options.max_requests + COMPLETION_SLOTS); + let (sender, requests) = + mailbox::channel(options.max_requests + COMPLETION_SLOTS + SNAPSHOT_HINT_SLOTS); let (events, receiver) = mpsc::channel(options.max_events); let closing = Arc::new(AtomicBool::new(false)); let terminal = Arc::new(Terminal::new()); @@ -220,6 +224,7 @@ impl CellGroupRuntime { domain: domain.clone(), budget: Budget::new(options.max_request_bytes, options.max_requests), completions: Budget::new(options.max_ready_bytes, COMPLETION_SLOTS), + snapshot_hints: Budget::new(SNAPSHOT_HINT_SLOTS * 4096, SNAPSHOT_HINT_SLOTS), read_pins: Budget::new(options.max_read_pin_bytes, options.max_pending_reads), options: options.clone(), requests: sender, @@ -278,7 +283,10 @@ impl CellGroupRuntime { ) -> Result { self.admit()?; let completion = matches!(command, Command::Apply(..) | Command::Quarantine); - let budget = if completion { + let snapshot_hint = matches!(command, Command::SnapshotStatus(..)); + let budget = if snapshot_hint { + &self.inner.snapshot_hints + } else if completion { &self.inner.completions } else { &self.inner.budget @@ -294,7 +302,7 @@ impl CellGroupRuntime { if let Some(permit) = budget.acquire(bytes) { break permit; } - if !completion { + if !completion && !snapshot_hint { return Err(RuntimeError::Overloaded); } crate::asyncrt::select! { @@ -635,6 +643,21 @@ impl CellGroupRuntime { .map(|_| ()) } + pub(crate) async fn report_snapshot( + &self, + history: GroupHistory, + member: u64, + delivered: bool, + ) -> Result<(), RuntimeError> { + self.call( + Some(history), + Command::SnapshotStatus(member, delivered), + 4096, + ) + .await + .map(|_| ()) + } + pub(crate) async fn report_unreachable( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 83eb5b2ba..8b9b35909 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -13,6 +13,8 @@ struct Group { queued: bool, deferred: VecDeque, completion: Option, + // One status per current peer, retaining node-wide hint item/byte admission. + snapshot_hints: BTreeMap, inbound: Arc, tick: bool, write: Option>, @@ -55,6 +57,7 @@ impl Group { self.write = None; self.write_bytes = 0; self.tick = false; + self.snapshot_hints.clear(); for request in self.deferred.drain(..) { reply(request, Err(error.clone())); } @@ -188,6 +191,7 @@ impl Worker { queued: false, deferred: VecDeque::new(), completion: None, + snapshot_hints: BTreeMap::new(), inbound: Budget::new( (self.options.max_request_bytes / 4).max(self.options.max_message_bytes + 4096), (self.options.max_requests / 8).clamp(1, 32), @@ -228,6 +232,21 @@ impl Worker { reply(request, group.owner.status().map(Response::Status)); return; } + if let Command::SnapshotStatus(member, delivered) = request.command { + if let Err(error) = group.owner.validate_snapshot_member(member) { + reply(request, Err(error)); + return; + } + let Request { + reply: answer, + _permit, + .. + } = request; + group.snapshot_hints.insert(member, (delivered, _permit)); + group.wake(history, &mut self.order); + let _ = answer.send(Ok(Response::Unit)); + return; + } if let Command::AdmitStep(message) = &request.command { if let Err(error) = group.owner.validate_message(message) { reply(request, Err(error)); @@ -313,6 +332,19 @@ impl Worker { made = true; } } + // Hints never wait behind ordinary admission or consume an + // apply-completion slot. Their protocol mutation still waits + // until the exact in-flight Ready/persistence phase has settled. + if group.owner.idle() { + if let Some((member, (delivered, _permit))) = group.snapshot_hints.pop_first() { + // A committed membership change may have removed this + // peer while the hint was queued; it then has no progress. + if group.owner.validate_snapshot_member(member).is_ok() { + group.owner.report_snapshot(member, delivered)?; + } + made = true; + } + } if collect && group.owner.idle() && group.pending.is_none() { if group.tick { match group.owner.tick() { @@ -482,6 +514,7 @@ impl Worker { || group.settled.is_some() || (group.failed.is_none() && (group.tick + || !group.snapshot_hints.is_empty() || !group.deferred.is_empty() || group.completion.is_some() || group.owner.has_work())) diff --git a/crates/cellar/durability/raft/group_snapshot_transport/codec.rs b/crates/cellar/durability/raft/group_snapshot_transport/codec.rs new file mode 100644 index 000000000..a1081f1a5 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/codec.rs @@ -0,0 +1,61 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Part; +use sha2::{Digest, Sha256}; + +pub(in crate::raft::transport) const START_MAGIC: &[u8; 4] = b"CLGS"; +pub(super) const DATA_MAGIC: &[u8; 4] = b"CLGD"; +pub(super) const ACK_MAGIC: &[u8; 4] = b"CLGA"; +pub(super) const START: &[u8] = b"cellar-group-snapshot-start-1"; +pub(super) const DATA: &[u8] = b"cellar-group-snapshot-data-1"; +pub(super) const ACK: &[u8] = b"cellar-group-snapshot-admitted-1"; +pub(super) const CHUNK_BYTES: usize = 64 * 1024; +pub(super) const MESSAGE_BYTES: usize = 128 * 1024; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Start { + pub route: Route, + pub digest: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Packet { + pub sequence: u64, + pub part: Option, + pub offset: u64, + pub length: u64, + pub digest: [u8; 32], +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Ack { + pub sequence: u64, +} + +pub(super) fn digest(body: &[u8]) -> [u8; 32] { + Sha256::digest(body).into() +} + +pub(super) fn message(body: &[u8], route: &Route) -> Result { + if body.len() > MESSAGE_BYTES { + return Err(TransportError::Limit); + } + frame::scan_message(body, 0)?; + let message = Message::parse_from_bytes(body).map_err(|_| TransportError::Protocol)?; + let (source, target) = route.decode_group()?; + if message.get_msg_type() != MessageType::MsgSnapshot + || !message.has_snapshot() + || message.from != source.member.get() + || message.to != target.member.get() + || message.term == 0 + || message.get_snapshot().is_empty() + || !message.entries.is_empty() + || !message.context.is_empty() + { + return Err(TransportError::Identity); + } + Ok(message) +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs new file mode 100644 index 000000000..a62f24f07 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -0,0 +1,180 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete checkpoints on dedicated authenticated data connections. A stream +//! reply proves admission only; Raft emits its own response after persistence. +use super::*; +use crate::raft::cell_groups::{ + CellGroupApplier, CellGroupCheckpointError, CellGroupCheckpointPool, +}; +use std::sync::Weak; +mod codec; +mod receive; +mod send; +mod workers; +pub(super) use codec::START_MAGIC; +use codec::*; + +pub(super) struct Endpoint { + applier: CellGroupApplier, + route: Arc>, + active: AtomicBool, +} + +/// Retain with the group's controller registration. Dropping this handle revokes +/// new and in-flight streams; already admitted Raft work keeps its own owners. +pub struct CellGroupCheckpointRegistration { + state: Weak, + endpoint: Arc, +} + +impl Drop for CellGroupCheckpointRegistration { + fn drop(&mut self) { + self.endpoint.active.store(false, Ordering::Release); + let Some(state) = self.state.upgrade() else { + return; + }; + if let Ok(mut registry) = state.registry.lock() { + let history = self.endpoint.applier.history(); + if registry + .group_checkpoints + .get(&history) + .is_some_and(|old| old.ptr_eq(&Arc::downgrade(&self.endpoint))) + { + registry.group_checkpoints.remove(&history); + } + }; + } +} + +impl RaftTransport { + /// Configure once per node, before accepting snapshots. Share this same pool + /// with local checkpoint publication. Scratch comes from the owning log. + pub fn enable_group_checkpoints( + &self, + pool: CellGroupCheckpointPool, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + state.runtime.shared()?; + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if registry.group_checkpoint_pool.is_some() { + return Err(TransportError::Identity); + } + registry.group_checkpoint_pool = Some(pool); + Ok(()) + } + + pub fn register_group_checkpoint( + &self, + applier: CellGroupApplier, + ) -> Result { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + if !applier.belongs_to_engine(&runtime.log().owner()) { + return Err(TransportError::Identity); + } + let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let history = applier.history(); + let route = registry + .physical_groups + .get(&history) + .ok_or(TransportError::UnknownGroup)? + .clone(); + if registry + .group_checkpoints + .get(&history) + .and_then(Weak::upgrade) + .is_some() + { + return Err(TransportError::Identity); + } + if registry.group_checkpoints.len() >= state.options.max_groups { + return Err(TransportError::Overloaded); + } + let endpoint = Arc::new(Endpoint { + applier, + route, + active: AtomicBool::new(true), + }); + registry + .group_checkpoints + .insert(history, Arc::downgrade(&endpoint)); + Ok(CellGroupCheckpointRegistration { + state: Arc::downgrade(state), + endpoint, + }) + } +} + +impl State { + fn checkpoint_endpoint( + &self, + history: &GroupHistory, + ) -> Result<(Arc, CellGroupCheckpointPool), TransportError> { + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let pool = registry + .group_checkpoint_pool + .clone() + .ok_or(TransportError::Refused)?; + let endpoint = registry + .group_checkpoints + .get(history) + .and_then(Weak::upgrade) + .ok_or(TransportError::Refused)?; + if !endpoint.active.load(Ordering::Acquire) + || !registry + .physical_groups + .get(history) + .is_some_and(|route| Arc::ptr_eq(route, &endpoint.route)) + { + return Err(TransportError::StalePeer); + } + Ok((endpoint, pool)) + } + + fn checkpoint_current( + &self, + endpoint: &Arc, + wire: &Route, + ) -> Result, TransportError> { + let (_, target) = wire.decode_group()?; + let (current, _) = self.checkpoint_endpoint(&target.scope)?; + if !Arc::ptr_eq(¤t, endpoint) { + return Err(TransportError::StalePeer); + } + self.authorize_physical(wire) + } +} + +fn checkpoint_error(error: CellGroupCheckpointError) -> TransportError { + match error { + CellGroupCheckpointError::Busy => TransportError::Overloaded, + CellGroupCheckpointError::Limit => TransportError::Limit, + other => TransportError::Invalid(other.to_string()), + } +} + +// Covers two in-flight chunk windows, the producer's copy, protobuf decoding, +// authentication/header copies and reply framing. Physical metadata and image +// workspaces are separately charged by the node-wide checkpoint pool. +fn wire_charge(options: &TransportOptions) -> usize { + 4 * CHUNK_BYTES + 4 * MESSAGE_BYTES + 4 * options.max_header_bytes +} + +struct Admission { + _node: Permit, + _peer: Permit, + _transfer: Permit, +} + +pub(super) fn revoke(registry: &mut Registry, history: &GroupHistory) { + if let Some(endpoint) = registry + .group_checkpoints + .remove(history) + .and_then(|weak| weak.upgrade()) + { + endpoint.active.store(false, Ordering::Release); + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/receive.rs b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs new file mode 100644 index 000000000..5a760b235 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +impl State { + pub(in crate::raft::transport) async fn receive_group_checkpoint< + S: AsyncRead + AsyncWrite + Unpin, + >( + self: &Arc, + stream: &mut S, + exporter: &[u8; 32], + prefix: [u8; frame::PREFIX], + ) -> Result<(), TransportError> { + let _connection = self + .snapshot_connections + .clone() + .try_acquire_owned() + .map_err(|_| TransportError::Overloaded)?; + let wire = snapshots::Wire { + auth: &self.auth, + exporter, + options: &self.options, + max_body: MESSAGE_BYTES, + }; + let head = self + .bounded( + None, + snapshots::read_bulk_head( + stream, + Some(prefix), + START_MAGIC, + MESSAGE_BYTES, + &self.options, + ), + ) + .await?; + let start: Start = wire.decode(START_MAGIC, START, &head, None)?; + let binding = head.tag; + let pool = self.authorize_physical(&start.route)?; + let (_, target) = start.route.decode_group()?; + let (endpoint, checkpoints) = self.checkpoint_endpoint(&target.scope)?; + let charge = wire_charge(&self.options); + let admission = Arc::new(Admission { + _node: self + .incoming + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: pool + .incoming + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: self + .snapshot_receive_budget + .acquire(snapshots::memory_bound(&self.options)) + .ok_or(TransportError::Overloaded)?, + }); + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(self.options.snapshot_timeout_ms); + let mut body = vec![0; head.payload_len]; + self.snapshot_io(&pool, deadline, async { + stream.read_exact(&mut body).await?; + Ok(()) + }) + .await?; + if digest(&body) != start.digest { + return Err(TransportError::Authentication); + } + let message = codec::message(&body, &start.route)?; + drop(body); + self.checkpoint_current(&endpoint, &start.route)?; + let receiver = workers::receiver(self, endpoint.clone(), checkpoints, admission.clone())?; + let mut sequence = 0_u64; + loop { + self.checkpoint_current(&endpoint, &start.route)?; + let head = self + .snapshot_io( + &pool, + deadline, + snapshots::read_bulk_head( + stream, + None, + DATA_MAGIC, + CHUNK_BYTES.min(self.options.snapshot_chunk_bytes), + &self.options, + ), + ) + .await?; + let packet: Packet = wire.decode(DATA_MAGIC, DATA, &head, Some(&binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay); + } + let mut bytes = vec![0; head.payload_len]; + self.snapshot_io(&pool, deadline, async { + stream.read_exact(&mut bytes).await?; + Ok(()) + }) + .await?; + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication); + } + self.checkpoint_current(&endpoint, &start.route)?; + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) + } + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.snapshot_io(&pool, deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + drop(receiver.input); + let prepared = self + .snapshot_io(&pool, deadline, async { + receiver.result.await.map_err(|_| TransportError::Closed)? + }) + .await?; + self.checkpoint_current(&endpoint, &start.route)?; + // The prepared closure carries physical pins and checkpoint admission + // into the bounded runtime request and its exact persistence completion. + self.snapshot_io(&pool, deadline, async { + self.runtime + .shared()? + .step_checkpoint(target.scope, message, prepared) + .await?; + Ok(()) + }) + .await?; + self.checkpoint_current(&endpoint, &start.route)?; + self.accepted.fetch_add(1, Ordering::Relaxed); + let ack = wire.encode(ACK_MAGIC, ACK, &binding, &Ack { sequence }, &[])?; + self.snapshot_io(&pool, deadline, async { + write_head(stream, &ack).await?; + stream.flush().await?; + Ok(()) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/send.rs b/crates/cellar/durability/raft/group_snapshot_transport/send.rs new file mode 100644 index 000000000..c189a4f56 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/send.rs @@ -0,0 +1,234 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl RaftTransport { + pub(in crate::raft::transport) fn submit_group_checkpoint( + &self, + history: &GroupHistory, + message: Message, + ) -> Result { + let state = &self.handle.0; + let (endpoint, checkpoints) = state.checkpoint_endpoint(history)?; + let group = state + .runtime + .shared()? + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + let local = group.identity(); + let peer = endpoint + .route + .peers + .iter() + .find(|p| p.replica.member.get() == message.to) + .filter(|p| p.replica.node != local.node) + .cloned() + .ok_or(TransportError::Identity)?; + shared::permitted(&group, &[local, &peer.replica])?; + let route = Route::new_group( + local, + state.auth.source(), + &peer.replica, + &peer.label, + endpoint.route.schema_digest, + )?; + if message.compute_size() as usize > MESSAGE_BYTES { + return Err(TransportError::Limit); + } + let body = message + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + codec::message(&body, &route)?; + let pool = state + .registry + .lock() + .map_err(|_| TransportError::Closed)? + .pools + .get(&peer.replica.node) + .cloned() + .ok_or(TransportError::StalePeer)?; + let charge = wire_charge(&state.options); + let admission = Arc::new(Admission { + _node: state + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: pool + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: state + .snapshot_source_budget + .acquire(snapshots::memory_bound(&state.options)) + .ok_or(TransportError::Overloaded)?, + }); + let guard = state.reserve_task()?; + let mut chunks = workers::source( + state, + endpoint.clone(), + checkpoints, + message.get_snapshot().clone(), + admission.clone(), + )?; + let (result, wait) = oneshot::channel(); + let state = state.clone(); + state.clone().spawn_reserved(guard, async move { + let _admission = admission; + let outcome = state + .send_group_checkpoint(&endpoint, &peer, &pool, route, body, &mut chunks) + .await; + // Closing the channel wakes a blocked physical producer before any + // runtime hint can await a busy consensus owner. + drop(chunks); + let delivered = outcome.is_ok(); + if !delivered { + state.failed.fetch_add(1, Ordering::Relaxed); + } + let _ = result.send(outcome); + let hint = SendRoute::Physical { + route: endpoint.route.clone(), + peer: peer.clone(), + }; + // Failure feedback is required for Raft to leave Snapshot state. + // Retain this admitted transfer through reserved hint admission; + // only transport shutdown may cancel it, not an I/O reply timeout. + let mut stop = state.stop.subscribe(); + if !*stop.borrow() { + crate::asyncrt::select! { + _ = stop.changed() => (), + _ = state.runtime.delivery_hint(&hint, peer.replica.member.get(), delivered, true) => (), + } + } + })?; + Ok(Delivery { result: wait }) + } +} + +impl State { + async fn send_group_checkpoint( + &self, + endpoint: &Arc, + peer: &RaftPeer, + pool: &Arc, + route: Route, + body: Vec, + chunks: &mut mpsc::Receiver, + ) -> Result<(), TransportError> { + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(self.options.snapshot_timeout_ms); + let _connection = self.outgoing_connections.acquire(Lane::Data); + let mut stream = self + .snapshot_io(pool, deadline, async { + let tcp = TcpStream::connect(peer.data_address).await?; + tcp.set_nodelay(true)?; + let tcp = self.observed_socket(tcp, metrics::Lane::Data); + let name = rustls::pki_types::ServerName::try_from(peer.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + tokio_rustls::TlsConnector::from(self.tls.client.clone()) + .connect(name, tcp) + .await + .map_err(|e| TransportError::Tls(e.to_string())) + }) + .await?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], frame::EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + let wire = snapshots::Wire { + auth: &self.auth, + exporter: &exporter, + options: &self.options, + max_body: MESSAGE_BYTES, + }; + if !endpoint.active.load(Ordering::Acquire) || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + let start = wire.encode( + START_MAGIC, + START, + &[], + &Start { + route, + digest: digest(&body), + }, + &body, + )?; + let binding = start.tag; + self.snapshot_io(pool, deadline, async { + write_head(&mut stream, &start).await?; + stream.write_all(&body).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + drop(body); + let mut sequence = 0_u64; + loop { + if !endpoint.active.load(Ordering::Acquire) + || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + let next = self + .snapshot_io(pool, deadline, async { + chunks.recv().await.ok_or(TransportError::Closed) + }) + .await?; + let (part, offset, length, bytes) = match next { + workers::Output::Chunk(chunk) => { + (Some(chunk.part), chunk.offset, chunk.length, chunk.bytes) + } + workers::Output::End(result) => { + result?; + (None, 0, 0, Vec::new()) + } + }; + if bytes.len() > CHUNK_BYTES.min(self.options.snapshot_chunk_bytes) { + return Err(TransportError::Limit); + } + let ending = part.is_none(); + let header = Packet { + sequence, + part, + offset, + length, + digest: digest(&bytes), + }; + let head = wire.encode(DATA_MAGIC, DATA, &binding, &header, &bytes)?; + self.snapshot_io(pool, deadline, async { + write_head(&mut stream, &head).await?; + stream.write_all(&bytes).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + let head = self + .snapshot_io( + pool, + deadline, + snapshots::read_bulk_head(&mut stream, None, ACK_MAGIC, 0, &self.options), + ) + .await?; + let ack: Ack = wire.decode(ACK_MAGIC, ACK, &head, Some(&binding))?; + if ack.sequence != sequence { + return Err(TransportError::Replay); + } + if !endpoint.active.load(Ordering::Acquire) || !self.current_physical(&endpoint.route, peer) + { + return Err(TransportError::StalePeer); + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/workers.rs b/crates/cellar/durability/raft/group_snapshot_transport/workers.rs new file mode 100644 index 000000000..2440ee798 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/workers.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Blocking owners retain task accounting and wire admission through actual I/O. +//! Channel closure aborts a partial receive; only an explicit End can seal it. +use super::*; +use crate::raft::cell_groups::{checkpoint::wire::Chunk, PreparedCellGroupCheckpoint}; + +pub(super) enum Input { + Chunk(Chunk), + End, +} + +pub(super) enum Output { + Chunk(Chunk), + End(Result<(), TransportError>), +} + +pub(super) fn source( + state: &Arc, + endpoint: Arc, + pool: CellGroupCheckpointPool, + snapshot: ::raft::prelude::Snapshot, + admission: Arc, +) -> Result, TransportError> { + let guard = state.reserve_task()?; + let (tx, rx) = mpsc::channel(1); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let result = (|| { + let mut source = endpoint.applier.capture_checkpoint(&pool, &snapshot)?; + source.stream(|chunk| { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked source")); + } + tx.blocking_send(Output::Chunk(Chunk::copy(chunk))) + .map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::new( + std::io::ErrorKind::BrokenPipe, + "snapshot connection closed", + )) + }) + }) + })() + .map_err(checkpoint_error); + let _ = tx.blocking_send(Output::End(result)); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(rx) +} + +pub(super) struct Receive { + pub input: mpsc::Sender, + pub result: oneshot::Receiver>, +} + +pub(super) fn receiver( + state: &Arc, + endpoint: Arc, + pool: CellGroupCheckpointPool, + admission: Arc, +) -> Result { + let guard = state.reserve_task()?; + let scratch = state.runtime.log().snapshot_scratch(); + let directory = state.runtime.log().limits().directory.clone(); + let (input, mut rx) = mpsc::channel(1); + let (tx, result) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let outcome = (|| { + let mut receiver = pool.receive(endpoint.applier.clone(), scratch, directory)?; + loop { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked receiver")); + } + match rx.blocking_recv() { + Some(Input::Chunk(chunk)) => chunk.append(&mut receiver)?, + Some(Input::End) => return receiver.finish(), + None => return Err(CellGroupCheckpointError::Invalid("incomplete stream")), + } + } + })() + .map_err(checkpoint_error); + let _ = tx.send(outcome); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(Receive { input, result }) +} diff --git a/crates/cellar/durability/raft/snapshot_transport.rs b/crates/cellar/durability/raft/snapshot_transport.rs index 8fdfa15a0..e01105411 100644 --- a/crates/cellar/durability/raft/snapshot_transport.rs +++ b/crates/cellar/durability/raft/snapshot_transport.rs @@ -528,7 +528,7 @@ impl State { Ok((route, peer, pool)) } - async fn snapshot_io( + pub(super) async fn snapshot_io( &self, pool: &Pool, deadline: u64, diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index 011bb89c4..5c5f7ff83 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -12,6 +12,8 @@ use cellar_logic::cell_group::GroupHistory; #[path = "transport_runtime.rs"] mod runtime_adapter; use runtime_adapter::TransportRuntime; +#[path = "group_snapshot_transport/mod.rs"] +mod group_snapshots; #[path = "transport_shared.rs"] mod shared; use super::transport_frame::{self as frame, Disposition, FrameHead, RequestHeader, Route}; @@ -20,6 +22,7 @@ use crate::asyncrt::DomainToken; use crate::peer_auth::PeerAuth; use ::raft::prelude::{Message, MessageType}; use cellar_logic::consensus::{ClusterId, GroupScope, NamespaceId, NodeId, ReplicaIdentity}; +pub use group_snapshots::CellGroupCheckpointRegistration; use protobuf::Message as _; use serde::{Deserialize, Serialize}; pub(crate) use shared::PhysicalRegistration; @@ -295,6 +298,8 @@ struct Registry { tasks: usize, groups: BTreeMap>, physical_groups: BTreeMap>>, + group_checkpoint_pool: Option, + group_checkpoints: BTreeMap>, query_groups: BTreeMap>, cell_query_groups: BTreeMap>>, command_origins: BTreeMap<(ClusterId, NamespaceId, NodeId), Arc>, @@ -525,6 +530,8 @@ impl RaftTransport { tasks: 0, groups: BTreeMap::new(), physical_groups: BTreeMap::new(), + group_checkpoint_pool: None, + group_checkpoints: BTreeMap::new(), query_groups: BTreeMap::new(), cell_query_groups: BTreeMap::new(), command_origins: BTreeMap::new(), @@ -1409,6 +1416,14 @@ impl State { .map_err(|error| TransportError::Tls(error.to_string()))?; return self.receive_seed(&mut stream, &exporter, prefix).await; } + if &prefix[..4] == group_snapshots::START_MAGIC { + if lane != Lane::Data || sequence != 0 { + return Err(TransportError::Protocol); + } + return self + .receive_group_checkpoint(&mut stream, &exporter, prefix) + .await; + } if &prefix[..4] == snapshots::REQUEST_MAGIC { if lane != Lane::Data || sequence != 0 { return Err(TransportError::Protocol); diff --git a/crates/cellar/durability/raft/transport_runtime.rs b/crates/cellar/durability/raft/transport_runtime.rs index aac228432..d493007e9 100644 --- a/crates/cellar/durability/raft/transport_runtime.rs +++ b/crates/cellar/durability/raft/transport_runtime.rs @@ -84,7 +84,12 @@ impl TransportRuntime { } Ok(()) } - (Self::Shared(runtime), SendRoute::Physical { route, .. }) if !snapshot => { + (Self::Shared(runtime), SendRoute::Physical { route, .. }) => { + if snapshot { + return runtime + .report_snapshot(route.scope, member, delivered) + .await; + } if !delivered { runtime.report_unreachable(route.scope, member).await?; } diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 37b7a74cc..43c9548cc 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -67,6 +67,7 @@ impl RaftTransport { Ok(PhysicalRegistration { state: Arc::downgrade(&self.handle.0), route, + checkpoint: None, }) } @@ -182,6 +183,7 @@ impl RaftTransport { } let starts = state.prepare_pools(&mut registry, &physical)?; let route = Arc::new(route); + group_snapshots::revoke(&mut registry, &route.scope); registry.physical_groups.insert(route.scope, route.clone()); registry.prune_pools(); drop(registry); @@ -190,6 +192,7 @@ impl RaftTransport { drop(PhysicalRegistration { state: Arc::downgrade(state), route, + checkpoint: None, }); } return Err(error); @@ -202,13 +205,14 @@ impl RaftTransport { state.runtime.shared()?; let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; + group_snapshots::revoke(&mut registry, history); registry.physical_groups.remove(history); registry.prune_pools(); Ok(()) } /// Queue a physical-group frame on existing bounded per-node lanes. Delivery - /// acknowledges receiver admission only. Complete group snapshots require CG-05. + /// acknowledges receiver admission only. Snapshots use the bounded checkpoint stream. pub fn submit_physical( &self, history: &GroupHistory, @@ -217,7 +221,7 @@ impl RaftTransport { let state = &self.handle.0; let runtime = state.runtime.shared()?; if message.get_msg_type() == MessageType::MsgSnapshot { - return Err(TransportError::Refused); + return self.submit_group_checkpoint(history, message); } let lane = frame::classify(&message)?; let size = message.compute_size() as usize; @@ -408,6 +412,7 @@ impl Registry { pub(crate) struct PhysicalRegistration { state: std::sync::Weak, route: Arc, + pub(crate) checkpoint: Option, } impl Drop for PhysicalRegistration { @@ -421,6 +426,7 @@ impl Drop for PhysicalRegistration { .get(&self.route.scope) .is_some_and(|r| Arc::ptr_eq(r, &self.route)) { + group_snapshots::revoke(&mut registry, &self.route.scope); registry.physical_groups.remove(&self.route.scope); registry.prune_pools(); } diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index c87a1478b..6d598fe08 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,31 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_refuses_invalid_streams_and_preserves_other_group_progress() -> Result<()> { + runtime_fixture::tcp_protocol().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_accepted_transfer_survives_dropped_delivery() -> Result<()> { + runtime_fixture::tcp_cancelled_delivery().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_failed_transfer_allows_raft_retry() -> Result<()> { + runtime_fixture::tcp_retry().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_materialized_catches_up_and_reopens() -> Result<()> { + runtime_fixture::tcp_publication(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_tcp_extents_catch_up_and_reopen() -> Result<()> { + runtime_fixture::tcp_publication(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_local_publication_preserves_suffix_and_catches_up() -> Result<()> { runtime_fixture::local_publication(false).await diff --git a/crates/cellar/tests/support/cell_group_checkpoint_local.rs b/crates/cellar/tests/support/cell_group_checkpoint_local.rs index beeeb0b7e..d0d85b7c7 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint_local.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint_local.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Local publication and actual Raft-generated snapshot catch-up. Transfer uses -//! the native streaming callback; this does not qualify authenticated snapshot RPC. +//! Local publication, retained suffixes and real Raft-generated catch-up over +//! native streaming and authenticated TLS, followed by cold source recovery. use super::*; use raft::{GetEntriesContext, Storage}; @@ -60,6 +60,36 @@ impl RuntimeCluster { } pub(crate) async fn local_publication(rooted: bool) -> Result<()> { + publication(rooted, Transfer::Native).await +} + +pub(crate) async fn tcp_publication(rooted: bool) -> Result<()> { + publication(rooted, Transfer::Tcp).await +} + +pub(crate) async fn tcp_protocol() -> Result<()> { + publication(false, Transfer::Probes).await +} + +pub(crate) async fn tcp_cancelled_delivery() -> Result<()> { + publication(false, Transfer::Cancelled).await +} + +pub(crate) async fn tcp_retry() -> Result<()> { + publication(false, Transfer::Retry).await +} + +enum Transfer { + Native, + Tcp, + Probes, + Cancelled, + Retry, +} + +async fn publication(rooted: bool, transfer: Transfer) -> Result<()> { + let tcp = !matches!(transfer, Transfer::Native); + let probes = matches!(transfer, Transfer::Probes); let (mut cluster, _) = checkpoint_fixture::populated_partition(rooted).await?; let route = cluster.birth.route().clone(); let group = cluster.voters[0].log.cell_group(&route.history)?; @@ -214,6 +244,22 @@ pub(crate) async fn local_publication(rooted: bool) -> Result<()> { GroupEntryLimits::default(), 100, )?; + let source_applier = if tcp { + Some(CellGroupApplier::new( + group.clone(), + cluster.voters[0].pages.take().context("source pages")?, + checkpoint_receive::config( + cluster.voters[0].images.clone(), + cluster.voters[0].extents.clone(), + cluster._root.path(), + )?, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) + } else { + None + }; let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1000).await?; let sql = CellGroupSqlPool::start( &fixture.cluster.voters[2].log, @@ -228,12 +274,80 @@ pub(crate) async fn local_publication(rooted: bool) -> Result<()> { )?; fixture.appliers.insert((2, route.history), (sql, applier)); fixture.snapshot_sources.insert((0, route.history), anchor); + let mut registrations = Vec::new(); + let mut transport_pools = Vec::new(); + if let Some(source_applier) = source_applier { + fixture.appliers.insert( + (0, route.history), + ( + CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?, + source_applier, + ), + ); + let other = if probes { + Some(fixture.add_group(91).await?) + } else { + None + }; + fixture.connect_tcp().await?; + for n in [0, 2] { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let transport = &fixture.transports[n]; + transport.enable_group_checkpoints(pool.clone())?; + assert!(matches!( + transport.enable_group_checkpoints(pool.clone()), + Err(TransportError::Identity) + )); + let applier = fixture.appliers[&(n, route.history)].1.clone(); + registrations.push(transport.register_group_checkpoint(applier.clone())?); + assert!(matches!( + transport.register_group_checkpoint(applier), + Err(TransportError::Identity) + )); + transport_pools.push(pool); + } + assert!(matches!( + fixture.transports[1] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone()), + Err(TransportError::Identity) + )); + if let Some(other) = other { + network_probes::probe( + &mut fixture, + route.history, + &snapshot, + &transport_pools[1], + other, + &mut registrations, + ) + .await?; + } + } + fixture.retry_snapshot = matches!(transfer, Transfer::Retry); + if fixture.retry_snapshot { + drop(registrations.pop().context("receiver registration")?); + } + fixture.drop_snapshot_delivery = matches!(transfer, Transfer::Cancelled); fixture.runtimes[0].campaign(route.history).await?; fixture.drive(None).await?; assert!( fixture.snapshot_transfers >= 1, "Raft must generate a snapshot after compaction" ); + if fixture.retry_snapshot { + assert!(fixture.failed_snapshot); + assert!(fixture.transports[0].usage().failed >= 1); + } let restored = fixture.cluster.voters[2].log.cell_group(&route.history)?; assert_eq!(restored.checkpoint_manifest()?.as_ref(), Some(&cut)); let source = group.group_manifest()?.context("source root")?; @@ -250,7 +364,11 @@ pub(crate) async fn local_publication(rooted: bool) -> Result<()> { drop(restored); drop(root); drop(group); + drop(registrations); fixture.close().await?; + for pool in transport_pools { + assert_eq!(pool.usage().sources, 0); + } // Reopen the actual leader engine and physical closures. Cache-only suffix // preservation cannot pass this check if publication cleaned durable entries. let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); diff --git a/crates/cellar/tests/support/cell_group_network_probes.rs b/crates/cellar/tests/support/cell_group_network_probes.rs new file mode 100644 index 000000000..55742d958 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_network_probes.rs @@ -0,0 +1,281 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independent wire client exercises authenticated framing on real TLS sockets. +use super::*; +use hmac::{Hmac, Mac}; +use protobuf::Message as _; +use serde_json::{json, Value}; +use tokio::io::{AsyncReadExt, AsyncWriteExt}; +use tokio::net::TcpStream; +use tokio_rustls::client::TlsStream; + +struct Connection { + stream: TlsStream, + exporter: [u8; 32], +} + +impl Connection { + async fn open(address: std::net::SocketAddr) -> Result { + let mut roots = rustls::RootCertStore::empty(); + roots.add( + include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into(), + )?; + let mut config = rustls::ClientConfig::builder_with_provider(Arc::new( + rustls::crypto::ring::default_provider(), + )) + .with_protocol_versions(&[&rustls::version::TLS13])? + .with_root_certificates(roots) + .with_no_client_auth(); + config.alpn_protocols = vec![b"cellar-raft-1".to_vec()]; + let stream = tokio_rustls::TlsConnector::from(Arc::new(config)) + .connect( + rustls::pki_types::ServerName::try_from("localhost")?, + TcpStream::connect(address).await?, + ) + .await?; + let exporter = + stream + .get_ref() + .1 + .export_keying_material([0; 32], b"EXPORTER-cellar-raft-1", None)?; + Ok(Self { stream, exporter }) + } + + async fn frame( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + header: Value, + body: &[u8], + ) -> Result<[u8; 32]> { + let header = serde_json::to_vec(&header)?; + let mut prefix = [0; 20]; + prefix[..4].copy_from_slice(magic); + prefix[4..8].copy_from_slice(&1u32.to_be_bytes()); + prefix[8..12].copy_from_slice(&(header.len() as u32).to_be_bytes()); + prefix[12..].copy_from_slice(&(body.len() as u64).to_be_bytes()); + let mut mac = Hmac::::new_from_slice(&[17; 32])?; + for part in [purpose, &self.exporter, binding, &prefix, &header] { + mac.update(&(part.len() as u64).to_be_bytes()); + mac.update(part); + } + let tag: [u8; 32] = mac.finalize().into_bytes().into(); + self.stream.write_all(&prefix).await?; + self.stream.write_all(&header).await?; + self.stream.write_all(&tag).await?; + self.stream.write_all(body).await?; + self.stream.flush().await?; + Ok(tag) + } + + async fn refused(mut self) -> Result<()> { + let mut byte = [0]; + let result = + tokio::time::timeout(Duration::from_secs(5), self.stream.read(&mut byte)).await?; + assert!( + !matches!(result, Ok(n) if n > 0), + "refused stream must not acknowledge admission" + ); + Ok(()) + } +} + +fn peer(peer: &RaftPeer) -> Value { + json!({"node":peer.replica.node.bytes(), "disk":peer.replica.disk.bytes(), "process":peer.replica.process.bytes(), "member":peer.replica.member.get().to_string(), "label":peer.label}) +} + +pub(super) async fn probe( + fixture: &mut RuntimeCluster, + history: GroupHistory, + snapshot: &raft::prelude::Snapshot, + pool: &CellGroupCheckpointPool, + other: GroupHistory, + registrations: &mut [CellGroupCheckpointRegistration], +) -> Result<()> { + let route = fixture.tcp_route(history)?; + let group = fixture.cluster.voters[2].log.cell_group(&history)?; + let before = group.group_manifest()?; + let mut message = Message { + from: 1, + to: 3, + term: snapshot.get_metadata().term, + ..Default::default() + }; + message.set_msg_type(MessageType::MsgSnapshot); + message.set_snapshot(snapshot.clone()); + let body = message.write_to_bytes()?; + let start = json!({"route": { + "scope": {"kind":"Group", "identity":[history.cluster.bytes(), history.namespace.bytes(), history.group.bytes(), history.incarnation.bytes()]}, + "schema":route.schema_digest, "source":peer(&route.peers[0]), "target":peer(&route.peers[2]) + }, "digest": <[u8; 32]>::from(Sha256::digest(&body))}); + let address = fixture.transports[2].data_address(); + for case in 0..7 { + let mut connection = Connection::open(address).await?; + let mut header = start.clone(); + match case { + 0 => { + connection.exporter[0] ^= 1; + } + 1 => { + header["route"]["source"]["process"] = json!(vec![244; 16]); + } + 2 => { + header["digest"] = json!(vec![0; 32]); + } + _ => (), + } + let binding = connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + header, + &body, + ) + .await?; + if case >= 3 { + let sequence = if case == 3 { 1 } else { 0 }; + let binding = if case == 4 { [241; 32] } else { binding }; + // Case 5 sends an empty header part, case 6 a premature End. + let part = if case == 6 { + Value::Null + } else { + json!({"kind":"Header"}) + }; + connection.frame(b"CLGD", b"cellar-group-snapshot-data-1", &binding, + json!({"sequence":sequence, "part":part, "offset":0, "length":0, "digest":<[u8; 32]>::from(Sha256::digest([]))}), &[]).await?; + } + connection.refused().await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + assert_eq!( + group.group_manifest()?, + before, + "fault {case} changed selected root" + ); + assert_eq!(group.checkpoint_manifest()?, None); + } + // Authenticated start with a declared oversized data body refuses before + // allocation or waiting for that body. + let mut connection = Connection::open(address).await?; + connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + start.clone(), + &body, + ) + .await?; + let mut oversized = [0; 20]; + oversized[..4].copy_from_slice(b"CLGD"); + oversized[4..8].copy_from_slice(&1u32.to_be_bytes()); + oversized[8..12].copy_from_slice(&1u32.to_be_bytes()); + oversized[12..].copy_from_slice(&65537u64.to_be_bytes()); + connection.stream.write_all(&oversized).await?; + connection.refused().await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + // A disconnected stream cannot use channel closure as successful End. + let mut connection = Connection::open(address).await?; + let binding = connection + .frame( + b"CLGS", + b"cellar-group-snapshot-start-1", + &[], + start.clone(), + &body, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources == 0 { + tokio::task::yield_now().await; + } + }) + .await?; + // The checkpoint owns receive admission while another group on the same + // runtimes/transport/engines elects, commits and applies a real cell write. + fixture.runtimes[0].campaign(other).await?; + fixture.settle(&[other]).await?; + let other_route = fixture.cluster.voters[0] + .log + .cell_group(&other)? + .group_manifest()? + .context("other root")? + .route() + .clone(); + let bytes = images::image( + None, + "CREATE TABLE progress(n); INSERT INTO progress VALUES(17);", + )?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let accepted = fixture.runtimes[0] + .propose( + other, + request( + &other_route, + super::history(&other_route, "unrelated")?, + 1, + &cut, + )?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(10), async { + while fixture.applied[&other] + .iter() + .any(|index| *index < accepted.position.index) + { + fixture.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert_eq!(pool.usage().sources, 1); + // Revocation while a stream is admitted must refuse its authenticated End. + fixture.transports[2].remove_physical_group(&history)?; + connection.frame(b"CLGD", b"cellar-group-snapshot-data-1", &binding, + json!({"sequence":0, "part":null, "offset":0, "length":0, "digest":<[u8; 32]>::from(Sha256::digest([]))}), &[]).await?; + connection.refused().await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + fixture.transports[2].install_physical_group(route)?; + // Replacing the old guard after registering must not remove the new owner. + registrations[1] = fixture.transports[2] + .register_group_checkpoint(fixture.appliers[&(2, history)].1.clone())?; + let mut connection = Connection::open(address).await?; + connection + .frame(b"CLGS", b"cellar-group-snapshot-start-1", &[], start, &body) + .await?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources == 0 { + tokio::task::yield_now().await; + } + }) + .await?; + drop(connection); + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + assert_eq!(group.group_manifest()?, before); + assert_eq!(group.checkpoint_manifest()?, None); + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index a83d2adad..469b1c991 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -7,7 +7,12 @@ use std::collections::BTreeMap; #[path = "cell_group_checkpoint_local.rs"] mod checkpoint_local; -pub(super) use checkpoint_local::{local_publication, local_shutdown}; +#[path = "cell_group_network_probes.rs"] +mod network_probes; +pub(super) use checkpoint_local::{ + local_publication, local_shutdown, tcp_cancelled_delivery, tcp_protocol, tcp_publication, + tcp_retry, +}; #[path = "cell_group_snapshot_owner.rs"] mod snapshot_owner; @@ -26,6 +31,10 @@ pub(super) use tcp::physical_transport; struct RuntimeCluster { snapshot_sources: BTreeMap<(usize, GroupHistory), PreparedGroupManifest>, snapshot_transfers: usize, + drop_snapshot_delivery: bool, + retry_snapshot: bool, + failed_snapshot: bool, + retry_checkpoint: Option, transports: Vec, appliers: BTreeMap<(usize, GroupHistory), (CellGroupSqlPool, CellGroupApplier)>, cluster: Cluster, @@ -89,6 +98,10 @@ impl RuntimeCluster { Ok(Self { snapshot_sources: BTreeMap::new(), snapshot_transfers: 0, + drop_snapshot_delivery: false, + retry_snapshot: false, + failed_snapshot: false, + retry_checkpoint: None, transports: Vec::new(), appliers: BTreeMap::new(), cluster, @@ -157,11 +170,43 @@ impl RuntimeCluster { let messages = std::mem::take(&mut self.network); if !self.transports.is_empty() { for (history, message) in messages { + let snapshot = message.get_msg_type() == MessageType::MsgSnapshot; let transport = &self.transports[message.from as usize - 1]; - transport - .submit_physical(&history, message)? - .accepted() - .await?; + let delivery = match transport.submit_physical(&history, message.clone()) { + Ok(delivery) => delivery, + Err(TransportError::Overloaded) if self.retry_snapshot && snapshot => { + self.network.push_back((history, message)); + continue; + } + Err(error) => return Err(error.into()), + }; + if snapshot && self.drop_snapshot_delivery { + drop(delivery); + } else { + let result = delivery.accepted().await; + if snapshot && self.retry_snapshot && !self.failed_snapshot { + assert!( + result.is_err(), + "revoked receiver must refuse the first actual Raft snapshot" + ); + self.failed_snapshot = true; + let to = message.to as usize - 1; + self.retry_checkpoint = + Some(self.transports[to].register_group_checkpoint( + self.appliers[&(to, history)].1.clone(), + )?); + // Do not requeue this failed snapshot. Raft must create + // another after its delivery failure is consumed. + continue; + } + result?; + } + if snapshot { + self.snapshot_transfers += 1; + } + } + if self.retry_snapshot && self.snapshot_transfers == 0 { + self.runtimes[0].tick().await?; } return Ok(()); } @@ -259,6 +304,7 @@ impl RuntimeCluster { drop(self.runtimes); drop(self.extra); drop(self.snapshot_sources); + drop(self.retry_checkpoint); self.cluster.close().await } } diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 874b21081..b04c1d52a 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -243,6 +243,42 @@ engine installation, retained-checkpoint selection, transport admission and membership catch-up are not implemented by this source API. Successful streaming supplies no receiver-install, voter, serving, archive-coverage or reclamation proof. +The shared transport has a dedicated checkpoint stream on its TLS data listener. +Configure one node `CellGroupCheckpointPool` with `enable_group_checkpoints`, shared +with local publication. Controller registration retains the exact applier and +physical route; route replacement/removal or registration drop revokes subsequent +stream admission. Ordinary consensus frames still reject snapshot payloads. + +Marker-1 start frames bind the physical group/incarnation, permanent member, +node/disk/process identities, schema and snapshot envelope to the TLS exporter. +Each chunk has a separate authenticated purpose, binds the start tag, and carries +a monotonic sequence, part identity, offset, length and body digest. Header and +payload limits are checked before allocation; chunks are at most 64 KiB. Extent +identities resolve against the receiver's already verified manifest. An explicit +authenticated End is required; EOF, cancellation, mixed cuts and incomplete +closures cannot finish a receive. Source capture pins the exact engine-retained +snapshot even when the current applied root has advanced. + +Two bounded channel windows connect the network to blocking physical owners. +Node/peer data budgets and source/receive slots remain owned through actual I/O, +even after a connection or delivery waiter disappears. The checkpoint pool +separately charges physical closure metadata/workspace; flat images use the log's +shared scratch pool. Transport close joins these owners. Membership and exact +registration checks run throughout the stream and again before runtime admission. +The verified closure enters `step_checkpoint`; its stream reply proves admission +only. Raft releases its acknowledgement after the matching persistence completion. + +Snapshot delivery feedback has 256 reserved node-wide slots and 1 MiB, separate +from ordinary calls and apply completions. The runtime coalesces one status per +known peer while retaining its reservation, then applies it after the in-flight +Ready has settled. A completed transfer remains owned until feedback admission or +transport shutdown; an I/O response timeout cannot silently discard required +failure feedback. Immediate snapshot queue refusals also report failure, allowing +Raft to leave its paused snapshot state and generate a retry. Automatic local +checkpoint selection/publication and whole-group membership orchestration remain +separate integration work. No transfer reply grants new voter, serving, archival +or reclamation authority. + An archive publishes a complete checkpoint plus a contiguous committed tail to K. The tail addresses cells and validates each base chain; administrative entries also replay. Do not rerun commands. Publish coverage only after all dependencies diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 325a1f6d4..c4db71ebb 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1582,7 +1582,9 @@ "crates/cellar/tests/support/cell_group_sql_apply.rs", "crates/cellar/tests/support/cell_group_snapshot_owner.rs", "crates/cellar/durability/raft/cell_groups/checkpoint/local.rs", - "crates/cellar/tests/support/cell_group_checkpoint_local.rs" + "crates/cellar/tests/support/cell_group_checkpoint_local.rs", + "crates/cellar/application/consensus/mod.rs", + "crates/cellar/tests/support/cell_group_network_probes.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 4c098efc35d8a81589689e8e6a0c09a3d4627dc7 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 17:21:20 -0700 Subject: [PATCH 128/568] docs: qualify CG-05 checkpoint transport and record next gaps --- .../multi-raft/cell-groups/baseline.json | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 69 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 63 ++++++++++++++++- .../multi-raft/cell-groups/protocols.md | 7 +- 4 files changed, 109 insertions(+), 34 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index c444059c9..e544df70e 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "742b1e6d4e7155e16fb1b4eba0cce3b9056944ae", + "source_revision": "ceb7ed3149202660855d32df1ec5feed7775e068", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 75169cd04..fcdfc3cb4 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -30,8 +30,10 @@ limits. Receiver verification and atomic storage installation now pass their partial gate, including cold recovery of the current and older checkpoint roots. Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate. Local checkpoint publication and native Raft-generated catch-up now pass their -partial gate. Continue with bounded authenticated snapshot transport and its -controller lifecycle, then whole-group membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic +partial gate. Authenticated checkpoint streams now pass their partial transport gate. +Continue with automatic controller-owned checkpoint publication and the remaining +fault/pressure boundaries, then whole-group membership and lost-disk recovery. +CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. @@ -70,14 +72,26 @@ leader engine/root recovery and re-stream, plus held publication shutdown in bot image formats. Materialized source recovery uses its fixture filename resolver; extent recovery and destination replay use production adapters. -The most critical gap is bounded authenticated snapshot RPC and controller-owned -source/publication lifecycle. Native streaming callbacks do not qualify network -framing/authentication, retries/cancellation, failed snapshot reporting or automatic -checkpoint policy. Ordinary peer snapshot ingress remains closed. Preserve the -post-persistence receiver acknowledgement boundary and exact retained cut. Then -qualify receiver crash/sync-error/cancellation and root-slot pressure, learner -catch-up, joint membership and lost-disk replacement. Snapshots grant no new voter -or serving authority. All five declared CG-05 acceptance cases remain required. +The latest transport checkpoint at `ceb7ed31` passes 227 selected Rust cases, +8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-snapshot-transport/` binds 306 source hashes +and 5 copied executables. Five TLS cases qualify both physical image forms, +real Raft-generated catch-up and retry after refusal, suffix/cold recovery, +dropped delivery ownership, negative authenticated framing, revocation/replacement +and unrelated-group progress during an admitted receive. Controller registrations +retain exact appliers/routes. Reserved coalesced snapshot feedback survives +ordinary request pressure; transport ownership lasts through feedback admission +or shutdown. The wire acknowledgement remains admission only. See +[progress](progress.md) for exact limits and retained intermediate diagnostics. + +The most critical gap is automatic controller-owned checkpoint selection and +publication. Coalesce work and coordinate with advancing apply; a continuously hot +group must not invalidate every captured cut. Bound retries and keep healthy groups +progressing. Then qualify receiver crash/sync-error/cancellation, stalled physical +work, hint saturation and retained-root slot pressure, whole-group learners/joint +membership and lost-disk replacement. Ordinary peer frames still refuse snapshots; +only the dedicated verified stream enters `step_checkpoint`. Snapshots grant no new +voter or serving authority. All five declared CG-05 acceptance cases remain required. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands and 37 validator tests. Transfer @@ -87,8 +101,8 @@ cases cover concurrent real quorum writes while streaming a held cut, both image representations, empty enrollment, obsolete-page retention/release, bounds, cancelled physical ownership, stale/foreign/unpublished roots, corruption and closed engines. Those are the preceding embedded source tests; the receiver slice above extends -storage coverage. Snapshot RPC, membership recovery and new process/RustFS/load -qualification remain open. +storage coverage. At that source checkpoint, snapshot RPC, membership recovery +and new process/RustFS/load qualification were still open. The following records describe the preceding incremental checkpoints. @@ -637,14 +651,13 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Complete CG-05 bounded authenticated snapshot transport. Connect exact retained - checkpoint capture, source leasing, framing and destination verification to the - controller/peer transport. Carry route/member authentication, byte/item budgets, - cancellation, retries and joined physical ownership through the full transfer. - Add the source publication policy and report actual snapshot failures to Raft. - Admit the destination through `step_checkpoint`; use its post-persistence Raft - acknowledgement, never the admission return value, as completion. Exercise real - physical TLS catch-up while unrelated groups continue making progress. +1. Complete CG-05 automatic checkpoint policy and remaining transport pressure cuts. + Configure the node-wide checkpoint pool explicitly. Coalesce source capture and + publication, coordinate with apply so hot groups cannot starve checkpointing, + and bound retries/backoff. Retain the local suffix and exact immutable cut. + Qualify stalled physical I/O, cancellation/close, reserved hint saturation and + root-slot pressure while unrelated groups progress. Keep the stream admission + acknowledgement separate from Raft's post-persistence acknowledgement. 2. Finish CG-05 membership and recovery. Qualify the outstanding receiver fault cuts and root-slot pressure, then whole-group learner catch-up, joint membership and lost-disk replacement. Run every exact acceptance case before DONE. @@ -662,14 +675,14 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Start transport integration in `transport_frame.rs`, `transport_runtime.rs`, -`snapshot_transport.rs` and the shared controller. Existing snapshot pulls use -cell-scoped identities; preserve physical group-history, permanent member and -incarnation checks for shared groups. Use `checkpoint/mod.rs::capture_snapshot` -for the exact retained cut. Local publication already has its own owner persistence -phase and `SnapshotChange::CellGroupCheckpoint`; never reuse receiver installation -to publish a local leader checkpoint. Capture/publication refusal is retryable only -under an explicit bounded policy; a missing receiver response remains ambiguous. +Start automatic policy in the shared controller and `checkpoint/mod.rs`, using +`publish_checkpoint` and its exact persistence completion. Current publication +requires the captured cut to remain the current applied root; explicitly resolve +that coordination before enabling thresholds for hot groups. The authenticated +path in `group_snapshot_transport/` already owns bounded source/receiver streaming, +route checks, cancellation and snapshot feedback. Extend its pressure tests rather +than adding a second transport. Never reuse receiver installation to publish a local +leader checkpoint. Missing network responses remain ambiguous. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 25617f9df..54fe21783 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,67 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 authenticated checkpoint transport + +Qualified source: `ceb7ed3149202660855d32df1ec5feed7775e068`. All 227 selected Rust cases pass across shared +execution/catalog/storage and the existing single-cell storage/snapshot suites. +All 8 gate commands pass, including all-target Clippy for Cellar/core/LTX, +formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. The partial record at +`.cellar/cell-groups-evidence/CG-05-snapshot-transport/checkpoint.json` binds +306 source hashes, 5 copied executables, frozen plan inputs and command logs. +Transfer this ignored directory with its predecessors. CG-05 remains IN_PROGRESS. + +Dedicated TLS data streams now carry complete checkpoints between physical group +peers. Start and chunk purposes bind the exact history/incarnation, node/disk/process, +member, schema, snapshot envelope, ordered part identity and byte digest to the +TLS exporter and start tag. Bounded headers and <=64 KiB chunks precede allocation; +an authenticated End is mandatory. Extent identities resolve from the receiver's +verified manifest. The exact retained source cut streams on a blocking owner; +receiver verification and sealing run on another. Wire, physical workspace and +scratch reservations stay with their owners through cancellation and shutdown. + +Controller registrations retain the applier/route endpoint. Removing/replacing a +route or dropping its registration revokes stream admission. The independently +verified closure enters `step_checkpoint`; the wire reply is admission only. +The existing exact Ready/persistence completion still controls Raft's acknowledgement. +A node explicitly enables the shared checkpoint pool; automatic checkpoint +selection/publication is not yet wired into the controller. + +Snapshot status feedback now has a separate 256-slot/1-MiB node budget and reserved +queue space. It coalesces by known peer while retaining admission, then mutates +Raft progress after the current Ready settles. Transport ownership lasts through +feedback admission or shutdown, so ordinary queue contention or an I/O reply timeout +cannot silently discard required snapshot failure feedback. Immediate controller +snapshot refusals also retain feedback; ordinary unreachable hints remain advisory. + +Five new real-TLS cases cover actual RawNode-generated catch-up in both image +representations, suffix replay/cold source reopen, a dropped delivery waiter, and a +refused first transfer followed by Raft's own successful retry. The harness does +not replay that failed message. Independent wire probes refuse invalid exporter, +process, body digest, transfer binding, sequence, chunk size and premature End; +revocation during an admitted receive leaves the old root selected. Replacing the +old registration guard preserves its replacement. Disconnect releases receive +admission; a different group commits and applies a real cell write while that +admission is occupied. Materialized source cold recovery still uses the fixture's +generation-ID resolver; extent recovery and destination replay use production adapters. + +The initial visibility and fixture trait-import compile diagnostics are retained. +The first all-target Clippy run was deliberately interrupted when review identified +the snapshot-feedback admission gap; only the final committed-source gate counts. +No assertion was weakened. Existing native held-sync/owner tests remain regression +coverage, rather than an independent TLS receiver crash or power-loss qualification. + +The most critical implementation gap is automatic controller-owned checkpoint +selection/publication: coalesce work, coordinate with advancing apply, and bound +capture/publication retries so hot groups can checkpoint without starving unrelated +ones. Then qualify source/receiver pressure, stalled physical I/O and hint saturation, +receiver process-kill/sync-error cuts, and retained-root slot exhaustion. Complete +whole-group learner/joint configuration and lost-disk replacement before the full +CG-05 gate. All five declared acceptance cases remain required. CG-06 archive/reclaim +and later public cutover/legacy removal remain open. No new shared-group RustFS, +load, performance-parity or zero-quorum DR qualification is claimed. + ## CG-05 local publication and native Raft snapshot catch-up Qualified source: `742b1e6d4e7155e16fb1b4eba0cce3b9056944ae`. All 222 selected Rust cases pass across shared @@ -46,7 +107,7 @@ materialized cold-recovery failure from using the wrong resolver is retained wit the compile diagnostics. No assertion was weakened. Joined runtime shutdown during accepted local publication is covered in both representations. -The critical gap is bounded authenticated snapshot RPC and its controller-owned +The gap recorded at this local-publication checkpoint was authenticated RPC and its controller-owned source/publication lifecycle. These cases use native streaming callbacks; they do not qualify network framing/authentication, transfer cancellation/retries, failed snapshot reporting, automated checkpoint policy, learners/joint membership or disk diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index b04c1d52a..6c7381560 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -255,9 +255,10 @@ Each chunk has a separate authenticated purpose, binds the start tag, and carrie a monotonic sequence, part identity, offset, length and body digest. Header and payload limits are checked before allocation; chunks are at most 64 KiB. Extent identities resolve against the receiver's already verified manifest. An explicit -authenticated End is required; EOF, cancellation, mixed cuts and incomplete -closures cannot finish a receive. Source capture pins the exact engine-retained -snapshot even when the current applied root has advanced. +authenticated End is required. EOF or cancellation before End cannot finish a +receive; mixed cuts and incomplete closures refuse. After verified admission, +connection loss does not establish whether installation completed. Source capture +pins the exact engine-retained snapshot even when the current applied root advances. Two bounded channel windows connect the network to blocking physical owners. Node/peer data budgets and source/receive slots remain owned through actual I/O, From c27dfb416784804b30a38b1a90812c607ed3b00f Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 17:37:42 -0700 Subject: [PATCH 129/568] feat: checkpoint active cell groups at controller apply boundaries --- .../cell_groups/controller/checkpoints.rs | 251 ++++++++++++++++++ .../raft/cell_groups/controller/failure.rs | 3 + .../raft/cell_groups/controller/mod.rs | 40 ++- .../raft/cell_groups/controller/worker.rs | 92 ++++++- .../cellar/durability/raft/cell_groups/mod.rs | 7 +- .../durability/raft/cell_groups/owner.rs | 29 ++ .../raft/cell_groups/runtime/mod.rs | 26 +- .../raft/cell_groups/runtime/worker.rs | 16 +- .../raft/group_snapshot_transport/mod.rs | 10 + .../tests/native_cell_groups_execution.rs | 15 ++ .../tests/support/cell_group_controller.rs | 13 +- .../cell_group_controller_checkpoints.rs | 89 +++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../multi-raft/cell-groups/protocols.md | 31 ++- docs/design/multi-raft/cell-groups/tasks.json | 4 +- 16 files changed, 601 insertions(+), 31 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs create mode 100644 crates/cellar/tests/support/cell_group_controller_checkpoints.rs diff --git a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs new file mode 100644 index 000000000..67e59b3b5 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs @@ -0,0 +1,251 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Checkpoint the preceding applied cut while retaining its successor ticket. +//! One bounded slot spans capture, reserved admission and durable publication. +use super::*; +use crate::raft::cell_groups::{ + CellGroupApply, CellGroupCheckpointError, CellGroupCheckpointPool, + PreparedLocalCellGroupCheckpoint, +}; +use cellar_logic::cell_group::GroupHistory; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupCheckpointPolicy { + /// Entries since the persisted checkpoint (birth index before the first). + /// Selection occurs before a successor apply; an idle final tail is retained. + pub interval_entries: u64, + /// Node-wide captures, admissions and pending syncs combined. + pub max_parallel: usize, +} + +pub(super) struct Config { + policy: CellGroupCheckpointPolicy, + pool: CellGroupCheckpointPool, +} + +impl Config { + pub fn new( + limits: CellGroupControllerLimits, + transport: &RaftTransport, + ) -> Result, CellGroupControllerError> { + let Some(policy) = limits.checkpoints else { + return Ok(None); + }; + if policy.interval_entries == 0 + || policy.max_parallel == 0 + || policy.max_parallel > limits.max_groups.min(256) + { + return Err(CellGroupControllerError::Invalid("checkpoint policy")); + } + Ok(Some(Self { + policy, + pool: transport + .group_checkpoint_pool() + .map_err(CellGroupControllerError::Transport)?, + })) + } +} + +struct Entry { + work: Box, + admitted: bool, + published: bool, +} + +type Captured = ( + GroupHistory, + Result, +); +type Admitted = (GroupHistory, Result<(), RuntimeError>); + +pub(super) enum Completed { + Captured(Captured), + Admitted(Admitted), +} + +#[derive(Default)] +pub(super) struct Manager { + entries: BTreeMap, + retry_at: BTreeMap, + captures: FuturesUnordered>, + publications: FuturesUnordered>, +} + +impl Manager { + pub fn contains(&self, group: &GroupHistory) -> bool { + self.entries.contains_key(group) + } + + pub fn running(&self) -> bool { + !self.captures.is_empty() || !self.publications.is_empty() + } + + pub async fn next(&mut self) -> Completed { + crate::asyncrt::select! { + Some(done) = self.captures.next(), if !self.captures.is_empty() => Completed::Captured(done), + Some(done) = self.publications.next(), if !self.publications.is_empty() => Completed::Admitted(done), + } + } + + pub fn start( + &mut self, + work: &CellGroupApply, + now: u64, + shared: &Shared, + ) -> Result { + let Some(config) = &shared.checkpoints else { + return Ok(false); + }; + let history = work.base.marker().route().history; + if self.entries.len() >= config.policy.max_parallel + || self.retry_at.get(&history).is_some_and(|at| *at > now) + { + return Ok(false); + } + let group = shared + .runtime + .log() + .cell_group(&history) + .map_err(RuntimeError::from)?; + let previous = group + .checkpoint_manifest() + .map_err(RuntimeError::from)? + .map_or(1, |root| root.position().index.get()); + let index = work.base.marker().position().index.get(); + if index.saturating_sub(previous) < config.policy.interval_entries { + return Ok(false); + } + let pool = config.pool.clone(); + let root = work.base.clone(); + let task = shared + .domain + .blocking(move || pool.capture(&group, &root)?.prepare_publication()) + .map_err(|_| CellGroupControllerError::Closed)?; + self.entries.insert( + history, + Entry { + work: Box::new(work.clone()), + admitted: false, + published: false, + }, + ); + self.captures.push(Box::pin(async move { + ( + history, + task.await + .map_err(|_| CellGroupControllerError::Closed) + .and_then(|result| { + result + .map_err(|error| CellGroupControllerError::Checkpoint(Arc::new(error))) + }), + ) + })); + shared + .active_checkpoints + .store(self.entries.len(), Ordering::Release); + Ok(true) + } + + pub fn publish( + &mut self, + group: GroupHistory, + checkpoint: PreparedLocalCellGroupCheckpoint, + runtime: &CellGroupRuntime, + ) -> Result<(), CellGroupControllerError> { + let entry = self + .entries + .get(&group) + .ok_or(CellGroupControllerError::Invalid( + "checkpoint capture ticket", + ))?; + let ticket = entry.work.ticket.clone(); + let runtime = runtime.clone(); + self.publications.push(Box::pin(async move { + ( + group, + runtime + .checkpoint_before_apply(group, ticket, checkpoint) + .await, + ) + })); + Ok(()) + } + + pub fn observed( + &mut self, + group: GroupHistory, + index: u64, + shared: &Shared, + ) -> Result>, CellGroupControllerError> { + let Some(entry) = self.entries.get_mut(&group) else { + return Ok(None); + }; + if entry.work.base.marker().position().index.get() != index { + return Err(CellGroupControllerError::Invalid( + "checkpoint publication cut", + )); + } + entry.published = true; + Ok(if entry.admitted { + self.remove(group, shared) + } else { + None + }) + } + + pub fn accept( + &mut self, + group: GroupHistory, + shared: &Shared, + ) -> Result>, CellGroupControllerError> { + let entry = self + .entries + .get_mut(&group) + .ok_or(CellGroupControllerError::Invalid( + "checkpoint admission ticket", + ))?; + entry.admitted = true; + Ok(if entry.published { + self.remove(group, shared) + } else { + None + }) + } + + pub fn retry( + &mut self, + group: GroupHistory, + now: u64, + shared: &Shared, + ) -> Option> { + self.retry_at + .insert(group, now.saturating_add(shared.limits.retry_interval_ms)); + shared.checkpoint_retries.fetch_add(1, Ordering::Relaxed); + self.remove(group, shared) + } + + pub fn remove(&mut self, group: GroupHistory, shared: &Shared) -> Option> { + let work = self.entries.remove(&group).map(|entry| entry.work); + shared + .active_checkpoints + .store(self.entries.len(), Ordering::Release); + work + } + + /// A cancelled controller waiter cannot detach a physical capture. Join it + /// before closing the runtime's engine claim. Admitted writes belong to the + /// runtime and are joined by runtime.close(). Retain tickets until both joins. + pub async fn join(&mut self) { + while self.captures.next().await.is_some() {} + self.publications.clear(); + } +} + +pub(super) fn retryable(error: &CellGroupControllerError) -> bool { + matches!(error, CellGroupControllerError::Checkpoint(error) if matches!(error.as_ref(), CellGroupCheckpointError::Busy)) + || matches!( + error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + ) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/failure.rs b/crates/cellar/durability/raft/cell_groups/controller/failure.rs index 195645946..1212f9f3d 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/failure.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/failure.rs @@ -44,6 +44,9 @@ pub(super) fn node_wide(error: &CellGroupControllerError) -> bool { } CellGroupControllerError::Runtime(error) => runtime(error), CellGroupControllerError::Apply(error) => execution(error), + CellGroupControllerError::Checkpoint(error) => { + matches!(error.as_ref(), super::super::CellGroupCheckpointError::Log(error) if storage(error)) + } CellGroupControllerError::Command(error) => dispatch(&error.cause), CellGroupControllerError::Enrollment(error) => dispatch(&error.cause), CellGroupControllerError::Closed | CellGroupControllerError::Transport(_) => true, diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 507edcbf6..7d7e9e515 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -4,6 +4,8 @@ //! Admitted command ownership survives response cancellation and expiry. mod calls; +mod checkpoints; +pub use checkpoints::CellGroupCheckpointPolicy; mod commands; mod enrollments; mod failure; @@ -45,6 +47,9 @@ pub struct CellGroupControllerLimits { /// Invocation/input/options retention. SQL capture, apply and result bytes /// retain the SQL pool's independent admission as well. pub max_command_bytes: usize, + /// Automatic compaction at an apply boundary. Requires the transport's + /// shared checkpoint pool. None leaves local publication to its caller. + pub checkpoints: Option, } #[derive(Clone, Debug)] @@ -52,6 +57,7 @@ pub enum CellGroupControllerError { Invalid(&'static str), Runtime(RuntimeError), Apply(Arc), + Checkpoint(Arc), Transport(TransportError), Catalog(Arc), Command(Arc), @@ -82,6 +88,9 @@ pub struct CellGroupControllerUsage { pub active_apply: usize, pub pending_apply: usize, pub applied: u64, + pub active_checkpoints: usize, + pub checkpoints_published: u64, + pub checkpoint_retries: u64, pub apply_retries: u64, /// Immediate queue/route refusals. The transport tracks later I/O failures. pub send_refusals: u64, @@ -105,6 +114,10 @@ struct Shared { active_apply: AtomicUsize, pending_apply: AtomicUsize, applied: AtomicU64, + checkpoints: Option, + active_checkpoints: AtomicUsize, + checkpoints_published: AtomicU64, + checkpoint_retries: AtomicU64, apply_retries: AtomicU64, send_refusals: AtomicU64, closing: AtomicBool, @@ -198,6 +211,7 @@ impl CellGroupController { { return Err(CellGroupControllerError::Invalid("controller owner")); } + let checkpoints = checkpoints::Config::new(limits, &transport)?; let mut roster = BTreeMap::new(); for applier in appliers { if !applier.belongs_to_engine(&runtime.log().owner()) @@ -224,6 +238,10 @@ impl CellGroupController { active_apply: AtomicUsize::new(0), pending_apply: AtomicUsize::new(0), applied: AtomicU64::new(0), + checkpoints, + active_checkpoints: AtomicUsize::new(0), + checkpoints_published: AtomicU64::new(0), + checkpoint_retries: AtomicU64::new(0), apply_retries: AtomicU64::new(0), send_refusals: AtomicU64::new(0), closing: AtomicBool::new(false), @@ -237,17 +255,17 @@ impl CellGroupController { let worker_domain = domain.clone(); // Infrastructure cleanup is not cancelled with application tasks. The // worker retains tickets/appliers until all physical owners have joined. + let mut worker = worker::Worker::new( + runtime, + events, + transport, + sql, + roster, + receiver, + shared.clone(), + )?; domain .spawn_cleanup(async move { - let mut worker = worker::Worker::new( - runtime, - events, - transport, - sql, - roster, - receiver, - shared.clone(), - ); let result = std::panic::AssertUnwindSafe(worker.run(&worker_domain)) .catch_unwind() .await @@ -262,6 +280,7 @@ impl CellGroupController { .lock() .unwrap_or_else(|e| e.into_inner()) .take(); + shared.active_checkpoints.store(0, Ordering::Release); shared.active_apply.store(0, Ordering::Release); shared.pending_apply.store(0, Ordering::Release); shared.complete(result.and(joined)); @@ -305,6 +324,9 @@ impl CellGroupController { active_apply: shared.active_apply.load(Ordering::Acquire), pending_apply: shared.pending_apply.load(Ordering::Acquire), applied: shared.applied.load(Ordering::Relaxed), + active_checkpoints: shared.active_checkpoints.load(Ordering::Acquire), + checkpoints_published: shared.checkpoints_published.load(Ordering::Relaxed), + checkpoint_retries: shared.checkpoint_retries.load(Ordering::Relaxed), apply_retries: shared.apply_retries.load(Ordering::Relaxed), send_refusals: shared.send_refusals.load(Ordering::Relaxed), closing: shared.closing.load(Ordering::Acquire), diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index a98f7dfaa..15b36b1e0 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -45,6 +45,8 @@ pub(super) struct Worker { registrations: FuturesUnordered>, routes: BTreeMap, progress: tokio::sync::watch::Sender<()>, + checkpoints: checkpoints::Manager, + _checkpoint_routes: Vec, } impl Worker { @@ -56,8 +58,18 @@ impl Worker { roster: BTreeMap, requests: tokio::sync::mpsc::Receiver, shared: Arc, - ) -> Self { - Self { + ) -> Result { + let mut checkpoint_routes = Vec::new(); + if shared.checkpoints.is_some() { + for applier in roster.values() { + checkpoint_routes.push( + transport + .register_group_checkpoint(applier.clone()) + .map_err(CellGroupControllerError::Transport)?, + ); + } + } + Ok(Self { runtime, events, transport, @@ -78,10 +90,12 @@ impl Worker { provisioning: provision::Manager::default(), routes: BTreeMap::new(), progress: tokio::sync::watch::channel(()).0, - } + checkpoints: checkpoints::Manager::default(), + _checkpoint_routes: checkpoint_routes, + }) } - fn launch(&mut self, now: u64) { + fn launch(&mut self, now: u64) -> Result<(), CellGroupControllerError> { // Rotate the bounded roster queue. A busy group cannot monopolize the // next available apply slot, and successors never run concurrently. for _ in 0..self.pending.len() { @@ -95,10 +109,20 @@ impl Worker { self.retained_apply.push(job.work); continue; } - if job.retry_at > now || self.active.contains(&job.group) { + if job.retry_at > now + || self.active.contains(&job.group) + || self.checkpoints.contains(&job.group) + { self.pending.push_back(job); continue; } + let checkpoint_started = match &job.work { + Work::Prepare(work) => self.checkpoints.start(work, now, &self.shared)?, + Work::Publish(_) => false, + }; + if checkpoint_started { + continue; + } self.active.insert(job.group); let sql = self.sql.clone(); let applier = self.roster[&job.group].clone(); @@ -115,6 +139,7 @@ impl Worker { self.shared .pending_apply .store(self.pending.len(), Ordering::Release); + Ok(()) } async fn quarantine( @@ -151,7 +176,7 @@ impl Worker { let now = domain .mono_ms() .map_err(|_| CellGroupControllerError::Closed)?; - self.launch(now); + self.launch(now)?; let timer = domain .sleep_until(wake_at) .map_err(|_| CellGroupControllerError::Closed)?; @@ -217,6 +242,23 @@ impl Worker { Ok(None) => (), } } + completed = self.checkpoints.next(), if self.checkpoints.running() => { + match completed { + checkpoints::Completed::Captured((group, result)) => match result { + Ok(checkpoint) if !self.quarantined.contains(&group) => self.checkpoints.publish(group, checkpoint, &self.runtime)?, + Ok(_) => { + if let Some(work) = self.checkpoints.remove(group, &self.shared) { self.retained_apply.push(Work::Prepare(work)); } + } + Err(error) => self.checkpoint_failed(group, error, now).await?, + }, + checkpoints::Completed::Admitted((group, result)) => match result { + Ok(()) => { + if let Some(work) = self.checkpoints.accept(group, &self.shared)? { self.resume_apply(group, work, now); } + } + Err(error) => self.checkpoint_failed(group, error.into(), now).await?, + }, + } + } Some(()) = self.hints.next(), if !self.hints.is_empty() => (), event = self.events.recv() => { let event = event.ok_or(CellGroupControllerError::Closed)?; @@ -278,7 +320,11 @@ impl Worker { self.shared.applied.fetch_add(1, Ordering::Relaxed); self.progress.send_replace(()); } - CellGroupEffect::CheckpointPublished { .. } => { + CellGroupEffect::CheckpointPublished { manifest } => { + self.shared.checkpoints_published.fetch_add(1, Ordering::Relaxed); + if let Some(work) = self.checkpoints.observed(group, manifest.position().index.get(), &self.shared)? { + self.resume_apply(group, work, now); + } // Local compaction does not advance applied state. self.progress.send_replace(()); } @@ -290,6 +336,37 @@ impl Worker { } } + fn resume_apply(&mut self, group: GroupHistory, work: Box, now: u64) { + if self.quarantined.contains(&group) { + self.retained_apply.push(Work::Prepare(work)); + } else { + self.pending.push_back(Pending { + group, + work: Work::Prepare(work), + retry_at: now, + }); + } + } + + async fn checkpoint_failed( + &mut self, + group: GroupHistory, + error: CellGroupControllerError, + now: u64, + ) -> Result<(), CellGroupControllerError> { + if checkpoints::retryable(&error) { + if let Some(work) = self.checkpoints.retry(group, now, &self.shared) { + self.resume_apply(group, work, now); + } + } else { + if let Some(work) = self.checkpoints.remove(group, &self.shared) { + self.retained_apply.push(Work::Prepare(work)); + } + self.quarantine(group, error).await?; + } + Ok(()) + } + pub(super) async fn finish(&mut self) -> Result<(), CellGroupControllerError> { self.requests.close(); while self.requests.try_recv().is_ok() {} @@ -299,6 +376,7 @@ impl Worker { let _ = completed.finish(true); } self.provisioning.join().await; + self.checkpoints.join().await; self.transport.close().await; let sql = self .sql diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index dab91f3ae..c224c3896 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -25,9 +25,10 @@ pub use checkpoint::{ mod controller; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, - CellGroupController, CellGroupControllerError, CellGroupControllerLimits, - CellGroupControllerUsage, CellGroupProvisionCall, CellGroupProvisioning, CellGroupRegistration, - CellGroupRegistrationCall, CellGroupRegistrationError, + CellGroupCheckpointPolicy, CellGroupController, CellGroupControllerError, + CellGroupControllerLimits, CellGroupControllerUsage, CellGroupProvisionCall, + CellGroupProvisioning, CellGroupRegistration, CellGroupRegistrationCall, + CellGroupRegistrationError, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index a3fd1bf59..ded2ba324 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -495,6 +495,35 @@ impl CellGroupOwner { if !self.idle() || self.applying.is_some() { return Err(RuntimeError::NotReady); } + self.select_local_checkpoint(checkpoint) + } + + /// The controller holds this exact successor ticket without starting apply. + /// Publishing its base cannot race another applied root or snapshot install. + /// Pending Ready work may persist a later suffix; it cannot change this cut. + pub(super) fn checkpoint_before_apply( + &mut self, + ticket: &CellGroupTicket, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + self.live()?; + if self.phase.is_some() || self.checkpoint.is_some() || self.waiting.is_some() { + return Err(RuntimeError::NotReady); + } + if self + .applying + .as_ref() + .is_none_or(|active| !active.matches(ticket)) + { + return Err(RuntimeError::StaleCompletion); + } + self.select_local_checkpoint(checkpoint) + } + + fn select_local_checkpoint( + &mut self, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { let root = checkpoint.root(); if root.marker() != self.current.marker() || !Arc::ptr_eq(&root.engine_owner, &self.group.owner) diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index a62a850f7..9c56792a4 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -102,6 +102,7 @@ enum Command { Step(Message), Checkpoint(Message, PreparedCellGroupCheckpoint), PublishCheckpoint(PreparedLocalCellGroupCheckpoint), + CheckpointBeforeApply(CellGroupTicket, PreparedLocalCellGroupCheckpoint), AdmitStep(Message), Propose(Vec), ProposeAtTerm(cellar_logic::consensus::Term, Vec), @@ -282,7 +283,10 @@ impl CellGroupRuntime { bytes: usize, ) -> Result { self.admit()?; - let completion = matches!(command, Command::Apply(..) | Command::Quarantine); + let completion = matches!( + command, + Command::Apply(..) | Command::CheckpointBeforeApply(..) | Command::Quarantine + ); let snapshot_hint = matches!(command, Command::SnapshotStatus(..)); let budget = if snapshot_hint { &self.inner.snapshot_hints @@ -555,6 +559,26 @@ impl CellGroupRuntime { .map(|_| ()) } + pub(crate) async fn checkpoint_before_apply( + &self, + history: GroupHistory, + ticket: CellGroupTicket, + checkpoint: PreparedLocalCellGroupCheckpoint, + ) -> Result<(), RuntimeError> { + let bytes = checkpoint + .root() + .reserved_bytes() + .saturating_add(checkpoint.snapshot().compute_size() as usize) + .saturating_add(4096); + self.call( + Some(history), + Command::CheckpointBeforeApply(ticket, checkpoint), + bytes, + ) + .await + .map(|_| ()) + } + /// Queue a verified snapshot on this group owner. Admission is not a /// persistence receipt; only its post-fence Raft response acknowledges it. pub async fn step_checkpoint( diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 8b9b35909..f190171e5 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -279,7 +279,10 @@ impl Worker { let _ = answer.send(Ok(Response::Unit)); return; } - if matches!(request.command, Command::Apply(..)) { + if matches!( + request.command, + Command::Apply(..) | Command::CheckpointBeforeApply(..) + ) { if group.completion.is_some() { reply(request, Err(RuntimeError::Overloaded)); return; @@ -321,10 +324,15 @@ impl Worker { group.queued = false; if group.failed.is_none() { if let Some(request) = group.completion.take() { - let Command::Apply(ticket, next) = &request.command else { - return Err(RuntimeError::Protocol("completion queue".into())); + let result = match &request.command { + Command::Apply(ticket, next) => { + group.owner.complete_apply(ticket.clone(), (**next).clone()) + } + Command::CheckpointBeforeApply(ticket, checkpoint) => group + .owner + .checkpoint_before_apply(ticket, checkpoint.clone()), + _ => return Err(RuntimeError::Protocol("completion queue".into())), }; - let result = group.owner.complete_apply(ticket.clone(), (**next).clone()); if matches!(result, Err(RuntimeError::NotReady)) { group.completion = Some(request); } else { diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs index a62f24f07..441b5754e 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -64,6 +64,16 @@ impl RaftTransport { Ok(()) } + pub(crate) fn group_checkpoint_pool(&self) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry + .group_checkpoint_pool + .clone() + .ok_or(TransportError::Refused) + } + pub fn register_group_checkpoint( &self, applier: CellGroupApplier, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 6d598fe08..b7e0dff81 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,21 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_compacts_materialized_groups_during_writes() -> Result<()> { + runtime_fixture::automatic_checkpoints(false, false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_compacts_extent_groups_during_writes() -> Result<()> { + runtime_fixture::automatic_checkpoints(true, false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_busy_pool_preserves_apply_and_other_group_progress() -> Result<()> { + runtime_fixture::automatic_checkpoints(false, true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_tcp_refuses_invalid_streams_and_preserves_other_group_progress() -> Result<()> { runtime_fixture::tcp_protocol().await diff --git a/crates/cellar/tests/support/cell_group_controller.rs b/crates/cellar/tests/support/cell_group_controller.rs index bc1b6c58f..4d5721ff7 100644 --- a/crates/cellar/tests/support/cell_group_controller.rs +++ b/crates/cellar/tests/support/cell_group_controller.rs @@ -15,6 +15,7 @@ pub(super) fn limits() -> CellGroupControllerLimits { retry_interval_ms: 5, max_commands: 2, max_command_bytes: 64 * 1024, + checkpoints: None, } } @@ -26,7 +27,9 @@ pub(super) async fn start_with_limits( fixture: &mut RuntimeCluster, bounds: CellGroupControllerLimits, ) -> Result> { - fixture.connect_tcp().await?; + if fixture.transports.is_empty() { + fixture.connect_tcp().await?; + } let mut controllers = Vec::new(); for (node, events) in fixture.events.drain(..).enumerate() { let selected: Vec<_> = fixture @@ -53,7 +56,7 @@ pub(super) async fn start_with_limits( Ok(controllers) } -async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { +pub(super) async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<()> { tokio::time::timeout(Duration::from_secs(15), async { while !predicate().await? { tokio::time::sleep(Duration::from_millis(2)).await; @@ -63,7 +66,11 @@ async fn wait_for(mut predicate: impl AsyncFnMut() -> Result) -> Result<() .await? } -async fn applied(fixture: &RuntimeCluster, group: GroupHistory, index: LogIndex) -> Result<()> { +pub(super) async fn applied( + fixture: &RuntimeCluster, + group: GroupHistory, + index: LogIndex, +) -> Result<()> { wait_for(async || { for runtime in &fixture.runtimes { if runtime.status(group).await?.applied.index < index { diff --git a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs new file mode 100644 index 000000000..2d066084d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs @@ -0,0 +1,89 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Production controller owns every event after startup; no fixture pumping. +use super::*; +use raft::Storage; + +pub(crate) async fn automatic_checkpoints(rooted: bool, pressure: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut held = if pressure { + Some(pools[0].receive( + fixture.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?) + } else { + None + }; + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + .get(); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(23);")?; + for round in 0..12 { + // Interleaved writes to two groups share one checkpoint slot per node. + // All voters must apply every successor, including under pool pressure. + for (group, format) in [(route.history, rooted), (second, false)] { + let log = fixture.cluster.voters[0].log.cell_group(&group)?; + let marker = log.group_manifest()?.context("selected")?; + let cell = history(marker.route(), &format!("auto-{round}"))?; + let accepted = fixture.runtimes[0] + .propose( + group, + apply::enrollment_bytes(marker.route(), &cell, &bytes, format)?, + ) + .await?; + controller::applied(&fixture, group, accepted.position.index).await?; + } + for controller in &controllers { + assert!(!controller.usage().closing); + assert!(controller.usage().active_checkpoints <= 1); + assert_eq!(controller.usage().quarantined_groups, 0); + } + if pressure && round == 4 { + assert!(controllers[0].usage().checkpoint_retries > 0); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + assert_eq!(pools[0].usage().sources, 1); + drop(held.take()); + tokio::time::sleep(Duration::from_millis(bounds.retry_interval_ms + 1)).await; + } + } + for (node, controller) in controllers.iter().enumerate() { + controller::wait_for(async || Ok(controller.usage().active_checkpoints == 0)).await?; + assert!(controller.usage().checkpoints_published >= 4); + for group in [route.history, second] { + let log = fixture.cluster.voters[node].log.cell_group(&group)?; + let current = log.group_manifest()?.context("applied root")?; + let checkpoint = log.checkpoint_manifest()?.context("automatic checkpoint")?; + assert!(checkpoint.position().index.get() > initial); + assert!(checkpoint.position().index < current.position().index); + assert!(current.position().index.get() - checkpoint.position().index.get() <= 2); + assert_eq!(log.first_index()?, checkpoint.position().index.get() + 1); + assert_eq!(log.last_index()?, current.position().index.get()); + } + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + } + drop(controllers); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 469b1c991..51591d113 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -807,3 +807,5 @@ pub(super) use sql::{ provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, provisioning_retries_retained_owner, }; + +pub(super) use sql::automatic_checkpoints; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f4e3630d5..8ff57f082 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -591,3 +591,7 @@ pub(crate) use provision::{ provisioning_recovers_selected_roots, provisioning_refusals_preserve_healthy_group, provisioning_retries_retained_owner, }; + +#[path = "cell_group_controller_checkpoints.rs"] +mod controller_checkpoints; +pub(crate) use controller_checkpoints::automatic_checkpoints; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 6c7381560..919276232 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -275,11 +275,36 @@ known peer while retaining its reservation, then applies it after the in-flight Ready has settled. A completed transfer remains owned until feedback admission or transport shutdown; an I/O response timeout cannot silently discard required failure feedback. Immediate snapshot queue refusals also report failure, allowing -Raft to leave its paused snapshot state and generate a retry. Automatic local -checkpoint selection/publication and whole-group membership orchestration remain -separate integration work. No transfer reply grants new voter, serving, archival +Raft to leave its paused snapshot state and generate a retry. Whole-group +membership orchestration remains separate integration work. No transfer reply grants new voter, serving, archival or reclamation authority. +The controller's optional `CellGroupCheckpointPolicy` selects a checkpoint before +starting a successor apply when the configured entry interval has elapsed. It +uses the transport's already configured pool and owns initial checkpoint endpoint +registrations. One node slot spans blocking capture, admission and durable +publication; each selected group retains its exact successor ticket. The current +root therefore cannot advance during capture. A private completion-lane request +checks that ticket and the current root, and can publish despite ordinary request +or apply pressure. Pending Raft work may still append a suffix. Only the matching +`CheckpointPublished` event releases the successor to physical apply; an admission +reply alone does not. Reply/event ordering is handled in either order. + +Pool contention or transient admission overload releases the successor and backs +off until a later apply boundary using the controller's node timer. There is no +per-cell task or unbounded retry loop. Capture jobs are joined before releasing the +runtime's engine claim; accepted writes remain owned by the runtime through its +join. Hard checkpoint failures retain the ticket and quarantine the group, while +shared log uncertainty closes the node. Quarantined groups can retain checkpoint +capacity until joined restart, like their registration capacity. + +This is an entry-count policy for active groups. It retains the final idle tail, +and time/byte triggers, fairness under continuously saturated checkpoint capacity, +controller checkpoint crash/held-sync cuts and complete root-pressure qualification +remain open. It supplies no S3 archive coverage proof. Manual publication remains +available to lower-level owners; this policy does not introduce a second serving +runtime or weaken exact-cut publication checks. + An archive publishes a complete checkpoint plus a contiguous committed tail to K. The tail addresses cells and validates each base chain; administrative entries also replay. Do not rerun commands. Publish coverage only after all dependencies diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index c4db71ebb..d6f1b25e2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1584,7 +1584,9 @@ "crates/cellar/durability/raft/cell_groups/checkpoint/local.rs", "crates/cellar/tests/support/cell_group_checkpoint_local.rs", "crates/cellar/application/consensus/mod.rs", - "crates/cellar/tests/support/cell_group_network_probes.rs" + "crates/cellar/tests/support/cell_group_network_probes.rs", + "crates/cellar/tests/support/cell_group_controller.rs", + "crates/cellar/tests/support/cell_group_controller_checkpoints.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 70d52448c2f0526db38b274ccc4a224975b86935 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 17:58:24 -0700 Subject: [PATCH 130/568] docs: record CG-05 checkpoint policy evidence and remaining gaps --- docs/design/multi-raft/cell-groups/handoff.md | 61 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 49 ++++++++++++++- 2 files changed, 83 insertions(+), 27 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index fcdfc3cb4..58d10593d 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -31,8 +31,9 @@ partial gate, including cold recovery of the current and older checkpoint roots. Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate. Local checkpoint publication and native Raft-generated catch-up now pass their partial gate. Authenticated checkpoint streams now pass their partial transport gate. -Continue with automatic controller-owned checkpoint publication and the remaining -fault/pressure boundaries, then whole-group membership and lost-disk recovery. +Automatic entry-count checkpointing at controller apply boundaries now passes its +partial gate. Continue with scheduler/persistence pressure qualification, then +whole-group membership and lost-disk recovery. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track @@ -84,14 +85,25 @@ ordinary request pressure; transport ownership lasts through feedback admission or shutdown. The wire acknowledgement remains admission only. See [progress](progress.md) for exact limits and retained intermediate diagnostics. -The most critical gap is automatic controller-owned checkpoint selection and -publication. Coalesce work and coordinate with advancing apply; a continuously hot -group must not invalidate every captured cut. Bound retries and keep healthy groups -progressing. Then qualify receiver crash/sync-error/cancellation, stalled physical -work, hint saturation and retained-root slot pressure, whole-group learners/joint -membership and lost-disk replacement. Ordinary peer frames still refuse snapshots; -only the dedicated verified stream enters `step_checkpoint`. Snapshots grant no new -voter or serving authority. All five declared CG-05 acceptance cases remain required. +The latest controller checkpoint at `c27dfb41` passes 230 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` binds 308 source +hashes and 5 copied executables. The controller holds a successor apply ticket, +captures its preceding durable root on a blocking owner, and publishes through +reserved completion capacity. Exact ticket/root validation prevents capture from +racing current apply. The matching persistence event releases successor work; +the runtime admission reply alone cannot. One slot spans this lifecycle. The +transport's shared checkpoint pool and initial endpoint registrations are reused. +Busy admission backs off until a later apply boundary, allowing writes to continue. + +The three new cases prove repeated compaction across two groups in both image +forms and progress/recovery while the checkpoint pool is saturated. The policy +retains the final idle tail. Continuous-saturation fairness, exhausted apply and +completion budgets, held sync/cancelled close, checkpoint failure/quarantine capacity +and retained-root pressure are the critical next qualification gaps. Follow with +receiver crash/sync-error cuts, whole-group learners/joint membership and lost-disk +replacement. All five declared CG-05 acceptance cases remain required. See +[progress](progress.md) for limits and intermediate failures. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands and 37 validator tests. Transfer @@ -651,13 +663,13 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Complete CG-05 automatic checkpoint policy and remaining transport pressure cuts. - Configure the node-wide checkpoint pool explicitly. Coalesce source capture and - publication, coordinate with apply so hot groups cannot starve checkpointing, - and bound retries/backoff. Retain the local suffix and exact immutable cut. - Qualify stalled physical I/O, cancellation/close, reserved hint saturation and - root-slot pressure while unrelated groups progress. Keep the stream admission - acknowledgement separate from Raft's post-persistence acknowledgement. +1. Qualify CG-05 checkpoint scheduling and remaining transport pressure cuts. + Automatic entry-count selection now holds the successor ticket until exact + publication. Exercise continuous saturation fairness, exhausted apply/completion + budgets, held sync, cancelled close, quarantine capacity and retained-root + pressure. Keep unrelated-group progress and bounded ownership observable. + Add idle/time/byte triggers only with an explicit cut-reservation contract. + Keep stream admission separate from Raft's post-persistence acknowledgement. 2. Finish CG-05 membership and recovery. Qualify the outstanding receiver fault cuts and root-slot pressure, then whole-group learner catch-up, joint membership and lost-disk replacement. Run every exact acceptance case before DONE. @@ -675,14 +687,13 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Start automatic policy in the shared controller and `checkpoint/mod.rs`, using -`publish_checkpoint` and its exact persistence completion. Current publication -requires the captured cut to remain the current applied root; explicitly resolve -that coordination before enabling thresholds for hot groups. The authenticated -path in `group_snapshot_transport/` already owns bounded source/receiver streaming, -route checks, cancellation and snapshot feedback. Extend its pressure tests rather -than adding a second transport. Never reuse receiver installation to publish a local -leader checkpoint. Missing network responses remain ambiguous. +Continue in `controller/checkpoints.rs` and the private `checkpoint_before_apply` +completion path. The policy currently selects the predecessor at an apply boundary; +manual `publish_checkpoint` still requires the exact idle current root. Preserve +both contracts, independent engine validation, and the local suffix. Extend the +existing controller and authenticated transport fixtures for pressure/failure cuts. +Do not reuse receiver installation to publish a local leader checkpoint. Missing +network responses remain ambiguous. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 54fe21783..1883278c4 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,51 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 automatic checkpoints at controller apply boundaries + +Qualified source: `c27dfb416784804b30a38b1a90812c607ed3b00f`. All 230 selected Rust cases and 8 +gate commands pass, including all-target Clippy for Cellar/core/LTX, the five +native suites, format/layout, dependency isolation, docs, preflight and 37 validator +tests. `.cellar/cell-groups-evidence/CG-05-controller-checkpoints/checkpoint.json` +binds 308 source hashes, 5 copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +An explicit controller policy selects an applied cut before its successor starts +physical apply. One node slot spans blocking capture, reserved publication admission +and persistence. The controller holds the exact successor ticket, so a hot group's +current root cannot advance during capture. A private completion-lane operation +checks that ticket, engine and exact current root while preserving any later log +suffix. Only the matching persisted `CheckpointPublished` event releases successor +apply; admission/event ordering is handled either way. The existing public manual +publication checks remain unchanged. + +The policy uses the transport's configured node pool and owns initial checkpoint +endpoint registrations. Busy pool/admission retries back off until a later apply +boundary, letting apply proceed. Capture joins precede engine-claim release; +accepted publications remain runtime-owned through shutdown. Hard failures retain +tickets and quarantine the group; shared log uncertainty closes the node. Usage +reports pending checkpoints, actual publications and transient retries. + +Three new cases use production controllers and real TCP Raft traffic after startup, +with no fixture event pumping. Repeated interleaved writes to two groups compact +successive prefixes in both image formats. A held receiver saturates the leader's +checkpoint pool while both groups continue applying; releasing it permits later +checkpoint publication. Every voter retains the applied suffix and advances its +checkpoint; joined close releases all pool admission. Initial polling borrow-conflict +and duplicate-TCP fixture failures are retained; the same assertions passed after +repair. No assertion was weakened. + +**Reflection and next step:** this entry-count policy retains the idle final tail. +The critical next gap is scheduler/persistence pressure qualification: continuous +saturation fairness, exhausted apply/completion budgets, held sync and cancelled +close, checkpoint failure/quarantine capacity, and retained-root slot pressure. +Add deterministic coverage at those boundaries before time/byte-trigger expansion. +Then complete receiver crash/sync-error cuts, whole-group learner/joint membership +and lost-disk replacement, and run all five declared CG-05 acceptance cases. CG-06 +archive/reclamation follows; upload completion alone cannot authorize reclaim. +No new shared-group RustFS, load, power-loss or zero-quorum DR qualification is +claimed. Public cutover and legacy removal remain open. + ## CG-05 authenticated checkpoint transport Qualified source: `ceb7ed3149202660855d32df1ec5feed7775e068`. All 227 selected Rust cases pass across shared @@ -61,8 +106,8 @@ the snapshot-feedback admission gap; only the final committed-source gate counts No assertion was weakened. Existing native held-sync/owner tests remain regression coverage, rather than an independent TLS receiver crash or power-loss qualification. -The most critical implementation gap is automatic controller-owned checkpoint -selection/publication: coalesce work, coordinate with advancing apply, and bound +The next gap recorded at this transport checkpoint was automatic controller-owned +checkpoint selection/publication: coalesce work, coordinate with advancing apply, and bound capture/publication retries so hot groups can checkpoint without starving unrelated ones. Then qualify source/receiver pressure, stalled physical I/O and hint saturation, receiver process-kill/sync-error cuts, and retained-root slot exhaustion. Complete From 1814b27896c3c21166186fe8fbd1a1a6bdbb721a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 18:25:20 -0700 Subject: [PATCH 131/568] feat: persist whole-group membership and fence leadership handoff --- crates/cellar/durability/raft/cache.rs | 29 ++ .../durability/raft/cell_groups/apply/mod.rs | 20 +- .../raft/cell_groups/enrollment_authority.rs | 10 + .../raft/cell_groups/execution_authority.rs | 6 + .../durability/raft/cell_groups/membership.rs | 209 ++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/owner.rs | 205 +++++++- .../raft/cell_groups/read_authority.rs | 9 +- .../raft/cell_groups/runtime/mod.rs | 28 ++ .../raft/cell_groups/runtime/worker.rs | 6 + .../raft/group_snapshot_transport/send.rs | 6 +- crates/cellar/durability/raft/member_state.rs | 16 + .../durability/raft/transport_shared.rs | 4 +- .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_membership.rs | 466 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 4 + docs/design/multi-raft/cell-groups/handoff.md | 39 +- .../multi-raft/cell-groups/protocols.md | 38 ++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- 20 files changed, 1090 insertions(+), 27 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/membership.rs create mode 100644 crates/cellar/tests/support/cell_group_membership.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 934ef8690..b4e11d961 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -808,6 +808,35 @@ impl CachedGroup { next.group_manifest = Some(marker.clone()); next.manifest_pins = Some(prepared.images.clone()); } + if let (Some(previous), Some(conf)) = (&self.group_manifest, &write.conf_state) { + let prepared = write.group_manifest.as_ref().ok_or_else(|| { + LogError::Invalid("group configuration lacks complete root".into()) + })?; + let marker = prepared.marker(); + let entry = self + .entries + .get(&marker.position().index.get()) + .filter(|entry| entry.index <= self.hard_state.commit) + .ok_or_else(|| { + LogError::Invalid("group configuration lacks committed entry".into()) + })?; + let expected = super::cell_groups::membership::transition( + entry, + previous.route(), + &self.conf_state, + &self.membership, + )?; + if *conf != expected.conf + || write.membership.as_ref() != Some(&expected.membership) + || marker.root() != previous.root() + || marker.membership_revision() != entry.index + || marker.position().term.get() != entry.term + { + return Err(LogError::Invalid( + "group configuration/root transition".into(), + )); + } + } if let Some(conf) = &write.conf_state { if next.applied.position.index <= self.applied.position.index && conf != &self.conf_state diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 5a5cd5edf..27e50cf10 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -138,7 +138,11 @@ impl CellGroupApplier { if !Arc::ptr_eq(&self.group.owner, &work.base.engine_owner) || work.base.marker().route().history != self.history() || work.ticket.position() != entry_position(&work.entry) - || work.entry.get_entry_type() != ::raft::prelude::EntryType::EntryNormal + || !matches!( + work.entry.get_entry_type(), + ::raft::prelude::EntryType::EntryNormal + | ::raft::prelude::EntryType::EntryConfChangeV2 + ) { return Err(ExecutionError::Invalid("group apply identity/entry")); } @@ -178,9 +182,21 @@ impl CellGroupApplier { max_effect_rows, } = &mut *owner; // Check page owner and contiguous prefix before any image allocation/I/O. - let unchanged = tree + let mut unchanged = tree .prepare_unchanged(pages, &work.base, work.ticket.position()) .map_err(storage)?; + if work.entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { + let cache = self.group.read().map_err(storage)?; + super::membership::transition( + &work.entry, + tree.route(), + &cache.conf_state, + &cache.membership, + ) + .map_err(storage)?; + unchanged.marker.membership_revision = work.entry.index; + return Ok(unchanged); + } if work.entry.data.is_empty() { return Ok(unchanged); } diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs index 0c31e846a..d9e0986e2 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -131,6 +131,16 @@ impl CellEnrollmentAuthority { return Err(RuntimeError::NotReady); } self.state.check()?; + let cache = self.state.group.read()?; + if !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == self.state.group.identity().member.get()) + { + return Err(RuntimeError::NotReady); + } if self.leader_term.load(Ordering::Acquire) != self.state.term.get() { return Err(RuntimeError::NotReady); } diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index 193305a01..fffde1d66 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -109,6 +109,12 @@ impl CellExecutionAuthority { &self.leaf.version.history.cell, ); if cache.hard_state.term != self.term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == self.group.identity().member.get()) || current.route() != &self.route || !cache.membership.permits_group(self.group.identity()) || cache diff --git a/crates/cellar/durability/raft/cell_groups/membership.rs b/crates/cellar/durability/raft/cell_groups/membership.rs new file mode 100644 index 000000000..ffe028337 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/membership.rs @@ -0,0 +1,209 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Group-bound configuration entries and deterministic, authority-free replay. +//! Raft progress changes only after the complete configuration/root write syncs. +use super::GroupManifest; +use crate::raft::{ + member_state::{GroupMembership, JoiningMember}, + storage::LogError, +}; +use cellar_logic::{cell_group::GroupRoute, consensus::LogIndex}; +use protobuf::Message as _; +use raft::{ + prelude::{ConfChangeV2, ConfState, Entry, EntryType}, + restore_configuration, Changer, ProgressTracker, +}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; + +pub(crate) const MAX_CHANGE_BYTES: usize = 40 * 1024; +const MAX_CONTEXT_BYTES: usize = 36 * 1024; + +fn invalid() -> LogError { + LogError::Invalid("cell-group configuration binding or transition".into()) +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Intent { + domain: String, + route: [u8; 32], + intent: String, +} + +fn binding(route: &GroupRoute) -> Result<[u8; 32], LogError> { + // Canonical marker-1 genesis projects the complete route, including range, + // epoch, birth, schema and all history IDs, without a mutable applied cut. + let genesis = GroupManifest::genesis(route.clone()).map_err(|_| invalid())?; + Ok(Sha256::digest(serde_json::to_vec(&genesis).map_err(|_| invalid())?).into()) +} + +/// An explicit proposal bound to one route and observed membership revision. +/// Losing the admission reply requires inspecting committed membership; callers +/// must not manufacture a new revision to blindly repeat an ambiguous change. +#[derive(Clone, Debug)] +pub struct CellGroupMembershipChange { + pub(super) change: ConfChangeV2, +} + +impl CellGroupMembershipChange { + pub fn new( + route: &GroupRoute, + membership: &GroupMembership, + mut change: ConfChangeV2, + joins: &[JoiningMember], + ) -> Result { + if !change.context.is_empty() || change.changes.len() > 32 { + return Err(invalid()); + } + membership.bind_change(&mut change, joins)?; + let intent = Intent { + domain: "cellar-cell-group-membership-1".into(), + route: binding(route)?, + intent: String::from_utf8(change.context.to_vec()).map_err(|_| invalid())?, + }; + change.context = serde_json::to_vec(&intent).map_err(|_| invalid())?.into(); + validate(&change, route)?; + Ok(Self { change }) + } + + pub(crate) fn bytes(&self) -> usize { + self.change.compute_size() as usize + 4096 + } +} + +/// Bound repeated fields before protobuf allocation. Unknown fields and duplicate +/// singular fields refuse; each nested member change has a fixed small bound. +fn scan(bytes: &[u8]) -> Result<(), LogError> { + fn varint(bytes: &[u8], at: &mut usize) -> Result { + let mut value = 0; + for shift in (0..70).step_by(7) { + let byte = *bytes.get(*at).ok_or_else(invalid)?; + *at += 1; + if shift == 63 && byte > 1 { + return Err(invalid()); + } + value |= u64::from(byte & 127) << shift; + if byte < 128 { + return Ok(value); + } + } + Err(invalid()) + } + if bytes.len() > MAX_CHANGE_BYTES { + return Err(LogError::Overloaded); + } + let (mut at, mut count, mut transition, mut context) = (0, 0, false, false); + while at < bytes.len() { + match varint(bytes, &mut at)? { + 8 if !transition => { + transition = true; + if varint(bytes, &mut at)? > 2 { + return Err(invalid()); + } + } + tag @ (18 | 26) => { + let length = usize::try_from(varint(bytes, &mut at)?).map_err(|_| invalid())?; + if tag == 18 { + count += 1; + if count > 32 || length > 32 { + return Err(invalid()); + } + } else { + if context || length > MAX_CONTEXT_BYTES { + return Err(invalid()); + } + context = true; + } + at = at + .checked_add(length) + .filter(|end| *end <= bytes.len()) + .ok_or_else(invalid)?; + } + _ => return Err(invalid()), + } + } + Ok(()) +} + +fn validate(change: &ConfChangeV2, route: &GroupRoute) -> Result { + if change.compute_size() as usize > MAX_CHANGE_BYTES + || change.changes.len() > 32 + || change.get_unknown_fields().iter().next().is_some() + || change + .changes + .iter() + .any(|item| item.get_unknown_fields().iter().next().is_some()) + { + return Err(invalid()); + } + let mut decoded = change.clone(); + if change.context.is_empty() { + if !change.changes.is_empty() + || change.get_transition() != raft::prelude::ConfChangeTransition::Auto + { + return Err(invalid()); + } + return Ok(decoded); // Only a committed auto-leave is accepted by transition(). + } + if change.context.len() > MAX_CONTEXT_BYTES { + return Err(invalid()); + } + let intent: Intent = serde_json::from_slice(&change.context).map_err(|_| invalid())?; + if intent.domain != "cellar-cell-group-membership-1" + || intent.route != binding(route)? + || serde_json::to_vec(&intent).map_err(|_| invalid())? != change.context + { + return Err(invalid()); + } + decoded.context = intent.intent.into_bytes().into(); + Ok(decoded) +} + +pub(crate) fn decode(entry: &Entry, route: &GroupRoute) -> Result { + if entry.get_entry_type() != EntryType::EntryConfChangeV2 { + return Err(invalid()); + } + scan(&entry.data)?; + let change = ConfChangeV2::parse_from_bytes(&entry.data).map_err(|_| invalid())?; + // Reject duplicate nested fields and noncanonical encodings as well as + // the outer framing checked before allocation. All local writers use this + // same protobuf serializer, including Raft's empty automatic leave. + if change.write_to_bytes().map_err(|_| invalid())? != entry.data { + return Err(invalid()); + } + validate(&change, route) +} + +pub(crate) struct Transition { + pub conf: ConfState, + pub membership: GroupMembership, +} + +pub(crate) fn transition( + entry: &Entry, + route: &GroupRoute, + before: &ConfState, + membership: &GroupMembership, +) -> Result { + let change = decode(entry, route)?; + if change.context.is_empty() && (before.voters_outgoing.is_empty() || !before.auto_leave) { + return Err(invalid()); + } + membership.admit_group(&change, before)?; + let mut tracker = ProgressTracker::new(1); + restore_configuration(&mut tracker, 1, before).map_err(|_| invalid())?; + let mut changer = Changer::new(&tracker); + let (conf, progress) = if change.leave_joint() { + changer.leave_joint() + } else if let Some(auto) = change.enter_joint() { + changer.enter_joint(auto, &change.changes) + } else { + changer.simple(&change.changes) + } + .map_err(|_| invalid())?; + tracker.apply_conf(conf, progress, 1); + let conf = tracker.conf().to_conf_state(); + let membership = membership.applied(&change, before, &conf, LogIndex::new(entry.index))?; + Ok(Transition { conf, membership }) +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index c224c3896..488ae5a83 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -50,6 +50,8 @@ mod enrollment; mod enrollment_apply; mod image_pin; mod manifest; +pub(crate) mod membership; +pub use membership::CellGroupMembershipChange; mod owner; mod page_store; mod page_wire; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index ded2ba324..c88e64516 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -287,7 +287,16 @@ impl CellGroupOwner { } fn refresh_execution_fence(&mut self) { - let term = if self.node.raft.state == StateRole::Leader { + let term = if self.node.raft.state == StateRole::Leader + && self.node.raft.lead_transferee.is_none() + && self + .node + .raft + .prs() + .conf() + .voters() + .contains(self.node.raft.id) + { self.node.raft.term } else { 0 @@ -412,6 +421,14 @@ impl CellGroupOwner { applied: self.current.marker().position(), committed: LogIndex::new(cache.hard_state.commit), ready: self.node.raft.state == StateRole::Leader + && self.node.raft.lead_transferee.is_none() + && self + .node + .raft + .prs() + .conf() + .voters() + .contains(self.node.raft.id) && self.phase.is_none() && self.checkpoint.is_none() && self.apply_bytes < self.options.max_group_apply_bytes @@ -432,6 +449,40 @@ impl CellGroupOwner { Ok(()) } + /// Admit a handoff to a caught-up incoming voter. Completion is observed + /// through leadership status; admission alone cannot prove an election. + pub fn transfer_leader(&mut self, member: u64) -> Result<(), RuntimeError> { + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let status = self.status()?; + let cache = self.group.read()?; + if member == self.node.raft.id || !cache.conf_state.voters.contains(&member) { + return Err(RuntimeError::Invalid( + "leadership target is not another incoming voter".into(), + )); + } + if !status.ready + || status.applied.index != status.committed + || !self.apply.is_empty() + || self.applying.is_some() + || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied + || self + .node + .raft + .prs() + .get(member) + .is_none_or(|p| p.matched < self.node.raft.raft_log.last_index()) + { + return Err(RuntimeError::NotReady); + } + drop(cache); + self.node.transfer_leader(member); + self.refresh_execution_fence(); + Ok(()) + } + pub fn tick(&mut self) -> Result<(), RuntimeError> { self.live()?; // The scheduler coalesces a deferred tick until pending Ready work is @@ -622,6 +673,101 @@ impl CellGroupOwner { Ok(()) } + /// Propose a route/revision-bound whole-group configuration. The returned + /// position is admission only; the persisted applied root proves completion. + pub fn configure( + &mut self, + change: super::CellGroupMembershipChange, + ) -> Result { + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !self.status()?.ready + || !self.apply.is_empty() + || self.applying.is_some() + || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied + { + return Err(RuntimeError::NotReady); + } + let before = self.node.raft.raft_log.last_index(); + let entry = Entry { + entry_type: EntryType::EntryConfChangeV2, + index: before + 1, + term: self.node.raft.term, + data: change + .change + .write_to_bytes() + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + .into(), + ..Default::default() + }; + if entry.data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let cache = self.group.read()?; + let next = super::membership::transition( + &entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + if crate::raft::member_state::members(&next.conf).len() > self.options.max_members_per_group + { + return Err(RuntimeError::Overloaded); + } + for item in &change.change.changes { + if item.get_change_type() == ConfChangeType::AddNode + && !cache.conf_state.voters.contains(&item.node_id) + && self + .node + .raft + .prs() + .get(item.node_id) + .is_none_or(|p| p.matched < cache.hard_state.commit) + { + return Err(RuntimeError::NotReady); + } + } + drop(cache); + let expected = entry.data.clone(); + if let Err(error) = self.node.propose_conf_change(Vec::new(), change.change) { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); + } + let result = (|| { + let observed = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol("missing configuration proposal".into()))?; + if observed.index != before + 1 + || observed.get_entry_type() != EntryType::EntryConfChangeV2 + || observed.data != expected + { + return Err(RuntimeError::Protocol( + "configuration proposal changed".into(), + )); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(observed), + digest: entry_digest(observed)?, + }) + })(); + if let Err(error) = &result { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + result + } + /// Protocol admission only. The shared cell executor must serialize each /// cell's preparation, reserve physical work and retain ambiguous outcomes. /// The first prepared submission permanently closes this byte-only entry @@ -1114,13 +1260,42 @@ impl CellGroupOwner { if !submitted { let mut write = GroupWrite::new(&self.group); write.group_manifest = Some((*next).clone()); + let entry = self.apply.front().ok_or(RuntimeError::StaleCompletion)?; + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + let cache = self.group.read()?; + let transition = super::membership::transition( + entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + write.conf_state = Some(transition.conf); + write.membership = Some(transition.membership); + } self.phase = Some(Phase::Apply(ticket, next, true)); return self.persistence(write).map(Some); } if self.group.group_manifest()?.as_ref() != Some(next.marker()) { return Err(RuntimeError::StaleCompletion); } + let entry = self.apply.front().ok_or(RuntimeError::StaleCompletion)?; + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + let change = + super::membership::decode(entry, self.current.marker().route())?; + let conf = self.node.apply_conf_change(&change)?; + if conf != self.group.initial_state()?.conf_state { + return Err(RuntimeError::StaleCompletion); + } + self.refresh_execution_fence(); + } + let last = self.node.raft.raft_log.last_index(); self.node.advance_apply_to(ticket.position.index.get()); + if self.node.raft.raft_log.last_index() > last { + // Raft appended an automatic joint-leave entry. Drive + // replication without waiting for a later write/tick; + // Ready still fences messages behind its log sync. + self.node.raft.bcast_append(); + } let entry = self .apply .pop_front() @@ -1235,7 +1410,25 @@ impl CellGroupOwner { next: &PreparedGroupManifest, ) -> Result<(), RuntimeError> { let invalid = || RuntimeError::Invalid("group apply does not match committed entry".into()); - if entry.get_entry_type() != EntryType::EntryNormal { + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + let cache = self.group.read()?; + super::membership::transition( + entry, + self.current.marker().route(), + &cache.conf_state, + &cache.membership, + )?; + return if next.marker().root() == self.current.marker().root() + && next.marker().membership_revision() == entry.index + { + Ok(()) + } else { + Err(invalid()) + }; + } + if entry.get_entry_type() != EntryType::EntryNormal + || next.marker().membership_revision() != self.current.marker().membership_revision() + { return Err(invalid()); } if entry.data.is_empty() { @@ -1361,11 +1554,17 @@ impl CellGroupOwner { || entry.term == 0 || entry.term < term || entry.term > message.term - || entry.get_entry_type() != EntryType::EntryNormal + || !matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) || entry.data.len() > self.options.max_proposal_bytes { return Err(invalid()); } + if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + super::membership::decode(entry, self.current.marker().route())?; + } term = entry.term; } } diff --git a/crates/cellar/durability/raft/cell_groups/read_authority.rs b/crates/cellar/durability/raft/cell_groups/read_authority.rs index 60670f825..5f700e656 100644 --- a/crates/cellar/durability/raft/cell_groups/read_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/read_authority.rs @@ -104,7 +104,14 @@ impl CellReadAuthority { .is_none_or(|m| m.route() != &s.route) || !cache.membership.permits_group(s.group.identity()) || s.leader.as_ref().is_some_and(|(term, leader)| { - cache.hard_state.term != term.get() || leader.load(Ordering::Acquire) != term.get() + cache.hard_state.term != term.get() + || leader.load(Ordering::Acquire) != term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == s.group.identity().member.get()) }) { return Err(RuntimeError::NotReady); diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 9c56792a4..84321fbae 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -95,6 +95,8 @@ enum Command { Attach(Box), Quarantine, Campaign, + TransferLeader(u64), + Configure(CellGroupMembershipChange), Execution(cellar_logic::cell_group::CellHistory), Enrollment(cellar_logic::cell_group::CellHistory), EnrollmentState(cellar_logic::cell_group::CellHistory), @@ -526,6 +528,32 @@ impl CellGroupRuntime { } } + pub async fn configure( + &self, + history: GroupHistory, + change: CellGroupMembershipChange, + ) -> Result { + let bytes = change.bytes(); + match self + .call(Some(history), Command::Configure(change), bytes) + .await? + { + Response::Proposal(proposal) => Ok(proposal), + _ => Err(RuntimeError::Protocol("configuration reply".into())), + } + } + + /// Admit whole-group leadership handoff; observe status for completion. + pub async fn transfer_leader( + &self, + history: GroupHistory, + member: u64, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::TransferLeader(member), 4096) + .await + .map(|_| ()) + } + pub async fn campaign(&self, history: GroupHistory) -> Result<(), RuntimeError> { self.call(Some(history), Command::Campaign, 4096) .await diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index f190171e5..18f2d144e 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -407,6 +407,12 @@ impl Worker { .owner .reconciliation_authority(&cell) .map(|authority| Response::Reconciliation(Box::new(authority))), + Command::Configure(change) => { + group.owner.configure(change).map(Response::Proposal) + } + Command::TransferLeader(member) => { + group.owner.transfer_leader(member).map(|()| Response::Unit) + } Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } diff --git a/crates/cellar/durability/raft/group_snapshot_transport/send.rs b/crates/cellar/durability/raft/group_snapshot_transport/send.rs index c189a4f56..a61f0f8de 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/send.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/send.rs @@ -21,9 +21,11 @@ impl RaftTransport { .peers .iter() .find(|p| p.replica.member.get() == message.to) - .filter(|p| p.replica.node != local.node) .cloned() - .ok_or(TransportError::Identity)?; + .ok_or(TransportError::StalePeer)?; + if peer.replica.node == local.node { + return Err(TransportError::Identity); + } shared::permitted(&group, &[local, &peer.replica])?; let route = Route::new_group( local, diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index 8de797d91..3d58d5b65 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -298,6 +298,22 @@ impl GroupMembership { Ok(intent) } + pub(crate) fn admit_group( + &self, + change: &ConfChangeV2, + conf: &ConfState, + ) -> Result<(), LogError> { + let intent = self.intent(change)?; + if self.fleet.is_some() + || self.repair.is_some() + || intent.fleet.is_some() + || intent.repair.is_some() + { + return Err(invalid()); + } + self.admit(change, conf) + } + pub(crate) fn admit(&self, change: &ConfChangeV2, conf: &ConfState) -> Result<(), LogError> { if change.changes.len() > MAX_MEMBERS { return Err(LogError::Overloaded); diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 43c9548cc..150c4fd46 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -247,7 +247,9 @@ impl RaftTransport { .peers .iter() .find(|p| p.replica.member.get() == message.to) - .ok_or(TransportError::Identity)? + // A committed learner can precede discovery of its process route. + // Treat that interval as unavailable delivery, not node corruption. + .ok_or(TransportError::StalePeer)? .clone(); if peer.replica.node == local.node { return Err(TransportError::Identity); diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index b7e0dff81..8824c937f 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,21 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_materialized_joint_changes_preserve_cells() -> Result<()> { + runtime_fixture::membership_lifecycle(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_extent_joint_changes_preserve_cells() -> Result<()> { + runtime_fixture::membership_lifecycle(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_group_configuration_waits_for_sync_and_joins_close() -> Result<()> { + runtime_fixture::membership_sync().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_controller_compacts_materialized_groups_during_writes() -> Result<()> { runtime_fixture::automatic_checkpoints(false, false).await diff --git a/crates/cellar/tests/support/cell_group_membership.rs b/crates/cellar/tests/support/cell_group_membership.rs new file mode 100644 index 000000000..f6bf42e17 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_membership.rs @@ -0,0 +1,466 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Whole-group configuration changes over actual TCP Raft and production apply. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::OperationId; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage, +}; + +fn change(member: u64, kind: ConfChangeType, transition: ConfChangeTransition) -> ConfChangeV2 { + ConfChangeV2 { + changes: vec![ConfChangeSingle { + node_id: member, + change_type: kind, + ..Default::default() + }] + .into(), + transition, + ..Default::default() + } +} + +async fn configure_at( + fixture: &RuntimeCluster, + leader: usize, + route: &GroupRoute, + command: ConfChangeV2, + joins: &[JoiningMember], +) -> Result { + let group = fixture.cluster.voters[leader] + .log + .cell_group(&route.history)?; + let label = format!("{command:?}"); + let request = CellGroupMembershipChange::new(route, &group.membership()?, command, joins)?; + let accepted = tokio::time::timeout(Duration::from_secs(10), async { + loop { + match fixture.runtimes[leader] + .configure(route.history, request.clone()) + .await + { + Ok(accepted) => return Ok::<_, anyhow::Error>(accepted), + // Retry only a definite pre-admission refusal, preserving the + // same observed revision. A lost response is never retried. + Err(RuntimeError::NotReady) => tokio::task::yield_now().await, + Err(error) => return Err(error.into()), + } + } + }) + .await??; + controller::applied(fixture, route.history, accepted.position.index) + .await + .with_context(|| format!("configuration apply: {label}"))?; + Ok(accepted) +} + +async fn configure( + fixture: &RuntimeCluster, + route: &GroupRoute, + command: ConfChangeV2, + joins: &[JoiningMember], +) -> Result { + configure_at(fixture, 0, route, command, joins).await +} + +pub(crate) async fn membership_lifecycle(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + for transport in &fixture.transports { + transport.enable_group_checkpoints(CellGroupCheckpointPool::new( + checkpoint_fixture::limits(), + )?)?; + } + let mut limits = controller::limits(); + limits.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, limits).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let original = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + let root = group.group_manifest()?.context("root")?.root(); + let demote = change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ); + let stale = CellGroupMembershipChange::new(&route, &group.membership()?, demote.clone(), &[])?; + let foreign = fixture.extra[&second][0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second root")?; + let wrong_group = + CellGroupMembershipChange::new(foreign.route(), &group.membership()?, demote.clone(), &[])?; + let last = group.last_index()?; + assert!(fixture.runtimes[0] + .configure(route.history, wrong_group) + .await + .is_err()); + assert_eq!(group.last_index()?, last); + let entered = configure(&fixture, &route, demote, &[]).await?; + for voter in &fixture.cluster.voters { + let log = voter.log.cell_group(&route.history)?; + let conf = log.initial_state()?.conf_state; + assert_eq!(conf.voters, vec![1, 2]); + assert_eq!(conf.voters_outgoing, vec![1, 2, 3]); + assert_eq!(conf.learners_next, vec![3]); + assert_eq!(log.membership()?.revision(), entered.position.index); + assert_eq!(log.group_manifest()?.context("root")?.root(), root); + } + let last = group.last_index()?; + assert!(fixture.runtimes[0] + .configure(route.history, stale) + .await + .is_err()); + assert_eq!(group.last_index()?, last); + let last = group.last_index()?; + let term = fixture.runtimes[0].status(route.history).await?.term.get(); + let mut malformed = group + .entries( + entered.position.index.get(), + entered.position.index.get() + 1, + None::, + raft::GetEntriesContext::empty(false), + )? + .remove(0); + let mut bytes = malformed.data.to_vec(); + bytes.extend_from_slice(&[8, 0]); // Duplicate transition with conflicting value. + malformed.data = bytes.into(); + malformed.index = last + 1; + malformed.term = term + 1; + let message = raft::prelude::Message { + msg_type: raft::prelude::MessageType::MsgAppend, + from: 2, + to: 1, + term: term + 1, + index: last, + log_term: group.term(last)?, + entries: vec![malformed].into(), + ..Default::default() + }; + assert!(fixture.runtimes[0] + .step(route.history, message) + .await + .is_err()); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.term.get(), + term + ); + assert_eq!(group.last_index()?, last); + configure(&fixture, &route, ConfChangeV2::default(), &[]).await?; + assert_eq!(group.initial_state()?.conf_state.learners, vec![3]); + + let join = JoiningMember { + member: MemberId::new(4)?, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + }; + configure( + &fixture, + &route, + change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + std::slice::from_ref(&join), + ) + .await?; + assert_eq!( + group + .membership()? + .joined(join.member)? + .context("bound learner")? + .0, + join + ); + // Checkpoint sends have the same retryable route-discovery refusal as + // ordinary append traffic; no stream or source owner is admitted. + let snapshot = raft::prelude::Message { + msg_type: raft::prelude::MessageType::MsgSnapshot, + from: 1, + to: 4, + ..Default::default() + }; + assert!(matches!( + fixture.transports[0].submit_physical(&route.history, snapshot), + Err(TransportError::StalePeer) + )); + controller::wait_for(async || Ok(controllers[0].usage().send_refusals > 0)).await?; + assert!( + !controllers[0].usage().closing, + "discovery lag must not close the node" + ); + let promotion = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change(4, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + )?; + let last = group.last_index()?; + assert!(matches!( + fixture.runtimes[0] + .configure(route.history, promotion) + .await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.last_index()?, last); + configure( + &fixture, + &route, + change(3, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + configure( + &fixture, + &route, + change(4, ConfChangeType::RemoveNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + let reused = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[join], + )?; + assert!(fixture.runtimes[0] + .configure(route.history, reused) + .await + .is_err()); + assert_eq!(group.membership()?.high_water(), 4); + + // Implicit joint consensus must generate and durably apply its own empty + // leave entry. The harness never manufactures that automatic command. + let implicit = configure( + &fixture, + &route, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Implicit, + ), + &[], + ) + .await?; + controller::wait_for(async || { + for voter in &fixture.cluster.voters { + let log = voter.log.cell_group(&route.history)?; + if log.membership()?.revision() <= implicit.position.index + || !log.initial_state()?.conf_state.voters_outgoing.is_empty() + { + return Ok(false); + } + } + Ok(true) + }) + .await + .context("automatic joint leave did not finish")?; + configure( + &fixture, + &route, + change(3, ConfChangeType::AddNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + assert_eq!( + fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await? + .cell(), + original.cell() + ); + + // Move leadership using the real TimeoutNow protocol. The fixture does + // not advance election ticks or inject votes to manufacture this handoff. + assert!(fixture.runtimes[0] + .transfer_leader(route.history, 4) + .await + .is_err()); + controller::wait_for(async || { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + assert!(original.check().is_err()); + controller::wait_for(async || Ok(fixture.runtimes[1].status(route.history).await?.ready)) + .await?; + let producer = fixture.runtimes[1] + .execution_authority(route.history, a.clone()) + .await?; + let cell = history(&route, "after-new-leader")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[1] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(&fixture, route.history, accepted.position.index).await?; + // Demote the new current leader. Captured producer authority must be + // revoked while another group on the original node continues serving. + configure_at( + &fixture, + 1, + &route, + change( + 2, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + configure_at(&fixture, 1, &route, ConfChangeV2::default(), &[]).await?; + assert!(producer.check().is_err()); + assert!(!fixture.runtimes[1].status(route.history).await?.ready); + assert!(fixture.runtimes[0].status(second).await?.ready); + let second_root = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second")?; + let cell = history(second_root.route(), "unrelated")?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(second_root.route(), &cell, &bytes, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + let conf = group.initial_state()?.conf_state; + let membership = group.membership()?; + assert_eq!(conf.voters, vec![1, 3]); + assert_eq!(conf.learners, vec![2]); + assert_eq!(membership.high_water(), 4); + assert!(group.checkpoint_manifest()?.is_some()); + for controller in &controllers { + assert_eq!(controller.usage().quarantined_groups, 0); + controller.close().await?; + } + drop(controllers); + fixture.cluster.voters[0].log.close().await?; + let reopened = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-1/log"), + NodeId::new([1; 16])?, + DiskLineage::new([11; 16])?, + ProcessGeneration::new([21; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let restored = reopened.cell_group(&route.history)?; + assert_eq!(restored.initial_state()?.conf_state, conf); + assert_eq!(restored.membership()?, membership); + assert_eq!( + restored + .group_manifest()? + .context("restored root")? + .membership_revision(), + membership.revision().get() + ); + // Metadata reopening alone is not permission to serve or a disk-replacement proof. + reopened.close().await?; + fixture.close().await +} + +pub(crate) async fn membership_sync() -> Result<()> { + for close in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("before")?; + let membership = group.membership()?; + let conf = group.initial_state()?.conf_state; + let request = CellGroupMembershipChange::new( + &route, + &membership, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + )?; + let accepted = fixture.runtimes[0] + .configure(route.history, request) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(10), async { + while !held.contains_key(&route.history) { + fixture.pump(&[route.history], &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let work = held.remove(&route.history).context("configuration apply")?; + let mut incomplete = GroupWrite::new(&group); + incomplete.conf_state = Some(conf.clone()); + incomplete.membership = Some(membership.clone()); + assert!(fixture.cluster.voters[0] + .log + .persist_cells(OperationId::new(991), vec![incomplete]) + .await + .is_err()); + let (sql, applier) = fixture.appliers[&(0, route.history)].clone(); + let prepared = sql.prepare_apply(&applier, work).await?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + prepared.complete(&fixture.runtimes[0]).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.initial_state()?.conf_state, conf); + assert_eq!(group.membership()?, membership); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before.position() + ); + if close { + assert!( + tokio::time::timeout(Duration::from_millis(30), fixture.runtimes[0].close()) + .await + .is_err() + ); + } + drop(release); + if close { + fixture.runtimes[0].close().await?; + } else { + fixture.drive(Some(accepted.position.index)).await?; + } + assert_eq!(group.membership()?.revision(), accepted.position.index); + assert_eq!( + group + .group_manifest()? + .context("after")? + .membership_revision(), + accepted.position.index.get() + ); + assert_eq!(group.initial_state()?.conf_state.learners_next, vec![3]); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 51591d113..a33e1fe2c 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -809,3 +809,5 @@ pub(super) use sql::{ }; pub(super) use sql::automatic_checkpoints; + +pub(super) use sql::{membership_lifecycle, membership_sync}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 8ff57f082..f41ea23c0 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -595,3 +595,7 @@ pub(crate) use provision::{ #[path = "cell_group_controller_checkpoints.rs"] mod controller_checkpoints; pub(crate) use controller_checkpoints::automatic_checkpoints; + +#[path = "cell_group_membership.rs"] +mod membership; +pub(crate) use membership::{membership_lifecycle, membership_sync}; diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 58d10593d..659aeb3f2 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -663,16 +663,20 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Qualify CG-05 checkpoint scheduling and remaining transport pressure cuts. - Automatic entry-count selection now holds the successor ticket until exact - publication. Exercise continuous saturation fairness, exhausted apply/completion - budgets, held sync, cancelled close, quarantine capacity and retained-root - pressure. Keep unrelated-group progress and bounded ownership observable. - Add idle/time/byte triggers only with an explicit cut-reservation contract. - Keep stream admission separate from Raft's post-persistence acknowledgement. -2. Finish CG-05 membership and recovery. Qualify the outstanding receiver fault - cuts and root-slot pressure, then whole-group learner catch-up, joint membership - and lost-disk replacement. Run every exact acceptance case before DONE. +1. Finish CG-05 fresh learner seeding and lost-disk replacement. Bind a seed + capability to committed learner membership, permanent node/disk identity and + the complete verified checkpoint. Existing bootstrap permits admit only birth + members; do not weaken them to accept an untrusted metadata observation. Bring + up the new learner through production transport, install every cell, catch up, + promote through joint consensus and retire the old member. Prove replacement + after deleting that voter's disk, minority refusal and one-voter-loss tolerance. +2. Complete CG-05 qualification while delivering those membership features. + Preserve all five exact acceptance cases. Add receiver crash/sync-error cuts, + continuous checkpoint saturation fairness, exhausted apply/completion budgets, + held sync/cancelled close, quarantine capacity and retained-root pressure. + Keep unrelated-group progress and bounded physical ownership observable. + Idle/time/byte checkpoint triggers require an explicit cut-reservation contract; + they must not displace fresh-member recovery as the next feature priority. 3. Deliver CG-06 archive/reclamation with verified complete checkpoints and contiguous tails. Upload completion alone cannot authorize reclaiming local recovery dependencies or an S3 durability acknowledgement. @@ -687,13 +691,14 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Continue in `controller/checkpoints.rs` and the private `checkpoint_before_apply` -completion path. The policy currently selects the predecessor at an apply boundary; -manual `publish_checkpoint` still requires the exact idle current root. Preserve -both contracts, independent engine validation, and the local suffix. Extend the -existing controller and authenticated transport fixtures for pressure/failure cuts. -Do not reuse receiver installation to publish a local leader checkpoint. Missing -network responses remain ambiguous. +Continue from `cell_groups/membership.rs`, the owner/runtime configuration path, +`directory/bootstrap.rs` and the existing checkpoint receiver/installation path. +Keep configuration, membership revision and complete root atomic. A captured +metadata observation, transport admission reply or lost response cannot create +new serving authority. Reuse permanent member claims and the node checkpoint pool +for learner bootstrap; retain physical closure owners through cancellation and +joined shutdown. Extend the production-controller membership fixture rather than +manually manufacturing committed transitions. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 919276232..7e161b283 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -204,6 +204,44 @@ Request-driven provisioning/lazy birth and configuration lifecycle remain open. The current lane serializes RPC exchanges per peer; this is not load qualification. +## Whole-group membership + +The shared owner accepts `CellGroupMembershipChange` through bounded runtime +admission. Its marker-1 domain binds the complete route and the observed membership +revision. Explicit changes use V2 entries; legacy V1 and unbound changes refuse. +Decoding bounds wire bytes, repeated changes and context before protobuf allocation. +New member IDs must exceed the permanent high-water mark and first enter as +learners with node/disk bindings. A removed ID cannot be reused. Promotion requires +the leader to observe that learner replicated through its durable committed cut. +These checks do not provision a fresh learner or grant it serving authority. + +Configuration entries occupy the same ordered apply lane as cell mutations. Pure +replay computes the next Raft configuration and membership without changing the +live RawNode. The complete group root advances its applied position and membership +revision while preserving every cell leaf and version. Engine persistence validates +the committed entry and atomically syncs configuration, membership and root. Only +the matching persistence completion changes live Raft progress and advances apply. +Explicit and implicit joint changes are supported; Raft's context-free automatic +leave is accepted only while the persisted configuration requests auto-leave. + +Admission returns a log position and digest, not a committed membership proof. +After a lost reply, inspect committed membership instead of rebinding a new revision +and blindly repeating the operation. A new binding may precede discovery of its +physical process route; missing peer routes are retryable stale-peer failures. +Authentication and active membership still gate transport admission. + +`transfer_leader` admits a handoff only to another caught-up incoming voter after +local committed apply settles. It revokes captured producer authority during the +handoff; observe the new leader's current-term readiness for completion. Removed +or demoted leaders lose producer and leader-bound read authority at synced cache +publication, even before the owner consumes its completion. Membership alone never +allows a learner to produce writes. + +Fresh learner seeding, initial-member replacement, lost-disk rebuild and managed +fleet orchestration remain required CG-05 work. Existing bootstrap permits still +admit only initial birth members. A metadata reopen is not a fresh-member seed, +quorum proof, verified physical recovery or serving permission. + ## Apply, checkpoint and archive For entry N, verify the cell base/sequence, stage the result, sync its immutable diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index d6f1b25e2..ffd579a22 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1586,7 +1586,8 @@ "crates/cellar/application/consensus/mod.rs", "crates/cellar/tests/support/cell_group_network_probes.rs", "crates/cellar/tests/support/cell_group_controller.rs", - "crates/cellar/tests/support/cell_group_controller_checkpoints.rs" + "crates/cellar/tests/support/cell_group_controller_checkpoints.rs", + "crates/cellar/tests/support/cell_group_membership.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From efb00dd8d6928f2edb6d2459a21c7556368aa35a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 18:46:03 -0700 Subject: [PATCH 132/568] docs: qualify group membership and prioritize fresh learner recovery --- docs/design/multi-raft/cell-groups/README.md | 11 +++- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 34 ++++++++--- .../design/multi-raft/cell-groups/progress.md | 58 +++++++++++++++++++ 4 files changed, 96 insertions(+), 13 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 7e9ebd247..1b127db08 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -7,8 +7,10 @@ CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and CG-04 shared execution have passed their scoped local evidence gates. CG-05 is in progress: checkpoint capture, receiver verification, atomic storage installation, exact Raft Ready acceptance, local publication and native Raft-generated -catch-up pass their scoped regression gates. Authenticated snapshot RPC, automated -controller lifecycle and membership recovery remain open. The production host has not yet cut over to shared groups. +catch-up, authenticated snapshot RPC and automatic controller checkpoint policy +pass their scoped regression gates. Durable joint membership and leadership handoff +also pass a partial gate. Fresh learner bootstrap and lost-disk replacement are the +next feature priorities. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -55,7 +57,10 @@ CG-05's partial source checkpoint is retained in `.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`, followed by owner/runtime acceptance in `.cellar/cell-groups-evidence/CG-05-snapshot-owner/`. These partial checkpoints and local publication in -`.cellar/cell-groups-evidence/CG-05-local-checkpoint/` do not complete CG-05. +`.cellar/cell-groups-evidence/CG-05-local-checkpoint/`, authenticated transport in +`.cellar/cell-groups-evidence/CG-05-snapshot-transport/`, automatic policy in +`.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` and membership in +`.cellar/cell-groups-evidence/CG-05-group-membership/` do not complete CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index e544df70e..650605b8c 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ceb7ed3149202660855d32df1ec5feed7775e068", + "source_revision": "1814b27896c3c21166186fe8fbd1a1a6bdbb721a", "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,7 +19,7 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "a961606bd7b2af30028b5956117e6f3c2e38f42b6caf485e334433c4dea4dff2", + "sha256": "8a33262be0c281ae3492043dbf72aa5a8ff6b8f2a84be4e6ba93ce4551082a3c", "line": 442, "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n}\n" }, diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 659aeb3f2..d2e666474 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -32,8 +32,9 @@ Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate Local checkpoint publication and native Raft-generated catch-up now pass their partial gate. Authenticated checkpoint streams now pass their partial transport gate. Automatic entry-count checkpointing at controller apply boundaries now passes its -partial gate. Continue with scheduler/persistence pressure qualification, then -whole-group membership and lost-disk recovery. +partial gate. Durable whole-group membership and leadership handoff now pass +an additional partial gate. Continue with fresh learner bootstrap and lost-disk +replacement; retain the remaining pressure/fault checks in the full CG-05 gate. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track @@ -85,6 +86,24 @@ ordinary request pressure; transport ownership lasts through feedback admission or shutdown. The wire acknowledgement remains admission only. See [progress](progress.md) for exact limits and retained intermediate diagnostics. +The latest membership checkpoint at `1814b278` passes 233 selected Rust +cases, 8 gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-group-membership/` binds 310 source hashes +and 5 copied executables. Route/revision-bound configuration entries atomically +sync membership/configuration and the unchanged complete cell root before live +Raft changes. Learner bindings, promotion checks, explicit/automatic joint changes, +leader handoff, demotion fencing and retryable discovery lag are implemented. +Two real-TCP controller cases cover both image forms; a third holds configuration +sync through continued operation and cancelled/joined shutdown. Automatic joint +leave now replicates immediately without another write or tick. + +Fresh learner 4 is intentionally not started: seed admission and lost-disk rebuild +remain the next feature-critical work. Metadata reopen is not complete physical +recovery or serving permission. Existing bootstrap permits still require birth +membership. Preserve all five CG-05 cases, outstanding crash/pressure cuts and +shared RustFS/load qualification. See [progress](progress.md) for exact evidence +and the retained intermediate failures. + The latest controller checkpoint at `c27dfb41` passes 230 selected Rust cases, 8 gate commands and 37 validator tests. Its record under `.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` binds 308 source @@ -100,9 +119,9 @@ The three new cases prove repeated compaction across two groups in both image forms and progress/recovery while the checkpoint pool is saturated. The policy retains the final idle tail. Continuous-saturation fairness, exhausted apply and completion budgets, held sync/cancelled close, checkpoint failure/quarantine capacity -and retained-root pressure are the critical next qualification gaps. Follow with -receiver crash/sync-error cuts, whole-group learners/joint membership and lost-disk -replacement. All five declared CG-05 acceptance cases remain required. See +and retained-root pressure remain qualification gaps. Fresh learner bootstrap and +lost-disk replacement are the next feature priorities; receiver crash/sync-error +cuts also remain required. All five declared CG-05 acceptance cases remain required. See [progress](progress.md) for limits and intermediate failures. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands @@ -671,8 +690,9 @@ Do not let plan maintenance replace implementation. promote through joint consensus and retire the old member. Prove replacement after deleting that voter's disk, minority refusal and one-voter-loss tolerance. 2. Complete CG-05 qualification while delivering those membership features. - Preserve all five exact acceptance cases. Add receiver crash/sync-error cuts, - continuous checkpoint saturation fairness, exhausted apply/completion budgets, + Preserve all five exact acceptance cases. Add joint-transition crash cuts, + receiver crash/sync-error cuts, continuous checkpoint saturation fairness, + exhausted apply/completion budgets, held sync/cancelled close, quarantine capacity and retained-root pressure. Keep unrelated-group progress and bounded physical ownership observable. Idle/time/byte checkpoint triggers require an explicit cut-reservation contract; diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1883278c4..6b4b6a0fa 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,64 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 durable membership and leadership handoff + +Qualified source: `1814b27896c3c21166186fe8fbd1a1a6bdbb721a`. All 233 selected Rust cases and 8 +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, format/layout, dependency isolation, docs, source preflight and 37 validator +tests. `.cellar/cell-groups-evidence/CG-05-group-membership/checkpoint.json` binds +310 source hashes, 5 copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +The owner/runtime now accepts canonical, bounded V2 membership intents bound to +one complete route and observed membership revision. Pure replay computes the +configuration without changing the live RawNode. Ordered apply preserves every +cell leaf while advancing the group position and membership revision. The engine +independently verifies the committed entry and atomically syncs configuration, +permanent member bindings and the complete root. Only its matching completion +changes live Raft progress and advances apply. Explicit and implicit joint changes +use this same boundary. Raft-generated automatic leave starts replication immediately; +it does not wait for a later write or tick. + +Fresh IDs first enter as learners with node/disk bindings. The permanent high-water +mark prevents retired-ID reuse. Promotion requires leader-observed replication +through its durable committed cut. Handoff targets another caught-up incoming voter +and revokes captured producer authority while it proceeds. Synced voter checks +also fence demoted/removed leaders before the owner consumes cache publication. +New learner routes may lag committed membership: ordinary and checkpoint sends +return retryable stale-peer failures without closing the node. + +Three new tests cover production controllers and real TCP Raft in both physical +image forms, plus held configuration sync and cancelled/joined runtime close. +Lifecycle cases preserve cell identity/version, exercise explicit and automatic +joint changes, stale/foreign/malformed refusal, learner binding, unsafe-promotion +refusal, retired-ID refusal, real leadership transfer, new-leader writes, +self-demotion fencing, unrelated-group writes and automatic checkpoints. Joined +engine metadata reopen retains the exact final configuration and revision. No +fixture event pumping follows controller startup in these two lifecycle cases. +The separate held-sync case proves old root/configuration/applied state remain +selected until sync and that shutdown joins the accepted write. + +Initial private Raft accessor and missing-import compile failures are retained. +The lifecycle timeout reproduced delayed automatic joint-leave replication; the +owner now broadcasts the Raft-generated entry through normal Ready persistence. +The same lifecycle assertions pass. Diagnostic logs, including the timeout +backtrace, remain alongside the final committed-source gate. Preflight also caught +the changed storage-engine anchor; review pinned its new validation to the tested +revision. Its initial refusal and passing rerun are retained; no gate was weakened. + +**Reflection and next step:** fresh learner 4 is deliberately not started in this +slice. The feature-critical gap is a seed/bootstrap capability tied to committed +learner membership, its permanent node/disk claim and a complete verified root. +Implement actual new-host installation/catch-up/promotion, then replace a voter +after deleting its disk. Metadata reopen alone is not physical recovery or serving +permission. Prove one-voter-loss tolerance and minority refusal and complete all +five declared CG-05 cases. Remaining receiver crash/sync-error and controller +pressure/fairness cuts stay required; they do not replace fresh-member recovery +as the next feature priority. Then deliver CG-06 verified archive/reclamation, +CG-08 effects/caches and CG-07 public SDK/fleet cutover. No shared-group RustFS, +load, power-loss or zero-quorum recovery qualification is claimed by this gate. + ## CG-05 automatic checkpoints at controller apply boundaries Qualified source: `c27dfb416784804b30a38b1a90812c607ed3b00f`. All 230 selected Rust cases and 8 From 0c3769eccbb845a5943ef60e771c107f1800d9f2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 19:11:09 -0700 Subject: [PATCH 133/568] feat: authorize whole-group learner seeds with fresh quorum --- .../raft/cell_groups/checkpoint/mod.rs | 2 + .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/owner.rs | 41 +- .../raft/cell_groups/runtime/mod.rs | 55 ++- .../raft/cell_groups/runtime/reads.rs | 72 +++- .../raft/cell_groups/runtime/worker.rs | 31 +- .../raft/cell_groups/seed_authority.rs | 212 ++++++++++ .../tests/native_cell_groups_execution.rs | 15 + .../tests/support/cell_group_membership.rs | 10 +- .../tests/support/cell_group_runtime.rs | 2 + .../cellar/tests/support/cell_group_seed.rs | 383 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 4 + .../multi-raft/cell-groups/protocols.md | 22 +- docs/design/multi-raft/cell-groups/tasks.json | 3 +- 14 files changed, 806 insertions(+), 48 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/seed_authority.rs create mode 100644 crates/cellar/tests/support/cell_group_seed.rs diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 66571e9ec..3a8d55d2d 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -32,6 +32,7 @@ const STREAM_WORKSPACE: usize = EXTENT_BYTES as usize + 1024 * 1024; #[derive(Debug)] pub enum CellGroupCheckpointError { + Authority(crate::raft::completion::RuntimeError), Invalid(&'static str), Limit, Busy, @@ -51,6 +52,7 @@ impl std::fmt::Display for CellGroupCheckpointError { impl std::error::Error for CellGroupCheckpointError { fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { match self { + Self::Authority(error) => Some(error), Self::Manifest(error) => Some(error), Self::Log(error) => Some(error), Self::Image(error) => Some(error), diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 488ae5a83..3557da189 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -59,6 +59,8 @@ mod pins; mod proposal_admission; mod read_authority; pub use read_authority::{CellReadAuthority, CellReadConsistency}; +mod seed_authority; +pub use seed_authority::{CellGroupSeedAuthority, CellGroupSeedSource}; mod retention; mod root; mod runtime; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index c88e64516..6b065a782 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -19,7 +19,7 @@ use crate::raft::{ use ::raft::{prelude::*, Config, LightReady, RawNode, Ready, StateRole, Storage}; use cellar_logic::{ cell_group::*, - consensus::{LogIndex, LogPosition, OperationId, Term}, + consensus::{LogIndex, LogPosition, OperationId, ReplicaIdentity, Term}, }; use protobuf::Message as _; use std::{ @@ -382,6 +382,45 @@ impl CellGroupOwner { }) } + pub(super) fn check_seed_target( + &self, + target: &ReplicaIdentity, + ) -> Result<(), RuntimeError> { + self.live()?; + super::seed_authority::check_target(&self.group, target).map(|_| ()) + } + + pub(super) fn select_seed( + &self, + target: &ReplicaIdentity, + term: Option, + barrier: Option, + ) -> Result { + self.live()?; + let status = self.status()?; + if self.node.raft.state != raft::StateRole::Leader || Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if !status.ready + || status.applied.index < index + || self.current.marker().membership_revision() > index.get() + || super::seed_authority::check_target(&self.group, target)? > index + { + return Err(RuntimeError::NotReady); + } + Ok(super::seed_authority::SeedSelection { + group: self.group.clone(), + root: self.current.clone(), + target: target.clone(), + barrier: LogPosition { + term: status.term, + index, + }, + leader: self.leader_term.clone(), + }) + } + pub fn enrollment_state( &self, cell: CellHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 84321fbae..7c1f7d003 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -132,6 +132,7 @@ enum Response { Reconciliation(Box), Status(CellGroupStatus), Read(Box), + Seed(Box), } struct Request { @@ -426,6 +427,49 @@ impl CellGroupRuntime { consistency: CellReadConsistency, deadline: u64, ) -> Result { + match self + .select_authority(history, reads::ReadKind::Cell(cell, consistency), deadline) + .await? + { + Response::Read(authority) => { + authority.check( + self.inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)?, + )?; + Ok(*authority) + } + _ => Err(RuntimeError::Protocol("cell read authority reply".into())), + } + } + + /// Authorize one complete-group seed from a fresh quorum ReadIndex. Only a + /// committed, node/disk-bound new learner is eligible. This source authority + /// does not admit the destination, authenticate a peer, or grant a vote. + pub async fn seed_authority( + &self, + target: cellar_logic::consensus::ReplicaIdentity, + deadline: u64, + ) -> Result { + match self + .select_authority(target.scope, reads::ReadKind::Seed(target), deadline) + .await? + { + Response::Seed(authority) => { + authority.check()?; + Ok(*authority) + } + _ => Err(RuntimeError::Protocol("group seed authority reply".into())), + } + } + + async fn select_authority( + &self, + history: GroupHistory, + kind: reads::ReadKind, + deadline: u64, + ) -> Result { self.admit()?; if self .inner @@ -442,8 +486,7 @@ impl CellGroupRuntime { .acquire(4096) .ok_or(RuntimeError::Overloaded)?; let request = reads::ReadRequest { - cell, - consistency, + kind, deadline, reservation, }; @@ -454,13 +497,7 @@ impl CellGroupRuntime { .map_err(|_| RuntimeError::Closed)?; crate::asyncrt::select! { result = self.call(Some(history), Command::Read(Box::new(request)), 4096) => { - match result? { - Response::Read(authority) => { - authority.check(self.inner.domain.mono_ms().map_err(|_| RuntimeError::Closed)?)?; - Ok(*authority) - } - _ => Err(RuntimeError::Protocol("cell read authority reply".into())), - } + result } _ = timer => Err(RuntimeError::DeadlineExceeded), } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs index de75fb687..4638a8c6f 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -3,6 +3,7 @@ //! Bounded invocation-owned reads. No pending read retains peer/apply capacity. use super::super::read_authority::ReadSelection; +use super::super::seed_authority::{CellGroupSeedAuthority, SeedSelection}; use super::*; use cellar_logic::{ cell_group::CellHistory, @@ -17,12 +18,30 @@ pub(in crate::raft::cell_groups) fn reserved_context(context: &[u8]) -> bool { } pub(super) struct ReadRequest { - pub cell: CellHistory, - pub consistency: CellReadConsistency, + pub kind: ReadKind, pub deadline: u64, pub reservation: Permit, } +pub(super) enum ReadKind { + Cell(CellHistory, CellReadConsistency), + Seed(cellar_logic::consensus::ReplicaIdentity), +} + +impl ReadKind { + pub fn linearizable(&self) -> bool { + matches!( + self, + Self::Seed(_) | Self::Cell(_, CellReadConsistency::Linearizable) + ) + } +} + +pub(super) enum Selection { + Cell(ReadSelection), + Seed(SeedSelection), +} + struct Pending { history: GroupHistory, request: ReadRequest, @@ -71,13 +90,18 @@ impl Reads { if now >= request.deadline { return Err(RuntimeError::DeadlineExceeded); } - if !history.same_namespace(&request.cell) { - return Err(RuntimeError::Invalid("cell read namespace".into())); - } - if let CellReadConsistency::AtLeast(minimum) = &request.consistency { - if minimum.history != request.cell { - return Err(RuntimeError::Invalid("cell read version history".into())); + match &request.kind { + ReadKind::Cell(cell, consistency) => { + if !history.same_namespace(cell) { + return Err(RuntimeError::Invalid("cell read namespace".into())); + } + if let CellReadConsistency::AtLeast(minimum) = consistency { + if &minimum.history != cell { + return Err(RuntimeError::Invalid("cell read version history".into())); + } + } } + ReadKind::Seed(target) => owner.check_seed_target(target)?, } self.groups.retain(|_, budget| budget.strong_count() != 0); let budget = self @@ -94,7 +118,7 @@ impl Reads { let mut context = PREFIX.to_vec(); context.extend_from_slice(&self.nonce); context.extend_from_slice(&self.sequence.to_be_bytes()); - let term = if matches!(request.consistency, CellReadConsistency::Linearizable) { + let term = if request.kind.linearizable() { let status = owner.status()?; if !status.ready { return Err(RuntimeError::NotReady); @@ -141,12 +165,13 @@ impl Reads { pub fn progress( &mut self, now: u64, + domain: &crate::asyncrt::DomainToken, mut select: impl FnMut( GroupHistory, &ReadRequest, Option, Option, - ) -> Result, + ) -> Result, ) -> bool { let mut made = false; self.pending.retain(|_, pending| { @@ -179,16 +204,23 @@ impl Reads { .pending .remove(&context) .expect("selected pending read"); - let result = result - .and_then(|selection| { - CellReadAuthority::new( - selection, - pending.request.deadline, - pending.request.reservation, - pending.group, - ) - }) - .map(|authority| Response::Read(Box::new(authority))); + let result = result.and_then(|selection| match selection { + Selection::Cell(selection) => CellReadAuthority::new( + selection, + pending.request.deadline, + pending.request.reservation, + pending.group, + ) + .map(|authority| Response::Read(Box::new(authority))), + Selection::Seed(selection) => CellGroupSeedAuthority::new( + selection, + pending.request.deadline, + domain.clone(), + pending.request.reservation, + pending.group, + ) + .map(|authority| Response::Seed(Box::new(authority))), + }); let _ = pending.reply.send(result); made = true; } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 18f2d144e..a0242729b 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -373,7 +373,7 @@ impl Worker { } = request; let mut answer = Some(answer); let new_read = matches!(command, Command::ReadIndex(_)) - || matches!(&command, Command::Read(request) if matches!(request.consistency, CellReadConsistency::Linearizable)) + || matches!(&command, Command::Read(request) if request.kind.linearizable()) || matches!(&command, Command::Step(m) if m.get_msg_type()==::raft::prelude::MessageType::MsgReadIndex); let result = if new_read && self.reads >= self.options.max_pending_reads { Err(RuntimeError::Overloaded) @@ -695,17 +695,24 @@ impl Worker { } let mut made = self.progress()? | self.start_batch()? | self.flush(); let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; - made |= self - .cell_reads - .progress(now, |history, request, term, barrier| { - let group = self.groups.get(&history).ok_or(RuntimeError::NotFound)?; - if let Some(error) = &group.failed { - return Err(error.clone()); - } - group - .owner - .select_read(&request.cell, &request.consistency, term, barrier) - }); + made |= + self.cell_reads + .progress(now, &self.domain, |history, request, term, barrier| { + let group = self.groups.get(&history).ok_or(RuntimeError::NotFound)?; + if let Some(error) = &group.failed { + return Err(error.clone()); + } + match &request.kind { + reads::ReadKind::Cell(cell, consistency) => group + .owner + .select_read(cell, consistency, term, barrier) + .map(reads::Selection::Cell), + reads::ReadKind::Seed(target) => group + .owner + .select_seed(target, term, barrier) + .map(reads::Selection::Seed), + } + }); let deadline = if self.retry_at > now { tick_at.min(self.retry_at) } else { diff --git a/crates/cellar/durability/raft/cell_groups/seed_authority.rs b/crates/cellar/durability/raft/cell_groups/seed_authority.rs new file mode 100644 index 000000000..cad987ebb --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/seed_authority.rs @@ -0,0 +1,212 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh quorum authorization for a complete-group learner seed. Destination +//! authentication, permanent claims and installation are separate boundaries. + +use super::{ + CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, + CellGroupCheckpointPool, GroupManifest, PreparedGroupManifest, +}; +use crate::{ + asyncrt::DomainToken, + raft::{budget::Permit, cache::GroupLog, completion::RuntimeError}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{LogIndex, LogPosition, ReplicaIdentity}, +}; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +pub(super) struct SeedSelection { + pub group: GroupLog, + pub root: PreparedGroupManifest, + pub target: ReplicaIdentity, + pub barrier: LogPosition, + pub leader: Arc, +} + +/// Reject birth members, retired IDs, other disks, and members in either voter +/// set. A joining learner has one permanent physical binding in the applied log. +pub(super) fn check_target( + group: &GroupLog, + target: &ReplicaIdentity, +) -> Result { + let cache = group.read()?; + let local = group.identity(); + let id = target.member.get(); + if target.scope != local.scope + || target.node == local.node + || !cache.conf_state.learners.contains(&id) + || cache.conf_state.voters.contains(&id) + || cache.conf_state.voters_outgoing.contains(&id) + || cache.conf_state.learners_next.contains(&id) + || !cache.membership.permits_group(target) + { + return Err(RuntimeError::Invalid("group seed target".into())); + } + let (binding, joined_at) = cache + .membership + .joined(target.member)? + .ok_or_else(|| RuntimeError::Invalid("group seed requires a joined learner".into()))?; + if binding.node != target.node || binding.disk != target.disk { + return Err(RuntimeError::Invalid("group seed member binding".into())); + } + Ok(joined_at) +} + +/// One non-cloneable, deadline-bound ReadIndex result. Holding this value pins a +/// complete applied root and charges the existing node and group read budgets. +/// Capture and stream on an admitted blocking owner, retaining it through actual +/// completion even when a network waiter disconnects. +pub struct CellGroupSeedAuthority { + selection: SeedSelection, + deadline: u64, + domain: DomainToken, + _node: Permit, + _group: Permit, +} + +impl CellGroupSeedAuthority { + pub(super) fn new( + selection: SeedSelection, + deadline: u64, + domain: DomainToken, + mut node: Permit, + mut group: Permit, + ) -> Result { + let bytes = selection + .root + .reserved_bytes() + .checked_add(4096) + .ok_or(RuntimeError::Overloaded)?; + if !node.grow_to(bytes) || !group.grow_to(bytes) { + return Err(RuntimeError::Overloaded); + } + let authority = Self { + selection, + deadline, + domain, + _node: node, + _group: group, + }; + authority.check()?; + Ok(authority) + } + + pub fn target(&self) -> &ReplicaIdentity { + &self.selection.target + } + + pub fn barrier(&self) -> LogPosition { + self.selection.barrier + } + + pub fn deadline(&self) -> u64 { + self.deadline + } + + /// Leadership, applied membership and deadline remain live constraints. + /// This check cannot replace the fresh ReadIndex that created the value. + pub fn check(&self) -> Result<(), RuntimeError> { + if self.domain.mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let s = &self.selection; + let cache = s.group.read()?; + let marker = s.root.marker(); + let local = s.group.identity(); + let term = s.barrier.term.get(); + if s.leader.load(Ordering::Acquire) != term + || cache.hard_state.term != term + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != marker.route()) + || cache.membership.revision().get() != marker.membership_revision() + || !cache.membership.permits_group(local) + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == local.member.get()) + { + return Err(RuntimeError::NotReady); + } + drop(cache); + if check_target(&s.group, &s.target)? > s.barrier.index { + return Err(RuntimeError::NotReady); + } + Ok(()) + } + + /// Capture the exact selected root. If apply advanced before capture, refuse + /// and obtain a fresh authority. Ordinary writes after capture may continue; + /// the admitted source retains its immutable cut. Membership changes revoke it. + pub fn capture( + self, + pool: &CellGroupCheckpointPool, + ) -> Result { + self.check().map_err(CellGroupCheckpointError::Authority)?; + let checkpoint = pool.capture(&self.selection.group, &self.selection.root)?; + self.check().map_err(CellGroupCheckpointError::Authority)?; + Ok(CellGroupSeedSource { + authority: self, + checkpoint, + }) + } +} + +/// A one-shot complete checkpoint bound to one joining process and fresh quorum +/// barrier. Neither its metadata nor a completed stream grants installation or +/// voting authority. A receiver must require an authenticated successful End. +pub struct CellGroupSeedSource { + authority: CellGroupSeedAuthority, + checkpoint: CellGroupCheckpoint, +} + +impl CellGroupSeedSource { + pub fn target(&self) -> &ReplicaIdentity { + self.authority.target() + } + + pub fn barrier(&self) -> LogPosition { + self.authority.barrier() + } + + pub fn marker(&self) -> &GroupManifest { + self.checkpoint.marker() + } + + pub fn check(&self) -> Result<(), RuntimeError> { + self.authority.check() + } + + pub fn snapshot(&self) -> Result { + self.check().map_err(CellGroupCheckpointError::Authority)?; + self.checkpoint.snapshot() + } + + /// No lock is held across the sink. A revoked or expired stream cannot report + /// success, including when the last sink call stalls beyond the deadline. + pub fn stream( + &mut self, + mut sink: impl FnMut(CellGroupCheckpointChunk<'_>) -> Result<(), CellGroupCheckpointError>, + ) -> Result<(), CellGroupCheckpointError> { + self.check().map_err(CellGroupCheckpointError::Authority)?; + let authority = &self.authority; + self.checkpoint.stream(|chunk| { + authority + .check() + .map_err(CellGroupCheckpointError::Authority)?; + sink(chunk)?; + authority + .check() + .map_err(CellGroupCheckpointError::Authority) + })?; + self.check().map_err(CellGroupCheckpointError::Authority) + } +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 8824c937f..1d5d14c9a 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,21 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_materialized_requires_quorum_and_bound_membership() -> Result<()> { + runtime_fixture::seed_source(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_extents_require_quorum_and_bound_membership() -> Result<()> { + runtime_fixture::seed_source(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_group_refuses_isolated_leader_and_expired_stream() -> Result<()> { + runtime_fixture::seed_minority().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn membership_group_materialized_joint_changes_preserve_cells() -> Result<()> { runtime_fixture::membership_lifecycle(false).await diff --git a/crates/cellar/tests/support/cell_group_membership.rs b/crates/cellar/tests/support/cell_group_membership.rs index f6bf42e17..e48f61898 100644 --- a/crates/cellar/tests/support/cell_group_membership.rs +++ b/crates/cellar/tests/support/cell_group_membership.rs @@ -9,7 +9,11 @@ use raft::{ Storage, }; -fn change(member: u64, kind: ConfChangeType, transition: ConfChangeTransition) -> ConfChangeV2 { +pub(super) fn change( + member: u64, + kind: ConfChangeType, + transition: ConfChangeTransition, +) -> ConfChangeV2 { ConfChangeV2 { changes: vec![ConfChangeSingle { node_id: member, @@ -22,7 +26,7 @@ fn change(member: u64, kind: ConfChangeType, transition: ConfChangeTransition) - } } -async fn configure_at( +pub(super) async fn configure_at( fixture: &RuntimeCluster, leader: usize, route: &GroupRoute, @@ -55,7 +59,7 @@ async fn configure_at( Ok(accepted) } -async fn configure( +pub(super) async fn configure( fixture: &RuntimeCluster, route: &GroupRoute, command: ConfChangeV2, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index a33e1fe2c..81457d5b6 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -811,3 +811,5 @@ pub(super) use sql::{ pub(super) use sql::automatic_checkpoints; pub(super) use sql::{membership_lifecycle, membership_sync}; + +pub(super) use sql::{seed_minority, seed_source}; diff --git a/crates/cellar/tests/support/cell_group_seed.rs b/crates/cellar/tests/support/cell_group_seed.rs new file mode 100644 index 000000000..91f3be15a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed.rs @@ -0,0 +1,383 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Live-quorum seed source boundaries; no fresh destination is admitted here. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::ReplicaIdentity; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; + +fn target(history: GroupHistory) -> Result> { + Ok(ReplicaIdentity { + scope: history, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + member: MemberId::new(4)?, + process: ProcessGeneration::new([24; 16])?, + }) +} + +async fn join(fixture: &RuntimeCluster, route: &GroupRoute) -> Result<()> { + let target = target(route.history)?; + membership::configure( + fixture, + route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: target.member, + node: target.node, + disk: target.disk, + }], + ) + .await?; + Ok(()) +} + +async fn authorize( + fixture: &RuntimeCluster, + history: GroupHistory, +) -> Result { + Ok(fixture.runtimes[0] + .seed_authority(target(history)?, reads::deadline(fixture, 10_000).await?) + .await?) +} + +async fn capture( + fixture: &RuntimeCluster, + history: GroupHistory, + pool: &CellGroupCheckpointPool, +) -> Result { + let authority = authorize(fixture, history).await?; + let pool = pool.clone(); + Ok(fixture + .cluster + .domain + .blocking(move || authority.capture(&pool))? + .await??) +} + +pub(crate) async fn seed_source(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let original = group.group_manifest()?.context("root")?; + assert!(fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await + .is_err()); + join(&fixture, &route).await?; + for wrong_disk in [false, true] { + let mut invalid = target(route.history)?; + if wrong_disk { + invalid.disk = DiskLineage::new([99; 16])?; + } else { + invalid.node = NodeId::new([99; 16])?; + } + assert!(fixture.runtimes[0] + .seed_authority(invalid, reads::deadline(&fixture, 1000).await?) + .await + .is_err()); + } + let mut voter = target(route.history)?; + voter.member = MemberId::new(1)?; + voter.node = NodeId::new([1; 16])?; + voter.disk = DiskLineage::new([11; 16])?; + assert!(fixture.runtimes[0] + .seed_authority(voter, reads::deadline(&fixture, 1000).await?) + .await + .is_err()); + assert!(matches!( + fixture.runtimes[1] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::NotReady | RuntimeError::NotLeader) + )); + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = capture(&fixture, route.history, &pool).await?; + assert_eq!(source.target(), &target(route.history)?); + assert_eq!(source.marker().root(), original.root()); + assert!(source.marker().position().index >= source.barrier().index); + assert!(source.barrier().index >= group.membership()?.revision()); + let expected = source.snapshot()?; + let receiver_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let applier = fixture.appliers[&(2, route.history)].1.clone(); + let scratch = super::super::checkpoint_receive::scratch()?; + let directory = fixture.cluster._root.path().to_path_buf(); + let prepared = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut receiver = receiver_pool.receive(applier, scratch, directory)?; + source.stream(|chunk| receiver.append(chunk))?; + assert!(source.stream(|_| Ok(())).is_err(), "one-shot stream"); + Ok(receiver.finish()?) + })? + .await??; + assert_eq!(prepared.snapshot(), &expected); + assert_eq!(prepared.marker().root(), original.root()); + drop(prepared); + assert_eq!(pool.usage().sources, 0); + + // A captured complete cut may stream while later cell enrollment advances + // the group. An authority not yet captured must instead refuse that old cut. + let old = authorize(&fixture, route.history).await?; + let mut captured = capture(&fixture, route.history, &pool).await?; + let before = captured.marker().clone(); + let cell = history(&route, "after-seed-capture")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(&fixture, route.history, accepted.position.index).await?; + let stale_pool = pool.clone(); + assert!(fixture + .cluster + .domain + .blocking(move || old.capture(&stale_pool))? + .await? + .is_err()); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + captured.stream(|chunk| { + if let CellGroupCheckpointPart::Materialized(cell) + | CellGroupCheckpointPart::ExtentManifest(cell) = chunk.part + { + assert_ne!(cell.as_str(), "after-seed-capture"); + } + Ok(()) + })?; + assert_eq!(captured.marker(), &before); + Ok(()) + })? + .await??; + + // A fresh ReadIndex also works without a cell; seeding cannot be implemented + // by choosing an arbitrary enrolled cell or replaying its cached observation. + let empty_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("empty group")? + .route() + .clone(); + join(&fixture, &empty_route).await?; + let mut empty = capture(&fixture, second, &pool).await?; + assert_eq!(empty.marker().root().cells(), 0); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let mut header = 0; + empty.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + header += chunk.bytes.len(); + Ok(()) + })?; + assert!(header > 0); + Ok(()) + })? + .await??; + + // Source authority shares the existing per-group read limit; another group's + // progress and fresh barrier remain available when this group fills it. + let mut held = Vec::new(); + for _ in 0..32 { + held.push(authorize(&fixture, route.history).await?); + } + assert!(matches!( + fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await, + Err(RuntimeError::Overloaded) + )); + drop(authorize(&fixture, second).await?); + drop(held); + let revoked = authorize(&fixture, route.history).await?; + let mut streaming = capture(&fixture, route.history, &pool).await?; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, hold) = std::sync::mpsc::sync_channel(1); + let worker = fixture.cluster.domain.blocking(move || { + let mut entered = Some(entered); + streaming.stream(|_| { + let _ = entered + .take() + .expect("only the first chunk is allowed") + .send(()); + hold.recv_timeout(Duration::from_secs(10)) + .map_err(|e| CellGroupCheckpointError::Sink(std::io::Error::other(e)))?; + Ok(()) + }) + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(pool.usage().sources, 1); + membership::configure( + &fixture, + &route, + membership::change(4, ConfChangeType::RemoveNode, ConfChangeTransition::Auto), + &[], + ) + .await?; + assert!(revoked.check().is_err()); + assert_eq!( + pool.usage().sources, + 1, + "revocation cannot release a blocked physical owner" + ); + release.send(())?; + let refused = worker.await?; + assert_eq!(pool.usage().sources, 0); + assert!(matches!( + revoked.capture(&pool), + Err(CellGroupCheckpointError::Authority(_)) + )); + assert!(matches!( + refused, + Err(CellGroupCheckpointError::Authority(_)) + )); + assert!(fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 1000).await? + ) + .await + .is_err()); + + let revoked = capture(&fixture, second, &pool).await?; + controller::wait_for( + async || match fixture.runtimes[0].transfer_leader(second, 2).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(e) => Err(e.into()), + }, + ) + .await?; + assert!(revoked.check().is_err()); + controller::wait_for(async || Ok(fixture.runtimes[1].status(second).await?.ready)).await?; + let new = fixture.runtimes[1] + .seed_authority(target(second)?, reads::deadline(&fixture, 1000).await?) + .await?; + assert!(new.barrier().term > revoked.barrier().term); + drop(new); + drop(revoked); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} + +pub(crate) async fn seed_minority() -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(false, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + join(&fixture, &route).await?; + let second_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second")? + .route() + .clone(); + join(&fixture, &second_route).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // A timeout during the final sink call must refuse End, even if the group + // remains leader and there are no further chunks at which to notice expiry. + let authority = fixture.runtimes[0] + .seed_authority(target(second)?, reads::deadline(&fixture, 500).await?) + .await?; + let capture_pool = pool.clone(); + let expired = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut source = authority.capture(&capture_pool)?; + assert_eq!(source.marker().root().cells(), 0); + let result = source.stream(|chunk| { + assert_eq!(chunk.part, CellGroupCheckpointPart::Header); + assert_eq!(chunk.offset, 0); + assert_eq!( + chunk.length, + chunk.bytes.len() as u64, + "only the final header chunk" + ); + std::thread::sleep(Duration::from_millis(600)); + Ok(()) + }); + Ok(result) + })? + .await??; + assert!(matches!( + expired, + Err(CellGroupCheckpointError::Authority( + RuntimeError::DeadlineExceeded + )) + )); + // Drain an earlier successful barrier, then isolate this group on all routes. + // The runtime still reports ready because fixture clocks do not auto-tick; + // a fresh seed must prove quorum instead of trusting that cached status. + drop(authorize(&fixture, route.history).await?); + for transport in &fixture.transports { + transport.remove_physical_group(&route.history)?; + } + assert!(fixture.runtimes[0].status(route.history).await?.ready); + for _ in 0..3 { + assert!(matches!( + fixture.runtimes[0] + .seed_authority( + target(route.history)?, + reads::deadline(&fixture, 100).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + } + let authority = authorize(&fixture, second).await?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes( + &second_route, + &history(&second_route, "healthy")?, + &images::image(None, "CREATE TABLE value(n);")?, + false, + )?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + for controller in &controllers { + controller.close().await?; + } + assert!( + authority.check().is_err(), + "shutdown revokes a selected source" + ); + drop(authority); + drop(controllers); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f41ea23c0..cdc6704de 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -599,3 +599,7 @@ pub(crate) use controller_checkpoints::automatic_checkpoints; #[path = "cell_group_membership.rs"] mod membership; pub(crate) use membership::{membership_lifecycle, membership_sync}; + +#[path = "cell_group_seed.rs"] +mod seed; +pub(crate) use seed::{seed_minority, seed_source}; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 7e161b283..815f799f7 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -237,8 +237,26 @@ or demoted leaders lose producer and leader-bound read authority at synced cache publication, even before the owner consumes its completion. Membership alone never allows a learner to produce writes. -Fresh learner seeding, initial-member replacement, lost-disk rebuild and managed -fleet orchestration remain required CG-05 work. Existing bootstrap permits still +`CellGroupRuntime::seed_authority` selects a complete applied root only after a +fresh correlated group ReadIndex. It works without an enrolled cell. The target +must be a committed new learner with the exact node/disk binding, outside both +voter sets. The selected cut covers the barrier and its membership revision. +Pending requests and retained roots share the bounded node/group read budgets. + +The non-cloneable authority captures the exact selected root through the node's +checkpoint pool on a blocking owner. If apply advances before capture, the caller +must obtain a fresh authority. After capture, ordinary cell writes may continue +while the source streams its immutable cut. Membership changes, term changes, +leadership transfer, shutdown and deadline expiry revoke the source. Every sink +call has checks before and after it; the final completion also checks authority. +An incomplete or revoked stream cannot produce a successful End. The source binds +the requested process identity but does not authenticate it; the future seed RPC +must bind that identity to its authenticated connection and process revocation. + +This is a source boundary, not a destination bootstrap permit. Authenticated seed +exchange, permanent destination claims, new-member installation/attachment, +initial-member replacement, lost-disk rebuild and managed fleet orchestration +remain required CG-05 work. Existing bootstrap permits still admit only initial birth members. A metadata reopen is not a fresh-member seed, quorum proof, verified physical recovery or serving permission. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index ffd579a22..2f51e0915 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1587,7 +1587,8 @@ "crates/cellar/tests/support/cell_group_network_probes.rs", "crates/cellar/tests/support/cell_group_controller.rs", "crates/cellar/tests/support/cell_group_controller_checkpoints.rs", - "crates/cellar/tests/support/cell_group_membership.rs" + "crates/cellar/tests/support/cell_group_membership.rs", + "crates/cellar/tests/support/cell_group_seed.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 9185e8cee3c70633c25a5d88825bfdd0259bb93a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 19:30:10 -0700 Subject: [PATCH 134/568] docs: qualify learner seed sources and prioritize destination bootstrap --- docs/design/multi-raft/cell-groups/README.md | 6 ++- docs/design/multi-raft/cell-groups/handoff.md | 30 +++++++++++-- .../design/multi-raft/cell-groups/progress.md | 43 +++++++++++++++++++ 3 files changed, 73 insertions(+), 6 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 1b127db08..f75b14b06 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -9,8 +9,10 @@ CG-05 is in progress: checkpoint capture, receiver verification, atomic storage installation, exact Raft Ready acceptance, local publication and native Raft-generated catch-up, authenticated snapshot RPC and automatic controller checkpoint policy pass their scoped regression gates. Durable joint membership and leadership handoff -also pass a partial gate. Fresh learner bootstrap and lost-disk replacement are the -next feature priorities. The production host has not yet cut over to shared groups. +also pass a partial gate. A learner seed source authorized by a fresh quorum read +now covers empty groups, bounded capture of complete roots and stream revocation. +Authenticated destination bootstrap, catch-up/promotion and lost-disk replacement +are the next feature priorities. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index d2e666474..3bcbcd278 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -41,6 +41,27 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. +The latest seed-source checkpoint at `0c3769ec` passes 236 selected Rust +cases, eight gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-source/` binds 312 source hashes and five +copied executables. `seed_authority` uses a fresh group ReadIndex, including for +empty groups, and binds a complete applied root to a committed joining learner. +Capture requires that exact root; the source then tolerates ordinary writes while +revoking on membership/leadership/shutdown/deadline changes. Read and checkpoint +budgets remain owned through physical work. Both image forms, receiver verification, +stale/invalid targets, bounded admission, held-sink revocation, final-chunk expiry, +isolated ready-leader refusal and independent group progress pass local tests. + +Continue from `cell_groups/seed_authority.rs`, `directory/bootstrap.rs` and the +existing authenticated checkpoint stream/atomic receiver path. The immediate gap is +authenticated seed exchange plus permanent fresh-disk claims and joined-member +installation/attachment. The source API binds but does not authenticate the requested +process and cannot authorize a destination. Catch-up/promotion and deleted-voter-disk +replacement remain unimplemented integration, not qualified by the source tests. +Coordinate capture with an apply boundary or bounded fresh-selection retries; +continuous-write capture fairness remains unqualified. Keep all five CG-05 cases +and shared RustFS/load/crash gates open. + The receiver checkpoint at `cb164e02` passes 226 selected Rust cases, 9 gate commands and 37 validator tests. Its evidence binds 295 source hashes and 6 copied executables under @@ -682,8 +703,9 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Finish CG-05 fresh learner seeding and lost-disk replacement. Bind a seed - capability to committed learner membership, permanent node/disk identity and +1. Finish CG-05 fresh learner seeding and lost-disk replacement. Reuse the + implemented fresh-ReadIndex source authority; authenticate the seed exchange + and bind destination installation to permanent node/disk/engine claims and the complete verified checkpoint. Existing bootstrap permits admit only birth members; do not weaken them to accept an untrusted metadata observation. Bring up the new learner through production transport, install every cell, catch up, @@ -711,8 +733,8 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Continue from `cell_groups/membership.rs`, the owner/runtime configuration path, -`directory/bootstrap.rs` and the existing checkpoint receiver/installation path. +Continue from `cell_groups/seed_authority.rs`, the authenticated checkpoint +transport, `directory/bootstrap.rs` and the atomic checkpoint installation path. Keep configuration, membership revision and complete root atomic. A captured metadata observation, transport admission reply or lost response cannot create new serving authority. Reuse permanent member claims and the node checkpoint pool diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 6b4b6a0fa..2aabcbe80 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,49 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 live-quorum learner seed source + +Qualified source: `0c3769eccbb845a5943ef60e771c107f1800d9f2`. All 236 selected Rust cases and eight +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. `.cellar/cell-groups-evidence/CG-05-seed-source/checkpoint.json` +binds 312 source hashes, five copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +A fresh correlated group ReadIndex now authorizes a complete applied seed root, +including a group with no cells. Only a committed new learner with the exact +node/disk binding outside both voter sets qualifies. Pending requests and retained +roots share existing node/group read budgets. No per-cell worker or new serving +mode is introduced. Capture consumes a non-cloneable authority and requires the +exact still-current root; if apply advanced first, a fresh selection is required. +Once captured, ordinary writes may continue while the immutable cut streams. +Membership changes, leadership loss/handoff, shutdown and deadline expiry revoke +the source. Checks before and after sink calls and final completion prevent a +revoked stream from reporting a successful End. + +Three new cases use production controllers and real TCP quorum traffic. Both +image forms stream complete cuts into existing receiver verification. Tests cover +uncommitted/wrong/retired member refusal, follower refusal, stale capture, empty +groups, per-group admission exhaustion, independent group progress, leadership +transfer and shutdown. Removing membership while a source blocks in its sink +revokes completion but retains pool admission until physical work exits. Expiry +during an empty group's final header chunk also refuses completion. An isolated +leader still reporting ready cannot issue a fresh seed; another group continues +real writes. Initial and strengthened targeted runs passed and are retained. + +**Reflection and next step:** source authorization is now implemented, but no new +destination is admitted by this API. The requested process identity is bound, not +authenticated. The next feature-critical slice is authenticated seed RPC with +process revocation, permanent destination node/disk/engine claims, complete atomic +installation and joined-member runtime attachment. Coordinate capture with the +controller apply boundary or bound fresh-selection retries under sustained writes; +source refusal is safe but does not establish capture fairness. Then demonstrate +real catch-up, promotion and replacement after deleting a voter disk, one-voter-loss tolerance +and minority write refusal. Preserve all five CG-05 cases and remaining receiver, +joint-transition and pressure/crash cuts. CG-06 verified archive/reclamation follows, +then CG-08 effects/caches and CG-07 public SDK/fleet cutover. Shared-group RustFS, +load, power-loss and zero-quorum DR remain unqualified. + ## CG-05 durable membership and leadership handoff Qualified source: `1814b27896c3c21166186fe8fbd1a1a6bdbb721a`. All 233 selected Rust cases and 8 From ab2a308a937f349d3482a825d4b669d042cb03c3 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 20:35:22 -0700 Subject: [PATCH 135/568] feat: install authenticated cell-group learner seeds Stage exact permanent member and engine claims without startup authority; atomically install a complete verified root and local seed receipt. Retain physical owners through cancellation and reconcile unknown results using the same page store. Exercise deleted-voter-disk replacement, catch-up, promotion, leadership handoff and cold recovery in both image modes. Compare validated canonical configurations after joint leave, since RawNode and pure replay can enumerate equivalent voter sets differently. --- crates/cellar/application/consensus/mod.rs | 4 +- crates/cellar/durability/raft/cache.rs | 20 + .../raft/cell_groups/apply/recovery.rs | 25 +- .../raft/cell_groups/checkpoint/mod.rs | 45 ++ .../raft/cell_groups/directory/bootstrap.rs | 24 +- .../cell_groups/directory/joined_bootstrap.rs | 152 +++++ .../durability/raft/cell_groups/owner.rs | 11 +- .../raft/group_snapshot_transport/mod.rs | 3 + .../group_snapshot_transport/seed/client.rs | 194 ++++++ .../group_snapshot_transport/seed/grant.rs | 86 +++ .../group_snapshot_transport/seed/install.rs | 136 ++++ .../raft/group_snapshot_transport/seed/mod.rs | 155 +++++ .../group_snapshot_transport/seed/server.rs | 186 +++++ .../group_snapshot_transport/seed/work.rs | 119 ++++ crates/cellar/durability/raft/records.rs | 43 +- .../cellar/durability/raft/snapshot_seed.rs | 1 + crates/cellar/durability/raft/storage.rs | 51 +- crates/cellar/durability/raft/transport.rs | 18 +- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_install.rs | 642 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 4 + 22 files changed, 1902 insertions(+), 29 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs create mode 100644 crates/cellar/tests/support/cell_group_seed_install.rs diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 3710843d5..acca8dbc6 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -211,8 +211,8 @@ pub mod verification { CellGroupBootstrap, GroupBootstrap, LogError, ReplicaLog, SyncReceipt, }; pub use crate::raft::transport::{ - CellGroupCheckpointRegistration, Delivery, GroupRoute, Lane, RaftPeer, RaftTransport, - TransportError, TransportUsage, + CellGroupCheckpointRegistration, CellGroupSeedError, CellGroupSeedSession, Delivery, + GroupRoute, Lane, RaftPeer, RaftTransport, TransportError, TransportUsage, }; pub use crate::raft::transport_options::{ NamespaceCommandOrigin, NamespaceInspectionOrigin, NamespaceQueryOrigin, RaftTls, diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index b4e11d961..5b02f6604 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -106,6 +106,7 @@ pub enum SnapshotChange { #[derive(Clone)] pub struct GroupWrite { pub(crate) consensus_pin: Option>, + pub(crate) seed_grant: Option>, pub group: GroupLog, pub entries: Vec, pub hard_state: Option, @@ -120,6 +121,7 @@ impl GroupWrite { pub fn new(group: &GroupLog) -> Self { Self { consensus_pin: None, + seed_grant: None, group: group.clone(), entries: Vec::new(), hard_state: None, @@ -449,6 +451,8 @@ pub(crate) struct CachedGroup { pub group_manifest: Option, pub manifest_pins: Option, pub checkpoint_manifest: Option, + /// Local receipt; never imported from another node's checkpoint. + pub group_seed: Option, } impl CachedGroup { @@ -506,6 +510,11 @@ impl CachedGroup { } pub fn validate(&self, max_entries: usize) -> Result<(), LogError> { + if self.group_seed.is_some_and(|index| { + index <= 1 || index > self.applied.position.index.get() || self.group_manifest.is_none() + }) { + return Err(LogError::Corrupt("invalid local group seed receipt".into())); + } validate_conf(&self.conf_state)?; self.membership .validate(&self.conf_state, self.applied.position.index.get()) @@ -579,6 +588,16 @@ impl CachedGroup { history: GroupHistory, ) -> Result { let mut next = self.clone(); + if let Some(grant) = &write.seed_grant { + grant.check(&write.group.owner, history, write.group.identity())?; + let Some(SnapshotChange::CellGroup(prepared)) = &write.snapshot else { + return Err(LogError::WrongIdentity); + }; + if prepared.snapshot() != grant.snapshot() || !write.entries.is_empty() { + return Err(LogError::WrongIdentity); + } + next.group_seed = Some(prepared.marker().position().index.get()); + } let group_install = matches!(&write.snapshot, Some(SnapshotChange::CellGroup(_))); let group_checkpoint = matches!( &write.snapshot, @@ -938,6 +957,7 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { group_manifest: None, manifest_pins: None, checkpoint_manifest: None, + group_seed: None, applied: DurableApply { position: LogPosition { index: LogIndex::new(1), diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs index b693a908c..ded56cc04 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -31,13 +31,22 @@ impl CellGroupApplier { limits, max_effect_rows, )?; - let marker = applier + let prepared = applier.recover_selected()?; + Ok((applier, prepared)) + } + + /// Reconcile the current and retained checkpoint closures using this same + /// physical store owner after an unknown installation result. Run on the + /// owning blocking domain. Never reopen a page directory while its selected + /// roots still hold pins. This grants no catalog, quorum or serving authority. + pub fn recover_selected(&self) -> Result { + let marker = self .group .group_manifest() .map_err(storage)? .ok_or(ExecutionError::Invalid("recovery manifest missing"))?; let prepared = { - let mut owner = applier + let mut owner = self .storage .lock() .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; @@ -48,7 +57,7 @@ impl CellGroupApplier { .. } = &mut *owner; tree.prepare_complete( - &applier.group, + &self.group, pages, marker.root(), marker.position(), @@ -61,8 +70,8 @@ impl CellGroupApplier { ) .map_err(storage)? }; - if let Some(checkpoint) = applier.group.checkpoint_manifest().map_err(storage)? { - let mut owner = applier + if let Some(checkpoint) = self.group.checkpoint_manifest().map_err(storage)? { + let mut owner = self .storage .lock() .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; @@ -73,7 +82,7 @@ impl CellGroupApplier { .. } = &mut *owner; tree.prepare_complete( - &applier.group, + &self.group, pages, checkpoint.root(), checkpoint.position(), @@ -86,10 +95,10 @@ impl CellGroupApplier { ) .map_err(storage)?; } - if applier.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) { + if self.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) { return Err(ExecutionError::Invalid("recovery selection changed")); } - Ok((applier, prepared)) + Ok(prepared) } } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 3a8d55d2d..0cb04a60a 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -24,6 +24,51 @@ pub use receive::CellGroupCheckpointReceiver; use sha2::{Digest, Sha256}; use std::sync::Arc; +/// Validate authenticated seed metadata before any destination mapping exists. +pub(crate) fn seed_marker( + snapshot: &raft::prelude::Snapshot, + target: &cellar_logic::consensus::ReplicaIdentity, + barrier: cellar_logic::consensus::LogIndex, +) -> Result { + let header = Header::decode(&snapshot.data)?; + let conf = header.configuration()?; + let id = target.member.get(); + let joined = header.membership.joined(target.member)?; + if header.snapshot()? != *snapshot + || header.root.route().history != target.scope + || barrier.get() == 0 + || header.root.position().index < barrier + || header.membership.revision() > barrier + || !conf.learners.contains(&id) + || conf.voters.contains(&id) + || conf.voters_outgoing.contains(&id) + || conf.learners_next.contains(&id) + || !header.membership.permits_group(target) + || joined.is_none_or(|(binding, at)| { + at > barrier || binding.node != target.node || binding.disk != target.disk + }) + { + return Err(CellGroupCheckpointError::Invalid("seed descriptor/member")); + } + Ok(header.root) +} + +pub(crate) fn seed_birth( + snapshot: &raft::prelude::Snapshot, + route: &cellar_logic::cell_group::GroupRoute, + initial: &raft::prelude::ConfState, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + if header.root.route() != route + || !header + .membership + .same_birth(&GroupMembership::initial(initial)) + { + return Err(CellGroupCheckpointError::Invalid("seed birth")); + } + Ok(()) +} + const HEADER_LIMIT: usize = 64 * 1024; // Covers one verified extent, page/extent metadata encoding, serialization diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs index 600d580c7..1adb179fd 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -1,6 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Initial member admission after real genesis and a permanent physical claim. +//! Member admission after real genesis and a permanent physical claim. use super::*; use crate::raft::{ @@ -10,7 +10,10 @@ use crate::raft::{ use cellar_logic::consensus::MemberId; use serde::{Deserialize, Serialize}; -/// Initial member startup permission, not quorum or application-serving authority. +#[path = "joined_bootstrap.rs"] +mod joined; + +/// Durable member startup permission, not quorum or application-serving authority. /// The shared runtime must verify the complete selected image/manifest closure /// and revalidate this permit before attachment. It must obtain current-term /// quorum authority separately before accepting application work. @@ -19,6 +22,7 @@ pub struct CellGroupBootstrapPermit { birth: CellGroupBirth, directory: RangeDirectory, log: ReplicaLog, + joined: bool, } impl CellGroupBootstrapPermit { @@ -35,10 +39,17 @@ impl CellGroupBootstrapPermit { } pub async fn revalidate(&self) -> Result<(), CatalogError> { - self.directory - .bootstrap_group_member(&self.birth, &self.log) - .await - .map(|_| ()) + if self.joined { + self.directory + .bootstrap_joined_group_member(&self.birth, self.group.identity().member, &self.log) + .await + .map(|_| ()) + } else { + self.directory + .bootstrap_group_member(&self.birth, &self.log) + .await + .map(|_| ()) + } } } @@ -156,6 +167,7 @@ impl RangeDirectory { birth: birth.clone(), directory: self.clone(), log: log.clone(), + joined: false, }) }) .await diff --git a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs new file mode 100644 index 000000000..b9786e3c8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs @@ -0,0 +1,152 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A staging mapping never grants startup. Only a locally fenced authenticated +//! install plus the exact permanent remote claim can resume a joined member. +use super::*; +use crate::raft::transport::CellGroupSeedGrant; + +impl RangeDirectory { + pub(crate) async fn stage_group_learner( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + ) -> Result, CatalogError> { + self.unmanaged_birth()?; + self.bounded(async { + grant.check_log(log).map_err(CatalogError::Log)?; + self.require_current_group_birth(birth).await?; + let _node = self.namespace.require_node_storage(log).await?; + let member = grant.local().member; + let bootstrap = CellGroupBootstrap { + route: birth.route().clone(), + member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }; + grant + .initial_configuration(&bootstrap) + .map_err(CatalogError::Log)?; + let expected = self.joined_claim(birth, member, log)?; + let key = self.group_claim_key(birth.route().history, member); + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.as_ref().is_some_and(|(bytes, _)| bytes != &expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + match log.cell_group(&birth.route().history) { + Err(LogError::NotFound) if remote.is_some() => { + return Err(CatalogError::LostStableState) + } + Err(LogError::NotFound) | Ok(_) => (), + Err(error) => return Err(CatalogError::Log(error)), + } + // Stage before publishing the remote claim. An interrupted operation + // can reconcile this same engine; an existing claim can never reset it. + let group = log + .stage_cell_group(bootstrap, grant.clone()) + .await + .map_err(CatalogError::Log)?; + let observed = self + .namespace + .inner + .create_and_observe_bounded(&key, &expected, wire::LIMIT) + .await?; + if observed != expected { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + grant.check_log(log).map_err(CatalogError::Log)?; + Ok(group) + }) + .await + } + + fn joined_claim( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + ) -> Result, CatalogError> { + let history = birth.route().history; + record::encode( + wire::GROUP_CLAIM, + &Claim { + namespace: self.namespace.root_digest, + group: history.group.bytes(), + incarnation: history.incarnation.bytes(), + birth: birth.route().birth_digest, + member: member.get(), + node: log.limits().node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }, + wire::LIMIT, + ) + } + + /// Resume a previously installed joined member. A staged empty mapping, an + /// imported snapshot or a recreated disk cannot substitute for the local + /// authenticated-install receipt and permanent engine claim. The runtime + /// still verifies the whole physical closure before attaching this permit. + pub async fn bootstrap_joined_group_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.require_current_group_birth(birth).await?; + let _node = self.namespace.require_node_storage(log).await?; + let group = log + .cell_group(&birth.route().history) + .map_err(CatalogError::Log)?; + let check = || -> Result<(), CatalogError> { + let cache = group.read().map_err(CatalogError::Log)?; + let joined = cache.membership.joined(member).map_err(CatalogError::Log)?; + if group.identity().member != member + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != birth.route()) + || joined.is_none_or(|(binding, at)| { + binding.node != log.limits().node + || binding.disk != log.limits().disk + || cache.group_seed.is_none_or(|seed| seed < at.get()) + }) + { + return Err(CatalogError::Conflict( + "joined group has no durable seed receipt", + )); + } + drop(cache); + check_member(&group) + }; + check()?; + let key = self.group_claim_key(birth.route().history, member); + let expected = self.joined_claim(birth, member, log)?; + let remote = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await?; + if remote.is_none_or(|(bytes, _)| bytes != expected) { + return Err(CatalogError::Conflict("permanent group member claim")); + } + self.require_current_group_birth(birth).await?; + check()?; + Ok(CellGroupBootstrapPermit { + group, + birth: birth.clone(), + directory: self.clone(), + log: log.clone(), + joined: true, + }) + }) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 6b065a782..6c3e1cf52 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -1322,7 +1322,16 @@ impl CellGroupOwner { let change = super::membership::decode(entry, self.current.marker().route())?; let conf = self.node.apply_conf_change(&change)?; - if conf != self.group.initial_state()?.conf_state { + // RawNode retains its historical insertion order while + // pure replay restores the persisted configuration first. + // Compare validated sets, including joint/auto-leave + // state, rather than hash iteration order after a move. + let durable = self.group.initial_state()?.conf_state; + if crate::raft::manifest::encode_conf(&conf) + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + != crate::raft::manifest::encode_conf(&durable) + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + { return Err(RuntimeError::StaleCompletion); } self.refresh_execution_fence(); diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs index 441b5754e..baa223aee 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -9,10 +9,12 @@ use crate::raft::cell_groups::{ use std::sync::Weak; mod codec; mod receive; +pub(crate) mod seed; mod send; mod workers; pub(super) use codec::START_MAGIC; use codec::*; +pub use seed::{CellGroupSeedError, CellGroupSeedSession}; pub(super) struct Endpoint { applier: CellGroupApplier, @@ -161,6 +163,7 @@ impl State { fn checkpoint_error(error: CellGroupCheckpointError) -> TransportError { match error { CellGroupCheckpointError::Busy => TransportError::Overloaded, + CellGroupCheckpointError::Authority(error) => TransportError::Runtime(error), CellGroupCheckpointError::Limit => TransportError::Limit, other => TransportError::Invalid(other.to_string()), } diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs new file mode 100644 index 000000000..cb6d54f69 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs @@ -0,0 +1,194 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl RaftTransport { + /// Authenticate a live-quorum source, then reconcile a non-serving staging + /// mapping and permanent claim. Retry failures with a fresh exchange and the + /// same identities; an existing remote claim never permits recreating a disk. + pub async fn prepare_group_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + ) -> Result { + let state = self.handle.0.clone(); + let log = state.runtime.shared()?.log().clone(); + let options = log.limits(); + if local.node != options.node + || local.disk != options.disk + || local.process != options.process + || local.scope != source.replica.scope + || local.scope != birth.route().history + || local.node == source.replica.node + || local.member == source.replica.member + { + return Err(TransportError::Identity.into()); + } + let deadline = deadline.min(self.seed_deadline()?); + let guard = state.reserve_task()?; + let process = state.seed_peer_epoch(&RaftPeerNode::from(&source))?; + let metadata = state + .seed_metadata + .acquire(seeds::META_BYTES) + .ok_or(TransportError::Overloaded)?; + let admission = Arc::new(ReceiveAdmission { + _wire: state + .incoming + .lane(Lane::Data) + .acquire(wire_charge(&state.options)) + .ok_or(TransportError::Overloaded)?, + _transfer: state + .snapshot_receive_budget + .acquire(snapshots::memory_bound(&state.options)) + .ok_or(TransportError::Overloaded)?, + }); + // Refuse before remote or local mutations if the node has no receiver pool. + let pool = self.group_checkpoint_pool()?; + let now = state.domain.mono_ms().map_err(|_| TransportError::Closed)?; + let timeout_ms = deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; + let nonce = state + .sequence + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| n.checked_add(1)) + .map_err(|_| TransportError::Closed)? + .checked_add(1) + .ok_or(TransportError::Closed)?; + let request = Request { + route: Route::new_group( + &local, + state.auth.source(), + &source.replica, + &source.label, + birth.route().schema_digest, + )?, + nonce: nonce.to_string(), + timeout_ms, + }; + let connection = state.outgoing_connections.acquire(Lane::Data); + let mut stream = state + .seed_process_io(&process, deadline, async { + let tcp = TcpStream::connect(source.data_address).await?; + tcp.set_nodelay(true)?; + let socket = state.observed_socket(tcp, metrics::Lane::Data); + let name = rustls::pki_types::ServerName::try_from(source.server_name.clone()) + .map_err(|_| TransportError::Identity)?; + tokio_rustls::TlsConnector::from(state.tls.client.clone()) + .connect(name, socket) + .await + .map_err(|e| TransportError::Tls(e.to_string())) + }) + .await?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], EXPORT_LABEL, None) + .map_err(|e| TransportError::Tls(e.to_string()))?; + let wire = snapshots::Wire { + auth: &state.auth, + exporter: &exporter, + options: &state.options, + max_body: META_BYTES, + }; + let head = wire.encode(REQUEST_MAGIC, REQUEST, &[], &request, &[])?; + let (reply, body, binding) = state + .seed_process_until(&process, deadline, async { + write_head(&mut stream, &head).await?; + stream.flush().await?; + let received = snapshots::read_bulk_head( + &mut stream, + None, + REPLY_MAGIC, + META_BYTES, + &state.options, + ) + .await?; + let reply: Reply = wire.decode(REPLY_MAGIC, REPLY, &received, Some(&head.tag))?; + if reply.route != request.route.reversed() + || reply.request != head.tag + || reply.nonce != request.nonce + { + return Err(TransportError::Authentication); + } + let mut body = vec![0; received.payload_len]; + stream.read_exact(&mut body).await?; + if reply.digest != digest(&body) { + return Err(TransportError::Authentication); + } + Ok((reply, body, received.tag)) + }) + .await?; + let barrier = LogIndex::new(frame::decimal(&reply.barrier)?); + let snapshot = ::raft::prelude::Snapshot::parse_from_bytes(&body) + .map_err(|_| TransportError::Protocol)?; + if snapshot + .write_to_bytes() + .map_err(|_| TransportError::Protocol)? + != body + { + return Err(TransportError::Protocol.into()); + } + let root = checkpoint::seed_marker(&snapshot, &local, barrier)?; + let grant = Arc::new(Grant { + local, + root, + snapshot, + deadline, + owner: log.owner(), + domain: state.domain.clone(), + process, + stop: state.stop.subscribe(), + _metadata: metadata, + _request: Arc::new(guard), + }); + let group = state + .domain + .run(directory.stage_group_learner(&birth, &log, grant.clone())) + .await + .map_err(|_| TransportError::Closed)??; + group + .consensus_claimed + .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) + .map_err(|_| RuntimeError::AlreadyAttached) + .map_err(TransportError::from)?; + let lease = Arc::new(Lease(group.consensus_claimed.clone())); + let (command, ready) = oneshot::channel(); + let (response, result) = oneshot::channel(); + let exchange = Exchange { + state: state.clone(), + stream, + exporter, + binding, + grant, + group: group.clone(), + lease, + directory, + birth, + pool, + admission, + _connection: connection, + operation: OperationId::new(nonce), + }; + let guard = state.reserve_task()?; + state.clone().spawn_reserved(guard, async move { + let outcome = async { + let applier = state + .seed_process_until(&exchange.grant.process, deadline, async { + ready.await.map_err(|_| TransportError::Closed) + }) + .await?; + exchange.install(applier).await + } + .await; + let _ = response.send(outcome); + })?; + Ok(CellGroupSeedSession { + group, + command, + result, + }) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs new file mode 100644 index 000000000..727ffe6ef --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +/// Only a verified TLS response constructs this capability. It remains live +/// through queued physical installation, including cancelled callers. +pub(crate) struct Grant { + pub(super) local: ReplicaIdentity, + pub(super) root: GroupManifest, + pub(super) snapshot: ::raft::prelude::Snapshot, + pub(super) deadline: u64, + pub(super) owner: Arc<()>, + pub(super) domain: DomainToken, + pub(super) process: watch::Receiver, + pub(super) stop: watch::Receiver, + pub(super) _metadata: Permit, + // Erase the transport graph while retaining its joined task ownership. + pub(super) _request: Arc, +} + +impl Grant { + pub(crate) fn local(&self) -> &ReplicaIdentity { + &self.local + } + + pub(crate) fn snapshot(&self) -> &::raft::prelude::Snapshot { + &self.snapshot + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + history: GroupHistory, + local: &ReplicaIdentity, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(owner, &self.owner) + || history != self.local.scope + || local.member != self.local.member + || local.node != self.local.node + || local.disk != self.local.disk + || local.process != self.local.process + || *self.process.borrow() + || *self.stop.borrow() + || !self.domain.mono_ms().is_ok_and(|now| now < self.deadline) + { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(crate) fn check_log(&self, log: &ReplicaLog) -> Result<(), LogError> { + let options = log.limits(); + self.check( + &log.owner(), + self.local.scope, + &ReplicaIdentity { + scope: self.local.scope, + node: options.node, + disk: options.disk, + process: options.process, + member: self.local.member, + }, + ) + } + + pub(crate) fn initial_configuration( + &self, + bootstrap: &CellGroupBootstrap, + ) -> Result<::raft::prelude::ConfState, LogError> { + let conf = ::raft::prelude::ConfState { + voters: bootstrap.voters.iter().map(|id| id.get()).collect(), + learners: bootstrap.learners.iter().map(|id| id.get()).collect(), + ..Default::default() + }; + crate::raft::cache::validate_conf(&conf)?; + crate::raft::member_state::GroupMembership::initial(&conf).validate(&conf, 1)?; + if bootstrap.member != self.local.member + || &bootstrap.route != self.root.route() + || crate::raft::member_state::members(&conf).contains(&bootstrap.member.get()) + { + return Err(LogError::WrongIdentity); + } + checkpoint::seed_birth(&self.snapshot, &bootstrap.route, &conf) + .map_err(|e| LogError::Invalid(e.to_string()))?; + Ok(conf) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs new file mode 100644 index 000000000..27147262f --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs @@ -0,0 +1,136 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +impl Exchange { + pub(super) async fn install( + mut self, + applier: CellGroupApplier, + ) -> Result { + let state = &self.state; + let log = state.runtime.log(); + self.grant.check_log(log)?; + if applier.history() != self.group.identity().scope + || !applier.belongs_to_engine(&log.owner()) + { + return Err(TransportError::Identity.into()); + } + let receiver = work::receiver( + state, + applier, + self.pool.clone(), + self.admission.clone(), + self.grant.clone(), + )?; + let wire = snapshots::Wire { + auth: &state.auth, + exporter: &self.exporter, + options: &state.options, + max_body: CHUNK_BYTES, + }; + let head = wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 0 }, &[])?; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + write_head(&mut self.stream, &head).await?; + self.stream.flush().await?; + Ok(()) + }) + .await?; + let mut sequence = 0_u64; + loop { + self.grant.check_log(log)?; + let head = state + .seed_process_io( + &self.grant.process, + self.grant.deadline, + snapshots::read_bulk_head( + &mut self.stream, + None, + CHUNK_MAGIC, + CHUNK_BYTES.min(state.options.snapshot_chunk_bytes), + &state.options, + ), + ) + .await?; + let packet: Packet = wire.decode(CHUNK_MAGIC, CHUNK, &head, Some(&self.binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay.into()); + } + let mut bytes = vec![0; head.payload_len]; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + self.stream.read_exact(&mut bytes).await?; + Ok(()) + }) + .await?; + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication.into()); + } + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol.into()); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) + } + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol.into()); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.grant.check_log(log)?; + state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + drop(receiver.input); + let prepared = state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + receiver.result.await.map_err(|_| TransportError::Closed)? + }) + .await?; + self.grant.check_log(log)?; + if prepared.snapshot() != self.grant.snapshot() { + return Err(TransportError::Authentication.into()); + } + let current = prepared.manifest().clone(); + let mut write = GroupWrite::new(&self.group); + let metadata = prepared.snapshot().get_metadata(); + write.hard_state = Some(::raft::prelude::HardState { + term: metadata.term, + commit: metadata.index, + ..Default::default() + }); + write.snapshot = Some(SnapshotChange::CellGroup(prepared)); + write.seed_grant = Some(self.grant.clone()); + write.consensus_pin = Some(self.lease.clone()); + // Once accepted, retain the network task, grant and consensus lease until + // actual persistence finishes. Caller cancellation/transport close cannot + // turn an in-flight fence into an unowned or successful empty bootstrap. + log.persist_cells(self.operation, vec![write]).await?; + let permit = self + .directory + .bootstrap_joined_group_member(&self.birth, self.grant.local.member, log) + .await?; + drop(self.lease); + Ok((permit, current)) + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs new file mode 100644 index 000000000..b7146292d --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs @@ -0,0 +1,155 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Pull one authenticated complete-group seed into a non-serving staged member. +use super::*; +use crate::raft::{ + cache::{GroupLog, GroupWrite, SnapshotChange}, + catalog::{CatalogError, CellGroupBirth, CellGroupBootstrapPermit, RangeDirectory}, + cell_groups::{checkpoint, CellGroupSeedAuthority, GroupManifest, PreparedGroupManifest}, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{LogIndex, OperationId}; +mod client; +mod grant; +mod install; +mod server; +mod work; +pub(crate) use grant::Grant; + +pub(in crate::raft::transport) const REQUEST_MAGIC: &[u8; 4] = b"CLGJ"; +pub(in crate::raft::transport) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-group-seed-1"; +const REPLY_MAGIC: &[u8; 4] = b"CLGK"; +const READY_MAGIC: &[u8; 4] = b"CLGL"; +const CHUNK_MAGIC: &[u8; 4] = b"CLGM"; +const REQUEST: &[u8] = b"cellar-group-seed-request-1"; +const REPLY: &[u8] = b"cellar-group-seed-grant-1"; +const READY: &[u8] = b"cellar-group-seed-ready-1"; +const CHUNK: &[u8] = b"cellar-group-seed-chunk-1"; +const META_BYTES: usize = 64 * 1024; + +type Stream = tokio_rustls::client::TlsStream>; +type Installed = (CellGroupBootstrapPermit, PreparedGroupManifest); + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Request { + route: Route, + nonce: String, + timeout_ms: u64, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Reply { + route: Route, + request: [u8; 32], + nonce: String, + barrier: String, + digest: [u8; 32], +} + +#[derive(Debug)] +pub enum CellGroupSeedError { + Transport(TransportError), + Catalog(CatalogError), + Log(LogError), + Checkpoint(CellGroupCheckpointError), +} + +impl std::fmt::Display for CellGroupSeedError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group seed: {self:?}") + } +} + +impl std::error::Error for CellGroupSeedError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Transport(e) => e, + Self::Catalog(e) => e, + Self::Log(e) => e, + Self::Checkpoint(e) => e, + }) + } +} + +impl From for CellGroupSeedError { + fn from(e: TransportError) -> Self { + Self::Transport(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: CatalogError) -> Self { + Self::Catalog(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: LogError) -> Self { + Self::Log(e) + } +} + +impl From for CellGroupSeedError { + fn from(e: CellGroupCheckpointError) -> Self { + Self::Checkpoint(e) + } +} + +/// A staged, non-serving destination. Dropping it before install closes the +/// exchange; retry with a fresh grant and the same permanent engine/member IDs. +/// Recover the staged group with `CellGroupApplier::recover` before install so +/// the selected base and its physical retention owner are established, even for +/// an empty genesis. Once install is accepted, cancellation does not abandon +/// queued engine work. +/// An unknown result must be reconciled through bootstrap_joined_group_member. +pub struct CellGroupSeedSession { + group: GroupLog, + command: oneshot::Sender, + result: oneshot::Receiver>, +} + +impl CellGroupSeedSession { + pub fn group(&self) -> &GroupLog { + &self.group + } + + /// Use the same stores on retry. Only a complete authenticated End, verified + /// physical closure and actual engine fence can return the startup permit. + pub async fn install(self, applier: CellGroupApplier) -> Result { + self.command + .send(applier) + .map_err(|_| TransportError::Closed)?; + self.result.await.map_err(|_| TransportError::Closed)? + } +} + +struct Lease(Arc); + +impl Drop for Lease { + fn drop(&mut self) { + self.0.store(false, Ordering::Release); + } +} + +struct ReceiveAdmission { + _wire: Permit, + _transfer: Permit, +} + +struct Exchange { + state: Arc, + stream: Stream, + exporter: [u8; 32], + binding: [u8; 32], + grant: Arc, + group: GroupLog, + lease: Arc, + directory: RangeDirectory, + birth: CellGroupBirth, + pool: CellGroupCheckpointPool, + admission: Arc, + _connection: resources::ConnectionOwner, + operation: OperationId, +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs new file mode 100644 index 000000000..9b4c0a01c --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/server.rs @@ -0,0 +1,186 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; + +impl State { + pub(in crate::raft::transport) async fn receive_group_seed< + S: AsyncRead + AsyncWrite + Unpin, + >( + self: &Arc, + stream: &mut S, + exporter: &[u8; 32], + prefix: [u8; frame::PREFIX], + ) -> Result<(), TransportError> { + let _connection = self + .snapshot_connections + .clone() + .try_acquire_owned() + .map_err(|_| TransportError::Overloaded)?; + let _metadata = self + .seed_metadata + .acquire(seeds::META_BYTES) + .ok_or(TransportError::Overloaded)?; + let wire = snapshots::Wire { + auth: &self.auth, + exporter, + options: &self.options, + max_body: META_BYTES, + }; + let head = self + .bounded( + None, + snapshots::read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), + ) + .await?; + let request: Request = wire.decode(REQUEST_MAGIC, REQUEST, &head, None)?; + if frame::decimal(&request.nonce)? == 0 + || request.timeout_ms == 0 + || request.timeout_ms > 3_600_000 + { + return Err(TransportError::Protocol); + } + let peer = self.authorize_physical(&request.route)?; + let (target, source) = request.route.decode_group()?; + let (endpoint, pool) = self.checkpoint_endpoint(&source.scope)?; + let deadline = self + .domain + .mono_ms() + .map_err(|_| TransportError::Closed)? + .saturating_add(request.timeout_ms.min(self.options.snapshot_timeout_ms)); + let charge = wire_charge(&self.options); + let admission = Arc::new(Admission { + _node: self + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _peer: peer + .outgoing + .lane(Lane::Data) + .acquire(charge) + .ok_or(TransportError::Overloaded)?, + _transfer: self + .snapshot_source_budget + .acquire(snapshots::memory_bound(&self.options)) + .ok_or(TransportError::Overloaded)?, + }); + let authority = self + .seed_until(deadline, async { + Ok(self + .runtime + .shared()? + .seed_authority(target.clone(), deadline) + .await?) + }) + .await?; + let barrier = authority.barrier().index; + self.checkpoint_current(&endpoint, &request.route)?; + let work::Source { + metadata, + mut chunks, + completed, + } = work::source(self, endpoint.clone(), authority, pool, admission.clone())?; + let snapshot = self + .seed_until(deadline, async { + metadata.await.map_err(|_| TransportError::Closed)? + }) + .await?; + self.checkpoint_current(&endpoint, &request.route)?; + let body = snapshot + .write_to_bytes() + .map_err(|_| TransportError::Protocol)?; + if body.len() > META_BYTES { + return Err(TransportError::Limit); + } + let reply = wire.encode( + REPLY_MAGIC, + REPLY, + &head.tag, + &Reply { + route: request.route.reversed(), + request: head.tag, + nonce: request.nonce, + barrier: barrier.get().to_string(), + digest: digest(&body), + }, + &body, + )?; + let binding = reply.tag; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &reply).await?; + stream.write_all(&body).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + drop(body); + // The destination may need catalog I/O and store creation before it can + // accept bytes. Wait within the original total deadline, not one I/O tick. + let ready = self + .seed_until( + deadline, + snapshots::read_bulk_head(stream, None, READY_MAGIC, 0, &self.options), + ) + .await?; + let ready: Ack = wire.decode(READY_MAGIC, READY, &ready, Some(&binding))?; + if ready.sequence != 0 { + return Err(TransportError::Replay); + } + let _admission = admission; + let mut sequence = 0_u64; + let mut completed = Some(completed); + loop { + self.checkpoint_current(&endpoint, &request.route)?; + let next = self + .snapshot_io(&peer, deadline, async { + chunks.recv().await.ok_or(TransportError::Closed) + }) + .await?; + let (part, offset, length, bytes) = match next { + workers::Output::Chunk(chunk) => { + (Some(chunk.part), chunk.offset, chunk.length, chunk.bytes) + } + workers::Output::End(result) => { + result?; + (None, 0, 0, Vec::new()) + } + }; + if bytes.len() > CHUNK_BYTES.min(self.options.snapshot_chunk_bytes) { + return Err(TransportError::Limit); + } + let ending = part.is_none(); + let final_source = if ending { + let completed = completed.take().ok_or(TransportError::Protocol)?; + let source = self + .seed_until(deadline, async { + completed.await.map_err(|_| TransportError::Closed) + }) + .await?; + source.check()?; + Some(source) + } else { + None + }; + let packet = Packet { + sequence, + part, + offset, + length, + digest: digest(&bytes), + }; + let head = wire.encode(CHUNK_MAGIC, CHUNK, &binding, &packet, &bytes)?; + self.checkpoint_current(&endpoint, &request.route)?; + self.snapshot_io(&peer, deadline, async { + write_head(stream, &head).await?; + stream.write_all(&bytes).await?; + stream.flush().await?; + Ok(()) + }) + .await?; + if let Some(source) = final_source { + source.check()?; + return Ok(()); + } + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + } + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs new file mode 100644 index 000000000..30feeaa93 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/work.rs @@ -0,0 +1,119 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. +use super::*; +use crate::raft::cell_groups::checkpoint::wire::Chunk; + +pub(super) struct Source { + pub metadata: oneshot::Receiver>, + pub chunks: mpsc::Receiver, + pub completed: oneshot::Receiver, +} + +pub(super) fn source( + state: &Arc, + endpoint: Arc, + authority: CellGroupSeedAuthority, + pool: CellGroupCheckpointPool, + admission: Arc, +) -> Result { + let guard = state.reserve_task()?; + let (metadata, result) = oneshot::channel(); + let (tx, rx) = mpsc::channel(1); + let (finished, completed) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let captured = authority + .capture(&pool) + .and_then(|source| { + let snapshot = source.snapshot()?; + Ok((source, snapshot)) + }) + .map_err(checkpoint_error); + let (mut source, snapshot) = match captured { + Ok(value) => value, + Err(error) => { + let _ = metadata.send(Err(error)); + return; + } + }; + if metadata.send(Ok(snapshot)).is_err() { + return; + } + let result = source + .stream(|chunk| { + if !endpoint.active.load(Ordering::Acquire) { + return Err(CellGroupCheckpointError::Invalid("revoked seed endpoint")); + } + tx.blocking_send(workers::Output::Chunk(Chunk::copy(chunk))) + .map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::new( + std::io::ErrorKind::BrokenPipe, + "seed connection closed", + )) + }) + }) + .map_err(checkpoint_error); + let success = result.is_ok(); + if tx.blocking_send(workers::Output::End(result)).is_ok() && success { + let _ = finished.send(source); + } + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(Source { + metadata: result, + chunks: rx, + completed, + }) +} + +pub(super) fn receiver( + state: &Arc, + applier: CellGroupApplier, + pool: CellGroupCheckpointPool, + admission: Arc, + grant: Arc, +) -> Result { + let guard = state.reserve_task()?; + let log = state.runtime.log().clone(); + let scratch = log.snapshot_scratch(); + let directory = log.limits().directory.clone(); + let (input, mut rx) = mpsc::channel(1); + let (tx, result) = oneshot::channel(); + state + .domain + .blocking(move || { + let _guard = guard; + let _admission = admission; + let outcome = (|| { + let mut receiver = pool.receive(applier, scratch, directory)?; + loop { + grant.check_log(&log)?; + match rx.blocking_recv() { + Some(workers::Input::Chunk(chunk)) => chunk.append(&mut receiver)?, + Some(workers::Input::End) => { + grant.check_log(&log)?; + let prepared = receiver.finish()?; + grant.check_log(&log)?; + if prepared.snapshot() != grant.snapshot() { + return Err(CellGroupCheckpointError::Invalid( + "seed descriptor changed", + )); + } + return Ok(prepared); + } + None => { + return Err(CellGroupCheckpointError::Invalid("incomplete seed stream")) + } + } + } + })() + .map_err(checkpoint_error); + let _ = tx.send(outcome); + }) + .map_err(|_| TransportError::Closed)? + .detach(); + Ok(workers::Receive { input, result }) +} diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index 669565638..11f4f4515 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -235,7 +235,31 @@ pub(crate) enum ApplicationRecord { #[serde(rename = "cellar-image-apply-1")] Image(Option), #[serde(rename = "cellar-group-apply-1")] - Group(Box), + Group(Box), +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct GroupApplication { + manifest: super::cell_groups::GroupManifest, + // Required, local to this mapping/engine, and never snapshot payload. + seed: LocalGroupSeed, +} + +#[derive(Serialize, Deserialize)] +#[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] +enum LocalGroupSeed { + Absent, + Installed { index: u64 }, +} + +impl LocalGroupSeed { + fn index(&self) -> Option { + match self { + Self::Absent => None, + Self::Installed { index } => Some(*index), + } + } } impl GroupRecord { @@ -251,7 +275,12 @@ impl GroupRecord { applied_index: cache.applied.position.index.get(), applied_term: cache.applied.position.term.get(), application: match &cache.group_manifest { - Some(marker) => ApplicationRecord::Group(Box::new(marker.clone())), + Some(marker) => ApplicationRecord::Group(Box::new(GroupApplication { + manifest: marker.clone(), + seed: cache.group_seed.map_or(LocalGroupSeed::Absent, |index| { + LocalGroupSeed::Installed { index } + }), + })), None => ApplicationRecord::Image(cache.applied.image.clone()), }, }) @@ -261,14 +290,15 @@ impl GroupRecord { if self.mapping.format != 1 { return Err(LogError::Corrupt("unsupported group marker".into())); } - let (image, group_manifest) = match &self.application { - ApplicationRecord::Image(image) => (image.clone(), None), - ApplicationRecord::Group(marker) => { + let (image, group_manifest, group_seed) = match &self.application { + ApplicationRecord::Image(image) => (image.clone(), None, None), + ApplicationRecord::Group(application) => { + let marker = &application.manifest; let scope = self.mapping.history()?; if !marker.matches_group(&scope, self.mapping.birth_digest) { return Err(LogError::WrongIdentity); } - (None, Some(marker.as_ref().clone())) + (None, Some(marker.clone()), application.seed.index()) } }; let snapshot = Snapshot::parse_from_bytes(&self.snapshot).map_err(corrupt)?; @@ -290,6 +320,7 @@ impl GroupRecord { snapshot, entries: Default::default(), group_manifest, + group_seed, manifest_pins: None, checkpoint_manifest, applied: DurableApply { diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index 3e7af8791..f4de5847e 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -296,6 +296,7 @@ impl PreparedSeed { group_manifest: None, manifest_pins: None, checkpoint_manifest: None, + group_seed: None, applied: crate::raft::cache::DurableApply { position: LogPosition { index: LogIndex::new(self.descriptor.index), diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 9a6763f19..937738470 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -245,7 +245,11 @@ enum Work { Option>, Reply, ), - CreateCells(CellGroupBootstrap, Reply>), + CreateCells( + CellGroupBootstrap, + Option>, + Reply>, + ), Seeded(Box, Reply), Persist(OperationId, Vec, Reply), PersistCells( @@ -560,10 +564,22 @@ impl ReplicaLog { bootstrap: CellGroupBootstrap, ) -> Result, LogError> { bootstrap.configuration()?; - self.request(16 * 1024, |reply| Work::CreateCells(bootstrap, reply)) + self.request(16 * 1024, |reply| Work::CreateCells(bootstrap, None, reply)) .await } + pub(crate) async fn stage_cell_group( + &self, + bootstrap: CellGroupBootstrap, + grant: Arc, + ) -> Result, LogError> { + grant.check_log(self)?; + self.request(96 * 1024, |reply| { + Work::CreateCells(bootstrap, Some(grant), reply) + }) + .await + } + /// Read occupied identities and locally durable completed membership under /// the engine catalog lock and short per-group cache reads. This catalog is /// append-only for an engine instance; retired bindings remain. @@ -1238,8 +1254,24 @@ impl Worker { fn create_cells( &mut self, bootstrap: CellGroupBootstrap, + seed: Option<&super::transport::CellGroupSeedGrant>, ) -> Result, LogError> { - let conf = bootstrap.configuration()?; + let conf = if let Some(seed) = seed { + seed.check( + &self.owner, + bootstrap.route.history, + &ReplicaIdentity { + scope: bootstrap.route.history, + node: self.options.node, + disk: self.options.disk, + process: self.options.process, + member: bootstrap.member, + }, + )?; + seed.initial_configuration(&bootstrap)? + } else { + bootstrap.configuration()? + }; if let Some(group) = self .groups .read() @@ -1419,6 +1451,15 @@ impl Worker { ); } } + for write in &writes { + if let Some(grant) = &write.seed_grant { + grant.check( + &self.owner, + self.mappings[&write.group.key].history()?, + write.group.identity(), + )?; + } + } for publication in &mut publications { publication.arm(); } @@ -1557,8 +1598,8 @@ impl Worker { let _ = reply.send(result); fatal } - Work::CreateCells(bootstrap, reply) => { - let result = self.create_cells(bootstrap); + Work::CreateCells(bootstrap, seed, reply) => { + let result = self.create_cells(bootstrap, seed.as_deref()); let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); if fatal.is_some() { self.closed.store(true, Ordering::Release); diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index 5c5f7ff83..e7496a2b3 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -22,7 +22,10 @@ use crate::asyncrt::DomainToken; use crate::peer_auth::PeerAuth; use ::raft::prelude::{Message, MessageType}; use cellar_logic::consensus::{ClusterId, GroupScope, NamespaceId, NodeId, ReplicaIdentity}; -pub use group_snapshots::CellGroupCheckpointRegistration; +pub(crate) use group_snapshots::seed::Grant as CellGroupSeedGrant; +pub use group_snapshots::{ + CellGroupCheckpointRegistration, CellGroupSeedError, CellGroupSeedSession, +}; use protobuf::Message as _; use serde::{Deserialize, Serialize}; pub(crate) use shared::PhysicalRegistration; @@ -1416,6 +1419,19 @@ impl State { .map_err(|error| TransportError::Tls(error.to_string()))?; return self.receive_seed(&mut stream, &exporter, prefix).await; } + if &prefix[..4] == group_snapshots::seed::REQUEST_MAGIC { + if lane != Lane::Data || sequence != 0 { + return Err(TransportError::Protocol); + } + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], group_snapshots::seed::EXPORT_LABEL, None) + .map_err(|error| TransportError::Tls(error.to_string()))?; + return self + .receive_group_seed(&mut stream, &exporter, prefix) + .await; + } if &prefix[..4] == group_snapshots::START_MAGIC { if lane != Lane::Data || sequence != 0 { return Err(TransportError::Protocol); diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 1d5d14c9a..400c37e6b 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,16 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_install_materialized_replaces_deleted_voter_and_resumes() -> Result<()> { + Box::pin(runtime_fixture::seed_install(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_install_extents_replaces_deleted_voter_and_resumes() -> Result<()> { + Box::pin(runtime_fixture::seed_install(true)).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_group_materialized_requires_quorum_and_bound_membership() -> Result<()> { runtime_fixture::seed_source(false).await diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 81457d5b6..b952d27e6 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -813,3 +813,5 @@ pub(super) use sql::automatic_checkpoints; pub(super) use sql::{membership_lifecycle, membership_sync}; pub(super) use sql::{seed_minority, seed_source}; + +pub(super) use sql::seed_install; diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs new file mode 100644 index 000000000..82869e14a --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -0,0 +1,642 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh learner staging, physical install, catch-up and deleted-disk replacement. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::{OperationId, ReplicaIdentity}; +use cellar_ltx::internal::NodeExtentStore; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage, +}; +use std::path::Path; + +fn target(history: GroupHistory) -> Result> { + Ok(ReplicaIdentity { + scope: history, + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([14; 16])?, + member: MemberId::new(4)?, + process: ProcessGeneration::new([24; 16])?, + }) +} + +fn stores(base: &Path, rooted: bool) -> Result<(ManifestPageStore, CellGroupApplyImages)> { + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let extents = if rooted { + Some(NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?) + } else { + None + }; + Ok(( + pages, + super::super::checkpoint_receive::config(images, extents, base)?, + )) +} + +async fn survivors(fixture: &RuntimeCluster, history: GroupHistory, index: LogIndex) -> Result<()> { + controller::wait_for(async || { + for runtime in &fixture.runtimes[..2] { + if runtime.status(history).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn configure( + fixture: &RuntimeCluster, + route: &GroupRoute, + command: ConfChangeV2, +) -> Result { + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let change = CellGroupMembershipChange::new(route, &group.membership()?, command, &[])?; + let accepted = tokio::time::timeout(Duration::from_secs(10), async { + loop { + match fixture.runtimes[0] + .configure(route.history, change.clone()) + .await + { + Ok(accepted) => return Ok::<_, anyhow::Error>(accepted), + Err(RuntimeError::NotReady) => { + fixture.runtimes[0].tick().await?; + tokio::time::sleep(Duration::from_millis(5)).await; + } + Err(error) => return Err(error.into()), + } + } + }) + .await??; + survivors(fixture, route.history, accepted.position.index).await?; + Ok(accepted.position.index) +} + +async fn prepare( + fixture: &RuntimeCluster, + transport: &RaftTransport, + source: RaftPeer, +) -> Result { + let deadline = reads::deadline(fixture, 10_000).await?; + for attempt in 0..16 { + let result = Box::pin(transport.prepare_group_seed( + target(source.replica.scope)?, + source.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + deadline, + )) + .await; + match result { + Ok(session) => return Ok(session), + // A dropped exchange may still own the sole physical source slot. + // This staging operation reconciles the SAME identities and engine; + // retry only a transient transport refusal, within the original deadline. + Err(CellGroupSeedError::Transport( + TransportError::Io(_) | TransportError::Overloaded, + )) if attempt < 15 => tokio::time::sleep(Duration::from_millis(10)).await, + Err(error) => return Err(error.into()), + } + } + unreachable!("bounded retries always return") +} + +pub(crate) async fn seed_install(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + Box::pin(replace(fixture, route, a, b, rooted)).await +} + +async fn replace( + mut fixture: RuntimeCluster, + route: GroupRoute, + a: CellHistory, + b: CellHistory, + rooted: bool, +) -> Result<()> { + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + for transport in &fixture.transports { + transport.enable_group_checkpoints(CellGroupCheckpointPool::new( + checkpoint_fixture::limits(), + )?)?; + } + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let local = target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + + let mut physical = fixture.tcp_route(route.history)?; + // Remove a real voter's entire fixture disk before starting the transfer. + controllers.remove(2).close().await?; + fixture.appliers.retain(|(node, _), _| *node != 2); + fixture.cluster.voters[2].log.close().await?; + std::fs::remove_dir_all(fixture.cluster._root.path().join("node-3"))?; + if rooted { + std::fs::remove_dir_all(fixture.cluster._root.path().join("rooted-2"))?; + } + let erased = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-3/log"), + NodeId::new([3; 16])?, + DiskLineage::new([13; 16])?, + ProcessGeneration::new([33; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert!( + matches!( + fixture + .cluster + .domain + .run( + fixture + .cluster + .directory + .bootstrap_group_member(&fixture.cluster.birth, &erased) + ) + .await?, + Err(CatalogError::LostStableState) + ), + "old member identity must not recreate a lost disk" + ); + erased.close().await?; + + let base = fixture.cluster._root.path().join("node-4"); + let gate = fault::Gate::new(None); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + physical.revision = 2; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let mut registrations = Vec::new(); + for node in 0..2 { + fixture.transports[node].install_physical_group(physical.clone())?; + registrations.push( + fixture.transports[node] + .register_group_checkpoint(fixture.appliers[&(node, route.history)].1.clone())?, + ); + } + // A stale ready flag cannot replace a fresh source quorum. Keep the seed + // connection discoverable while disconnecting the remaining voting peer. + let mut isolated = physical.clone(); + isolated.revision = 3; + isolated.peers.retain(|peer| peer.replica.member.get() != 2); + fixture.transports[1].remove_physical_group(&route.history)?; + fixture.transports[0].install_physical_group(isolated)?; + registrations[0] = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + assert!(fixture.runtimes[0].status(route.history).await?.ready); + assert!(Box::pin(transport.prepare_group_seed( + local.clone(), + physical.peers[0].clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 100).await? + )) + .await + .is_err()); + assert!( + matches!(log.cell_group(&route.history), Err(LogError::NotFound)), + "unproven source must not create a stage" + ); + physical.revision = 4; + for (node, registration) in registrations.iter_mut().enumerate() { + fixture.transports[node].install_physical_group(physical.clone())?; + *registration = fixture.transports[node] + .register_group_checkpoint(fixture.appliers[&(node, route.history)].1.clone())?; + } + let baseline_tasks = transport.usage().tasks; + let source = physical.peers[0].clone(); + let session = Box::pin(prepare(&fixture, &transport, source.clone())) + .await + .context("authenticate and stage fresh learner")?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "empty stage must never grant startup" + ); + assert!( + fixture + .cluster + .domain + .run( + fixture + .cluster + .directory + .bootstrap_group_member(&fixture.cluster.birth, &log) + ) + .await? + .is_err(), + "learner is not a birth member" + ); + let group = session.group().clone(); + let (pages, images) = stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, staged_root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(staged_root.marker().root().cells(), 0); + drop(staged_root); + drop(session); + controller::wait_for(async || Ok(transport.usage().tasks == baseline_tasks)) + .await + .context("cancel staged exchange")?; + // Even a fully verified generic snapshot install cannot mint the local + // authenticated seed receipt. Metadata and physical bytes alone are insufficient. + let authority = fixture.runtimes[0] + .seed_authority(local.clone(), reads::deadline(&fixture, 10_000).await?) + .await?; + let capture_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let receiver_pool = pool.clone(); + let receiver_applier = applier.clone(); + let directory = base.clone(); + let prepared = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut source = authority.capture(&capture_pool)?; + let mut receiver = receiver_pool.receive( + receiver_applier, + super::super::checkpoint_receive::scratch()?, + directory, + )?; + source.stream(|chunk| receiver.append(chunk))?; + Ok(receiver.finish()?) + })? + .await??; + let mut write = GroupWrite::new(&group); + write.hard_state = Some(raft::prelude::HardState { + term: prepared.snapshot().get_metadata().term, + commit: prepared.snapshot().get_metadata().index, + ..Default::default() + }); + write.snapshot = Some(SnapshotChange::CellGroup(prepared)); + log.persist_cells(OperationId::new(1801), vec![write]) + .await?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "generic snapshot must not mint local seed receipt" + ); + let session = Box::pin(prepare(&fixture, &transport, source)) + .await + .context("retry same staged engine")?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source cut")?; + + // Mutations after source capture must arrive through the Raft suffix. + let later = history(&route, "after-seed-cut")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + survivors(&fixture, route.history, accepted.position.index).await?; + let other = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second group")?; + let unrelated = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes( + other.route(), + &history(other.route(), "while-seeding")?, + &bytes, + false, + )?, + ) + .await?; + survivors(&fixture, second, unrelated.position.index).await?; + let release = gate.arm(fault::HOLD_SYNC); + let installer = applier.clone(); + let pending = fixture + .cluster + .domain + .spawn(async move { session.install(installer).await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "receipt cannot precede the storage fence" + ); + assert_eq!( + pool.usage().sources, + 1, + "physical receiver retained through sync" + ); + let (permit, root) = if rooted { + drop(release); + pending.await?? + } else { + pending.abort(); + assert!(pending.await.is_err()); + assert_eq!( + pool.usage().sources, + 1, + "caller cancellation cannot release physical owner" + ); + drop(release); + controller::wait_for(async || Ok(transport.usage().tasks == baseline_tasks)).await?; + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await??; + let recovering = applier.clone(); + let root = fixture + .cluster + .domain + .blocking(move || recovering.recover_selected())? + .await??; + (permit, root) + }; + assert_eq!(root.marker(), &expected); + assert!(root.pin_cell(&a.cell).is_some()); + assert!(root.pin_cell(&b.cell).is_some()); + assert!(root.pin_cell(&later.cell).is_none()); + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + transport.install_physical_group(physical)?; + runtime.attach(permit, root).await?; + assert!( + !runtime.status(route.history).await?.ready, + "learner cannot serve writes" + ); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + vec![applier.clone()], + controller::limits(), + ) + .await?; + controller::wait_for(async || { + fixture.runtimes[0].tick().await?; + Ok(runtime.status(route.history).await?.applied.index >= accepted.position.index) + }) + .await + .context("fresh learner catches up suffix")?; + assert_eq!( + group.group_manifest()?.context("caught up")?.root(), + fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("leader")? + .root() + ); + + let joint = ConfChangeV2 { + transition: ConfChangeTransition::Explicit, + changes: vec![ + ConfChangeSingle { + node_id: 4, + change_type: ConfChangeType::AddNode, + ..Default::default() + }, + ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::RemoveNode, + ..Default::default() + }, + ] + .into(), + ..Default::default() + }; + let entered = configure(&fixture, &route, joint) + .await + .context("promote caught-up replacement")?; + controller::wait_for( + async || Ok(runtime.status(route.history).await?.applied.index >= entered), + ) + .await?; + let mut joint_conf = group.initial_state()?.conf_state; + joint_conf.voters.sort_unstable(); + joint_conf.voters_outgoing.sort_unstable(); + assert_eq!(joint_conf.voters, vec![1, 2, 4]); + assert_eq!(joint_conf.voters_outgoing, vec![1, 2, 3]); + let left = configure(&fixture, &route, ConfChangeV2::default()).await?; + controller::wait_for(async || Ok(runtime.status(route.history).await?.applied.index >= left)) + .await?; + assert!(group.initial_state()?.conf_state.voters_outgoing.is_empty()); + assert!(group.initial_state()?.conf_state.learners.is_empty()); + controller::wait_for(async || { + match fixture.runtimes[0].transfer_leader(route.history, 4).await { + Ok(()) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + Err(error) => Err(error.into()), + } + }) + .await?; + controller::wait_for(async || Ok(runtime.status(route.history).await?.ready)).await?; + let final_cell = history(&route, "replacement-leader")?; + let committed = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &final_cell, &bytes, rooted)?, + ) + .await?; + survivors(&fixture, route.history, committed.position.index).await?; + controller::wait_for(async || { + Ok(runtime.status(route.history).await?.applied.index >= committed.position.index) + }) + .await?; + let current = group.group_manifest()?.context("replacement root")?; + let checkpoint = group + .checkpoint_manifest()? + .context("installed checkpoint")?; + controller.close().await?; + drop((controller, sql, applier)); + log.close().await?; + drop((transport, runtime, group, log)); + let resumed = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([25; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &resumed, + )) + .await??; + let resumed_group = resumed.cell_group(&route.history)?; + assert_eq!( + resumed_group.checkpoint_manifest()?.as_ref(), + Some(&checkpoint) + ); + let (pages, images) = stores(&base, rooted)?; + let (applier, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + resumed_group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker(), ¤t); + for cell in [&a, &b, &later, &final_cell] { + assert!(root.pin_cell(&cell.cell).is_some()); + } + let owner = fixture + .cluster + .domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [95; 32], + )) + .await??; + assert!( + !owner.status()?.ready, + "durable receipt cannot renew quorum authority" + ); + drop((owner, applier)); + resumed.close().await?; + for controller in controllers { + controller.close().await?; + } + drop(registrations); + assert_eq!(pool.usage().sources, 0); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index cdc6704de..f33a428d4 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -603,3 +603,7 @@ pub(crate) use membership::{membership_lifecycle, membership_sync}; #[path = "cell_group_seed.rs"] mod seed; pub(crate) use seed::{seed_minority, seed_source}; + +#[path = "cell_group_seed_install.rs"] +mod seed_install; +pub(crate) use seed_install::seed_install; From 06dda48da82d4651ed7e6b7a6b2fee354bbe6911 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 21:04:10 -0700 Subject: [PATCH 136/568] docs: qualify learner replacement and prioritize failure gates --- docs/design/multi-raft/cell-groups/README.md | 5 +- .../multi-raft/cell-groups/baseline.json | 12 +-- docs/design/multi-raft/cell-groups/handoff.md | 85 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 55 ++++++++++++ .../multi-raft/cell-groups/protocols.md | 51 ++++++++--- docs/design/multi-raft/cell-groups/tasks.json | 3 +- 6 files changed, 155 insertions(+), 56 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index f75b14b06..9f5ad5af8 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -11,8 +11,9 @@ catch-up, authenticated snapshot RPC and automatic controller checkpoint policy pass their scoped regression gates. Durable joint membership and leadership handoff also pass a partial gate. A learner seed source authorized by a fresh quorum read now covers empty groups, bounded capture of complete roots and stream revocation. -Authenticated destination bootstrap, catch-up/promotion and lost-disk replacement -are the next feature priorities. The production host has not yet cut over to shared groups. +Authenticated destination staging/install, caught-up promotion and replacement +after deleting a voter disk now pass native tests in both image forms. Remaining +CG-05 crash/pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 650605b8c..c02abea07 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "1814b27896c3c21166186fe8fbd1a1a6bdbb721a", - "runtime_revision": "1be27d234d4674d4d014785c60cecbfe394bdab9", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open.", + "source_revision": "ab2a308a937f349d3482a825d4b669d042cb03c3", + "runtime_revision": "ab2a308a937f349d3482a825d4b669d042cb03c3", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "8a33262be0c281ae3492043dbf72aa5a8ff6b8f2a84be4e6ba93ce4551082a3c", - "line": 442, - "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n}\n" + "sha256": "81ca3a200c2438c7a924d25cdb1a86e17e9a71e182f31df200d02ac882da142a", + "line": 444, + "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 3bcbcd278..d5f54666b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -33,15 +33,33 @@ Local checkpoint publication and native Raft-generated catch-up now pass their partial gate. Authenticated checkpoint streams now pass their partial transport gate. Automatic entry-count checkpointing at controller apply boundaries now passes its partial gate. Durable whole-group membership and leadership handoff now pass -an additional partial gate. Continue with fresh learner bootstrap and lost-disk -replacement; retain the remaining pressure/fault checks in the full CG-05 gate. +an additional partial gate. Authenticated fresh-learner installation, catch-up, +promotion and deleted-voter-disk replacement now pass a native partial gate. +Complete the remaining fault/pressure checks and all five CG-05 acceptance cases. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. -The latest seed-source checkpoint at `0c3769ec` passes 236 selected Rust +The latest learner-install checkpoint at `ab2a308a` passes 238 Rust cases, +eight gate commands and 37 validator tests. Its record under +`.cellar/cell-groups-evidence/CG-05-learner-install-final/` binds 320 source hashes and +five copied executables. Authenticated seed exchange, permanent claims, non-serving +staging, atomic local receipt, joined-member bootstrap and same-owner unknown-result +reconciliation are implemented. Both image forms replace a deleted voter disk, +catch up the post-capture suffix, promote/retire through joint consensus, transfer +leadership and commit new state. Cold recovery checks both retained and current +closures. Held-sync receipt refusal and materialized caller cancellation pass. +The tests caught and fixed noncanonical configuration equality after joint leave. + +Next complete seed-specific authentication/revocation faults, sync-error/process-kill +and joint-transition cuts, restarted-peer reconnection, minority receipt refusal, +large/empty group bounds and all five exact CG-05 cases. Preserve the original +acceptance requirements. Source capture fairness and controller pressure remain +open; no managed fleet, shared RustFS/load/power-loss/zero-quorum proof is claimed. + +The earlier seed-source checkpoint at `0c3769ec` passes 236 selected Rust cases, eight gate commands and 37 validator tests. Its record under `.cellar/cell-groups-evidence/CG-05-seed-source/` binds 312 source hashes and five copied executables. `seed_authority` uses a fresh group ReadIndex, including for @@ -52,15 +70,10 @@ budgets remain owned through physical work. Both image forms, receiver verificat stale/invalid targets, bounded admission, held-sink revocation, final-chunk expiry, isolated ready-leader refusal and independent group progress pass local tests. -Continue from `cell_groups/seed_authority.rs`, `directory/bootstrap.rs` and the -existing authenticated checkpoint stream/atomic receiver path. The immediate gap is -authenticated seed exchange plus permanent fresh-disk claims and joined-member -installation/attachment. The source API binds but does not authenticate the requested -process and cannot authorize a destination. Catch-up/promotion and deleted-voter-disk -replacement remain unimplemented integration, not qualified by the source tests. -Coordinate capture with an apply boundary or bounded fresh-selection retries; -continuous-write capture fairness remains unqualified. Keep all five CG-05 cases -and shared RustFS/load/crash gates open. +That source-only checkpoint did not authorize destination startup. The newer +learner-install checkpoint above supplies authenticated exchange, permanent claims, +installation, catch-up/promotion and native deleted-disk replacement. Continuous-write +capture fairness and the full five-case CG-05/RustFS/load/crash gates remain open. The receiver checkpoint at `cb164e02` passes 226 selected Rust cases, 9 gate commands and 37 validator tests. Its evidence binds 295 source hashes @@ -118,10 +131,10 @@ Two real-TCP controller cases cover both image forms; a third holds configuratio sync through continued operation and cancelled/joined shutdown. Automatic joint leave now replicates immediately without another write or tick. -Fresh learner 4 is intentionally not started: seed admission and lost-disk rebuild -remain the next feature-critical work. Metadata reopen is not complete physical -recovery or serving permission. Existing bootstrap permits still require birth -membership. Preserve all five CG-05 cases, outstanding crash/pressure cuts and +That earlier membership fixture intentionally did not start learner 4. The newer +learner-install fixture above covers that integration. Metadata reopen still cannot +replace complete physical recovery or serving permission. Birth-member bootstrap +retains its original contract; joined bootstrap requires the new local seed receipt. Preserve all five CG-05 cases, outstanding crash/pressure cuts and shared RustFS/load qualification. See [progress](progress.md) for exact evidence and the retained intermediate failures. @@ -140,9 +153,9 @@ The three new cases prove repeated compaction across two groups in both image forms and progress/recovery while the checkpoint pool is saturated. The policy retains the final idle tail. Continuous-saturation fairness, exhausted apply and completion budgets, held sync/cancelled close, checkpoint failure/quarantine capacity -and retained-root pressure remain qualification gaps. Fresh learner bootstrap and -lost-disk replacement are the next feature priorities; receiver crash/sync-error -cuts also remain required. All five declared CG-05 acceptance cases remain required. See +and retained-root pressure remain qualification gaps. The newer learner-install +checkpoint above covers native fresh-disk replacement; receiver crash/sync-error +cuts remain required. All five declared CG-05 acceptance cases remain required. See [progress](progress.md) for limits and intermediate failures. The source checkpoint at `b15da415` passes 138 Rust cases, eight gate commands @@ -703,22 +716,20 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Finish CG-05 fresh learner seeding and lost-disk replacement. Reuse the - implemented fresh-ReadIndex source authority; authenticate the seed exchange - and bind destination installation to permanent node/disk/engine claims and - the complete verified checkpoint. Existing bootstrap permits admit only birth - members; do not weaken them to accept an untrusted metadata observation. Bring - up the new learner through production transport, install every cell, catch up, - promote through joint consensus and retire the old member. Prove replacement - after deleting that voter's disk, minority refusal and one-voter-loss tolerance. -2. Complete CG-05 qualification while delivering those membership features. +1. Finish the full CG-05 acceptance gate around the implemented fresh-learner + replacement path. Extend `group_snapshot_transport/seed/` and its production + controller fixture with seed-specific malformed/replayed/wrong-process traffic, + process revocation, sync failures and process-kill cuts. Reconnect a restarted + joined member, prove minority write/receipt refusal and joint-transition restart. + Preserve permanent claims, local install receipts and complete physical recovery. +2. Complete CG-05 qualification around the delivered membership features. Preserve all five exact acceptance cases. Add joint-transition crash cuts, receiver crash/sync-error cuts, continuous checkpoint saturation fairness, exhausted apply/completion budgets, held sync/cancelled close, quarantine capacity and retained-root pressure. Keep unrelated-group progress and bounded physical ownership observable. Idle/time/byte checkpoint triggers require an explicit cut-reservation contract; - they must not displace fresh-member recovery as the next feature priority. + keep the exact five-case task gate ahead of the archive feature dependency. 3. Deliver CG-06 archive/reclamation with verified complete checkpoints and contiguous tails. Upload completion alone cannot authorize reclaiming local recovery dependencies or an S3 durability acknowledgement. @@ -733,14 +744,14 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Continue from `cell_groups/seed_authority.rs`, the authenticated checkpoint -transport, `directory/bootstrap.rs` and the atomic checkpoint installation path. -Keep configuration, membership revision and complete root atomic. A captured -metadata observation, transport admission reply or lost response cannot create -new serving authority. Reuse permanent member claims and the node checkpoint pool -for learner bootstrap; retain physical closure owners through cancellation and -joined shutdown. Extend the production-controller membership fixture rather than -manually manufacturing committed transitions. +Continue from `group_snapshot_transport/seed/`, +`cell_groups/directory/joined_bootstrap.rs`, `cell_groups/apply/recovery.rs` and +`tests/support/cell_group_seed_install.rs`. Seed metadata and an imported snapshot +cannot mint the local authenticated-install receipt. Keep configuration, membership, +receipt and complete root atomic. Reconcile unknown results through the exact +permanent claim and the existing page owner; retain physical owners through +cancellation and joined shutdown. Use canonical configuration equality when +comparing replay with live RawNode membership. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 2aabcbe80..4a1088b76 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,61 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 authenticated learner installation and replacement + +Qualified source: `ab2a308a937f349d3482a825d4b669d042cb03c3`. All 238 selected Rust cases and eight +gate commands pass, including all-target Clippy for Cellar/core/LTX, five native +suites, formatting/layout, dependency isolation, docs, source preflight and 37 +plan-validator tests. `.cellar/cell-groups-evidence/CG-05-learner-install-final/checkpoint.json` +binds 320 source hashes, five copied executables, frozen plan inputs and logs. +Transfer this ignored evidence with its predecessors. CG-05 remains IN_PROGRESS. + +A dedicated authenticated seed exchange now binds the exact physical identities, +fresh group ReadIndex barrier and canonical complete checkpoint. Destination +staging reconciles permanent node/disk/engine and member claims but cannot serve. +Only a verified complete transfer followed by the actual engine fence installs a +local seed receipt atomically with configuration, membership and both root cuts. +The receipt never travels in snapshots; generic snapshot installation cannot mint +it. Joined-member bootstrap requires this receipt, the unchanged permanent claim +and active membership. Startup still needs physical recovery and fresh quorum. +The internal marker-1 group application record now requires `manifest` and `seed`; +the previous bare-manifest shape refuses. No legacy reader or data deletion is added. + +Two new tests use production controllers, real TLS, independent native engines +and SQLite/LTX in both image forms. They delete voter 3's fixture disk before +transfer and prove its old identity cannot recreate storage. An isolated source +cannot create a stage. Staged cancellation retries retain exact identities and +the original deadline; transient source-slot ownership is allowed to finish. +Initial-member bootstrap and generic-snapshot bootstrap refuse the staged learner. +A held engine sync cannot publish the receipt. Cancelling the materialized caller +retains physical owners through sync; `recover_selected` reconciles using the same +page store without bypassing its lock. The new learner installs every prior cell, +catches up a later Raft suffix while another group progresses, joins the voter set +through joint consensus, retires member 3, takes leadership and commits another +cell. Cold reopen verifies current and retained checkpoints, then starts a new +process owner that is correctly not quorum-ready. + +The tests also exposed a production joint-leave bug: RawNode's historical hash +iteration order can differ from pure configuration replay. Owner completion now +compares validated canonical configurations, preserving all sets and auto-leave +state. Fixture setup was split to avoid an oversized combined debug future; no +stack limit or safety assertion was relaxed. Intermediate failures and the final +passing runs are retained, and temporary debug probes were removed. The first +broad gate hit eight macOS socket-address allocation errors; all ten remote RPC +cases passed unchanged in a targeted recheck, followed by the complete final gate. +Host samples showed thousands of transient loopback sockets. The failed broad +run and diagnostics are retained; no global network settings were changed. + +**Reflection and next step:** the fresh-disk replacement path is implemented and +locally exercised. The immediate remaining CG-05 work is adversarial seed-frame +and process-revocation coverage, sync-error/process-kill and joint-transition crash +cuts, restarted-peer reconnection, minority write/receipt refusal, and the complete +five-case gate including empty-cell and pressure/pin scenarios. Sustained-write +capture fairness and controller pressure cuts remain open. Complete those gates, +then deliver CG-06 verified S3 archive/reclamation, CG-08 effects/caches and CG-07 +SDK/fleet cutover. Managed fleet orchestration, shared-group RustFS/load, power-loss +and zero-quorum DR remain unqualified; native disk deletion is not those proofs. + ## CG-05 live-quorum learner seed source Qualified source: `0c3769eccbb845a5943ef60e771c107f1800d9f2`. All 236 selected Rust cases and eight diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 815f799f7..b4a427c19 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -249,16 +249,47 @@ must obtain a fresh authority. After capture, ordinary cell writes may continue while the source streams its immutable cut. Membership changes, term changes, leadership transfer, shutdown and deadline expiry revoke the source. Every sink call has checks before and after it; the final completion also checks authority. -An incomplete or revoked stream cannot produce a successful End. The source binds -the requested process identity but does not authenticate it; the future seed RPC -must bind that identity to its authenticated connection and process revocation. - -This is a source boundary, not a destination bootstrap permit. Authenticated seed -exchange, permanent destination claims, new-member installation/attachment, -initial-member replacement, lost-disk rebuild and managed fleet orchestration -remain required CG-05 work. Existing bootstrap permits still -admit only initial birth members. A metadata reopen is not a fresh-member seed, -quorum proof, verified physical recovery or serving permission. +An incomplete or revoked stream cannot produce a successful End. The source API +binds the requested process identity; `prepare_group_seed` authenticates it through +a separate TLS-exporter-bound seed exchange. Requests, grants, readiness and chunks +have distinct marker-1 domains. The reply binds the request tag, nonce, complete +physical route, fresh barrier and canonical checkpoint descriptor. Chunk sequence, +digests and explicit End are authenticated within that exchange. Peer replacement, +transport shutdown and the original deadline revoke pending destination work. + +The destination reconciles the namespace node pin and permanent member claim +against the exact node/disk/engine. Its synced staging mapping retains the original +birth configuration and grants no startup permission. An existing remote claim +cannot recreate a missing mapping. `CellGroupSeedSession` exposes the staged group +only to construct its physical stores; dropping an uninstalled session abandons +the exchange and leaves a non-serving stage that needs a fresh authenticated retry. + +The receiver verifies and syncs the entire physical closure before a single engine +fence installs configuration, membership, complete root, retained checkpoint and a +local authenticated-install receipt. That receipt is required in the marker-1 +`cellar-group-apply-1` storage record (`manifest` plus `seed`); the old bare-manifest +shape refuses decoding. The receipt never travels in a snapshot and generic +snapshot installation cannot mint it. No old-format reader or user-data deletion +is introduced. Live grant and exact engine/process checks run before entering the +fence. Once physical persistence begins, cancellation and close retain the grant, +image owners, task reservation and consensus lease until actual completion. + +`CellGroupApplier::recover_selected` reconciles an unknown install outcome using +the existing physical store owner; selected pins prevent reopening that directory +while it is still owned. It verifies current and retained checkpoint closures +without introducing catalog or quorum authority. + +`bootstrap_joined_group_member` resumes only a locally receipted install with the +same permanent remote claim and active node/disk-bound membership. Full physical +recovery still verifies both current and retained checkpoint closures before +attachment. Birth-member bootstrap continues to admit only original members. +Neither API grants current-term quorum authority. A new learner must catch up +through ordinary Raft replication before promotion; a lost install response is +reconciled from durable local state instead of assuming failure or resetting disk. + +Native transfer/replacement qualification is recorded in progress.md. Managed +fleet orchestration, sustained-write capture fairness, the complete CG-05 fault +matrix and deployment qualification remain open. ## Apply, checkpoint and archive diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 2f51e0915..4d29111c0 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1588,7 +1588,8 @@ "crates/cellar/tests/support/cell_group_controller.rs", "crates/cellar/tests/support/cell_group_controller_checkpoints.rs", "crates/cellar/tests/support/cell_group_membership.rs", - "crates/cellar/tests/support/cell_group_seed.rs" + "crates/cellar/tests/support/cell_group_seed.rs", + "crates/cellar/tests/support/cell_group_seed_install.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From de552f6daabf786c14fe9df97565538ba5d302b4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 21:15:32 -0700 Subject: [PATCH 137/568] test: qualify cell-group seed refusal and voter reconnection --- .../support/cell_group_network_probes.rs | 59 +++- .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_install.rs | 253 ++++++++++++++++-- .../tests/support/cell_group_seed_probes.rs | 142 ++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- 5 files changed, 426 insertions(+), 33 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_seed_probes.rs diff --git a/crates/cellar/tests/support/cell_group_network_probes.rs b/crates/cellar/tests/support/cell_group_network_probes.rs index 55742d958..6f7ff8d7f 100644 --- a/crates/cellar/tests/support/cell_group_network_probes.rs +++ b/crates/cellar/tests/support/cell_group_network_probes.rs @@ -9,13 +9,20 @@ use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpStream; use tokio_rustls::client::TlsStream; -struct Connection { +pub(super) struct Connection { stream: TlsStream, - exporter: [u8; 32], + pub(super) exporter: [u8; 32], } impl Connection { async fn open(address: std::net::SocketAddr) -> Result { + Self::open_with_exporter(address, b"EXPORTER-cellar-raft-1").await + } + + pub(super) async fn open_with_exporter( + address: std::net::SocketAddr, + label: &[u8], + ) -> Result { let mut roots = rustls::RootCertStore::empty(); roots.add( include_bytes!("../../transport/tcp/fixtures/ca.der") @@ -35,15 +42,14 @@ impl Connection { TcpStream::connect(address).await?, ) .await?; - let exporter = - stream - .get_ref() - .1 - .export_keying_material([0; 32], b"EXPORTER-cellar-raft-1", None)?; + let exporter = stream + .get_ref() + .1 + .export_keying_material([0; 32], label, None)?; Ok(Self { stream, exporter }) } - async fn frame( + pub(super) async fn frame( &mut self, magic: &[u8; 4], purpose: &[u8], @@ -71,7 +77,40 @@ impl Connection { Ok(tag) } - async fn refused(mut self) -> Result<()> { + /// Independently check the server's framing and exporter-bound MAC before + /// using a reply in a later probe. Bounds apply before allocating either body. + pub(super) async fn receive( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + ) -> Result<(Value, Vec, [u8; 32])> { + tokio::time::timeout(Duration::from_secs(5), async { + let mut prefix = [0; 20]; + self.stream.read_exact(&mut prefix).await?; + assert_eq!(&prefix[..4], magic); + assert_eq!(&prefix[4..8], &1u32.to_be_bytes()); + let header_len = u32::from_be_bytes(prefix[8..12].try_into()?) as usize; + let body_len = u64::from_be_bytes(prefix[12..].try_into()?); + assert!(header_len <= 64 * 1024 && body_len <= 64 * 1024); + let mut header = vec![0; header_len]; + self.stream.read_exact(&mut header).await?; + let mut tag = [0; 32]; + self.stream.read_exact(&mut tag).await?; + let mut mac = Hmac::::new_from_slice(&[17; 32])?; + for part in [purpose, &self.exporter, binding, &prefix, &header] { + mac.update(&(part.len() as u64).to_be_bytes()); + mac.update(part); + } + mac.verify_slice(&tag)?; + let mut body = vec![0; body_len as usize]; + self.stream.read_exact(&mut body).await?; + Ok((serde_json::from_slice(&header)?, body, tag)) + }) + .await? + } + + pub(super) async fn refused(mut self) -> Result<()> { let mut byte = [0]; let result = tokio::time::timeout(Duration::from_secs(5), self.stream.read(&mut byte)).await?; @@ -83,7 +122,7 @@ impl Connection { } } -fn peer(peer: &RaftPeer) -> Value { +pub(super) fn peer(peer: &RaftPeer) -> Value { json!({"node":peer.replica.node.bytes(), "disk":peer.replica.disk.bytes(), "process":peer.replica.process.bytes(), "member":peer.replica.member.get().to_string(), "label":peer.label}) } diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index b952d27e6..92e3491c4 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -9,6 +9,8 @@ use std::collections::BTreeMap; mod checkpoint_local; #[path = "cell_group_network_probes.rs"] mod network_probes; +#[path = "cell_group_seed_probes.rs"] +mod seed_probes; pub(super) use checkpoint_local::{ local_publication, local_shutdown, tcp_cancelled_delivery, tcp_protocol, tcp_publication, tcp_retry, diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs index 82869e14a..50d8a0440 100644 --- a/crates/cellar/tests/support/cell_group_seed_install.rs +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -130,11 +130,15 @@ async fn replace( ) -> Result<()> { let second = controller::add_second(&mut fixture, route.history).await?; fixture.connect_tcp().await?; - for transport in &fixture.transports { - transport.enable_group_checkpoints(CellGroupCheckpointPool::new( - checkpoint_fixture::limits(), - )?)?; - } + let source_pools = fixture + .transports + .iter() + .map(|transport| -> Result<_> { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + Ok(pool) + }) + .collect::>>()?; let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; fixture.runtimes[0].campaign(second).await?; controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; @@ -240,6 +244,14 @@ async fn replace( .register_group_checkpoint(fixture.appliers[&(node, route.history)].1.clone())?, ); } + super::super::seed_probes::source_protocol( + &fixture, + &physical, + &source_pools[0], + &mut registrations[0], + ) + .await + .context("seed protocol probes")?; // A stale ready flag cannot replace a fresh source quorum. Keep the seed // connection discoverable while disconnecting the remaining voting peer. let mut isolated = physical.clone(); @@ -324,6 +336,53 @@ async fn replace( controller::wait_for(async || Ok(transport.usage().tasks == baseline_tasks)) .await .context("cancel staged exchange")?; + let staged = group.group_manifest()?; + let revoked = Box::pin(prepare(&fixture, &transport, source.clone())).await?; + fixture.transports[0].remove_physical_group(&route.history)?; + assert!( + revoked.install(applier.clone()).await.is_err(), + "revoked source cannot install its previously signed cut" + ); + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline_tasks && source_pools[0].usage().sources == 0) + }) + .await?; + assert_eq!(group.group_manifest()?, staged); + assert_eq!(group.checkpoint_manifest()?, None); + assert_eq!(pool.usage().sources, 0); + fixture.transports[0].install_physical_group(physical.clone())?; + registrations[0] = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + let expired = Box::pin(transport.prepare_group_seed( + local.clone(), + source.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 500).await?, + )) + .await?; + // Wait for the bounded exchange to expire while holding the user session. + // No sleep or fabricated process clock establishes the refusal. + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline_tasks && source_pools[0].usage().sources == 0) + }) + .await?; + assert!( + expired.install(applier.clone()).await.is_err(), + "expired session cannot acquire a fresh install deadline" + ); + assert_eq!(group.group_manifest()?, staged); + assert_eq!(group.checkpoint_manifest()?, None); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err()); // Even a fully verified generic snapshot install cannot mint the local // authenticated seed receipt. Metadata and physical bytes alone are insufficient. let authority = fixture.runtimes[0] @@ -465,7 +524,7 @@ async fn replace( assert!(root.pin_cell(&b.cell).is_some()); assert!(root.pin_cell(&later.cell).is_none()); assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); - transport.install_physical_group(physical)?; + transport.install_physical_group(physical.clone())?; runtime.attach(permit, root).await?; assert!( !runtime.status(route.history).await?.ready, @@ -617,22 +676,11 @@ async fn replace( for cell in [&a, &b, &later, &final_cell] { assert!(root.pin_cell(&cell.cell).is_some()); } - let owner = fixture - .cluster - .domain - .run(CellGroupOwner::start( - permit, - root, - RaftRuntimeOptions::default(), - [95; 32], - )) - .await??; - assert!( - !owner.status()?.ready, - "durable receipt cannot renew quorum authority" - ); - drop((owner, applier)); - resumed.close().await?; + Box::pin(reconnect( + &fixture, physical, resumed, permit, root, applier, rooted, + )) + .await + .context("replacement process reconnect")?; for controller in controllers { controller.close().await?; } @@ -640,3 +688,164 @@ async fn replace( assert_eq!(pool.usage().sources, 0); fixture.close().await } + +/// Start a new process from the installed disk and use only production transport +/// and controllers to reacquire quorum, then prove a minority cannot commit. +async fn reconnect( + fixture: &RuntimeCluster, + mut physical: cellar::application::consensus::verification::GroupRoute, + log: ReplicaLog, + permit: CellGroupBootstrapPermit, + root: PreparedGroupManifest, + applier: CellGroupApplier, + rooted: bool, +) -> Result<()> { + let route = root.marker().route().clone(); + let group = log.cell_group(&route.history)?; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, root).await?; + assert!( + !runtime.status(route.history).await?.ready, + "disk receipt cannot renew quorum" + ); + let transport = tcp::start(&runtime, 3).await?; + transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(checkpoint_fixture::limits())?)?; + physical.revision += 1; + physical.peers.retain(|peer| peer.replica.member.get() != 3); + let peer = physical + .peers + .iter_mut() + .find(|peer| peer.replica.member.get() == 4) + .context("replacement route")?; + peer.replica = group.identity().clone(); + peer.control_address = transport.control_address(); + peer.data_address = transport.data_address(); + peer.query_address = transport.query_address(); + for source in &fixture.transports[..2] { + source + .install_physical_group(physical.clone()) + .context("refresh survivor route after process restart")?; + } + transport + .install_physical_group(physical) + .context("install restarted voter route")?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql, + vec![applier], + controller::limits(), + ) + .await?; + runtime.campaign(route.history).await?; + controller::wait_for(async || { + // Every fixture voter has automatic ticking disabled. Survivors must + // also advance past the old process's leader lease before campaigning + // can succeed. If one wins first, use the real caught-up handoff path. + for survivor in &fixture.runtimes[..2] { + survivor.tick().await?; + if survivor.status(route.history).await?.ready { + match survivor.transfer_leader(route.history, 4).await { + Ok(()) | Err(RuntimeError::NotReady) => (), + Err(error) => return Err(error.into()), + } + } + } + runtime.tick().await?; + Ok(runtime.status(route.history).await?.ready) + }) + .await + .context("restarted voter regains quorum")?; + let cell = history(&route, "after-process-restart")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let accepted = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + survivors(fixture, route.history, accepted.position.index).await?; + controller::wait_for(async || { + Ok(runtime.status(route.history).await?.applied.index >= accepted.position.index) + }) + .await?; + drop( + runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + reads::deadline(fixture, 5000).await?, + ) + .await?, + ); + let before = group.group_manifest()?.context("quorum root")?; + for source in &fixture.transports[..2] { + source.remove_physical_group(&route.history)?; + } + transport.remove_physical_group(&route.history)?; + assert!( + runtime.status(route.history).await?.ready, + "exercise stale ready status" + ); + let pending_cell = history(&route, "isolated-cannot-commit")?; + let pending = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &pending_cell, &bytes, rooted)?, + ) + .await?; + // Admission is not a commit. A fresh correlated quorum read must fail even + // though the old leader can still enqueue a proposal during check-quorum lag. + assert!(matches!( + runtime + .read_authority( + route.history, + cell, + CellReadConsistency::Linearizable, + reads::deadline(fixture, 150).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + assert!(runtime.status(route.history).await?.applied.index < pending.position.index); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + controller::wait_for(async || { + runtime.tick().await?; + Ok(!runtime.status(route.history).await?.ready) + }) + .await?; + assert!(matches!( + runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &pending_cell, &bytes, rooted)? + ) + .await, + Err(RuntimeError::NotLeader | RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + controller.close().await?; + log.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_seed_probes.rs b/crates/cellar/tests/support/cell_group_seed_probes.rs new file mode 100644 index 000000000..e050f6bd3 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_probes.rs @@ -0,0 +1,142 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Independent authenticated seed requests against the real source listener. +use super::*; +use network_probes::{peer, Connection}; +use serde_json::{json, Value}; + +pub(super) async fn source_protocol( + fixture: &RuntimeCluster, + route: &cellar::application::consensus::verification::GroupRoute, + pool: &CellGroupCheckpointPool, + registration: &mut CellGroupCheckpointRegistration, +) -> Result<()> { + let source = &fixture.transports[0]; + let before = fixture.cluster.voters[0] + .log + .cell_group(&route.scope)? + .group_manifest()?; + let target = route.peers.last().context("joining peer")?; + assert_eq!(target.replica.member.get(), 4); + let history = route.scope; + let request = json!({"route": { + "scope": {"kind":"Group", "identity":[history.cluster.bytes(), history.namespace.bytes(), history.group.bytes(), history.incarnation.bytes()]}, + "schema":route.schema_digest, "source":peer(target), "target":peer(&route.peers[0]) + }, "nonce":"1", "timeout_ms":10000}); + for case in 0..11 { + let mut connection = + Connection::open_with_exporter(source.data_address(), b"EXPORTER-cellar-group-seed-1") + .await?; + let mut invalid = request.clone(); + match case { + 0 => connection.exporter[0] ^= 1, + 1 => invalid["route"]["source"]["process"] = json!(vec![244; 16]), + 2 => invalid["route"]["source"]["disk"] = json!(vec![244; 16]), + 3 => invalid["route"]["scope"]["identity"][3] = json!(vec![244; 16]), + 4 => invalid["route"]["schema"] = json!(vec![244; 32]), + 5 => invalid["nonce"] = json!("0"), + 6 => invalid["nonce"] = json!("01"), + 7 => invalid["timeout_ms"] = json!(0), + 8 => invalid["timeout_ms"] = json!(3_600_001), + 9 => invalid["extra"] = json!(true), + // A current voter cannot request a fresh learner seed. + 10 => invalid["route"]["source"] = peer(&route.peers[1]), + _ => unreachable!(), + } + connection + .frame(b"CLGJ", b"cellar-group-seed-request-1", &[], invalid, &[]) + .await?; + connection + .refused() + .await + .with_context(|| format!("seed request fault {case}"))?; + released(source, pool).await?; + } + let mut previous = None; + for case in 0..5 { + let mut connection = + Connection::open_with_exporter(source.data_address(), b"EXPORTER-cellar-group-seed-1") + .await?; + let tag = connection + .frame( + b"CLGJ", + b"cellar-group-seed-request-1", + &[], + request.clone(), + &[], + ) + .await?; + let (reply, body, binding) = connection + .receive(b"CLGK", b"cellar-group-seed-grant-1", &tag) + .await?; + assert_eq!(reply["request"], json!(tag)); + assert_eq!(reply["nonce"], request["nonce"]); + assert_eq!(reply["route"]["source"], request["route"]["target"]); + assert_eq!(reply["route"]["target"], request["route"]["source"]); + assert_eq!( + reply["digest"], + json!(<[u8; 32]>::from(Sha256::digest(&body))) + ); + assert_eq!( + pool.usage().sources, + 1, + "complete source remains pinned awaiting Ready" + ); + let mut ready = json!({"sequence":0}); + let bound = match case { + 0 => { + ready["sequence"] = json!(1); + binding + } + 1 => previous.context("previous authenticated reply")?, + 2 => { + ready["extra"] = Value::Bool(true); + binding + } + // The captured source registration is revoked while the peer has + // its signed descriptor. Its Ready must not revive the old owner. + 3 => { + source.remove_physical_group(&history)?; + binding + } + 4 => { + drop(connection); + released(source, pool).await?; + continue; + } + _ => unreachable!(), + }; + connection + .frame(b"CLGL", b"cellar-group-seed-ready-1", &bound, ready, &[]) + .await?; + connection + .refused() + .await + .with_context(|| format!("seed Ready fault {case}"))?; + released(source, pool).await?; + if case == 3 { + source.install_physical_group(route.clone())?; + *registration = + source.register_group_checkpoint(fixture.appliers[&(0, history)].1.clone())?; + } + previous = Some(binding); + } + assert_eq!( + fixture.cluster.voters[0] + .log + .cell_group(&history)? + .group_manifest()?, + before + ); + Ok(()) +} + +async fn released(transport: &RaftTransport, pool: &CellGroupCheckpointPool) -> Result<()> { + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage().sources != 0 || transport.usage().data_items != 0 { + tokio::task::yield_now().await; + } + }) + .await?; + Ok(()) +} diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 4d29111c0..563e86ef4 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1589,7 +1589,8 @@ "crates/cellar/tests/support/cell_group_controller_checkpoints.rs", "crates/cellar/tests/support/cell_group_membership.rs", "crates/cellar/tests/support/cell_group_seed.rs", - "crates/cellar/tests/support/cell_group_seed_install.rs" + "crates/cellar/tests/support/cell_group_seed_install.rs", + "crates/cellar/tests/support/cell_group_seed_probes.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 7eb32ff2976544445b02b439c20d21a72259fe11 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 21:28:51 -0700 Subject: [PATCH 138/568] docs: record cell-group seed fault and reconnect qualification --- docs/design/multi-raft/cell-groups/README.md | 6 ++- docs/design/multi-raft/cell-groups/handoff.md | 34 +++++++++---- .../design/multi-raft/cell-groups/progress.md | 50 +++++++++++++++++++ 3 files changed, 79 insertions(+), 11 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 9f5ad5af8..00cbdfa8f 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -12,8 +12,10 @@ pass their scoped regression gates. Durable joint membership and leadership hand also pass a partial gate. A learner seed source authorized by a fresh quorum read now covers empty groups, bounded capture of complete roots and stream revocation. Authenticated destination staging/install, caught-up promotion and replacement -after deleting a voter disk now pass native tests in both image forms. Remaining -CG-05 crash/pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. +after deleting a voter disk now pass native tests in both image forms. Seed request +refusal, revoked/expired sessions, retained-peer reconnection and minority commit/read +refusal also pass. Remaining CG-05 receiver faults, crash/pressure and full acceptance +gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index d5f54666b..aa64a1163 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,24 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. -The latest learner-install checkpoint at `ab2a308a` passes 238 Rust cases, +The latest seed-fault/reconnect checkpoint at `de552f6d` passes 238 Rust +cases and eight commands. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-fault-reconnect/` binds 321 source hashes +and five copied executables. Both image forms refuse eleven invalid seed requests, +five post-reply source faults and revoked/expired destination sessions without +minting startup permission. After retained cold recovery, fresh runtime/transport/ +controller owners reconnect over TLS with a new process generation, regain quorum +and commit. Isolation cannot advance applied state or satisfy a linearizable read; +after stepdown new proposals refuse. This is one OS process with explicit routes +and clock advancement, not managed fleet or process-kill qualification. + +Next cover seed reply/chunk faults at the receiver, process-generation revocation, +sync-error/process-kill and joint-transition cuts, minority signed receipts, +large/empty group bounds and all five exact CG-05 cases. Source capture fairness +and controller pressure remain open. Shared RustFS/load/power-loss/zero-quorum +proof is not supplied by these native tests. + +The earlier learner-install checkpoint at `ab2a308a` passes 238 Rust cases, eight gate commands and 37 validator tests. Its record under `.cellar/cell-groups-evidence/CG-05-learner-install-final/` binds 320 source hashes and five copied executables. Authenticated seed exchange, permanent claims, non-serving @@ -53,11 +70,9 @@ leadership and commit new state. Cold recovery checks both retained and current closures. Held-sync receipt refusal and materialized caller cancellation pass. The tests caught and fixed noncanonical configuration equality after joint leave. -Next complete seed-specific authentication/revocation faults, sync-error/process-kill -and joint-transition cuts, restarted-peer reconnection, minority receipt refusal, -large/empty group bounds and all five exact CG-05 cases. Preserve the original -acceptance requirements. Source capture fairness and controller pressure remain -open; no managed fleet, shared RustFS/load/power-loss/zero-quorum proof is claimed. +The newer checkpoint above extends this installation slice with source wire faults, +revoked/expired destination sessions, retained-peer reconnection and minority +commit/read refusal. Preserve the remaining acceptance requirements listed above. The earlier seed-source checkpoint at `0c3769ec` passes 236 selected Rust cases, eight gate commands and 37 validator tests. Its record under @@ -718,9 +733,10 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented fresh-learner replacement path. Extend `group_snapshot_transport/seed/` and its production - controller fixture with seed-specific malformed/replayed/wrong-process traffic, - process revocation, sync failures and process-kill cuts. Reconnect a restarted - joined member, prove minority write/receipt refusal and joint-transition restart. + controller fixture with malformed/replayed seed replies/chunks at the receiver, + process-generation revocation, sync failures and process-kill cuts. Qualify + minority signed receipt refusal and joint-transition restart. Native retained + peer reconnection and minority commit/read refusal now pass in both image forms. Preserve permanent claims, local install receipts and complete physical recovery. 2. Complete CG-05 qualification around the delivered membership features. Preserve all five exact acceptance cases. Add joint-transition crash cuts, diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 4a1088b76..4f2117a40 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,56 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 seed refusal and restarted voter reconnection + +Qualified source: `de552f6daabf786c14fe9df97565538ba5d302b4`. All 238 Rust cases and eight gate commands +pass, including all-target Clippy for Cellar/core/LTX, five native suites, +formatting/layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. `.cellar/cell-groups-evidence/CG-05-seed-fault-reconnect/checkpoint.json` +binds 321 source hashes, five copied executables, frozen plan inputs and +retained intermediate logs. Independent hash verification passes. These strengthen +two existing replacement cases; the test count is unchanged. CG-05 remains IN_PROGRESS. + +Both image modes now exercise independent wire probes against the production TLS +seed source. Eleven invalid requests cover exporter authentication, stale process, +wrong disk/incarnation/schema, zero and noncanonical nonce, zero and excessive +timeout, unknown fields and an ineligible voter. Five post-reply scenarios cover +wrong Ready sequence, a previous exchange's binding, unknown Ready fields, source +endpoint revocation and receiver disconnect. The probe independently verifies reply +framing, exporter-bound MAC, correlation and digest. Every scenario releases the +source pin and wire admission. A normal complete transfer later succeeds in the +same fixture, so refusal is not inferred from an unusable source. + +Production destination sessions also refuse source revocation after the signed +descriptor and expiry while the user retains the session. Neither changes the +staged root, installs a checkpoint or grants joined-member bootstrap. The same +engine and identities then complete installation and replacement. + +After cold recovery, the replacement now starts a new runtime, transport and +controller with a new process generation. Explicit physical routes remove retired +voter 3 and advertise the new endpoint. Survivors reconnect over TLS, the replacement +regains quorum and commits another cell. A linearizable read succeeds. Isolating +that group then leaves a deliberately stale ready flag: a proposal is admitted, +but cannot advance applied state or the selected root, and a fresh linearizable +read expires. After check-quorum stepdown, new proposals refuse. Admission remains +distinct from commitment. The fixture uses no event pumping after controller start. + +Two intermediate failures were fixture mistakes, retained in the evidence: the +first route still advertised retired voter 3; the second advanced only the restarted +voter's clock while survivor leader leases remained frozen. The final fixture +advances every voter and uses normal caught-up leadership handoff if a survivor +wins first. No production check or safety assertion was weakened. + +**Reflection and next step:** native reconnect and minority commit/read refusal are +now covered. Next test malformed/replayed seed replies and chunks at the receiver, +source process-generation revocation during admitted work, and sync failures or +process kills across seed installation and joint configuration. Signed receipt +refusal, large/empty group pressure, retained pins, sustained capture fairness and +the five exact CG-05 acceptance cases remain open. These tests restart runtime +owners in one OS process; they do not qualify managed fleet orchestration, +independent-process failure, power loss, shared-group RustFS/load or zero-quorum +recovery. Complete CG-05 before CG-06 archive/reclamation and CG-08/CG-07 cutover. + ## CG-05 authenticated learner installation and replacement Qualified source: `ab2a308a937f349d3482a825d4b669d042cb03c3`. All 238 selected Rust cases and eight From 3db498eacf4bd0621b7a0ecdf11ee68fe2a3ed32 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 21:47:51 -0700 Subject: [PATCH 139/568] fix: end checkpoint streams when physical verification fails --- .../raft/group_snapshot_transport/receive.rs | 127 ++-- .../group_snapshot_transport/seed/install.rs | 142 +++-- .../tests/native_cell_groups_execution.rs | 10 + .../support/cell_group_network_probes.rs | 30 +- .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_install.rs | 7 +- .../tests/support/cell_group_seed_receiver.rs | 573 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 4 + .../multi-raft/cell-groups/protocols.md | 7 + docs/design/multi-raft/cell-groups/tasks.json | 3 +- 10 files changed, 777 insertions(+), 128 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_seed_receiver.rs diff --git a/crates/cellar/durability/raft/group_snapshot_transport/receive.rs b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs index 5a760b235..496ba3c9b 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/receive.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/receive.rs @@ -75,72 +75,85 @@ impl State { self.checkpoint_current(&endpoint, &start.route)?; let receiver = workers::receiver(self, endpoint.clone(), checkpoints, admission.clone())?; let mut sequence = 0_u64; - loop { - self.checkpoint_current(&endpoint, &start.route)?; - let head = self - .snapshot_io( - &pool, - deadline, - snapshots::read_bulk_head( - stream, - None, - DATA_MAGIC, - CHUNK_BYTES.min(self.options.snapshot_chunk_bytes), - &self.options, - ), - ) + let mut result = receiver.result; + let receiving = async { + loop { + self.checkpoint_current(&endpoint, &start.route)?; + let head = self + .snapshot_io( + &pool, + deadline, + snapshots::read_bulk_head( + stream, + None, + DATA_MAGIC, + CHUNK_BYTES.min(self.options.snapshot_chunk_bytes), + &self.options, + ), + ) + .await?; + let packet: Packet = wire.decode(DATA_MAGIC, DATA, &head, Some(&binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay); + } + let mut bytes = vec![0; head.payload_len]; + self.snapshot_io(&pool, deadline, async { + stream.read_exact(&mut bytes).await?; + Ok(()) + }) .await?; - let packet: Packet = wire.decode(DATA_MAGIC, DATA, &head, Some(&binding))?; - if packet.sequence != sequence { - return Err(TransportError::Replay); - } - let mut bytes = vec![0; head.payload_len]; - self.snapshot_io(&pool, deadline, async { - stream.read_exact(&mut bytes).await?; - Ok(()) - }) - .await?; - if digest(&bytes) != packet.digest { - return Err(TransportError::Authentication); - } - self.checkpoint_current(&endpoint, &start.route)?; - let input = match packet.part { - Some(part) => { - if bytes.is_empty() { - return Err(TransportError::Protocol); - } - workers::Input::Chunk(Chunk { - part, - offset: packet.offset, - length: packet.length, - bytes, - }) + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication); } - None => { - if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { - return Err(TransportError::Protocol); + self.checkpoint_current(&endpoint, &start.route)?; + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) } - workers::Input::End + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.snapshot_io(&pool, deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; } - }; - let ending = matches!(input, workers::Input::End); - self.snapshot_io(&pool, deadline, async { - receiver - .input - .send(input) - .await - .map_err(|_| TransportError::Closed) - }) - .await?; - if ending { - break; + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; } - sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + Ok::<(), TransportError>(()) + }; + // Verification runs on a physical worker. Its failure must also end a + // peer that keeps the next header or body open without an explicit End. + crate::asyncrt::select_biased! { "physical checkpoint verification failure precedes more network input"; + outcome = &mut result => { + outcome.map_err(|_| TransportError::Closed)??; + return Err(TransportError::Protocol); + }, + outcome = receiving => outcome?, } drop(receiver.input); let prepared = self .snapshot_io(&pool, deadline, async { - receiver.result.await.map_err(|_| TransportError::Closed)? + result.await.map_err(|_| TransportError::Closed)? }) .await?; self.checkpoint_current(&endpoint, &start.route)?; diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs index 27147262f..6c2d15895 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs @@ -28,83 +28,99 @@ impl Exchange { options: &state.options, max_body: CHUNK_BYTES, }; - let head = wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 0 }, &[])?; - state - .seed_process_io(&self.grant.process, self.grant.deadline, async { - write_head(&mut self.stream, &head).await?; - self.stream.flush().await?; - Ok(()) - }) - .await?; - let mut sequence = 0_u64; - loop { - self.grant.check_log(log)?; - let head = state - .seed_process_io( - &self.grant.process, - self.grant.deadline, - snapshots::read_bulk_head( - &mut self.stream, - None, - CHUNK_MAGIC, - CHUNK_BYTES.min(state.options.snapshot_chunk_bytes), - &state.options, - ), - ) - .await?; - let packet: Packet = wire.decode(CHUNK_MAGIC, CHUNK, &head, Some(&self.binding))?; - if packet.sequence != sequence { - return Err(TransportError::Replay.into()); - } - let mut bytes = vec![0; head.payload_len]; + let mut result = receiver.result; + let receiving = async { + let head = wire.encode(READY_MAGIC, READY, &self.binding, &Ack { sequence: 0 }, &[])?; state .seed_process_io(&self.grant.process, self.grant.deadline, async { - self.stream.read_exact(&mut bytes).await?; + write_head(&mut self.stream, &head).await?; + self.stream.flush().await?; Ok(()) }) .await?; - if digest(&bytes) != packet.digest { - return Err(TransportError::Authentication.into()); - } - let input = match packet.part { - Some(part) => { - if bytes.is_empty() { - return Err(TransportError::Protocol.into()); - } - workers::Input::Chunk(Chunk { - part, - offset: packet.offset, - length: packet.length, - bytes, + let mut sequence = 0_u64; + loop { + self.grant.check_log(log)?; + let head = state + .seed_process_io( + &self.grant.process, + self.grant.deadline, + snapshots::read_bulk_head( + &mut self.stream, + None, + CHUNK_MAGIC, + CHUNK_BYTES.min(state.options.snapshot_chunk_bytes), + &state.options, + ), + ) + .await?; + let packet: Packet = wire.decode(CHUNK_MAGIC, CHUNK, &head, Some(&self.binding))?; + if packet.sequence != sequence { + return Err(TransportError::Replay.into()); + } + let mut bytes = vec![0; head.payload_len]; + state + .seed_process_io(&self.grant.process, self.grant.deadline, async { + self.stream.read_exact(&mut bytes).await?; + Ok(()) }) + .await?; + if digest(&bytes) != packet.digest { + return Err(TransportError::Authentication.into()); } - None => { - if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { - return Err(TransportError::Protocol.into()); + let input = match packet.part { + Some(part) => { + if bytes.is_empty() { + return Err(TransportError::Protocol.into()); + } + workers::Input::Chunk(Chunk { + part, + offset: packet.offset, + length: packet.length, + bytes, + }) } - workers::Input::End + None => { + if !bytes.is_empty() || packet.offset != 0 || packet.length != 0 { + return Err(TransportError::Protocol.into()); + } + workers::Input::End + } + }; + let ending = matches!(input, workers::Input::End); + self.grant.check_log(log)?; + state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + receiver + .input + .send(input) + .await + .map_err(|_| TransportError::Closed) + }) + .await?; + if ending { + break; } - }; - let ending = matches!(input, workers::Input::End); - self.grant.check_log(log)?; - state - .seed_process_until(&self.grant.process, self.grant.deadline, async { - receiver - .input - .send(input) - .await - .map_err(|_| TransportError::Closed) - }) - .await?; - if ending { - break; + sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; } - sequence = sequence.checked_add(1).ok_or(TransportError::Limit)?; + Ok::<(), CellGroupSeedError>(()) + }; + // A physical verifier can reject a part while the peer keeps the next + // header/body open. Observe it throughout network work rather than + // retaining the connection and admission until the peer deadline. + crate::asyncrt::select_biased! { "physical seed verification failure precedes more network input"; + outcome = &mut result => { + outcome.map_err(|_| TransportError::Closed)??; + // Successful physical completion requires the explicit End + // sent by the receiving branch, which completes in that poll. + return Err(TransportError::Protocol.into()); + }, + outcome = receiving => outcome?, } drop(receiver.input); let prepared = state .seed_process_until(&self.grant.process, self.grant.deadline, async { - receiver.result.await.map_err(|_| TransportError::Closed)? + result.await.map_err(|_| TransportError::Closed)? }) .await?; self.grant.check_log(log)?; diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 400c37e6b..2f4c01753 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,16 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_receiver_materialized_refuses_forged_transfers_and_revoked_process() -> Result<()> { + Box::pin(runtime_fixture::seed_receiver_faults(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_receiver_extents_refuse_forged_transfers_and_revoked_process() -> Result<()> { + Box::pin(runtime_fixture::seed_receiver_faults(true)).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_install_materialized_replaces_deleted_voter_and_resumes() -> Result<()> { Box::pin(runtime_fixture::seed_install(false)).await diff --git a/crates/cellar/tests/support/cell_group_network_probes.rs b/crates/cellar/tests/support/cell_group_network_probes.rs index 6f7ff8d7f..d88dc046f 100644 --- a/crates/cellar/tests/support/cell_group_network_probes.rs +++ b/crates/cellar/tests/support/cell_group_network_probes.rs @@ -9,8 +9,8 @@ use tokio::io::{AsyncReadExt, AsyncWriteExt}; use tokio::net::TcpStream; use tokio_rustls::client::TlsStream; -pub(super) struct Connection { - stream: TlsStream, +pub(super) struct Connection> { + stream: S, pub(super) exporter: [u8; 32], } @@ -48,6 +48,18 @@ impl Connection { .export_keying_material([0; 32], label, None)?; Ok(Self { stream, exporter }) } +} + +impl Connection { + pub(super) fn from_stream(stream: S, exporter: [u8; 32]) -> Self { + Self { stream, exporter } + } + + pub(super) async fn raw(&mut self, bytes: &[u8]) -> Result<()> { + self.stream.write_all(bytes).await?; + self.stream.flush().await?; + Ok(()) + } pub(super) async fn frame( &mut self, @@ -151,7 +163,7 @@ pub(super) async fn probe( "schema":route.schema_digest, "source":peer(&route.peers[0]), "target":peer(&route.peers[2]) }, "digest": <[u8; 32]>::from(Sha256::digest(&body))}); let address = fixture.transports[2].data_address(); - for case in 0..7 { + for case in 0..8 { let mut connection = Connection::open(address).await?; let mut header = start.clone(); match case { @@ -184,10 +196,18 @@ pub(super) async fn probe( } else { json!({"kind":"Header"}) }; + // A nonempty Header with an invalid offset reaches the physical + // verifier. Keep the socket open without End: worker failure must + // close admission without waiting for another network packet. + let bytes: &[u8] = if case == 7 { &[0] } else { &[] }; + let offset = usize::from(case == 7); connection.frame(b"CLGD", b"cellar-group-snapshot-data-1", &binding, - json!({"sequence":sequence, "part":part, "offset":0, "length":0, "digest":<[u8; 32]>::from(Sha256::digest([]))}), &[]).await?; + json!({"sequence":sequence, "part":part, "offset":offset, "length":bytes.len(), "digest":<[u8; 32]>::from(Sha256::digest(bytes))}), bytes).await?; } - connection.refused().await?; + connection + .refused() + .await + .with_context(|| format!("checkpoint fault {case}"))?; tokio::time::timeout(Duration::from_secs(5), async { while pool.usage().sources != 0 || fixture.transports[2].usage().data_items != 0 { tokio::task::yield_now().await; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 92e3491c4..6c75d3dae 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -817,3 +817,5 @@ pub(super) use sql::{membership_lifecycle, membership_sync}; pub(super) use sql::{seed_minority, seed_source}; pub(super) use sql::seed_install; + +pub(super) use sql::seed_receiver_faults; diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs index 50d8a0440..381b648c9 100644 --- a/crates/cellar/tests/support/cell_group_seed_install.rs +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -11,7 +11,7 @@ use raft::{ }; use std::path::Path; -fn target(history: GroupHistory) -> Result> { +pub(super) fn target(history: GroupHistory) -> Result> { Ok(ReplicaIdentity { scope: history, node: NodeId::new([4; 16])?, @@ -21,7 +21,10 @@ fn target(history: GroupHistory) -> Result> { }) } -fn stores(base: &Path, rooted: bool) -> Result<(ManifestPageStore, CellGroupApplyImages)> { +pub(super) fn stores( + base: &Path, + rooted: bool, +) -> Result<(ManifestPageStore, CellGroupApplyImages)> { let filesystem = Arc::new(DirectFileSystem); let pages = ManifestPageStore::open( &base.join("pages"), diff --git a/crates/cellar/tests/support/cell_group_seed_receiver.rs b/crates/cellar/tests/support/cell_group_seed_receiver.rs new file mode 100644 index 000000000..82e1a359b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_receiver.rs @@ -0,0 +1,573 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An independent authenticated peer sends adversarial replies to the production receiver. +use super::super::network_probes::Connection; +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use protobuf::Message as _; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use serde_json::{json, Value}; +use tokio::net::TcpListener; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Fault { + Exporter, + Correlation, + Nonce, + Process, + Digest, + Barrier, + Decimal, + ExtraReply, + Descriptor, + ReplyLimit, + Binding, + Sequence, + ChunkDigest, + ExtraChunk, + Offset, + EarlyEnd, + ChunkLimit, + Disconnect, + Replay, + EndOffset, + EndBody, + Valid, + Revoked, +} + +impl Fault { + fn reply(self) -> bool { + matches!( + self, + Self::Exporter + | Self::Correlation + | Self::Nonce + | Self::Process + | Self::Digest + | Self::Barrier + | Self::Decimal + | Self::ExtraReply + | Self::Descriptor + | Self::ReplyLimit + ) + } +} + +fn expected_error(fault: Fault, error: &CellGroupSeedError) -> bool { + use CellGroupSeedError::Transport; + use TransportError::*; + matches!( + (fault, error), + ( + Fault::Exporter + | Fault::Correlation + | Fault::Nonce + | Fault::Process + | Fault::Digest + | Fault::Binding + | Fault::ChunkDigest, + Transport(Authentication) + ) | ( + Fault::Decimal + | Fault::ExtraReply + | Fault::Descriptor + | Fault::ExtraChunk + | Fault::EndOffset + | Fault::EndBody, + Transport(Protocol) + ) | (Fault::ReplyLimit | Fault::ChunkLimit, Transport(Limit)) + | (Fault::Sequence | Fault::Replay, Transport(Replay)) + | (Fault::Offset | Fault::EarlyEnd, Transport(Invalid(_))) + | (Fault::Disconnect, Transport(Io(_) | Closed)) + | (Fault::Barrier, CellGroupSeedError::Checkpoint(_)) + | (Fault::Revoked, Transport(StalePeer | Closed)) + ) +} + +struct Part { + header: Value, + bytes: Vec, +} + +struct Capture { + body: Vec, + barrier: u64, + parts: Vec, +} + +async fn capture(fixture: &RuntimeCluster, history: GroupHistory) -> Result> { + let authority = fixture.runtimes[0] + .seed_authority( + seed_install::target(history)?, + reads::deadline(fixture, 10_000).await?, + ) + .await?; + let barrier = authority.barrier().index.get(); + fixture.cluster.domain.blocking(move || { + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = authority.capture(&pool)?; + let body = source.snapshot()?.write_to_bytes()?; + let mut parts = Vec::new(); + let mut retained = 0; + // Only this small adversarial test peer buffers a captured cut. Production + // transfer remains streamed; bound this fixture before retaining bytes. + source.stream(|chunk| { + retained += chunk.bytes.len(); + assert!(retained < 2 * 1024 * 1024); + let part = match chunk.part { + CellGroupCheckpointPart::Header => json!({"kind":"Header"}), + CellGroupCheckpointPart::Page(digest) => json!({"kind":"Page", "digest":digest}), + CellGroupCheckpointPart::Materialized(cell) => json!({"kind":"Materialized", "cell":cell.as_str()}), + CellGroupCheckpointPart::ExtentManifest(cell) => json!({"kind":"ExtentManifest", "cell":cell.as_str()}), + CellGroupCheckpointPart::Extent { cell, ordinal, id } => json!({"kind":"Extent", "cell":cell.as_str(), "ordinal":ordinal, "digest":id.bytes()}), + }; + parts.push(Part { header: json!({"sequence":parts.len(), "part":part, + "offset":chunk.offset, "length":chunk.length, "digest":<[u8;32]>::from(Sha256::digest(chunk.bytes))}), bytes:chunk.bytes.to_vec() }); + Ok(()) + })?; + Ok(Arc::new(Capture { body, barrier, parts })) + })?.await? +} + +fn acceptor() -> Result { + let mut config = rustls::ServerConfig::builder_with_provider(Arc::new( + rustls::crypto::ring::default_provider(), + )) + .with_protocol_versions(&[&rustls::version::TLS13])? + .with_no_client_auth() + .with_single_cert( + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?; + config.alpn_protocols = vec![b"cellar-raft-1".to_vec()]; + Ok(tokio_rustls::TlsAcceptor::from(Arc::new(config))) +} + +async fn serve(listener: Arc, cut: Arc, fault: Fault) -> Result<()> { + tokio::time::timeout(Duration::from_secs(10), async { + let (tcp, _) = listener.accept().await?; + let stream = acceptor()?.accept(tcp).await?; + let exporter = stream.get_ref().1.export_keying_material( + [0; 32], + b"EXPORTER-cellar-group-seed-1", + None, + )?; + let mut wire = Connection::from_stream(stream, exporter); + let (request, _, request_tag) = wire + .receive(b"CLGJ", b"cellar-group-seed-request-1", &[]) + .await?; + let mut route = request["route"].clone(); + route["source"] = request["route"]["target"].clone(); + route["target"] = request["route"]["source"].clone(); + let mut body = cut.body.clone(); + let mut reply = json!({"route":route, "request":request_tag, "nonce":request["nonce"], + "barrier":cut.barrier.to_string(), "digest":<[u8;32]>::from(Sha256::digest(&body))}); + match fault { + Fault::Exporter => wire.exporter[0] ^= 1, + Fault::Correlation => reply["request"] = json!(vec![0; 32]), + Fault::Nonce => reply["nonce"] = json!("999999"), + Fault::Process => reply["route"]["source"]["process"] = json!(vec![244; 16]), + Fault::Digest => reply["digest"] = json!(vec![0; 32]), + Fault::Barrier => reply["barrier"] = json!(u64::MAX.to_string()), + Fault::Decimal => reply["barrier"] = json!("01"), + Fault::ExtraReply => reply["extra"] = json!(true), + Fault::Descriptor => { + body = vec![255]; + reply["digest"] = json!(<[u8; 32]>::from(Sha256::digest(&body))); + } + Fault::ReplyLimit => { + oversized(&mut wire, b"CLGK").await?; + return wire.refused().await; + } + _ => (), + } + let binding = wire + .frame( + b"CLGK", + b"cellar-group-seed-grant-1", + &request_tag, + reply, + &body, + ) + .await?; + if fault.reply() || fault == Fault::Revoked { + return wire.refused().await; + } + let (ready, body, _) = wire + .receive(b"CLGL", b"cellar-group-seed-ready-1", &binding) + .await?; + assert_eq!(ready, json!({"sequence":0})); + assert!(body.is_empty()); + if fault == Fault::ChunkLimit { + oversized(&mut wire, b"CLGM").await?; + return wire.refused().await; + } + if fault == Fault::EarlyEnd { + wire.frame(b"CLGM", b"cellar-group-seed-chunk-1", &binding, end(0), &[]) + .await?; + return wire.refused().await; + } + for (i, part) in cut.parts.iter().enumerate() { + let mut header = part.header.clone(); + let mut bound = binding; + if i == 0 { + match fault { + Fault::Binding => bound[0] ^= 1, + Fault::Sequence => header["sequence"] = json!(1), + Fault::ChunkDigest => header["digest"] = json!(vec![0; 32]), + Fault::ExtraChunk => header["extra"] = json!(true), + Fault::Offset => header["offset"] = json!(1), + _ => (), + } + } + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &bound, + header, + &part.bytes, + ) + .await?; + if i == 0 && fault == Fault::Disconnect { + return Ok(()); + } + if i == 0 + && matches!( + fault, + Fault::Binding + | Fault::Sequence + | Fault::ChunkDigest + | Fault::ExtraChunk + | Fault::Offset + | Fault::Replay + ) + { + if fault == Fault::Replay { + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &binding, + part.header.clone(), + &part.bytes, + ) + .await?; + } + return wire.refused().await; + } + } + let mut header = end(cut.parts.len()); + let body = if fault == Fault::EndBody { + vec![1] + } else { + vec![] + }; + if fault == Fault::EndOffset { + header["offset"] = json!(1); + } + header["digest"] = json!(<[u8; 32]>::from(Sha256::digest(&body))); + wire.frame( + b"CLGM", + b"cellar-group-seed-chunk-1", + &binding, + header, + &body, + ) + .await?; + wire.refused().await // The protocol has no install acknowledgement on this stream. + }) + .await? +} + +fn end(sequence: usize) -> Value { + json!({"sequence":sequence, "part":null, "offset":0, "length":0, "digest":<[u8;32]>::from(Sha256::digest([]))}) +} + +async fn oversized( + wire: &mut Connection, + magic: &[u8; 4], +) -> Result<()> { + let mut prefix = [0; 20]; + prefix[..4].copy_from_slice(magic); + prefix[4..8].copy_from_slice(&1u32.to_be_bytes()); + prefix[8..12].copy_from_slice(&1u32.to_be_bytes()); + prefix[12..].copy_from_slice(&65537u64.to_be_bytes()); + wire.raw(&prefix).await +} + +pub(crate) async fn seed_receiver_faults(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + Box::pin(run(fixture, route, rooted)).await +} + +async fn run(mut fixture: RuntimeCluster, route: GroupRoute, rooted: bool) -> Result<()> { + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let cut = capture(&fixture, route.history).await?; + let listener = Arc::new(TcpListener::bind("127.0.0.1:0").await?); + let mut peer = fixture.tcp_route(route.history)?.peers[0].clone(); + peer.data_address = listener.local_addr()?; + let base = fixture.cluster._root.path().join("seed-receiver"); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + let baseline = transport.usage().tasks; + let mut applier: Option = None; + for fault in [ + Fault::Exporter, + Fault::Correlation, + Fault::Nonce, + Fault::Process, + Fault::Digest, + Fault::Barrier, + Fault::Decimal, + Fault::ExtraReply, + Fault::Descriptor, + Fault::ReplyLimit, + Fault::Binding, + Fault::Sequence, + Fault::ChunkDigest, + Fault::ExtraChunk, + Fault::Offset, + Fault::EarlyEnd, + Fault::ChunkLimit, + Fault::Disconnect, + Fault::Replay, + Fault::EndOffset, + Fault::EndBody, + Fault::Valid, + ] { + let service = fixture + .cluster + .domain + .spawn(serve(listener.clone(), cut.clone(), fault))?; + let prepared = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 5000).await?, + )) + .await; + if fault.reply() { + let error = prepared + .err() + .with_context(|| format!("{fault:?} accepted descriptor"))?; + assert!(expected_error(fault, &error), "{fault:?}: {error:?}"); + assert!( + matches!(log.cell_group(&route.history), Err(LogError::NotFound)), + "{fault:?} created a stage" + ); + } else { + let session = prepared.with_context(|| format!("prepare {fault:?}"))?; + if applier.is_none() { + let group = session.group().clone(); + let (pages, images) = seed_install::stores(&base, rooted)?; + let (owner, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker().root().cells(), 0); + applier = Some(owner); + } + let result = session.install(applier.as_ref().unwrap().clone()).await; + if fault == Fault::Valid { + let (_permit, root) = result?; + assert_eq!(root.marker().root().cells(), 2); + } else { + let error = result + .err() + .with_context(|| format!("{fault:?} installed"))?; + assert!(expected_error(fault, &error), "{fault:?}: {error:?}"); + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.context("stage")?.root().cells(), 0); + assert!(group.checkpoint_manifest()?.is_none()); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err()); + } + } + service + .await? + .with_context(|| format!("independent peer {fault:?}"))?; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await + .with_context(|| format!("release {fault:?}"))?; + assert_eq!(transport.usage().data_items, 0); + } + // Capture a later valid cut, then replace the source process while its signed + // grant is admitted at the destination but before installation begins. + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes( + &route, + &history(&route, "revoked-later-cut")?, + &bytes, + rooted, + )?, + ) + .await?; + controller::applied(&fixture, route.history, proposal.position.index).await?; + let later = capture(&fixture, route.history).await?; + let before = log.cell_group(&route.history)?.group_manifest()?; + let checkpoint = log.cell_group(&route.history)?.checkpoint_manifest()?; + let service = fixture + .cluster + .domain + .spawn(serve(listener, later, Fault::Revoked))?; + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 5000).await?, + )) + .await?; + controllers.remove(0).close().await?; + let new_process = ProcessGeneration::new([99; 16])?; + let options = ReplicaLogOptions::new( + fixture.cluster._root.path().join("node-1/log"), + peer.replica.node, + peer.replica.disk, + new_process, + LogOpenMode::Resume, + ); + fixture.cluster.voters[0].log.close().await?; + let restarted = fixture + .cluster + .domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + let trusted = RaftPeerNode::from(&peer); + let mut latest = trusted.clone(); + latest.process = new_process; + let view = fixture + .cluster + .domain + .run(async { + let catalog = Catalog::new(fixture.cluster.store.clone(), CatalogOptions::default())?; + let namespace = catalog + .open_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + )) + .await?; + let registration = namespace + .register_node_storage(&restarted, Duration::from_secs(5)) + .await?; + Ok::<_, anyhow::Error>( + registration + .publication(latest, Duration::from_secs(5)) + .await? + .publish(Duration::from_secs(5)) + .await?, + ) + }) + .await??; + transport.accept_node_view(view, &trusted)?; + let error = session + .install(applier.clone().unwrap()) + .await + .err() + .context("revoked grant installed")?; + assert!( + expected_error(Fault::Revoked, &error), + "revocation: {error:?}" + ); + service.await??; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await?; + assert_eq!(log.cell_group(&route.history)?.group_manifest()?, before); + assert_eq!( + log.cell_group(&route.history)?.checkpoint_manifest()?, + checkpoint + ); + assert!(matches!( + Box::pin(transport.prepare_group_seed( + local, + peer, + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 1000).await? + )) + .await, + Err(CellGroupSeedError::Transport(TransportError::StalePeer)) + )); + restarted.close().await?; + transport.close().await; + runtime.close().await?; + drop((events, applier)); + log.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f33a428d4..e34cb76f0 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -607,3 +607,7 @@ pub(crate) use seed::{seed_minority, seed_source}; #[path = "cell_group_seed_install.rs"] mod seed_install; pub(crate) use seed_install::seed_install; + +#[path = "cell_group_seed_receiver.rs"] +mod seed_receiver; +pub(crate) use seed_receiver::seed_receiver_faults; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index b4a427c19..da30b7132 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -347,6 +347,13 @@ receive; mixed cuts and incomplete closures refuse. After verified admission, connection loss does not establish whether installation completed. Source capture pins the exact engine-retained snapshot even when the current applied root advances. +Both seed and checkpoint receivers observe the physical verifier while reading +network headers/bodies and feeding its bounded input channel. A rejected part +closes the exchange even if the peer sends no more bytes; it must not retain +network admission until the peer deadline. Successful verification still requires +the authenticated End and the matching closure. This does not cancel engine +persistence after it has been admitted or replace its actual durability fence. + Two bounded channel windows connect the network to blocking physical owners. Node/peer data budgets and source/receive slots remain owned through actual I/O, even after a connection or delivery waiter disappears. The checkpoint pool diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 563e86ef4..6c3ba8dbf 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1590,7 +1590,8 @@ "crates/cellar/tests/support/cell_group_membership.rs", "crates/cellar/tests/support/cell_group_seed.rs", "crates/cellar/tests/support/cell_group_seed_install.rs", - "crates/cellar/tests/support/cell_group_seed_probes.rs" + "crates/cellar/tests/support/cell_group_seed_probes.rs", + "crates/cellar/tests/support/cell_group_seed_receiver.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 150df15a7dccf8a07e07785a75578e8e5a72684b Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 21:59:03 -0700 Subject: [PATCH 140/568] docs: record cell-group receiver fault qualification --- docs/design/multi-raft/cell-groups/README.md | 6 ++- docs/design/multi-raft/cell-groups/handoff.md | 32 +++++++++--- .../design/multi-raft/cell-groups/progress.md | 52 +++++++++++++++++++ 3 files changed, 80 insertions(+), 10 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 00cbdfa8f..0c13227e4 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -14,8 +14,10 @@ now covers empty groups, bounded capture of complete roots and stream revocation Authenticated destination staging/install, caught-up promotion and replacement after deleting a voter disk now pass native tests in both image forms. Seed request refusal, revoked/expired sessions, retained-peer reconnection and minority commit/read -refusal also pass. Remaining CG-05 receiver faults, crash/pressure and full acceptance -gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. +refusal also pass. Receiver reply/chunk faults and pre-install process revocation +now pass; physical verifier failures promptly close blocked seed/checkpoint streams. +Remaining CG-05 queued-persistence, crash/pressure and full acceptance gates precede +verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index aa64a1163..45b2d8003 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,24 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. -The latest seed-fault/reconnect checkpoint at `de552f6d` passes 238 Rust +The latest receiver-fault checkpoint at `3db498ea` passes 167 selected Rust +cases across shared execution/catalog/storage and eight gate commands. Its record +under `.cellar/cell-groups-evidence/CG-05-seed-receiver-faults/` binds 322 source +hashes and three executables. Two new cases cover ten malformed reply and eleven +chunk/end scenarios in both image forms, followed by successful same-engine install. +A newer published NodeView revokes an admitted pre-install grant and stale source +hints. Current and retained roots remain unchanged. Both seed and ordinary +checkpoint loops now observe physical verifier failure while network input is +pending; reproduced held-open offset failures no longer retain admission until +timeout. Unchanged legacy snapshot/storage suites were not rerun in this scoped gate. + +Next qualify revocation while transferring or queued for persistence, sync-error +and process-kill installation outcomes, joint-transition restart, signed minority +receipt refusal, large/empty group bounds and all five exact CG-05 cases. Native +pre-install process revocation does not qualify the queued/entered persistence +boundary or managed fleet, independent-process, RustFS/load and zero-quorum recovery. + +The earlier seed-fault/reconnect checkpoint at `de552f6d` passes 238 Rust cases and eight commands. Its record under `.cellar/cell-groups-evidence/CG-05-seed-fault-reconnect/` binds 321 source hashes and five copied executables. Both image forms refuse eleven invalid seed requests, @@ -53,11 +70,9 @@ and commit. Isolation cannot advance applied state or satisfy a linearizable rea after stepdown new proposals refuse. This is one OS process with explicit routes and clock advancement, not managed fleet or process-kill qualification. -Next cover seed reply/chunk faults at the receiver, process-generation revocation, -sync-error/process-kill and joint-transition cuts, minority signed receipts, -large/empty group bounds and all five exact CG-05 cases. Source capture fairness -and controller pressure remain open. Shared RustFS/load/power-loss/zero-quorum -proof is not supplied by these native tests. +The newer receiver checkpoint above adds reply/chunk faults and pre-install +process revocation. The remaining queued/entered persistence and full-task gaps +are listed above. Source capture fairness and controller pressure remain open. The earlier learner-install checkpoint at `ab2a308a` passes 238 Rust cases, eight gate commands and 37 validator tests. Its record under @@ -733,8 +748,9 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented fresh-learner replacement path. Extend `group_snapshot_transport/seed/` and its production - controller fixture with malformed/replayed seed replies/chunks at the receiver, - process-generation revocation, sync failures and process-kill cuts. Qualify + controller fixture with process-generation revocation during transfer and queued + persistence, sync failures and process-kill cuts. Reply/chunk faults and + pre-install process revocation now pass in both image forms. Qualify minority signed receipt refusal and joint-transition restart. Native retained peer reconnection and minority commit/read refusal now pass in both image forms. Preserve permanent claims, local install receipts and complete physical recovery. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 4f2117a40..55b031967 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 receiver faults and physical verifier completion + +Qualified source: `3db498eacf4bd0621b7a0ecdf11ee68fe2a3ed32`. All 167 selected Rust cases pass across shared +execution, catalog and storage. All eight commands pass: all-target Clippy for +Cellar/core/LTX, those three native suites, formatting/layout, dependency isolation, +docs, source preflight and 37 plan-validator tests. The source-bound record is +`.cellar/cell-groups-evidence/CG-05-seed-receiver-faults/checkpoint.json`, with +322 source hashes, three copied executables, frozen plan inputs and retained +intermediate logs. Independent hash verification passes. The unchanged legacy +snapshot/storage suites were not repeated in this scoped gate; their earlier +results remain in the predecessor record. CG-05 remains IN_PROGRESS. + +Adversarial receive tests exposed a production resource-lifetime defect in both +seed and ordinary group-checkpoint transport: a physical worker could reject a +chunk while the network loop waited for the peer's next header. The connection +and network admission survived until timeout. Both loops now observe physical +completion throughout network reads and bounded-channel sends. Failure closes the +exchange promptly. Explicit End, complete closure verification and the real engine +fence remain necessary for installation. Work already admitted to persistence is +still owned until completion. Both original failures are retained in the evidence. + +Two new cases use independent TLS test peers in both image modes. Their payload +comes from a real quorum-authorized complete source; the deliberately small fixture +buffers less than 2 MiB of chunk payload and is not a production streaming +benchmark. Ten faulty +replies cover exporter authentication, correlation, nonce, source process, payload +digest, impossible/noncanonical barrier, unknown fields, invalid descriptor and +oversized payload. They cannot create a destination stage. Eleven chunk/end faults +cover wrong binding/sequence/digest, unknown fields, bad physical offset, premature +End, oversized chunk, disconnect after the first part, replay, and malformed End +offset/body. They cannot change the staged root, install a checkpoint or authorize +bootstrap. Physical and wire admission release after every case. A valid transfer +then installs both cells on the same engine and identities. + +The fixture captures a later cut and admits its signed descriptor, then closes and +reopens the source engine with a new process generation. Normal catalog/storage +publication produces a newer NodeView. Accepting that view revokes the pending +session; both current and retained destination roots stay unchanged and an old +source hint explicitly returns StalePeer. This is revocation before install begins, +not a process-kill or revocation during queued/entered engine persistence. The +ordinary checkpoint protocol also has a held-open malformed-offset regression. + +**Reflection and next step:** seed reply/chunk refusal and pre-install process +revocation now have direct native coverage, and verifier failure no longer retains +network admission until timeout. Next qualify revocation during transfer/queued +persistence, sync-error/process-kill installation outcomes and joint-transition +restart. Signed minority receipt refusal, large/empty group pressure, retained pins, +sustained capture fairness and all five exact CG-05 cases remain required. No +managed fleet, independent-process, shared-group RustFS/load/power-loss/zero-quorum +qualification is claimed. Complete CG-05 before verified S3 archive/reclamation +and the effects/cache plus SDK/fleet cutover tasks. + ## CG-05 seed refusal and restarted voter reconnection Qualified source: `de552f6daabf786c14fe9df97565538ba5d302b4`. All 238 Rust cases and eight gate commands From 374f981683558a11ea43628e7c94b868019dfbf9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:06:54 -0700 Subject: [PATCH 141/568] test: qualify cell-group seed persistence and revocation --- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_runtime.rs | 2 + .../support/cell_group_seed_persistence.rs | 376 ++++++++++++++++++ .../tests/support/cell_group_seed_receiver.rs | 106 +++-- crates/cellar/tests/support/cell_group_sql.rs | 4 + .../multi-raft/cell-groups/protocols.md | 8 + docs/design/multi-raft/cell-groups/tasks.json | 3 +- 7 files changed, 464 insertions(+), 45 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_seed_persistence.rs diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 2f4c01753..887c3da18 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,16 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_persistence_materialized_revocation_and_sync_outcomes() -> Result<()> { + Box::pin(runtime_fixture::seed_persistence(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_persistence_extents_revocation_and_sync_outcomes() -> Result<()> { + Box::pin(runtime_fixture::seed_persistence(true)).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_receiver_materialized_refuses_forged_transfers_and_revoked_process() -> Result<()> { Box::pin(runtime_fixture::seed_receiver_faults(false)).await diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 6c75d3dae..bc3dac183 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -819,3 +819,5 @@ pub(super) use sql::{seed_minority, seed_source}; pub(super) use sql::seed_install; pub(super) use sql::seed_receiver_faults; + +pub(super) use sql::seed_persistence; diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs new file mode 100644 index 000000000..e062d89ec --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -0,0 +1,376 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated seed grants cross the real engine queue and filesystem fence. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use cellar_logic::consensus::OperationId; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use raft::Storage; +use tokio::net::TcpListener; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Cut { + QueuedRevocation, + EnteredRevocation, + SyncError, + AfterSyncError, +} + +pub(crate) async fn seed_persistence(rooted: bool) -> Result<()> { + for cut in [ + Cut::QueuedRevocation, + Cut::EnteredRevocation, + Cut::SyncError, + Cut::AfterSyncError, + ] { + Box::pin(run(rooted, cut)) + .await + .with_context(|| format!("seed persistence {cut:?}"))?; + } + Ok(()) +} + +fn queued(log: &ReplicaLog) -> Result { + let sample = log.performance_snapshot(); + assert!(sample.available && !sample.saturated); + let queue = sample + .operations + .iter() + .find(|op| op.name == "engine_queue") + .context("queue metric")?; + Ok(queue.started - queue.succeeded - queue.failed - queue.abandoned) +} + +async fn run(rooted: bool, cut: Cut) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let mut controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let captured = seed_receiver::capture(&fixture, route.history).await?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + let listener = Arc::new(TcpListener::bind("127.0.0.1:0").await?); + let mut peer = fixture.tcp_route(route.history)?.peers[0].clone(); + peer.data_address = listener.local_addr()?; + let base = fixture.cluster._root.path().join("seed-persistence"); + let gate = fault::Gate::new(None); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + let baseline = transport.usage().tasks; + let service = fixture.cluster.domain.spawn(seed_receiver::serve( + listener, + captured, + seed_receiver::Fault::Valid, + ))?; + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + peer.clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 10_000).await?, + )) + .await?; + let group = session.group().clone(); + let staged = group.group_manifest()?.context("staged root")?; + let (pages, images) = seed_install::stores(&base, rooted).context("open destination stores")?; + let recovering = group.clone(); + let (applier, root) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(root.marker(), &staged); + drop(root); + + let release = gate.arm(match cut { + Cut::QueuedRevocation | Cut::EnteredRevocation => fault::HOLD_SYNC, + Cut::SyncError => fault::FAIL_SYNC, + Cut::AfterSyncError => fault::FAIL_AFTER_SYNC, + }); + // Hold an unrelated node revision fence ahead of the seed. No other local + // group/controller is attached, so the next queued request is this install. + let ahead = if cut == Cut::QueuedRevocation { + let writer = log.clone(); + let ahead = fixture + .cluster + .domain + .spawn(async move { writer.reserve_node_revision().await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(queued(&log)?, 0); + Some(ahead) + } else { + None + }; + let installer = applier.clone(); + let mut pending = fixture + .cluster + .domain + .spawn(async move { session.install(installer).await })?; + let mut restarted_source = None; + if matches!(cut, Cut::QueuedRevocation | Cut::EnteredRevocation) { + if cut == Cut::QueuedRevocation { + controller::wait_for(async || Ok(queued(&log)? == 1)).await?; + } else { + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&staged)); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(pool.usage().sources, 1, "physical closure remains owned"); + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await? + .is_err(), + "no receipt before the engine fence" + ); + restarted_source = Some( + seed_receiver::revoke_source(&fixture, &mut controllers, &transport, &peer, &route) + .await?, + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err(), + "revocation cannot abandon entered/queued physical work" + ); + assert_eq!(pool.usage().sources, 1); + drop(release); + } else { + // The one-shot sync error consumes the armed fault before completion. + // Retain its release guard until the result has been observed. + } + let result = pending.await?; + if let Some(ahead) = ahead { + ahead.await??; + } + match cut { + Cut::QueuedRevocation => assert!( + matches!( + result, + Err(CellGroupSeedError::Log(LogError::WrongIdentity)) + ), + "queued revoked grant: {:?}", + result.err() + ), + Cut::EnteredRevocation => { + let (_permit, installed) = result?; + assert_eq!(installed.marker(), &expected); + assert!(installed.pin_cell(&a.cell).is_some() && installed.pin_cell(&b.cell).is_some()); + } + Cut::SyncError | Cut::AfterSyncError => assert!( + matches!(result, Err(CellGroupSeedError::Log(LogError::Io(_)))), + "sync must report uncertainty: {:?}", + result.err() + ), + } + service.await??; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await?; + assert_eq!(transport.usage().data_items, 0); + if cut == Cut::QueuedRevocation { + assert_eq!( + group.group_manifest()?.as_ref(), + Some(&staged), + "failed work cannot publish cache state" + ); + assert!(group.checkpoint_manifest()?.is_none()); + } + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!( + matches!(group.group_manifest(), Err(LogError::Closed)), + "uncertain engine cannot expose a selected root" + ); + assert!(matches!(group.checkpoint_manifest(), Err(LogError::Closed))); + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await? + .is_err(), + "uncertain engine cannot grant bootstrap" + ); + assert!( + matches!( + log.persist_cells(OperationId::new(5101), vec![GroupWrite::new(&group)]) + .await, + Err(LogError::Closed) + ), + "uncertain engine refuses later writes" + ); + } + transport.close().await; + let runtime_closed = runtime.close().await; + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!( + matches!(runtime_closed, Err(RuntimeError::Storage(LogError::Closed))), + "failed engine must propagate through joined runtime close: {runtime_closed:?}" + ); + } else { + runtime_closed?; + } + drop((events, applier, group, runtime, transport)); + let closed = log.close().await; + if matches!(cut, Cut::SyncError | Cut::AfterSyncError) { + assert!(matches!(closed, Err(LogError::Io(_)))); + } else { + closed?; + } + drop(log); + + // Cold reopen resolves the on-disk outcome independently of the failed + // caller and old cache. A pre-sync error can leave either complete root; + // an error after successful sync must retain the complete authenticated seed. + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([94; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + let selected = group.group_manifest()?.context("reopened root")?; + match cut { + Cut::QueuedRevocation => assert_eq!(selected, staged), + Cut::EnteredRevocation | Cut::AfterSyncError => assert_eq!(selected, expected), + Cut::SyncError => assert!(selected == staged || selected == expected), + } + let (pages, images) = seed_install::stores(&base, rooted) + .context("reopen destination stores after joined close")?; + let recovering = group.clone(); + let (applier, recovered) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + let permit = fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log, + )) + .await?; + if selected == expected { + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&expected)); + assert!(recovered.pin_cell(&a.cell).is_some() && recovered.pin_cell(&b.cell).is_some()); + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit?, recovered.clone()).await?; + assert!( + !runtime.status(route.history).await?.ready, + "seed receipt is not quorum authority" + ); + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = + apply::enrollment_bytes(&route, &history(&route, "without-quorum")?, &bytes, rooted)?; + assert!(matches!( + runtime.propose(route.history, proposal).await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + runtime.close().await?; + drop(events); + } else { + assert!(permit.is_err()); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(recovered.marker().root().cells(), 0); + } + println!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); + drop((applier, recovered, group)); + log.close().await?; + if let Some(source) = restarted_source { + source.close().await?; + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_seed_receiver.rs b/crates/cellar/tests/support/cell_group_seed_receiver.rs index 82e1a359b..4c6fa75a7 100644 --- a/crates/cellar/tests/support/cell_group_seed_receiver.rs +++ b/crates/cellar/tests/support/cell_group_seed_receiver.rs @@ -10,7 +10,7 @@ use serde_json::{json, Value}; use tokio::net::TcpListener; #[derive(Clone, Copy, Debug, PartialEq, Eq)] -enum Fault { +pub(super) enum Fault { Exporter, Correlation, Nonce, @@ -90,13 +90,16 @@ struct Part { bytes: Vec, } -struct Capture { +pub(super) struct Capture { body: Vec, barrier: u64, parts: Vec, } -async fn capture(fixture: &RuntimeCluster, history: GroupHistory) -> Result> { +pub(super) async fn capture( + fixture: &RuntimeCluster, + history: GroupHistory, +) -> Result> { let authority = fixture.runtimes[0] .seed_authority( seed_install::target(history)?, @@ -149,7 +152,11 @@ fn acceptor() -> Result { Ok(tokio_rustls::TlsAcceptor::from(Arc::new(config))) } -async fn serve(listener: Arc, cut: Arc, fault: Fault) -> Result<()> { +pub(super) async fn serve( + listener: Arc, + cut: Arc, + fault: Fault, +) -> Result<()> { tokio::time::timeout(Duration::from_secs(10), async { let (tcp, _) = listener.accept().await?; let stream = acceptor()?.accept(tcp).await?; @@ -487,6 +494,55 @@ async fn run(mut fixture: RuntimeCluster, route: GroupRoute, rooted: bool) -> Re reads::deadline(&fixture, 5000).await?, )) .await?; + let restarted = revoke_source(&fixture, &mut controllers, &transport, &peer, &route).await?; + let error = session + .install(applier.clone().unwrap()) + .await + .err() + .context("revoked grant installed")?; + assert!( + expected_error(Fault::Revoked, &error), + "revocation: {error:?}" + ); + service.await??; + controller::wait_for(async || { + Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) + }) + .await?; + assert_eq!(log.cell_group(&route.history)?.group_manifest()?, before); + assert_eq!( + log.cell_group(&route.history)?.checkpoint_manifest()?, + checkpoint + ); + assert!(matches!( + Box::pin(transport.prepare_group_seed( + local, + peer, + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 1000).await? + )) + .await, + Err(CellGroupSeedError::Transport(TransportError::StalePeer)) + )); + restarted.close().await?; + transport.close().await; + runtime.close().await?; + drop((events, applier)); + log.close().await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await +} + +pub(super) async fn revoke_source( + fixture: &RuntimeCluster, + controllers: &mut Vec, + transport: &RaftTransport, + peer: &RaftPeer, + route: &GroupRoute, +) -> Result { controllers.remove(0).close().await?; let new_process = ProcessGeneration::new([99; 16])?; let options = ReplicaLogOptions::new( @@ -502,7 +558,7 @@ async fn run(mut fixture: RuntimeCluster, route: GroupRoute, rooted: bool) -> Re .domain .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) .await??; - let trusted = RaftPeerNode::from(&peer); + let trusted = RaftPeerNode::from(peer); let mut latest = trusted.clone(); latest.process = new_process; let view = fixture @@ -531,43 +587,5 @@ async fn run(mut fixture: RuntimeCluster, route: GroupRoute, rooted: bool) -> Re }) .await??; transport.accept_node_view(view, &trusted)?; - let error = session - .install(applier.clone().unwrap()) - .await - .err() - .context("revoked grant installed")?; - assert!( - expected_error(Fault::Revoked, &error), - "revocation: {error:?}" - ); - service.await??; - controller::wait_for(async || { - Ok(transport.usage().tasks == baseline && pool.usage().sources == 0) - }) - .await?; - assert_eq!(log.cell_group(&route.history)?.group_manifest()?, before); - assert_eq!( - log.cell_group(&route.history)?.checkpoint_manifest()?, - checkpoint - ); - assert!(matches!( - Box::pin(transport.prepare_group_seed( - local, - peer, - fixture.cluster.directory.clone(), - fixture.cluster.birth.clone(), - reads::deadline(&fixture, 1000).await? - )) - .await, - Err(CellGroupSeedError::Transport(TransportError::StalePeer)) - )); - restarted.close().await?; - transport.close().await; - runtime.close().await?; - drop((events, applier)); - log.close().await?; - for controller in controllers { - controller.close().await?; - } - fixture.close().await + Ok(restarted) } diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index e34cb76f0..4370db8f3 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -611,3 +611,7 @@ pub(crate) use seed_install::seed_install; #[path = "cell_group_seed_receiver.rs"] mod seed_receiver; pub(crate) use seed_receiver::seed_receiver_faults; + +#[path = "cell_group_seed_persistence.rs"] +mod seed_persistence; +pub(crate) use seed_persistence::seed_persistence; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index da30b7132..67100708e 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -273,6 +273,14 @@ snapshot installation cannot mint it. No old-format reader or user-data deletion is introduced. Live grant and exact engine/process checks run before entering the fence. Once physical persistence begins, cancellation and close retain the grant, image owners, task reservation and consensus lease until actual completion. +A queued install rechecks the grant on the engine worker before its fence and +refuses a newly revoked source without publishing the prepared root or seed receipt. +Revocation after entering the fence cannot roll back already issued writes. A +successful fence may retain the complete authenticated seed despite that late +revocation; the local receipt still grants neither voting promotion nor current +quorum authority. A sync error leaves the live cache unpublished and closes the +engine to further mutation. Cold recovery must reconcile the selected complete +root and local receipt together; an error response alone does not prove rollback. `CellGroupApplier::recover_selected` reconciles an unknown install outcome using the existing physical store owner; selected pins prevent reopening that directory diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 6c3ba8dbf..fa915bc60 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1595,7 +1595,8 @@ ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", - "crates/cellar/tests/native_cell_groups_snapshot.rs" + "crates/cellar/tests/native_cell_groups_snapshot.rs", + "crates/cellar/tests/support/cell_group_seed_persistence.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 3ea90fb5d67d7c66aa842781788c403aa2afc80a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:09:10 -0700 Subject: [PATCH 142/568] test: report seed persistence cuts through native output --- crates/cellar/tests/support/cell_group_seed_persistence.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs index e062d89ec..2923136e6 100644 --- a/crates/cellar/tests/support/cell_group_seed_persistence.rs +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -363,7 +363,7 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { assert!(group.checkpoint_manifest()?.is_none()); assert_eq!(recovered.marker().root().cells(), 0); } - println!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); + cellar::note!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); drop((applier, recovered, group)); log.close().await?; if let Some(source) = restarted_source { From b80c8078c9c51fff80f1e256b4908ea2301569e4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:14:39 -0700 Subject: [PATCH 143/568] docs: record seed persistence recovery qualification --- docs/design/multi-raft/cell-groups/README.md | 3 +- docs/design/multi-raft/cell-groups/handoff.md | 33 +++++++++----- .../design/multi-raft/cell-groups/progress.md | 45 +++++++++++++++++++ 3 files changed, 70 insertions(+), 11 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 0c13227e4..05ec2e111 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -16,7 +16,8 @@ after deleting a voter disk now pass native tests in both image forms. Seed requ refusal, revoked/expired sessions, retained-peer reconnection and minority commit/read refusal also pass. Receiver reply/chunk faults and pre-install process revocation now pass; physical verifier failures promptly close blocked seed/checkpoint streams. -Remaining CG-05 queued-persistence, crash/pressure and full acceptance gates precede +Queued/entered seed revocation and sync-error recovery now pass in both image forms. +Remaining CG-05 process-kill, joint-restart, pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 45b2d8003..9baebe421 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,22 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent-process/power-loss and load remain unqualified. -The latest receiver-fault checkpoint at `3db498ea` passes 167 selected Rust +The latest persistence checkpoint at `3ea90fb5` passes 9 selected seed tests +and eight scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-seed-persistence/` binds 323 source hashes +and one executable. In both image forms, queued source-process revocation refuses +before the seed fence; revocation after entering the fence retains physical owners +and reconciles the complete seed. Sync errors before/after the underlying sync +close all engine admission and cold recovery verifies the complete root and local +receipt together. Recovered learners still refuse valid proposals without quorum. +Production receiver/storage code was unchanged. These are native one-process +fixtures, not process-kill, power-loss, managed fleet or RustFS qualification. + +The immediate next gap is process-kill installation and joint-transition recovery; +transfer-time revocation, signed minority receipts, pressure and the full CG-05 +acceptance gate also remain open. Broader predecessor test results are historical. + +The earlier receiver-fault checkpoint at `3db498ea` passes 167 selected Rust cases across shared execution/catalog/storage and eight gate commands. Its record under `.cellar/cell-groups-evidence/CG-05-seed-receiver-faults/` binds 322 source hashes and three executables. Two new cases cover ten malformed reply and eleven @@ -53,11 +68,9 @@ checkpoint loops now observe physical verifier failure while network input is pending; reproduced held-open offset failures no longer retain admission until timeout. Unchanged legacy snapshot/storage suites were not rerun in this scoped gate. -Next qualify revocation while transferring or queued for persistence, sync-error -and process-kill installation outcomes, joint-transition restart, signed minority -receipt refusal, large/empty group bounds and all five exact CG-05 cases. Native -pre-install process revocation does not qualify the queued/entered persistence -boundary or managed fleet, independent-process, RustFS/load and zero-quorum recovery. +The newer persistence checkpoint above adds queued/entered revocation and sync +errors. Transfer-time revocation, process-kill outcomes, joint-transition restart, +signed minority receipts, pressure and all five exact CG-05 cases remain open. The earlier seed-fault/reconnect checkpoint at `de552f6d` passes 238 Rust cases and eight commands. Its record under @@ -748,15 +761,15 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented fresh-learner replacement path. Extend `group_snapshot_transport/seed/` and its production - controller fixture with process-generation revocation during transfer and queued - persistence, sync failures and process-kill cuts. Reply/chunk faults and - pre-install process revocation now pass in both image forms. Qualify + controller fixture with process-kill install cuts and process-generation + revocation during transfer. Reply/chunk faults, pre-install and queued/entered + revocation, and sync-error recovery now pass in both image forms. Qualify minority signed receipt refusal and joint-transition restart. Native retained peer reconnection and minority commit/read refusal now pass in both image forms. Preserve permanent claims, local install receipts and complete physical recovery. 2. Complete CG-05 qualification around the delivered membership features. Preserve all five exact acceptance cases. Add joint-transition crash cuts, - receiver crash/sync-error cuts, continuous checkpoint saturation fairness, + receiver process-kill cuts, continuous checkpoint saturation fairness, exhausted apply/completion budgets, held sync/cancelled close, quarantine capacity and retained-root pressure. Keep unrelated-group progress and bounded physical ownership observable. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 55b031967..9473e6575 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,51 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 seed persistence revocation and sync outcomes + +Qualified source: `3ea90fb5d67d7c66aa842781788c403aa2afc80a`. All 9 selected seed tests pass, +including two new tests covering four persistence cuts in each image form. +Eight scoped commands pass: Clippy for the execution integration target, selected +seed tests, formatting, layout, dependency isolation, docs, source preflight and +37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-seed-persistence/checkpoint.json` binds +323 source hashes, one copied executable, frozen plan inputs and retained +intermediate logs. Independent hash verification passes. Broader predecessor +results are historical; this checkpoint does not rerun the full shared suites. + +A held node-revision fence and the actual engine queue metric establish that the +seed install is queued. Reopening the source engine with a fresh process and +publishing its NodeView through the catalog revokes the admitted grant. The +engine refuses it with WrongIdentity before publication. Both live and cold +recovered destination roots remain the empty stage, with no checkpoint or startup +receipt. The pending install retains physical receive admission until the held +engine work completes. + +A separate cut revokes the source after the seed enters the filesystem fence. +The install retains its owners until sync completes and preserves the complete +seed despite late revocation. The receipt cannot appear before the fence. +Cold recovery revalidates the selected and retained roots, both cell images and +learner membership. A recovered learner refuses a valid enrollment without quorum. + +Injected failures immediately before and after the underlying sync return typed +I/O errors. The uncertain engine refuses metadata reads, further writes and +bootstrap; joined runtime/storage shutdown reports the failure. Reopening verifies +the complete root and local receipt together. The pre-sync cut permits either +complete old or complete new selection; the post-sync error requires the new +seed. Both observed pre-sync runs recovered the seed, illustrating why failure +cannot imply rollback. This uses an independently authenticated TLS fixture fed +by a real quorum-authorized cut; production receiver/storage behavior was unchanged. + +**Reflection and next step:** queued and entered process revocation plus sync-error +reconciliation now have direct coverage in both image forms. The most critical +remaining gap is actual process-kill installation and joint-transition recovery. +Next add child-process cuts around the seed fence and verify complete recovery, +then finish transfer-time revocation, signed minority receipts, large/empty-group +pressure and all five exact CG-05 acceptance cases. No OS process kill, power loss, +managed fleet, shared-group RustFS/load or zero-quorum recovery is qualified here. +CG-05 remains IN_PROGRESS; archive/reclamation and production hard cutover remain +required downstream work. + ## CG-05 receiver faults and physical verifier completion Qualified source: `3db498eacf4bd0621b7a0ecdf11ee68fe2a3ed32`. All 167 selected Rust cases pass across shared From b997a48ba686fceb2531cd96ae2734c9bed84fc9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:21:48 -0700 Subject: [PATCH 144/568] test: recover cell-group seeds after process kills --- .../tests/native_cell_groups_execution.rs | 16 + .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_crash.rs | 396 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 4 + docs/design/multi-raft/cell-groups/tasks.json | 3 +- 5 files changed, 420 insertions(+), 1 deletion(-) create mode 100644 crates/cellar/tests/support/cell_group_seed_crash.rs diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 887c3da18..7b67bf58e 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,22 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_crash_materialized_recovers_atomic_install() -> Result<()> { + Box::pin(runtime_fixture::seed_crash(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_crash_extents_recovers_atomic_install() -> Result<()> { + Box::pin(runtime_fixture::seed_crash(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by seed crash tests"] +async fn seed_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::seed_process_fixture()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_persistence_materialized_revocation_and_sync_outcomes() -> Result<()> { Box::pin(runtime_fixture::seed_persistence(false)).await diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index bc3dac183..4bec85ed3 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -821,3 +821,5 @@ pub(super) use sql::seed_install; pub(super) use sql::seed_receiver_faults; pub(super) use sql::seed_persistence; + +pub(super) use sql::{seed_crash, seed_process_fixture}; diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs new file mode 100644 index 000000000..ccf9ed616 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -0,0 +1,396 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned process kills around an authenticated, whole-group learner install. +//! Frozen synthetic catalog bytes are fixture input, not an S3 recovery protocol. +use super::*; +use cellar::application::consensus::verification::JoiningMember; +use futures_util::TryStreamExt; +use object_store::{path::Path as ObjectPath, ObjectStore}; +use raft::prelude::{ConfChangeTransition, ConfChangeType}; +use raft::Storage; +use serde::{Deserialize, Serialize}; +use std::io::Write; +use std::path::Path; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Oracle { + rooted: bool, + cut: String, + staged: GroupManifest, + expected: GroupManifest, + objects: Vec<(String, Vec)>, +} + +struct Child(std::process::Child); + +impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +pub(crate) async fn seed_crash(rooted: bool) -> Result<()> { + for cut in ["staged", "before-sync", "after-sync", "published"] { + let directory = tempfile::tempdir()?; + let root = directory.path(); + let output = root.join("child.log"); + let file = std::fs::File::create(&output)?; + let mut child = Child( + std::process::Command::new(std::env::current_exe()?) + .args([ + "--ignored", + "--exact", + "seed_process_fixture", + "--nocapture", + "--test-threads=1", + ]) + .env("CELLAR_GROUP_SEED_CRASH_ROOT", root) + .env( + "CELLAR_GROUP_SEED_CRASH_MODE", + if rooted { "extents" } else { "materialized" }, + ) + .env("CELLAR_GROUP_SEED_CRASH_CUT", cut) + .stdout(file.try_clone()?) + .stderr(file) + .spawn()?, + ); + let wait = tokio::time::timeout(Duration::from_secs(30), async { + loop { + let reached = match std::fs::read(root.join("signal")) { + Ok(bytes) => bytes == b"reached", + Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, + Err(error) => return Err(error.into()), + }; + if reached { + break; + } + if let Some(status) = child.0.try_wait()? { + anyhow::bail!("child exited before {cut}: {status}"); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await; + let reached = wait.map_err(anyhow::Error::from).and_then(|r| r); + reached.with_context(|| { + format!( + "{rooted}/{cut}: {}", + std::fs::read_to_string(&output).unwrap_or_default() + ) + })?; + assert_eq!(std::fs::read(root.join("signal"))?, b"reached"); + child.0.kill()?; + let status = child.0.wait()?; + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + assert_eq!(status.signal(), Some(9)); + } + assert!(!status.success()); + Box::pin(recover(root, rooted, cut)) + .await + .with_context(|| format!("cold recovery {rooted}/{cut}"))?; + } + Ok(()) +} + +fn save(path: &Path, bytes: &[u8]) -> Result<()> { + let mut file = std::fs::File::create(path)?; + file.write_all(bytes)?; + file.sync_all()?; + Ok(()) +} + +async fn stop_at(root: &Path) -> Result<()> { + save(&root.join("signal"), b"reached")?; + std::future::pending::<()>().await; + unreachable!("parent kills the held child") +} + +pub(crate) async fn seed_process_fixture() -> Result<()> { + let root = std::path::PathBuf::from(std::env::var("CELLAR_GROUP_SEED_CRASH_ROOT")?); + let rooted = match std::env::var("CELLAR_GROUP_SEED_CRASH_MODE")?.as_str() { + "materialized" => false, + "extents" => true, + _ => anyhow::bail!("invalid crash image mode"), + }; + let cut = std::env::var("CELLAR_GROUP_SEED_CRASH_CUT")?; + anyhow::ensure!( + matches!( + cut.as_str(), + "staged" | "before-sync" | "after-sync" | "published" + ), + "invalid crash cut" + ); + anyhow::ensure!( + root.is_dir() && !root.join("signal").exists(), + "fresh owned parent directory required" + ); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup_in( + rooted, + &class, + tempfile::tempdir_in(&root)?, + Arc::new(DirectFileSystem), + )) + .await?; + let _controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let local = seed_install::target(route.history)?; + membership::configure( + &fixture, + &route, + membership::change( + 4, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Auto, + ), + &[JoiningMember { + member: local.member, + node: local.node, + disk: local.disk, + }], + ) + .await?; + let base = root.join("learner"); + let gate = fault::Gate::new(Some(root.join("signal"))); + let log = fixture + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let (runtime, _events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + transport.enable_group_checkpoints(pool.clone())?; + // The complete transfer uses the production source and destination over TLS. + // Learner enrollment is committed before its authenticated route is installed. + let mut physical = fixture.tcp_route(route.history)?; + physical.revision += 1; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + fixture.transports[0].enable_group_checkpoints(source_pool.clone())?; + for peer in &fixture.transports { + peer.install_physical_group(physical.clone())?; + } + let _registration = fixture.transports[0] + .register_group_checkpoint(fixture.appliers[&(0, route.history)].1.clone())?; + let session = Box::pin(transport.prepare_group_seed( + local, + physical.peers[0].clone(), + fixture.cluster.directory.clone(), + fixture.cluster.birth.clone(), + reads::deadline(&fixture, 10_000).await?, + )) + .await?; + let group = session.group().clone(); + let staged = group.group_manifest()?.context("staged root")?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + // The catalog is external to the learner disk. Freeze its synthetic objects + // before the fault, including the permanent destination claim. The parent + // never manufactures a replacement claim from the recovered disk. + let mut objects = Vec::new(); + let listed = fixture + .cluster + .store + .list(None) + .try_collect::>() + .await?; + anyhow::ensure!(listed.len() <= 128, "bounded fixture catalog"); + let mut bytes = 0; + for item in listed { + let body = fixture + .cluster + .store + .get(&item.location) + .await? + .bytes() + .await? + .to_vec(); + bytes += body.len(); + anyhow::ensure!(bytes < 1024 * 1024, "bounded fixture catalog bytes"); + objects.push((item.location.to_string(), body)); + } + save( + &root.join("oracle.json"), + &serde_json::to_vec(&Oracle { + rooted, + cut: cut.clone(), + staged, + expected: expected.clone(), + objects, + })?, + )?; + if cut == "staged" { + return stop_at(&root).await; + } + let (pages, images) = seed_install::stores(&base, rooted)?; + let (applier, empty) = fixture + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(empty.marker().root().cells(), 0); + drop(empty); + let _release = match cut.as_str() { + "before-sync" => Some(gate.arm(fault::CRASH_BEFORE_SYNC)), + "after-sync" => Some(gate.arm(fault::CRASH_AFTER_SYNC)), + _ => None, + }; + let (_permit, installed) = session.install(applier).await?; + assert_eq!( + cut, "published", + "a held physical fence cannot complete before kill" + ); + assert_eq!(installed.marker(), &expected); + stop_at(&root).await +} + +async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.cut, cut); + let route = oracle.expected.route().clone(); + let local = seed_install::target(route.history)?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let store = Arc::new(object_store::memory::InMemory::new()); + for (name, bytes) in oracle.objects { + store.put(&ObjectPath::from(name), bytes.into()).await?; + } + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = domain + .run(directory.materialize_selected_group(&route)) + .await??; + let base = root.join("learner"); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([94; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + let selected = group.group_manifest()?.context("recovered selected root")?; + match cut { + "staged" => assert_eq!(selected, oracle.staged), + "before-sync" => assert!(selected == oracle.staged || selected == oracle.expected), + "after-sync" | "published" => assert_eq!(selected, oracle.expected), + _ => unreachable!(), + } + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &selected); + let permit = domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await?; + if selected == oracle.expected { + assert_eq!( + group.checkpoint_manifest()?.as_ref(), + Some(&oracle.expected) + ); + assert_eq!(recovered.pinned_images(), 2); + for name in ["registered-a", "registered-b"] { + let cell = history(&route, name)?; + let (leaf, _) = recovered.pin_cell(&cell.cell).context("recovered cell")?; + assert_eq!(leaf.version.history, cell); + } + assert_eq!(group.initial_state()?.conf_state.learners, vec![4]); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit?, recovered.clone()).await?; + assert!(!runtime.status(route.history).await?.ready); + let bytes = images::image(None, "CREATE TABLE value(n);")?; + let proposal = + apply::enrollment_bytes(&route, &history(&route, "without-quorum")?, &bytes, rooted)?; + assert!(matches!( + runtime.propose(route.history, proposal).await, + Err(RuntimeError::NotReady) + )); + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.expected)); + runtime.close().await?; + drop(events); + } else { + assert!(permit.is_err(), "empty stage has no local seed receipt"); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(recovered.pinned_images(), 0); + } + cellar::note!( + "seed crash rooted={rooted} cut={cut} killed=true recovered_cells={} recovered_index={}", + selected.root().cells(), + selected.position().index.get() + ); + drop((applier, recovered, group)); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 4370db8f3..0002bc023 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -615,3 +615,7 @@ pub(crate) use seed_receiver::seed_receiver_faults; #[path = "cell_group_seed_persistence.rs"] mod seed_persistence; pub(crate) use seed_persistence::seed_persistence; + +#[path = "cell_group_seed_crash.rs"] +mod seed_crash; +pub(crate) use seed_crash::{seed_crash, seed_process_fixture}; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index fa915bc60..17a5d93f2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1596,7 +1596,8 @@ "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", "crates/cellar/tests/native_cell_groups_snapshot.rs", - "crates/cellar/tests/support/cell_group_seed_persistence.rs" + "crates/cellar/tests/support/cell_group_seed_persistence.rs", + "crates/cellar/tests/support/cell_group_seed_crash.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 035449f25fc1b6d45aafe08f6726d5cbbb5b71f5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:25:39 -0700 Subject: [PATCH 145/568] docs: record cell-group seed process crash qualification --- docs/design/multi-raft/cell-groups/README.md | 3 +- docs/design/multi-raft/cell-groups/handoff.md | 37 +++++++++++----- .../design/multi-raft/cell-groups/progress.md | 44 +++++++++++++++++++ 3 files changed, 73 insertions(+), 11 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 05ec2e111..68ae47d22 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -17,7 +17,8 @@ refusal, revoked/expired sessions, retained-peer reconnection and minority commi refusal also pass. Receiver reply/chunk faults and pre-install process revocation now pass; physical verifier failures promptly close blocked seed/checkpoint streams. Queued/entered seed revocation and sync-error recovery now pass in both image forms. -Remaining CG-05 process-kill, joint-restart, pressure and full acceptance gates precede +Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. +Remaining CG-05 mid-stream cuts, joint restart, pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 9baebe421..870eb01e9 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -40,9 +40,25 @@ CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. -Shared RustFS, independent-process/power-loss and load remain unqualified. - -The latest persistence checkpoint at `3ea90fb5` passes 9 selected seed tests +Shared RustFS, independent voter failure domains, power loss and load remain unqualified. + +The latest seed-process checkpoint at `b997a48b` passes 11 selected seed tests +and eight scoped commands. The record under +`.cellar/cell-groups-evidence/CG-05-seed-process-crash/` binds 324 source hashes +and one executable. Two parent tests perform eight owned SIGKILL runs across both +image forms: durable stage, before engine sync, after sync and after publication. +Production TLS source/receiver transfer real quorum-authorized seeds. Cold reopen +checks the complete selected/retained root and local receipt; a stage refuses +startup and a recovered learner refuses valid writes without quorum. + +The frozen synthetic catalog preserves pre-fault external claims only for the +fixture. It is not S3 durability or a recovery API. All source voters and the +learner share the killed child; independent voter failure domains, power loss, +mid-stream cuts, joint restart, RustFS/load and zero-quorum recovery remain open. +The next critical work is retained-disk joint-consensus restart and completion, +then the remaining bounded-pressure and exact CG-05 acceptance cases. + +The earlier persistence checkpoint at `3ea90fb5` passes 9 selected seed tests and eight scoped commands. Its record under `.cellar/cell-groups-evidence/CG-05-seed-persistence/` binds 323 source hashes and one executable. In both image forms, queued source-process revocation refuses @@ -53,9 +69,9 @@ receipt together. Recovered learners still refuse valid proposals without quorum Production receiver/storage code was unchanged. These are native one-process fixtures, not process-kill, power-loss, managed fleet or RustFS qualification. -The immediate next gap is process-kill installation and joint-transition recovery; -transfer-time revocation, signed minority receipts, pressure and the full CG-05 -acceptance gate also remain open. Broader predecessor test results are historical. +The newer seed-process checkpoint above covers the staged and engine-boundary +process-kill cases. Joint-transition recovery, mid-stream revocation/crash, signed +minority receipts, pressure and the full CG-05 gate remain open. The earlier receiver-fault checkpoint at `3db498ea` passes 167 selected Rust cases across shared execution/catalog/storage and eight gate commands. Its record @@ -761,15 +777,16 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented fresh-learner replacement path. Extend `group_snapshot_transport/seed/` and its production - controller fixture with process-kill install cuts and process-generation - revocation during transfer. Reply/chunk faults, pre-install and queued/entered - revocation, and sync-error recovery now pass in both image forms. Qualify + controller fixture with retained-disk joint-consensus restart/completion and + process-generation revocation during transfer. Reply/chunk faults, pre-install + and queued/entered revocation, sync-error recovery and staged/engine-boundary + process kills now pass in both image forms. Qualify minority signed receipt refusal and joint-transition restart. Native retained peer reconnection and minority commit/read refusal now pass in both image forms. Preserve permanent claims, local install receipts and complete physical recovery. 2. Complete CG-05 qualification around the delivered membership features. Preserve all five exact acceptance cases. Add joint-transition crash cuts, - receiver process-kill cuts, continuous checkpoint saturation fairness, + mid-stream receiver process-kill cuts, continuous checkpoint saturation fairness, exhausted apply/completion budgets, held sync/cancelled close, quarantine capacity and retained-root pressure. Keep unrelated-group progress and bounded physical ownership observable. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 9473e6575..5958fdbfb 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,50 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 seed installation across process kills + +Qualified source: `b997a48ba686fceb2531cd96ae2734c9bed84fc9`. All 11 selected seed tests pass, +including two new parent tests covering four process-kill cuts in each image form. +One ignored child fixture is invoked explicitly by each parent; it is not counted +as a standalone pass. Eight scoped commands pass: Clippy for the execution target, +seed tests, formatting, layout, dependency isolation, docs, source preflight and +37 plan-validator tests. The source-bound record +`.cellar/cell-groups-evidence/CG-05-seed-process-crash/checkpoint.json` binds +324 source hashes, one copied executable, frozen plan inputs and intermediate +logs. Independent hash verification passes. Broader predecessor suites remain +historical results, not additional current-source passes. + +Each owned child starts a real three-voter shared group with two SQLite cells, +commits a fresh learner and transfers its seed using the production TLS sender and +receiver. The parent waits for a complete reached-cut signal and kills only that +child, checks its Unix exit signal is SIGKILL, and joins it. Cuts occur after durable +staging, before the destination engine sync, after that sync and after publication. +The child and parent execute the same test binary, retained with this checkpoint. + +Cold recovery opens the original learner disk with a new process identity and +verifies the selected root plus physical closure. The staged-only cut has no +checkpoint or authenticated local receipt and refuses startup. The pre-sync cut +allows either complete old or complete new state; both observed runs retained the +new seed. Post-sync and publication cuts require the complete seed, both cells, +retained checkpoint, learner membership and local receipt. Attaching a recovered +learner still refuses a valid enrollment without quorum and cannot advance its root. + +The parent restores a bounded frozen copy of this fixture's synthetic catalog, +including the permanent destination claim captured before the fault. It never +manufactures a new claim from recovered disk contents. This fixture catalog is +not an object-store durability implementation or S3 disaster-recovery protocol. +All source voters and the destination share the killed child process. The tests +qualify these learner-install crash cuts, not independent voter domains, device +power loss, mid-stream crash cuts, joint-transition restart, RustFS or zero-quorum +recovery. Production runtime behavior was unchanged. + +**Reflection and next step:** seed engine-boundary process kills now have direct +native evidence. The next critical gap is restarting and completing joint consensus +from retained disks. Follow with mid-stream cancellation/crash and pressure coverage, +map the existing proof to all five exact CG-05 cases, and complete that gate before +CG-06 verified S3 archive/reclamation. CG-05 remains IN_PROGRESS and the full hard +cutover remains unfinished. + ## CG-05 seed persistence revocation and sync outcomes Qualified source: `3ea90fb5d67d7c66aa842781788c403aa2afc80a`. All 9 selected seed tests pass, From 5f30cc6b28ae01c2e6411507ea11716ef94e6456 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:34:52 -0700 Subject: [PATCH 146/568] test: resume explicit joint consensus after process loss --- .../tests/native_cell_groups_execution.rs | 16 + .../tests/support/cell_group_crash_fixture.rs | 143 +++++++ .../support/cell_group_membership_crash.rs | 379 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_crash.rs | 136 +------ crates/cellar/tests/support/cell_group_sql.rs | 6 + docs/design/multi-raft/cell-groups/tasks.json | 4 +- 7 files changed, 555 insertions(+), 131 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_crash_fixture.rs create mode 100644 crates/cellar/tests/support/cell_group_membership_crash.rs diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 7b67bf58e..675db6946 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,22 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_crash_materialized_recovers_and_completes_joint_consensus() -> Result<()> { + Box::pin(runtime_fixture::membership_crash(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_crash_extents_recover_and_complete_joint_consensus() -> Result<()> { + Box::pin(runtime_fixture::membership_crash(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by membership crash tests"] +async fn membership_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::membership_process_fixture()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_crash_materialized_recovers_atomic_install() -> Result<()> { Box::pin(runtime_fixture::seed_crash(false)).await diff --git a/crates/cellar/tests/support/cell_group_crash_fixture.rs b/crates/cellar/tests/support/cell_group_crash_fixture.rs new file mode 100644 index 000000000..d91f7b821 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_crash_fixture.rs @@ -0,0 +1,143 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Owned crash children and bounded synthetic catalog snapshots for native tests. +//! These objects are fixture inputs, never a production recovery protocol. +use super::*; +use futures_util::TryStreamExt; +use object_store::{path::Path as ObjectPath, ObjectStore}; +use std::{io::Write, path::Path}; + +struct Child(std::process::Child); + +impl Drop for Child { + fn drop(&mut self) { + let _ = self.0.kill(); + let _ = self.0.wait(); + } +} + +pub(super) async fn kill_at(root: &Path, fixture: &str, rooted: bool, cut: &str) -> Result<()> { + let output = root.join("child.log"); + let file = std::fs::File::create(&output)?; + let mut child = Child( + std::process::Command::new(std::env::current_exe()?) + .args([ + "--ignored", + "--exact", + fixture, + "--nocapture", + "--test-threads=1", + ]) + .env("CELLAR_GROUP_CRASH_ROOT", root) + .env( + "CELLAR_GROUP_CRASH_MODE", + if rooted { "extents" } else { "materialized" }, + ) + .env("CELLAR_GROUP_CRASH_CUT", cut) + .stdout(file.try_clone()?) + .stderr(file) + .spawn()?, + ); + let wait = tokio::time::timeout(Duration::from_secs(30), async { + loop { + let reached = match std::fs::read(root.join("signal")) { + Ok(bytes) => bytes == b"reached", + Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, + Err(error) => return Err(error.into()), + }; + if reached { + break; + } + if let Some(status) = child.0.try_wait()? { + anyhow::bail!("child exited before {cut}: {status}"); + } + tokio::time::sleep(Duration::from_millis(5)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await; + let reached = wait.map_err(anyhow::Error::from).and_then(|r| r); + reached.with_context(|| { + format!( + "{rooted}/{cut}: {}", + std::fs::read_to_string(&output).unwrap_or_default() + ) + })?; + assert_eq!(std::fs::read(root.join("signal"))?, b"reached"); + child.0.kill()?; + let status = child.0.wait()?; + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + assert_eq!(status.signal(), Some(9)); + } + assert!(!status.success()); + Ok(()) +} + +pub(super) fn save(path: &Path, bytes: &[u8]) -> Result<()> { + let mut file = std::fs::File::create(path)?; + file.write_all(bytes)?; + file.sync_all()?; + Ok(()) +} + +pub(super) async fn stop_at(root: &Path) -> Result<()> { + save(&root.join("signal"), b"reached")?; + std::future::pending::<()>().await; + unreachable!("parent kills the held child") +} + +pub(super) fn parameters() -> Result<(std::path::PathBuf, bool, String)> { + let root = std::path::PathBuf::from(std::env::var("CELLAR_GROUP_CRASH_ROOT")?); + let rooted = match std::env::var("CELLAR_GROUP_CRASH_MODE")?.as_str() { + "materialized" => false, + "extents" => true, + _ => anyhow::bail!("invalid crash image mode"), + }; + let cut = std::env::var("CELLAR_GROUP_CRASH_CUT")?; + anyhow::ensure!( + root.is_dir() && !root.join("signal").exists(), + "fresh owned parent directory required" + ); + Ok((root, rooted, cut)) +} + +pub(super) async fn capture_catalog(store: &dyn ObjectStore) -> Result)>> { + let mut objects = Vec::new(); + let listed = store.list(None).try_collect::>().await?; + anyhow::ensure!(listed.len() <= 128, "bounded fixture catalog"); + let mut bytes = 0; + for item in listed { + let body = store.get(&item.location).await?.bytes().await?.to_vec(); + bytes += body.len(); + anyhow::ensure!(bytes < 1024 * 1024, "bounded fixture catalog bytes"); + objects.push((item.location.to_string(), body)); + } + Ok(objects) +} + +pub(super) async fn restore_catalog( + objects: Vec<(String, Vec)>, + route: &GroupRoute, + domain: &ExecutionDomain, +) -> Result<(RangeDirectory, CellGroupBirth)> { + let store = Arc::new(object_store::memory::InMemory::new()); + for (name, bytes) in objects { + store.put(&ObjectPath::from(name), bytes.into()).await?; + } + let catalog = Catalog::new(store, CatalogOptions::default())?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let birth = domain + .run(directory.materialize_selected_group(route)) + .await??; + Ok((directory, birth)) +} diff --git a/crates/cellar/tests/support/cell_group_membership_crash.rs b/crates/cellar/tests/support/cell_group_membership_crash.rs new file mode 100644 index 000000000..a070b4f35 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_membership_crash.rs @@ -0,0 +1,379 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Retained voter disks resume joint consensus through production TCP controllers. +use super::*; +use raft::{ + prelude::{ConfChangeTransition, ConfChangeType, ConfChangeV2, ConfState}, + Storage, +}; +use serde::{Deserialize, Serialize}; +use std::path::Path; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Oracle { + rooted: bool, + joint: bool, + source_directory: String, + expected: GroupManifest, + objects: Vec<(String, Vec)>, +} + +fn configuration(conf: &ConfState, joint: bool) { + assert_eq!(conf.voters, vec![1, 2]); + if joint { + assert_eq!(conf.voters_outgoing, vec![1, 2, 3]); + assert_eq!(conf.learners_next, vec![3]); + assert!(conf.learners.is_empty()); + } else { + assert!(conf.voters_outgoing.is_empty()); + assert!(conf.learners_next.is_empty()); + assert_eq!(conf.learners, vec![3]); + } + assert!(!conf.auto_leave); +} + +pub(crate) async fn membership_crash(rooted: bool) -> Result<()> { + for cut in ["joint", "left"] { + let directory = tempfile::tempdir()?; + crash_fixture::kill_at(directory.path(), "membership_process_fixture", rooted, cut).await?; + Box::pin(recover(directory.path(), rooted, cut == "joint")) + .await + .with_context(|| format!("membership restart rooted={rooted} cut={cut}"))?; + } + Ok(()) +} + +pub(crate) async fn membership_process_fixture() -> Result<()> { + let (root, rooted, cut) = crash_fixture::parameters()?; + anyhow::ensure!( + cut == "joint" || cut == "left", + "invalid membership crash cut" + ); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup_in( + rooted, + &class, + tempfile::tempdir_in(&root)?, + Arc::new(DirectFileSystem), + )) + .await?; + let _controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + membership::configure( + &fixture, + &route, + membership::change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + if cut == "left" { + membership::configure(&fixture, &route, ConfChangeV2::default(), &[]).await?; + } + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("source root")?; + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + configuration(&group.initial_state()?.conf_state, cut == "joint"); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + } + let source_directory = fixture + .cluster + ._root + .path() + .file_name() + .context("fixture directory")? + .to_str() + .context("fixture directory name")? + .to_owned(); + let objects = crash_fixture::capture_catalog(fixture.cluster.store.as_ref()).await?; + crash_fixture::save( + &root.join("oracle.json"), + &serde_json::to_vec(&Oracle { + rooted, + joint: cut == "joint", + source_directory, + expected, + objects, + })?, + )?; + crash_fixture::stop_at(&root).await +} + +struct Node { + log: ReplicaLog, + runtime: CellGroupRuntime, + transport: RaftTransport, + controller: CellGroupController, +} + +async fn applied(nodes: &[Node], history: GroupHistory, index: LogIndex) -> Result<()> { + controller::wait_for(async || { + for node in nodes { + if node.runtime.status(history).await?.applied.index < index { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn enroll( + nodes: &[Node], + leader: usize, + route: &GroupRoute, + rooted: bool, + name: &str, +) -> Result { + let cell = history(route, name)?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(91);")?; + let accepted = nodes[leader] + .runtime + .propose( + route.history, + apply::enrollment_bytes(route, &cell, &bytes, rooted)?, + ) + .await?; + applied(nodes, route.history, accepted.position.index).await?; + Ok(accepted.position.index) +} + +async fn recover(root: &Path, rooted: bool, joint: bool) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.joint, joint); + let name = Path::new(&oracle.source_directory); + anyhow::ensure!( + name.components().count() == 1 + && matches!( + name.components().next(), + Some(std::path::Component::Normal(_)) + ), + "owned source directory name" + ); + let disk = root.join(name); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let route = oracle.expected.route().clone(); + let (directory, birth) = + crash_fixture::restore_catalog(oracle.objects, &route, &domain).await?; + let mut nodes = Vec::new(); + let mut peers = Vec::new(); + for n in 1..=3u8 { + let base = disk.join(format!("node-{n}")); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([n + 90; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.expected)); + configuration(&group.initial_state()?.conf_state, joint); + assert_eq!( + group.membership()?.revision().get(), + oracle.expected.membership_revision() + ); + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let extents = if rooted { + Some(NodeExtentStore::open( + disk.join(format!("rooted-{}", n - 1)), + filesystem, + Default::default(), + )?) + } else { + None + }; + let config = super::super::checkpoint_receive::config(images, extents, &base)?; + let recovering = group.clone(); + let (applier, recovered) = domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &oracle.expected); + assert_eq!(recovered.pinned_images(), 2); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered).await?; + assert!( + !runtime.status(route.history).await?.ready, + "retained joint membership is not fresh quorum authority" + ); + let transport = tcp::start(&runtime, usize::from(n - 1)).await?; + peers.push(RaftPeer { + replica: group.identity().clone(), + label: format!("node-{n}"), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql, + vec![applier], + controller::limits(), + ) + .await?; + nodes.push(Node { + log, + runtime, + transport, + controller, + }); + } + let physical = cellar::application::consensus::verification::GroupRoute { + scope: route.history, + schema_digest: route.schema_digest, + revision: 1, + peers, + }; + for node in &nodes { + node.transport.install_physical_group(physical.clone())?; + } + nodes[0].runtime.campaign(route.history).await?; + let mut leader = None; + controller::wait_for(async || { + for (i, node) in nodes.iter().enumerate() { + if node.runtime.status(route.history).await?.ready { + leader = Some(i); + return Ok(true); + } + node.runtime.tick().await?; + } + Ok(false) + }) + .await + .context("retained voters regain current-term quorum")?; + let leader = leader.context("actual ready leader")?; + assert!( + leader < 2, + "demoted/outgoing-only member cannot lead the incoming configuration" + ); + enroll(&nodes, leader, &route, rooted, "after-joint-restart").await?; + for node in &nodes { + configuration( + &node + .log + .cell_group(&route.history)? + .initial_state()? + .conf_state, + joint, + ); + } + if joint { + let group = nodes[leader].log.cell_group(&route.history)?; + let request = CellGroupMembershipChange::new( + &route, + &group.membership()?, + ConfChangeV2::default(), + &[], + )?; + let left = nodes[leader] + .runtime + .configure(route.history, request) + .await?; + applied(&nodes, route.history, left.position.index).await?; + } + for node in &nodes { + configuration( + &node + .log + .cell_group(&route.history)? + .initial_state()? + .conf_state, + false, + ); + } + let final_index = enroll(&nodes, leader, &route, rooted, "after-joint-completion").await?; + for node in &nodes { + let manifest = node + .log + .cell_group(&route.history)? + .group_manifest()? + .context("final root")?; + assert_eq!(manifest.root().cells(), 4); + assert!(manifest.position().index >= final_index); + } + for name in [ + "registered-a", + "registered-b", + "after-joint-restart", + "after-joint-completion", + ] { + let cell = history(&route, name)?; + let deadline = domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let selected = nodes[leader] + .runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await?; + assert_eq!(selected.cell().version.history, cell); + assert_eq!(selected.cell().version.sequence.get(), 1); + } + assert!(!nodes[2].runtime.status(route.history).await?.ready); + cellar::note!("membership crash rooted={rooted} joint_at_kill={joint} killed=true recovered_replicas=3 final_cells=4 final_index={}", final_index.get()); + for node in nodes { + node.controller.close().await?; + node.log.close().await?; + } + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 4bec85ed3..fa460a2e6 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -823,3 +823,5 @@ pub(super) use sql::seed_receiver_faults; pub(super) use sql::seed_persistence; pub(super) use sql::{seed_crash, seed_process_fixture}; + +pub(super) use sql::{membership_crash, membership_process_fixture}; diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs index ccf9ed616..ff5040979 100644 --- a/crates/cellar/tests/support/cell_group_seed_crash.rs +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -2,14 +2,12 @@ //! Owned process kills around an authenticated, whole-group learner install. //! Frozen synthetic catalog bytes are fixture input, not an S3 recovery protocol. +use super::crash_fixture::{save, stop_at}; use super::*; use cellar::application::consensus::verification::JoiningMember; -use futures_util::TryStreamExt; -use object_store::{path::Path as ObjectPath, ObjectStore}; use raft::prelude::{ConfChangeTransition, ConfChangeType}; use raft::Storage; use serde::{Deserialize, Serialize}; -use std::io::Write; use std::path::Path; #[derive(Serialize, Deserialize)] @@ -22,74 +20,11 @@ struct Oracle { objects: Vec<(String, Vec)>, } -struct Child(std::process::Child); - -impl Drop for Child { - fn drop(&mut self) { - let _ = self.0.kill(); - let _ = self.0.wait(); - } -} - pub(crate) async fn seed_crash(rooted: bool) -> Result<()> { for cut in ["staged", "before-sync", "after-sync", "published"] { let directory = tempfile::tempdir()?; let root = directory.path(); - let output = root.join("child.log"); - let file = std::fs::File::create(&output)?; - let mut child = Child( - std::process::Command::new(std::env::current_exe()?) - .args([ - "--ignored", - "--exact", - "seed_process_fixture", - "--nocapture", - "--test-threads=1", - ]) - .env("CELLAR_GROUP_SEED_CRASH_ROOT", root) - .env( - "CELLAR_GROUP_SEED_CRASH_MODE", - if rooted { "extents" } else { "materialized" }, - ) - .env("CELLAR_GROUP_SEED_CRASH_CUT", cut) - .stdout(file.try_clone()?) - .stderr(file) - .spawn()?, - ); - let wait = tokio::time::timeout(Duration::from_secs(30), async { - loop { - let reached = match std::fs::read(root.join("signal")) { - Ok(bytes) => bytes == b"reached", - Err(error) if error.kind() == std::io::ErrorKind::NotFound => false, - Err(error) => return Err(error.into()), - }; - if reached { - break; - } - if let Some(status) = child.0.try_wait()? { - anyhow::bail!("child exited before {cut}: {status}"); - } - tokio::time::sleep(Duration::from_millis(5)).await; - } - Ok::<_, anyhow::Error>(()) - }) - .await; - let reached = wait.map_err(anyhow::Error::from).and_then(|r| r); - reached.with_context(|| { - format!( - "{rooted}/{cut}: {}", - std::fs::read_to_string(&output).unwrap_or_default() - ) - })?; - assert_eq!(std::fs::read(root.join("signal"))?, b"reached"); - child.0.kill()?; - let status = child.0.wait()?; - #[cfg(unix)] - { - use std::os::unix::process::ExitStatusExt; - assert_eq!(status.signal(), Some(9)); - } - assert!(!status.success()); + crash_fixture::kill_at(root, "seed_process_fixture", rooted, cut).await?; Box::pin(recover(root, rooted, cut)) .await .with_context(|| format!("cold recovery {rooted}/{cut}"))?; @@ -97,27 +32,8 @@ pub(crate) async fn seed_crash(rooted: bool) -> Result<()> { Ok(()) } -fn save(path: &Path, bytes: &[u8]) -> Result<()> { - let mut file = std::fs::File::create(path)?; - file.write_all(bytes)?; - file.sync_all()?; - Ok(()) -} - -async fn stop_at(root: &Path) -> Result<()> { - save(&root.join("signal"), b"reached")?; - std::future::pending::<()>().await; - unreachable!("parent kills the held child") -} - pub(crate) async fn seed_process_fixture() -> Result<()> { - let root = std::path::PathBuf::from(std::env::var("CELLAR_GROUP_SEED_CRASH_ROOT")?); - let rooted = match std::env::var("CELLAR_GROUP_SEED_CRASH_MODE")?.as_str() { - "materialized" => false, - "extents" => true, - _ => anyhow::bail!("invalid crash image mode"), - }; - let cut = std::env::var("CELLAR_GROUP_SEED_CRASH_CUT")?; + let (root, rooted, cut) = crash_fixture::parameters()?; anyhow::ensure!( matches!( cut.as_str(), @@ -125,10 +41,6 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { ), "invalid crash cut" ); - anyhow::ensure!( - root.is_dir() && !root.join("signal").exists(), - "fresh owned parent directory required" - ); let class = RaftClass::new("Counter", 1, [81; 32])?; let (mut fixture, route, _, _) = Box::pin(registered::setup_in( rooted, @@ -218,28 +130,7 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { // The catalog is external to the learner disk. Freeze its synthetic objects // before the fault, including the permanent destination claim. The parent // never manufactures a replacement claim from the recovered disk. - let mut objects = Vec::new(); - let listed = fixture - .cluster - .store - .list(None) - .try_collect::>() - .await?; - anyhow::ensure!(listed.len() <= 128, "bounded fixture catalog"); - let mut bytes = 0; - for item in listed { - let body = fixture - .cluster - .store - .get(&item.location) - .await? - .bytes() - .await? - .to_vec(); - bytes += body.len(); - anyhow::ensure!(bytes < 1024 * 1024, "bounded fixture catalog bytes"); - objects.push((item.location.to_string(), body)); - } + let objects = crash_fixture::capture_catalog(fixture.cluster.store.as_ref()).await?; save( &root.join("oracle.json"), &serde_json::to_vec(&Oracle { @@ -291,23 +182,8 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { let route = oracle.expected.route().clone(); let local = seed_install::target(route.history)?; let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - let store = Arc::new(object_store::memory::InMemory::new()); - for (name, bytes) in oracle.objects { - store.put(&ObjectPath::from(name), bytes.into()).await?; - } - let catalog = Catalog::new(store, CatalogOptions::default())?; - let namespace = domain - .run(catalog.open_namespace(NamespaceConfig::new( - route.history.cluster, - route.history.namespace, - cellar_logic::NamespaceMode::Raft, - route.schema_digest, - ))) - .await??; - let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; - let birth = domain - .run(directory.materialize_selected_group(&route)) - .await??; + let (directory, birth) = + crash_fixture::restore_catalog(oracle.objects, &route, &domain).await?; let base = root.join("learner"); let log = domain .run(ReplicaLog::open( diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 0002bc023..4a6a509ee 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -619,3 +619,9 @@ pub(crate) use seed_persistence::seed_persistence; #[path = "cell_group_seed_crash.rs"] mod seed_crash; pub(crate) use seed_crash::{seed_crash, seed_process_fixture}; + +#[path = "cell_group_crash_fixture.rs"] +mod crash_fixture; +#[path = "cell_group_membership_crash.rs"] +mod membership_crash; +pub(crate) use membership_crash::{membership_crash, membership_process_fixture}; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 17a5d93f2..424028a43 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1597,7 +1597,9 @@ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", "crates/cellar/tests/native_cell_groups_snapshot.rs", "crates/cellar/tests/support/cell_group_seed_persistence.rs", - "crates/cellar/tests/support/cell_group_seed_crash.rs" + "crates/cellar/tests/support/cell_group_seed_crash.rs", + "crates/cellar/tests/support/cell_group_crash_fixture.rs", + "crates/cellar/tests/support/cell_group_membership_crash.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From ef100f0794b77a0b64f95e87880dddd1e14326aa Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:42:29 -0700 Subject: [PATCH 147/568] docs: record explicit joint-consensus restart qualification --- docs/design/multi-raft/cell-groups/README.md | 3 +- docs/design/multi-raft/cell-groups/handoff.md | 52 ++++++++++++------- .../design/multi-raft/cell-groups/progress.md | 45 ++++++++++++++++ 3 files changed, 79 insertions(+), 21 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 68ae47d22..b19cdc850 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -18,7 +18,8 @@ refusal also pass. Receiver reply/chunk faults and pre-install process revocatio now pass; physical verifier failures promptly close blocked seed/checkpoint streams. Queued/entered seed revocation and sync-error recovery now pass in both image forms. Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. -Remaining CG-05 mid-stream cuts, joint restart, pressure and full acceptance gates precede +Explicit joint-consensus SIGKILL/restart and completion now pass in both image forms. +Remaining CG-05 mid-stream cuts, pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 870eb01e9..ed2e2a8ec 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,24 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest seed-process checkpoint at `b997a48b` passes 11 selected seed tests +The latest joint-restart checkpoint at `5f30cc6b` passes 16 selected +seed/membership tests and nine scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-joint-process-restart/` binds 326 source +hashes and one executable. Four owned SIGKILL scenarios cover replicated explicit +joint entry and exit in both image forms. Three retained replica disks reopen +with new process identities; complete roots, membership revisions and incoming/ +outgoing voter and learner sets match before admission. Production TLS/controllers +regain quorum, commit a cell, finish the pending transition, commit another cell +and satisfy linearizable reads for all four cells at unchanged logical sequences. + +Production runtime code is unchanged. Shared crash/catalog fixture helpers keep +the seed regressions passing. These are replicated-boundary cuts with all original +replicas in one killed child; every in-flight/automatic-transition crash boundary, +independent domains, power loss, RustFS/load and zero-quorum remain unqualified. +Next address large-group transfer/pin pressure and consolidate all five exact +CG-05 acceptance cases, retaining the remaining mid-stream and lifecycle faults. + +The earlier seed-process checkpoint at `b997a48b` passes 11 selected seed tests and eight scoped commands. The record under `.cellar/cell-groups-evidence/CG-05-seed-process-crash/` binds 324 source hashes and one executable. Two parent tests perform eight owned SIGKILL runs across both @@ -55,8 +72,8 @@ The frozen synthetic catalog preserves pre-fault external claims only for the fixture. It is not S3 durability or a recovery API. All source voters and the learner share the killed child; independent voter failure domains, power loss, mid-stream cuts, joint restart, RustFS/load and zero-quorum recovery remain open. -The next critical work is retained-disk joint-consensus restart and completion, -then the remaining bounded-pressure and exact CG-05 acceptance cases. +The newer joint-restart checkpoint above covers retained-disk explicit transition +restart/completion. Bounded pressure and the exact CG-05 gate remain open. The earlier persistence checkpoint at `3ea90fb5` passes 9 selected seed tests and eight scoped commands. Its record under @@ -775,23 +792,18 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Finish the full CG-05 acceptance gate around the implemented fresh-learner - replacement path. Extend `group_snapshot_transport/seed/` and its production - controller fixture with retained-disk joint-consensus restart/completion and - process-generation revocation during transfer. Reply/chunk faults, pre-install - and queued/entered revocation, sync-error recovery and staged/engine-boundary - process kills now pass in both image forms. Qualify - minority signed receipt refusal and joint-transition restart. Native retained - peer reconnection and minority commit/read refusal now pass in both image forms. - Preserve permanent claims, local install receipts and complete physical recovery. -2. Complete CG-05 qualification around the delivered membership features. - Preserve all five exact acceptance cases. Add joint-transition crash cuts, - mid-stream receiver process-kill cuts, continuous checkpoint saturation fairness, - exhausted apply/completion budgets, - held sync/cancelled close, quarantine capacity and retained-root pressure. - Keep unrelated-group progress and bounded physical ownership observable. - Idle/time/byte checkpoint triggers require an explicit cut-reservation contract; - keep the exact five-case task gate ahead of the archive feature dependency. +1. Finish the full CG-05 acceptance gate around the implemented checkpoint, + learner and membership paths. Add bounded large-group transfer and retained + reader/checkpoint-pin pressure, then consolidate the existing proof into all + five exact cases. Preserve permanent claims, local install receipts, complete + physical recovery and unrelated-group progress. +2. Complete remaining CG-05 fault/lifecycle requirements: mid-stream process + revocation and crash, minority signed receipt refusal, continuous capture + fairness, exhausted apply/completion budgets, held sync/cancelled close, + quarantine capacity and retained-root pressure. Explicit joint restart after + replicated entry/exit now passes; this does not qualify every in-flight or + automatic-transition crash cut. Idle/time/byte checkpoint triggers require an + explicit cut-reservation contract. Keep the full task gate ahead of CG-06. 3. Deliver CG-06 archive/reclamation with verified complete checkpoints and contiguous tails. Upload completion alone cannot authorize reclaiming local recovery dependencies or an S3 durability acknowledgement. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 5958fdbfb..5a7312ccd 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,51 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 explicit joint-consensus restart and completion + +Qualified source: `5f30cc6b28ae01c2e6411507ea11716ef94e6456`. All 16 selected seed/membership +tests pass. Two new parent cases cover four owned process kills: after all replicas +apply explicit joint entry and after all apply joint exit, in both image forms. +Two ignored child fixtures are invoked by their parents and are not standalone +passes. Nine scoped commands pass: Clippy for the execution target, seed tests, +membership tests, formatting, layout, dependency isolation, docs, source preflight +and 37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-joint-process-restart/checkpoint.json` binds +326 source hashes, one executable, frozen plan inputs and intermediate logs. +Independent hash verification passes; broader predecessor suites remain historical. + +The original group has three voters and two independent SQLite cells. Explicit +joint entry demotes member 3: incoming voters are 1/2, outgoing voters remain +1/2/3 and learner-next is 3. The child is killed with SIGKILL after the selected +replicated boundary. Every original replica disk is reopened with a new process +identity, preserving its permanent claims. Before runtime admission, recovery +checks the exact complete root, membership revision and both configuration sets. +After joint exit, the recovered configuration instead contains voters 1/2 and +learner 3, with no outgoing voters or learner-next entries. + +Production controllers and TLS transport establish a fresh leader. The recovered +group commits a new cell under its restored configuration, explicitly completes +any pending joint transition, and commits another cell after completion. All three +replicas apply the four-cell state. Fresh linearizable reads verify both original +and both new cell histories with per-cell sequence 1; group log progress does not +change those logical sequences. The demoted member remains non-serving. + +The seed and membership suites share owned-child SIGKILL/join and bounded frozen +synthetic-catalog helpers. Seed crash regression cases remain green. Production +runtime behavior was unchanged. All original replicas share the killed child and +all recovered replicas share the parent; the frozen catalog is test input, not a +production recovery backend. These cuts occur after replicated entry/exit, not +at every in-flight configuration fence or automatic-transition crash boundary. +Independent failure domains, device power loss, RustFS/load and zero-quorum +recovery remain unqualified. + +**Reflection and next step:** explicit joint membership now survives process loss +and completes with fresh quorum authority. The next critical gap is bounded +large-group transfer and retained-pin pressure, followed by consolidating the +existing evidence into all five exact CG-05 acceptance cases. Mid-stream fault, +minority receipt and controller-pressure requirements remain open. Complete CG-05 +before CG-06 verified S3 archive/reclamation; the full hard cutover remains pending. + ## CG-05 seed installation across process kills Qualified source: `b997a48ba686fceb2531cd96ae2734c9bed84fc9`. All 11 selected seed tests pass, From a2036e5074066eb7585ea82a2eb62331368ccca5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 22:52:25 -0700 Subject: [PATCH 148/568] test: bound shared-group checkpoint pressure and reader retention --- .../tests/native_cell_groups_execution.rs | 10 + .../tests/support/cell_group_checkpoint.rs | 38 ++- .../support/cell_group_checkpoint_pressure.rs | 297 ++++++++++++++++++ .../support/cell_group_checkpoint_receive.rs | 2 +- .../tests/support/cell_group_runtime.rs | 4 + docs/design/multi-raft/cell-groups/tasks.json | 3 +- 6 files changed, 346 insertions(+), 8 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_checkpoint_pressure.rs diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 675db6946..4f093527c 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,16 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_pressure_materialized_preserves_bounds_and_reader() -> Result<()> { + runtime_fixture::transfer_pressure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_pressure_extents_preserves_bounds_and_reader() -> Result<()> { + runtime_fixture::transfer_pressure(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn membership_crash_materialized_recovers_and_completes_joint_consensus() -> Result<()> { Box::pin(runtime_fixture::membership_crash(false)).await diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs index 2bac46cbf..571581b78 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -41,6 +41,16 @@ async fn populate( rooted: bool, lagging: bool, partition: bool, +) -> Result<(Cluster, BTreeMap>)> { + populate_images(rooted, lagging, partition, 0, 4_202_496).await +} + +pub(super) async fn populate_images( + rooted: bool, + lagging: bool, + partition: bool, + additional: usize, + padding: usize, ) -> Result<(Cluster, BTreeMap>)> { let mut cluster = Cluster::new().await?; cluster.elect().await?; @@ -56,14 +66,20 @@ async fn populate( } } let mut expected = BTreeMap::new(); - for name in ["alpha", "beta", "empty"] { - let cell = history(&route, name)?; + let names = ["alpha", "beta", "empty"] + .into_iter() + .map(str::to_owned) + .chain((0..additional).map(|n| format!("cell-{n:03}"))); + for (ordinal, name) in names.enumerate() { + let cell = history(&route, &name)?; let bytes = if name == "empty" { Vec::new() } else { - // More than one verified extent; the two images also exceed one - // source workspace together. Source transfer never aggregates them. - images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(1); CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob(4202496));")? + // Keep the original two-image fixture unchanged. Pressure cohorts + // use unique contents so other cells cannot accidentally retain an + // old reader's extent after that cell advances. + let value = if additional == 0 { 1 } else { ordinal + 1 }; + images::image(None, &format!("CREATE TABLE value(n); INSERT INTO value VALUES({value}); CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob({padding}));"))? }; let encoded = if bytes.is_empty() { CellGroupEntry { @@ -135,6 +151,16 @@ fn extent_commitment( } async fn advance(cluster: &mut Cluster, name: &str, before: &[u8], rooted: bool) -> Result<()> { + advance_partition(cluster, name, before, rooted, None).await +} + +pub(super) async fn advance_partition( + cluster: &mut Cluster, + name: &str, + before: &[u8], + rooted: bool, + excluded: Option, +) -> Result<()> { let route = cluster.birth.route().clone(); let cell = history(&route, name)?; let after = images::image(Some(before), "UPDATE value SET n=n+1")?; @@ -153,7 +179,7 @@ async fn advance(cluster: &mut Cluster, name: &str, before: &[u8], rooted: bool) .as_mut() .unwrap() .propose(entry.encode(&route, GroupEntryLimits::default())?)?; - cluster.drive(false).await?; + cluster.drive_partition(false, excluded).await?; Ok(()) } diff --git a/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs b/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs new file mode 100644 index 000000000..b728772c1 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_pressure.rs @@ -0,0 +1,297 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded physical transfer and read views; quorum admission is tested separately. +use super::*; +use cellar_ltx::stream_snapshot::{ScratchLimits, ScratchPool}; + +fn digest(pin: &CellImagePin) -> Result<[u8; 32]> { + let mut hash = Sha256::new(); + match pin { + CellImagePin::Empty => {} + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut offset = 0; + while offset < reader.length() { + let size = (reader.length() - offset).min(64 * 1024) as usize; + hash.update(reader.read_exact_at(offset, size)?); + offset += size as u64; + } + } + CellImagePin::Extents(pin) => { + for n in 0..pin.manifest().geometry().extent_count() { + hash.update( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + } + } + Ok(hash.finalize().into()) +} + +pub(crate) async fn transfer_pressure(rooted: bool) -> Result<()> { + // 31 populated cells plus empty enrollment: aggregate payload exceeds both + // source workspace and one receiver image spool, without collecting a stream. + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 29, 524_288).await?; + let source_group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let root = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + assert_eq!(root.pinned_images(), 32); + let marker = root.marker().clone(); + let (_, reader) = root + .pin_cell(&CellId::new("alpha")?) + .context("reader pin")?; + let old_image = match &reader { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("node-1/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).context("reader extent")?; + cluster._root.path().join("extents-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )) + } + CellImagePin::Empty => anyhow::bail!("populated reader"), + }; + let old_page = cluster._root.path().join("node-1/pages").join(format!( + "{}.page", + hex::encode(marker.root().digest().unwrap()) + )); + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = source_pool.capture(&source_group, &root)?; + let source_usage = source_pool.usage(); + let payload: usize = expected.values().map(Vec::len).sum(); + assert!(payload > source_usage.retained_bytes); + assert!(matches!( + source_pool.capture(&source_group, &root), + Err(CellGroupCheckpointError::Busy) + )); + drop(root); + + let applier = checkpoint_receive::applier(&mut cluster)?; + let destination = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let before = destination.group_manifest()?.context("receiver base")?; + let limits = CellGroupCheckpointLimits { + max_sources: 2, + ..checkpoint_fixture::limits() + }; + let pool = CellGroupCheckpointPool::new(limits)?; + let image_bytes = expected["alpha"].len() as u64; + let scratch = ScratchPool::new(ScratchLimits { + max_files: 1, + max_bytes: image_bytes, + })?; + let directory = cluster._root.path().to_owned(); + let mut receiver = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + let receive_usage = pool.usage(); + assert!(receive_usage.retained_bytes <= limits.max_retained_bytes); + // Verify byte admission independently of item admission, including rollback + // when a second full receiver cannot reserve its metadata/workspace. + let bytes_pool = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 2, + max_retained_bytes: receive_usage.retained_bytes, + ..limits + })?; + let reserved = bytes_pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + assert!(matches!( + bytes_pool.receive(applier.clone(), scratch.clone(), directory.clone()), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(bytes_pool.usage(), receive_usage); + drop(reserved); + assert_eq!(bytes_pool.usage().retained_bytes, 0); + + let (entered, entered_rx) = tokio::sync::oneshot::channel(); + let (release, release_rx) = std::sync::mpsc::channel(); + let observed_pool = pool.clone(); + let observed_scratch = scratch.clone(); + let worker = cluster.domain.blocking(move || -> Result<_> { + let mut entered = Some(entered); + let mut transferred = 0; + let mut peak_scratch = 0; + source.stream(|chunk| { + let image = matches!( + chunk.part, + CellGroupCheckpointPart::Materialized(_) | CellGroupCheckpointPart::Extent { .. } + ); + assert!(chunk.bytes.len() <= limits.max_chunk_bytes); + transferred += chunk.bytes.len(); + receiver.append(chunk)?; + let usage = observed_scratch.usage().unwrap(); + assert!(usage.reserved_files <= 1 && usage.reserved_bytes <= image_bytes); + peak_scratch = peak_scratch.max(usage.reserved_bytes); + assert_eq!(observed_pool.usage(), receive_usage); + if image { + if let Some(entered) = entered.take() { + let _ = entered.send(()); + release_rx.recv().map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("released")) + })?; + } + } + Ok(()) + })?; + Ok((receiver.finish()?, transferred, peak_scratch)) + })?; + tokio::time::timeout(Duration::from_secs(10), entered_rx).await??; + assert_eq!(pool.usage(), receive_usage); + assert_eq!(source_pool.usage(), source_usage); + let spare = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + assert!(matches!( + pool.receive(applier.clone(), scratch.clone(), directory.clone()), + Err(CellGroupCheckpointError::Busy) + )); + drop(spare); + assert_eq!(pool.usage(), receive_usage); + if !rooted { + // A different node can produce another valid stream, but this destination + // has only one shared image spool. Refusal must not refund the first one. + let peer_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut peer_source = peer_pool.capture( + &cluster.voters[1] + .log + .cell_group(&cluster.birth.route().history)?, + cluster.voters[1].owner.as_ref().unwrap().manifest(), + )?; + let mut competing = pool.receive(applier.clone(), scratch.clone(), directory.clone())?; + cluster + .domain + .blocking(move || { + assert!(matches!( + peer_source.stream(|chunk| competing.append(chunk)), + Err(CellGroupCheckpointError::Image( + cellar_ltx::generation::GenerationError::Snapshot( + cellar_ltx::stream_snapshot::Error::ScratchFull + ) + )) + )); + })? + .await?; + assert_eq!(pool.usage(), receive_usage); + assert_eq!(peer_pool.usage().sources, 0); + assert_eq!(scratch.usage()?.reserved_files, 1); + assert_eq!(scratch.usage()?.reserved_bytes, image_bytes); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + } + assert_eq!( + scratch.usage()?.reserved_bytes, + if rooted { 0 } else { image_bytes } + ); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + // Receiver is deliberately detached. The other two real voters still commit + // mutations while its physical stream is blocked midway through an image. + for name in ["alpha", "beta"] { + checkpoint_fixture::advance_partition(&mut cluster, name, &expected[name], rooted, Some(2)) + .await?; + } + for voter in &mut cluster.voters[..2] { + assert!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > marker.position().index + ); + for name in ["alpha", "beta"] { + assert_eq!( + voter + .owner + .as_ref() + .unwrap() + .manifest() + .pin_cell(&CellId::new(name)?) + .context("advanced cell")? + .0 + .version + .sequence + .get(), + 2 + ); + } + voter.pages.as_mut().unwrap().reconcile_retained(&[])?; + voter.pages.as_mut().unwrap().reclaim(1024)?; + voter.images.retain_only(&[])?; + voter.images.reclaim(1024)?; + if let Some(store) = &voter.extents { + store.retain_only(&[])?; + store.reclaim(1024)?; + } + } + assert!(old_page.exists()); + release.send(())?; + let (prepared, transferred, peak_scratch) = worker.await??; + assert!(transferred >= payload); + assert_eq!(peak_scratch, if rooted { 0 } else { image_bytes }); + assert_eq!(source_pool.usage().sources, 0); + assert_eq!(source_pool.usage().retained_bytes, 0); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + assert_eq!(scratch.usage()?.reserved_files, 0); + assert_eq!( + pool.usage(), + receive_usage, + "prepared closure still owns admission" + ); + assert_eq!(prepared.marker(), &marker); + assert_eq!(destination.group_manifest()?.as_ref(), Some(&before)); + let source_voter = &mut cluster.voters[0]; + source_voter.pages.as_mut().unwrap().reclaim(1024)?; + source_voter.pages.as_mut().unwrap().reclaim(1024)?; + assert!( + !old_page.exists(), + "released transfer no longer pins its old tree" + ); + source_voter.images.reclaim(1024)?; + if let Some(store) = &source_voter.extents { + store.reclaim(1024)?; + } + assert!( + old_image.exists(), + "reader alone retains its obsolete image" + ); + cluster + .domain + .blocking(move || -> Result<()> { + // The old physical reader survives even after the transfer and its tree + // pin are gone. Check all destination images independently, in bounded reads. + assert_eq!( + digest(&reader)?, + <[u8; 32]>::from(Sha256::digest(&expected["alpha"])) + ); + for (name, bytes) in expected { + let (leaf, pin) = prepared + .manifest() + .pin_cell(&CellId::new(name)?) + .context("received cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + assert_eq!(digest(&pin)?, <[u8; 32]>::from(Sha256::digest(bytes))); + } + Ok(()) + })? + .await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(pool.usage().retained_bytes, 0); + let source_voter = &mut cluster.voters[0]; + source_voter.images.reclaim(1024)?; + if let Some(store) = &source_voter.extents { + store.reclaim(1024)?; + } + assert!( + !old_image.exists(), + "released reader permits obsolete image reclamation" + ); + drop(applier); + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs index 59bb51fc8..fc7b65438 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint_receive.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint_receive.rs @@ -41,7 +41,7 @@ pub(super) fn config( }) } -fn applier(cluster: &mut Cluster) -> Result { +pub(super) fn applier(cluster: &mut Cluster) -> Result { let route = cluster.birth.route(); let voter = &mut cluster.voters[2]; drop(voter.owner.take()); diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index fa460a2e6..0f59b75a2 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -22,6 +22,10 @@ pub(super) use snapshot_owner::{ checkpoint_bounds, checkpoint_matching_log, checkpoint_owner, checkpoint_shutdown, }; +#[path = "cell_group_checkpoint_pressure.rs"] +mod checkpoint_pressure; +pub(super) use checkpoint_pressure::transfer_pressure; + #[path = "cell_group_checkpoint_receive.rs"] mod checkpoint_receive; pub(super) use checkpoint_receive::{install_checkpoint, refuse_checkpoint}; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 424028a43..e64ff68c8 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1599,7 +1599,8 @@ "crates/cellar/tests/support/cell_group_seed_persistence.rs", "crates/cellar/tests/support/cell_group_seed_crash.rs", "crates/cellar/tests/support/cell_group_crash_fixture.rs", - "crates/cellar/tests/support/cell_group_membership_crash.rs" + "crates/cellar/tests/support/cell_group_membership_crash.rs", + "crates/cellar/tests/support/cell_group_checkpoint_pressure.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 646bebf67ebdf8c72107253dec26934f9d1ee1df Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 23:00:13 -0700 Subject: [PATCH 149/568] docs: record bounded checkpoint pressure qualification --- docs/design/multi-raft/cell-groups/README.md | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 25 ++++++++-- .../design/multi-raft/cell-groups/progress.md | 49 +++++++++++++++++++ 3 files changed, 73 insertions(+), 5 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index b19cdc850..c1e0c0e99 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -19,7 +19,9 @@ now pass; physical verifier failures promptly close blocked seed/checkpoint stre Queued/entered seed revocation and sync-error recovery now pass in both image forms. Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. Explicit joint-consensus SIGKILL/restart and completion now pass in both image forms. -Remaining CG-05 mid-stream cuts, pressure and full acceptance gates precede +Bounded 32-cell transfer, scratch contention and old-reader image retention/release +now pass in both image forms. Remaining CG-05 mid-stream cuts, controller pressure +and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index ed2e2a8ec..8ed7789f5 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,24 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest joint-restart checkpoint at `5f30cc6b` passes 16 selected +The latest pressure checkpoint at `a2036e50` passes all 142 native +shared-execution tests and eight scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-checkpoint-pressure/` binds +327 source hashes and one executable. Two new cases transfer 32 cells with +bounded chunks and scratch in both image forms. A held receiver does not prevent +two original voters from advancing other cell versions. Separate item/byte limits +and materialized scratch contention refuse without refunding live owners. Complete +received digests remain at the captured cut and do not publish before installation. + +The old source tree becomes reclaimable after transfer release. A unique old cell +image stays readable through its independent physical pin, then is actually deleted +after that reader releases. Production runtime is unchanged. The pressure fixture +uses direct physical streaming in one process; it is not a measured RSS/TLS/load or +RustFS qualification. The complete execution run also retains the earlier seed and +explicit joint-consensus crash coverage. Next finish missing mid-stream and +controller-lifecycle faults and all five exact CG-05 acceptance cases. + +The earlier joint-restart checkpoint at `5f30cc6b` passes 16 selected seed/membership tests and nine scoped commands. Its record under `.cellar/cell-groups-evidence/CG-05-joint-process-restart/` binds 326 source hashes and one executable. Four owned SIGKILL scenarios cover replicated explicit @@ -793,9 +810,9 @@ limits, then identify the most critical remaining gap and the concrete next step Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented checkpoint, - learner and membership paths. Add bounded large-group transfer and retained - reader/checkpoint-pin pressure, then consolidate the existing proof into all - five exact cases. Preserve permanent claims, local install receipts, complete + learner and membership paths. Bounded 32-cell transfer and physical reader + retention/release now pass. Finish the remaining fault/lifecycle requirements + below, then consolidate the proof into all five exact cases. Preserve permanent claims, local install receipts, complete physical recovery and unrelated-group progress. 2. Complete remaining CG-05 fault/lifecycle requirements: mid-stream process revocation and crash, minority signed receipt refusal, continuous capture diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 5a7312ccd..bd57e4593 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,55 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 bounded checkpoint transfer and reader retention + +Qualified source: `a2036e5074066eb7585ea82a2eb62331368ccca5`. All 142 native shared-execution tests +pass, including two new pressure cases; two ignored child fixtures are invoked by +their process-crash parents and are not standalone passes. Eight scoped commands +pass: execution-target Clippy, the complete execution suite, formatting, layout, +dependency isolation, docs, source preflight and 37 plan-validator tests. The +record `.cellar/cell-groups-evidence/CG-05-checkpoint-pressure/checkpoint.json` +binds 327 source hashes, one executable, frozen plan inputs and intermediate +logs. Independent hash verification passes. Other integration suites were not +rerun at this source. + +Both image forms transfer 32 cells, including empty enrollment, through the +production source and independent receiver. The aggregate payload exceeds the +source reservation; emitted chunks stay within 4093 bytes. Materialized reception +uses one image-sized scratch file across the entire group. A second valid stream +refuses with `ScratchFull` without refunding the first receiver's reservation or +publishing partial state. Extent reception uses no materialized scratch file. +Independent receiver item and byte saturation refuse without leaking admission. + +While the transfer is held inside its first image, two original voters commit +changes to two cells and attempt physical reclamation. The captured tree remains +present until the source releases it. The receiver verifies every original cell's +digest and sequence at the captured cut; it never substitutes the newer images. +A completed but unpublished receiver still leaves the prior durable root selected +and retains its admission. Source, receiver and scratch usage return to zero only +after their physical owners finish. + +An independent old physical reader remains valid after the source and its tree +pin are gone. The pressure cohort has unique cell contents, so other cells cannot +accidentally preserve that obsolete image. Reclamation keeps it while the reader +exists, and deletes it after the reader releases. This proves actual retention +and release in both materialized and extent stores. + +Production runtime behavior is unchanged. This is a direct physical-stream test +with three real engines in one process, not a TLS pressure benchmark or measured +RSS ceiling. The independent read pin is a physical view, not new quorum authority. +It does not qualify continuous-write capture fairness, all controller saturation +paths, independent failure domains, RustFS or zero-quorum recovery. Earlier seed +and explicit joint-consensus process-crash cases also pass in the full suite. + +**Reflection and next step:** bounded 32-cell transfer and reader-image retention +now have native evidence. The critical remaining CG-05 work is the missing +mid-stream revocation/crash and controller-lifecycle pressure coverage, followed by +all five exact acceptance cases at one source. Keep minority signed-receipt refusal, +continuous capture fairness and retained-checkpoint lifecycle pressure explicit. +Complete that task gate before CG-06 verified S3 archive/reclamation; production +SDK/fleet cutover and final legacy removal remain pending. + ## CG-05 explicit joint-consensus restart and completion Qualified source: `5f30cc6b28ae01c2e6411507ea11716ef94e6456`. All 16 selected seed/membership From 6e56f21f2209ce3f02d78ffeeb3150a763a1476e Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 23:09:25 -0700 Subject: [PATCH 150/568] test: fence interrupted shared-group seed dependencies --- .../tests/native_cell_groups_execution.rs | 10 +++ .../tests/support/cell_group_runtime.rs | 2 +- .../tests/support/cell_group_seed_crash.rs | 89 ++++++++++++++++++- .../tests/support/cell_group_seed_install.rs | 9 +- .../support/cell_group_seed_persistence.rs | 69 +++++++++++--- crates/cellar/tests/support/cell_group_sql.rs | 2 +- .../tests/support/raft_storage_fault.rs | 63 +++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- 8 files changed, 228 insertions(+), 19 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 4f093527c..27fa6e865 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,16 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_transfer_materialized_revocation_retains_physical_owner() -> Result<()> { + Box::pin(runtime_fixture::seed_transfer_revocation(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_transfer_extents_revocation_retains_physical_owner() -> Result<()> { + Box::pin(runtime_fixture::seed_transfer_revocation(true)).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_pressure_materialized_preserves_bounds_and_reader() -> Result<()> { runtime_fixture::transfer_pressure(false).await diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 0f59b75a2..712c7d2c4 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -824,7 +824,7 @@ pub(super) use sql::seed_install; pub(super) use sql::seed_receiver_faults; -pub(super) use sql::seed_persistence; +pub(super) use sql::{seed_persistence, seed_transfer_revocation}; pub(super) use sql::{seed_crash, seed_process_fixture}; diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs index ff5040979..61f72950e 100644 --- a/crates/cellar/tests/support/cell_group_seed_crash.rs +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -21,7 +21,14 @@ struct Oracle { } pub(crate) async fn seed_crash(rooted: bool) -> Result<()> { - for cut in ["staged", "before-sync", "after-sync", "published"] { + for cut in [ + "staged", + "image-prefix", + "image-synced", + "before-sync", + "after-sync", + "published", + ] { let directory = tempfile::tempdir()?; let root = directory.path(); crash_fixture::kill_at(root, "seed_process_fixture", rooted, cut).await?; @@ -37,7 +44,7 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { anyhow::ensure!( matches!( cut.as_str(), - "staged" | "before-sync" | "after-sync" | "published" + "staged" | "image-prefix" | "image-synced" | "before-sync" | "after-sync" | "published" ), "invalid crash cut" ); @@ -144,7 +151,14 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { if cut == "staged" { return stop_at(&root).await; } - let (pages, images) = seed_install::stores(&base, rooted)?; + let image_gate = fault::Gate::new(None); + let (pages, images) = seed_install::stores_with_filesystem( + &base, + rooted, + Arc::new(fault::FaultFileSystem { + gate: image_gate.clone(), + }), + )?; let (applier, empty) = fixture .cluster .domain @@ -166,6 +180,51 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { "after-sync" => Some(gate.arm(fault::CRASH_AFTER_SYNC)), _ => None, }; + if matches!(cut.as_str(), "image-prefix" | "image-synced") { + let _release_image = image_gate.arm(if cut == "image-prefix" { + fault::HOLD_DEPENDENCY_PREFIX + } else { + fault::HOLD_DEPENDENCY_SYNC + }); + let _installer = fixture + .cluster + .domain + .spawn(async move { session.install(applier).await })?; + tokio::time::timeout(Duration::from_secs(5), image_gate.entered.notified()).await?; + let dependency = image_gate + .dependency_cut + .lock() + .unwrap() + .clone() + .context("physical dependency cut")?; + assert!(dependency.1 > 0); + let group = log.cell_group(&route.history)?; + assert_eq!( + group + .group_manifest()? + .context("staged at image cut")? + .root() + .cells(), + 0 + ); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(pool.usage().sources, 1); + assert!(fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + MemberId::new(4)?, + &log + )) + .await? + .is_err()); + save( + &root.join("dependency.json"), + &serde_json::to_vec(&dependency)?, + )?; + return stop_at(&root).await; + } let (_permit, installed) = session.install(applier).await?; assert_eq!( cut, "published", @@ -179,6 +238,28 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; assert_eq!(oracle.rooted, rooted); assert_eq!(oracle.cut, cut); + if matches!(cut, "image-prefix" | "image-synced") { + let (path, length): (std::path::PathBuf, u64) = + serde_json::from_slice(&std::fs::read(root.join("dependency.json"))?)?; + assert!(length > 0); + if rooted { + assert_eq!(path.parent(), Some(root.join("learner/extents").as_path())); + assert_eq!(std::fs::metadata(&path)?.len(), length); + let name = path + .file_name() + .context("extent file")? + .to_str() + .context("extent name")?; + let complete = u64::from_str_radix(&name[2..10], 16)?; + if cut == "image-prefix" { + assert!(length < complete); + } else { + assert_eq!(length, complete); + } + } else { + assert_eq!(path, std::path::PathBuf::from("snapshot.scratch")); + } + } let route = oracle.expected.route().clone(); let local = seed_install::target(route.history)?; let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); @@ -200,7 +281,7 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { let group = log.cell_group(&route.history)?; let selected = group.group_manifest()?.context("recovered selected root")?; match cut { - "staged" => assert_eq!(selected, oracle.staged), + "staged" | "image-prefix" | "image-synced" => assert_eq!(selected, oracle.staged), "before-sync" => assert!(selected == oracle.staged || selected == oracle.expected), "after-sync" | "published" => assert_eq!(selected, oracle.expected), _ => unreachable!(), diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs index 381b648c9..412142838 100644 --- a/crates/cellar/tests/support/cell_group_seed_install.rs +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -25,7 +25,14 @@ pub(super) fn stores( base: &Path, rooted: bool, ) -> Result<(ManifestPageStore, CellGroupApplyImages)> { - let filesystem = Arc::new(DirectFileSystem); + stores_with_filesystem(base, rooted, Arc::new(DirectFileSystem)) +} + +pub(super) fn stores_with_filesystem( + base: &Path, + rooted: bool, + filesystem: Arc, +) -> Result<(ManifestPageStore, CellGroupApplyImages)> { let pages = ManifestPageStore::open( &base.join("pages"), filesystem.clone(), diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs index 2923136e6..26f2acc91 100644 --- a/crates/cellar/tests/support/cell_group_seed_persistence.rs +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -10,12 +10,17 @@ use tokio::net::TcpListener; #[derive(Clone, Copy, Debug, PartialEq, Eq)] enum Cut { + StreamingRevocation, QueuedRevocation, EnteredRevocation, SyncError, AfterSyncError, } +pub(crate) async fn seed_transfer_revocation(rooted: bool) -> Result<()> { + Box::pin(run(rooted, Cut::StreamingRevocation)).await +} + pub(crate) async fn seed_persistence(rooted: bool) -> Result<()> { for cut in [ Cut::QueuedRevocation, @@ -113,7 +118,12 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { .await?; let group = session.group().clone(); let staged = group.group_manifest()?.context("staged root")?; - let (pages, images) = seed_install::stores(&base, rooted).context("open destination stores")?; + let (pages, images) = seed_install::stores_with_filesystem( + &base, + rooted, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .context("open destination stores")?; let recovering = group.clone(); let (applier, root) = fixture .cluster @@ -133,6 +143,7 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { drop(root); let release = gate.arm(match cut { + Cut::StreamingRevocation => fault::HOLD_DEPENDENCY_PREFIX, Cut::QueuedRevocation | Cut::EnteredRevocation => fault::HOLD_SYNC, Cut::SyncError => fault::FAIL_SYNC, Cut::AfterSyncError => fault::FAIL_AFTER_SYNC, @@ -157,7 +168,11 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { .domain .spawn(async move { session.install(installer).await })?; let mut restarted_source = None; - if matches!(cut, Cut::QueuedRevocation | Cut::EnteredRevocation) { + let mut early_result = None; + if matches!( + cut, + Cut::StreamingRevocation | Cut::QueuedRevocation | Cut::EnteredRevocation + ) { if cut == Cut::QueuedRevocation { controller::wait_for(async || Ok(queued(&log)? == 1)).await?; } else { @@ -183,23 +198,54 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { seed_receiver::revoke_source(&fixture, &mut controllers, &transport, &peer, &route) .await?, ); - assert!( - tokio::time::timeout(Duration::from_millis(30), &mut pending) - .await - .is_err(), - "revocation cannot abandon entered/queued physical work" - ); + if cut == Cut::StreamingRevocation { + assert!( + gate.dependency_cut + .lock() + .unwrap() + .as_ref() + .context("held dependency")? + .1 + > 0 + ); + early_result = Some(tokio::time::timeout(Duration::from_secs(5), &mut pending).await??); + assert!( + transport.usage().data_items > 0, + "failed network waiter cannot refund the physical receive owner" + ); + assert!(pool.usage().retained_bytes > 0); + } else { + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut pending) + .await + .is_err(), + "revocation cannot abandon entered/queued physical work" + ); + } assert_eq!(pool.usage().sources, 1); drop(release); } else { // The one-shot sync error consumes the armed fault before completion. // Retain its release guard until the result has been observed. } - let result = pending.await?; + let result = match early_result { + Some(result) => result, + None => pending.await?, + }; if let Some(ahead) = ahead { ahead.await??; } match cut { + Cut::StreamingRevocation => assert!( + matches!( + result, + Err(CellGroupSeedError::Transport( + TransportError::StalePeer | TransportError::Closed + )) + ), + "streaming revoked grant: {:?}", + result.err() + ), Cut::QueuedRevocation => assert!( matches!( result, @@ -225,7 +271,8 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { }) .await?; assert_eq!(transport.usage().data_items, 0); - if cut == Cut::QueuedRevocation { + assert_eq!(pool.usage().retained_bytes, 0); + if matches!(cut, Cut::StreamingRevocation | Cut::QueuedRevocation) { assert_eq!( group.group_manifest()?.as_ref(), Some(&staged), @@ -300,7 +347,7 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { let group = log.cell_group(&route.history)?; let selected = group.group_manifest()?.context("reopened root")?; match cut { - Cut::QueuedRevocation => assert_eq!(selected, staged), + Cut::StreamingRevocation | Cut::QueuedRevocation => assert_eq!(selected, staged), Cut::EnteredRevocation | Cut::AfterSyncError => assert_eq!(selected, expected), Cut::SyncError => assert!(selected == staged || selected == expected), } diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 4a6a509ee..0ea4b194a 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -614,7 +614,7 @@ pub(crate) use seed_receiver::seed_receiver_faults; #[path = "cell_group_seed_persistence.rs"] mod seed_persistence; -pub(crate) use seed_persistence::seed_persistence; +pub(crate) use seed_persistence::{seed_persistence, seed_transfer_revocation}; #[path = "cell_group_seed_crash.rs"] mod seed_crash; diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index 33059eefd..893c70c09 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -28,6 +28,8 @@ pub const HOLD_IMAGE_READ: u8 = 11; pub const HOLD_SNAPSHOT_TEMP: u8 = 12; pub const HOLD_EXTENT_READ: u8 = 13; pub const FAIL_AFTER_SYNC: u8 = 14; +pub const HOLD_DEPENDENCY_PREFIX: u8 = 15; +pub const HOLD_DEPENDENCY_SYNC: u8 = 16; pub struct Gate { pub mode: AtomicU8, @@ -37,6 +39,7 @@ pub struct Gate { signal: Option, pub sync_path: Mutex>, pub image_read_paths: Mutex>, + pub dependency_cut: Mutex>, } impl Gate { @@ -49,6 +52,7 @@ impl Gate { signal, sync_path: Mutex::new(None), image_read_paths: Mutex::new(vec![]), + dependency_cut: Mutex::new(None), }) } @@ -114,6 +118,27 @@ struct FaultFile { image: bool, } +impl FaultFile { + fn dependency(&self) -> bool { + self.path == Path::new("snapshot.scratch") + || self + .path + .parent() + .and_then(Path::file_name) + .is_some_and(|name| name == "extents") + && self + .path + .file_name() + .is_some_and(|name| name.to_string_lossy().starts_with("e-")) + } + + fn pause_dependency(&mut self) -> io::Result<()> { + *self.gate.dependency_cut.lock().unwrap() = + Some((self.path.clone(), self.inner.file_len()?)); + self.gate.pause() + } +} + struct TrackedLock { _lock: HostFileLock, gate: Arc, @@ -138,6 +163,28 @@ impl HostFileIo for FaultFile { } fn write_all_at(&mut self, offset: u64, bytes: &[u8]) -> io::Result<()> { + if self.dependency() + && offset == 0 + && bytes.len() > 1 + && self + .gate + .mode + .compare_exchange( + HOLD_DEPENDENCY_PREFIX, + OFF, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + let prefix = bytes.len() / 2; + self.inner.write_all_at(offset, &bytes[..prefix])?; + self.inner.sync_all()?; + self.pause_dependency()?; + return self + .inner + .write_all_at(offset + prefix as u64, &bytes[prefix..]); + } if self.log && self .gate @@ -180,6 +227,22 @@ impl HostFileIo for FaultFile { } fn sync_all(&mut self) -> io::Result<()> { + if self.dependency() + && self + .gate + .mode + .compare_exchange( + HOLD_DEPENDENCY_SYNC, + OFF, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_ok() + { + self.inner.sync_all()?; + self.pause_dependency()?; + return Ok(()); + } if self.log { let mode = self.gate.mode.load(Ordering::Acquire); let mode = if matches!( diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index e64ff68c8..54f779e14 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1591,7 +1591,8 @@ "crates/cellar/tests/support/cell_group_seed.rs", "crates/cellar/tests/support/cell_group_seed_install.rs", "crates/cellar/tests/support/cell_group_seed_probes.rs", - "crates/cellar/tests/support/cell_group_seed_receiver.rs" + "crates/cellar/tests/support/cell_group_seed_receiver.rs", + "crates/cellar/tests/support/raft_storage_fault.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs", From 04c8f9c810565179509ceb62f8352eba0ff5aaa8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 23:17:23 -0700 Subject: [PATCH 151/568] docs: record interrupted seed transfer qualification --- docs/design/multi-raft/cell-groups/README.md | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 56 ++++++++++++++++--- .../design/multi-raft/cell-groups/progress.md | 49 ++++++++++++++++ 3 files changed, 100 insertions(+), 9 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index c1e0c0e99..206691b4a 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -20,8 +20,8 @@ Queued/entered seed revocation and sync-error recovery now pass in both image fo Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. Explicit joint-consensus SIGKILL/restart and completion now pass in both image forms. Bounded 32-cell transfer, scratch contention and old-reader image retention/release -now pass in both image forms. Remaining CG-05 mid-stream cuts, controller pressure -and full acceptance gates precede +now pass in both image forms. Partial/synced seed-dependency SIGKILL cuts and transfer-time source-process +revocation now pass. Remaining CG-05 controller pressure and full acceptance gates precede verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 8ed7789f5..02a192493 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -42,7 +42,46 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest pressure checkpoint at `a2036e50` passes all 142 native +The latest interrupted-transfer checkpoint at `6e56f21f` passes all 144 +shared-execution tests and nineteen native Raft-storage tests, with eight scoped commands. +Its record under `.cellar/cell-groups-evidence/CG-05-seed-transfer-cuts/` binds +330 source hashes and two executables. The seed crash matrix now includes +12 owned SIGKILL scenarios: four new cases hold partial or synced dependency bytes +in both image modes. Cold recovery preserves the empty stage and refuses startup +without a local install receipt. The production TLS source and receiver perform +these interrupted transfers; the parent checks reached cuts and retained extent +lengths before cold reopen. + +Two new revocation tests replace the source process while a physical image write +is held. A refused network waiter does not refund checkpoint/transport admission; +physical release drains it without publishing any seed. Cold recovery again +refuses startup. These revocation cases use an independently authenticated captured +peer. Production runtime code is unchanged. No automatic orphan cleanup/retry, +all-wire-boundary, power-loss, independent-domain or RustFS/load claim is made. +Next audit the five required CG-05 oracles, deliver their exact test target and +close the remaining controller/lifecycle saturation requirements. + +The starting fixture mapping for the exact target is below. This is an audit map, +not evidence that the named target exists or its five gates have passed. Reuse the +real fixture owners and direct assertions, without launching another test binary +as an oracle or replacing any required case. + +| Required case | Existing behavior to audit and reuse | +| --- | --- | +| `snapshot_contains_every_cell_at_cut` | `checkpoint_fixture::coherent` captures both forms while quorum writes advance. | +| `partial_snapshot_never_serves` | `refuse_checkpoint`, seed receiver faults and seed crash cuts reject partial/corrupt/foreign state. | +| `whole_group_learner_and_joint_change` | `seed_install`, `membership_lifecycle` and `membership_crash` cover learner promotion, replacement, quorum loss and explicit joint restart. | +| `snapshot_empty_cell_and_extent_restart` | `tcp_publication` transfers the complete populated/empty closure and reopens retained checkpoint plus current suffix in both forms. | +| `snapshot_memory_and_pin_bounds` | `transfer_pressure` and `local_shutdown` retain bounded physical work and old reader images through concurrent apply/shutdown. | + +The native execution test currently keeps reusable imports, `Voter`, `Cluster` +and image/apply helpers between the `images` module declaration and +`three_cells_one_rawnode_per_voter`. Extract those helpers without changing their +semantics so the new snapshot target shares the same real engine fixture. Preserve +the root module paths used by the support files and the exact ignored child names +used by owned-process tests. + +The earlier pressure checkpoint at `a2036e50` passes all 142 native shared-execution tests and eight scoped commands. Its record under `.cellar/cell-groups-evidence/CG-05-checkpoint-pressure/` binds 327 source hashes and one executable. Two new cases transfer 32 cells with @@ -811,13 +850,16 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented checkpoint, learner and membership paths. Bounded 32-cell transfer and physical reader - retention/release now pass. Finish the remaining fault/lifecycle requirements - below, then consolidate the proof into all five exact cases. Preserve permanent claims, local install receipts, complete + retention/release now pass, as do partial/synced dependency crash cuts and + transfer-time source-process revocation. Audit all five exact oracles and + deliver the declared native_cell_groups_snapshot target, closing remaining + fault/lifecycle requirements below. Preserve permanent claims, local install receipts, complete physical recovery and unrelated-group progress. -2. Complete remaining CG-05 fault/lifecycle requirements: mid-stream process - revocation and crash, minority signed receipt refusal, continuous capture - fairness, exhausted apply/completion budgets, held sync/cancelled close, - quarantine capacity and retained-root pressure. Explicit joint restart after +2. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt + refusal, continuous capture fairness, exhausted apply/completion budgets, held sync/cancelled close, + quarantine capacity and retained-root pressure. Track automatic orphan cleanup + and fresh-seed retry after interrupted dependency writes; safe refusal alone + does not prove retry availability. Explicit joint restart after replicated entry/exit now passes; this does not qualify every in-flight or automatic-transition crash cut. Idle/time/byte checkpoint triggers require an explicit cut-reservation contract. Keep the full task gate ahead of CG-06. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index bd57e4593..4e2f28de1 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,55 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 interrupted seed dependency writes and process revocation + +Qualified source: `6e56f21f2209ce3f02d78ffeeb3150a763a1476e`. All 144 native shared-execution tests +and nineteen native Raft-storage tests pass. Three ignored child fixtures are invoked +by their process-crash parents and are not standalone passes. Eight scoped commands +pass: Clippy for both targets, both complete suites, formatting, layout, dependency +isolation, docs, source preflight and 37 plan-validator tests. The record +`.cellar/cell-groups-evidence/CG-05-seed-transfer-cuts/checkpoint.json` binds +330 source hashes, two executables, frozen plan inputs and intermediate logs. +Independent hash verification passes. Other suites were not rerun at this source. + +The authenticated seed crash matrix now runs six cuts in each image form: staged, +partial dependency write, synced dependency, before engine sync, after engine sync +and publication. Four new SIGKILL scenarios use the production TLS source/receiver +and hold either an anonymous materialized-image spool or an extent payload. The +partial cut writes and syncs a prefix; the second cut finishes the file sync but +stops before complete group installation. The child observes nonzero physical +bytes, the unchanged empty stage, no checkpoint and refused bootstrap before +signaling the parent to kill it. The parent joins the killed child, checks the +recorded extent file length against its encoded full length, then cold-opens the +original learner disk with a new process identity. Both new cuts select the empty +stage with no authenticated local receipt and refuse startup. + +Two new transfer-revocation cases hold a physical dependency write and replace +the source process. The network waiter returns stale/closed authority while the +physical receiver remains admitted: both checkpoint metadata and transport data +reservations remain charged. Releasing the write lets the receiver observe the +revocation and finish cleanup; admission drains without publishing a root or +checkpoint. Cold recovery independently confirms the empty stage and refused +bootstrap. These cases use an independently authenticated captured peer; the +crash cases use the production source server. + +The shared filesystem fixture adds two opt-in dependency faults. Default behavior +and existing fault modes are unchanged. Production runtime code is unchanged. +All original voters and the learner share the killed child; this is not device +power-loss or independent-domain qualification. These cuts do not cover every +wire/manifest boundary, automatic cleanup and fresh-seed retry after a partial +orphan, measured load, RustFS or zero-quorum recovery. Existing pressure, joint +restart and seed installation cases remain green in the complete execution run. + +**Reflection and next step:** interrupted dependency writes now have direct native +failure and restart evidence. The next critical step is to audit the five exact +CG-05 acceptance oracles against these fixtures, implement their required test +target and close remaining controller/lifecycle saturation gaps. Keep unverified +orphan cleanup/retry, continuous capture fairness, minority signed receipts and +retained-checkpoint pressure explicit. Broader fault/load requirements remain in +the release matrix. Complete CG-05 before CG-06 verified S3 archive/reclamation; +production SDK/fleet cutover and final legacy removal remain pending. + ## CG-05 bounded checkpoint transfer and reader retention Qualified source: `a2036e5074066eb7585ea82a2eb62331368ccca5`. All 142 native shared-execution tests From f076731a2d384bff65914828409c01506438c4a2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 23:34:20 -0700 Subject: [PATCH 152/568] test: add exact cell-group snapshot acceptance cases Share the real engine fixture and strengthen held-install writes and empty-cell recovery checks. Construct large fixture futures outside the composite poll frame to avoid debug stack overflow without changing stack limits or assertions. --- .../tests/native_cell_groups_execution.rs | 582 +---------------- .../tests/native_cell_groups_snapshot.rs | 105 ++++ .../support/cell_group_checkpoint_local.rs | 15 + .../support/cell_group_engine_fixture.rs | 584 ++++++++++++++++++ .../support/cell_group_membership_crash.rs | 2 +- .../tests/support/cell_group_seed_crash.rs | 2 +- .../tests/support/cell_group_seed_install.rs | 54 +- .../support/cell_group_seed_persistence.rs | 2 +- docs/design/multi-raft/cell-groups/tasks.json | 3 +- 9 files changed, 740 insertions(+), 609 deletions(-) create mode 100644 crates/cellar/tests/native_cell_groups_snapshot.rs create mode 100644 crates/cellar/tests/support/cell_group_engine_fixture.rs diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 27fa6e865..bbe447efb 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -428,288 +428,11 @@ async fn rooted_enrollment_recovery_after_leadership_change() -> Result<()> { #[allow(dead_code)] mod images; -use anyhow::{Context, Result}; -use cellar::application::consensus::verification::cell_groups::GroupRoute; -use cellar::application::consensus::{ - verification::{cell_groups::*, *}, - *, -}; -use cellar::sdk::ExecutionDomain; -use cellar_ltx::{ - apply::{ImageState, ImageUpdate, LtxSegment}, - generation::{GenerationPin, GenerationStore, ImageId}, - DirectFileSystem, TXID, -}; -use raft::prelude::{Entry, Message, MessageType}; -use sha2::{Digest, Sha256}; -use std::{collections::VecDeque, sync::Arc, time::Duration}; - -struct Voter { - log: ReplicaLog, - owner: Option, - pages: Option, - images: GenerationStore, - extents: Option, - tree: ManifestTree, - gate: Arc, - paused: bool, - held: Option, -} - -struct Cluster { - _root: tempfile::TempDir, - domain: ExecutionDomain, - directory: RangeDirectory, - store: Arc, - birth: CellGroupBirth, - voters: Vec, - network: VecDeque, - reads: Vec<(usize, Term, LogIndex, Vec)>, - options: RaftRuntimeOptions, -} - -impl Cluster { - async fn new() -> Result { - Self::with_options(RaftRuntimeOptions::default()).await - } - - async fn with_options(options: RaftRuntimeOptions) -> Result { - Box::pin(Self::with_storage( - options, - tempfile::tempdir()?, - Arc::new(DirectFileSystem), - )) - .await - } - - async fn with_storage( - options: RaftRuntimeOptions, - root: tempfile::TempDir, - filesystem: Arc, - ) -> Result { - Box::pin(Self::with_storage_schema( - options, root, filesystem, [6; 32], - )) - .await - } - - async fn with_storage_schema( - options: RaftRuntimeOptions, - root: tempfile::TempDir, - filesystem: Arc, - schema: [u8; 32], - ) -> Result { - let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - let history = GroupHistory { - cluster: ClusterId::new([1; 16])?, - namespace: NamespaceId::new([2; 16])?, - group: GroupId::new([3; 16])?, - incarnation: IncarnationId::new([4; 16])?, - }; - let store = Arc::new(object_store::memory::InMemory::new()); - let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; - let namespace = domain - .run(catalog.open_namespace(NamespaceConfig::new( - history.cluster, - history.namespace, - cellar_logic::NamespaceMode::Raft, - schema, - ))) - .await??; - let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; - let request = directory.prepare_group_birth(CellGroupBirthSpec { - history, - range: CellRange::all(), - epoch: RouteEpoch::new(1)?, - creation: ProposalId::new([5; 16])?, - voters: (1..=3) - .map(|n| InitialMember { - member: MemberId::new(n).unwrap(), - node: NodeId::new([n as u8; 16]).unwrap(), - }) - .collect(), - learners: vec![], - })?; - domain - .run(directory.reserve_group_birth(&request)) - .await??; - let init = directory - .prepare_initialization(ProposalId::new([7; 16])?, &[request.route().clone()])?; - domain.run(directory.publish(&init)).await??; - let birth = domain - .run(directory.materialize_group_birth(&request)) - .await??; - let mut voters = Vec::new(); - for n in 1..=3 { - let base = root.path().join(format!("node-{n}")); - let gate = fault::Gate::new(None); - let log = domain - .run(ReplicaLog::open( - ReplicaLogOptions::new( - base.join("log"), - NodeId::new([n; 16])?, - DiskLineage::new([n + 10; 16])?, - ProcessGeneration::new([20; 16])?, - LogOpenMode::Create, - ), - Arc::new(fault::FaultFileSystem { gate: gate.clone() }), - )) - .await??; - let permit = domain - .run(directory.bootstrap_group_member(&birth, &log)) - .await??; - let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; - let mut pages = ManifestPageStore::open( - &base.join("pages"), - filesystem.clone(), - 1024, - 16 * 1024 * 1024, - 16, - )?; - let marker = permit.group().group_manifest()?.context("genesis root")?; - let prepared = tree.prepare_complete( - permit.group(), - &mut pages, - marker.root(), - marker.position(), - marker.membership_revision(), - |_| panic!("empty genesis"), - )?; - let owner = domain - .run(CellGroupOwner::start( - permit, - prepared, - options.clone(), - [n; 32], - )) - .await??; - let images = - GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; - voters.push(Voter { - log, - owner: Some(owner), - pages: Some(pages), - images, - extents: None, - tree, - gate, - paused: false, - held: None, - }); - } - Ok(Self { - _root: root, - domain, - directory, - store, - birth, - voters, - network: VecDeque::new(), - reads: vec![], - options, - }) - } +include!("support/cell_group_engine_fixture.rs"); - async fn permit(&self, voter: usize) -> Result { - Ok(self - .domain - .run( - self.directory - .bootstrap_group_member(&self.birth, &self.voters[voter].log), - ) - .await??) - } - - async fn drive(&mut self, isolated: bool) -> Result<()> { - self.drive_partition(isolated, None).await - } - - async fn drive_partition(&mut self, isolated: bool, excluded: Option) -> Result<()> { - for _ in 0..2000 { - let mut progress = false; - for (n, voter) in self.voters.iter_mut().enumerate() { - if excluded == Some(n) { - continue; - } - let event = voter.owner.as_mut().unwrap().poll()?; - let Some(event) = event else { continue }; - progress = true; - match event { - CellGroupEffect::Persist(work) => { - let done = work.run().await?; - voter.owner.as_mut().unwrap().complete_persist(done)?; - } - CellGroupEffect::Message(message) => { - if !isolated { - self.network.push_back(message); - } - } - CellGroupEffect::Apply(apply) => { - if voter.paused { - assert!(voter.held.replace(*apply).is_none()); - } else { - let prepared = prepare(voter, &apply)?; - voter - .owner - .as_mut() - .unwrap() - .complete_apply(apply.ticket, prepared)?; - } - } - CellGroupEffect::Applied { ticket, manifest } => { - assert_eq!(ticket.position(), manifest.position()); - assert_eq!( - voter.log.cell_groups()?[0].group_manifest()?.as_ref(), - Some(manifest.as_ref()) - ); - } - CellGroupEffect::CheckpointInstalled { manifest } - | CellGroupEffect::CheckpointPublished { manifest } => { - assert_eq!( - voter.log.cell_groups()?[0].group_manifest()?.as_ref(), - Some(manifest.as_ref()) - ); - } - CellGroupEffect::ReadIndex { - term, - index, - context, - } => self.reads.push((n, term, index, context)), - } - } - for _ in 0..self.network.len() { - let message = self.network.pop_front().unwrap(); - let target = message.to as usize - 1; - if excluded == Some(target) || excluded == Some(message.from as usize - 1) { - continue; - } - match self.voters[target] - .owner - .as_mut() - .unwrap() - .step(message.clone()) - { - Ok(()) => progress = true, - Err(RuntimeError::NotReady) => self.network.push_back(message), - Err(error) => return Err(error.into()), - } - } - // A bounded receiver may refuse messages until a held apply settles. - // Keep those messages for the next pump instead of busy-spinning. - if !progress { - return Ok(()); - } - } - anyhow::bail!("group did not quiesce") - } - - async fn elect(&mut self) -> Result<()> { - self.voters[0].owner.as_mut().unwrap().campaign()?; - self.drive(false).await?; - assert!(self.voters[0].owner.as_ref().unwrap().status()?.ready); - Ok(()) - } +use raft::prelude::Entry; +impl Cluster { async fn restart(&mut self, n: usize) -> Result<()> { let mut old = self.voters.remove(n); drop(old.held.take()); @@ -802,305 +525,6 @@ impl Cluster { ); Ok(()) } - - async fn close(self) -> Result<()> { - for mut voter in self.voters { - drop(voter.owner.take()); - voter.log.close().await?; - } - self.domain.shutdown().await; - Ok(()) - } -} - -fn image_id(history: &CellHistory, sequence: u64) -> Result { - let mut hash = Sha256::new(); - hash.update(history.cell.as_str()); - hash.update(sequence.to_be_bytes()); - Ok(ImageId::new(hash.finalize()[..16].try_into()?)?) -} - -fn apply_image( - store: &GenerationStore, - history: &CellHistory, - sequence: u64, - base: Option<&GenerationPin>, - image: LtxImage, - files: &[GroupLtx<'_>], -) -> Result { - let header = cellar_ltx::ltx::Header::parse(files.last().context("LTX")?.bytes)?; - let ImageCommitment::Sha256(digest) = image.commitment else { - anyhow::bail!("materialized commitment required") - }; - let result = ImageState { - epoch: image.epoch, - txid: TXID(image.txid), - checksum: image.checksum, - digest, - page_size: header.page_size, - length: u64::from(header.page_size) * u64::from(header.commit), - }; - let update = ImageUpdate { - base: match base { - Some(pin) => pin.state(), - None => ImageState::empty(image.epoch)?, - }, - result, - segments: files - .iter() - .map(|f| LtxSegment { - bytes: f.bytes, - min_txid: TXID(f.min_txid), - max_txid: TXID(f.max_txid), - page_count: f.page_count as usize, - decoded_bytes: f.decoded_bytes, - }) - .collect(), - }; - Ok(store.prepare(image_id(history, sequence)?, base, &update)?) -} - -fn prepare_cell_image( - images: &GenerationStore, - extents: Option<&cellar_ltx::internal::NodeExtentStore>, - cell: &CellHistory, - sequence: u64, - base: Option<&CellImagePin>, - image: LtxImage, - files: &[GroupLtx<'_>], -) -> Result { - use cellar_ltx::internal::{ - ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, - }; - if image.txid == 0 { - anyhow::ensure!(base.is_none() && files.is_empty()); - return Ok(EnrollmentImage { - pin: CellImagePin::Empty, - geometry: CellImageGeometry { - length: 0, - page_size: 512, - }, - effects: EffectSummary::default(), - }); - } - let (pin, geometry) = match image.commitment { - ImageCommitment::Sha256(_) => { - let base = match base { - Some(CellImagePin::Materialized(pin)) => Some(pin), - None => None, - _ => anyhow::bail!("materialized base"), - }; - let pin = apply_image(images, cell, sequence, base, image, files)?; - let geometry = CellImageGeometry { - length: pin.state().length, - page_size: pin.state().page_size, - }; - (CellImagePin::Materialized(pin), geometry) - } - ImageCommitment::Extents(result) => { - let store = extents.context("extent store")?; - let base = match base { - Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), - None => None, - _ => anyhow::bail!("extent base"), - }; - let binding = ImageBinding { - history: cell_image_history_digest(cell), - schema: [6; 32], - epoch: image.epoch, - txid: image.txid, - checksum: image.checksum, - }; - let empty = ExtentManifest::new( - ImageBinding { - txid: 0, - checksum: 0, - ..binding - }, - Geometry::new(0, 0)?, - &[], - )?; - let before = base - .map(|pin| pin.reference()) - .unwrap_or(ExtentRootReference::new( - empty.root(), - empty.encode()?.len().try_into()?, - )?); - let header = cellar_ltx::ltx::Header::parse(files.last().context("extent LTX")?.bytes)?; - let geometry = Geometry::new( - header.page_size, - u64::from(header.page_size) * u64::from(header.commit), - )?; - let update = ImageRootUpdate { - base: before, - result, - binding, - geometry, - segments: files - .iter() - .map(|file| LtxSegment { - bytes: file.bytes, - min_txid: TXID(file.min_txid), - max_txid: TXID(file.max_txid), - page_count: file.page_count as usize, - decoded_bytes: file.decoded_bytes, - }) - .collect(), - }; - let prepared = store.prepare_image_root( - base, - &update, - &images.apply_limits(), - &mut vec![0; EXTENT_BYTES as usize], - )?; - ( - CellImagePin::Extents(Arc::new(prepared.image)), - CellImageGeometry { - length: geometry.length(), - page_size: geometry.page_size(), - }, - ) - } - }; - Ok(EnrollmentImage { - pin, - geometry, - effects: EffectSummary::default(), - }) -} - -fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { - if apply.entry.data.is_empty() { - return Ok(voter.tree.prepare_unchanged( - voter.pages.as_ref().context("fixture page owner")?, - &apply.base, - apply.ticket.position(), - )?); - } - let decoded = CellGroupEntry::decode( - &apply.entry.data, - apply.base.marker().route(), - GroupEntryLimits::default(), - )?; - match decoded.command { - GroupCommand::Enroll { cell, image, files } => { - let (next, _) = voter.tree.prepare_enrollment( - voter.pages.as_mut().context("fixture page owner")?, - &apply.base, - EnrollmentApply { - position: apply.ticket.position(), - phase: EnrollmentPhase::Active, - entry: &apply.entry.data, - limits: GroupEntryLimits::default(), - }, - |_| { - prepare_cell_image( - &voter.images, - voter.extents.as_ref(), - &cell, - 1, - None, - image, - &files, - ) - .map_err(|_| ManifestError::Invalid("fixture initial LTX")) - }, - )?; - Ok(next) - } - GroupCommand::Mutation(mutation) => { - let (old, base) = apply.base.pin_cell(&mutation.cell.cell).context("cell")?; - let prepared = prepare_cell_image( - &voter.images, - voter.extents.as_ref(), - &mutation.cell, - mutation.sequence.get(), - Some(&base), - mutation.result, - &mutation.files, - )?; - let leaf = CellManifestLeaf { - enrollment: old.enrollment.clone(), - version: CellVersion { - history: mutation.cell, - sequence: mutation.sequence, - image: mutation.result.into(), - }, - geometry: prepared.geometry, - last_mutation: apply.ticket.position(), - witness: CommitWitness { - group: decoded.binding.group, - position: apply.ticket.position(), - proposal: decoded.binding.proposal, - }, - effects: old.effects, - }; - Ok(voter.tree.prepare_update( - voter.pages.as_mut().context("fixture page owner")?, - &apply.base, - leaf, - Some(&old.version), - prepared.pin, - )?) - } - _ => anyhow::bail!("unexpected administration"), - } -} - -fn history(route: &GroupRoute, name: &str) -> Result { - Ok(CellHistory { - cluster: route.history.cluster, - namespace: route.history.namespace, - cell: CellId::new(name)?, - incarnation: IncarnationId::new([8; 16])?, - }) -} - -fn request( - route: &GroupRoute, - cell: CellHistory, - sequence: u64, - cut: &images::Cut, -) -> Result> { - let image = |state: ImageState| LtxImage { - epoch: state.epoch, - txid: state.txid.0, - checksum: state.checksum, - commitment: state.digest.into(), - }; - let files = vec![GroupLtx { - min_txid: cut.base.txid.0 + 1, - max_txid: cut.result.txid.0, - page_count: cut.pages.len() as u64, - decoded_bytes: cut.pages.len() as u64 * 512, - bytes: &cut.bytes, - }]; - let command = if sequence == 1 { - GroupCommand::Enroll { - cell, - image: image(cut.result), - files, - } - } else { - GroupCommand::Mutation(GroupMutation { - cell, - sequence: MutationSequence::new(sequence)?, - base: image(cut.base), - result: image(cut.result), - files, - }) - }; - Ok(CellGroupEntry { - binding: GroupEntryBinding { - group: route.history, - route_epoch: route.epoch, - birth_digest: route.birth_digest, - schema_digest: route.schema_digest, - proposal: ProposalId::new([sequence as u8; 16])?, - }, - command, - } - .encode(route, GroupEntryLimits::default())?) } #[tokio::test(flavor = "multi_thread", worker_threads = 3)] diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs new file mode 100644 index 000000000..7b8d5af8b --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -0,0 +1,105 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! CG-05 acceptance over the same real engines, SQLite images and TLS controllers +//! as native_cell_groups_execution. Each named case retains direct fixture oracles. +#![allow(clippy::disallowed_methods)] + +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; +// These shared fixture modules expose other execution scenarios to the sibling +// integration target. Only the CG-05 scenarios are selected in this target. +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + +include!("support/cell_group_engine_fixture.rs"); + +// Construct large fixture futures outside the composite test's poll frame. +// Boxing at the call site alone still leaves their construction temporaries on +// that frame in debug builds, consuming stack while a nested fixture is polled. +#[inline(never)] +fn fixture>>( + make: impl FnOnce() -> F, +) -> std::pin::Pin> { + Box::pin(make()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_contains_every_cell_at_cut() -> Result<()> { + // Hold an actual source while two other cell mutations commit, then verify + // every original image and manifest page at one cut in both representations. + checkpoint_fixture::coherent().await?; + checkpoint_fixture::refusals().await?; + checkpoint_fixture::corruption().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn partial_snapshot_never_serves() -> Result<()> { + runtime_fixture::refuse_checkpoint().await?; + for rooted in [false, true] { + runtime_fixture::checkpoint_owner(rooted).await?; + Box::pin(runtime_fixture::seed_receiver_faults(rooted)).await?; + Box::pin(runtime_fixture::seed_transfer_revocation(rooted)).await?; + Box::pin(runtime_fixture::seed_persistence(rooted)).await?; + // These parents launch this same executable's ignored child and verify + // actual retained disks after owned SIGKILL at all six install cuts. + Box::pin(runtime_fixture::seed_crash(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn whole_group_learner_and_joint_change() -> Result<()> { + for rooted in [false, true] { + // Erase one voter, install every cell into a fresh learner, require + // catch-up before promotion, reconnect and refuse isolated commits. + fixture(|| runtime_fixture::seed_source(rooted)).await?; + fixture(|| runtime_fixture::seed_install(rooted)).await?; + fixture(|| runtime_fixture::membership_lifecycle(rooted)).await?; + fixture(|| runtime_fixture::membership_crash(rooted)).await?; + } + fixture(runtime_fixture::membership_sync).await?; + fixture(runtime_fixture::seed_minority).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_empty_cell_and_extent_restart() -> Result<()> { + for rooted in [false, true] { + // Raft generates the snapshot over TLS after prefix compaction. The + // selected/retained closures include empty enrollment and later writes, + // and cold reopen verifies the exact checkpoint plus surviving suffix. + runtime_fixture::tcp_publication(rooted).await?; + runtime_fixture::install_checkpoint(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn snapshot_memory_and_pin_bounds() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::transfer_pressure(rooted).await?; + } + checkpoint_fixture::ownership().await?; + runtime_fixture::local_shutdown().await?; + runtime_fixture::checkpoint_shutdown().await?; + runtime_fixture::automatic_checkpoints(false, true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by partial_snapshot_never_serves"] +async fn seed_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::seed_process_fixture()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "owned child process controlled by whole_group_learner_and_joint_change"] +async fn membership_process_fixture() -> Result<()> { + Box::pin(runtime_fixture::membership_process_fixture()).await +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint_local.rs b/crates/cellar/tests/support/cell_group_checkpoint_local.rs index d0d85b7c7..bcb39f105 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint_local.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint_local.rs @@ -426,6 +426,21 @@ async fn publication(rooted: bool, transfer: Transfer) -> Result<()> { .blocking(move || recover_source(recovering, pages, config, rooted))? .await??; assert_eq!(recovered.marker(), &source); + assert_eq!(recovered.pinned_images(), 4); + let (empty, empty_image) = recovered + .pin_cell(&CellId::new("empty")?) + .context("retained empty enrollment")?; + assert_eq!(empty.version.sequence.get(), 1); + assert_eq!(empty.version.image.txid, 0); + assert!(matches!(empty_image, CellImagePin::Empty)); + for name in ["alpha", "beta", "suffix"] { + let (leaf, image) = recovered + .pin_cell(&CellId::new(name)?) + .context("retained populated cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + assert!(!matches!(image, CellImagePin::Empty)); + } + let mut retained = pool.capture_snapshot(&group, &recovered, &snapshot)?; assert_eq!(retained.marker(), &cut); domain diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs new file mode 100644 index 000000000..581c77c10 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -0,0 +1,584 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +// Shared at each integration target root to preserve the support modules' paths. +use anyhow::{Context, Result}; +use cellar::application::consensus::verification::cell_groups::GroupRoute; +use cellar::application::consensus::{ + verification::{cell_groups::*, *}, + *, +}; +use cellar::sdk::ExecutionDomain; +use cellar_ltx::{ + apply::{ImageState, ImageUpdate, LtxSegment}, + generation::{GenerationPin, GenerationStore, ImageId}, + DirectFileSystem, TXID, +}; +use raft::prelude::{Message, MessageType}; +use sha2::{Digest, Sha256}; +use std::{collections::VecDeque, sync::Arc, time::Duration}; + +struct Voter { + log: ReplicaLog, + owner: Option, + pages: Option, + images: GenerationStore, + extents: Option, + tree: ManifestTree, + gate: Arc, + paused: bool, + held: Option, +} + +struct Cluster { + _root: tempfile::TempDir, + domain: ExecutionDomain, + directory: RangeDirectory, + store: Arc, + birth: CellGroupBirth, + voters: Vec, + network: VecDeque, + reads: Vec<(usize, Term, LogIndex, Vec)>, + options: RaftRuntimeOptions, +} + +impl Cluster { + async fn new() -> Result { + Self::with_options(RaftRuntimeOptions::default()).await + } + + async fn with_options(options: RaftRuntimeOptions) -> Result { + Box::pin(Self::with_storage( + options, + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + )) + .await + } + + async fn with_storage( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + ) -> Result { + Box::pin(Self::with_storage_schema( + options, root, filesystem, [6; 32], + )) + .await + } + + async fn with_storage_schema( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + ) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let store = Arc::new(object_store::memory::InMemory::new()); + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let namespace = domain + .run(catalog.open_namespace(NamespaceConfig::new( + history.cluster, + history.namespace, + cellar_logic::NamespaceMode::Raft, + schema, + ))) + .await??; + let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + creation: ProposalId::new([5; 16])?, + voters: (1..=3) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + domain + .run(directory.reserve_group_birth(&request)) + .await??; + let init = directory + .prepare_initialization(ProposalId::new([7; 16])?, &[request.route().clone()])?; + domain.run(directory.publish(&init)).await??; + let birth = domain + .run(directory.materialize_group_birth(&request)) + .await??; + let mut voters = Vec::new(); + for n in 1..=3 { + let base = root.path().join(format!("node-{n}")); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([20; 16])?, + LogOpenMode::Create, + ), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let tree = ManifestTree::new(birth.route().clone(), ManifestLimits::default())?; + let mut pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let marker = permit.group().group_manifest()?.context("genesis root")?; + let prepared = tree.prepare_complete( + permit.group(), + &mut pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| panic!("empty genesis"), + )?; + let owner = domain + .run(CellGroupOwner::start( + permit, + prepared, + options.clone(), + [n; 32], + )) + .await??; + let images = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + voters.push(Voter { + log, + owner: Some(owner), + pages: Some(pages), + images, + extents: None, + tree, + gate, + paused: false, + held: None, + }); + } + Ok(Self { + _root: root, + domain, + directory, + store, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }) + } + + async fn permit(&self, voter: usize) -> Result { + Ok(self + .domain + .run( + self.directory + .bootstrap_group_member(&self.birth, &self.voters[voter].log), + ) + .await??) + } + + async fn drive(&mut self, isolated: bool) -> Result<()> { + self.drive_partition(isolated, None).await + } + + async fn drive_partition(&mut self, isolated: bool, excluded: Option) -> Result<()> { + for _ in 0..2000 { + let mut progress = false; + for (n, voter) in self.voters.iter_mut().enumerate() { + if excluded == Some(n) { + continue; + } + let event = voter.owner.as_mut().unwrap().poll()?; + let Some(event) = event else { continue }; + progress = true; + match event { + CellGroupEffect::Persist(work) => { + let done = work.run().await?; + voter.owner.as_mut().unwrap().complete_persist(done)?; + } + CellGroupEffect::Message(message) => { + if !isolated { + self.network.push_back(message); + } + } + CellGroupEffect::Apply(apply) => { + if voter.paused { + assert!(voter.held.replace(*apply).is_none()); + } else { + let prepared = prepare(voter, &apply)?; + voter + .owner + .as_mut() + .unwrap() + .complete_apply(apply.ticket, prepared)?; + } + } + CellGroupEffect::Applied { ticket, manifest } => { + assert_eq!(ticket.position(), manifest.position()); + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(manifest.as_ref()) + ); + } + CellGroupEffect::CheckpointInstalled { manifest } + | CellGroupEffect::CheckpointPublished { manifest } => { + assert_eq!( + voter.log.cell_groups()?[0].group_manifest()?.as_ref(), + Some(manifest.as_ref()) + ); + } + CellGroupEffect::ReadIndex { + term, + index, + context, + } => self.reads.push((n, term, index, context)), + } + } + for _ in 0..self.network.len() { + let message = self.network.pop_front().unwrap(); + let target = message.to as usize - 1; + if excluded == Some(target) || excluded == Some(message.from as usize - 1) { + continue; + } + match self.voters[target] + .owner + .as_mut() + .unwrap() + .step(message.clone()) + { + Ok(()) => progress = true, + Err(RuntimeError::NotReady) => self.network.push_back(message), + Err(error) => return Err(error.into()), + } + } + // A bounded receiver may refuse messages until a held apply settles. + // Keep those messages for the next pump instead of busy-spinning. + if !progress { + return Ok(()); + } + } + anyhow::bail!("group did not quiesce") + } + + async fn elect(&mut self) -> Result<()> { + self.voters[0].owner.as_mut().unwrap().campaign()?; + self.drive(false).await?; + assert!(self.voters[0].owner.as_ref().unwrap().status()?.ready); + Ok(()) + } + + async fn close(self) -> Result<()> { + for mut voter in self.voters { + drop(voter.owner.take()); + voter.log.close().await?; + } + self.domain.shutdown().await; + Ok(()) + } +} + +fn image_id(history: &CellHistory, sequence: u64) -> Result { + let mut hash = Sha256::new(); + hash.update(history.cell.as_str()); + hash.update(sequence.to_be_bytes()); + Ok(ImageId::new(hash.finalize()[..16].try_into()?)?) +} + +fn apply_image( + store: &GenerationStore, + history: &CellHistory, + sequence: u64, + base: Option<&GenerationPin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + let header = cellar_ltx::ltx::Header::parse(files.last().context("LTX")?.bytes)?; + let ImageCommitment::Sha256(digest) = image.commitment else { + anyhow::bail!("materialized commitment required") + }; + let result = ImageState { + epoch: image.epoch, + txid: TXID(image.txid), + checksum: image.checksum, + digest, + page_size: header.page_size, + length: u64::from(header.page_size) * u64::from(header.commit), + }; + let update = ImageUpdate { + base: match base { + Some(pin) => pin.state(), + None => ImageState::empty(image.epoch)?, + }, + result, + segments: files + .iter() + .map(|f| LtxSegment { + bytes: f.bytes, + min_txid: TXID(f.min_txid), + max_txid: TXID(f.max_txid), + page_count: f.page_count as usize, + decoded_bytes: f.decoded_bytes, + }) + .collect(), + }; + Ok(store.prepare(image_id(history, sequence)?, base, &update)?) +} + +fn prepare_cell_image( + images: &GenerationStore, + extents: Option<&cellar_ltx::internal::NodeExtentStore>, + cell: &CellHistory, + sequence: u64, + base: Option<&CellImagePin>, + image: LtxImage, + files: &[GroupLtx<'_>], +) -> Result { + use cellar_ltx::internal::{ + ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, + }; + if image.txid == 0 { + anyhow::ensure!(base.is_none() && files.is_empty()); + return Ok(EnrollmentImage { + pin: CellImagePin::Empty, + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + effects: EffectSummary::default(), + }); + } + let (pin, geometry) = match image.commitment { + ImageCommitment::Sha256(_) => { + let base = match base { + Some(CellImagePin::Materialized(pin)) => Some(pin), + None => None, + _ => anyhow::bail!("materialized base"), + }; + let pin = apply_image(images, cell, sequence, base, image, files)?; + let geometry = CellImageGeometry { + length: pin.state().length, + page_size: pin.state().page_size, + }; + (CellImagePin::Materialized(pin), geometry) + } + ImageCommitment::Extents(result) => { + let store = extents.context("extent store")?; + let base = match base { + Some(CellImagePin::Extents(pin)) => Some(pin.as_ref()), + None => None, + _ => anyhow::bail!("extent base"), + }; + let binding = ImageBinding { + history: cell_image_history_digest(cell), + schema: [6; 32], + epoch: image.epoch, + txid: image.txid, + checksum: image.checksum, + }; + let empty = ExtentManifest::new( + ImageBinding { + txid: 0, + checksum: 0, + ..binding + }, + Geometry::new(0, 0)?, + &[], + )?; + let before = base + .map(|pin| pin.reference()) + .unwrap_or(ExtentRootReference::new( + empty.root(), + empty.encode()?.len().try_into()?, + )?); + let header = cellar_ltx::ltx::Header::parse(files.last().context("extent LTX")?.bytes)?; + let geometry = Geometry::new( + header.page_size, + u64::from(header.page_size) * u64::from(header.commit), + )?; + let update = ImageRootUpdate { + base: before, + result, + binding, + geometry, + segments: files + .iter() + .map(|file| LtxSegment { + bytes: file.bytes, + min_txid: TXID(file.min_txid), + max_txid: TXID(file.max_txid), + page_count: file.page_count as usize, + decoded_bytes: file.decoded_bytes, + }) + .collect(), + }; + let prepared = store.prepare_image_root( + base, + &update, + &images.apply_limits(), + &mut vec![0; EXTENT_BYTES as usize], + )?; + ( + CellImagePin::Extents(Arc::new(prepared.image)), + CellImageGeometry { + length: geometry.length(), + page_size: geometry.page_size(), + }, + ) + } + }; + Ok(EnrollmentImage { + pin, + geometry, + effects: EffectSummary::default(), + }) +} + +fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { + if apply.entry.data.is_empty() { + return Ok(voter.tree.prepare_unchanged( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + apply.ticket.position(), + )?); + } + let decoded = CellGroupEntry::decode( + &apply.entry.data, + apply.base.marker().route(), + GroupEntryLimits::default(), + )?; + match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let (next, _) = voter.tree.prepare_enrollment( + voter.pages.as_mut().context("fixture page owner")?, + &apply.base, + EnrollmentApply { + position: apply.ticket.position(), + phase: EnrollmentPhase::Active, + entry: &apply.entry.data, + limits: GroupEntryLimits::default(), + }, + |_| { + prepare_cell_image( + &voter.images, + voter.extents.as_ref(), + &cell, + 1, + None, + image, + &files, + ) + .map_err(|_| ManifestError::Invalid("fixture initial LTX")) + }, + )?; + Ok(next) + } + GroupCommand::Mutation(mutation) => { + let (old, base) = apply.base.pin_cell(&mutation.cell.cell).context("cell")?; + let prepared = prepare_cell_image( + &voter.images, + voter.extents.as_ref(), + &mutation.cell, + mutation.sequence.get(), + Some(&base), + mutation.result, + &mutation.files, + )?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: prepared.geometry, + last_mutation: apply.ticket.position(), + witness: CommitWitness { + group: decoded.binding.group, + position: apply.ticket.position(), + proposal: decoded.binding.proposal, + }, + effects: old.effects, + }; + Ok(voter.tree.prepare_update( + voter.pages.as_mut().context("fixture page owner")?, + &apply.base, + leaf, + Some(&old.version), + prepared.pin, + )?) + } + _ => anyhow::bail!("unexpected administration"), + } +} + +fn history(route: &GroupRoute, name: &str) -> Result { + Ok(CellHistory { + cluster: route.history.cluster, + namespace: route.history.namespace, + cell: CellId::new(name)?, + incarnation: IncarnationId::new([8; 16])?, + }) +} + +fn request( + route: &GroupRoute, + cell: CellHistory, + sequence: u64, + cut: &images::Cut, +) -> Result> { + let image = |state: ImageState| LtxImage { + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + commitment: state.digest.into(), + }; + let files = vec![GroupLtx { + min_txid: cut.base.txid.0 + 1, + max_txid: cut.result.txid.0, + page_count: cut.pages.len() as u64, + decoded_bytes: cut.pages.len() as u64 * 512, + bytes: &cut.bytes, + }]; + let command = if sequence == 1 { + GroupCommand::Enroll { + cell, + image: image(cut.result), + files, + } + } else { + GroupCommand::Mutation(GroupMutation { + cell, + sequence: MutationSequence::new(sequence)?, + base: image(cut.base), + result: image(cut.result), + files, + }) + }; + Ok(CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: ProposalId::new([sequence as u8; 16])?, + }, + command, + } + .encode(route, GroupEntryLimits::default())?) +} diff --git a/crates/cellar/tests/support/cell_group_membership_crash.rs b/crates/cellar/tests/support/cell_group_membership_crash.rs index a070b4f35..2d7e6dc85 100644 --- a/crates/cellar/tests/support/cell_group_membership_crash.rs +++ b/crates/cellar/tests/support/cell_group_membership_crash.rs @@ -369,7 +369,7 @@ async fn recover(root: &Path, rooted: bool, joint: bool) -> Result<()> { assert_eq!(selected.cell().version.sequence.get(), 1); } assert!(!nodes[2].runtime.status(route.history).await?.ready); - cellar::note!("membership crash rooted={rooted} joint_at_kill={joint} killed=true recovered_replicas=3 final_cells=4 final_index={}", final_index.get()); + eprintln!("membership crash rooted={rooted} joint_at_kill={joint} killed=true recovered_replicas=3 final_cells=4 final_index={}", final_index.get()); for node in nodes { node.controller.close().await?; node.log.close().await?; diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs index 61f72950e..c9dd94608 100644 --- a/crates/cellar/tests/support/cell_group_seed_crash.rs +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -341,7 +341,7 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { assert!(group.checkpoint_manifest()?.is_none()); assert_eq!(recovered.pinned_images(), 0); } - cellar::note!( + eprintln!( "seed crash rooted={rooted} cut={cut} killed=true recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get() diff --git a/crates/cellar/tests/support/cell_group_seed_install.rs b/crates/cellar/tests/support/cell_group_seed_install.rs index 412142838..8f826eb9d 100644 --- a/crates/cellar/tests/support/cell_group_seed_install.rs +++ b/crates/cellar/tests/support/cell_group_seed_install.rs @@ -447,7 +447,34 @@ async fn replace( .group_manifest()? .context("source cut")?; - // Mutations after source capture must arrive through the Raft suffix. + let release = gate.arm(fault::HOLD_SYNC); + let installer = applier.clone(); + let pending = fixture + .cluster + .domain + .spawn(async move { session.install(installer).await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert!( + fixture + .cluster + .domain + .run(fixture.cluster.directory.bootstrap_joined_group_member( + &fixture.cluster.birth, + local.member, + &log + )) + .await? + .is_err(), + "receipt cannot precede the storage fence" + ); + assert_eq!( + pool.usage().sources, + 1, + "physical receiver retained through sync" + ); + // While the replacement receiver is physically held at its install fence, + // both the surviving group and an unrelated group must keep committing. + // These later mutations must reach the learner through the Raft suffix. let later = history(&route, "after-seed-cut")?; let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(73);")?; let accepted = fixture.runtimes[0] @@ -474,31 +501,6 @@ async fn replace( ) .await?; survivors(&fixture, second, unrelated.position.index).await?; - let release = gate.arm(fault::HOLD_SYNC); - let installer = applier.clone(); - let pending = fixture - .cluster - .domain - .spawn(async move { session.install(installer).await })?; - tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; - assert!( - fixture - .cluster - .domain - .run(fixture.cluster.directory.bootstrap_joined_group_member( - &fixture.cluster.birth, - local.member, - &log - )) - .await? - .is_err(), - "receipt cannot precede the storage fence" - ); - assert_eq!( - pool.usage().sources, - 1, - "physical receiver retained through sync" - ); let (permit, root) = if rooted { drop(release); pending.await?? diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs index 26f2acc91..ed1af4f72 100644 --- a/crates/cellar/tests/support/cell_group_seed_persistence.rs +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -410,7 +410,7 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { assert!(group.checkpoint_manifest()?.is_none()); assert_eq!(recovered.marker().root().cells(), 0); } - cellar::note!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); + eprintln!("verified seed persistence rooted={rooted} cut={cut:?} recovered_cells={} recovered_index={}", selected.root().cells(), selected.position().index.get()); drop((applier, recovered, group)); log.close().await?; if let Some(source) = restarted_source { diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 54f779e14..4f591d54a 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1601,7 +1601,8 @@ "crates/cellar/tests/support/cell_group_seed_crash.rs", "crates/cellar/tests/support/cell_group_crash_fixture.rs", "crates/cellar/tests/support/cell_group_membership_crash.rs", - "crates/cellar/tests/support/cell_group_checkpoint_pressure.rs" + "crates/cellar/tests/support/cell_group_checkpoint_pressure.rs", + "crates/cellar/tests/support/cell_group_engine_fixture.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 997a53b3d6ee20ddff37987e7f00511c6e01469a Mon Sep 17 00:00:00 2001 From: forhappy Date: Fri, 30 Oct 2026 23:52:55 -0700 Subject: [PATCH 153/568] docs: record cell-group snapshot acceptance gate --- docs/design/multi-raft/cell-groups/README.md | 13 ++-- docs/design/multi-raft/cell-groups/handoff.md | 53 +++++++++++----- .../design/multi-raft/cell-groups/progress.md | 60 +++++++++++++++++++ 3 files changed, 105 insertions(+), 21 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 206691b4a..74a00d5e0 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -21,7 +21,8 @@ Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. Explicit joint-consensus SIGKILL/restart and completion now pass in both image forms. Bounded 32-cell transfer, scratch contention and old-reader image retention/release now pass in both image forms. Partial/synced seed-dependency SIGKILL cuts and transfer-time source-process -revocation now pass. Remaining CG-05 controller pressure and full acceptance gates precede +revocation now pass. All five exact native acceptance cases and their prescribed command gate now pass. +Remaining CG-05 lifecycle/availability qualification precedes verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. @@ -99,13 +100,13 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff The implementation executor should continue from the existing code and retained -evidence. Review the working diff before resuming CG-04; intermediate owner and -storage checkpoints do not replace its full shared-execution gate. +evidence. Resume CG-05 from its exact acceptance gate and remaining lifecycle/availability +obligations; intermediate checkpoints do not establish full task completion. -Continue CG-04 after reading +Continue CG-05 after reading [progress](progress.md) and inspecting the working diff. -CG-01 through CG-03 are DONE only at their named contract/storage/admission -boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; +CG-01 through CG-04 are DONE only at their named contract/storage/admission +and shared-execution boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; an IN_PROGRESS task is resumed from its existing brief and working state. Each task's complete source-bound evidence gate must pass before changing its status to DONE. Continue in the generated dependency order afterward. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 02a192493..fe5a560ef 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -35,14 +35,39 @@ Automatic entry-count checkpointing at controller apply boundaries now passes it partial gate. Durable whole-group membership and leadership handoff now pass an additional partial gate. Authenticated fresh-learner installation, catch-up, promotion and deleted-voter-disk replacement now pass a native partial gate. -Complete the remaining fault/pressure checks and all five CG-05 acceptance cases. +All five CG-05 acceptance cases now pass; complete the remaining fault/pressure +and retry-availability obligations before closing the task. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest interrupted-transfer checkpoint at `6e56f21f` passes all 144 +The latest exact-acceptance checkpoint at `f076731a` delivers +`native_cell_groups_snapshot` with all five required native oracles. Each passes +individually with `--exact --test-threads=1`, and all ten prescribed task commands +pass. Fifteen total commands include 144 execution regression cases and package +checks. The record under `.cellar/cell-groups-evidence/CG-05-snapshot-acceptance/` +binds 332 source hashes and two executables; both independent hashes and the +plan's full `check_evidence` validator pass. CG-05 remains IN_PROGRESS pending the +controller/lifecycle and orphan-retry audit; the new gate is not a release claim. + +Shared root-level engine helpers now live in `cell_group_engine_fixture.rs`; +the existing execution-only restart method stays in the original target. Exact +cases compose the real fixtures and preserve owned-child names. Writes now commit +in the replacement and unrelated groups while a learner install fence is held; +cold restart explicitly checks empty and populated cell sequences/images. The +composite membership test constructs futures outside its poll frame after phase +probes reproduced a debug stack overflow. The standalone case passed and the +fixed complete composite passes; no stack limits, timeouts or assertions changed. + +Next close the declared controller checkpoint lifecycle/saturation gaps and audit +staged orphan reconciliation before authenticated retry. Preserve fresh quorum, +physical-owner retention and selected-root checks. Production runtime code was +unchanged in this checkpoint; RustFS/load, independent-domain and public host +cutover qualification remain open. + +The earlier interrupted-transfer checkpoint at `6e56f21f` passes all 144 shared-execution tests and nineteen native Raft-storage tests, with eight scoped commands. Its record under `.cellar/cell-groups-evidence/CG-05-seed-transfer-cuts/` binds 330 source hashes and two executables. The seed crash matrix now includes @@ -58,11 +83,11 @@ physical release drains it without publishing any seed. Cold recovery again refuses startup. These revocation cases use an independently authenticated captured peer. Production runtime code is unchanged. No automatic orphan cleanup/retry, all-wire-boundary, power-loss, independent-domain or RustFS/load claim is made. -Next audit the five required CG-05 oracles, deliver their exact test target and -close the remaining controller/lifecycle saturation requirements. +That checkpoint preceded the exact acceptance target delivered above; its +controller/lifecycle and retry-availability limits remain explicit. -The starting fixture mapping for the exact target is below. This is an audit map, -not evidence that the named target exists or its five gates have passed. Reuse the +The earlier starting fixture mapping for the exact target is below. The newer +checkpoint above implements that target and records its exact five-case gate. Reuse the real fixture owners and direct assertions, without launching another test binary as an oracle or replacing any required case. @@ -74,12 +99,10 @@ as an oracle or replacing any required case. | `snapshot_empty_cell_and_extent_restart` | `tcp_publication` transfers the complete populated/empty closure and reopens retained checkpoint plus current suffix in both forms. | | `snapshot_memory_and_pin_bounds` | `transfer_pressure` and `local_shutdown` retain bounded physical work and old reader images through concurrent apply/shutdown. | -The native execution test currently keeps reusable imports, `Voter`, `Cluster` -and image/apply helpers between the `images` module declaration and -`three_cells_one_rawnode_per_voter`. Extract those helpers without changing their -semantics so the new snapshot target shares the same real engine fixture. Preserve -the root module paths used by the support files and the exact ignored child names -used by owned-process tests. +The shared imports, `Voter`, `Cluster` and image/apply helpers are now extracted +without semantic changes into `cell_group_engine_fixture.rs`. Both integration +targets include them at their root, preserving support-module paths and exact +ignored child names used by owned-process tests. The earlier pressure checkpoint at `a2036e50` passes all 142 native shared-execution tests and eight scoped commands. Its record under @@ -851,9 +874,9 @@ Do not let plan maintenance replace implementation. 1. Finish the full CG-05 acceptance gate around the implemented checkpoint, learner and membership paths. Bounded 32-cell transfer and physical reader retention/release now pass, as do partial/synced dependency crash cuts and - transfer-time source-process revocation. Audit all five exact oracles and - deliver the declared native_cell_groups_snapshot target, closing remaining - fault/lifecycle requirements below. Preserve permanent claims, local install receipts, complete + transfer-time source-process revocation. The declared snapshot target and all + five exact cases now pass their prescribed command gate. Close the remaining + fault/lifecycle and availability requirements below before task completion. Preserve permanent claims, local install receipts, complete physical recovery and unrelated-group progress. 2. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt refusal, continuous capture fairness, exhausted apply/completion budgets, held sync/cancelled close, diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 4e2f28de1..a471d5fd4 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,66 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 exact native snapshot acceptance target + +Qualified source: `f076731a2d384bff65914828409c01506438c4a2`. The declared +`native_cell_groups_snapshot` target now contains all five required named cases, +plus two ignored child entry points used by its actual process-kill parents. +Every required case passes separately with `--exact --test-threads=1`; inventory +and exact one-test results pass the plan's evidence validator. The +execution target's 144 regression cases also pass. Fifteen commands pass: all ten +prescribed CG-05 commands, the execution regression suite, dependency isolation, +docs, source preflight and 37 plan-validator tests. The prescribed commands include +all-target checks and Clippy for cellar/cellar-logic, formatting and layout. + +The record `.cellar/cell-groups-evidence/CG-05-snapshot-acceptance/checkpoint.json` +binds 332 source hashes, two executables, frozen plan inputs and intermediate +logs. Independent hash verification and `check_evidence` pass. No required case +or command was removed or renamed. This record does not mark the task complete; +remaining lifecycle/availability gaps below keep CG-05 IN_PROGRESS. + +The cases directly compose existing real-engine, SQLite/LTX, TLS and owned-crash +fixtures. They do not infer an oracle from another test process's success. The +engine helper extraction preserves its body byte-for-byte after keeping the +execution-only import and restart method in the original target. The process-kill +children retain their exact names and run the same snapshot executable. Fixture +progress messages use ordinary test capture, leaving the validator's required +per-case pass lines unambiguous while retaining failure diagnostics. + +| Exact case | Direct native coverage | +| --- | --- | +| `snapshot_contains_every_cell_at_cut` | Both image forms, two concurrent cell advances, exact original closure, stale/foreign cut refusal and corrupt physical bytes. | +| `partial_snapshot_never_serves` | Receiver corruption/partial/foreign/budget failures, exact Raft persistence, forged TLS seed data, revoked source, sync failures and twelve owned seed SIGKILL scenarios. | +| `whole_group_learner_and_joint_change` | Fresh quorum seed authority, deleted-voter replacement, complete learner install and catch-up before promotion, explicit/implicit joint changes, leader handoff, four explicit joint SIGKILL scenarios, sync ownership and minority refusal. | +| `snapshot_empty_cell_and_extent_restart` | Raft-generated TLS catch-up, empty and populated enrollment, retained checkpoint/current suffix cold recovery and missing closure refusal in both forms. | +| `snapshot_memory_and_pin_bounds` | 32-cell transfers, scratch/metadata admission, unique old reader preservation and actual deletion after release, cancellation, joined publication/installation shutdown and controller pool contention. | + +The audit strengthened two fixture oracles. Replacement-group and unrelated-group +writes now commit while the receiver is physically held at its install fence; +the learner later obtains those cell changes through the Raft suffix. Cold +checkpoint restart explicitly verifies empty enrollment remains at sequence 1, +transaction 0 and an empty image, while alpha/beta/suffix retain populated images +at sequence 1. + +The initial composite membership test overflowed its stack even with boxed +futures. Phase probes located the failure entering seed installation, whose +standalone counterpart passed. A non-inlined future factory releases construction +temporaries before polling nested fixtures; the complete original case then +passes in both image modes. No stack limit, timeout, assertion or fault was +relaxed. Temporary phase probes were removed and their logs retained. Production +runtime behavior is unchanged. + +**Reflection and next step:** the exact test target and its prescribed command +gate are delivered. The next concrete test should hold controller checkpoint capture, fill its +lifecycle capacity, then cancel close and verify that physical owners remain +charged until release and shutdown joins them. Follow with staged-orphan cleanup +and authenticated retry on the same disk; safe +refusal of interrupted state alone does not prove availability. Retain explicit +gaps for continuous capture fairness, minority signed receipts and retained-root +pressure. This is native qualification, not real RustFS/load, independent domains +or production SDK/fleet cutover. Complete the remaining CG-05 obligations before +CG-06 archive/reclamation; the full hard cutover remains pending. + ## CG-05 interrupted seed dependency writes and process revocation Qualified source: `6e56f21f2209ce3f02d78ffeeb3150a763a1476e`. All 144 native shared-execution tests From 0847930ceb22a1cc364b53b8324ce872c5201b91 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 00:02:25 -0700 Subject: [PATCH 154/568] test: qualify controller checkpoint shutdown ownership --- .../tests/native_cell_groups_execution.rs | 10 ++ .../tests/native_cell_groups_snapshot.rs | 2 + .../cell_group_controller_checkpoints.rs | 135 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 +- crates/cellar/tests/support/cell_group_sql.rs | 2 +- .../tests/support/raft_storage_fault.rs | 19 ++- 6 files changed, 167 insertions(+), 3 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index bbe447efb..0a8a48544 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -141,6 +141,16 @@ async fn checkpoint_controller_busy_pool_preserves_apply_and_other_group_progres runtime_fixture::automatic_checkpoints(false, true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_materialized_holds_sync_and_joins_cancelled_close() -> Result<()> { + runtime_fixture::controller_checkpoint_shutdown(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_extents_hold_sync_and_join_cancelled_close() -> Result<()> { + runtime_fixture::controller_checkpoint_shutdown(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_tcp_refuses_invalid_streams_and_preserves_other_group_progress() -> Result<()> { runtime_fixture::tcp_protocol().await diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 7b8d5af8b..28bd62285 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -89,6 +89,8 @@ async fn snapshot_memory_and_pin_bounds() -> Result<()> { checkpoint_fixture::ownership().await?; runtime_fixture::local_shutdown().await?; runtime_fixture::checkpoint_shutdown().await?; + fixture(|| runtime_fixture::controller_checkpoint_shutdown(false)).await?; + fixture(|| runtime_fixture::controller_checkpoint_shutdown(true)).await?; runtime_fixture::automatic_checkpoints(false, true).await } diff --git a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs index 2d066084d..4fd787aac 100644 --- a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs +++ b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs @@ -87,3 +87,138 @@ pub(crate) async fn automatic_checkpoints(rooted: bool, pressure: bool) -> Resul drop(controllers); fixture.close().await } + +pub(crate) async fn controller_checkpoint_shutdown(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[1].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[1].status(second).await?.ready)).await?; + let elected = fixture.runtimes[1].status(second).await?.applied.index; + controller::applied(&fixture, second, elected).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("preceding durable cut")?; + assert!(group.checkpoint_manifest()?.is_none()); + let controller = controllers[0].clone(); + let checkpoint_pool = pools[0].clone(); + let gate = fixture.cluster.voters[0].gate.clone(); + // Pass ordinary append/commit fences. Hold a sync only after the controller + // owns both its lifecycle slot and the captured checkpoint reservation. + let release = gate.arm_sync_when(move || { + controller.usage().active_checkpoints == 1 && checkpoint_pool.usage().sources == 1 + }); + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(29);")?; + let cell = history(&route, "checkpoint-successor")?; + let successor = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(controllers[0].usage().active_checkpoints, 1); + let charged = pools[0].usage(); + assert_eq!(charged.sources, 1); + assert!(charged.retained_bytes > 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before.position() + ); + assert!(matches!( + pools[0].receive( + fixture.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + ), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pools[0].usage(), charged); + + // This node's shared engine is stalled. The other two real voters still + // form a quorum and must advance the other group through repeated writes. + for round in 0..4 { + let selected = fixture.cluster.voters[1] + .log + .cell_group(&second)? + .group_manifest()? + .context("healthy group")?; + let cell = history(selected.route(), &format!("while-checkpoint-sync-{round}"))?; + let accepted = fixture.runtimes[1] + .propose( + second, + apply::enrollment_bytes(selected.route(), &cell, &bytes, false)?, + ) + .await?; + controller::wait_for(async || { + for runtime in &fixture.runtimes[1..] { + if runtime.status(second).await?.applied.index < accepted.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert_eq!(controllers[0].usage().active_checkpoints, 1); + assert_eq!(pools[0].usage(), charged); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + } + // Cancelling both close and wait leaves the cleanup task, source pins and + // exclusive engine owner alive until the actual sync returns. + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].wait()) + .await + .is_err() + ); + assert!(controllers[0].usage().closing); + assert_eq!(controllers[0].usage().active_checkpoints, 1); + assert_eq!(pools[0].usage(), charged); + assert!(matches!( + CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + assert!(apply::pool(&fixture, 0).is_err()); + drop(release); + tokio::time::timeout(Duration::from_secs(5), controllers[0].close()).await??; + assert_eq!(controllers[0].usage().active_checkpoints, 0); + assert_eq!(pools[0].usage().sources, 0); + assert_eq!(pools[0].usage().retained_bytes, 0); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&before)); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.last_index()? >= successor.position.index.get()); + let (replacement, events) = CellGroupRuntime::start( + fixture.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default(), + ) + .await?; + replacement.close().await?; + drop(events); + for controller in controllers { + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + } + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 712c7d2c4..c5805ab1b 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -814,7 +814,7 @@ pub(super) use sql::{ provisioning_retries_retained_owner, }; -pub(super) use sql::automatic_checkpoints; +pub(super) use sql::{automatic_checkpoints, controller_checkpoint_shutdown}; pub(super) use sql::{membership_lifecycle, membership_sync}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 0ea4b194a..6bc917280 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -594,7 +594,7 @@ pub(crate) use provision::{ #[path = "cell_group_controller_checkpoints.rs"] mod controller_checkpoints; -pub(crate) use controller_checkpoints::automatic_checkpoints; +pub(crate) use controller_checkpoints::{automatic_checkpoints, controller_checkpoint_shutdown}; #[path = "cell_group_membership.rs"] mod membership; diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index 893c70c09..b3b786760 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -40,6 +40,7 @@ pub struct Gate { pub sync_path: Mutex>, pub image_read_paths: Mutex>, pub dependency_cut: Mutex>, + sync_when: Mutex bool + Send + Sync>>>, } impl Gate { @@ -53,15 +54,29 @@ impl Gate { sync_path: Mutex::new(None), image_read_paths: Mutex::new(vec![]), dependency_cut: Mutex::new(None), + sync_when: Mutex::new(None), }) } pub fn arm(self: &Arc, mode: u8) -> Release { + *self.sync_when.lock().unwrap() = None; *self.released.lock().unwrap() = false; self.mode.store(mode, Ordering::Release); Release(self.clone()) } + /// Hold a real log sync only after the owning lifecycle is entered. Earlier + /// Raft append/commit syncs pass normally; no sync-count assumption is needed. + pub fn arm_sync_when( + self: &Arc, + when: impl Fn() -> bool + Send + Sync + 'static, + ) -> Release { + *self.released.lock().unwrap() = false; + *self.sync_when.lock().unwrap() = Some(Arc::new(when)); + self.mode.store(HOLD_SYNC, Ordering::Release); + Release(self.clone()) + } + fn pause(&self) -> io::Result<()> { if let Some(signal) = &self.signal { let mut file = std::fs::File::create(signal)?; @@ -82,6 +97,7 @@ pub struct Release(Arc); impl Drop for Release { fn drop(&mut self) { self.0.mode.store(OFF, Ordering::Release); + self.0.sync_when.lock().unwrap().take(); *self.0.released.lock().unwrap() = true; self.0.release.notify_all(); } @@ -243,7 +259,8 @@ impl HostFileIo for FaultFile { self.pause_dependency()?; return Ok(()); } - if self.log { + let sync_when = self.gate.sync_when.lock().unwrap().clone(); + if self.log && sync_when.is_none_or(|when| when()) { let mode = self.gate.mode.load(Ordering::Acquire); let mode = if matches!( mode, From 370dcf3465b3161c9e69e8b1f398bc9d49450c0c Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 00:12:43 -0700 Subject: [PATCH 155/568] test: keep checkpoint sync selection in its fixture --- .../support/cell_group_controller_checkpoints.rs | 5 +++-- .../cellar/tests/support/raft_storage_fault.rs | 16 ++-------------- 2 files changed, 5 insertions(+), 16 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs index 4fd787aac..1cf5dc20d 100644 --- a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs +++ b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs @@ -117,9 +117,10 @@ pub(crate) async fn controller_checkpoint_shutdown(rooted: bool) -> Result<()> { let gate = fixture.cluster.voters[0].gate.clone(); // Pass ordinary append/commit fences. Hold a sync only after the controller // owns both its lifecycle slot and the captured checkpoint reservation. - let release = gate.arm_sync_when(move || { + *gate.sync_when.lock().unwrap() = Some(Arc::new(move || { controller.usage().active_checkpoints == 1 && checkpoint_pool.usage().sources == 1 - }); + })); + let release = gate.arm(fault::HOLD_SYNC); let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(29);")?; let cell = history(&route, "checkpoint-successor")?; let successor = fixture.runtimes[0] diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index b3b786760..226a59779 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -40,7 +40,8 @@ pub struct Gate { pub sync_path: Mutex>, pub image_read_paths: Mutex>, pub dependency_cut: Mutex>, - sync_when: Mutex bool + Send + Sync>>>, + /// Optional sync selection, installed before arming and cleared on release. + pub sync_when: Mutex bool + Send + Sync>>>, } impl Gate { @@ -59,24 +60,11 @@ impl Gate { } pub fn arm(self: &Arc, mode: u8) -> Release { - *self.sync_when.lock().unwrap() = None; *self.released.lock().unwrap() = false; self.mode.store(mode, Ordering::Release); Release(self.clone()) } - /// Hold a real log sync only after the owning lifecycle is entered. Earlier - /// Raft append/commit syncs pass normally; no sync-count assumption is needed. - pub fn arm_sync_when( - self: &Arc, - when: impl Fn() -> bool + Send + Sync + 'static, - ) -> Release { - *self.released.lock().unwrap() = false; - *self.sync_when.lock().unwrap() = Some(Arc::new(when)); - self.mode.store(HOLD_SYNC, Ordering::Release); - Release(self.clone()) - } - fn pause(&self) -> io::Result<()> { if let Some(signal) = &self.signal { let mut file = std::fs::File::create(signal)?; From 84f032b33a874f62310eebd95d93f3194a7758c2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 00:23:36 -0700 Subject: [PATCH 156/568] docs: record controller checkpoint shutdown qualification --- docs/design/multi-raft/cell-groups/README.md | 3 +- docs/design/multi-raft/cell-groups/handoff.md | 38 +++++++++++--- .../design/multi-raft/cell-groups/progress.md | 52 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 9 +++- 4 files changed, 94 insertions(+), 8 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 74a00d5e0..1206b66c5 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -22,7 +22,8 @@ Explicit joint-consensus SIGKILL/restart and completion now pass in both image f Bounded 32-cell transfer, scratch contention and old-reader image retention/release now pass in both image forms. Partial/synced seed-dependency SIGKILL cuts and transfer-time source-process revocation now pass. All five exact native acceptance cases and their prescribed command gate now pass. -Remaining CG-05 lifecycle/availability qualification precedes +Controller publication through cancelled shutdown now passes in both image forms. +Remaining CG-05 recovery availability and broader pressure qualification precedes verified S3 archive/reclamation. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index fe5a560ef..4be5bf80e 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -43,7 +43,31 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest exact-acceptance checkpoint at `f076731a` delivers +The latest controller-publication checkpoint at `370dcf34` passes 146 +execution cases, nineteen Raft-storage cases and the updated exact resource-bound +snapshot case, with ten scoped commands. Its record under +`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/` binds +332 source hashes and three executables; independent hashes pass. The other +four exact snapshot cases and all-target checks remain at their earlier source. +CG-05 remains IN_PROGRESS and production runtime code is unchanged. + +Both image forms now hold the actual publication sync after lifecycle/source +admission, fill checkpoint capacity and cancel both close and wait. Slot, source, +bytes and engine/SQL ownership remain retained until physical release. The healthy +two-voter quorum commits four writes to the other group during the stall. Release +publishes the preceding checkpoint, leaves the successor unapplied, drains +admission and permits a replacement runtime. Existing fault modes retain their +default behavior; the predicate is opt-in and released by its guard. + +The next concrete implementation is interrupted-seed orphan reconciliation and +fresh authenticated retry. Extent startup inventories partial files without +trusting them; immutable import refuses a partial existing object. Supply complete +node references before collecting unselected orphans, preserve selected current +and checkpoint roots plus live pins, and verify retry on the same disk. A single +group must not reconcile or delete another group's shared objects. Continue the +remaining controller crash/budget/fairness and retained-root pressure qualification. + +The earlier exact-acceptance checkpoint at `f076731a` delivers `native_cell_groups_snapshot` with all five required native oracles. Each passes individually with `--exact --test-threads=1`, and all ten prescribed task commands pass. Fifteen total commands include 144 execution regression cases and package @@ -61,8 +85,9 @@ composite membership test constructs futures outside its poll frame after phase probes reproduced a debug stack overflow. The standalone case passed and the fixed complete composite passes; no stack limits, timeouts or assertions changed. -Next close the declared controller checkpoint lifecycle/saturation gaps and audit -staged orphan reconciliation before authenticated retry. Preserve fresh quorum, +The pending-publication cancelled-shutdown cut now passes above. Continue +remaining controller lifecycle/saturation cuts and staged orphan reconciliation +before authenticated retry. Preserve fresh quorum, physical-owner retention and selected-root checks. Production runtime code was unchanged in this checkpoint; RustFS/load, independent-domain and public host cutover qualification remain open. @@ -878,9 +903,10 @@ Do not let plan maintenance replace implementation. five exact cases now pass their prescribed command gate. Close the remaining fault/lifecycle and availability requirements below before task completion. Preserve permanent claims, local install receipts, complete physical recovery and unrelated-group progress. -2. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt - refusal, continuous capture fairness, exhausted apply/completion budgets, held sync/cancelled close, - quarantine capacity and retained-root pressure. Track automatic orphan cleanup +2. Pending-publication sync and cancelled controller shutdown now pass in both + image forms. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt + refusal, continuous capture fairness, exhausted apply/completion budgets, other + capture/admission cancellation boundaries, quarantine capacity and retained-root pressure. Track automatic orphan cleanup and fresh-seed retry after interrupted dependency writes; safe refusal alone does not prove retry availability. Explicit joint restart after replicated entry/exit now passes; this does not qualify every in-flight or diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index a471d5fd4..41e8c753f 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 controller publication ownership through cancelled shutdown + +Qualified source: `370dcf3465b3161c9e69e8b1f398bc9d49450c0c`. Both new controller publication tests pass +in the complete native execution suite: 146 cases, plus nineteen native +Raft-storage cases and the updated exact `snapshot_memory_and_pin_bounds` case. +Ten scoped commands pass, including Clippy for all three integration targets, +formatting, layout, dependency isolation, docs, source preflight and 37 plan +validator tests. The other four snapshot acceptance cases and all-target checks +were not rerun at this source; their complete earlier gate remains below. + +The record +`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/checkpoint.json` +binds 332 source hashes, three copied executables, frozen plan inputs and +intermediate trial outputs. Independent source/output/executable hashes pass. +This partial record is not a combined full task gate and keeps CG-05 IN_PROGRESS. +Production runtime behavior is unchanged. The first scoped gate passed all +selected Rust cases but failed Clippy on an unused helper in the shared legacy +storage target. That method was replaced with fixture-local predicate setup. +The failed source, outputs and three executables remain hash-bound in +`previous_attempt`; no assertions or fault scenarios were removed. + +The filesystem fault now supports an opt-in sync predicate. Earlier ordinary +append/commit syncs pass. Once the controller owns its checkpoint lifecycle slot +and captured source, the next actual log sync holds. The test waits for all voters +to persist the other group's election before arming this cut; it does not guess a +sync count. Both materialized and extent cases check the following directly: + +- Source count, retained bytes and the one controller slot stay charged. Another + receiver refuses with `Busy`; the prior root and applied position stay selected. +- Four writes to the other group commit and apply on the two healthy voters while + this node's shared engine is stalled. No checkpoint completion is credited early. +- Cancelled close and wait retain physical ownership. Replacement runtime and SQL + pool ownership refuse until the real sync returns. +- Release joins controller shutdown, selects the captured complete checkpoint, + keeps the committed successor unapplied and drains checkpoint admission. A new + runtime can then acquire the engine. The exact resource-bound acceptance case + directly composes both scenarios. + +This qualifies pending publication through cancelled controller shutdown in both +image forms. It does not close every controller crash, completion-budget, capture +fairness or retained-root pressure schedule. Native process tests do not qualify +independent domains, power loss, real RustFS/load or the SDK/fleet cutover. + +**Reflection and next step:** interrupted-seed retry is now the critical recovery +availability gap. The extent inventory deliberately keeps partial known files +charged after reopen, and immutable import verifies existing names instead of +silently overwriting them. Retry needs complete node-reference reconciliation +before orphan collection, preserving all selected current/checkpoint roots and +live pins across groups. Do not add group-local deletion of shared objects or +weaken corruption refusal. Implement and test that reconciliation and a fresh +authenticated retry before closing CG-05 and proceeding to CG-06 archive/reclaim. + ## CG-05 exact native snapshot acceptance target Qualified source: `f076731a2d384bff65914828409c01506438c4a2`. The declared diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 67100708e..bf31bdfc7 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -400,9 +400,16 @@ join. Hard checkpoint failures retain the ticket and quarantine the group, while shared log uncertainty closes the node. Quarantined groups can retain checkpoint capacity until joined restart, like their registration capacity. +Native controller tests hold publication at the real engine sync after lifecycle +and checkpoint-source admission. Cancelled close and wait retain both owners and +the engine/SQL claims through physical completion. The healthy two-voter quorum +continues another group's writes during the stalled node's engine sync. Release +joins shutdown, selects the captured checkpoint and permits a replacement runtime. +Both image forms run this cut; other controller crash/budget schedules remain open. + This is an entry-count policy for active groups. It retains the final idle tail, and time/byte triggers, fairness under continuously saturated checkpoint capacity, -controller checkpoint crash/held-sync cuts and complete root-pressure qualification +controller checkpoint crash cuts and complete root-pressure qualification remain open. It supplies no S3 archive coverage proof. Manual publication remains available to lower-level owners; this policy does not introduce a second serving runtime or weaken exact-cut publication checks. From 5cfa9fb652c32cc75934ef3d0a32e22ffc009f35 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 01:21:11 -0700 Subject: [PATCH 157/568] feat: reclaim node orphans for cell-group seed retry Require idle-node ownership and the complete current/checkpoint inventory, retain live pins, and join cancelled maintenance. Qualify owned partial/synced dependency crashes, fresh-quorum TLS retry, live suffix catch-up and retained restart while keeping unseeded routing refused. --- crates/cellar-ltx/src/generation.rs | 5 + .../raft/cell_groups/apply/recovery.rs | 65 +-- .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/reclaim.rs | 346 ++++++++++++++ crates/cellar/durability/raft/storage.rs | 85 ++++ .../tests/native_cell_groups_execution.rs | 20 + .../tests/native_cell_groups_snapshot.rs | 2 + .../tests/support/cell_group_crash_fixture.rs | 17 +- .../tests/support/cell_group_node_reclaim.rs | 394 ++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_seed_crash.rs | 22 +- .../tests/support/cell_group_seed_retry.rs | 445 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 6 + docs/design/multi-raft/cell-groups/tasks.json | 5 +- 14 files changed, 1371 insertions(+), 45 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/reclaim.rs create mode 100644 crates/cellar/tests/support/cell_group_node_reclaim.rs create mode 100644 crates/cellar/tests/support/cell_group_seed_retry.rs diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index b418dd784..0c2d2befa 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -629,6 +629,11 @@ impl GenerationStore { self.inner.limits.apply } + /// Compare exact open directory owners without performing I/O. + pub fn same_inventory(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.inner, &other.inner) + } + /// Whether this pin belongs to this exact open directory owner. Logical /// image IDs and digests alone do not establish local file ownership. /// This does not select the image as an authoritative retained reference. diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs index ded56cc04..e03c1e650 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -40,22 +40,30 @@ impl CellGroupApplier { /// owning blocking domain. Never reopen a page directory while its selected /// roots still hold pins. This grants no catalog, quorum or serving authority. pub fn recover_selected(&self) -> Result { + let mut owner = self + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + Ok(self.recover_roots(&mut owner)?.0) + } + + pub(in crate::raft::cell_groups) fn recover_roots( + &self, + owner: &mut Storage, + ) -> Result<(PreparedGroupManifest, Option), ExecutionError> { let marker = self .group .group_manifest() .map_err(storage)? .ok_or(ExecutionError::Invalid("recovery manifest missing"))?; - let prepared = { - let mut owner = self - .storage - .lock() - .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; - let Storage { - tree, - pages, - images, - .. - } = &mut *owner; + let checkpoint = self.group.checkpoint_manifest().map_err(storage)?; + let Storage { + tree, + pages, + images, + .. + } = owner; + let mut prepare = |marker: &GroupManifest| { tree.prepare_complete( &self.group, pages, @@ -68,37 +76,16 @@ impl CellGroupApplier { .map_err(|error| ManifestError::Io(std::io::Error::other(error))) }, ) - .map_err(storage)? + .map_err(storage) }; - if let Some(checkpoint) = self.group.checkpoint_manifest().map_err(storage)? { - let mut owner = self - .storage - .lock() - .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; - let Storage { - tree, - pages, - images, - .. - } = &mut *owner; - tree.prepare_complete( - &self.group, - pages, - checkpoint.root(), - checkpoint.position(), - checkpoint.membership_revision(), - |leaf| { - images - .recover_image(leaf, checkpoint.route().schema_digest) - .map_err(|error| ManifestError::Io(std::io::Error::other(error))) - }, - ) - .map_err(storage)?; - } - if self.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) { + let current = prepare(&marker)?; + let retained = checkpoint.as_ref().map(&mut prepare).transpose()?; + if self.group.group_manifest().map_err(storage)?.as_ref() != Some(&marker) + || self.group.checkpoint_manifest().map_err(storage)? != checkpoint + { return Err(ExecutionError::Invalid("recovery selection changed")); } - Ok(prepared) + Ok((current, retained)) } } diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 3557da189..11ecc1275 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -58,7 +58,9 @@ mod page_wire; mod pins; mod proposal_admission; mod read_authority; +pub(crate) mod reclaim; pub use read_authority::{CellReadAuthority, CellReadConsistency}; +pub use reclaim::{CellGroupReclaimError, CellGroupReclaimLimits, CellGroupReclaimReport}; mod seed_authority; pub use seed_authority::{CellGroupSeedAuthority, CellGroupSeedSource}; mod retention; diff --git a/crates/cellar/durability/raft/cell_groups/reclaim.rs b/crates/cellar/durability/raft/cell_groups/reclaim.rs new file mode 100644 index 000000000..105a889f4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/reclaim.rs @@ -0,0 +1,346 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reconcile all engine-selected closures before bounded orphan collection. +use super::*; +use crate::raft::{cache::GroupLog, execution::ExecutionError, storage::LogError}; +use cellar_logic::cell_group::GroupHistory; +use cellar_ltx::{ + generation::{GenerationError, GenerationStore, ImageId}, + internal::{ExtentRootReference, NodeExtentStore, StoreError, EXTENT_BYTES}, +}; +use std::{ + collections::{BTreeMap, BTreeSet}, + sync::{Arc, TryLockError}, +}; + +/// One maintenance request's aggregate bounds, including duplicate retained cuts. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupReclaimLimits { + pub max_selected_cells: u32, + pub max_files: usize, + pub max_page_scan: usize, +} + +impl Default for CellGroupReclaimLimits { + fn default() -> Self { + Self { + max_selected_cells: 4096, + max_files: 256, + max_page_scan: 256, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupReclaimReport { + pub groups: usize, + pub selected_roots: usize, + pub selected_cells: u32, + pub pages_scanned: usize, + pub page_files: usize, + pub image_files: usize, +} + +#[derive(Debug)] +pub enum CellGroupReclaimError { + Invalid(&'static str), + Limit, + Busy, + Log(LogError), + Recovery(ExecutionError), + Manifest(ManifestError), + Image(GenerationError), + Extent(StoreError), +} + +impl std::fmt::Display for CellGroupReclaimError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell-group storage reclamation: {self:?}") + } +} + +impl std::error::Error for CellGroupReclaimError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Log(e) => Some(e), + Self::Recovery(e) => Some(e), + Self::Manifest(e) => Some(e), + Self::Image(e) => Some(e), + Self::Extent(e) => Some(e), + _ => None, + } + } +} + +type Result = std::result::Result; + +pub(crate) struct ReclaimCells { + appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, +} + +impl ReclaimCells { + pub(crate) fn new( + mut appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, + ) -> Result { + if appliers.is_empty() + || appliers.len() > 4096 + || retained.len() > 4096 + || !(1..=1_000_000).contains(&limits.max_selected_cells) + || !(1..=4096).contains(&limits.max_files) + || !(1..=4096).contains(&limits.max_page_scan) + { + return Err(CellGroupReclaimError::Limit); + } + appliers.sort_by_key(CellGroupApplier::history); + if appliers + .windows(2) + .any(|a| a[0].history() == a[1].history()) + { + return Err(CellGroupReclaimError::Invalid("duplicate group applier")); + } + Ok(Self { + appliers, + retained, + limits, + }) + } + + pub(crate) fn reservation_bytes(&self) -> Result { + // Full recovered trees, reference inventories, one extent verifier and + // bounded group/retained-root owners stay charged through physical work. + ManifestPins::maximum_bytes(self.limits.max_selected_cells) + .checked_mul(2) + .and_then(|n| n.checked_add(self.appliers.len().checked_mul(4096)?)) + .and_then(|n| n.checked_add(self.retained.len().checked_mul(1024)?)) + .and_then(|n| n.checked_add(EXTENT_BYTES as usize + 65536)) + .ok_or(CellGroupReclaimError::Limit) + } + + /// Runs on the engine worker: group creation/publication cannot interleave. + pub(crate) fn run( + self, + engine: &Arc<()>, + groups: Vec>, + ) -> Result { + if groups.len() != self.appliers.len() + || groups.iter().zip(&self.appliers).any(|(g, a)| { + g.identity() != a.group.identity() + || !Arc::ptr_eq(&g.owner, engine) + || !a.belongs_to_engine(engine) + }) + { + return Err(CellGroupReclaimError::Invalid( + "incomplete or foreign engine inventory", + )); + } + let mut owners = self + .appliers + .iter() + .map(|a| { + a.storage.try_lock().map_err(|e| match e { + TryLockError::WouldBlock => CellGroupReclaimError::Busy, + TryLockError::Poisoned(_) => { + CellGroupReclaimError::Invalid("poisoned group storage") + } + }) + }) + .collect::>>()?; + let mut report = CellGroupReclaimReport { + groups: groups.len(), + ..Default::default() + }; + // Refuse the aggregate cell bound before recovery allocates any trees. + for group in &groups { + let current = group + .group_manifest() + .map_err(CellGroupReclaimError::Log)? + .ok_or(CellGroupReclaimError::Invalid("selected manifest missing"))?; + let checkpoint = group + .checkpoint_manifest() + .map_err(CellGroupReclaimError::Log)?; + for marker in std::iter::once(¤t).chain(checkpoint.as_ref()) { + report.selected_cells = report + .selected_cells + .checked_add(marker.root().cells()) + .filter(|n| *n <= self.limits.max_selected_cells) + .ok_or(CellGroupReclaimError::Limit)?; + report.selected_roots += 1; + } + } + for root in &self.retained { + report.selected_cells = report + .selected_cells + .checked_add(root.marker().root().cells()) + .filter(|n| *n <= self.limits.max_selected_cells) + .ok_or(CellGroupReclaimError::Limit)?; + report.selected_roots += 1; + } + let mut roots = Vec::with_capacity(groups.len()); + let mut stores = Vec::::new(); + for (applier, owner) in self.appliers.iter().zip(&mut owners) { + let (current, checkpoint) = applier + .recover_roots(owner) + .map_err(CellGroupReclaimError::Recovery)?; + let mut selected = vec![current]; + selected.extend(checkpoint); + let extras: Vec<_> = self + .retained + .iter() + .filter(|r| r.marker().route().history == applier.history()) + .cloned() + .collect(); + for extra in &extras { + if !Arc::ptr_eq(&extra.engine_owner, engine) + || !Arc::ptr_eq(&extra.page_owner, &owner.pages.owner) + { + return Err(CellGroupReclaimError::Invalid("retained root owner")); + } + } + selected.extend(extras); + let index = match stores.iter().position(|s| s.same(&owner.images)) { + Some(index) => index, + None => { + stores.push(Images::new(&owner.images)); + stores.len() - 1 + } + }; + for root in &selected { + stores[index].retain(root)?; + } + roots.push(selected); + } + if self.retained.iter().any(|r| { + !self + .appliers + .iter() + .any(|a| a.history() == r.marker().route().history) + }) { + return Err(CellGroupReclaimError::Invalid( + "retained root outside engine", + )); + } + // No deletion until every selected closure and owner has been checked. + for store in &stores { + store.reconcile()?; + } + for (applier, owner) in self.appliers.iter().zip(&mut owners) { + let extras: Vec<_> = self + .retained + .iter() + .filter(|r| r.marker().route().history == applier.history()) + .cloned() + .collect(); + owner + .pages + .reconcile_retained(&extras) + .map_err(CellGroupReclaimError::Manifest)?; + } + let mut files = self.limits.max_files; + let mut scan = self.limits.max_page_scan; + for owner in &mut owners { + if scan == 0 || files == 0 { + break; + } + let pages = owner + .pages + .reclaim(scan.min(files)) + .map_err(CellGroupReclaimError::Manifest)?; + scan -= pages.scanned; + files -= pages.pages; + report.pages_scanned += pages.scanned; + report.page_files += pages.pages; + } + for store in &stores { + if files == 0 { + break; + } + let removed = store.reclaim(files)?; + files -= removed; + report.image_files += removed; + } + // Retain every recovered tree until all file operations finish. + drop(roots); + Ok(report) + } +} + +enum Images { + Materialized(GenerationStore, BTreeSet), + Extents(NodeExtentStore, BTreeMap<[u8; 32], ExtentRootReference>), +} + +impl Images { + fn new(images: &CellGroupApplyImages) -> Self { + match images { + CellGroupApplyImages::Materialized { images, .. } => { + Self::Materialized(images.clone(), BTreeSet::new()) + } + CellGroupApplyImages::Extents { images, .. } => { + Self::Extents(images.clone(), BTreeMap::new()) + } + } + } + + fn same(&self, images: &CellGroupApplyImages) -> bool { + match (self, images) { + (Self::Materialized(a, _), CellGroupApplyImages::Materialized { images: b, .. }) => { + a.same_inventory(b) + } + (Self::Extents(a, _), CellGroupApplyImages::Extents { images: b, .. }) => { + a.same_inventory(b) + } + _ => false, + } + } + + fn retain(&mut self, root: &PreparedGroupManifest) -> Result<()> { + root.images.visit(&mut |_, _, pin, _, _| { + match (&mut *self, pin) { + (_, CellImagePin::Empty) => (), + (Self::Materialized(store, refs), CellImagePin::Materialized(pin)) + if store.owns(pin) => + { + refs.insert(pin.id()); + } + (Self::Extents(store, refs), CellImagePin::Extents(pin)) + if store.owns_image(pin) => + { + let reference = pin.reference(); + if refs + .insert(reference.root().bytes(), reference) + .is_some_and(|old| old != reference) + { + return Err(CellGroupReclaimError::Invalid("conflicting extent root")); + } + } + _ => return Err(CellGroupReclaimError::Invalid("image inventory owner")), + } + Ok(()) + }) + } + + fn reconcile(&self) -> Result<()> { + match self { + Self::Materialized(store, refs) => store + .retain_only(&refs.iter().copied().collect::>()) + .map_err(CellGroupReclaimError::Image), + Self::Extents(store, refs) => store + .retain_only(&refs.values().copied().collect::>()) + .map_err(CellGroupReclaimError::Extent), + } + } + + fn reclaim(&self, files: usize) -> Result { + match self { + Self::Materialized(store, _) => store + .reclaim(files) + .map(|r| r.files) + .map_err(CellGroupReclaimError::Image), + Self::Extents(store, _) => store.reclaim(files).map_err(CellGroupReclaimError::Extent), + } + } +} diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 937738470..93d6ed6bf 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -5,6 +5,10 @@ use super::cache::{ genesis, validate_conf, CachedGroup, GroupLog, GroupWrite, SnapshotChange, StoredGroup, }; +use super::cell_groups::{ + reclaim::ReclaimCells, CellGroupApplier, CellGroupReclaimError, CellGroupReclaimLimits, + CellGroupReclaimReport, PreparedGroupManifest, +}; use super::filesystem::EngineFileSystem; use super::fleet_stop::{Guard, LifecycleRecord, LifecycleUpdate}; use super::options::{LogOpenMode, ReplicaLogOptions}; @@ -257,6 +261,11 @@ enum Work { Vec>, Reply>, ), + ReclaimCells( + Box, + MaintenanceLease, + Reply>, + ), Purge(Reply<()>), } @@ -266,6 +275,21 @@ pub(crate) struct RuntimeLease(Arc); /// Held by the SQL owner, physical workers and retained response admission. pub(crate) struct SqlPoolLease(Arc); +// Maintenance performs no consensus writes of its own. It may refund an +// unqueued reservation, but an entered operation retains both claims to exit. +struct MaintenanceLease { + runtime: Option, + _sql: Arc, +} + +impl Drop for MaintenanceLease { + fn drop(&mut self) { + if let Some(runtime) = self.runtime.take() { + runtime.release(); + } + } +} + impl Drop for SqlPoolLease { fn drop(&mut self) { self.0.sql_claimed.store(false, Ordering::Release); @@ -547,6 +571,50 @@ impl ReplicaLog { .collect() } + /// Reconcile every physical group in this engine, then collect a bounded + /// batch of unselected image/page files. Supply one exact applier per group, + /// including unattached/retired groups, and every additional durable archive + /// or transition root. Image stores must belong exclusively to this node + /// engine. Live reader/transfer pins remain protected independently. + /// + /// Runtime and SQL ownership must have joined shutdown before maintenance; + /// the engine reserves both claims until verification/deletion finishes. + /// A busy applier refuses instead of waiting. Cancellation retains work and admission; + /// log.close() joins physical completion. This supplies no serving, archive + /// coverage, log truncation or voting authority. + pub async fn reclaim_cell_storage( + &self, + appliers: Vec, + retained: Vec, + limits: CellGroupReclaimLimits, + ) -> Result { + let work = ReclaimCells::new(appliers, retained, limits)?; + let bytes = work.reservation_bytes()?; + let runtime = self.claim_runtime().map_err(|e| match e { + LogError::AlreadyExists => CellGroupReclaimError::Busy, + e => CellGroupReclaimError::Log(e), + })?; + let sql = match self.claim_sql_pool() { + Ok(sql) => sql, + Err(error) => { + runtime.release(); + return Err(match error { + LogError::AlreadyExists => CellGroupReclaimError::Busy, + e => CellGroupReclaimError::Log(e), + }); + } + }; + let lease = MaintenanceLease { + runtime: Some(runtime), + _sql: sql, + }; + self.request(bytes, |reply| { + Work::ReclaimCells(Box::new(work), lease, reply) + }) + .await + .map_err(CellGroupReclaimError::Log)? + } + pub fn cell_group(&self, history: &GroupHistory) -> Result, LogError> { self.admit()?; self.inner @@ -1618,6 +1686,23 @@ impl Worker { let _ = reply.send(result); fatal } + Work::ReclaimCells(work, lease, reply) => { + let result = (|| { + let groups = self + .groups + .read() + .map_err(|_| LogError::Closed)? + .values() + .map(|g| g.as_cells().cloned()) + .collect::, _>>()?; + Ok(work.run(&self.owner, groups)) + })(); + // Physical stores quarantine their own uncertain deletes; + // a refused maintenance request cannot relabel log state. + drop(lease); + let _ = reply.send(result); + None + } Work::Seeded(seed, reply) => { let result = self.seeded(*seed); let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 0a8a48544..20677c97f 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,26 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_retry_materialized_reclaims_interrupted_disk() -> Result<()> { + Box::pin(runtime_fixture::seed_retry(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn seed_retry_extents_reclaim_interrupted_disk() -> Result<()> { + Box::pin(runtime_fixture::seed_retry(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_reclaim_materialized_preserves_complete_shared_inventory() -> Result<()> { + Box::pin(runtime_fixture::shared_inventory(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn node_reclaim_extents_preserve_complete_shared_inventory() -> Result<()> { + Box::pin(runtime_fixture::shared_inventory(true)).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_transfer_materialized_revocation_retains_physical_owner() -> Result<()> { Box::pin(runtime_fixture::seed_transfer_revocation(false)).await diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 28bd62285..7bb72c0d0 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -51,6 +51,7 @@ async fn partial_snapshot_never_serves() -> Result<()> { // These parents launch this same executable's ignored child and verify // actual retained disks after owned SIGKILL at all six install cuts. Box::pin(runtime_fixture::seed_crash(rooted)).await?; + fixture(|| runtime_fixture::seed_retry(rooted)).await?; } Ok(()) } @@ -85,6 +86,7 @@ async fn snapshot_empty_cell_and_extent_restart() -> Result<()> { async fn snapshot_memory_and_pin_bounds() -> Result<()> { for rooted in [false, true] { runtime_fixture::transfer_pressure(rooted).await?; + fixture(|| runtime_fixture::shared_inventory(rooted)).await?; } checkpoint_fixture::ownership().await?; runtime_fixture::local_shutdown().await?; diff --git a/crates/cellar/tests/support/cell_group_crash_fixture.rs b/crates/cellar/tests/support/cell_group_crash_fixture.rs index d91f7b821..f4eeed434 100644 --- a/crates/cellar/tests/support/cell_group_crash_fixture.rs +++ b/crates/cellar/tests/support/cell_group_crash_fixture.rs @@ -122,11 +122,24 @@ pub(super) async fn restore_catalog( route: &GroupRoute, domain: &ExecutionDomain, ) -> Result<(RangeDirectory, CellGroupBirth)> { + let (directory, birth, _) = restore_catalog_store(objects, route, domain).await?; + Ok((directory, birth)) +} + +pub(super) async fn restore_catalog_store( + objects: Vec<(String, Vec)>, + route: &GroupRoute, + domain: &ExecutionDomain, +) -> Result<( + RangeDirectory, + CellGroupBirth, + Arc, +)> { let store = Arc::new(object_store::memory::InMemory::new()); for (name, bytes) in objects { store.put(&ObjectPath::from(name), bytes.into()).await?; } - let catalog = Catalog::new(store, CatalogOptions::default())?; + let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; let namespace = domain .run(catalog.open_namespace(NamespaceConfig::new( route.history.cluster, @@ -139,5 +152,5 @@ pub(super) async fn restore_catalog( let birth = domain .run(directory.materialize_selected_group(route)) .await??; - Ok((directory, birth)) + Ok((directory, birth, store)) } diff --git a/crates/cellar/tests/support/cell_group_node_reclaim.rs b/crates/cellar/tests/support/cell_group_node_reclaim.rs new file mode 100644 index 000000000..d7ccf9964 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_node_reclaim.rs @@ -0,0 +1,394 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete engine inventory over two groups sharing the same physical images. +use super::*; +use cellar_ltx::internal::{ExtentId, ExtentManifest, Geometry, ImageBinding, EXTENT_BYTES}; +use std::path::PathBuf; + +pub(super) fn limits() -> CellGroupReclaimLimits { + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + } +} + +fn appliers(f: &RuntimeCluster) -> Vec { + f.appliers + .iter() + .filter(|((node, _), _)| *node == 0) + .map(|(_, (_, a))| a.clone()) + .collect() +} + +async fn images_for( + f: &RuntimeCluster, + cell: &CellHistory, + node: usize, +) -> Result { + Ok(match f.cell_options_at(cell, node).await? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: Default::default(), + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }) +} + +async fn enroll(f: &RuntimeCluster, group: GroupHistory, name: &str, rooted: bool) -> Result<()> { + let selected = f.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("group")?; + let cell = history(selected.route(), name)?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(113);", + )?; + let accepted = f.runtimes[0] + .propose( + group, + apply::enrollment_bytes(selected.route(), &cell, &bytes, rooted)?, + ) + .await?; + controller::applied(f, group, accepted.position.index).await +} + +fn physical_image( + images: &CellGroupApplyImages, + base: PathBuf, + cell: CellHistory, + number: u8, +) -> Result<(CellImagePin, PathBuf)> { + let bytes = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({number});"), + )?; + let state = images::Cut::new(&[], &bytes, 0, true, false).result; + match images { + CellGroupApplyImages::Materialized { images, .. } => { + let source = base.join(format!("private-{number}.sqlite")); + std::fs::write(&source, &bytes)?; + let id = ImageId::new([number; 16])?; + let pin = images.prepare_image_file(id, &source, state)?; + images.retire(id)?; + Ok(( + CellImagePin::Materialized(pin), + base.join("node-1/images") + .join(format!("{}.sqlite", hex::encode(id.bytes()))), + )) + } + CellGroupApplyImages::Extents { images, .. } => { + let ids = bytes + .chunks(EXTENT_BYTES as usize) + .map(ExtentId::of) + .collect::, _>>()?; + let manifest = ExtentManifest::new( + ImageBinding { + history: cell_image_history_digest(&cell), + schema: [81; 32], + epoch: state.epoch, + txid: state.txid.0, + checksum: state.checksum, + }, + Geometry::new(state.page_size, state.length)?, + &ids, + )?; + let view = images.stage_transient_root(manifest.clone())?; + for (id, chunk) in ids.iter().zip(bytes.chunks(EXTENT_BYTES as usize)) { + images.put_extent_bytes(*id, chunk)?; + } + let pin = images.seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize])?; + drop(view); + let id = ids[0]; + Ok(( + CellImagePin::Extents(Arc::new(pin)), + base.join("rooted-0").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )), + )) + } + } +} + +fn read(pin: &CellImagePin) -> Result<[u8; 32]> { + let mut hash = Sha256::new(); + match pin { + CellImagePin::Materialized(pin) => { + let mut r = pin.open_reader()?; + let mut offset = 0; + while offset < r.length() { + let count = (r.length() - offset).min(65536) as usize; + hash.update(r.read_exact_at(offset, count)?); + offset += count as u64; + } + } + CellImagePin::Extents(pin) => { + for n in 0..pin.manifest().geometry().extent_count() { + hash.update( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + } + CellImagePin::Empty => (), + } + Ok(hash.finalize().into()) +} + +pub(crate) async fn shared_inventory(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = f.add_group(31).await?; + let selected_images = images_for(&f, &a, 0).await?; + for node in 0..3 { + let images = images_for(&f, &a, node).await?; + let pool = f.appliers[&(node, route.history)].0.clone(); + let voter = &mut f.extra.get_mut(&second).context("second")?[node]; + let applier = CellGroupApplier::new( + voter.log.cell_group(&second)?, + voter.pages.take().context("page store")?, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + f.appliers.insert((node, second), (pool, applier)); + } + let mut controllers = controller::start_with_limits(&mut f, controller::limits()).await?; + f.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(second).await?.ready)).await?; + enroll(&f, second, "other-group", rooted).await?; + let mut selected = Vec::new(); + for group in [route.history, second] { + let applier = f.appliers[&(0, group)].1.clone(); + let log = f.cluster.voters[0].log.cell_group(&group)?; + let (root, checkpoint) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let root = applier.recover_selected()?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = pool.capture(&log, &root)?.prepare_publication()?; + Ok((root, checkpoint)) + })? + .await??; + let marker = root.marker().clone(); + f.runtimes[0].publish_checkpoint(group, checkpoint).await?; + controller::wait_for(async || { + Ok(f.cluster.voters[0] + .log + .cell_group(&group)? + .checkpoint_manifest()? + .as_ref() + == Some(&marker)) + }) + .await?; + selected.push(root.marker().clone()); + drop(root); + } + enroll(&f, route.history, "after-checkpoint", rooted).await?; + enroll(&f, second, "other-after-checkpoint", rooted).await?; + let images = selected_images.clone(); + let base = f.cluster._root.path().to_path_buf(); + let cell = history(&route, "private-reader")?; + let (reader, reader_path, orphan_path) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let (reader, path) = physical_image(&images, base.clone(), cell.clone(), 211)?; + let (orphan, orphan_path) = physical_image(&images, base, cell, 212)?; + drop(orphan); + Ok((reader, path, orphan_path)) + })? + .await??; + let expected = read(&reader)?; + let selected_appliers = appliers(&f); + let log = &f.cluster.voters[0].log; + assert!(matches!( + log.reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await, + Err(CellGroupReclaimError::Busy) + )); + controllers.remove(0).close().await?; + f.appliers.retain(|(node, _), _| *node != 0); + assert!(matches!( + log.reclaim_cell_storage(vec![selected_appliers.clone()[0].clone()], vec![], limits()) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + assert!(orphan_path.exists()); + assert!(matches!( + log.reclaim_cell_storage( + vec![selected_appliers.clone()[0].clone(); 2], + vec![], + limits() + ) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + let foreign = f.appliers[&(1, route.history)].1.clone(); + assert!(matches!( + log.reclaim_cell_storage( + vec![foreign, selected_appliers.clone()[1].clone()], + vec![], + limits() + ) + .await, + Err(CellGroupReclaimError::Invalid(_)) + )); + let mut small = limits(); + small.max_selected_cells = 1; + assert!(matches!( + log.reclaim_cell_storage(selected_appliers.clone(), vec![], small) + .await, + Err(CellGroupReclaimError::Limit) + )); + assert!(orphan_path.exists()); + let mut deleted = 0; + for _ in 0..32 { + let report = log + .reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await?; + assert_eq!(report.groups, 2); + assert_eq!(report.selected_roots, 4); + assert!(report.page_files + report.image_files <= 1); + assert!(report.pages_scanned <= 64); + deleted += report.image_files; + assert_eq!(read(&reader)?, expected); + if !orphan_path.exists() { + break; + } + } + assert!(deleted > 0 && !orphan_path.exists()); + assert!(reader_path.exists()); + for marker in &selected { + let applier = selected_appliers + .iter() + .find(|a| a.history() == marker.route().history) + .context("group applier")? + .clone(); + let current = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + for name in if marker.route().history == route.history { + vec!["registered-a", "registered-b"] + } else { + vec!["other-group"] + } { + let (_, pin) = current + .pin_cell(&CellId::new(name)?) + .context("retained cell")?; + read(&pin)?; + } + } + drop(reader); + for _ in 0..32 { + log.reclaim_cell_storage(selected_appliers.clone(), vec![], limits()) + .await?; + if !reader_path.exists() { + break; + } + } + assert!(!reader_path.exists()); + for (index, group) in [route.history, second].into_iter().enumerate() { + assert_eq!( + log.cell_group(&group)?.checkpoint_manifest()?.as_ref(), + Some(&selected[index]) + ); + let applier = selected_appliers + .iter() + .find(|a| a.history() == group) + .context("group applier")? + .clone(); + let current = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert!(current.marker().position().index > selected[index].position().index); + } + // Queue maintenance behind a real engine fence, then lose its waiter. + // Neither cancellation nor a cancelled close may refund physical ownership. + let image_store = selected_images.clone(); + let base = f.cluster._root.path().to_path_buf(); + let cell = history(&route, "cancelled-orphan")?; + let orphan = f + .cluster + .domain + .blocking(move || -> Result<_> { + let (pin, path) = physical_image(&image_store, base, cell, 213)?; + drop(pin); + Ok(path) + })? + .await??; + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let writer = (*log).clone(); + let ahead = f + .cluster + .domain + .spawn(async move { writer.reserve_node_revision().await })?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let writer = (*log).clone(); + let owned = selected_appliers.clone(); + let pending = f.cluster.domain.spawn(async move { + writer + .reclaim_cell_storage( + owned, + vec![], + CellGroupReclaimLimits { + max_files: 4096, + max_page_scan: 4096, + ..limits() + }, + ) + .await + })?; + controller::wait_for(async || { + let sample = log.performance_snapshot(); + assert!(sample.available && !sample.saturated); + let queue = sample + .operations + .iter() + .find(|o| o.name == "engine_queue") + .context("engine queue")?; + Ok(queue.started - queue.succeeded - queue.failed - queue.abandoned == 1) + }) + .await?; + pending.abort(); + assert!(pending.await.is_err()); + assert!(matches!( + CellGroupRuntime::start((*log).clone(), RaftRuntimeOptions::default()).await, + Err(RuntimeError::Storage(LogError::AlreadyExists)) + )); + assert!(apply::pool(&f, 0).is_err()); + assert!(tokio::time::timeout(Duration::from_millis(30), log.close()) + .await + .is_err()); + assert!(orphan.exists()); + drop(release); + ahead.await??; + tokio::time::timeout(Duration::from_secs(5), log.close()).await??; + assert!( + !orphan.exists(), + "queued collection must finish after waiter cancellation" + ); + for c in controllers { + c.close().await?; + } + drop((selected, selected_appliers)); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index c5805ab1b..dbba7fd3b 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -829,3 +829,5 @@ pub(super) use sql::{seed_persistence, seed_transfer_revocation}; pub(super) use sql::{seed_crash, seed_process_fixture}; pub(super) use sql::{membership_crash, membership_process_fixture}; + +pub(super) use sql::{seed_retry, shared_inventory}; diff --git a/crates/cellar/tests/support/cell_group_seed_crash.rs b/crates/cellar/tests/support/cell_group_seed_crash.rs index c9dd94608..f955e57c6 100644 --- a/crates/cellar/tests/support/cell_group_seed_crash.rs +++ b/crates/cellar/tests/support/cell_group_seed_crash.rs @@ -14,6 +14,7 @@ use std::path::Path; #[serde(deny_unknown_fields)] struct Oracle { rooted: bool, + source_root: std::path::PathBuf, cut: String, staged: GroupManifest, expected: GroupManifest, @@ -44,7 +45,14 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { anyhow::ensure!( matches!( cut.as_str(), - "staged" | "image-prefix" | "image-synced" | "before-sync" | "after-sync" | "published" + "staged" + | "image-prefix" + | "image-synced" + | "retry-prefix" + | "retry-synced" + | "before-sync" + | "after-sync" + | "published" ), "invalid crash cut" ); @@ -142,6 +150,7 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { &root.join("oracle.json"), &serde_json::to_vec(&Oracle { rooted, + source_root: fixture.cluster._root.path().to_path_buf(), cut: cut.clone(), staged, expected: expected.clone(), @@ -180,8 +189,11 @@ pub(crate) async fn seed_process_fixture() -> Result<()> { "after-sync" => Some(gate.arm(fault::CRASH_AFTER_SYNC)), _ => None, }; - if matches!(cut.as_str(), "image-prefix" | "image-synced") { - let _release_image = image_gate.arm(if cut == "image-prefix" { + if matches!( + cut.as_str(), + "image-prefix" | "image-synced" | "retry-prefix" | "retry-synced" + ) { + let _release_image = image_gate.arm(if cut.ends_with("prefix") { fault::HOLD_DEPENDENCY_PREFIX } else { fault::HOLD_DEPENDENCY_SYNC @@ -351,3 +363,7 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { domain.shutdown().await; Ok(()) } + +#[path = "cell_group_seed_retry.rs"] +mod retry; +pub(crate) use retry::seed_retry; diff --git a/crates/cellar/tests/support/cell_group_seed_retry.rs b/crates/cellar/tests/support/cell_group_seed_retry.rs new file mode 100644 index 000000000..d2a738525 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_seed_retry.rs @@ -0,0 +1,445 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Resume the original voter disks and retry an owned, killed learner's disk. +use super::*; +use cellar_ltx::internal::NodeExtentStore; +use std::collections::VecDeque; + +pub(crate) async fn seed_retry(rooted: bool) -> Result<()> { + for cut in ["retry-prefix", "retry-synced"] { + let directory = tempfile::tempdir()?; + crash_fixture::kill_at(directory.path(), "seed_process_fixture", rooted, cut).await?; + Box::pin(retry(directory.path(), rooted, cut)) + .await + .with_context(|| format!("cold retry {rooted}/{cut}"))?; + } + Ok(()) +} + +async fn restore_source( + root: &Path, + oracle: &Oracle, + domain: ExecutionDomain, + directory: RangeDirectory, + birth: CellGroupBirth, + store: Arc, +) -> Result { + anyhow::ensure!(oracle.source_root.starts_with(root), "owned source disks"); + let mut voters = Vec::new(); + let mut appliers = Vec::new(); + let options = RaftRuntimeOptions::default(); + for n in 0..3 { + let base = oracle.source_root.join(format!("node-{}", n + 1)); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n as u8 + 1; 16])?, + DiskLineage::new([n as u8 + 11; 16])?, + ProcessGeneration::new([95; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let extents = if oracle.rooted { + Some(NodeExtentStore::open( + oracle.source_root.join(format!("rooted-{n}")), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let config = + super::super::super::checkpoint_receive::config(images.clone(), extents.clone(), root)?; + let group = log.cell_group(&oracle.expected.route().history)?; + let (applier, prepared) = domain + .blocking(move || { + CellGroupApplier::recover( + group, + pages, + config, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + prepared, + options.clone(), + [71; 32], + )) + .await??; + appliers.push(applier); + voters.push(Voter { + log, + owner: Some(owner), + pages: None, + images, + extents, + tree: ManifestTree::new(birth.route().clone(), ManifestLimits::default())?, + gate: fault::Gate::new(None), + paused: false, + held: None, + }); + } + let cluster = Cluster { + _root: tempfile::tempdir_in(root)?, + domain, + directory, + store, + birth, + voters, + network: VecDeque::new(), + reads: vec![], + options, + }; + let mut f = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + for (n, applier) in appliers.into_iter().enumerate() { + let pool = apply::pool(&f, n)?; + f.appliers.insert((n, applier.history()), (pool, applier)); + } + f.connect_tcp().await.context("connect cold source peers")?; + Ok(f) +} + +async fn retry(root: &Path, rooted: bool, cut: &str) -> Result<()> { + let oracle: Oracle = serde_json::from_slice(&std::fs::read(root.join("oracle.json"))?)?; + assert_eq!(oracle.rooted, rooted); + assert_eq!(oracle.cut, cut); + let (partial, length): (std::path::PathBuf, u64) = + serde_json::from_slice(&std::fs::read(root.join("dependency.json"))?)?; + assert!(length > 0); + if rooted { + assert_eq!( + partial.parent(), + Some(root.join("learner/extents").as_path()) + ); + assert_eq!(std::fs::metadata(&partial)?.len(), length); + let text = partial + .file_name() + .context("extent")? + .to_str() + .context("name")?; + let full = u64::from_str_radix(&text[2..10], 16)?; + if cut == "retry-prefix" { + assert!(length < full); + } else { + assert_eq!(length, full); + } + } + let route = oracle.expected.route().clone(); + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let (directory, birth, store) = + crash_fixture::restore_catalog_store(oracle.objects.clone(), &route, &domain).await?; + let mut f = Box::pin(restore_source( + root, + &oracle, + domain, + directory.clone(), + birth.clone(), + store, + )) + .await + .context("restore cold source quorum")?; + let controllers = controller::start_with_limits(&mut f, controller::limits()) + .await + .context("start cold source controllers")?; + f.runtimes[0] + .campaign(route.history) + .await + .context("campaign cold source")?; + controller::wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + assert!(f.runtimes[0].status(route.history).await?.term > oracle.expected.position().term); + let later = history(&route, "after-crash-before-retry")?; + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(317);", + )?; + let accepted = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &later, &bytes, rooted)?, + ) + .await?; + controller::applied(&f, route.history, accepted.position.index).await?; + let expected = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("fresh source cut")?; + assert_eq!(expected.root().cells(), 3); + let base = root.join("learner"); + let mut local = seed_install::target(route.history)?; + local.process = ProcessGeneration::new([94; 16])?; + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + local.process, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let engine = log.storage_instance()?; + let group = log.cell_group(&route.history)?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.staged)); + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, empty) = f + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(empty.pinned_images(), 0); + drop(empty); + assert!(f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await? + .is_err()); + let mut deleted = 0; + for _ in 0..16 { + let report = log + .reclaim_cell_storage( + vec![applier.clone()], + vec![], + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + }, + ) + .await?; + assert!(report.image_files + report.page_files <= 1); + deleted += report.image_files; + if report.image_files + report.page_files == 0 && (!rooted || !partial.exists()) { + break; + } + } + if rooted { + assert!(deleted > 0); + assert!(!partial.exists()); + } + assert_eq!(group.group_manifest()?.as_ref(), Some(&oracle.staged)); + assert!(f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await? + .is_err()); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + let transport = tcp::start(&runtime, 3).await?; + transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(checkpoint_fixture::limits())?)?; + let source_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + f.transports[0].enable_group_checkpoints(source_pool)?; + let mut physical = f.tcp_route(route.history)?; + physical.revision += 1; + physical.peers.push(RaftPeer { + replica: local.clone(), + label: "node-4".into(), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }); + for source in &f.transports { + source + .install_physical_group(physical.clone()) + .context("install fresh learner route at source")?; + } + assert!( + matches!( + transport.install_physical_group(physical.clone()), + Err(TransportError::StalePeer) + ), + "an empty stage cannot authorize ordinary group routing" + ); + let _registration = + f.transports[0].register_group_checkpoint(f.appliers[&(0, route.history)].1.clone())?; + + let session = Box::pin(transport.prepare_group_seed( + local.clone(), + physical.peers[0].clone(), + directory.clone(), + birth.clone(), + reads::deadline(&f, 10_000).await?, + )) + .await + .context("prepare fresh authenticated seed")?; + let (permit, installed) = session + .install(applier.clone()) + .await + .context("install retried seed")?; + assert_eq!(installed.marker(), &expected); + transport + .install_physical_group(physical.clone()) + .context("route authenticated installed learner")?; + for name in ["registered-a", "registered-b", "after-crash-before-retry"] { + assert!(installed.pin_cell(&CellId::new(name)?).is_some()); + } + assert_eq!(log.storage_instance()?, engine); + runtime.attach(permit, installed.clone()).await?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 1, + max_pending_cells: 1, + group_pending_cells: 1, + node_bytes: 1024 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let learner = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + vec![applier.clone()], + controller::limits(), + ) + .await?; + let live = history(&route, "live-after-retry")?; + let written = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &live, &bytes, rooted)?, + ) + .await?; + controller::applied(&f, route.history, written.position.index).await?; + controller::wait_for(async || { + f.runtimes[0].tick().await?; + Ok(runtime.status(route.history).await?.applied.index >= written.position.index) + }) + .await?; + let expected = f.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("live suffix source")?; + assert_eq!(expected.root().cells(), 4); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert_eq!( + group.checkpoint_manifest()?.as_ref(), + Some(installed.marker()) + ); + learner.close().await?; + drop((learner, sql, installed, transport, runtime, applier, group)); + log.close().await?; + drop(log); + // Reopen the same successful retry independently of every previous owner. + let log = f + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + base.join("log"), + local.node, + local.disk, + ProcessGeneration::new([96; 16])?, + LogOpenMode::Resume, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert_eq!(log.storage_instance()?, engine); + let group = log.cell_group(&route.history)?; + let (pages, images) = seed_install::stores(&base, rooted)?; + let recovering = group.clone(); + let (applier, recovered) = f + .cluster + .domain + .blocking(move || { + CellGroupApplier::recover( + recovering, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + ) + })? + .await??; + assert_eq!(recovered.marker(), &expected); + assert_eq!(recovered.pinned_images(), 4); + for name in [ + "registered-a", + "registered-b", + "after-crash-before-retry", + "live-after-retry", + ] { + let (leaf, _) = recovered + .pin_cell(&CellId::new(name)?) + .context("cold recovered cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + let permit = f + .cluster + .domain + .run(directory.bootstrap_joined_group_member(&birth, local.member, &log)) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, recovered.clone()).await?; + assert!( + !runtime.status(route.history).await?.ready, + "local seed receipt is not fresh quorum authority" + ); + runtime.close().await?; + drop((runtime, events, applier, recovered, group)); + log.close().await?; + for c in controllers { + c.close().await?; + } + drop(_registration); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 6bc917280..30e03eacf 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -625,3 +625,9 @@ mod crash_fixture; #[path = "cell_group_membership_crash.rs"] mod membership_crash; pub(crate) use membership_crash::{membership_crash, membership_process_fixture}; + +#[path = "cell_group_node_reclaim.rs"] +mod node_reclaim; +pub(crate) use node_reclaim::shared_inventory; + +pub(crate) use seed_crash::seed_retry; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 4f591d54a..ee5b75dc7 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1602,7 +1602,10 @@ "crates/cellar/tests/support/cell_group_crash_fixture.rs", "crates/cellar/tests/support/cell_group_membership_crash.rs", "crates/cellar/tests/support/cell_group_checkpoint_pressure.rs", - "crates/cellar/tests/support/cell_group_engine_fixture.rs" + "crates/cellar/tests/support/cell_group_engine_fixture.rs", + "crates/cellar/durability/raft/cell_groups/reclaim.rs", + "crates/cellar/tests/support/cell_group_node_reclaim.rs", + "crates/cellar/tests/support/cell_group_seed_retry.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 76dcfc26951f817a10586df2e78e3319a0a148f9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 01:44:42 -0700 Subject: [PATCH 158/568] docs: qualify cell-group orphan retry --- docs/design/multi-raft/cell-groups/README.md | 7 +- docs/design/multi-raft/cell-groups/handoff.md | 79 +++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 62 +++++++++++++++ .../multi-raft/cell-groups/protocols.md | 20 +++++ 4 files changed, 132 insertions(+), 36 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 1206b66c5..a45760eaf 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -23,8 +23,11 @@ Bounded 32-cell transfer, scratch contention and old-reader image retention/rele now pass in both image forms. Partial/synced seed-dependency SIGKILL cuts and transfer-time source-process revocation now pass. All five exact native acceptance cases and their prescribed command gate now pass. Controller publication through cancelled shutdown now passes in both image forms. -Remaining CG-05 recovery availability and broader pressure qualification precedes -verified S3 archive/reclamation. The production host has not yet cut over to shared groups. +Idle-node complete-inventory orphan collection and fresh authenticated retry on the +same interrupted learner disk now pass in both forms, including live suffix catch-up +and cold restart. Remaining capture fairness and controller/budget/root-pressure +qualification precedes verified S3 archive/reclamation. The production host has +not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 4be5bf80e..cf2c1eb77 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -36,36 +36,46 @@ partial gate. Durable whole-group membership and leadership handoff now pass an additional partial gate. Authenticated fresh-learner installation, catch-up, promotion and deleted-voter-disk replacement now pass a native partial gate. All five CG-05 acceptance cases now pass; complete the remaining fault/pressure -and retry-availability obligations before closing the task. +and capture-fairness obligations before closing the task. CG-06 follows with archival reclamation. Automatic production request/voter startup and public SDK/fleet binding belong to CG-07; effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest controller-publication checkpoint at `370dcf34` passes 146 -execution cases, nineteen Raft-storage cases and the updated exact resource-bound -snapshot case, with ten scoped commands. Its record under -`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/` binds -332 source hashes and three executables; independent hashes pass. The other -four exact snapshot cases and all-target checks remain at their earlier source. -CG-05 remains IN_PROGRESS and production runtime code is unchanged. - -Both image forms now hold the actual publication sync after lifecycle/source -admission, fill checkpoint capacity and cancel both close and wait. Slot, source, -bytes and engine/SQL ownership remain retained until physical release. The healthy -two-voter quorum commits four writes to the other group during the stall. Release -publishes the preceding checkpoint, leaves the successor unapplied, drains -admission and permits a replacement runtime. Existing fault modes retain their -default behavior; the predicate is opt-in and released by its guard. - -The next concrete implementation is interrupted-seed orphan reconciliation and -fresh authenticated retry. Extent startup inventories partial files without -trusting them; immutable import refuses a partial existing object. Supply complete -node references before collecting unselected orphans, preserve selected current -and checkpoint roots plus live pins, and verify retry on the same disk. A single -group must not reconcile or delete another group's shared objects. Continue the -remaining controller crash/budget/fairness and retained-root pressure qualification. +The latest checkpoint at `5cfa9fb6` adds production idle-node orphan +maintenance and authenticated retry on the same interrupted learner disk. All five +exact snapshot cases, 150 execution cases, nineteen Raft-storage cases and 38 +relevant LTX cases pass. All nineteen commands pass, including the complete CG-05 +prescribed gate and all-target Clippy for the affected crates. Its record under +`.cellar/cell-groups-evidence/CG-05-node-orphan-retry/` binds 335 source hashes, +six executables and frozen inputs; independent hashes and `check_evidence` pass. +CG-05 remains IN_PROGRESS pending broader lifecycle/pressure qualification. + +`ReplicaLog::reclaim_cell_storage` requires joined runtime/SQL ownership and one +exact applier per physical engine group, plus every additional durable retained +root. It verifies current/checkpoint closures before bounded deletion, merges +references across shared image stores, preserves live pins and holds both engine +claims through cancelled work. Missing/duplicate/foreign/oversized/busy inventories +refuse. Managed orchestration must supply the complete inventory and call this +idle-node boundary; it is not a serving-node collector or Raft log truncation API. + +Four new SIGKILL scenarios interrupt prefix/synced dependencies in both forms. +Original voter disks resume, establish fresh quorum and commit a newer cell; +the same learner disk collects unselected debris and receives a fresh TLS seed. +Ordinary routing refuses until the authenticated install receipt exists. The +learner catches a live suffix; another cold reopen verifies all four independent +cell sequences and current/checkpoint closures. Original voter disks remain intact. +Shared image tests preserve both groups and an independent reader, collect one +file per request and join cancellation. Continue capture fairness, remaining +controller crash/completion-budget and retained-root pressure qualification. + +The preceding controller-publication checkpoint at `370dcf34` retains the actual +held publication-sync and cancelled-close cuts in both forms. The healthy two-voter +quorum commits another group's writes while the stalled node retains its lifecycle +slot, captured source, bytes and engine/SQL ownership. Release selects the preceding +checkpoint and joins shutdown. Its source-bound record remains under +`.cellar/cell-groups-evidence/CG-05-controller-checkpoint-shutdown-final/`. The earlier exact-acceptance checkpoint at `f076731a` delivers `native_cell_groups_snapshot` with all five required native oracles. Each passes @@ -85,11 +95,10 @@ composite membership test constructs futures outside its poll frame after phase probes reproduced a debug stack overflow. The standalone case passed and the fixed complete composite passes; no stack limits, timeouts or assertions changed. -The pending-publication cancelled-shutdown cut now passes above. Continue -remaining controller lifecycle/saturation cuts and staged orphan reconciliation -before authenticated retry. Preserve fresh quorum, -physical-owner retention and selected-root checks. Production runtime code was -unchanged in this checkpoint; RustFS/load, independent-domain and public host +The pending-publication cancelled-shutdown and idle-node orphan/retry cuts +now pass above. Continue remaining controller lifecycle/saturation and capture +fairness qualification. Preserve fresh quorum, physical-owner retention and +complete selected-root checks. RustFS/load, independent-domain and public host cutover qualification remain open. The earlier interrupted-transfer checkpoint at `6e56f21f` passes all 144 @@ -906,9 +915,10 @@ Do not let plan maintenance replace implementation. 2. Pending-publication sync and cancelled controller shutdown now pass in both image forms. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt refusal, continuous capture fairness, exhausted apply/completion budgets, other - capture/admission cancellation boundaries, quarantine capacity and retained-root pressure. Track automatic orphan cleanup - and fresh-seed retry after interrupted dependency writes; safe refusal alone - does not prove retry availability. Explicit joint restart after + capture/admission cancellation boundaries, quarantine capacity and retained-root pressure. + Idle-node orphan collection and fresh-seed retry after interrupted dependency + writes now pass in both forms. Managed startup must invoke maintenance with all + groups and durable archive/transition references. Explicit joint restart after replicated entry/exit now passes; this does not qualify every in-flight or automatic-transition crash cut. Idle/time/byte checkpoint triggers require an explicit cut-reservation contract. Keep the full task gate ahead of CG-06. @@ -926,9 +936,10 @@ Do not let plan maintenance replace implementation. paths must not become permanent modes. Never derive cell sequence or witness from a group log position. Final qualification reruns all gates at one revision. -Continue from `group_snapshot_transport/seed/`, +Continue from `cell_groups/controller/checkpoints.rs`, +`group_snapshot_transport/seed/`, `cell_groups/reclaim.rs`, `cell_groups/directory/joined_bootstrap.rs`, `cell_groups/apply/recovery.rs` and -`tests/support/cell_group_seed_install.rs`. Seed metadata and an imported snapshot +`tests/support/cell_group_seed_retry.rs`. Seed metadata and an imported snapshot cannot mint the local authenticated-install receipt. Keep configuration, membership, receipt and complete root atomic. Reconcile unknown results through the exact permanent claim and the existing page owner; retain physical owners through diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 41e8c753f..ffde43893 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,68 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 idle-node orphan reclamation and authenticated cold retry + +Qualified source: `5cfa9fb652c32cc75934ef3d0a32e22ffc009f35`. `ReplicaLog::reclaim_cell_storage` +now reconciles the complete physical engine inventory and collects bounded batches +of unselected image/page files. Both materialized and extent images are supported. +The complete prescribed CG-05 command gate passes at this source: all five exact +snapshot cases, all-target build and Clippy, formatting and layout. The expanded +gate also passes 150 shared-execution cases, nineteen Raft-storage cases and +38 relevant LTX cases, plus dependency isolation, docs, source preflight and +37 plan-validator tests. All nineteen command logs pass. + +The record +`.cellar/cell-groups-evidence/CG-05-node-orphan-retry/checkpoint.json` +binds 335 source hashes, six copied executables, frozen plan inputs, the prior +checkpoint and intermediate trial outputs. Independent hashes and the plan's +`check_evidence` pass. CG-05 remains IN_PROGRESS; the native gate does not close +the remaining lifecycle/pressure obligations or qualify the production cutover. + +Maintenance requires joined runtime and SQL ownership and holds both exclusive +claims through physical completion. It runs on the existing engine worker, where +group creation/publication cannot interleave. It refuses missing, duplicate, +foreign, oversized or busy inventories. Before deletion it verifies every group's +current and retained checkpoint closures, merges image references across shared +stores and retains caller-supplied durable archive/transition roots. The node must +exclusively own those image stores. Existing live reader pins and file/directory +sync barriers still govern deletion and capacity refunds. This is idle-node orphan +maintenance; it supplies no archive coverage, Raft log truncation or serving authority. + +Two new shared-inventory cases use two groups with one image store in each form. +They retain both groups' current/checkpoint images, collect an actual orphan with +`max_files=1`, preserve and read an independent old image, then collect it after +its reader releases. A queued request retains engine/SQL claims after its waiter +is aborted, and cancelled engine close waits for physical completion. + +Two new retry cases run four additional owned SIGKILL scenarios: partial and synced +dependency writes in both image forms. The original voter disks reopen with fresh +process identities, establish a new term/quorum and commit a newer cell before +retry. The same learner disk collects its unselected transfer debris while its +empty stage still refuses bootstrap. Fresh TLS seed installation supplies the +authenticated local receipt; ordinary routing refuses before that receipt. The +learner then applies a live Raft suffix. A second cold reopen verifies all four +independent sequence-1 cells and both current/checkpoint closures. These cases +retain the voter disks; they do not exercise zero-quorum S3 recovery. + +The supplemental `generation` library filter selected zero tests. Its output is +retained as non-qualifying diagnostic evidence; the actual `private_clone` and +`extent_store` integration targets supply the relevant image-store regressions. +Compiler failures during fixture wiring and a correctly refused pre-seed route +are retained in the intermediate logs. Context probes identified the premature +ordinary route installation. The fixtures now assert that refusal and install the +route only after the authenticated receipt; source checkpoint registration follows +the fresh route. No production transport/fencing check or assertion was relaxed. + +**Reflection and next step:** same-disk orphan cleanup and authenticated retry are +implemented at the idle-node API boundary. Managed orchestration must eventually +invoke this boundary with every durable reference. Finish sustained-write capture +fairness and remaining controller/completion-budget/root-pressure qualification +before closing CG-05. Then implement CG-06 complete S3 checkpoint plus contiguous +tail coverage and safe local Raft log reclamation. CG-08 effects/cache lifetime and +CG-07 SDK/fleet cutover follow; topology, placement, hibernation, fenced recovery +under fresh incarnations, real RustFS/load and final legacy removal remain open. + ## CG-05 controller publication ownership through cancelled shutdown Qualified source: `370dcf3465b3161c9e69e8b1f398bc9d49450c0c`. Both new controller publication tests pass diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index bf31bdfc7..1e4e7c7a3 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -287,6 +287,26 @@ the existing physical store owner; selected pins prevent reopening that director while it is still owned. It verifies current and retained checkpoint closures without introducing catalog or quorum authority. +`ReplicaLog::reclaim_cell_storage` performs idle-node orphan maintenance after +runtime and SQL shutdown have joined. It reserves both exclusive claims until +physical completion and serializes on the existing engine worker. Callers supply +one exact applier for every physical group, including unattached/retired groups, +and every additional durable archive/transition root. The image stores must belong +exclusively to this engine. Missing, duplicate, foreign, oversized or busy +inventories refuse. It verifies all selected current and checkpoint closures and +merges references by exact shared-store owner before any deletion. Live pins +remain independent protection; file and directory barriers precede quota refund. +Limits bound selected cells, reclaimed files and page scans. Cancellation retains +queued work and claims; engine close joins completion. This API grants neither +Raft log truncation nor S3 coverage and does not run during normal serving. + +After interrupted dependency writes, collect only unselected debris, then obtain +a fresh seed from a fresh quorum. The empty stage still has no startup permission. +Install its ordinary process route only after authenticated seed installation +produces the local receipt; a staged grant is not ordinary membership authority. +Register source checkpoint endpoints after installing their current physical +routes so the captured transport route matches the fresh source process. + `bootstrap_joined_group_member` resumes only a locally receipted install with the same permanent remote claim and active node/disk-bound membership. Full physical recovery still verifies both current and retained checkpoint closures before From bad0387cb025fbba62f37aa42ba80b81d75366dd Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 01:58:08 -0700 Subject: [PATCH 159/568] fix: schedule checkpoints while SQL apply is full Keep the bounded pending scan alive when SQL apply capacity is occupied. Checkpoint capture and publication use independent reservations; two real image-barrier regressions reproduce the previous stall and retain SQL ownership until release. --- .../raft/cell_groups/controller/worker.rs | 10 +- .../tests/native_cell_groups_execution.rs | 10 ++ .../tests/native_cell_groups_snapshot.rs | 2 + .../cell_group_controller_checkpoints.rs | 159 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 4 +- crates/cellar/tests/support/cell_group_sql.rs | 4 +- 6 files changed, 184 insertions(+), 5 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 15b36b1e0..5efe4d70c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -99,9 +99,6 @@ impl Worker { // Rotate the bounded roster queue. A busy group cannot monopolize the // next available apply slot, and successors never run concurrently. for _ in 0..self.pending.len() { - if self.jobs.len() == self.limits.max_parallel_apply { - break; - } let Some(job) = self.pending.pop_front() else { break; }; @@ -123,6 +120,13 @@ impl Worker { if checkpoint_started { continue; } + // Capture/publication has its own bounded capacity and requires + // no SQL slot. Continue scanning when apply is full so a slow + // physical apply cannot prevent another group's checkpoint. + if self.jobs.len() == self.limits.max_parallel_apply { + self.pending.push_back(job); + continue; + } self.active.insert(job.group); let sql = self.sql.clone(); let applier = self.roster[&job.group].clone(); diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 20677c97f..426422eae 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -161,6 +161,16 @@ async fn checkpoint_controller_busy_pool_preserves_apply_and_other_group_progres runtime_fixture::automatic_checkpoints(false, true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_materialized_progresses_with_full_apply_capacity() -> Result<()> { + runtime_fixture::checkpoint_during_apply(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_controller_extents_progress_with_full_apply_capacity() -> Result<()> { + runtime_fixture::checkpoint_during_apply(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_controller_materialized_holds_sync_and_joins_cancelled_close() -> Result<()> { runtime_fixture::controller_checkpoint_shutdown(false).await diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 7bb72c0d0..9183b9349 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -91,6 +91,8 @@ async fn snapshot_memory_and_pin_bounds() -> Result<()> { checkpoint_fixture::ownership().await?; runtime_fixture::local_shutdown().await?; runtime_fixture::checkpoint_shutdown().await?; + fixture(|| runtime_fixture::checkpoint_during_apply(false)).await?; + fixture(|| runtime_fixture::checkpoint_during_apply(true)).await?; fixture(|| runtime_fixture::controller_checkpoint_shutdown(false)).await?; fixture(|| runtime_fixture::controller_checkpoint_shutdown(true)).await?; runtime_fixture::automatic_checkpoints(false, true).await diff --git a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs index 1cf5dc20d..4beeb5e60 100644 --- a/crates/cellar/tests/support/cell_group_controller_checkpoints.rs +++ b/crates/cellar/tests/support/cell_group_controller_checkpoints.rs @@ -4,6 +4,165 @@ use super::*; use raft::Storage; +struct ReleaseApply(std::sync::mpsc::SyncSender<()>); + +impl Drop for ReleaseApply { + fn drop(&mut self) { + let _ = self.0.send(()); + } +} + +pub(crate) async fn checkpoint_during_apply(rooted: bool) -> Result<()> { + let root = tempfile::tempdir()?; + let directory = if rooted { + root.path().join("rooted-0") + } else { + root.path().join("node-1/images") + }; + let faults = apply_faults::physical::Faults::new(directory, None); + let filesystem = Arc::new(apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = + Box::pin(registered::setup_in(rooted, &class, root, filesystem)).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let mut bounds = controller::limits(); + bounds.max_parallel_apply = 1; + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, bounds).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&second)?; + let route_b = group + .group_manifest()? + .context("second birth")? + .route() + .clone(); + let first_b = history(&route_b, "before-busy-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(43);")?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&route_b, &first_b, &bytes, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + let before_b = group.group_manifest()?.context("second durable cut")?; + assert!(group.checkpoint_manifest()?.is_none()); + + // Block actual image persistence on this node, not a consensus/log fence. + // The sole SQL apply slot and its retained physical owners remain occupied. + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, wait) = std::sync::mpsc::sync_channel(1); + let release = ReleaseApply(release); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + wait.recv_timeout(Duration::from_secs(30)) + .expect("release actual image persistence"); + })); + let before_a = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("first durable cut")?; + let held_cell = history(&route, "held-physical-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(97);")?; + let held = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &held_cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(controllers[0].usage().active_apply, 1); + let sql = &fixture.appliers[&(0, route.history)].0; + assert_eq!(sql.usage()?.apply_retained_operations, 1); + assert_eq!(sql.usage()?.completion_reservations, 1); + assert_eq!(controllers[0].usage().active_checkpoints, 0); + assert_eq!(pools[0].usage().sources, 0); + + let next_b = history(&route_b, "after-busy-apply")?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(67);")?; + let successor = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&route_b, &next_b, &bytes, false)?, + ) + .await?; + controller::wait_for(async || { + for runtime in &fixture.runtimes[1..] { + if runtime.status(second).await?.applied.index < successor.position.index { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert!(fixture.runtimes[0].status(second).await?.committed >= successor.position.index); + // The second group's committed successor makes its preceding complete cut + // due. Checkpoint capture/publication does not require an SQL apply slot. + tokio::time::timeout(Duration::from_secs(5), async { + while group.checkpoint_manifest()?.as_ref() != Some(&before_b) { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .with_context(|| format!( + "unrelated checkpoint must publish while the SQL apply slot is held: controller={:?}, checkpoint_pool={:?}, sql={:?}", + controllers[0].usage(), pools[0].usage(), + sql.usage().map(|usage| (usage.apply_retained_operations, usage.completion_reservations)) + ))??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before_b)); + assert_eq!(group.first_index()?, before_b.position().index.get() + 1); + assert!(group.last_index()? >= successor.position.index.get()); + assert_eq!( + fixture.runtimes[0].status(second).await?.applied, + before_b.position() + ); + assert_eq!( + fixture.runtimes[0].status(route.history).await?.applied, + before_a.position() + ); + assert_eq!(controllers[0].usage().active_apply, 1); + assert_eq!(sql.usage()?.apply_retained_operations, 1); + assert_eq!(sql.usage()?.completion_reservations, 1); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + drop(release); + controller::applied(&fixture, route.history, held.position.index).await?; + controller::applied(&fixture, second, successor.position.index).await?; + for runtime in &fixture.runtimes { + for (group, cell) in [ + (route.history, &held_cell), + (second, &first_b), + (second, &next_b), + ] { + let authority = runtime + .reconciliation_authority(group, cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + } + } + for (node, controller) in controllers.iter().enumerate() { + controller.close().await?; + assert_eq!(controller.usage().active_apply, 0); + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + } + fixture.close().await +} + pub(crate) async fn automatic_checkpoints(rooted: bool, pressure: bool) -> Result<()> { let class = RaftClass::new("Counter", 1, [81; 32])?; let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index dbba7fd3b..f4d9be243 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -814,7 +814,9 @@ pub(super) use sql::{ provisioning_retries_retained_owner, }; -pub(super) use sql::{automatic_checkpoints, controller_checkpoint_shutdown}; +pub(super) use sql::{ + automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, +}; pub(super) use sql::{membership_lifecycle, membership_sync}; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 30e03eacf..65933fefc 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -594,7 +594,9 @@ pub(crate) use provision::{ #[path = "cell_group_controller_checkpoints.rs"] mod controller_checkpoints; -pub(crate) use controller_checkpoints::{automatic_checkpoints, controller_checkpoint_shutdown}; +pub(crate) use controller_checkpoints::{ + automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, +}; #[path = "cell_group_membership.rs"] mod membership; From 6d2e0c3054521c79a6919afcefaf2cee2a5fd4c2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 02:12:33 -0700 Subject: [PATCH 160/568] docs: qualify checkpoint scheduling under SQL load --- docs/design/multi-raft/cell-groups/README.md | 5 ++- docs/design/multi-raft/cell-groups/handoff.md | 24 ++++++++++- .../design/multi-raft/cell-groups/progress.md | 43 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 8 ++++ 4 files changed, 76 insertions(+), 4 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index a45760eaf..6ba85b3c0 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -26,8 +26,9 @@ Controller publication through cancelled shutdown now passes in both image forms Idle-node complete-inventory orphan collection and fresh authenticated retry on the same interrupted learner disk now pass in both forms, including live suffix catch-up and cold restart. Remaining capture fairness and controller/budget/root-pressure -qualification precedes verified S3 archive/reclamation. The production host has -not yet cut over to shared groups. +qualification precedes verified S3 archive/reclamation. Checkpoint capture and +publication now progress while an unrelated physical apply occupies all SQL apply +capacity. The production host has not yet cut over to shared groups. See [implementation progress](progress.md) and the generated task index. Package validation is not implementation, benchmark or release qualification. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index cf2c1eb77..e6c8bab00 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -43,7 +43,25 @@ effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Trac these remaining capabilities in their owning tasks. Shared RustFS, independent voter failure domains, power loss and load remain unqualified. -The latest checkpoint at `5cfa9fb6` adds production idle-node orphan +The latest checkpoint at `bad0387c` removes checkpoint scheduling's +dependence on free SQL apply slots. The bounded pending scan still visits other +groups when SQL is full, starts checkpoint work within its own capacity, then +defers SQL work without increasing its limit. Two physical image-barrier cases +hold node-zero apply in both image forms while another group's preceding complete +checkpoint publishes. Healthy voters apply its successor; node zero retains its +current root, suffix and SQL ownership until release. Both new cases also run +inside the exact resource-bound snapshot case. + +All five exact snapshot cases, 152 execution cases and nineteen storage cases +pass, along with all sixteen commands including the prescribed gate. Its record +under `.cellar/cell-groups-evidence/CG-05-checkpoint-apply-scheduling/qualification/` +binds 335 source hashes and three executables. Both original and diagnostic +failures retain their source/output/executable hashes; independent verification +and `check_evidence` pass. No timeout, assertion or physical fence was weakened. +CG-05 remains IN_PROGRESS pending completion-lane saturation and sustained capture +fairness qualification. S3 archival, public cutover and deployment gates stay open. + +The preceding checkpoint at `5cfa9fb6` adds production idle-node orphan maintenance and authenticated retry on the same interrupted learner disk. All five exact snapshot cases, 150 execution cases, nineteen Raft-storage cases and 38 relevant LTX cases pass. All nineteen commands pass, including the complete CG-05 @@ -913,7 +931,9 @@ Do not let plan maintenance replace implementation. fault/lifecycle and availability requirements below before task completion. Preserve permanent claims, local install receipts, complete physical recovery and unrelated-group progress. 2. Pending-publication sync and cancelled controller shutdown now pass in both - image forms. Complete remaining CG-05 fault/lifecycle requirements: minority signed receipt + image forms. An unrelated checkpoint now publishes while the sole SQL apply + slot remains physically held, with all owners retained. Complete remaining + CG-05 fault/lifecycle requirements: minority signed receipt refusal, continuous capture fairness, exhausted apply/completion budgets, other capture/admission cancellation boundaries, quarantine capacity and retained-root pressure. Idle-node orphan collection and fresh-seed retry after interrupted dependency diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index ffde43893..75628c9ad 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,49 @@ CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 checkpoint progress while SQL apply capacity is full + +Qualified source: `bad0387cb025fbba62f37aa42ba80b81d75366dd`. The production controller now +scans its bounded pending queue even when every SQL apply slot is occupied. +Checkpoint capture/publication uses independent capacity and can start before +SQL work is deferred. This removes a reproduced dependency between unrelated +groups without increasing SQL concurrency or releasing physical ownership early. + +All five required snapshot cases, 152 shared-execution cases and nineteen +Raft-storage cases pass: 176 distinct Rust cases. All sixteen commands pass, +including the complete prescribed CG-05 native gate, all-target build/Clippy, +formatting/layout, dependency isolation, docs, source preflight and 37 plan-validator +tests. The record +`.cellar/cell-groups-evidence/CG-05-checkpoint-apply-scheduling/qualification/checkpoint.json` +binds 335 source hashes, three executables, frozen plan inputs and the prior +checkpoint. Independent hashes and `check_evidence` pass. CG-05 stays IN_PROGRESS. + +Two new cases hold actual node-zero image-directory persistence, once with +materialized apply and once with extent apply. The sole SQL apply slot and its +retained completion stay charged. A second group commits a successor, which the +other two voters apply. Node zero publishes that group's preceding complete +checkpoint while preserving its unchanged current root and unapplied suffix. +After physical release both groups apply, independent sequence-1 cells verify, +and controller shutdown joins all owners. The exact resource-bound snapshot case +directly runs both scenarios. + +Both cases failed before the production change at the same five-second checkpoint +deadline. A second reproduction first verified the other voters' applied successor; +diagnostics showed one active and one pending apply, zero active checkpoints and +zero checkpoint sources. The worker stopped scanning solely because SQL capacity +was full. Moving that capacity check after checkpoint admission makes both cases +pass. The initial and diagnostic reproductions retain their exact source copies, +outputs and executables; an intermediate diagnostic-format compiler failure is +also retained. No timeout, assertion, resource bound or persistence fence changed. + +**Reflection and next step:** checkpoint capture is now independent of occupied +SQL apply capacity. Publication with an exhausted completion lane and capture +fairness under sustained traffic still need direct qualification before closing +CG-05. Then deliver CG-06 complete S3 checkpoint/tail coverage and safe Raft log +reclamation, followed by effects/cache lifetime and SDK/fleet cutover. Real RustFS, +load, independent failure domains, zero-quorum recovery and final legacy removal +remain open. This gate does not combine earlier LTX binaries into its case count. + ## CG-05 idle-node orphan reclamation and authenticated cold retry Qualified source: `5cfa9fb652c32cc75934ef3d0a32e22ffc009f35`. `ReplicaLog::reclaim_cell_storage` diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 1e4e7c7a3..6f1c33e46 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -412,6 +412,14 @@ or apply pressure. Pending Raft work may still append a suffix. Only the matchin `CheckpointPublished` event releases the successor to physical apply; an admission reply alone does not. Reply/event ordering is handled in either order. +Checkpoint selection precedes the SQL apply-capacity check in the controller's +bounded pending scan. An occupied SQL slot therefore does not block another +group's capture/publication. SQL work stays queued when its capacity is full; +checkpoint lifecycle, source and completion admission remain independently bounded. +Native tests hold actual image-directory persistence in both image forms, publish +an unrelated complete checkpoint, retain its unapplied suffix, then release and +join the original physical apply. + Pool contention or transient admission overload releases the successor and backs off until a later apply boundary using the controller's node timer. There is no per-cell task or unbounded retry loop. Capture jobs are joined before releasing the From e0550acd95a78cdf1852a408620eb054ee505669 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 02:57:00 -0700 Subject: [PATCH 161/568] fix: preserve mixed-size completion admission Wake every capacity waiter and reserve bytes/items together so a rejected request cannot strand or spuriously refuse a fitting caller. Preserve the stored signal needed by late worker registration. Expose advisory lane usage and exact-ticket checkpoint publication. Qualify full completion admission, retained source ownership at real sync, and concurrent group capture progress before the writers finish. --- crates/cellar/durability/raft/budget.rs | 171 ++++++++++++---- .../cellar/durability/raft/cell_groups/mod.rs | 5 +- .../raft/cell_groups/runtime/mod.rs | 56 ++++- crates/cellar/durability/raft/options.rs | 5 +- .../tests/native_cell_groups_execution.rs | 20 ++ .../tests/native_cell_groups_snapshot.rs | 4 + .../support/cell_group_checkpoint_fairness.rs | 165 +++++++++++++++ .../support/cell_group_completion_pressure.rs | 193 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 8 + docs/design/multi-raft/cell-groups/tasks.json | 4 +- 11 files changed, 591 insertions(+), 42 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_checkpoint_fairness.rs create mode 100644 crates/cellar/tests/support/cell_group_completion_pressure.rs diff --git a/crates/cellar/durability/raft/budget.rs b/crates/cellar/durability/raft/budget.rs index 377513fc3..afe737fd4 100644 --- a/crates/cellar/durability/raft/budget.rs +++ b/crates/cellar/durability/raft/budget.rs @@ -2,13 +2,17 @@ //! Byte and item admission retained through queued work and event delivery. -use std::sync::atomic::{AtomicUsize, Ordering}; -use std::sync::Arc; +use std::sync::{Arc, Mutex, MutexGuard}; + +#[derive(Default)] +struct Usage { + bytes: usize, + items: usize, +} pub(crate) struct Budget { pub wake: tokio::sync::Notify, - bytes: AtomicUsize, - items: AtomicUsize, + usage: Mutex, max_bytes: usize, max_items: usize, } @@ -19,38 +23,49 @@ impl Budget { } pub(crate) fn usage(&self) -> (usize, usize) { - ( - self.bytes.load(Ordering::Acquire), - self.items.load(Ordering::Acquire), - ) + let usage = self.lock(); + (usage.bytes, usage.items) + } + + fn lock(&self) -> MutexGuard<'_, Usage> { + self.usage.lock().unwrap_or_else(|error| error.into_inner()) + } + + fn released(&self) { + // Requests have different byte sizes. A waiter that still cannot fit + // must not consume another request's only opportunity to make progress. + self.wake.notify_waiters(); + // Workers also inspect capacity before registering their select arm. + // Keep a stored signal when release precedes that registration. + self.wake.notify_one(); } pub fn new(max_bytes: usize, max_items: usize) -> Arc { Arc::new(Self { wake: tokio::sync::Notify::new(), - bytes: AtomicUsize::new(0), - items: AtomicUsize::new(0), + usage: Mutex::new(Usage::default()), max_bytes, max_items, }) } pub fn acquire(self: &Arc, bytes: usize) -> Option { - self.items - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { - n.checked_add(1).filter(|n| *n <= self.max_items) - }) - .ok()?; - if self + // Reserve both dimensions together. A rejected byte request must never + // temporarily exhaust the item count seen by a fitting caller. + let mut usage = self.lock(); + let next_bytes = usage .bytes - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { - n.checked_add(bytes).filter(|n| *n <= self.max_bytes) - }) - .is_err() - { - self.items.fetch_sub(1, Ordering::AcqRel); - return None; - } + .checked_add(bytes) + .filter(|n| *n <= self.max_bytes)?; + let next_items = usage + .items + .checked_add(1) + .filter(|n| *n <= self.max_items)?; + *usage = Usage { + bytes: next_bytes, + items: next_items, + }; + drop(usage); Some(Permit { owner: self.clone(), bytes, @@ -70,16 +85,15 @@ impl Permit { let Some(extra) = bytes.checked_sub(self.bytes) else { return false; }; - if self - .owner + let mut usage = self.owner.lock(); + let Some(next) = usage .bytes - .fetch_update(Ordering::AcqRel, Ordering::Acquire, |n| { - n.checked_add(extra).filter(|n| *n <= self.owner.max_bytes) - }) - .is_err() - { + .checked_add(extra) + .filter(|n| *n <= self.owner.max_bytes) + else { return false; - } + }; + usage.bytes = next; self.bytes = bytes; true } @@ -91,10 +105,12 @@ impl Permit { return false; } let released = self.bytes - bytes; + let mut usage = self.owner.lock(); + usage.bytes -= released; self.bytes = bytes; - self.owner.bytes.fetch_sub(released, Ordering::AcqRel); + drop(usage); if released != 0 { - self.owner.wake.notify_one(); + self.owner.released(); } true } @@ -102,15 +118,96 @@ impl Permit { impl Drop for Permit { fn drop(&mut self) { - self.owner.bytes.fetch_sub(self.bytes, Ordering::AcqRel); - self.owner.items.fetch_sub(1, Ordering::AcqRel); - self.owner.wake.notify_one(); + let mut usage = self.owner.lock(); + usage.bytes -= self.bytes; + usage.items -= 1; + drop(usage); + self.owner.released(); } } #[cfg(test)] mod tests { use super::*; + use futures_util::FutureExt; + + fn fitting_waiter_after_partial_release(shrink: bool) { + let budget = Budget::new(100, 3); + let mut held = budget.acquire(80).unwrap(); + let mut other = Some(budget.acquire(20).unwrap()); + let mut large = std::pin::pin!(budget.wake.notified()); + large.as_mut().enable(); + let mut small = std::pin::pin!(budget.wake.notified()); + small.as_mut().enable(); + assert!(budget.acquire(40).is_none()); + assert!(budget.acquire(20).is_none()); + if shrink { + assert!(held.shrink_to(60)); + } else { + drop(other.take()); + } + assert_eq!(budget.usage().0, 80); + // The first registered request still cannot use the released bytes. + // A fitting request must be woken without another capacity change. + assert!(large.as_mut().now_or_never().is_some()); + assert!(budget.acquire(40).is_none()); + assert!( + small.as_mut().now_or_never().is_some(), + "a non-fitting waiter must not consume the fitting request's only wake" + ); + let admitted = budget.acquire(20).unwrap(); + assert_eq!(budget.usage().0, 100); + drop(admitted); + drop(other); + drop(held); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn release_wakes_fitting_waiter_behind_a_larger_request() { + fitting_waiter_after_partial_release(false); + } + + #[test] + fn shrink_wakes_fitting_waiter_behind_a_larger_request() { + fitting_waiter_after_partial_release(true); + } + + #[test] + fn late_worker_observes_release_without_prior_registration() { + let budget = Budget::new(100, 1); + let held = budget.acquire(100).unwrap(); + drop(held); + assert!(budget.wake.notified().now_or_never().is_some()); + assert_eq!(budget.usage(), (0, 0)); + } + + #[test] + fn nonfitting_attempts_do_not_consume_available_item_capacity() { + let budget = Budget::new(100, 2); + let held = budget.acquire(80).unwrap(); + let start = std::sync::Barrier::new(2); + std::thread::scope(|scope| { + let worker = scope.spawn(|| { + start.wait(); + for _ in 0..100_000 { + assert!(budget.acquire(30).is_none()); + } + }); + start.wait(); + for _ in 0..100_000 { + let fitting = budget.acquire(20); + assert!( + fitting.is_some(), + "a rejected request must not charge an item against a fitting caller" + ); + drop(fitting); + } + worker.join().unwrap(); + }); + drop(held); + assert_eq!(budget.usage(), (0, 0)); + } #[test] fn grow_preserves_item_and_refuses_without_changing_accounting() { diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 11ecc1275..7212a96e0 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -67,7 +67,10 @@ mod retention; mod root; mod runtime; -pub use runtime::{CellGroupRuntime, CellGroupRuntimeEvent, CellGroupRuntimeEvents}; +pub use runtime::{ + CellGroupAdmissionUsage, CellGroupRuntime, CellGroupRuntimeEvent, CellGroupRuntimeEvents, + CellGroupRuntimeUsage, +}; pub use manifest::{ CellImageGeometry, CellManifestLeaf, EffectSummary, ManifestError, ManifestLimits, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 7c1f7d003..c3a856b48 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -27,6 +27,42 @@ use tokio::sync::{mpsc, oneshot, Notify}; const COMPLETION_SLOTS: usize = 8; const SNAPSHOT_HINT_SLOTS: usize = 256; +/// Advisory admission counters. Different lanes may be observed at different +/// moments; these values neither reserve capacity nor establish durability or +/// serving authority. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupAdmissionUsage { + pub items: usize, + pub bytes: usize, + pub max_items: usize, + pub max_bytes: usize, +} + +impl CellGroupAdmissionUsage { + fn read(budget: &Budget) -> Self { + let (bytes, items) = budget.usage(); + let (max_bytes, max_items) = budget.capacity(); + Self { + items, + bytes, + max_items, + max_bytes, + } + } +} + +/// Node-wide request/pin admission, retained through each lane's owned work. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupRuntimeUsage { + pub ordinary: CellGroupAdmissionUsage, + /// Eight slots and max_ready_bytes of request metadata, independently of + /// the worker's Ready/persistence accounting. Refunded at owner admission; + /// physical work retains its other source, root and storage owners. + pub completions: CellGroupAdmissionUsage, + pub snapshot_feedback: CellGroupAdmissionUsage, + pub read_pins: CellGroupAdmissionUsage, +} + pub struct CellGroupRuntimeEvent { group: GroupHistory, effect: Result, @@ -153,6 +189,17 @@ impl Drop for TransportClaim { } impl CellGroupRuntime { + /// Observe admission without requiring a free request slot. These gauges + /// may include work whose response waiter has disconnected. + pub fn usage(&self) -> CellGroupRuntimeUsage { + CellGroupRuntimeUsage { + ordinary: CellGroupAdmissionUsage::read(&self.inner.budget), + completions: CellGroupAdmissionUsage::read(&self.inner.completions), + snapshot_feedback: CellGroupAdmissionUsage::read(&self.inner.snapshot_hints), + read_pins: CellGroupAdmissionUsage::read(&self.inner.read_pins), + } + } + pub(crate) fn same_owner(&self, other: &Self) -> bool { Arc::ptr_eq(&self.inner, &other.inner) } @@ -624,7 +671,14 @@ impl CellGroupRuntime { .map(|_| ()) } - pub(crate) async fn checkpoint_before_apply( + /// Admit publication of the exact captured base of a runtime-issued apply + /// ticket, while its owner retains the successor without preparing it. + /// Uses reserved completion admission, waiting with its source/pins when + /// capacity is occupied. Success proves admission; CheckpointPublished or + /// selected engine recovery proves persistence. The same ticket must then + /// complete its successor. Cancelling before admission publishes no work; + /// admitted work stays owned by the runtime through joined shutdown. + pub async fn checkpoint_before_apply( &self, history: GroupHistory, ticket: CellGroupTicket, diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index 7441615fc..dca050a98 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -100,8 +100,9 @@ pub struct RaftRuntimeOptions { pub tick_interval_ms: Option, pub election_seed: Option<[u8; 32]>, pub max_groups: usize, - /// Ordinary requests. Eight additional queue slots and 32 KiB are reserved - /// for physical apply/snapshot completions so saturated work can settle. + /// Ordinary requests. Eight additional queue slots are reserved for physical + /// completions so saturated work can settle. Cell-group completion requests + /// bound retained manifest metadata with max_ready_bytes. /// Transport hint admission has a separate 256 slots/1 MiB; the worker /// coalesces retained hints to one small record per current group member. pub max_requests: usize, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 426422eae..62628d525 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -171,6 +171,26 @@ async fn checkpoint_controller_extents_progress_with_full_apply_capacity() -> Re runtime_fixture::checkpoint_during_apply(true).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_materialized_waits_for_full_completion_admission() -> Result<()> { + runtime_fixture::checkpoint_completion_pressure(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_extents_waits_for_full_completion_admission() -> Result<()> { + runtime_fixture::checkpoint_completion_pressure(true).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_materialized_concurrent_writers_share_capture_capacity() -> Result<()> { + runtime_fixture::concurrent_checkpoints(false).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_extents_concurrent_writers_share_capture_capacity() -> Result<()> { + runtime_fixture::concurrent_checkpoints(true).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_controller_materialized_holds_sync_and_joins_cancelled_close() -> Result<()> { runtime_fixture::controller_checkpoint_shutdown(false).await diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 9183b9349..60be8f8b2 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -93,6 +93,10 @@ async fn snapshot_memory_and_pin_bounds() -> Result<()> { runtime_fixture::checkpoint_shutdown().await?; fixture(|| runtime_fixture::checkpoint_during_apply(false)).await?; fixture(|| runtime_fixture::checkpoint_during_apply(true)).await?; + fixture(|| runtime_fixture::checkpoint_completion_pressure(false)).await?; + fixture(|| runtime_fixture::checkpoint_completion_pressure(true)).await?; + fixture(|| runtime_fixture::concurrent_checkpoints(false)).await?; + fixture(|| runtime_fixture::concurrent_checkpoints(true)).await?; fixture(|| runtime_fixture::controller_checkpoint_shutdown(false)).await?; fixture(|| runtime_fixture::controller_checkpoint_shutdown(true)).await?; runtime_fixture::automatic_checkpoints(false, true).await diff --git a/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs b/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs new file mode 100644 index 000000000..b3b768b76 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_checkpoint_fairness.rs @@ -0,0 +1,165 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Concurrent real enrollments compete for one shared checkpoint slot. Inspect +//! each group's durable progress at fixed cuts before either writer finishes. +use super::*; + +struct Trace { + cut: tokio::sync::Barrier, + resume: tokio::sync::Barrier, +} + +async fn write( + runtime: CellGroupRuntime, + route: GroupRoute, + rooted: bool, + trace: Arc, + controller: CellGroupController, +) -> Result { + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(83);")?; + let mut last = None; + for round in 0..32 { + let cell = history(&route, &format!("concurrent-checkpoint-{round}"))?; + let accepted = runtime + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &bytes, rooted)?, + ) + .await?; + tokio::time::timeout(Duration::from_secs(15), async { + while runtime.status(route.history).await?.applied.index < accepted.position.index { + tokio::time::sleep(Duration::from_millis(2)).await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + assert!(controller.usage().active_checkpoints <= 1); + assert_eq!(controller.usage().quarantined_groups, 0); + if round == 8 || round == 24 { + trace.cut.wait().await; + trace.resume.wait().await; + } + last = Some(accepted); + } + last.context("nonempty write trace") +} + +pub(crate) async fn concurrent_checkpoints(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let second = controller::add_second(&mut f, route.history).await?; + f.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in f.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let held = pools[0].receive( + f.appliers[&(0, route.history)].1.clone(), + checkpoint_receive::scratch()?, + f.cluster._root.path().into(), + )?; + let charged = pools[0].usage(); + let mut bounds = controller::limits(); + bounds.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 2, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut f, bounds).await?; + f.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(second).await?.ready)).await?; + let second_route = f.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("second route")? + .route() + .clone(); + let trace = Arc::new(Trace { + cut: tokio::sync::Barrier::new(3), + resume: tokio::sync::Barrier::new(3), + }); + let mut writers = Vec::new(); + for (route, rooted) in [(route.clone(), rooted), (second_route, false)] { + let runtime = f.runtimes[0].clone(); + let control = controllers[0].clone(); + let trace = trace.clone(); + let group = route.history; + writers.push(( + group, + f.cluster + .domain + .spawn(async move { write(runtime, route, rooted, trace, control).await })?, + )); + } + tokio::time::timeout(Duration::from_secs(15), trace.cut.wait()).await?; + assert_eq!(pools[0].usage(), charged); + assert!(controllers[0].usage().checkpoint_retries > 0); + assert_eq!(controllers[0].usage().checkpoints_published, 0); + let mut release_cuts = BTreeMap::new(); + for group in [route.history, second] { + let log = f.cluster.voters[0].log.cell_group(&group)?; + assert!(log.checkpoint_manifest()?.is_none()); + release_cuts.insert( + group, + log.group_manifest()?.context("held-slot cut")?.position(), + ); + } + drop(held); + trace.resume.wait().await; + // Each writer has sixteen more committed/applied entries before stopping + // at this cut. Both must obtain a newer checkpoint while its peer is active. + tokio::time::timeout(Duration::from_secs(15), trace.cut.wait()).await?; + for group in [route.history, second] { + let log = f.cluster.voters[0].log.cell_group(&group)?; + let current = log.group_manifest()?.context("concurrent applied cut")?; + let checkpoint = log + .checkpoint_manifest()? + .context("concurrent checkpoint")?; + assert!(checkpoint.position().index > release_cuts[&group].index); + assert!(checkpoint.position().index < current.position().index); + assert_eq!( + current.root().cells(), + if group == route.history { 27 } else { 25 } + ); + } + trace.resume.wait().await; + for (group, writer) in writers { + let last = writer.await??; + controller::applied(&f, group, last.position.index).await?; + } + for (node, controller) in controllers.iter().enumerate() { + assert_eq!(controller.usage().quarantined_groups, 0); + assert!(!controller.usage().closing); + for group in [route.history, second] { + let log = f.cluster.voters[node].log.cell_group(&group)?; + assert!(log.checkpoint_manifest()?.is_some()); + let current = log.group_manifest()?.context("final applied cut")?; + assert_eq!( + current.root().cells(), + if group == route.history { 34 } else { 32 } + ); + let (sql, applier) = &f.appliers[&(node, group)]; + let applier = applier.clone(); + let recovered = f + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert_eq!(recovered.marker(), ¤t); + for round in 0..32 { + let (leaf, _) = recovered + .pin_cell(&CellId::new(format!("concurrent-checkpoint-{round}"))?) + .context("independent concurrent cell")?; + assert_eq!(leaf.version.sequence.get(), 1); + } + assert!(sql.usage()?.completion_reservations <= 1); + } + controller.close().await?; + assert_eq!(controller.usage().active_checkpoints, 0); + assert_eq!(pools[node].usage().sources, 0); + assert_eq!(pools[node].usage().retained_bytes, 0); + } + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_completion_pressure.rs b/crates/cellar/tests/support/cell_group_completion_pressure.rs new file mode 100644 index 000000000..72ceab5d2 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_completion_pressure.rs @@ -0,0 +1,193 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real completed images fill the runtime's reserved admission before polling +//! its worker. A separately captured checkpoint must wait without losing pins. +use super::*; +use futures_util::FutureExt; +use raft::Storage; + +pub(crate) async fn checkpoint_completion_pressure(rooted: bool) -> Result<()> { + // A current-thread owner makes the admission burst deterministic even when + // this fixture is composed inside a multi-thread integration test. Both + // execution domains and the nested runtime join before this call returns. + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let result = domain + .blocking(move || -> Result<()> { + let runtime = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build()?; + runtime.block_on(Box::pin(run(rooted))) + })? + .await; + domain.shutdown().await; + result? +} + +async fn run(rooted: bool) -> Result<()> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut f, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let slots = f.runtimes[0].usage().completions.max_items; + assert_eq!(slots, 8, "qualify the actual fixed completion capacity"); + let mut groups = vec![route.history]; + for n in 0..slots { + groups.push(f.add_group(100 + n as u8).await?); + } + f.connect_tcp().await?; + for &group in &groups[1..] { + f.runtimes[0].campaign(group).await?; + } + f.settle(&groups).await?; + let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(71);")?; + let cut = images::Cut::new(&[], &bytes, 0, true, false); + let mut targets = BTreeMap::new(); + let mut cells = Vec::new(); + for &group in &groups[1..] { + let marker = f.cluster.voters[0] + .log + .cell_group(&group)? + .group_manifest()? + .context("completed-image group")?; + let cell = history(marker.route(), "completed-before-checkpoint")?; + let accepted = f.runtimes[0] + .propose(group, request(marker.route(), cell.clone(), 1, &cut)?) + .await?; + targets.insert(group, accepted.position); + cells.push((group, cell)); + } + let successor_cell = history(&route, "after-capacity-checkpoint")?; + let successor = f.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &successor_cell, &bytes, rooted)?, + ) + .await?; + let mut held = BTreeMap::new(); + tokio::time::timeout(Duration::from_secs(15), async { + while held.len() != groups.len() { + f.pump(&groups, &mut held).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + let work = held + .remove(&route.history) + .context("checkpoint successor")?; + let before = work.base.marker().clone(); + let group = f.cluster.voters[0].log.cell_group(&route.history)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source = pool.clone(); + let log = group.clone(); + let root = work.base.clone(); + let checkpoint = f + .cluster + .domain + .blocking(move || source.capture(&log, &root)?.prepare_publication())? + .await??; + let charged = pool.usage(); + assert_eq!(charged.sources, 1); + assert!(charged.retained_bytes > 0); + let mut prepared = Vec::new(); + for &group in &groups[1..] { + let work = held.remove(&group).context("completed-image ticket")?; + let mut voters = f.extra.remove(&group).context("physical image owners")?; + let (voters, work, next) = f + .cluster + .domain + .blocking(move || -> Result<_> { + let next = prepare(&mut voters[0], &work)?; + Ok((voters, work, next)) + })? + .await??; + f.extra.insert(group, voters); + prepared.push((group, work, next)); + } + let runtime = f.runtimes[0].clone(); + let mut pending = prepared + .into_iter() + .map(|(group, work, next)| runtime.complete_apply(group, work, next).boxed()) + .collect::>(); + // No owner turn occurs between these polls. All eight real completions are + // published to the mailbox, retaining their exact fsynced image/ticket. + for call in &mut pending { + assert!(call.as_mut().now_or_never().is_none()); + } + let usage = runtime.usage().completions; + assert_eq!(usage.items, usage.max_items); + assert!(usage.bytes > 0 && usage.bytes <= usage.max_bytes); + pending.push( + runtime + .checkpoint_before_apply(route.history, work.ticket.clone(), checkpoint) + .boxed(), + ); + assert!(pending + .last_mut() + .unwrap() + .as_mut() + .now_or_never() + .is_none()); + assert_eq!(runtime.usage().completions, usage); + assert_eq!(pool.usage(), charged); + assert!(group.checkpoint_manifest()?.is_none()); + + let gate = f.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let mut admitted = futures_util::future::try_join_all(pending).boxed(); + assert!(admitted.as_mut().now_or_never().is_none()); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + // A real persistence fence is now held. Polling the ninth request lets it + // acquire refunded completion admission while its source remains physical. + assert!(admitted.as_mut().now_or_never().is_none()); + runtime.status(route.history).await?; + tokio::time::timeout(Duration::from_secs(5), admitted).await??; + assert_eq!(runtime.usage().completions.items, 0); + assert_eq!(runtime.usage().completions.bytes, 0); + assert_eq!(pool.usage(), charged); + assert!(group.checkpoint_manifest()?.is_none()); + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert_eq!( + runtime.status(route.history).await?.applied, + before.position() + ); + drop(release); + controller::wait_for(async || Ok(group.checkpoint_manifest()?.as_ref() == Some(&before))) + .await?; + controller::wait_for(async || Ok(pool.usage().sources == 0)).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + assert!(group.last_index()? >= successor.position.index.get()); + let (sql, applier) = &f.appliers[&(0, route.history)]; + sql.prepare_apply(applier, work) + .await? + .complete(&runtime) + .await?; + f.drive(Some(successor.position.index)).await?; + tokio::time::timeout(Duration::from_secs(15), async { + while targets + .iter() + .any(|(group, position)| f.applied[group].iter().any(|index| *index < position.index)) + { + f.pump(&[], &mut BTreeMap::new()).await?; + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + for runtime in &f.runtimes { + for (group, cell) in &cells { + let authority = runtime + .reconciliation_authority(*group, cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + assert_eq!(authority.cell().witness.position, targets[group]); + } + let authority = runtime + .reconciliation_authority(route.history, successor_cell.clone()) + .await?; + assert_eq!(authority.cell().version.sequence.get(), 1); + assert_eq!(authority.cell().witness.position, successor.position); + assert_eq!(runtime.usage().completions.items, 0); + assert_eq!(runtime.usage().completions.bytes, 0); + } + assert_eq!(pool.usage().retained_bytes, 0); + f.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index f4d9be243..d225dde8a 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -814,6 +814,8 @@ pub(super) use sql::{ provisioning_retries_retained_owner, }; +pub(super) use sql::checkpoint_completion_pressure; +pub(super) use sql::concurrent_checkpoints; pub(super) use sql::{ automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, }; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 65933fefc..f014233b6 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -598,6 +598,14 @@ pub(crate) use controller_checkpoints::{ automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, }; +#[path = "cell_group_completion_pressure.rs"] +mod completion_pressure; +pub(crate) use completion_pressure::checkpoint_completion_pressure; + +#[path = "cell_group_checkpoint_fairness.rs"] +mod checkpoint_fairness; +pub(crate) use checkpoint_fairness::concurrent_checkpoints; + #[path = "cell_group_membership.rs"] mod membership; pub(crate) use membership::{membership_lifecycle, membership_sync}; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index ee5b75dc7..35311e6d9 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1605,7 +1605,9 @@ "crates/cellar/tests/support/cell_group_engine_fixture.rs", "crates/cellar/durability/raft/cell_groups/reclaim.rs", "crates/cellar/tests/support/cell_group_node_reclaim.rs", - "crates/cellar/tests/support/cell_group_seed_retry.rs" + "crates/cellar/tests/support/cell_group_seed_retry.rs", + "crates/cellar/tests/support/cell_group_completion_pressure.rs", + "crates/cellar/tests/support/cell_group_checkpoint_fairness.rs" ], "test_target": "native_cell_groups_snapshot", "steps": [ From 3888678e1706c9b75d712d9712d38c6f4f506ade Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 03:32:31 -0700 Subject: [PATCH 162/568] docs: close CG-05 native snapshot qualification --- docs/design/multi-raft/cell-groups/README.md | 67 ++++------ docs/design/multi-raft/cell-groups/handoff.md | 126 ++++++++---------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 70 +++++++++- .../multi-raft/cell-groups/protocols.md | 36 +++-- docs/design/multi-raft/cell-groups/tasks.json | 6 +- 6 files changed, 180 insertions(+), 127 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 6ba85b3c0..285df4d25 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -3,34 +3,15 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 contracts, CG-02 storage, CG-03 routing/admission preparation and -CG-04 shared execution have passed their scoped local evidence gates. -CG-05 is in progress: checkpoint capture, receiver verification, atomic storage -installation, exact Raft Ready acceptance, local publication and native Raft-generated -catch-up, authenticated snapshot RPC and automatic controller checkpoint policy -pass their scoped regression gates. Durable joint membership and leadership handoff -also pass a partial gate. A learner seed source authorized by a fresh quorum read -now covers empty groups, bounded capture of complete roots and stream revocation. -Authenticated destination staging/install, caught-up promotion and replacement -after deleting a voter disk now pass native tests in both image forms. Seed request -refusal, revoked/expired sessions, retained-peer reconnection and minority commit/read -refusal also pass. Receiver reply/chunk faults and pre-install process revocation -now pass; physical verifier failures promptly close blocked seed/checkpoint streams. -Queued/entered seed revocation and sync-error recovery now pass in both image forms. -Staged and engine-boundary seed SIGKILL recovery now passes in both image forms. -Explicit joint-consensus SIGKILL/restart and completion now pass in both image forms. -Bounded 32-cell transfer, scratch contention and old-reader image retention/release -now pass in both image forms. Partial/synced seed-dependency SIGKILL cuts and transfer-time source-process -revocation now pass. All five exact native acceptance cases and their prescribed command gate now pass. -Controller publication through cancelled shutdown now passes in both image forms. -Idle-node complete-inventory orphan collection and fresh authenticated retry on the -same interrupted learner disk now pass in both forms, including live suffix catch-up -and cold restart. Remaining capture fairness and controller/budget/root-pressure -qualification precedes verified S3 archive/reclamation. Checkpoint capture and -publication now progress while an unrelated physical apply occupies all SQL apply -capacity. The production host has not yet cut over to shared groups. +CG-01 through CG-05 have passed their scoped native evidence gates. CG-05's +complete checkpoint, authenticated learner/rebuild, membership, restart and +resource-ownership requirements pass at `e0550acd`. Its current gate includes +all five exact cases and 205 Rust regressions across eighteen commands. +CG-06 verified S3 checkpoint/tail coverage and safe log reclamation is next. +The production SDK/fleet has not yet cut over; topology, hibernation, complete S3 +recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. -Package validation is not implementation, benchmark or release qualification. +Package validation alone does not qualify implementation or a release. This delivery is the executable design and implementation plan. End-to-end implementation and hard cutover remain tracked as separate task and release gates. See the [executor handoff](handoff.md) for the current @@ -53,7 +34,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-05` emits a complete executor brief, source excerpts, + eligibility; `show CG-06` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -69,8 +50,10 @@ This checkout retains evidence under the ignored `.cellar/cell-groups-evidence/CG-04-qualified/` and its supporting `.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` directory. A fresh checkout must receive these retained directories at the same relative paths -before validating the four DONE checkpoints. Missing evidence fails validation; it is never treated as a pass. -CG-05's partial source checkpoint is retained in +before validating the first four DONE checkpoints. CG-05 additionally requires +`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/` and its +retained predecessor/reproduction records. Missing evidence fails validation. +CG-05's earlier partial source checkpoint is retained in `.cellar/cell-groups-evidence/CG-05-checkpoint-source/` and the receiver slice in `.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`, followed by owner/runtime acceptance in `.cellar/cell-groups-evidence/CG-05-snapshot-owner/`. These partial @@ -78,14 +61,15 @@ checkpoints and local publication in `.cellar/cell-groups-evidence/CG-05-local-checkpoint/`, authenticated transport in `.cellar/cell-groups-evidence/CG-05-snapshot-transport/`, automatic policy in `.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` and membership in -`.cellar/cell-groups-evidence/CG-05-group-membership/` do not complete CG-05. +`.cellar/cell-groups-evidence/CG-05-group-membership/` remain historical partial records; +the complete current-source gate above closes CG-05. The design, task briefs and validator unit tests are committed in this package. ```sh python3 docs/design/multi-raft/cell-groups/plan.py validate python3 docs/design/multi-raft/cell-groups/plan.py preflight python3 docs/design/multi-raft/cell-groups/plan.py ready -python3 docs/design/multi-raft/cell-groups/plan.py show CG-05 +python3 docs/design/multi-raft/cell-groups/plan.py show CG-06 python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' ``` @@ -104,17 +88,14 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -The implementation executor should continue from the existing code and retained -evidence. Resume CG-05 from its exact acceptance gate and remaining lifecycle/availability -obligations; intermediate checkpoints do not establish full task completion. - -Continue CG-05 after reading -[progress](progress.md) and inspecting the working diff. -CG-01 through CG-04 are DONE only at their named contract/storage/admission -and shared-execution boundaries; the remaining tasks still require implementation. `ready` lists eligible TODO tasks; -an IN_PROGRESS task is resumed from its existing brief and working state. Each -task's complete source-bound evidence gate must pass before changing its status -to DONE. Continue in the generated dependency order afterward. +Continue from the existing code and retained evidence. CG-01 through CG-05 are +DONE at their declared native boundaries; CG-06 is dependency-ready and requires +implementation. Read [progress](progress.md), then run `plan.py show CG-06` for the +complete archive/reclamation brief. Preserve every group's selected current and +checkpoint closure, exact cell identities and permanent physical claims. Only +verified complete remote coverage may settle S3 outputs or authorize archive-policy +reclamation. A task's current-source gate must pass before marking it DONE; final +release repeats the complete capability/profile matrix at one revision. Physical group genesis, typed replica handles, permanent initial-member claims, complete group inventory and invocation-bounded catalog-confirmed redirects are diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index e6c8bab00..9110af798 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,35 +15,37 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -CG-04 is now DONE at `69e41e86` after its complete scoped acceptance gate: -83 individual exact-case passes, all required commands, and the supporting 238-case -regression gate. `.cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json` -retains 98 distinct command records and 288 source hashes. The plan runner -validates the evidence and now selects CG-05 as the next eligible task. Preserve -all acceptance cases; the task-boundary audit removed no requirements or checks. - -CG-05 is now IN_PROGRESS. Its native source captures an exact selected complete -root, configuration and membership, and retains an admitted immutable closure -through bounded streaming and cancellation. Both image forms and empty enrollment -are represented. See the latest [progress](progress.md) for source evidence and -limits. Receiver verification and atomic storage installation now pass their -partial gate, including cold recovery of the current and older checkpoint roots. -Exact Raft snapshot acceptance and persistence now pass their owner/runtime gate. -Local checkpoint publication and native Raft-generated catch-up now pass their -partial gate. Authenticated checkpoint streams now pass their partial transport gate. -Automatic entry-count checkpointing at controller apply boundaries now passes its -partial gate. Durable whole-group membership and leadership handoff now pass -an additional partial gate. Authenticated fresh-learner installation, catch-up, -promotion and deleted-voter-disk replacement now pass a native partial gate. -All five CG-05 acceptance cases now pass; complete the remaining fault/pressure -and capture-fairness obligations before closing the task. -CG-06 follows with archival reclamation. Automatic -production request/voter startup and public SDK/fleet binding belong to CG-07; -effects/cache lifecycle belong to CG-08, and final legacy removal to CG-16. Track -these remaining capabilities in their owning tasks. -Shared RustFS, independent voter failure domains, power loss and load remain unqualified. - -The latest checkpoint at `bad0387c` removes checkpoint scheduling's +CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 +closes at `e0550acd` with all five exact snapshot cases and eighteen passing +commands, backed by 205 Rust cases, 337 source hashes and five executables. +Its source-bound record is +`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json`. +Independent hashes, full `check_evidence` and the closure audit in [progress](progress.md) +pass. CG-04's earlier 83 exact-case/238-regression gate remains retained at +`69e41e86`; no task requirement or acceptance oracle was removed. + +The final CG-05 slice fixes wrong-size wake starvation and transient item charges +from byte-rejected budget attempts. It exposes advisory runtime lane usage and +the existing exact-ticket checkpoint publication boundary. Both image forms fill +all eight completion slots with actual fsynced work, retain the ninth checkpoint +through admission and a held real engine fence, then finish its exact successor. +Two concurrent writers share one checkpoint slot through a bounded 64-enrollment +trace, inspect both newer checkpoints before the writers finish, and verify every +cell and both selected closures on all three nodes. Every new native scenario runs +directly inside the exact resource-bound acceptance case. + +Continue with CG-06 complete group S3 archival, verified checkpoint/tail coverage, +safe log reclamation and fenced unmanaged recovery. Follow with CG-08 effects/cache +lifetime, then CG-07 public SDK/fleet cutover. Idle-node orphan maintenance must be +invoked by managed startup with the complete durable-reference inventory. The +entry-count policy retains the idle tail; idle/time/byte checkpoint policy and +coordinated wake remain future work. Real RustFS/load, broader crash schedules, +independent domains, managed disaster recovery and final removal of the old host +remain required later gates. The full end-to-end goal is not complete. + +The following checkpoints retain their historical source and partial boundaries. + +The preceding checkpoint at `bad0387c` removes checkpoint scheduling's dependence on free SQL apply slots. The bounded pending scan still visits other groups when SQL is full, starts checkpoint work within its own capacity, then defers SQL work without increasing its limit. Two physical image-barrier cases @@ -923,48 +925,30 @@ boundary. At the end of each turn, record the completed behavior, evidence and limits, then identify the most critical remaining gap and the concrete next step. Do not let plan maintenance replace implementation. -1. Finish the full CG-05 acceptance gate around the implemented checkpoint, - learner and membership paths. Bounded 32-cell transfer and physical reader - retention/release now pass, as do partial/synced dependency crash cuts and - transfer-time source-process revocation. The declared snapshot target and all - five exact cases now pass their prescribed command gate. Close the remaining - fault/lifecycle and availability requirements below before task completion. Preserve permanent claims, local install receipts, complete - physical recovery and unrelated-group progress. -2. Pending-publication sync and cancelled controller shutdown now pass in both - image forms. An unrelated checkpoint now publishes while the sole SQL apply - slot remains physically held, with all owners retained. Complete remaining - CG-05 fault/lifecycle requirements: minority signed receipt - refusal, continuous capture fairness, exhausted apply/completion budgets, other - capture/admission cancellation boundaries, quarantine capacity and retained-root pressure. - Idle-node orphan collection and fresh-seed retry after interrupted dependency - writes now pass in both forms. Managed startup must invoke maintenance with all - groups and durable archive/transition references. Explicit joint restart after - replicated entry/exit now passes; this does not qualify every in-flight or - automatic-transition crash cut. Idle/time/byte checkpoint triggers require an - explicit cut-reservation contract. Keep the full task gate ahead of CG-06. -3. Deliver CG-06 archive/reclamation with verified complete checkpoints and - contiguous tails. Upload completion alone cannot authorize reclaiming local - recovery dependencies or an S3 durability acknowledgement. -4. Follow the declared dependency order through CG-08 and CG-07 for effect/cache - scheduling and production SDK/fleet cutover. Retain catalog and current-term - authorization, coalesced preparation, physical ownership after disconnect and - group-local quarantine. Shared storage uncertainty stays node-wide. A missing - receipt stays ambiguous; callbacks cannot be replayed automatically. -5. Continue topology, placement, hibernation and fresh-incarnation recovery gates. - Remove `TransportRuntime::Legacy`, cell-scoped transport, `QueryVersion::GroupCut` - and `RaftClass::invoke_group_query` with the old host in CG-16. These transitional - paths must not become permanent modes. Never derive cell sequence or witness - from a group log position. Final qualification reruns all gates at one revision. - -Continue from `cell_groups/controller/checkpoints.rs`, -`group_snapshot_transport/seed/`, `cell_groups/reclaim.rs`, -`cell_groups/directory/joined_bootstrap.rs`, `cell_groups/apply/recovery.rs` and -`tests/support/cell_group_seed_retry.rs`. Seed metadata and an imported snapshot -cannot mint the local authenticated-install receipt. Keep configuration, membership, -receipt and complete root atomic. Reconcile unknown results through the exact -permanent claim and the existing page owner; retain physical owners through -cancellation and joined shutdown. Use canonical configuration equality when -comparing replay with live RawNode membership. +1. Implement CG-06 complete-group checkpoints plus contiguous committed tails. + Replay cell-addressed LTX, enrollments, membership and administrative entries; + validate every cell base chain and dependency before publishing coverage. +2. Keep Quorum progress separate from S3 settlement. Bound archive lag/unarchived + bytes and tie actual local truncation/purge to covering checkpoints, configured + archive policy and all live group/reader/transition pins. Verify unmanaged + fresh-incarnation seeding through the existing fencing authority. +3. Follow CG-08 then CG-07 for effect/cache scheduling and production SDK/fleet + cutover. Preserve catalog/current-term authorization, captured inputs, exact + receipt reconciliation, cancellation ownership and group-local quarantine. +4. Complete CG-09 through CG-12 for irreversible transfer, split/merge, placement + and authority-checked coordinated hibernation/wake. Retain independent cell + histories and versions; never derive them from a physical group log position. +5. Build CG-14's real-infrastructure harness, then qualify managed/in-transition + S3 recovery in CG-13 and the complete crash/load/domain-loss matrix in CG-15. + CG-16 removes the old host, `TransportRuntime::Legacy`, cell-scoped transport, + `QueryVersion::GroupCut` and `RaftClass::invoke_group_query`, and reruns all gates + at one immutable source. These are required parts of the full hard cutover. + +Start from `cell_groups/checkpoint/`, `cell_groups/reclaim.rs`, existing `archive/` +and `disaster.rs`, plus `cache.rs`/`storage.rs` for group log cuts and pins. Read +`plan.py show CG-06` before creating its planned archive module/native target. +Preserve fresh quorum, permanent fencing authority and complete physical recovery; +source metadata and uploaded objects alone cannot establish those proofs. The opportunity is to reuse the existing shared consensus, SQL, read, apply and provisioning paths. Avoid adding per-cell workers or another serving mode while diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index e5fda9c48..5d1c9a54c 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -16,7 +16,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-02 | Persist complete group manifests with crash-safe cell image ownership | CG-01 | DONE | 5 | | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | -| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | IN_PROGRESS | 5 | +| CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | | CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 75628c9ad..8a9f5421a 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -5,11 +5,77 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). -CG-01 through CG-04 have passed their scoped gates; -CG-05 is IN_PROGRESS; CG-06 through CG-16 remain TODO. Intermediate contracts +CG-01 through CG-05 have passed their scoped native gates; +CG-06 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-05 complete native snapshot/rebuild gate + +CG-05 is DONE at `e0550acd95a78cdf1852a408620eb054ee505669` after the complete native gate +and a requirement-by-requirement closure audit. All five required cases pass +individually with exact one-test selection. The expanded gate passes 156 shared +execution cases, nineteen storage cases, six budget cases and nineteen legacy runtime +regressions: 205 distinct Rust cases across five executables. All eighteen +commands pass, including all-target build/Clippy, formatting/layout, dependency +isolation, docs, source preflight and 37 plan-validator tests. No required case, +oracle, command, deadline, configured resource limit or profile was weakened. + +The record +`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json` +binds 337 source hashes, five executables, frozen plan inputs, both budget +reproductions, intermediate trials and the preceding checkpoint. Independent +verification and `check_evidence` pass. Task status and the generated implementation +index now select CG-06 as the next dependency-ready task. + +| CG-05 requirement | Direct current native evidence | +| --- | --- | +| One complete same-cut root, membership/configuration and all cells, while later writes continue | `snapshot_contains_every_cell_at_cut`: both image forms, exact selected closure, concurrent cell advances and stale/foreign/corrupt refusal. | +| Verify all receiver dependencies before atomic installation; interrupted state never serves | `partial_snapshot_never_serves`: actual TLS/engine receivers, physical byte verification, missing/corrupt/foreign/budget refusal, revocation/sync cuts, owned seed SIGKILL and fresh authenticated same-disk retry. | +| Whole-group learner, joint membership, handoff and disk replacement preserve permanent claims/quorum | `whole_group_learner_and_joint_change`: complete learner install/catch-up before promotion, explicit/implicit joint changes, leadership handoff, deleted-voter replacement, joint restart and minority refusal. | +| Empty/populated cells and retained checkpoint/current suffix survive physical recovery | `snapshot_empty_cell_and_extent_restart`: Raft-generated TLS catch-up, both representations, independent sequence-1 cells, cold restart and missing-closure refusal. | +| Bounded transfer, live pins, cancellation, joined owners and unrelated-group progress | `snapshot_memory_and_pin_bounds`: 32-cell streaming/scratch pressure, old-reader retention/deletion, shared-store orphan reconciliation, held publication/install shutdown, full SQL apply capacity, eight-slot completion saturation and two concurrent writers through one checkpoint slot. | + +The final implementation fixes two admission faults. A partial refund used to +wake only the first registered waiter, even when its larger request still could +not fit. Release/shrink now wake registered waiters and retain a stored signal for +late worker registration. Separate item and byte atomics could also expose a +temporary item charge from a byte-rejected attempt. Both dimensions now reserve +and refund in one short mutex scope; notifications run after unlocking. Four new +regressions plus the existing two accounting cases pass. The original two wake +failures and subsequent concurrent-reservation failure retain exact source, +output and executable hashes. The notification-only probe passes the former +cases before demonstrating the second fault; the final Cargo gate validates both +fixes together. + +`CellGroupRuntime::usage` exposes advisory lane counters without request admission. +`checkpoint_before_apply` exposes the controller's existing exact-ticket publication +boundary for embedding owners. Eight real fsynced apply completions fill its fixed +admission; a ninth checkpoint retains its source while waiting. Admission refunds +before the actual storage fence, while the source remains charged. Releasing that +fence selects the complete checkpoint; the successor then applies. Both image +forms run directly in the resource acceptance case on explicitly owned, joined +current-thread runtimes. + +Two independent writers additionally enroll 32 real cells per group. While one +receiver holds node-zero capture capacity, both groups continue applying. After +release, both have newer checkpoints at the 25-entry cut, before either writer +finishes its remaining seven entries. All three nodes recover the selected current +and checkpoint closures and every independent sequence-1 cell. This is a bounded +native progress trace, not a measured starvation or latency bound for arbitrary +traffic. + +**Reflection and next step:** the snapshot/rebuild task's declared native +requirements now have implementation and current-source evidence. The next +critical feature task is CG-06: complete group S3 checkpoint plus contiguous +cell-addressed committed tail, independently verified coverage, safe local log +reclamation and fenced unmanaged seeding under fresh incarnations. CG-08 effects/ +cache lifetime and CG-07 SDK/fleet cutover follow. Entry-count policy retains its +idle final tail; idle/time/byte policy and coordinated wake remain future work. +Real RustFS, measured load, remaining crash/deployment schedules, independent +failure domains, managed zero-quorum recovery and removal of the old architecture +remain required in their later task/release gates. CG-06 through CG-16 remain TODO. + ## CG-05 checkpoint progress while SQL apply capacity is full Qualified source: `bad0387cb025fbba62f37aa42ba80b81d75366dd`. The production controller now diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 6f1c33e46..15ae60af1 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -406,7 +406,7 @@ starting a successor apply when the configured entry interval has elapsed. It uses the transport's already configured pool and owns initial checkpoint endpoint registrations. One node slot spans blocking capture, admission and durable publication; each selected group retains its exact successor ticket. The current -root therefore cannot advance during capture. A private completion-lane request +root therefore cannot advance during capture. The reserved completion-lane request checks that ticket and the current root, and can publish despite ordinary request or apply pressure. Pending Raft work may still append a suffix. Only the matching `CheckpointPublished` event releases the successor to physical apply; an admission @@ -420,6 +420,21 @@ Native tests hold actual image-directory persistence in both image forms, publis an unrelated complete checkpoint, retain its unapplied suffix, then release and join the original physical apply. +`CellGroupRuntime::checkpoint_before_apply` exposes this same exact-ticket +boundary for embedding owners. It waits for completion admission while retaining +the captured source; owner/engine validation still binds the current root and +ticket. Cancelling an unadmitted request queues no work. Admitted requests remain +runtime-owned through physical completion and joined shutdown. `usage()` reports +advisory ordinary/completion/feedback/read-pin charges and immutable limits without +request admission. Completion charges refund at owner admission, while physical +source/root/storage ownership continues. Gauge observations grant no authority. + +The shared budget reserves and refunds byte/item dimensions in one short lock +scope, preventing rejected byte attempts from exposing temporary item charges. +Release/shrink notify all registered request sizes and retain a stored signal for +workers that register after checking capacity. Native budget tests cover mixed- +size waiters, late registration and concurrent rejected/fitting attempts. + Pool contention or transient admission overload releases the successor and backs off until a later apply boundary using the controller's node timer. There is no per-cell task or unbounded retry loop. Capture jobs are joined before releasing the @@ -433,13 +448,18 @@ and checkpoint-source admission. Cancelled close and wait retain both owners and the engine/SQL claims through physical completion. The healthy two-voter quorum continues another group's writes during the stalled node's engine sync. Release joins shutdown, selects the captured checkpoint and permits a replacement runtime. -Both image forms run this cut; other controller crash/budget schedules remain open. - -This is an entry-count policy for active groups. It retains the final idle tail, -and time/byte triggers, fairness under continuously saturated checkpoint capacity, -controller checkpoint crash cuts and complete root-pressure qualification -remain open. It supplies no S3 archive coverage proof. Manual publication remains -available to lower-level owners; this policy does not introduce a second serving +Both image forms run this cut. Further native cases fill all eight completion +slots with real completed images, retain a ninth checkpoint through admission and +a held engine sync, then complete its exact successor. Two concurrent 32-entry +writers compete for one shared source slot; both advance checkpoints before the +writers finish, and all selected current/checkpoint closures recover on every node. +These are bounded functional traces; broader deployment/fault/load schedules remain +in the later qualification gates. + +This is an entry-count policy for active groups. It retains the final idle tail; +time/byte triggers and coordinated hibernation require their later explicit cut +reservation/wake contracts. It supplies no S3 archive coverage proof. Manual +publication remains available to lower-level owners; this policy does not introduce a second serving runtime or weaken exact-cut publication checks. An archive publishes a complete checkpoint plus a contiguous committed tail to K. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 35311e6d9..8f9a347f2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1558,7 +1558,7 @@ { "id": "CG-05", "title": "Snapshot and rebuild complete groups through membership changes", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -1688,7 +1688,9 @@ "I08", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From 650fc63892eab59e6fd5023fa90d5f361a8c7302 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 03:58:06 -0700 Subject: [PATCH 163/568] feat: archive complete cell-group checkpoints --- .../raft/cell_groups/archive/checkpoint.rs | 289 +++++++++ .../raft/cell_groups/archive/mod.rs | 241 +++++++ .../raft/cell_groups/archive/objects.rs | 127 ++++ .../raft/cell_groups/checkpoint/wire.rs | 2 +- .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../tests/native_cell_groups_archive.rs | 603 ++++++++++++++++++ .../tests/support/cell_group_archive_store.rs | 115 ++++ .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 2 +- .../multi-raft/cell-groups/protocols.md | 35 + docs/design/multi-raft/cell-groups/tasks.json | 9 +- 11 files changed, 1424 insertions(+), 6 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/objects.rs create mode 100644 crates/cellar/tests/native_cell_groups_archive.rs create mode 100644 crates/cellar/tests/support/cell_group_archive_store.rs diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs new file mode 100644 index 000000000..c4e3133d7 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -0,0 +1,289 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::{objects::Object, *}; +use std::io::Write; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Index { + domain: String, + marker: GroupManifest, + bytes: u64, + chunks: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Chunk { + part: wire::Part, + offset: u64, + length: u64, + object: Object, +} + +impl Index { + fn validate(&self, archive: &CellGroupArchive) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.domain != DOMAIN || self.marker.route().history != archive.history { + return Err(CellGroupArchiveError::Invalid("checkpoint domain/history")); + } + if self.chunks.is_empty() + || self.chunks.len() > archive.limits.max_chunks + || self.bytes > archive.limits.max_checkpoint_bytes + { + return Err(CellGroupArchiveError::Limit); + } + let mut total = 0u64; + let mut previous: Option<&Chunk> = None; + for chunk in &self.chunks { + if chunk.object.length == 0 + || chunk.object.length > CHUNK_BYTES as u64 + || chunk.length == 0 + || chunk.length > archive.limits.max_checkpoint_bytes + { + return Err(CellGroupArchiveError::Limit); + } + chunk + .offset + .checked_add(chunk.object.length) + .filter(|n| *n <= chunk.length) + .ok_or(CellGroupArchiveError::Invalid("chunk bounds"))?; + if let Some(before) = previous { + let before_end = before.offset + before.object.length; + if before_end < before.length { + if chunk.part != before.part + || chunk.length != before.length + || chunk.offset != before_end + { + return Err(CellGroupArchiveError::Invalid("chunk continuity")); + } + } else if chunk.offset != 0 { + return Err(CellGroupArchiveError::Invalid("part offset")); + } + } else if chunk.part != wire::Part::Header || chunk.offset != 0 { + return Err(CellGroupArchiveError::Invalid("first archive part")); + } + total = total + .checked_add(chunk.object.length) + .ok_or(CellGroupArchiveError::Limit)?; + previous = Some(chunk); + } + let last = previous.ok_or(CellGroupArchiveError::Invalid("empty archive"))?; + if total != self.bytes || last.offset + last.object.length != last.length { + return Err(CellGroupArchiveError::Invalid("incomplete archive index")); + } + Ok(()) + } +} + +struct Bounded { + bytes: Vec, + limit: usize, +} + +impl Write for Bounded { + fn write(&mut self, bytes: &[u8]) -> std::io::Result { + if self + .bytes + .len() + .checked_add(bytes.len()) + .is_none_or(|n| n > self.limit) + { + return Err(std::io::Error::other("archive index limit")); + } + self.bytes.extend_from_slice(bytes); + Ok(bytes.len()) + } + + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } +} + +impl CellGroupArchive { + /// Capture is supplied by the caller's node checkpoint pool. The bounded + /// producer keeps the full source pinned through physical reads. Losing the + /// async caller closes its channel; domain shutdown joins the physical owner. + /// Success confirms immutable bytes, without granting group coverage. + pub async fn upload_checkpoint( + &self, + domain: &ExecutionDomain, + source: CellGroupCheckpoint, + ) -> Result { + domain + .run(self.upload(domain, source)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn upload( + &self, + domain: &ExecutionDomain, + mut source: CellGroupCheckpoint, + ) -> Result { + if source.marker().route().history != self.history { + return Err(CellGroupArchiveError::Invalid("source history")); + } + let permit = self.admit()?; + let retained = permit.clone(); + let mut index = Index { + domain: DOMAIN.into(), + marker: source.marker().clone(), + bytes: 0, + chunks: Vec::new(), + }; + let (sender, mut receiver) = tokio::sync::mpsc::channel(1); + let producer = domain + .blocking(move || { + let _retained = retained; + source.stream(|chunk| { + if chunk.bytes.len() > CHUNK_BYTES { + return Err(CellGroupCheckpointError::Limit); + } + sender.blocking_send(wire::Chunk::copy(chunk)).map_err(|_| { + CellGroupCheckpointError::Sink(std::io::Error::other("archive stopped")) + }) + }) + }) + .map_err(|_| CellGroupArchiveError::Stopped)?; + let upload = async { + while let Some(chunk) = receiver.recv().await { + index.bytes = index + .bytes + .checked_add(chunk.bytes.len() as u64) + .filter(|n| *n <= self.limits.max_checkpoint_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + if index.chunks.len() >= self.limits.max_chunks { + return Err(CellGroupArchiveError::Limit); + } + let object = self.put("chunks", chunk.bytes, CHUNK_BYTES).await?; + index.chunks.push(Chunk { + part: chunk.part, + offset: chunk.offset, + length: chunk.length, + object, + }); + } + Ok(()) + } + .await; + drop(receiver); // Release a blocked producer before joining any error. + let produced = producer.await.map_err(CellGroupArchiveError::Worker)?; + upload?; + produced?; + let archive = self.clone(); + let retained = permit.clone(); + let bytes = domain + .blocking(move || { + let _retained = retained; + index.validate(&archive)?; + let mut output = Bounded { + bytes: Vec::new(), + limit: archive.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &index) + .map_err(|_| CellGroupArchiveError::Limit)?; + Ok::<_, CellGroupArchiveError>(output.bytes) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let object = self + .put("indexes", bytes, self.limits.max_index_bytes) + .await?; + drop(permit); + Ok(CellGroupArchiveReference { + digest: object.digest, + length: object.length, + }) + } + + /// Independently GET every object into the ordinary physical receiver. No + /// source pin or upload receipt substitutes for dependency verification. + /// A missing/corrupt cell refuses the whole cut. Ordinary errors join the + /// receiver; cancellation retains its permits until physical work finishes. + pub async fn verify_checkpoint( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveReference, + receiver: CellGroupCheckpointReceiver, + ) -> Result { + domain + .run(self.verify(domain, reference, receiver)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn verify( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveReference, + mut receiver: CellGroupCheckpointReceiver, + ) -> Result { + let permit = self.admit()?; + let bytes = self + .get( + "indexes", + &Object { + digest: reference.digest, + length: reference.length, + }, + self.limits.max_index_bytes, + ) + .await?; + let archive = self.clone(); + let retained = permit.clone(); + let index: Index = domain + .blocking(move || { + let _retained = retained; + let index: Index = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("checkpoint index"))?; + index.validate(&archive)?; + Ok::<_, CellGroupArchiveError>(index) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let marker = index.marker; + let (sender, mut incoming) = tokio::sync::mpsc::channel::(1); + let retained = permit.clone(); + let consumer = domain + .blocking(move || { + let _retained = retained; + while let Some(chunk) = incoming.blocking_recv() { + chunk.append(&mut receiver)?; + } + receiver.finish() + }) + .map_err(|_| CellGroupArchiveError::Stopped)?; + let download: Result<()> = async { + for chunk in index.chunks { + let bytes = self.get("chunks", &chunk.object, CHUNK_BYTES).await?; + sender + .send(wire::Chunk { + part: chunk.part, + offset: chunk.offset, + length: chunk.length, + bytes, + }) + .await + .map_err(|_| CellGroupArchiveError::Invalid("receiver refused archive"))?; + } + Ok(()) + } + .await; + drop(sender); + let received = consumer.await.map_err(CellGroupArchiveError::Worker)?; + download?; + let checkpoint = received?; + if checkpoint.marker() != &marker { + return Err(CellGroupArchiveError::Invalid("archive checkpoint binding")); + } + Ok(VerifiedCellGroupArchiveCheckpoint { + reference: reference.clone(), + checkpoint, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs new file mode 100644 index 000000000..1d44b4fb9 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -0,0 +1,241 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete group checkpoints in immutable object storage. Upload confirmation +//! and independently recovered physical state are separate capabilities. Neither +//! authorizes S3 settlement, log truncation, membership or disaster recovery. + +use super::{ + checkpoint::wire, CellGroupCheckpoint, CellGroupCheckpointError, CellGroupCheckpointReceiver, + GroupManifest, PreparedCellGroupCheckpoint, +}; +use crate::{ + asyncrt::ExecutionDomain, + raft::budget::{Budget, Permit}, +}; +use cellar_logic::cell_group::GroupHistory; +use object_store::{path::Path, ObjectStore}; +use serde::{Deserialize, Serialize}; +use sha2::{Digest, Sha256}; +use std::{fmt, sync::Arc}; + +mod checkpoint; +mod objects; + +const DOMAIN: &str = "cellar-cell-group-archive-checkpoint-1"; +const CHUNK_BYTES: usize = 64 * 1024; + +#[derive(Debug)] +pub enum CellGroupArchiveError { + Invalid(&'static str), + Limit, + Busy, + Stopped, + Timeout, + Checkpoint(CellGroupCheckpointError), + Storage(object_store::Error), + Worker(crate::asyncrt::TaskPanic), +} + +impl fmt::Display for CellGroupArchiveError { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "cell-group archive: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Checkpoint(e) => Some(e), + Self::Storage(e) => Some(e), + Self::Worker(e) => Some(e), + _ => None, + } + } +} + +impl From for CellGroupArchiveError { + fn from(error: CellGroupCheckpointError) -> Self { + Self::Checkpoint(error) + } +} + +type Result = std::result::Result; + +/// Share one archive pool across this node's group jobs. Clones share admission. +/// Checkpoint source/receiver pools separately account physical closures/scratch. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_index_bytes: usize, + pub max_chunks: usize, + pub max_checkpoint_bytes: u64, + /// Deadline for a complete individual PUT or GET, including its body stream. + pub io_timeout_ms: u64, +} + +impl Default for CellGroupArchiveLimits { + fn default() -> Self { + Self { + max_operations: 2, + max_retained_bytes: 32 * 1024 * 1024, + max_index_bytes: 1024 * 1024, + max_chunks: 4096, + max_checkpoint_bytes: 256 * 1024 * 1024, + io_timeout_ms: 10_000, + } + } +} + +impl CellGroupArchiveLimits { + fn workspace(self) -> Result { + if !(1..=4096).contains(&self.max_operations) + || !(1024..=8 * 1024 * 1024).contains(&self.max_index_bytes) + || !(1..=65_536).contains(&self.max_chunks) + || !(64 * 1024..=1024 * 1024 * 1024).contains(&self.max_checkpoint_bytes) + || !(1..=60_000).contains(&self.io_timeout_ms) + { + return Err(CellGroupArchiveError::Limit); + } + // Decoded records, bounded encoded index, GET/PUT windows and transport + // copies. The physical source or receiver has its own pool reservation. + self.max_index_bytes + .checked_mul(6) + .and_then(|n| n.checked_add(self.max_chunks.checked_mul(2048)?)) + .and_then(|n| n.checked_add(8 * CHUNK_BYTES)) + .ok_or(CellGroupArchiveError::Limit) + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct CellGroupArchiveUsage { + pub operations: usize, + pub retained_bytes: usize, +} + +/// An untrusted locator, suitable for persistence by a later archive-head owner. +/// Its digest authenticates the index bytes, not their complete recovery closure. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveReference { + digest: [u8; 32], + length: u64, +} + +impl CellGroupArchiveReference { + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn length(&self) -> u64 { + self.length + } +} + +/// Every referenced object was downloaded, hashed and physically recovered on +/// the destination. The contained checkpoint retains receiver-local pins. This +/// is a checkpoint-only proof, without a tail, published head or remote GC lease. +pub struct VerifiedCellGroupArchiveCheckpoint { + reference: CellGroupArchiveReference, + checkpoint: PreparedCellGroupCheckpoint, +} + +impl VerifiedCellGroupArchiveCheckpoint { + pub fn reference(&self) -> &CellGroupArchiveReference { + &self.reference + } + + pub fn checkpoint(&self) -> &PreparedCellGroupCheckpoint { + &self.checkpoint + } + + pub fn into_checkpoint(self) -> PreparedCellGroupCheckpoint { + self.checkpoint + } +} + +/// Group-owned archive namespace. No synthetic cell identity is involved. +#[derive(Clone)] +pub struct CellGroupArchive { + store: Arc, + prefix: Path, + history: GroupHistory, + limits: CellGroupArchiveLimits, + budget: Arc, + workspace: usize, +} + +/// Node-wide archive admission, shared by every group binding and both transfer +/// directions. Holding a group handle consumes no operation slot or I/O buffer. +#[derive(Clone)] +pub struct CellGroupArchivePool { + limits: CellGroupArchiveLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchivePool { + pub fn new(limits: CellGroupArchiveLimits) -> Result { + let workspace = limits.workspace()?; + if limits.max_retained_bytes < workspace { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn for_group( + &self, + store: Arc, + prefix: Path, + history: GroupHistory, + ) -> Result { + if prefix.as_ref().len() > 4096 { + return Err(CellGroupArchiveError::Limit); + } + let mut binding = Sha256::new(); + binding.update(b"cellar-cell-group-archive-history-1\0"); + binding.update(history.cluster.bytes()); + binding.update(history.namespace.bytes()); + binding.update(history.group.bytes()); + binding.update(history.incarnation.bytes()); + let prefix = prefix.child(DOMAIN).child(hex::encode(binding.finalize())); + Ok(CellGroupArchive { + store, + prefix, + history, + limits: self.limits, + budget: self.budget.clone(), + workspace: self.workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } +} + +impl CellGroupArchive { + /// Advisory node-wide counters shared with every binding from this pool. + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } + + fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/objects.rs b/crates/cellar/durability/raft/cell_groups/archive/objects.rs new file mode 100644 index 000000000..57c4f1075 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/objects.rs @@ -0,0 +1,127 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use futures_util::TryStreamExt; +use object_store::{PutMode, PutOptions}; +use std::time::Duration; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Object { + pub digest: [u8; 32], + pub length: u64, +} + +impl Object { + pub fn of(bytes: &[u8]) -> Self { + Self { + digest: Sha256::digest(bytes).into(), + length: bytes.len() as u64, + } + } + + pub fn verify(&self, bytes: &[u8]) -> Result<()> { + if bytes.len() as u64 != self.length + || <[u8; 32]>::from(Sha256::digest(bytes)) != self.digest + { + return Err(CellGroupArchiveError::Invalid("object digest/length")); + } + Ok(()) + } +} + +impl CellGroupArchive { + fn object_path(&self, kind: &str, object: &Object) -> Path { + self.prefix + .child(kind) + .child(format!("{}-{}", object.length, hex::encode(object.digest))) + } + + async fn io(&self, future: impl std::future::Future>) -> Result { + crate::asyncrt::timeout(Duration::from_millis(self.limits.io_timeout_ms), future) + .await + .map_err(|_| CellGroupArchiveError::Timeout)? + } + + // One deadline includes headers, every body chunk and EOF. Never allocate + // from unbounded remote metadata or accept a partial/ranged object response. + async fn read(&self, kind: &str, object: &Object, limit: usize) -> Result>> { + if object.length == 0 || object.length > limit as u64 { + return Err(CellGroupArchiveError::Limit); + } + let path = self.object_path(kind, object); + self.io(async { + let result = match self.store.get(&path).await { + Ok(result) => result, + Err(object_store::Error::NotFound { .. }) => return Ok(None), + Err(error) => return Err(CellGroupArchiveError::Storage(error)), + }; + if result.meta.size != object.length + || result.range.start != 0 + || result.range.end != object.length + { + return Err(CellGroupArchiveError::Invalid( + "object response length/range", + )); + } + let mut bytes = Vec::with_capacity(object.length as usize); + let mut stream = result.into_stream(); + let mut chunks = 0; + while let Some(chunk) = stream + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + { + chunks += 1; + if chunks > 16_384 + || bytes + .len() + .checked_add(chunk.len()) + .is_none_or(|n| n > object.length as usize) + { + return Err(CellGroupArchiveError::Limit); + } + bytes.extend_from_slice(&chunk); + } + object.verify(&bytes)?; + Ok(Some(bytes)) + }) + .await + } + + pub(super) async fn get(&self, kind: &str, object: &Object, limit: usize) -> Result> { + self.read(kind, object, limit) + .await? + .ok_or(CellGroupArchiveError::Invalid("missing archive object")) + } + + pub(super) async fn put(&self, kind: &str, bytes: Vec, limit: usize) -> Result { + if bytes.is_empty() || bytes.len() > limit { + return Err(CellGroupArchiveError::Limit); + } + let object = Object::of(&bytes); + // Reuse verified immutable objects without another PUT. A corrupt + // existing object refuses; overwriting could erase evidence of damage. + if self.read(kind, &object, limit).await?.is_some() { + return Ok(object); + } + let path = self.object_path(kind, &object); + let put = self + .io(async { + self.store + .put_opts(&path, bytes.into(), PutOptions::from(PutMode::Create)) + .await + .map_err(CellGroupArchiveError::Storage) + }) + .await; + // Lost/failed PUT responses are ambiguous. Only an exact GET confirms + // publication, including a concurrent conditional-create winner. + match self.read(kind, &object, limit).await? { + Some(_) => Ok(object), + None => match put { + Err(error) => Err(error), + Ok(_) => Err(CellGroupArchiveError::Invalid("unconfirmed archive object")), + }, + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs index 70693953b..296da0963 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs @@ -5,7 +5,7 @@ use super::*; use serde::{Deserialize, Serialize}; -#[derive(Serialize, Deserialize)] +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(tag = "kind", deny_unknown_fields)] pub(crate) enum Part { Header, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 7212a96e0..040e04212 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -15,6 +15,11 @@ pub use crate::raft::transport::{ }; mod apply; +mod archive; +pub use archive::{ + CellGroupArchive, CellGroupArchiveError, CellGroupArchiveLimits, CellGroupArchivePool, + CellGroupArchiveReference, CellGroupArchiveUsage, VerifiedCellGroupArchiveCheckpoint, +}; pub(crate) mod checkpoint; pub use checkpoint::{ CellGroupCheckpoint, CellGroupCheckpointChunk, CellGroupCheckpointError, diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs new file mode 100644 index 000000000..18b1c7e86 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -0,0 +1,603 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! CG-06 checkpoint archival slice. No tail, settlement, log reclamation or +//! fresh-incarnation disaster-recovery qualification is claimed by these cases. +#![allow(clippy::disallowed_methods)] +// The shared engine fixture also defines fault/membership helpers selected by +// sibling integration targets. They remain unused in this checkpoint-only gate. +#![allow(dead_code, unused_imports)] + +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; + +include!("support/cell_group_engine_fixture.rs"); + +use cellar_ltx::{ + generation::ReadServices, + internal::{CacheLimits, EXTENT_BYTES}, + stream_snapshot::{ScratchLimits, ScratchPool}, +}; +use futures_util::TryStreamExt; +use object_store::{path::Path, ObjectStore}; +use std::sync::atomic::Ordering; + +fn destination(cluster: &mut Cluster) -> Result { + let voter = &mut cluster.voters[2]; + drop(voter.owner.take()); + drop(voter.held.take()); + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = match voter.extents.clone() { + Some(images) => CellGroupApplyImages::Extents { + images, + services, + limits: Default::default(), + cache_directory: cluster._root.path().join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }, + None => CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services, + }, + }; + Ok(CellGroupApplier::new( + voter.log.cell_group(&cluster.birth.route().history)?, + voter.pages.take().context("destination pages")?, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) +} + +fn receiver( + pool: &CellGroupCheckpointPool, + applier: &CellGroupApplier, + cluster: &Cluster, +) -> Result { + Ok(pool.receive( + applier.clone(), + ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cluster._root.path().into(), + )?) +} + +fn source(pool: &CellGroupCheckpointPool, cluster: &Cluster) -> Result { + Ok(pool.capture( + &cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + )?) +} + +fn new_archive( + store: Arc, + prefix: Path, + history: GroupHistory, + limits: CellGroupArchiveLimits, +) -> Result { + Ok(CellGroupArchivePool::new(limits)?.for_group(store, prefix, history)?) +} + +async fn index( + store: &dyn ObjectStore, + reference: &CellGroupArchiveReference, +) -> Result<(Path, serde_json::Value)> { + let suffix = format!( + "/indexes/{}-{}", + reference.length(), + hex::encode(reference.digest()) + ); + let objects: Vec<_> = store.list(None).try_collect().await?; + let path = objects + .into_iter() + .find(|m| m.location.as_ref().ends_with(&suffix)) + .context("checkpoint index object")? + .location; + let bytes = store.get(&path).await?.bytes().await?; + Ok((path, serde_json::from_slice(&bytes)?)) +} + +fn cell_objects(index_path: &Path, index: &serde_json::Value, cell: &str) -> Vec { + let prefix = index_path.as_ref().split("/indexes/").next().unwrap(); + index["chunks"] + .as_array() + .unwrap() + .iter() + .filter(|c| c["part"]["cell"] == cell) + .map(|chunk| { + let digest: Vec = + serde_json::from_value(chunk["object"]["digest"].clone()).unwrap(); + Path::from(format!( + "{prefix}/chunks/{}-{}", + chunk["object"]["length"], + hex::encode(digest) + )) + }) + .collect() +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_roundtrip_and_reuse() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("qualification"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + store.lose_put_response.store(true, Ordering::Release); + let original = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let count = store.puts.load(Ordering::Acquire); + let repeat = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + assert_eq!(repeat, original); + assert_eq!( + store.puts.load(Ordering::Acquire), + count, + "unchanged cut must not PUT again" + ); + let (before_path, before_index) = index(store.as_ref(), &original).await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let latest = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (after_path, after_index) = index(store.as_ref(), &latest).await?; + for cell in ["beta", "cell-000"] { + let objects = cell_objects(&before_path, &before_index, cell); + assert!(!objects.is_empty()); + assert_eq!(objects, cell_objects(&after_path, &after_index, cell)); + } + assert_ne!(original, latest); + let marker = cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + let applier = destination(&mut cluster)?; + let receive_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // Prove the independent receiver uses remote data after its source engine + // has closed; no borrowed source pin can satisfy this recovery. + drop(cluster.voters[0].owner.take()); + cluster.voters[0].log.close().await?; + let verified = archive + .verify_checkpoint( + &cluster.domain, + &latest, + receiver(&receive_pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(verified.reference(), &latest); + assert_eq!(verified.checkpoint().marker(), &marker); + assert_eq!(archive.usage().operations, 0); + assert_eq!(sources.usage().sources, 0); + assert_eq!(receive_pool.usage().sources, 1); + for (cell, wanted) in expected { + let (_, pin) = verified + .checkpoint() + .manifest() + .pin_cell(&CellId::new(cell)?) + .context("recovered cell")?; + let bytes = cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes + .extend(pin.view().read_extent( + pin.manifest().extent(n).context("extent")?, + )?); + } + bytes + } + }) + })? + .await??; + assert_eq!(bytes, wanted); + } + drop(verified); + assert_eq!(receive_pool.usage().sources, 0); + cluster.close().await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let history = cluster.birth.route().history; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + // Chunk and encoded-index bounds are independent, and neither may leave + // a selected index or a blocked physical producer behind on failure. + for limits in [ + CellGroupArchiveLimits { + max_chunks: 1, + ..Default::default() + }, + CellGroupArchiveLimits { + max_index_bytes: 1024, + ..Default::default() + }, + ] { + let limited = new_archive(store.clone(), Path::from("limited"), history, limits)?; + assert!(matches!( + limited + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await, + Err(CellGroupArchiveError::Limit) + )); + assert_eq!(limited.usage().operations, 0); + assert_eq!(sources.usage().sources, 0); + let objects: Vec<_> = store + .list(Some(&Path::from("limited"))) + .try_collect() + .await?; + assert!(!objects + .iter() + .any(|m| m.location.as_ref().contains("/indexes/"))); + } + let node_pool = CellGroupArchivePool::new(CellGroupArchiveLimits { + max_operations: 1, + ..Default::default() + })?; + let archive = node_pool.for_group(store.clone(), Path::from("qualification"), history)?; + let other = node_pool.for_group( + store.clone(), + Path::from("qualification"), + GroupHistory { + group: GroupId::new([81; 16])?, + ..history + }, + )?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, encoded) = index(store.as_ref(), &reference).await?; + let dependency = cell_objects(&path, &encoded, "beta") + .into_iter() + .last() + .context("held beta")?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let gate = store.hold(dependency); + let mut pending = Box::pin(archive.verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + )); + assert!( + tokio::time::timeout(Duration::from_millis(50), &mut pending) + .await + .is_err() + ); + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(pool.usage().sources, 1); + assert_eq!(archive.usage().operations, 1); + let charged = archive.usage().retained_bytes; + assert!(charged > 0 && charged <= CellGroupArchiveLimits::default().max_retained_bytes); + // Node admission is shared by clones and both directions. Refusal must + // drop the rejected source without borrowing the active job's charge. + assert!(matches!( + archive + .clone() + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await, + Err(CellGroupArchiveError::Busy) + )); + assert_eq!(archive.usage().retained_bytes, charged); + assert_eq!(node_pool.usage().operations, 1); + let other_receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + assert!(matches!( + other + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&other_receivers, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Busy) + )); + assert_eq!(other_receivers.usage().sources, 0); + assert_eq!(other.usage().retained_bytes, charged); + assert_eq!(sources.usage().sources, 0); + drop(pending); + tokio::time::timeout(Duration::from_secs(5), async { + while archive.usage().operations != 0 || pool.usage().sources != 0 { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await?; + store.clear(); + gate.release.notify_one(); + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + drop(verified); + // A blocked GET is bounded by the configured complete-object deadline. + let gate = store.hold(path); + let timed = new_archive( + store.clone(), + Path::from("qualification"), + history, + CellGroupArchiveLimits { + io_timeout_ms: 20, + ..Default::default() + }, + )?; + assert!(matches!( + timed + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Timeout) + )); + assert_eq!(timed.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + store.clear(); + gate.release.notify_one(); + cluster.close().await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_missing_or_corrupt_cell_refuses() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("qualification"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, encoded) = index(store.as_ref(), &reference).await?; + let dependency = cell_objects(&path, &encoded, "beta") + .into_iter() + .last() + .context("beta dependency")?; + let bytes = store.get(&dependency).await?.bytes().await?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + store.delete(&dependency).await?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("missing archive object")) + )); + assert_eq!(pool.usage().sources, 0); + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 0x80; + store.inner.put(&dependency, corrupt.into()).await?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("object digest/length")) + )); + assert_eq!(pool.usage().sources, 0); + // Existing corrupt immutable payloads cannot be silently overwritten. + assert!(archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await + .is_err()); + store.inner.put(&dependency, bytes.into()).await?; + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(verified.checkpoint().marker().root().cells(), 4); + drop(verified); + assert_eq!(archive.usage().retained_bytes, 0); + cluster.close().await?; + } + Ok(()) +} + +async fn forged_index( + store: &dyn ObjectStore, + original_path: &Path, + value: &serde_json::Value, +) -> Result { + let bytes = serde_json::to_vec(value)?; + let digest: [u8; 32] = Sha256::digest(&bytes).into(); + let reference = serde_json::from_value(serde_json::json!({ + "digest": digest, "length": bytes.len(), + }))?; + let prefix = original_path + .as_ref() + .split("/indexes/") + .next() + .context("index prefix")?; + let path = Path::from(format!( + "{prefix}/indexes/{}-{}", + bytes.len(), + hex::encode(digest) + )); + store.put(&path, bytes.into()).await?; + Ok(reference) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { + for rooted in [false, true] { + let (mut cluster, _) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("qualification"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let reference = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let (path, original) = index(store.as_ref(), &reference).await?; + let group = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let base = group.group_manifest()?.context("destination base")?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for attack in 0..4 { + let mut value = original.clone(); + match attack { + 0 => value["domain"] = "cellar-single-cell-archive-1".into(), + 1 => value["chunks"][1]["offset"] = 1.into(), + 2 => { + let chunks = value["chunks"].as_array_mut().unwrap(); + chunks.retain(|chunk| chunk["part"]["cell"] != "beta"); + value["bytes"] = chunks + .iter() + .map(|c| c["object"]["length"].as_u64().unwrap()) + .sum::() + .into(); + } + 3 => value["marker"] = serde_json::to_value(&base)?, + _ => unreachable!(), + } + let forged = forged_index(store.as_ref(), &path, &value).await?; + let result = archive + .verify_checkpoint( + &cluster.domain, + &forged, + receiver(&pool, &applier, &cluster)?, + ) + .await; + match attack { + 0 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("checkpoint domain/history")) + )), + 1 => assert!(matches!(result, Err(CellGroupArchiveError::Invalid(_)))), + 2 => assert!(matches!( + result, + Err(CellGroupArchiveError::Checkpoint(_)) + | Err(CellGroupArchiveError::Invalid("receiver refused archive")) + )), + 3 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("archive checkpoint binding")) + )), + _ => unreachable!(), + } + assert_eq!(archive.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + assert_eq!(group.group_manifest()?.as_ref(), Some(&base)); + assert_eq!(group.checkpoint_manifest()?, None); + } + for length in [0, u64::MAX] { + let invalid = serde_json::from_value( + serde_json::json!({"digest": reference.digest(), "length": length}), + )?; + assert!(matches!( + archive + .verify_checkpoint( + &cluster.domain, + &invalid, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Limit) + )); + } + let verified = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_ne!(verified.checkpoint().marker(), &base); + assert_eq!(group.group_manifest()?.as_ref(), Some(&base)); + assert_eq!(group.checkpoint_manifest()?, None); + drop(verified); + cluster.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs new file mode 100644 index 000000000..7f7cdcd6b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -0,0 +1,115 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Instrument the actual in-memory object-store adapter, without replacing its +//! conditional-create semantics, bytes or physical checkpoint verification. +use futures_util::stream::BoxStream; +use object_store::{path::Path, *}; +use std::sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Arc, Mutex, +}; +use tokio::sync::Notify; + +#[derive(Debug, Default)] +pub struct Gate { + pub entered: Notify, + pub release: Notify, +} + +#[derive(Debug)] +pub struct Store { + pub inner: Arc, + pub puts: AtomicUsize, + pub lose_put_response: AtomicBool, + held: Mutex)>>, +} + +impl Store { + pub fn new() -> Arc { + Arc::new(Self { + inner: Arc::new(memory::InMemory::new()), + puts: AtomicUsize::new(0), + lose_put_response: AtomicBool::new(false), + held: Mutex::new(None), + }) + } + + pub fn hold(&self, path: Path) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held.lock().unwrap() = Some((path, gate.clone())); + gate + } + + pub fn clear(&self) { + self.held.lock().unwrap().take(); + } +} + +impl std::fmt::Display for Store { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("cell-group archive instrumented store") + } +} + +#[async_trait::async_trait] +impl ObjectStore for Store { + async fn put_opts( + &self, + path: &Path, + payload: PutPayload, + opts: PutOptions, + ) -> Result { + self.puts.fetch_add(1, Ordering::AcqRel); + let result = self.inner.put_opts(path, payload, opts).await?; + if self.lose_put_response.swap(false, Ordering::AcqRel) { + return Err(Error::Generic { + store: "lost PUT response", + source: "injected".into(), + }); + } + Ok(result) + } + + async fn put_multipart_opts( + &self, + path: &Path, + opts: PutMultipartOptions, + ) -> Result> { + self.inner.put_multipart_opts(path, opts).await + } + + async fn get_opts(&self, path: &Path, opts: GetOptions) -> Result { + let gate = self + .held + .lock() + .unwrap() + .as_ref() + .filter(|(held, _)| held == path) + .map(|(_, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + self.inner.get_opts(path, opts).await + } + + async fn delete(&self, path: &Path) -> Result<()> { + self.inner.delete(path).await + } + + fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { + self.inner.list(prefix) + } + + async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result { + self.inner.list_with_delimiter(prefix).await + } + + async fn copy(&self, from: &Path, to: &Path) -> Result<()> { + self.inner.copy(from, to).await + } + + async fn copy_if_not_exists(&self, from: &Path, to: &Path) -> Result<()> { + self.inner.copy_if_not_exists(from, to).await + } +} diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 5d1c9a54c..43b72ac5d 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -17,7 +17,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 | -| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | TODO | 5 | +| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | IN_PROGRESS | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | | CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | | CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | TODO | 7 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 8a9f5421a..c186ec2dd 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -6,7 +6,7 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). CG-01 through CG-05 have passed their scoped native gates; -CG-06 through CG-16 remain TODO. Intermediate contracts +CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 15ae60af1..31ae56866 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -468,6 +468,41 @@ also replay. Do not rerun commands. Publish coverage only after all dependencies are durable and independently verifiable. A dependency failure in one cell blocks group coverage. Unchanged immutable cell objects can be reused between cuts. +The initial CG-06 checkpoint archive boundary is implemented in +`raft/cell_groups/archive/`. A node owns one `CellGroupArchivePool`; group handles +and their clones share its item/byte admission across uploads and downloads. +Group history has its own archive namespace and marker-1 domain, without a +synthetic cell identity. One admitted physical producer streams a pinned complete +checkpoint through a one-chunk channel. An immutable bounded index names each +cell-addressed chunk, offset, length and SHA-256. Unchanged chunks are GET-verified +and reused without PUT. New conditional creates require exact GET confirmation, +including an ambiguous PUT response. Corrupt existing objects refuse overwrite. + +`CellGroupArchiveReference` is an untrusted persistent locator. Only +`verify_checkpoint` can produce `VerifiedCellGroupArchiveCheckpoint`: it downloads +and hashes every indexed object, checks continuity and limits, runs the ordinary +physical checkpoint receiver, and binds its complete recovered marker to the +index. Missing/corrupt cells or a substituted cut refuse the whole preparation. +The receiver does not change the engine-selected current/checkpoint root. Its +ordinary installation and Raft admission contracts remain required. The source +engine and its pins are unnecessary for this independent recovery. + +Source/receiver pools separately charge physical closures and scratch. Archive +admission charges bounded encoded/decoded metadata and I/O windows. Complete +individual PUT/GET operations, including body EOF, have domain-owned deadlines. +Ordinary failures join entered physical workers; caller cancellation closes their +bounded channels and retains permits until those workers exit. Domain shutdown +joins entered work. The pool must be shared by node orchestration when it is wired +in; constructing separate pools per group would defeat the node-wide bound. + +This checkpoint-only capability has no committed tail, conditionally published +head or remote GC lease. It grants no group S3 settlement, archive coverage, +reclamation or disaster-recovery authority. CG-06 still requires those owners, +complete administrative/cell tail replay, retention admission and fenced +fresh-incarnation seeding. Its executable scope now records the component's three +production files and the native archive target/store instrumentation, replacing +the originally planned leaf output without changing any required case or oracle. + Quorum writes may proceed during an S3 outage within explicit disk/unarchived-byte budgets. S3-mode outputs wait for confirmed coverage. Log reclamation requires the covering local checkpoint and configured archive retention proofs, plus transfer diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 8f9a347f2..4bf1059a5 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1706,7 +1706,7 @@ { "id": "CG-06", "title": "Archive and reclaim the complete group recovery history", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -1725,8 +1725,11 @@ "crates/cellar-ltx/src/" ], "creates": [ - "crates/cellar/durability/raft/cell_groups/archive.rs", - "crates/cellar/tests/native_cell_groups_archive.rs" + "crates/cellar/durability/raft/cell_groups/archive/mod.rs", + "crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs", + "crates/cellar/durability/raft/cell_groups/archive/objects.rs", + "crates/cellar/tests/native_cell_groups_archive.rs", + "crates/cellar/tests/support/cell_group_archive_store.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 9b23bce57cb7512e4dacbb74fb366b5863b35d1e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 04:17:49 -0700 Subject: [PATCH 164/568] docs: record group archive checkpoint gate --- docs/design/multi-raft/cell-groups/handoff.md | 20 ++++++++ .../design/multi-raft/cell-groups/progress.md | 48 +++++++++++++++++++ 2 files changed, 68 insertions(+) diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 9110af798..59b8979c9 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,6 +15,26 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). +The latest partial source is `650fc63892eab59e6fd5023fa90d5f361a8c7302`. +CG-06 is IN_PROGRESS: `raft/cell_groups/archive/` implements complete group +checkpoint objects, node-shared item/byte admission, immutable reuse/conditional +creation and independent physical recovery. Four exact archive cases and all +five snapshot regressions pass. All fifteen commands pass, including all-target +build/Clippy, formatting/layout, dependency/docs/preflight and 37 validator tests. +Its retained record under +`.cellar/cell-groups-evidence/CG-06-checkpoint-archive/qualification/` +binds 317 source hashes and two executables; independent verification passes. +See the current [progress](progress.md) and [archive contract](protocols.md#apply-checkpoint-and-archive). + +The checkpoint locator is untrusted; the verified receiver capability grants +neither group coverage nor S3 settlement, log deletion or fresh-incarnation +recovery. All five original CG-06 required cases and its complete task gate remain +pending. Next implement contiguous cell/control tails and a published, +GC-protected coverage owner, then bounded outage admission/S3 settlement, +archive-aware log reclamation and fenced unmanaged seeding. Shared dispatch still +refuses S3 durability. Current object framing follows bounded source chunks; +qualify coalescing and large-group bounds before performance/cost claims. + CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing commands, backed by 205 Rust cases, 337 source hashes and five executables. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index c186ec2dd..2db1b3b10 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,54 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 independently recovered object-store checkpoints + +Partial source checkpoint: `650fc63892eab59e6fd5023fa90d5f361a8c7302`. +CG-06 is IN_PROGRESS. The group-owned archive now stores complete checkpoints +without synthetic cell identities. One node pool bounds upload/download jobs +and retained metadata across all group handles. One-chunk channels connect +domain-owned physical sources/receivers to object-store I/O; source and receiver +pools separately retain closure and scratch admission through cancellation. + +An immutable bounded index binds the complete group marker and every ordered, +cell-addressed chunk. Existing objects are GET-verified and reused without PUT; +conditional creates require exact read-back even after an ambiguous PUT response. +Corrupt existing payloads refuse overwrite. Upload returns an untrusted locator. +Independent verification downloads every dependency, runs the physical receiver +and checks its complete marker against the index. Preparation leaves the selected +engine roots unchanged and grants no serving authority. + +All fifteen gate commands pass: four new archive cases selected individually, +all five existing snapshot acceptance cases, all-target build/Clippy, +formatting/layout, native dependency isolation, docs, CG-06 source preflight, +plan validation and 37 Python validator tests. Nine distinct Rust cases run across +two executables; ignored snapshot child entry points are exercised by their +process-crash parents. The source-bound record +`.cellar/cell-groups-evidence/CG-06-checkpoint-archive/qualification/checkpoint.json` +binds 317 source hashes, both executables, hashed logs, frozen plan inputs and +the preceding CG-05 checkpoint. An independent hash/source/selection audit passes. + +The new cases use real three-voter Raft Engine and SQLite/LTX with both materialized +and extent images, empty enrollment, and the actual in-memory object-store adapter +with instrumented faults. They verify unchanged-cut reuse without repeat PUTs, +recovery after source-engine closure, a missing/corrupt cell despite another +cell's later cut, ambiguous PUT confirmation, shared cross-group admission, +chunk/index bounds, cancellation, complete-GET timeout and malformed/rehashed +index refusal. Omitting a cell or substituting the advertised cut cannot produce +a verified checkpoint. The five original CG-06 required cases remain pending; +these narrower checkpoint tests do not replace them or their oracles. + +**Reflection and next step:** the critical gap is a complete checkpoint plus +contiguous committed cell/control tail and a published, GC-protected coverage +owner. Shared dispatch still refuses S3 durability. Only confirmed complete +coverage may release S3 outputs or feed the archive retention/reclamation policy. +Then implement bounded outage admission, actual shared-engine log reclamation +and fenced unmanaged seeding under fresh group/cell incarnations. Current objects +follow bounded source chunks; request coalescing and large-group limits need +qualification before any cost/throughput claim. Follow with CG-08 effects/cache +lifetime and CG-07 public SDK/fleet cutover. Real RustFS/load, managed recovery, +independent domains and final removal of the old architecture remain later gates. + ## CG-05 complete native snapshot/rebuild gate CG-05 is DONE at `e0550acd95a78cdf1852a408620eb054ee505669` after the complete native gate From 7a29c2a9bff5c2daddf37f53c50bc5d575e0b069 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 05:19:59 -0700 Subject: [PATCH 165/568] feat: replay committed cell-group archive tails --- crates/cellar-ltx/src/generation.rs | 52 ++ crates/cellar-ltx/src/generation_receive.rs | 12 +- .../raft/cell_groups/apply/images.rs | 23 +- .../durability/raft/cell_groups/apply/mod.rs | 125 +-- .../raft/cell_groups/apply/transition.rs | 133 +++ .../raft/cell_groups/archive/checkpoint.rs | 49 +- .../raft/cell_groups/archive/mod.rs | 11 +- .../raft/cell_groups/archive/replay.rs | 204 +++++ .../raft/cell_groups/archive/tail.rs | 480 +++++++++++ .../cell_groups/archive/tail_admission.rs | 89 ++ .../raft/cell_groups/archive/tail_codec.rs | 86 ++ .../durability/raft/cell_groups/membership.rs | 34 + .../cellar/durability/raft/cell_groups/mod.rs | 4 +- .../tests/native_cell_groups_archive.rs | 45 +- .../tests/support/cell_group_archive_tail.rs | 759 ++++++++++++++++++ .../tests/support/cell_group_checkpoint.rs | 15 +- .../support/cell_group_engine_fixture.rs | 10 + .../multi-raft/cell-groups/protocols.md | 47 ++ docs/design/multi-raft/cell-groups/tasks.json | 12 +- 19 files changed, 2046 insertions(+), 144 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/apply/transition.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/replay.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/tail.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs create mode 100644 crates/cellar/tests/support/cell_group_archive_tail.rs diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index 0c2d2befa..b1ef490bc 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -203,6 +203,20 @@ pub struct GenerationPin { } impl GenerationPin { + /// Independently inspect and sync this pinned destination, including when + /// its seal is already cached. Run on its joined blocking owner. This + /// verifies physical bytes; it supplies no publication or consensus right. + pub fn verify_and_sync(&self) -> Result<(), GenerationError> { + let mut file = self + .inner + .filesystem + .open(&self.inner.directory.join(final_name(self.id())))?; + apply::inspect(&mut file, &self.state(), None)?; + file.sync_all()?; + self.inner.filesystem.sync_all(&self.inner.directory)?; + Ok(()) + } + pub fn id(&self) -> ImageId { self.image.id } @@ -653,6 +667,44 @@ impl GenerationStore { self.prepare_chain(id, base, std::slice::from_ref(update)) } + /// Apply and verify the supplied delta even if the destination already + /// exists. A cached result cannot substitute for replay of untrusted LTX. + /// Existing images use an independently admitted anonymous scratch image; + /// no selected image or inventory retention flag is changed. + pub fn prepare_verified( + &self, + id: ImageId, + base: Option<&GenerationPin>, + update: &ImageUpdate<'_>, + ) -> Result { + match self.prepare(id, base, update) { + Ok(pin) => return Ok(pin), + Err(GenerationError::Exists) => (), + Err(error) => return Err(error), + } + let pin = self.pin_prepared(id, update.result)?; + let updates = std::slice::from_ref(update); + let peak = apply::peak_chain_bytes(updates, &self.inner.limits.apply)?; + let _scratch = self.inner.snapshot_scratch.acquire(peak, 1)?; + let mut file = self + .inner + .filesystem + .temporary_file(Some(&self.inner.directory))?; + let source = base.map(|pin| self.inner.directory.join(final_name(pin.id()))); + let actual = apply::apply_private_chain( + self.inner.filesystem.as_ref(), + source.as_deref(), + &mut file, + updates, + &self.inner.limits.apply, + )?; + if actual != update.result { + return Err(GenerationError::Identity); + } + pin.verify_and_sync()?; + Ok(pin) + } + /// Captured physical limits; callers can partition replay without relaxing /// the store's aggregate decode, temporary-image or generation budgets. pub fn limits(&self) -> GenerationLimits { diff --git a/crates/cellar-ltx/src/generation_receive.rs b/crates/cellar-ltx/src/generation_receive.rs index 85c1a2ab7..c7a07d6e4 100644 --- a/crates/cellar-ltx/src/generation_receive.rs +++ b/crates/cellar-ltx/src/generation_receive.rs @@ -82,17 +82,7 @@ impl GenerationReceive { // pin_prepared may reuse an in-memory seal. A transfer must // independently inspect the existing destination as well as // the incoming spool, while the pin prevents reclamation. - let mut destination = self - .store - .inner - .filesystem - .open(&self.store.inner.directory.join(final_name(self.id)))?; - apply::inspect(&mut destination, &self.expected, None)?; - destination.sync_all()?; - self.store - .inner - .filesystem - .sync_all(&self.store.inner.directory)?; + pin.verify_and_sync()?; return Ok(pin); } Err(GenerationError::Missing) => {} diff --git a/crates/cellar/durability/raft/cell_groups/apply/images.rs b/crates/cellar/durability/raft/cell_groups/apply/images.rs index ac084af6d..985664d85 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/images.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/images.rs @@ -73,6 +73,7 @@ impl CellGroupApplyImages { pub(super) fn prepare( &self, request: ImageApply<'_>, + verify_existing: bool, ) -> Result { let ImageApply { cell, @@ -147,13 +148,21 @@ impl CellGroupApplyImages { segments, }; let id = image_id(cell, sequence, update.result)?; - CellImagePin::Materialized(match images.prepare(id, base, &update) { - Ok(pin) => pin, - Err(cellar_ltx::generation::GenerationError::Exists) => { - images.pin_prepared(id, update.result).map_err(storage)? - } - Err(error) => return Err(storage(error)), - }) + if verify_existing { + CellImagePin::Materialized( + images + .prepare_verified(id, base, &update) + .map_err(storage)?, + ) + } else { + CellImagePin::Materialized(match images.prepare(id, base, &update) { + Ok(pin) => pin, + Err(cellar_ltx::generation::GenerationError::Exists) => { + images.pin_prepared(id, update.result).map_err(storage)? + } + Err(error) => return Err(storage(error)), + }) + } } (Self::Extents { images, limits, .. }, ImageCommitment::Extents(result)) => { let base = match base { diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 27e50cf10..768eb3ebd 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -5,18 +5,17 @@ pub(super) mod images; mod recovery; mod summary; +mod transition; pub use images::CellGroupApplyImages; use super::*; -use crate::application::consensus::group_protocol::{ - CellGroupEntry, GroupCommand, GroupEntryLimits, -}; +use crate::application::consensus::group_protocol::GroupEntryLimits; use crate::raft::{ cache::GroupLog, completion::{entry_digest, entry_position}, execution::{storage, ExecutionError}, }; -use cellar_logic::cell_group::{CommitWitness, GroupHistory}; +use cellar_logic::cell_group::GroupHistory; use std::sync::{Arc, Mutex}; pub(super) struct Storage { @@ -174,118 +173,16 @@ impl CellGroupApplier { ExecutionError::Invalid("poisoned group applier") } })?; - let Storage { - tree, - pages, - images, - limits, - max_effect_rows, - } = &mut *owner; - // Check page owner and contiguous prefix before any image allocation/I/O. - let mut unchanged = tree - .prepare_unchanged(pages, &work.base, work.ticket.position()) - .map_err(storage)?; - if work.entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { + let next = if work.entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { let cache = self.group.read().map_err(storage)?; - super::membership::transition( + owner.prepare_entry( + &work.base, &work.entry, - tree.route(), - &cache.conf_state, - &cache.membership, - ) - .map_err(storage)?; - unchanged.marker.membership_revision = work.entry.index; - return Ok(unchanged); - } - if work.entry.data.is_empty() { - return Ok(unchanged); - } - let decoded = CellGroupEntry::decode(&work.entry.data, work.base.marker().route(), *limits) - .map_err(storage)?; - let witness = CommitWitness { - group: self.history(), - position: work.ticket.position(), - proposal: decoded.binding.proposal, - }; - let next = match decoded.command { - GroupCommand::Enroll { cell, image, files } => { - let decision = decide_enrollment( - tree.route(), - EnrollmentPhase::Active, - &work.entry.data, - work.base - .pin_cell(&cell.cell) - .map(|(leaf, _)| &leaf.enrollment), - *limits, - ); - match decision { - Ok(EnrollmentDecision::Existing(_)) | Err(EnrollmentError::Conflict) => { - unchanged - } - Ok(EnrollmentDecision::Propose(record)) => { - let prepared = images.prepare(images::ImageApply { - cell: &cell, - sequence: 1, - schema: tree.route().schema_digest, - base: None, - image, - files: &files, - max_effect_rows: *max_effect_rows, - })?; - let leaf = CellManifestLeaf { - enrollment: record, - version: cellar_logic::cell_group::CellVersion { - history: cell, - sequence: cellar_logic::cell_group::MutationSequence::new(1) - .map_err(storage)?, - image: image.into(), - }, - geometry: prepared.geometry, - last_mutation: work.ticket.position(), - witness, - effects: prepared.effects, - }; - tree.prepare_update(pages, &work.base, leaf, None, prepared.pin) - .map_err(storage)? - } - Err(error) => return Err(storage(error)), - } - } - GroupCommand::Mutation(mutation) => { - let (old, base) = work - .base - .pin_cell(&mutation.cell.cell) - .ok_or(ExecutionError::Invalid("apply cell missing"))?; - mutation.check_base(&old.version).map_err(storage)?; - let prepared = images.prepare(images::ImageApply { - cell: &mutation.cell, - sequence: mutation.sequence.get(), - schema: tree.route().schema_digest, - base: Some(&base), - image: mutation.result, - files: &mutation.files, - max_effect_rows: *max_effect_rows, - })?; - let leaf = CellManifestLeaf { - enrollment: old.enrollment.clone(), - version: cellar_logic::cell_group::CellVersion { - history: mutation.cell, - sequence: mutation.sequence, - image: mutation.result.into(), - }, - geometry: prepared.geometry, - last_mutation: work.ticket.position(), - witness, - effects: prepared.effects, - }; - tree.prepare_update(pages, &work.base, leaf, Some(&old.version), prepared.pin) - .map_err(storage)? - } - _ => { - return Err(ExecutionError::Invalid( - "unsupported group apply administration", - )) - } + Some((&cache.conf_state, &cache.membership)), + false, + )? + } else { + owner.prepare_entry(&work.base, &work.entry, None, false)? }; self.check(work)?; Ok(next) diff --git a/crates/cellar/durability/raft/cell_groups/apply/transition.rs b/crates/cellar/durability/raft/cell_groups/apply/transition.rs new file mode 100644 index 000000000..35f945327 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/apply/transition.rs @@ -0,0 +1,133 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical transitions shared by live committed apply and archive data replay. +//! This module supplies preparation, never a persistence ticket or serving right. +use super::*; +use crate::application::consensus::group_protocol::{CellGroupEntry, GroupCommand}; +use crate::raft::member_state::GroupMembership; +use cellar_logic::cell_group::CommitWitness; +use raft::prelude::{ConfState, Entry}; + +impl Storage { + pub(in crate::raft::cell_groups) fn prepare_entry( + &mut self, + base: &PreparedGroupManifest, + entry: &Entry, + configuration: Option<(&ConfState, &GroupMembership)>, + verify_existing: bool, + ) -> Result { + let Storage { + tree, + pages, + images, + limits, + max_effect_rows, + } = self; + if entry.get_entry_type() == ::raft::prelude::EntryType::EntryConfChangeV2 { + let (conf, membership) = configuration.ok_or(ExecutionError::Invalid( + "configuration replay state missing", + ))?; + return tree + .prepare_membership(pages, base, entry, conf, membership) + .map_err(storage); + } + // Check page owner and contiguous prefix before any image allocation/I/O. + let unchanged = tree + .prepare_unchanged(pages, base, entry_position(entry)) + .map_err(storage)?; + if entry.data.is_empty() { + return Ok(unchanged); + } + let decoded = + CellGroupEntry::decode(&entry.data, base.marker().route(), *limits).map_err(storage)?; + let witness = CommitWitness { + group: base.marker().route().history, + position: entry_position(entry), + proposal: decoded.binding.proposal, + }; + let next = match decoded.command { + GroupCommand::Enroll { cell, image, files } => { + let decision = decide_enrollment( + tree.route(), + EnrollmentPhase::Active, + &entry.data, + base.pin_cell(&cell.cell).map(|(leaf, _)| &leaf.enrollment), + *limits, + ); + match decision { + Ok(EnrollmentDecision::Existing(_)) | Err(EnrollmentError::Conflict) => { + unchanged + } + Ok(EnrollmentDecision::Propose(record)) => { + let prepared = images.prepare( + images::ImageApply { + cell: &cell, + sequence: 1, + schema: tree.route().schema_digest, + base: None, + image, + files: &files, + max_effect_rows: *max_effect_rows, + }, + verify_existing, + )?; + let leaf = CellManifestLeaf { + enrollment: record, + version: cellar_logic::cell_group::CellVersion { + history: cell, + sequence: cellar_logic::cell_group::MutationSequence::new(1) + .map_err(storage)?, + image: image.into(), + }, + geometry: prepared.geometry, + last_mutation: entry_position(entry), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, base, leaf, None, prepared.pin) + .map_err(storage)? + } + Err(error) => return Err(storage(error)), + } + } + GroupCommand::Mutation(mutation) => { + let (old, image_base) = base + .pin_cell(&mutation.cell.cell) + .ok_or(ExecutionError::Invalid("apply cell missing"))?; + mutation.check_base(&old.version).map_err(storage)?; + let prepared = images.prepare( + images::ImageApply { + cell: &mutation.cell, + sequence: mutation.sequence.get(), + schema: tree.route().schema_digest, + base: Some(&image_base), + image: mutation.result, + files: &mutation.files, + max_effect_rows: *max_effect_rows, + }, + verify_existing, + )?; + let leaf = CellManifestLeaf { + enrollment: old.enrollment.clone(), + version: cellar_logic::cell_group::CellVersion { + history: mutation.cell, + sequence: mutation.sequence, + image: mutation.result.into(), + }, + geometry: prepared.geometry, + last_mutation: entry_position(entry), + witness, + effects: prepared.effects, + }; + tree.prepare_update(pages, base, leaf, Some(&old.version), prepared.pin) + .map_err(storage)? + } + _ => { + return Err(ExecutionError::Invalid( + "unsupported group apply administration", + )) + } + }; + Ok(next) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs index c4e3133d7..f4f7ae953 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -78,9 +78,9 @@ impl Index { } } -struct Bounded { - bytes: Vec, - limit: usize, +pub(super) struct Bounded { + pub(super) bytes: Vec, + pub(super) limit: usize, } impl Write for Bounded { @@ -220,9 +220,50 @@ impl CellGroupArchive { &self, domain: &ExecutionDomain, reference: &CellGroupArchiveReference, - mut receiver: CellGroupCheckpointReceiver, + receiver: CellGroupCheckpointReceiver, ) -> Result { let permit = self.admit()?; + self.verify_admitted(domain, reference, receiver, permit) + .await + } + + pub(super) async fn checkpoint_marker( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveReference, + permit: Arc, + ) -> Result { + let bytes = self + .get( + "indexes", + &Object { + digest: reference.digest, + length: reference.length, + }, + self.limits.max_index_bytes, + ) + .await?; + let archive = self.clone(); + domain + .blocking(move || { + let _permit = permit; + let index: Index = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("checkpoint index"))?; + index.validate(&archive)?; + Ok::<_, CellGroupArchiveError>(index.marker) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + } + + pub(super) async fn verify_admitted( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveReference, + mut receiver: CellGroupCheckpointReceiver, + permit: Arc, + ) -> Result { let bytes = self .get( "indexes", diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 1d44b4fb9..3cf0a65f0 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -1,6 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Complete group checkpoints in immutable object storage. Upload confirmation +//! Complete group checkpoints and committed tails in immutable object storage. Upload confirmation //! and independently recovered physical state are separate capabilities. Neither //! authorizes S3 settlement, log truncation, membership or disaster recovery. @@ -20,6 +20,13 @@ use std::{fmt, sync::Arc}; mod checkpoint; mod objects; +mod replay; +mod tail; +mod tail_admission; +mod tail_codec; +pub use replay::VerifiedCellGroupArchiveRecovery; +pub use tail::CellGroupArchiveTailReference; +pub use tail_admission::{CellGroupArchiveTailLimits, CellGroupArchiveTailPool}; const DOMAIN: &str = "cellar-cell-group-archive-checkpoint-1"; const CHUNK_BYTES: usize = 64 * 1024; @@ -32,6 +39,7 @@ pub enum CellGroupArchiveError { Stopped, Timeout, Checkpoint(CellGroupCheckpointError), + Replay(crate::raft::execution::ExecutionError), Storage(object_store::Error), Worker(crate::asyncrt::TaskPanic), } @@ -46,6 +54,7 @@ impl std::error::Error for CellGroupArchiveError { fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { match self { Self::Checkpoint(e) => Some(e), + Self::Replay(e) => Some(e), Self::Storage(e) => Some(e), Self::Worker(e) => Some(e), _ => None, diff --git a/crates/cellar/durability/raft/cell_groups/archive/replay.rs b/crates/cellar/durability/raft/cell_groups/archive/replay.rs new file mode 100644 index 000000000..674d0e07e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/replay.rs @@ -0,0 +1,204 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Non-serving data replay. Never constructs Raft tickets, changes a hard state, +//! installs a snapshot, invokes application callbacks or grants a quorum fence. +use super::super::{apply::Storage, CellGroupApplyImages, CellImagePin, PreparedGroupManifest}; +use super::{tail::Cut, *}; +use crate::raft::member_state::GroupMembership; +use raft::prelude::{ConfState, Entry, EntryType}; +use std::sync::atomic::{AtomicBool, Ordering}; + +/// Complete receiver-local state after independently replaying the selected +/// checkpoint and contiguous tail. Keeps physical pins and replay admission; +/// this supplies no published head, GC lease, serving right or S3 settlement. +pub struct VerifiedCellGroupArchiveRecovery { + reference: CellGroupArchiveTailReference, + checkpoint: VerifiedCellGroupArchiveCheckpoint, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + _reservation: Arc, +} + +impl VerifiedCellGroupArchiveRecovery { + pub fn reference(&self) -> &CellGroupArchiveTailReference { + &self.reference + } + + pub fn checkpoint(&self) -> &VerifiedCellGroupArchiveCheckpoint { + &self.checkpoint + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.root + } + + pub fn configuration(&self) -> &ConfState { + &self.configuration + } + + pub fn membership(&self) -> &GroupMembership { + &self.membership + } +} + +pub(super) fn replay( + reference: CellGroupArchiveTailReference, + checkpoint: VerifiedCellGroupArchiveCheckpoint, + cut: Cut, + entries: Vec, + limits: CellGroupArchiveTailLimits, + reservation: Arc, + cancelled: Arc, +) -> Result { + let applier = checkpoint.checkpoint().applier().clone(); + let selected = applier + .group + .group_manifest() + .map_err(CellGroupCheckpointError::from)?; + let mut root = checkpoint.checkpoint().manifest().clone(); + let mut conf = checkpoint + .checkpoint() + .snapshot() + .get_metadata() + .get_conf_state() + .clone(); + let mut membership = checkpoint.checkpoint().membership().clone(); + let mut owner = applier.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, + std::sync::TryLockError::Poisoned(_) => { + CellGroupArchiveError::Invalid("poisoned archive replay owner") + } + })?; + if owner.images.memory_bound() > limits.max_apply_bytes + || root.pinned_images() > limits.max_cells as usize + { + return Err(CellGroupArchiveError::Limit); + } + for entry in &entries { + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + applier + .group + .cache_revision() + .map_err(CellGroupCheckpointError::from)?; + // Control replay is the same canonical pure transition used by the live + // owner; cell bytes still go through the ordinary LTX/base-chain checks. + let change = if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + Some( + super::super::membership::transition( + entry, + root.marker().route(), + &conf, + &membership, + ) + .map_err(CellGroupCheckpointError::from)?, + ) + } else { + None + }; + root = owner + .prepare_entry(&root, entry, Some((&conf, &membership)), true) + .map_err(CellGroupArchiveError::Replay)?; + if root.pinned_images() > limits.max_cells as usize { + return Err(CellGroupArchiveError::Limit); + } + if let Some(change) = change { + conf = change.conf; + membership = change.membership; + } + } + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + let encoded = crate::raft::manifest::encode_conf(&conf) + .map_err(|_| CellGroupArchiveError::Invalid("replayed configuration"))?; + if root.marker() != &cut.marker || encoded != cut.conf || membership != cut.membership { + return Err(CellGroupArchiveError::Invalid("replayed complete cut")); + } + // Recheck every final physical image and page, even after warm replay. An + // in-memory pin or a cached destination is never a substitute for bytes. + let Storage { + tree, + pages, + images, + .. + } = &mut *owner; + let verified = tree + .prepare_complete( + &applier.group, + pages, + root.marker().root(), + root.marker().position(), + root.marker().membership_revision(), + |leaf| { + if cancelled.load(Ordering::Acquire) { + return Err(super::super::ManifestError::Invalid( + "archive replay stopped", + )); + } + let (_, pin) = root + .pin_cell(&leaf.version.history.cell) + .filter(|(expected, _)| *expected == leaf) + .ok_or(super::super::ManifestError::Invalid( + "replayed leaf missing", + ))?; + match (&*images, &pin) { + (_, CellImagePin::Empty) => (), + ( + CellGroupApplyImages::Materialized { .. }, + CellImagePin::Materialized(pin), + ) => { + pin.verify_and_sync().map_err(|e| { + super::super::ManifestError::Io(std::io::Error::other(e)) + })?; + } + (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(pin)) => { + if pin.reference().encoded_length() > 1024 * 1024 { + return Err(super::super::ManifestError::Invalid( + "replay extent metadata limit", + )); + } + images + .seal_image_root( + pin.manifest().clone(), + &mut vec![0; cellar_ltx::internal::EXTENT_BYTES as usize], + ) + .map_err(|e| { + super::super::ManifestError::Io(std::io::Error::other(e)) + })?; + } + _ => { + return Err(super::super::ManifestError::Invalid( + "replay representation", + )) + } + } + Ok(pin) + }, + ) + .map_err(CellGroupCheckpointError::from)?; + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + if applier + .group + .group_manifest() + .map_err(CellGroupCheckpointError::from)? + != selected + { + return Err(CellGroupArchiveError::Invalid( + "replay destination selection changed", + )); + } + drop(owner); + Ok(VerifiedCellGroupArchiveRecovery { + reference, + checkpoint, + root: verified, + configuration: conf, + membership, + _reservation: reservation, + }) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs new file mode 100644 index 000000000..a911ee093 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -0,0 +1,480 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::PreparedGroupManifest; +use super::{objects::Object, *}; +use crate::raft::{cache::GroupLog, member_state::GroupMembership}; +use raft::prelude::ConfState; +use std::sync::atomic::{AtomicBool, Ordering}; + +const TAIL_DOMAIN: &str = "cellar-cell-group-archive-tail-1"; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +/// An untrusted immutable-tail locator; recovery must verify its entire closure. +pub struct CellGroupArchiveTailReference { + digest: [u8; 32], + length: u64, +} + +impl CellGroupArchiveTailReference { + pub fn digest(&self) -> [u8; 32] { + self.digest + } + + pub fn length(&self) -> u64 { + self.length + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Cut { + pub marker: GroupManifest, + pub conf: Vec, + pub membership: GroupMembership, +} + +impl Cut { + pub(super) fn configuration(&self) -> Result { + crate::raft::manifest::decode_conf(&self.conf) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration")) + } + + fn validate(&self, history: GroupHistory, limits: CellGroupArchiveTailLimits) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.marker.route().history != history + || self.marker.root().cells() > limits.max_cells + || self.conf.len() > 64 * 1024 + || self.membership.revision().get() != self.marker.membership_revision() + { + return Err(CellGroupArchiveError::Invalid("tail cut binding/bounds")); + } + self.membership + .validate(&self.configuration()?, self.marker.position().index.get()) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Tail { + domain: String, + checkpoint: CellGroupArchiveReference, + base: GroupManifest, + cut: Cut, + bytes: u64, + entries: Vec, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct EntryObject { + index: u64, + term: u64, + object: Object, +} + +impl Tail { + fn validate( + &self, + archive: &CellGroupArchive, + limits: CellGroupArchiveTailLimits, + ) -> Result<()> { + self.base.check().map_err(CellGroupCheckpointError::from)?; + self.cut.validate(archive.history, limits)?; + if self.domain != TAIL_DOMAIN + || self.base.route() != self.cut.marker.route() + || self.checkpoint.length == 0 + || self.checkpoint.length > archive.limits.max_index_bytes as u64 + || self.entries.len() > limits.max_entries + || self.bytes > limits.max_tail_bytes as u64 + || self + .cut + .marker + .position() + .index + .get() + .checked_sub(self.base.position().index.get()) + != Some(self.entries.len() as u64) + { + return Err(CellGroupArchiveError::Invalid("tail domain/cuts/limits")); + } + let mut index = self.base.position().index.get(); + let mut term = self.base.position().term.get(); + let mut bytes = 0u64; + for entry in &self.entries { + index = index.checked_add(1).ok_or(CellGroupArchiveError::Limit)?; + if entry.index != index + || entry.term < term + || entry.object.length == 0 + || entry.object.length > limits.max_tail_bytes as u64 + { + return Err(CellGroupArchiveError::Invalid("tail continuity")); + } + term = entry.term; + bytes = bytes + .checked_add(entry.object.length) + .ok_or(CellGroupArchiveError::Limit)?; + } + if bytes != self.bytes || term != self.cut.marker.position().term.get() { + return Err(CellGroupArchiveError::Invalid("tail total/final term")); + } + Ok(()) + } +} + +struct Capture { + cut: Cut, + entries: Vec<(u64, u64, Vec)>, + _root: PreparedGroupManifest, +} + +fn capture( + group: GroupLog, + base: &GroupManifest, + current: PreparedGroupManifest, + limits: CellGroupArchiveTailLimits, +) -> Result { + if !Arc::ptr_eq(&group.owner, ¤t.engine_owner) + || base.route() != current.marker().route() + || current.marker().root().cells() > limits.max_cells + { + return Err(CellGroupArchiveError::Invalid("tail source owner/route")); + } + let cache = group.read().map_err(CellGroupCheckpointError::from)?; + let target = current.marker().position(); + let count = target + .index + .get() + .checked_sub(base.position().index.get()) + .filter(|n| *n <= limits.max_entries as u64) + .ok_or(CellGroupArchiveError::Limit)?; + if cache.group_manifest.as_ref() != Some(current.marker()) + || cache.applied.position != target + || cache.applied.image.is_some() + || cache.hard_state.commit < target.index.get() + || cache.term(target.index.get()) != Some(target.term.get()) + || cache.term(base.position().index.get()) != Some(base.position().term.get()) + { + return Err(CellGroupArchiveError::Invalid("tail source committed cut")); + } + let mut entries = Vec::with_capacity(count as usize); + let mut bytes = 0usize; + let mut index = base.position().index.get(); + for _ in 0..count { + index += 1; + let entry = cache + .entries + .get(&index) + .ok_or(CellGroupArchiveError::Invalid( + "tail source log hole/compaction", + ))?; + bytes = bytes + .checked_add(tail_codec::length(entry)?) + .filter(|n| *n <= limits.max_tail_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + entries.push(entry.clone()); + } + let cut = Cut { + marker: current.marker().clone(), + conf: crate::raft::manifest::encode_conf(&cache.conf_state) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration encoding"))?, + membership: cache.membership.clone(), + }; + drop(cache); // No engine/cache lock across encoding or physical/S3 work. + cut.validate(group.identity().scope, limits)?; + current.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if matches!(pin, super::super::CellImagePin::Extents(pin) if pin.reference().encoded_length() > 1024 * 1024) { + return Err(CellGroupArchiveError::Limit); + } + Ok(()) + })?; + let entries = entries + .iter() + .map(|entry| { + Ok(( + entry.index, + entry.term, + tail_codec::encode(entry, limits.max_tail_bytes)?, + )) + }) + .collect::>>()?; + Ok(Capture { + cut, + entries, + _root: current, + }) +} + +enum Origin<'a> { + Checkpoint(&'a CellGroupArchiveReference), + Tail(&'a CellGroupArchiveTailReference), +} + +struct Stop(Arc); + +impl Drop for Stop { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupArchive { + /// Copy an exact contiguous committed suffix under the engine cache lock, + /// then upload immutable entry objects without holding that lock. This + /// returns an untrusted locator; all base bytes and replay must still verify. + pub async fn upload_tail( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveTailPool, + checkpoint: &CellGroupArchiveReference, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + self.upload_from(domain, pool, Origin::Checkpoint(checkpoint), group, current) + .await + } + + /// Extend an immutable archived prefix while copying only the local suffix + /// after its last cut. This works after local prefix compaction; references + /// to prior entry objects are reused. Full closure verification is still + /// required, and the caller's publication owner must protect the base/prefix. + pub async fn extend_tail( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveTailPool, + previous: &CellGroupArchiveTailReference, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + self.upload_from(domain, pool, Origin::Tail(previous), group, current) + .await + } + + async fn upload_from( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveTailPool, + origin: Origin<'_>, + group: &GroupLog, + current: &PreparedGroupManifest, + ) -> Result { + domain + .run(async { + let io = self.admit()?; + let retention = pool.admit()?; + let limits = pool.limits; + let (checkpoint, base, worker_base, prefix, prefix_bytes) = match origin { + Origin::Checkpoint(checkpoint) => { + let marker = self + .checkpoint_marker(domain, checkpoint, io.clone()) + .await?; + (checkpoint.clone(), marker.clone(), marker, Vec::new(), 0) + } + Origin::Tail(previous) => { + let tail = self + .read_tail_index( + domain, + previous, + limits, + io.clone(), + retention.clone(), + ) + .await?; + ( + tail.checkpoint, + tail.base, + tail.cut.marker, + tail.entries, + tail.bytes, + ) + } + }; + let group = group.clone(); + let root = current.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + let captured = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + capture(group, &worker_base, root, limits) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let mut tail = Tail { + domain: TAIL_DOMAIN.into(), + checkpoint, + base, + cut: captured.cut, + bytes: prefix_bytes, + entries: prefix, + }; + let total_entries = tail + .entries + .len() + .checked_add(captured.entries.len()) + .ok_or(CellGroupArchiveError::Limit)?; + let total_bytes = captured + .entries + .iter() + .try_fold(tail.bytes, |sum, (_, _, bytes)| { + sum.checked_add(bytes.len() as u64) + }) + .ok_or(CellGroupArchiveError::Limit)?; + if total_entries > limits.max_entries || total_bytes > limits.max_tail_bytes as u64 + { + return Err(CellGroupArchiveError::Limit); + } + for (index, term, entry) in captured.entries { + let object = self.put("entries", entry, limits.max_tail_bytes).await?; + tail.bytes = tail + .bytes + .checked_add(object.length) + .ok_or(CellGroupArchiveError::Limit)?; + tail.entries.push(EntryObject { + index, + term, + object, + }); + } + tail.validate(self, limits)?; + let archive = self.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + let encoded = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + let mut output = checkpoint::Bounded { + bytes: Vec::new(), + limit: archive.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &tail) + .map_err(|_| CellGroupArchiveError::Limit)?; + Ok::<_, CellGroupArchiveError>(output.bytes) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let object = self + .put("tails", encoded, self.limits.max_index_bytes) + .await?; + Ok(CellGroupArchiveTailReference { + digest: object.digest, + length: object.length, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn read_tail_index( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveTailReference, + limits: CellGroupArchiveTailLimits, + io: Arc, + retention: Arc, + ) -> Result { + let bytes = self + .get( + "tails", + &Object { + digest: reference.digest, + length: reference.length, + }, + self.limits.max_index_bytes, + ) + .await?; + let archive = self.clone(); + let retained = retention.clone(); + let retained_io = io.clone(); + domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + let tail: Tail = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("tail index"))?; + tail.validate(&archive, limits)?; + Ok::<_, CellGroupArchiveError>(tail) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + } + + /// Independently recover the base and replay every cell/control entry. + /// Retains replay admission with the returned physical closure. No archive + /// head/GC protection, serving permission or S3 settlement is supplied here. + pub async fn verify_tail( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveTailPool, + reference: &CellGroupArchiveTailReference, + receiver: CellGroupCheckpointReceiver, + ) -> Result { + domain + .run(async { + let io = self.admit()?; + let retention = pool.admit()?; + let limits = pool.limits; + let tail = self + .read_tail_index(domain, reference, limits, io.clone(), retention.clone()) + .await?; + let mut encoded = Vec::with_capacity(tail.entries.len()); + for entry in &tail.entries { + encoded.push( + self.get("entries", &entry.object, limits.max_tail_bytes) + .await?, + ); + } + let retained = retention.clone(); + let retained_io = io.clone(); + let descriptors = tail.entries; + let entries = domain + .blocking(move || { + let (_retained, _io) = (retained, retained_io); + encoded + .into_iter() + .zip(descriptors) + .map(|(bytes, expected)| { + let entry = tail_codec::decode(&bytes, limits.max_tail_bytes)?; + if entry.index != expected.index || entry.term != expected.term { + return Err(CellGroupArchiveError::Invalid( + "tail entry descriptor", + )); + } + Ok(entry) + }) + .collect::>>() + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)??; + let base = self + .verify_admitted(domain, &tail.checkpoint, receiver, io.clone()) + .await?; + if base.checkpoint().marker() != &tail.base { + return Err(CellGroupArchiveError::Invalid("tail checkpoint binding")); + } + let stop = Stop(Arc::new(AtomicBool::new(false))); + let cancelled = stop.0.clone(); + let reference = reference.clone(); + let retained_io = io.clone(); + domain + .blocking(move || { + let _io = retained_io; + replay::replay( + reference, base, tail.cut, entries, limits, retention, cancelled, + ) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs new file mode 100644 index 000000000..aba40dcfc --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs @@ -0,0 +1,89 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::ManifestPins; +use super::*; + +/// Additional node-wide admission for committed-tail capture and independent +/// LTX replay. Share this pool across all group bindings. The ordinary archive +/// pool still charges I/O and indexes; checkpoint pools still charge their cuts. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveTailLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_entries: usize, + pub max_tail_bytes: usize, + pub max_cells: u32, + /// Must cover the destination applier's captured image workspace bound. + pub max_apply_bytes: usize, +} + +impl Default for CellGroupArchiveTailLimits { + fn default() -> Self { + Self { + max_operations: 1, + max_retained_bytes: 512 * 1024 * 1024, + max_entries: 1024, + max_tail_bytes: 16 * 1024 * 1024, + max_cells: 64, + max_apply_bytes: 128 * 1024 * 1024, + } + } +} + +#[derive(Clone)] +pub struct CellGroupArchiveTailPool { + pub(super) limits: CellGroupArchiveTailLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchiveTailPool { + pub fn new(limits: CellGroupArchiveTailLimits) -> Result { + if !(1..=4096).contains(&limits.max_operations) + || !(1..=65_536).contains(&limits.max_entries) + || !(1024..=512 * 1024 * 1024).contains(&limits.max_tail_bytes) + || !(1..=1_000_000).contains(&limits.max_cells) + || !(4 * 1024 * 1024..=512 * 1024 * 1024).contains(&limits.max_apply_bytes) + { + return Err(CellGroupArchiveError::Limit); + } + // Base, predecessor and successor roots can coexist while one image + // transition executes. Each cell may retain a <=1 MiB extent manifest. + let metadata = ManifestPins::maximum_bytes(limits.max_cells) + .checked_add( + (limits.max_cells as usize) + .checked_mul(1024 * 1024) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .and_then(|n| n.checked_mul(3)); + let workspace = metadata + .and_then(|n| n.checked_add(limits.max_tail_bytes.checked_mul(4)?)) + .and_then(|n| n.checked_add(limits.max_apply_bytes)) + .and_then(|n| n.checked_add(limits.max_entries.checked_mul(512)?)) + .and_then(|n| n.checked_add(1024 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + retained_bytes, + operations, + } + } + + pub(super) fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs b/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs new file mode 100644 index 000000000..86234c242 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fixed canonical entry framing bounds allocation before decoding. Includes +//! context and sync_log; no unknown/repeated protobuf fields are accepted. +use super::*; +use protobuf::Message as _; +use raft::prelude::{Entry, EntryType}; + +const ENTRY_DOMAIN: &[u8] = b"cellar-cell-group-archive-entry-1\0"; +const FIXED: usize = ENTRY_DOMAIN.len() + 26; + +pub(super) fn length(entry: &Entry) -> Result { + if entry.index == 0 + || entry.term == 0 + || entry.get_unknown_fields().iter().next().is_some() + || !matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) + { + return Err(CellGroupArchiveError::Invalid("archive entry metadata")); + } + entry + .data + .len() + .checked_add(entry.context.len()) + .and_then(|n| n.checked_add(FIXED)) + .ok_or(CellGroupArchiveError::Limit) +} + +pub(super) fn encode(entry: &Entry, limit: usize) -> Result> { + let length = length(entry)?; + if length > limit + || entry.data.len() > u32::MAX as usize + || entry.context.len() > u32::MAX as usize + { + return Err(CellGroupArchiveError::Limit); + } + let mut bytes = Vec::with_capacity(length); + bytes.extend_from_slice(ENTRY_DOMAIN); + bytes.extend_from_slice(&entry.index.to_be_bytes()); + bytes.extend_from_slice(&entry.term.to_be_bytes()); + bytes.push(entry.get_entry_type() as u8); + bytes.push(u8::from(entry.sync_log)); + bytes.extend_from_slice(&(entry.context.len() as u32).to_be_bytes()); + bytes.extend_from_slice(&(entry.data.len() as u32).to_be_bytes()); + bytes.extend_from_slice(&entry.context); + bytes.extend_from_slice(&entry.data); + Ok(bytes) +} + +pub(super) fn decode(bytes: &[u8], limit: usize) -> Result { + if bytes.len() < FIXED || bytes.len() > limit || !bytes.starts_with(ENTRY_DOMAIN) { + return Err(CellGroupArchiveError::Invalid("archive entry framing")); + } + let at = ENTRY_DOMAIN.len(); + // Every fixed slice below lies in the already checked header. + let index = u64::from_be_bytes(bytes[at..at + 8].try_into().unwrap()); + let term = u64::from_be_bytes(bytes[at + 8..at + 16].try_into().unwrap()); + let entry_type = match bytes[at + 16] { + 0 => EntryType::EntryNormal, + 2 => EntryType::EntryConfChangeV2, + _ => return Err(CellGroupArchiveError::Invalid("archive entry type")), + }; + let sync_log = match bytes[at + 17] { + 0 => false, + 1 => true, + _ => return Err(CellGroupArchiveError::Invalid("archive entry sync bit")), + }; + let context = u32::from_be_bytes(bytes[at + 18..at + 22].try_into().unwrap()) as usize; + let data = u32::from_be_bytes(bytes[at + 22..at + 26].try_into().unwrap()) as usize; + if context.checked_add(data).and_then(|n| n.checked_add(FIXED)) != Some(bytes.len()) { + return Err(CellGroupArchiveError::Invalid("archive entry lengths")); + } + let entry = Entry { + index, + term, + entry_type, + sync_log, + context: bytes[FIXED..FIXED + context].to_vec().into(), + data: bytes[FIXED + context..].to_vec().into(), + ..Default::default() + }; + length(&entry)?; + Ok(entry) +} diff --git a/crates/cellar/durability/raft/cell_groups/membership.rs b/crates/cellar/durability/raft/cell_groups/membership.rs index ffe028337..dff3bb096 100644 --- a/crates/cellar/durability/raft/cell_groups/membership.rs +++ b/crates/cellar/durability/raft/cell_groups/membership.rs @@ -180,6 +180,40 @@ pub(crate) struct Transition { pub membership: GroupMembership, } +impl super::ManifestTree { + /// Prepare the unchanged cell root and new configuration revision for one + /// canonical membership entry. This proves data preparation only; the Raft + /// owner must validate its committed entry/ticket and atomically persist the + /// matching configuration and membership before acknowledging application. + pub fn prepare_membership( + &self, + pages: &super::ManifestPageStore, + previous: &super::PreparedGroupManifest, + entry: &Entry, + before: &ConfState, + membership: &GroupMembership, + ) -> Result { + let mut prepared = self.prepare_unchanged( + pages, + previous, + crate::raft::completion::entry_position(entry), + )?; + if previous.marker().membership_revision() != membership.revision().get() { + return Err(super::ManifestError::Invalid( + "membership preparation revision", + )); + } + membership + .validate(before, previous.marker().position().index.get()) + .map_err(|e| super::ManifestError::Io(std::io::Error::other(e)))?; + transition(entry, self.route(), before, membership) + .map_err(|e| super::ManifestError::Io(std::io::Error::other(e)))?; + prepared.marker.membership_revision = entry.index; + prepared.marker.check()?; + Ok(prepared) + } +} + pub(crate) fn transition( entry: &Entry, route: &GroupRoute, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 040e04212..98b5941a1 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -18,7 +18,9 @@ mod apply; mod archive; pub use archive::{ CellGroupArchive, CellGroupArchiveError, CellGroupArchiveLimits, CellGroupArchivePool, - CellGroupArchiveReference, CellGroupArchiveUsage, VerifiedCellGroupArchiveCheckpoint, + CellGroupArchiveReference, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, + CellGroupArchiveTailReference, CellGroupArchiveUsage, VerifiedCellGroupArchiveCheckpoint, + VerifiedCellGroupArchiveRecovery, }; pub(crate) mod checkpoint; pub use checkpoint::{ diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 18b1c7e86..25c647274 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -1,10 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! CG-06 checkpoint archival slice. No tail, settlement, log reclamation or -//! fresh-incarnation disaster-recovery qualification is claimed by these cases. +//! CG-06 checkpoint and committed-tail data recovery. Archive-head protection, +//! S3 settlement, reclamation and fresh-incarnation recovery remain separate. #![allow(clippy::disallowed_methods)] // The shared engine fixture also defines fault/membership helpers selected by -// sibling integration targets. They remain unused in this checkpoint-only gate. +// sibling integration targets and are not all selected by this archive gate. #![allow(dead_code, unused_imports)] #[path = "support/cell_group_archive_store.rs"] @@ -18,9 +18,48 @@ mod fault; #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; +#[path = "support/cell_group_archive_tail.rs"] +mod tail_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tails_cover_enrollment_and_controls() -> Result<()> { + for rooted in [false, true] { + tail_fixture::enrollment_controls(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_missing_and_forged_dependencies_refuse() -> Result<()> { + for rooted in [false, true] { + tail_fixture::dependencies(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_warm_ltx_is_independently_replayed() -> Result<()> { + for rooted in [false, true] { + tail_fixture::warm_replay(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_cancelled_replay_retains_physical_owners() -> Result<()> { + tail_fixture::physical_cancellation().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_tail_extension_survives_local_prefix_compaction() -> Result<()> { + for rooted in [false, true] { + tail_fixture::extend_compacted(rooted).await?; + } + Ok(()) +} + use cellar_ltx::{ generation::ReadServices, internal::{CacheLimits, EXTENT_BYTES}, diff --git a/crates/cellar/tests/support/cell_group_archive_tail.rs b/crates/cellar/tests/support/cell_group_archive_tail.rs new file mode 100644 index 000000000..0add7a1cd --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_tail.rs @@ -0,0 +1,759 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use raft::prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}; + +async fn tail_index( + store: &dyn ObjectStore, + reference: &CellGroupArchiveTailReference, +) -> Result<(Path, serde_json::Value)> { + let suffix = format!( + "/tails/{}-{}", + reference.length(), + hex::encode(reference.digest()) + ); + let objects: Vec<_> = store.list(None).try_collect().await?; + let path = objects + .into_iter() + .find(|m| m.location.as_ref().ends_with(&suffix)) + .context("tail index")? + .location; + let body = store.get(&path).await?.bytes().await?; + Ok((path, serde_json::from_slice(&body)?)) +} + +fn entry_path(path: &Path, entry: &serde_json::Value) -> Result { + let digest: Vec = serde_json::from_value(entry["object"]["digest"].clone())?; + let prefix = path + .as_ref() + .split("/tails/") + .next() + .context("tail prefix")?; + Ok(Path::from(format!( + "{prefix}/entries/{}-{}", + entry["object"]["length"], + hex::encode(digest) + ))) +} + +async fn write_tail( + store: &dyn ObjectStore, + original: &Path, + tail: &serde_json::Value, +) -> Result { + let bytes = serde_json::to_vec(tail)?; + let digest: [u8; 32] = Sha256::digest(&bytes).into(); + let prefix = original + .as_ref() + .split("/tails/") + .next() + .context("tail prefix")?; + store + .put( + &Path::from(format!( + "{prefix}/tails/{}-{}", + bytes.len(), + hex::encode(digest) + )), + bytes.clone().into(), + ) + .await?; + Ok(serde_json::from_value( + serde_json::json!({"digest": digest, "length": bytes.len()}), + )?) +} + +async fn check_images( + cluster: &Cluster, + root: &PreparedGroupManifest, + expected: &std::collections::BTreeMap>, +) -> Result<()> { + assert_eq!(root.pinned_images(), expected.len()); + for (name, wanted) in expected { + let (_, pin) = root + .pin_cell(&CellId::new(name)?) + .context("archived cell")?; + let bytes = cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await??; + assert_eq!(&bytes, wanted); + } + Ok(()) +} + +pub(super) async fn enrollment_controls(rooted: bool) -> Result<()> { + let mut cluster = Cluster::new().await?; + cluster.elect().await?; + cluster.voters[2].paused = true; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("tail-qualification"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let mut expected = + checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 1, 8192).await?; + for name in ["alpha", "beta"] { + checkpoint_fixture::advance_partition(&mut cluster, name, &expected[name], rooted, Some(2)) + .await?; + expected.insert( + name.into(), + images::image(Some(&expected[name]), "UPDATE value SET n=n+1")?, + ); + } + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + // A genuine explicit joint demotion and leave preserve initial member 3 as + // a learner. Both changes commit on the two healthy voters, with no fake cell. + let mut joint = ConfChangeV2::default(); + joint.set_transition(ConfChangeTransition::Explicit); + joint.changes.push(ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::AddLearnerNode, + ..Default::default() + }); + let change = + CellGroupMembershipChange::new(cluster.birth.route(), &group.membership()?, joint, &[])?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .configure(change)?; + cluster + .drive(false) + .await + .context("explicit joint configuration apply")?; + let leave = CellGroupMembershipChange::new( + cluster.birth.route(), + &group.membership()?, + ConfChangeV2::default(), + &[], + )?; + cluster.voters[0].owner.as_mut().unwrap().configure(leave)?; + cluster.drive(false).await?; + // Real leadership turnover appends the term's genuine Raft no-op after all + // data/control entries. The original leader applies it as a follower. + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .transfer_leader(2)?; + cluster.drive(false).await?; + assert!(cluster.voters[1].owner.as_ref().unwrap().status()?.ready); + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let expected_membership = group.membership()?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (_, encoded) = tail_index(store.as_ref(), &reference).await?; + assert_eq!(encoded["entries"].as_array().unwrap().len(), 9); + assert_eq!(tails.usage().operations, 0); + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let receiver_group = cluster.voters[2] + .log + .cell_group(&cluster.birth.route().history)?; + let selected = receiver_group.group_manifest()?; + for voter in &mut cluster.voters[..2] { + drop(voter.owner.take()); + voter.log.close().await?; + } + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + assert_eq!(recovered.reference(), &reference); + assert!(recovered.configuration().voters_outgoing.is_empty()); + assert_eq!(recovered.configuration().learners, vec![3]); + assert_eq!(recovered.membership(), &expected_membership); + assert_eq!(recovered.manifest().marker().root().cells(), 4); + for name in ["alpha", "beta"] { + assert_eq!( + recovered + .manifest() + .pin_cell(&CellId::new(name)?) + .unwrap() + .0 + .version + .sequence + .get(), + 2 + ); + } + assert_eq!(receiver_group.group_manifest()?, selected); + assert_eq!(receiver_group.checkpoint_manifest()?, None); + assert_eq!( + tails.usage().operations, + 1, + "returned closure retains replay admission" + ); + assert_eq!(pool.usage().sources, 1); + check_images(&cluster, recovered.manifest(), &expected).await?; + drop(recovered); + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn dependencies(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("tail-qualification"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (path, encoded) = tail_index(store.as_ref(), &reference).await?; + let entry = entry_path(&path, &encoded["entries"][0])?; + let entry_bytes = store.get(&entry).await?.bytes().await?; + let (base_path, base_index) = index(store.as_ref(), &base).await?; + let dependency = cell_objects(&base_path, &base_index, "beta") + .into_iter() + .last() + .context("base beta")?; + let dependency_bytes = store.get(&dependency).await?.bytes().await?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for missing in [&dependency, &entry] { + store.inner.delete(missing).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("missing archive object")) + )); + let bytes = if missing == &dependency { + dependency_bytes.clone() + } else { + entry_bytes.clone() + }; + store.inner.put(missing, bytes.into()).await?; + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + } + let mut corrupt = entry_bytes.to_vec(); + corrupt[0] ^= 1; + store.inner.put(&entry, corrupt.into()).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid("object digest/length")) + )); + store.inner.put(&entry, entry_bytes.into()).await?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + check_images(&cluster, recovered.manifest(), &expected).await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + drop(recovered); + for attack in 0..3 { + let mut forged = encoded.clone(); + match attack { + 0 => forged["entries"] = serde_json::json!([]), + 1 => { + forged["entries"][0]["index"] = + serde_json::json!(current.marker().position().index.get() + 1) + } + 2 => forged["cut"]["marker"]["root"] = base_index["marker"]["root"].clone(), + _ => unreachable!(), + } + let forged = write_tail(store.as_ref(), &path, &forged).await?; + let result = archive + .verify_tail( + &cluster.domain, + &tails, + &forged, + receiver(&pool, &applier, &cluster)?, + ) + .await; + match attack { + 0 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("tail domain/cuts/limits")) + )), + 1 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("tail continuity")) + )), + 2 => assert!(matches!( + result, + Err(CellGroupArchiveError::Invalid("replayed complete cut")) + )), + _ => unreachable!(), + } + assert_eq!(tails.usage().operations, 0); + assert_eq!(pool.usage().sources, 0); + } + cluster.close().await?; + Ok(()) +} + +pub(super) async fn warm_replay(rooted: bool) -> Result<()> { + let (mut cluster, before) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("tail-warm"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition(&mut cluster, "alpha", &before["alpha"], rooted, Some(2)) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let (path, original) = tail_index(store.as_ref(), &reference).await?; + let entry_location = entry_path(&path, &original["entries"][0])?; + let entry = store.get(&entry_location).await?.bytes().await?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + let (_, final_pin) = recovered + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("warm result")?; + let (_, base_pin) = recovered + .checkpoint() + .checkpoint() + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("warm base")?; + drop(recovered); + let wrong_image = images::image(Some(&before["alpha"]), "UPDATE value SET n=n+17")?; + let wrong = images::Cut::new(&before["alpha"], &wrong_image, 1, true, false); + if let (CellImagePin::Materialized(base_pin), CellImagePin::Materialized(final_pin)) = + (&base_pin, &final_pin) + { + let store = cluster.voters[2].images.clone(); + let base_pin = base_pin.clone(); + let final_pin = final_pin.clone(); + let wrong = wrong.bytes.clone(); + let before = before["alpha"].clone(); + cluster + .domain + .blocking(move || -> Result<()> { + let mut cut = images::Cut::new(&before, &wrong_image, 1, true, false); + cut.bytes = wrong; + let mut update = cut.update(); + update.base = base_pin.state(); + update.result = final_pin.state(); + assert!( + matches!( + store.prepare(final_pin.id(), Some(&base_pin), &update), + Err(cellar_ltx::generation::GenerationError::Exists) + ), + "ordinary cached preparation does not replay the supplied LTX" + ); + assert!(store + .pin_prepared(final_pin.id(), final_pin.state()) + .is_ok()); + Ok(()) + })? + .await??; + } + // Re-encode a valid group command carrying a different valid SQLite delta, + // while retaining the original declared result. Hash the forged entry/index + // correctly so only actual LTX replay can reject it, including on warm pins. + let fixed = b"cellar-cell-group-archive-entry-1\0".len(); + let context = u32::from_be_bytes(entry[fixed + 18..fixed + 22].try_into()?) as usize; + let data = fixed + 26 + context; + let mut decoded = CellGroupEntry::decode( + &entry[data..], + cluster.birth.route(), + GroupEntryLimits::default(), + )?; + let GroupCommand::Mutation(mutation) = &mut decoded.command else { + anyhow::bail!("expected mutation tail") + }; + let bad = wrong.update(); + mutation.files = vec![GroupLtx { + min_txid: bad.segments[0].min_txid.0, + max_txid: bad.segments[0].max_txid.0, + page_count: bad.segments[0].page_count as u64, + decoded_bytes: bad.segments[0].decoded_bytes, + bytes: &wrong.bytes, + }]; + let command = decoded.encode(cluster.birth.route(), GroupEntryLimits::default())?; + let mut body = entry[..data].to_vec(); + body[fixed + 22..fixed + 26].copy_from_slice(&(command.len() as u32).to_be_bytes()); + body.extend(command); + let digest: [u8; 32] = Sha256::digest(&body).into(); + let mut forged = original.clone(); + forged["entries"][0]["object"] = serde_json::json!({"digest": digest, "length": body.len()}); + forged["bytes"] = serde_json::json!(body.len()); + let object = entry_path(&path, &forged["entries"][0])?; + store.inner.put(&object, body.into()).await?; + let forged = write_tail(store.as_ref(), &path, &forged).await?; + assert!(matches!( + archive + .verify_tail( + &cluster.domain, + &tails, + &forged, + receiver(&pool, &applier, &cluster)? + ) + .await, + Err(CellGroupArchiveError::Replay(_)) + )); + assert_eq!(tails.usage().operations, 0); + // A cached final destination with damaged bytes must also refuse. Repair it + // only in this owned fault fixture; verification must independently resync it. + let physical = match &final_pin { + CellImagePin::Materialized(pin) => cluster + ._root + .path() + .join("node-3/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), + CellImagePin::Extents(pin) => { + let id = pin.manifest().extent(0).context("result extent")?; + cluster._root.path().join("extents-2").join(format!( + "e-{:08x}-{}", + id.length(), + hex::encode(id.bytes()) + )) + } + CellImagePin::Empty => anyhow::bail!("expected populated result"), + }; + let bytes = std::fs::read(&physical)?; + let mut corrupt = bytes.clone(); + corrupt[100] ^= 0x80; + std::fs::write(&physical, corrupt)?; + assert!(archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)? + ) + .await + .is_err()); + assert_eq!(tails.usage().operations, 0); + std::fs::write(physical, bytes)?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), current.marker()); + drop(recovered); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn physical_cancellation() -> Result<()> { + let gate = fault::Gate::new(None); + let mut cluster = Cluster::with_storage( + Default::default(), + tempfile::tempdir()?, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .await?; + cluster.elect().await?; + cluster.voters[2].paused = true; + let expected = checkpoint_fixture::enroll_images(&mut cluster, false, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store, + Path::from("tail-cancellation"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + false, + Some(2), + ) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + let (_, pin) = recovered + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("physical replay result")?; + let CellImagePin::Materialized(pin) = pin else { + anyhow::bail!("expected materialized fixture") + }; + let path = cluster + ._root + .path() + .join("node-3/images") + .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))); + drop(recovered); + // Only the replayed result is selected; checkpoint/base reads cannot enter + // this gate. The warm path first recomputes incoming LTX into private scratch. + gate.image_read_paths.lock().unwrap().push(path); + let release = gate.arm(fault::HOLD_IMAGE_READ); + let mut pending = Box::pin(archive.verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("replay ended before selected physical read") + } + } + assert_eq!(archive.usage().operations, 1); + assert_eq!(tails.usage().operations, 1); + assert_eq!(pool.usage().sources, 1); + drop(pending); // Stop the waiter while its physical job remains entered. + assert_eq!(archive.usage().operations, 1); + assert_eq!(tails.usage().operations, 1); + assert_eq!(pool.usage().sources, 1); + checkpoint_fixture::advance_partition(&mut cluster, "beta", &expected["beta"], false, Some(2)) + .await?; + assert!( + cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > current.marker().position().index + ); + for voter in &mut cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + let mut shutdown = Box::pin(cluster.domain.shutdown()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut shutdown) + .await + .is_err(), + "domain shutdown must join the held physical replay" + ); + assert_eq!(tails.usage().operations, 1); + drop(release); + tokio::time::timeout(Duration::from_secs(5), &mut shutdown).await?; + drop(shutdown); + assert_eq!(archive.usage().retained_bytes, 0); + assert_eq!(tails.usage().retained_bytes, 0); + assert_eq!(pool.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn extend_compacted(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let store = archive_store::Store::new(); + let archive = new_archive( + store.clone(), + Path::from("tail-extension"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let first = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let prefix = archive + .upload_tail(&cluster.domain, &tails, &base, &group, &first) + .await?; + let (prefix_path, prefix_index) = tail_index(store.as_ref(), &prefix).await?; + let checkpoint = source(&sources, &cluster)?.prepare_publication()?; + cluster.voters[0] + .owner + .as_mut() + .unwrap() + .publish_checkpoint(checkpoint)?; + cluster.drive_partition(false, Some(2)).await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(first.marker())); + assert_eq!( + raft::Storage::first_index(&group)?, + first.marker().position().index.get() + 1 + ); + checkpoint_fixture::advance_partition(&mut cluster, "beta", &expected["beta"], rooted, Some(2)) + .await?; + expected.insert( + "beta".into(), + images::image(Some(&expected["beta"]), "UPDATE value SET n=n+1")?, + ); + let latest = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + assert!( + matches!( + archive + .upload_tail(&cluster.domain, &tails, &base, &group, &latest) + .await, + Err(CellGroupArchiveError::Invalid("tail source committed cut")) + ), + "the original local prefix is genuinely unavailable" + ); + let count = store.puts.load(Ordering::Acquire); + let reference = archive + .extend_tail(&cluster.domain, &tails, &prefix, &group, &latest) + .await?; + assert_eq!( + store.puts.load(Ordering::Acquire), + count + 2, + "only the new entry and complete index require PUT" + ); + let (_, index) = tail_index(store.as_ref(), &reference).await?; + assert_eq!(index["entries"].as_array().unwrap().len(), 2); + assert_eq!(index["entries"][0], prefix_index["entries"][0]); + assert_eq!(index["checkpoint"], prefix_index["checkpoint"]); + // The flattened extension names the original checkpoint and all immutable + // entry objects directly; the old tail index is not a recovery dependency. + store.inner.delete(&prefix_path).await?; + let applier = destination(&mut cluster)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for voter in &mut cluster.voters[..2] { + drop(voter.owner.take()); + voter.log.close().await?; + } + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&pool, &applier, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), latest.marker()); + check_images(&cluster, recovered.manifest(), &expected).await?; + drop(recovered); + assert_eq!(tails.usage().retained_bytes, 0); + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs index 571581b78..fd78a9ad2 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -55,6 +55,19 @@ pub(super) async fn populate_images( let mut cluster = Cluster::new().await?; cluster.elect().await?; cluster.voters[2].paused = lagging; + let expected = enroll_images(&mut cluster, rooted, partition, additional, padding).await?; + Ok((cluster, expected)) +} + +/// Enroll into an existing elected group so archive tests can capture its empty +/// checkpoint first. Existing source/snapshot fixtures call this same path. +pub(super) async fn enroll_images( + cluster: &mut Cluster, + rooted: bool, + partition: bool, + additional: usize, + padding: usize, +) -> Result>> { let route = cluster.birth.route().clone(); if rooted { for n in 0..3 { @@ -121,7 +134,7 @@ pub(super) async fn populate_images( .await?; expected.insert(name.to_owned(), bytes); } - Ok((cluster, expected)) + Ok(expected) } fn extent_commitment( diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 581c77c10..7cdb2e132 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -450,6 +450,16 @@ fn prepare_cell_image( } fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result { + if apply.entry.get_entry_type() == raft::prelude::EntryType::EntryConfChangeV2 { + let group = voter.log.cell_group(&apply.base.marker().route().history)?; + return Ok(voter.tree.prepare_membership( + voter.pages.as_ref().context("fixture page owner")?, + &apply.base, + &apply.entry, + &raft::Storage::initial_state(&group)?.conf_state, + &group.membership()?, + )?); + } if apply.entry.data.is_empty() { return Ok(voter.tree.prepare_unchanged( voter.pages.as_ref().context("fixture page owner")?, diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 31ae56866..2a2e22add 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -509,6 +509,53 @@ covering local checkpoint and configured archive retention proofs, plus transfer pins. Shared engine purge must preserve other groups' live ranges. Readers and in-flight installs retain object pins until their physical owners finish. +CG-06 additionally implements committed-tail capture and independent data replay. +`upload_tail` copies the exact contiguous suffix to the engine-selected applied +root under the cache read lock, then encodes/uploads without that lock. Entries +have bounded canonical framing, including context and sync metadata. The complete +tail index names the original checkpoint, every immutable entry object, and the +final root/configuration/permanent membership. Missing ranges, unsupported types, +invalid cuts and configured byte/entry/cell limits refuse. Upload still returns +an untrusted locator rather than coverage authority. + +`extend_tail` reuses an existing archived prefix and captures only the local +suffix after its cut. Its flattened index directly names the original checkpoint +and all entry objects, so the previous tail index is not a recovery dependency. +This permits continued archival after local prefix compaction. Combined limits +are checked before new uploads; a full bounded tail requires a new checkpoint. +Physical shared-engine file reclamation remains a separate unqualified boundary. + +`verify_tail` recovers every checkpoint dependency and replays all entries through +the physical transition code shared with live apply. Live apply retains its +original ticket/commit/owner checks before and after preparation. Canonical joint +membership, leave and no-op entries advance group metadata without synthetic +cells or independent cell-sequence changes. Replayed final root, configuration +and membership must match the index; every final page/image is independently +verified and synced. No application callback or engine installation occurs. + +Materialized replay uses `GenerationStore::prepare_verified`: an existing cached +result cannot replace validation of the incoming LTX. The delta is recomputed in +admitted anonymous scratch before the destination is independently verified and +synced. `GenerationPin::verify_and_sync` also preserves the receiver's existing +independent-destination checks. Extent replay applies every delta and rechecks +the complete final image. A valid but wrong LTX delta or warm destination damage +must refuse, even when metadata and object hashes are well formed. + +A node-wide `CellGroupArchiveTailPool` separately bounds copied suffixes, decoded +entries, image work and retained base/predecessor/successor metadata. Its limits +are explicit and do not change the ordinary archive/checkpoint pool limits. The +returned `VerifiedCellGroupArchiveRecovery` retains this admission with physical +pins. Cancellation stops further replay work and retains entered physical owners; +domain shutdown joins them. Typed contention and poisoned-owner failures differ. + +This remains data-only recovery through an existing destination engine/applier. +Production integration needs an isolated data-verification storage owner, which +must not depend on a serving engine's advancing selected cut or surviving voter +disks. Published/GC-protected archive coverage, S3 settlement/outage admission, +archive-aware file reclamation and fenced fresh-incarnation seeding remain CG-06 +work. The task scope now explicitly includes the shared checkpoint/engine fixture +changes and new transition/tail modules; all required cases and oracles are kept. + ## Split and merge Use the same transition state machine for one parent to multiple child ranges diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 4bf1059a5..78d401ee4 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1722,14 +1722,22 @@ ], "scope": [ "crates/cellar/durability/raft/", - "crates/cellar-ltx/src/" + "crates/cellar-ltx/src/", + "crates/cellar/tests/support/cell_group_checkpoint.rs", + "crates/cellar/tests/support/cell_group_engine_fixture.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/archive/mod.rs", "crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs", "crates/cellar/durability/raft/cell_groups/archive/objects.rs", "crates/cellar/tests/native_cell_groups_archive.rs", - "crates/cellar/tests/support/cell_group_archive_store.rs" + "crates/cellar/tests/support/cell_group_archive_store.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs", + "crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs", + "crates/cellar/durability/raft/cell_groups/archive/replay.rs", + "crates/cellar/durability/raft/cell_groups/apply/transition.rs", + "crates/cellar/tests/support/cell_group_archive_tail.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From e774ee4bf461bc48fcbe9067002d7c831cef867e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 05:48:02 -0700 Subject: [PATCH 166/568] docs: record committed-tail recovery gate --- docs/design/multi-raft/cell-groups/handoff.md | 55 ++++++++++------ .../design/multi-raft/cell-groups/progress.md | 64 +++++++++++++++++++ 2 files changed, 100 insertions(+), 19 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 59b8979c9..80e2e672b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,25 +15,42 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial source is `650fc63892eab59e6fd5023fa90d5f361a8c7302`. -CG-06 is IN_PROGRESS: `raft/cell_groups/archive/` implements complete group -checkpoint objects, node-shared item/byte admission, immutable reuse/conditional -creation and independent physical recovery. Four exact archive cases and all -five snapshot regressions pass. All fifteen commands pass, including all-target -build/Clippy, formatting/layout, dependency/docs/preflight and 37 validator tests. -Its retained record under -`.cellar/cell-groups-evidence/CG-06-checkpoint-archive/qualification/` -binds 317 source hashes and two executables; independent verification passes. -See the current [progress](progress.md) and [archive contract](protocols.md#apply-checkpoint-and-archive). - -The checkpoint locator is untrusted; the verified receiver capability grants -neither group coverage nor S3 settlement, log deletion or fresh-incarnation -recovery. All five original CG-06 required cases and its complete task gate remain -pending. Next implement contiguous cell/control tails and a published, -GC-protected coverage owner, then bounded outage admission/S3 settlement, -archive-aware log reclamation and fenced unmanaged seeding. Shared dispatch still -refuses S3 durability. Current object framing follows bounded source chunks; -qualify coalescing and large-group bounds before performance/cost claims. +The latest partial source is `7a29c2a9bff5c2daddf37f53c50bc5d575e0b069`. +CG-06 is IN_PROGRESS: complete immutable checkpoints plus contiguous committed +tails now recover every cell and configuration through independent LTX/data +replay. `extend_tail` reuses archived entry references after actual local prefix +compaction, with only a new suffix entry/index PUT in the tested two-entry case. +The obsolete tail index is deleted before recovery; actual physical shared-engine +file purge is not qualified. No callback, synthetic cell or engine install occurs. + +All 22 commands pass: nine exact archive cases, 156 execution cases, five snapshot +cases and four generation regressions (174 distinct Rust cases), all-target +build/Clippy, formatting/layout, dependencies/docs/preflight and 37 validator tests. +The record under +`.cellar/cell-groups-evidence/CG-06-committed-tail/qualification/` +binds 323 source hashes, four executables, frozen inputs, both fixture failures and +the preceding checkpoint. Independent hashes/source/exact-selection verification +passes. See [progress](progress.md) and the +[archive contract](protocols.md#apply-checkpoint-and-archive). + +The original `archive_tails_cover_enrollment_and_controls` case passes with an +empty checkpoint, four real enrollments, later mutations, explicit joint demotion/ +leave and production leadership-transfer no-op. The other four original CG-06 +cases and its complete task gate remain pending. A cached result cannot replace +incoming LTX replay; wrong but correctly hashed deltas and warm image corruption +refuse. Physical-read cancellation retains admission while another cell commits; +domain shutdown joins the held worker before refunding. Source/receiver/archive +limits, readiness assertions and the exact nine-entry control trace are unchanged. + +Uploaded locators and verified physical state grant no archive coverage, serving +right, S3 settlement or disaster-recovery authorization. Next isolate the data +verifier from a serving engine's advancing cut and surviving voter disks. Then +build published/GC-protected coverage, bounded outage admission/S3 settlement, +archive-aware file reclamation and fenced unmanaged seeding under fresh group/cell +incarnations. Shared dispatch still refuses S3 durability. Qualify framing/ +coalescing and large-group limits before cost/throughput claims. Actual RustFS, +zero-quorum erasure/fencing, managed recovery and the public hard cutover remain +later required work. The previous checkpoint-only record at `650fc638` is retained. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 2db1b3b10..07929062a 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,70 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 committed-tail replay and extension after compaction + +Partial source checkpoint: `7a29c2a9bff5c2daddf37f53c50bc5d575e0b069`. +CG-06 remains IN_PROGRESS. A complete immutable tail now binds its original group +checkpoint, every contiguous entry and the final root/configuration/permanent +membership. Capture copies only the exact committed/applied suffix under the +engine read lock; encoding and object-store work happen afterward. Independent +verification recovers the checkpoint and replays actual cell-addressed LTX, +enrollment, no-op, joint-membership and leave entries without business callbacks, +synthetic cells or engine installation. Final roots, images and membership must +verify against the advertised complete cut. + +`extend_tail` reuses existing entry objects and captures only the newer local +suffix. Both image forms publish a real local checkpoint and assert the old +prefix is compacted before extension. Starting again from the original local +cut refuses; extension succeeds with exactly two PUTs for the new entry and +replacement index. The old tail index is deleted before independent recovery, +proving that the flattened replacement directly names its complete closure. +This is logical prefix compaction, not qualification of physical log-file purge. + +Live apply retains its original ticket/commit/owner checks around shared physical +transition code. The new membership-root preparation API also lets the lower-level +fixture handle real configuration entries. Materialized archive replay recomputes +incoming LTX in admitted scratch when its destination is cached, then independently +verifies/syncs that destination. A warm seal cannot substitute for the supplied +delta. Extent replay and every final closure retain their independent checks. +Replay admission and pins stay charged with the result and through cancelled +physical work. A materialized case holds the exact warm result-image read, drops +the caller, commits another cell on the healthy quorum, and proves domain shutdown +waits for release before every admission charge refunds. + +All 22 commands pass at the committed source: nine archive cases selected +individually, 156 execution cases, five snapshot cases and four generation +regressions, plus all-target build/Clippy, formatting/layout, dependency isolation, +docs, source preflight, plan validation and 37 Python tests. The 174 distinct Rust +cases run across four executables. The retained record +`.cellar/cell-groups-evidence/CG-06-committed-tail/qualification/checkpoint.json` +binds 323 source hashes, four binaries, frozen plan inputs, hashed outputs, both +failed fixture stages and the preceding checkpoint. Independent source/artifact/ +exact-selection verification passes. The original +`archive_tails_cover_enrollment_and_controls` requirement passes with four real +enrollments, later independent mutations, explicit joint demotion/leave and a +real transferred-leader term no-op. The other four original CG-06 cases remain +pending; no oracle, resource limit, readiness assertion, exact entry count or +timeout was relaxed. + +The retained failures first show a fixture decoding configuration bytes as cell +commands, then a direct campaign refused by a healthy leader lease. The fixture +now shares canonical membership-root preparation and uses production leadership +transfer. Seven other cases already passed before that final setup correction. +Warm wrong-LTX tests positively observe the ordinary cached `Exists` shortcut, +then require archive replay to reject a correctly hashed but wrong SQLite delta. +Missing/corrupt base or entry bytes, omitted/noncontiguous ranges, forged final roots +and damaged warm destinations also refuse without changing selected engine roots. + +**Reflection and next step:** verification still takes an existing destination +engine/applier. Build an isolated data-verification store independent of a serving +engine's advancing cut and surviving voter disks, then implement published, +GC-protected coverage and S3 settlement with bounded outage admission. Continue +archive-aware physical log reclamation and fenced fresh-incarnation seeding. +Shared dispatch still refuses S3 durability. Real RustFS/load and zero-quorum +host/disk-loss qualification, managed recovery, public SDK/fleet cutover and final +legacy removal remain open; this native data-recovery gate does not qualify them. + ## CG-06 independently recovered object-store checkpoints Partial source checkpoint: `650fc63892eab59e6fd5023fa90d5f361a8c7302`. From 14955f2dc2b01a6f6ab6b75b0122c812e5de6765 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 06:19:34 -0700 Subject: [PATCH 167/568] feat: isolate cell-group archive recovery stores --- crates/cellar-ltx/src/generation.rs | 8 +- .../durability/raft/cell_groups/apply/mod.rs | 42 ++- .../raft/cell_groups/archive/checkpoint.rs | 6 +- .../raft/cell_groups/archive/data_store.rs | 62 ++++ .../raft/cell_groups/archive/mod.rs | 14 +- .../raft/cell_groups/archive/replay.rs | 113 +++---- .../raft/cell_groups/archive/tail.rs | 2 +- .../cell_groups/archive/tail_admission.rs | 2 +- .../raft/cell_groups/checkpoint/mod.rs | 2 + .../raft/cell_groups/checkpoint/receive.rs | 299 +++++++++++++----- .../cell_groups/checkpoint/receive_archive.rs | 134 ++++++++ .../raft/cell_groups/checkpoint/wire.rs | 4 +- .../cellar/durability/raft/cell_groups/mod.rs | 7 +- .../durability/raft/cell_groups/root.rs | 74 ++++- .../tests/native_cell_groups_archive.rs | 156 ++++++--- .../support/cell_group_archive_isolation.rs | 233 ++++++++++++++ .../tests/support/cell_group_archive_tail.rs | 34 +- .../multi-raft/cell-groups/protocols.md | 40 ++- docs/design/multi-raft/cell-groups/tasks.json | 5 +- 19 files changed, 980 insertions(+), 257 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/data_store.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs create mode 100644 crates/cellar/tests/support/cell_group_archive_isolation.rs diff --git a/crates/cellar-ltx/src/generation.rs b/crates/cellar-ltx/src/generation.rs index b1ef490bc..af524fce5 100644 --- a/crates/cellar-ltx/src/generation.rs +++ b/crates/cellar-ltx/src/generation.rs @@ -670,7 +670,8 @@ impl GenerationStore { /// Apply and verify the supplied delta even if the destination already /// exists. A cached result cannot substitute for replay of untrusted LTX. /// Existing images use an independently admitted anonymous scratch image; - /// no selected image or inventory retention flag is changed. + /// existing inventory retention is preserved. New private results are + /// reclaimable once their returned pins are released, like received images. pub fn prepare_verified( &self, id: ImageId, @@ -678,7 +679,10 @@ impl GenerationStore { update: &ImageUpdate<'_>, ) -> Result { match self.prepare(id, base, update) { - Ok(pin) => return Ok(pin), + Ok(pin) => { + self.retire(pin.id())?; + return Ok(pin); + } Err(GenerationError::Exists) => (), Err(error) => return Err(error), } diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index 768eb3ebd..d1faf7abe 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -26,6 +26,31 @@ pub(super) struct Storage { max_effect_rows: u32, } +impl Storage { + pub(in crate::raft::cell_groups) fn new( + route: cellar_logic::cell_group::GroupRoute, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + limits.validate().map_err(storage)?; + images.validate()?; + if !(1..=1_000_000).contains(&max_effect_rows) { + return Err(ExecutionError::Invalid("apply effect summary limit")); + } + let tree = ManifestTree::new(route, manifest_limits).map_err(storage)?; + Ok(Self { + tree, + pages, + images, + limits, + max_effect_rows, + }) + } +} + /// Owns the already opened manifest directory and shared physical image stores. /// Construction performs no I/O. Opening/recovering the stores belongs on the /// owning execution domain; transfer the same page owner used for startup. @@ -53,16 +78,17 @@ impl CellGroupApplier { .group_manifest() .map_err(storage)? .ok_or(ExecutionError::Invalid("group manifest missing"))?; - let tree = ManifestTree::new(marker.route().clone(), manifest_limits).map_err(storage)?; + let state = Storage::new( + marker.route().clone(), + pages, + images, + manifest_limits, + limits, + max_effect_rows, + )?; Ok(Self { group, - storage: Arc::new(Mutex::new(Storage { - tree, - pages, - images, - limits, - max_effect_rows, - })), + storage: Arc::new(Mutex::new(state)), }) } diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs index f4f7ae953..b15435dac 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -208,7 +208,7 @@ impl CellGroupArchive { &self, domain: &ExecutionDomain, reference: &CellGroupArchiveReference, - receiver: CellGroupCheckpointReceiver, + receiver: CellGroupArchiveCheckpointReceiver, ) -> Result { domain .run(self.verify(domain, reference, receiver)) @@ -220,7 +220,7 @@ impl CellGroupArchive { &self, domain: &ExecutionDomain, reference: &CellGroupArchiveReference, - receiver: CellGroupCheckpointReceiver, + receiver: CellGroupArchiveCheckpointReceiver, ) -> Result { let permit = self.admit()?; self.verify_admitted(domain, reference, receiver, permit) @@ -261,7 +261,7 @@ impl CellGroupArchive { &self, domain: &ExecutionDomain, reference: &CellGroupArchiveReference, - mut receiver: CellGroupCheckpointReceiver, + mut receiver: CellGroupArchiveCheckpointReceiver, permit: Arc, ) -> Result { let bytes = self diff --git a/crates/cellar/durability/raft/cell_groups/archive/data_store.rs b/crates/cellar/durability/raft/cell_groups/archive/data_store.rs new file mode 100644 index 000000000..d9d88a01e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/data_store.rs @@ -0,0 +1,62 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Private physical preparation, without a Raft mapping or voter identity. +use super::super::{apply::Storage, root::ArchiveManifestOwner, *}; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::execution::ExecutionError; +use cellar_logic::cell_group::GroupRoute; +use std::sync::{Arc, Mutex}; + +/// Owns archive verification pages/images for one expected route. Open dedicated +/// physical directories on the owning blocking domain, separately from serving +/// stores. No ReplicaLog, member claim, +/// bootstrap permission or application callback is created or required. +#[derive(Clone)] +pub struct CellGroupArchiveStore { + route: GroupRoute, + pub(in crate::raft::cell_groups) storage: Arc>, + pub(in crate::raft::cell_groups) owner: Arc, +} + +impl CellGroupArchiveStore { + pub fn new( + route: GroupRoute, + pages: ManifestPageStore, + images: CellGroupApplyImages, + manifest_limits: ManifestLimits, + entry_limits: GroupEntryLimits, + max_effect_rows: u32, + ) -> Result { + let state = Storage::new( + route.clone(), + pages, + images, + manifest_limits, + entry_limits, + max_effect_rows, + )?; + Ok(Self { + route, + storage: Arc::new(Mutex::new(state)), + owner: Arc::new(ArchiveManifestOwner::new()), + }) + } + + pub fn route(&self) -> &GroupRoute { + &self.route + } + + /// Refuse further receives/replay. Entered blocking work keeps its physical + /// pins/admission until it exits; its execution domain owns the join. + pub fn close(&self) { + self.owner.close(); + } + + pub(in crate::raft::cell_groups) fn require_live( + &self, + ) -> Result<(), CellGroupCheckpointError> { + self.owner + .require_live() + .map_err(CellGroupCheckpointError::from) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 3cf0a65f0..302948a0b 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -4,10 +4,10 @@ //! and independently recovered physical state are separate capabilities. Neither //! authorizes S3 settlement, log truncation, membership or disaster recovery. -use super::{ - checkpoint::wire, CellGroupCheckpoint, CellGroupCheckpointError, CellGroupCheckpointReceiver, - GroupManifest, PreparedCellGroupCheckpoint, +pub use super::checkpoint::{ + CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint, }; +use super::{checkpoint::wire, CellGroupCheckpoint, CellGroupCheckpointError, GroupManifest}; use crate::{ asyncrt::ExecutionDomain, raft::budget::{Budget, Permit}, @@ -19,6 +19,8 @@ use sha2::{Digest, Sha256}; use std::{fmt, sync::Arc}; mod checkpoint; +mod data_store; +pub use data_store::CellGroupArchiveStore; mod objects; mod replay; mod tail; @@ -146,7 +148,7 @@ impl CellGroupArchiveReference { /// is a checkpoint-only proof, without a tail, published head or remote GC lease. pub struct VerifiedCellGroupArchiveCheckpoint { reference: CellGroupArchiveReference, - checkpoint: PreparedCellGroupCheckpoint, + checkpoint: PreparedCellGroupArchiveCheckpoint, } impl VerifiedCellGroupArchiveCheckpoint { @@ -154,11 +156,11 @@ impl VerifiedCellGroupArchiveCheckpoint { &self.reference } - pub fn checkpoint(&self) -> &PreparedCellGroupCheckpoint { + pub fn checkpoint(&self) -> &PreparedCellGroupArchiveCheckpoint { &self.checkpoint } - pub fn into_checkpoint(self) -> PreparedCellGroupCheckpoint { + pub fn into_checkpoint(self) -> PreparedCellGroupArchiveCheckpoint { self.checkpoint } } diff --git a/crates/cellar/durability/raft/cell_groups/archive/replay.rs b/crates/cellar/durability/raft/cell_groups/archive/replay.rs index 674d0e07e..5e92940d5 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/replay.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/replay.rs @@ -51,12 +51,14 @@ pub(super) fn replay( reservation: Arc, cancelled: Arc, ) -> Result { - let applier = checkpoint.checkpoint().applier().clone(); - let selected = applier - .group - .group_manifest() - .map_err(CellGroupCheckpointError::from)?; + let store = checkpoint.checkpoint().store().clone(); + store.require_live()?; let mut root = checkpoint.checkpoint().manifest().clone(); + if !store.owner.owns(&root) { + return Err(CellGroupArchiveError::Invalid( + "archive replay physical owner", + )); + } let mut conf = checkpoint .checkpoint() .snapshot() @@ -64,7 +66,7 @@ pub(super) fn replay( .get_conf_state() .clone(); let mut membership = checkpoint.checkpoint().membership().clone(); - let mut owner = applier.storage.try_lock().map_err(|error| match error { + let mut owner = store.storage.try_lock().map_err(|error| match error { std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, std::sync::TryLockError::Poisoned(_) => { CellGroupArchiveError::Invalid("poisoned archive replay owner") @@ -79,10 +81,7 @@ pub(super) fn replay( if cancelled.load(Ordering::Acquire) { return Err(CellGroupArchiveError::Stopped); } - applier - .group - .cache_revision() - .map_err(CellGroupCheckpointError::from)?; + store.require_live()?; // Control replay is the same canonical pure transition used by the live // owner; cell bytes still go through the ordinary LTX/base-chain checks. let change = if entry.get_entry_type() == EntryType::EntryConfChangeV2 { @@ -126,72 +125,50 @@ pub(super) fn replay( .. } = &mut *owner; let verified = tree - .prepare_complete( - &applier.group, - pages, - root.marker().root(), - root.marker().position(), - root.marker().membership_revision(), - |leaf| { - if cancelled.load(Ordering::Acquire) { - return Err(super::super::ManifestError::Invalid( - "archive replay stopped", - )); + .prepare_archive_complete(&store.owner, pages, root.marker().clone(), &mut |leaf| { + if cancelled.load(Ordering::Acquire) { + return Err(super::super::ManifestError::Invalid( + "archive replay stopped", + )); + } + let (_, pin) = root + .pin_cell(&leaf.version.history.cell) + .filter(|(expected, _)| *expected == leaf) + .ok_or(super::super::ManifestError::Invalid( + "replayed leaf missing", + ))?; + match (&*images, &pin) { + (_, CellImagePin::Empty) => (), + (CellGroupApplyImages::Materialized { .. }, CellImagePin::Materialized(pin)) => { + pin.verify_and_sync() + .map_err(|e| super::super::ManifestError::Io(std::io::Error::other(e)))?; } - let (_, pin) = root - .pin_cell(&leaf.version.history.cell) - .filter(|(expected, _)| *expected == leaf) - .ok_or(super::super::ManifestError::Invalid( - "replayed leaf missing", - ))?; - match (&*images, &pin) { - (_, CellImagePin::Empty) => (), - ( - CellGroupApplyImages::Materialized { .. }, - CellImagePin::Materialized(pin), - ) => { - pin.verify_and_sync().map_err(|e| { - super::super::ManifestError::Io(std::io::Error::other(e)) - })?; - } - (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(pin)) => { - if pin.reference().encoded_length() > 1024 * 1024 { - return Err(super::super::ManifestError::Invalid( - "replay extent metadata limit", - )); - } - images - .seal_image_root( - pin.manifest().clone(), - &mut vec![0; cellar_ltx::internal::EXTENT_BYTES as usize], - ) - .map_err(|e| { - super::super::ManifestError::Io(std::io::Error::other(e)) - })?; - } - _ => { + (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(pin)) => { + if pin.reference().encoded_length() > 1024 * 1024 { return Err(super::super::ManifestError::Invalid( - "replay representation", - )) + "replay extent metadata limit", + )); } + images + .seal_image_root( + pin.manifest().clone(), + &mut vec![0; cellar_ltx::internal::EXTENT_BYTES as usize], + ) + .map_err(|e| super::super::ManifestError::Io(std::io::Error::other(e)))?; } - Ok(pin) - }, - ) + _ => { + return Err(super::super::ManifestError::Invalid( + "replay representation", + )) + } + } + Ok(pin) + }) .map_err(CellGroupCheckpointError::from)?; if cancelled.load(Ordering::Acquire) { return Err(CellGroupArchiveError::Stopped); } - if applier - .group - .group_manifest() - .map_err(CellGroupCheckpointError::from)? - != selected - { - return Err(CellGroupArchiveError::Invalid( - "replay destination selection changed", - )); - } + store.require_live()?; drop(owner); Ok(VerifiedCellGroupArchiveRecovery { reference, diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs index a911ee093..703f87ad4 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/tail.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -413,7 +413,7 @@ impl CellGroupArchive { domain: &ExecutionDomain, pool: &CellGroupArchiveTailPool, reference: &CellGroupArchiveTailReference, - receiver: CellGroupCheckpointReceiver, + receiver: CellGroupArchiveCheckpointReceiver, ) -> Result { domain .run(async { diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs index aba40dcfc..92c8628e0 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/tail_admission.rs @@ -13,7 +13,7 @@ pub struct CellGroupArchiveTailLimits { pub max_entries: usize, pub max_tail_bytes: usize, pub max_cells: u32, - /// Must cover the destination applier's captured image workspace bound. + /// Must cover the destination store's captured image workspace bound. pub max_apply_bytes: usize, } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 0cb04a60a..acc730fcb 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -15,12 +15,14 @@ mod header; mod local; mod prepared; mod receive; +mod receive_archive; pub(crate) mod wire; use header::Header; pub use local::PreparedLocalCellGroupCheckpoint; pub(crate) use prepared::persisted_marker; pub use prepared::PreparedCellGroupCheckpoint; pub use receive::CellGroupCheckpointReceiver; +pub use receive_archive::{CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint}; use sha2::{Digest, Sha256}; use std::sync::Arc; diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs index c752b70df..9621c7e6a 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -19,50 +19,223 @@ use std::{collections::BTreeMap, path::PathBuf}; // decoding, header/page encoding, scratch verification and traversal stacks. const RECEIVE_WORKSPACE: usize = 2 * EXTENT_BYTES as usize + 2 * 1024 * 1024; +pub(super) trait Target { + fn storage(&self) -> &Arc>; + + fn route(&self) -> &cellar_logic::cell_group::GroupRoute; + + fn require_live(&self) -> Result<()>; + + fn check_header(&self, header: &Header) -> Result<()>; + + fn check_finish(&self, header: &Header) -> Result<()>; + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut super::super::ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) + -> std::result::Result, + ) -> Result; +} + +struct EngineTarget { + applier: CellGroupApplier, + base: GroupManifest, +} + +impl Target for EngineTarget { + fn storage(&self) -> &Arc> { + &self.applier.storage + } + + fn route(&self) -> &cellar_logic::cell_group::GroupRoute { + self.base.route() + } + + fn require_live(&self) -> Result<()> { + self.applier.group.cache_revision()?; + Ok(()) + } + + fn check_header(&self, header: &Header) -> Result<()> { + if header.root.position().index < self.base.position().index { + return Err(invalid("receiver history/range/cut")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + Ok(()) + } + + fn check_finish(&self, header: &Header) -> Result<()> { + if self.applier.group.group_manifest()?.as_ref() != Some(&self.base) { + return Err(invalid("incomplete closure or changed receiver base")); + } + header + .membership + .succeeds(&self.applier.group.membership()?)?; + Ok(()) + } + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut super::super::ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) + -> std::result::Result, + ) -> Result { + let mut root = tree.prepare_complete( + &self.applier.group, + pages, + header.root.root(), + header.root.position(), + header.root.membership_revision(), + image, + )?; + root.base = Some(self.base.clone()); + Ok(root) + } +} + impl CellGroupCheckpointPool { - /// Share this node pool across both source and receiver owners. The existing - /// max_sources setting bounds their combined count. The scratch pool must - /// likewise be shared by all receivers using this volume. + /// Engine-bound receiving retains the exact selected base and every + /// membership/installation guard. Share source/receiver and scratch pools. pub fn receive( &self, applier: CellGroupApplier, scratch: ScratchPool, directory: PathBuf, ) -> Result { + let base = applier + .group + .group_manifest()? + .ok_or(invalid("receiver base"))?; + Ok(CellGroupCheckpointReceiver(State::new( + self, + EngineTarget { applier, base }, + scratch, + directory, + )?)) + } +} + +/// Engine-bound closure receiver. Finishing supplies storage preparation only; +/// the Raft owner's exact snapshot effect still controls atomic installation. +pub struct CellGroupCheckpointReceiver(State); + +pub(crate) trait WireReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result; + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()>; +} + +impl WireReceiver for CellGroupCheckpointReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + self.0.wire_extent(cell, ordinal, digest, length) + } + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } +} + +impl CellGroupCheckpointReceiver { + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } + + pub fn finish(self) -> Result { + let received = self.0.finish()?; + PreparedCellGroupCheckpoint::new( + received.target.applier, + received.root, + received.header, + received.reservation, + ) + } +} + +pub(super) struct Received { + pub target: T, + pub root: PreparedGroupManifest, + pub header: Header, + pub reservation: Permit, +} + +/// Common physical decoder. Destination-specific authority checks remain in +/// sealed typed targets, and each public wrapper has a distinct output type. +pub(super) struct State { + target: T, + scratch: ScratchPool, + directory: PathBuf, + limits: CellGroupCheckpointLimits, + header: Option
, + part: Option, + cell: Option, + images: BTreeMap, + pages: Vec, + last_key: Option, + transferred: u64, + failed: bool, + reservation: Permit, +} + +impl State { + pub(super) fn new( + pool: &CellGroupCheckpointPool, + target: T, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { if !directory.is_absolute() || directory.as_os_str().len() > 4096 { return Err(CellGroupCheckpointError::Invalid( "receive scratch directory", )); } - let cells = self.limits.max_cells as usize; - let extents = usize::try_from(self.limits.max_transfer_bytes.div_ceil(EXTENT_BYTES)) + let cells = pool.limits.max_cells as usize; + let extents = usize::try_from(pool.limits.max_transfer_bytes.div_ceil(EXTENT_BYTES)) .map_err(|_| CellGroupCheckpointError::Limit)?; - let bytes = super::super::ManifestPins::maximum_bytes(self.limits.max_cells) + let bytes = super::super::ManifestPins::maximum_bytes(pool.limits.max_cells) .checked_mul(3) .and_then(|n| n.checked_add(cells.checked_mul(4096)?)) .and_then(|n| n.checked_add(extents.checked_mul(2048)?)) .and_then(|n| n.checked_add(RECEIVE_WORKSPACE)) .ok_or(CellGroupCheckpointError::Limit)?; - let reservation = self + let reservation = pool .budget .acquire(bytes) .ok_or(CellGroupCheckpointError::Busy)?; - let base = applier - .group - .group_manifest()? - .ok_or(CellGroupCheckpointError::Invalid("receiver base"))?; - applier - .storage + target.require_live()?; + target + .storage() .lock() - .map_err(|_| CellGroupCheckpointError::Invalid("receiver page owner"))? + .map_err(|_| invalid("receiver page owner"))? .pages .require_live()?; - Ok(CellGroupCheckpointReceiver { - applier, - base, + Ok(Self { + target, scratch, directory, - limits: self.limits, + limits: pool.limits, header: None, part: None, cell: None, @@ -76,26 +249,6 @@ impl CellGroupCheckpointPool { } } -/// Owns all staged page/image pins and the full receive reservation. Appending -/// and finishing belong on the same joined blocking owner. No partial state is -/// published, and no source pin is accepted as a receiver durability proof. -pub struct CellGroupCheckpointReceiver { - applier: CellGroupApplier, - base: GroupManifest, - scratch: ScratchPool, - directory: PathBuf, - limits: CellGroupCheckpointLimits, - header: Option
, - part: Option, - cell: Option, - images: BTreeMap, - pages: Vec, - last_key: Option, - transferred: u64, - failed: bool, - reservation: Permit, -} - struct ReceivingCell { leaf: CellManifestLeaf, view: Option, @@ -141,7 +294,7 @@ fn invalid(message: &'static str) -> CellGroupCheckpointError { CellGroupCheckpointError::Invalid(message) } -impl CellGroupCheckpointReceiver { +impl State { pub(super) fn wire_extent( &self, cell: &CellId, @@ -176,7 +329,7 @@ impl CellGroupCheckpointReceiver { } fn append_inner(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { - self.applier.group.cache_revision()?; + self.target.require_live()?; if chunk.bytes.is_empty() || chunk.bytes.len() > self.limits.max_chunk_bytes { return Err(CellGroupCheckpointError::Limit); } @@ -259,8 +412,8 @@ impl CellGroupCheckpointReceiver { ) .map_err(|_| invalid("received image identity"))?; let storage = self - .applier - .storage + .target + .storage() .lock() .map_err(|_| invalid("receiver storage"))?; let super::super::CellGroupApplyImages::Materialized { images, .. } = @@ -332,15 +485,12 @@ impl CellGroupCheckpointReceiver { match (part.kind, part.data) { (Kind::Header, Data::Bytes(bytes)) => { let header = Header::decode(&bytes)?; - if header.root.route() != self.base.route() + if header.root.route() != self.target.route() || header.root.root().cells() > self.limits.max_cells - || header.root.position().index < self.base.position().index { return Err(invalid("receiver history/range/cut")); } - header - .membership - .succeeds(&self.applier.group.membership()?)?; + self.target.check_header(&header)?; self.header = Some(header); } (Kind::Page(digest), Data::Bytes(bytes)) => { @@ -365,8 +515,8 @@ impl CellGroupCheckpointReceiver { return Err(invalid("checkpoint leaf order/cut")); } let mut storage = self - .applier - .storage + .target + .storage() .lock() .map_err(|_| invalid("receiver storage"))?; storage.pages.put(digest, &bytes)?; @@ -401,7 +551,7 @@ impl CellGroupCheckpointReceiver { history: super::super::cell_image_history_digest( &pending.leaf.version.history, ), - schema: self.base.route().schema_digest, + schema: self.target.route().schema_digest, epoch: image.epoch, txid: image.txid, checksum: image.checksum, @@ -414,8 +564,8 @@ impl CellGroupCheckpointReceiver { return Err(invalid("extent image binding")); } let storage = self - .applier - .storage + .target + .storage() .lock() .map_err(|_| invalid("receiver storage"))?; let super::super::CellGroupApplyImages::Extents { images, .. } = &storage.images @@ -429,8 +579,8 @@ impl CellGroupCheckpointReceiver { let pending = self.cell.as_mut().ok_or(invalid("extent cell"))?; let view = pending.view.as_ref().ok_or(invalid("extent view"))?; let storage = self - .applier - .storage + .target + .storage() .lock() .map_err(|_| invalid("receiver storage"))?; let super::super::CellGroupApplyImages::Extents { images, .. } = @@ -461,9 +611,9 @@ impl CellGroupCheckpointReceiver { fn finish_cell(&mut self, pin: CellImagePin) -> Result<()> { let pending = self.cell.take().ok_or(invalid("missing receive cell"))?; - pin.check(&pending.leaf, self.base.route().schema_digest)?; + pin.check(&pending.leaf, self.target.route().schema_digest)?; let key = cell_routing_key( - self.base.route().history.namespace, + self.target.route().history.namespace, &pending.leaf.version.history.cell, ); if self.images.insert(key, (pending.leaf, pin)).is_some() { @@ -472,27 +622,22 @@ impl CellGroupCheckpointReceiver { Ok(()) } - /// Verify the complete authenticated tree against every independently sealed - /// destination image and sync its page closure. This still does not install - /// it: only the Raft owner's exact snapshot effect may publish the result. - pub fn finish(self) -> Result { + pub(super) fn finish(self) -> Result> { if self.failed || self.part.is_some() || self.cell.is_some() { return Err(invalid("incomplete checkpoint")); } let header = self.header.ok_or(invalid("missing checkpoint header"))?; if self.images.len() != header.root.root().cells() as usize || self.pages.len() != self.images.len() - || self.applier.group.group_manifest()?.as_ref() != Some(&self.base) { return Err(invalid("incomplete closure or changed receiver base")); } - header - .membership - .succeeds(&self.applier.group.membership()?)?; + self.target.require_live()?; + self.target.check_finish(&header)?; let root = { let mut storage = self - .applier - .storage + .target + .storage() .lock() .map_err(|_| invalid("receiver storage"))?; let tree = ManifestTree::new( @@ -502,13 +647,8 @@ impl CellGroupCheckpointReceiver { max_depth: 32, }, )?; - let mut root = tree.prepare_complete( - &self.applier.group, - &mut storage.pages, - header.root.root(), - header.root.position(), - header.root.membership_revision(), - |leaf| { + self.target + .prepare(&tree, &mut storage.pages, &header, &mut |leaf| { let key = cell_routing_key( header.root.route().history.namespace, &leaf.version.history.cell, @@ -520,11 +660,14 @@ impl CellGroupCheckpointReceiver { .ok_or(super::super::ManifestError::Invalid( "missing received cell image", )) - }, - )?; - root.base = Some(self.base); - root + })? }; - PreparedCellGroupCheckpoint::new(self.applier, root, header, self.reservation) + self.target.require_live()?; + Ok(Received { + target: self.target, + root, + header, + reservation: self.reservation, + }) } } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs new file mode 100644 index 000000000..4ae18ddd4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs @@ -0,0 +1,134 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::super::{ + CellGroupArchiveStore, CellManifestLeaf, ManifestError, ManifestPageStore, ManifestTree, +}; +use super::receive::{State, Target, WireReceiver}; +use super::*; +use cellar_logic::cell_group::GroupRoute; +use cellar_ltx::stream_snapshot::ScratchPool; +use std::{path::PathBuf, sync::Mutex}; + +impl Target for CellGroupArchiveStore { + fn storage(&self) -> &Arc> { + &self.storage + } + + fn route(&self) -> &GroupRoute { + CellGroupArchiveStore::route(self) + } + + fn require_live(&self) -> Result<()> { + CellGroupArchiveStore::require_live(self) + } + + fn check_header(&self, _header: &Header) -> Result<()> { + self.require_live() + } + + fn check_finish(&self, _header: &Header) -> Result<()> { + self.require_live() + } + + fn prepare( + &self, + tree: &ManifestTree, + pages: &mut ManifestPageStore, + header: &Header, + image: &mut dyn FnMut( + &CellManifestLeaf, + ) -> std::result::Result, + ) -> Result { + Ok(tree.prepare_archive_complete(&self.owner, pages, header.root.clone(), image)?) + } +} + +impl CellGroupCheckpointPool { + /// Receive into private data stores, without a Raft mapping, surviving voter + /// disk or applied-cut observation. Shares the node's source/receiver quota. + pub fn receive_archive( + &self, + store: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + Ok(CellGroupArchiveCheckpointReceiver(State::new( + self, store, scratch, directory, + )?)) + } +} + +/// Physical archive receiver. It cannot produce the engine-bound checkpoint +/// capability accepted by SnapshotChange or the Raft owner. +pub struct CellGroupArchiveCheckpointReceiver(State); + +impl WireReceiver for CellGroupArchiveCheckpointReceiver { + fn wire_extent( + &self, + cell: &CellId, + ordinal: u32, + digest: [u8; 32], + length: u64, + ) -> Result { + self.0.wire_extent(cell, ordinal, digest, length) + } + + fn append_chunk(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } +} + +impl CellGroupArchiveCheckpointReceiver { + pub fn append(&mut self, chunk: CellGroupCheckpointChunk<'_>) -> Result<()> { + self.0.append(chunk) + } + + pub fn finish(self) -> Result { + let received = self.0.finish()?; + let snapshot = received.header.snapshot()?; + Ok(PreparedCellGroupArchiveCheckpoint(Arc::new(Inner { + store: received.target, + root: received.root, + header: received.header, + snapshot, + _reservation: received.reservation, + }))) + } +} + +struct Inner { + store: CellGroupArchiveStore, + root: PreparedGroupManifest, + header: Header, + snapshot: raft::prelude::Snapshot, + _reservation: Permit, +} + +/// Complete archive-local closure with no engine publication or serving right. +/// Clones share physical owners and admission, and have a distinct type from a +/// received Raft snapshot. Private owner tokens refuse every live engine. +#[derive(Clone)] +pub struct PreparedCellGroupArchiveCheckpoint(Arc); + +impl PreparedCellGroupArchiveCheckpoint { + pub fn marker(&self) -> &GroupManifest { + self.0.root.marker() + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.0.root + } + + /// Snapshot metadata only; copying it does not grant installation permission. + pub fn snapshot(&self) -> &raft::prelude::Snapshot { + &self.0.snapshot + } + + pub fn store(&self) -> &CellGroupArchiveStore { + &self.0.store + } + + pub(crate) fn membership(&self) -> &GroupMembership { + &self.0.header.membership + } +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs index 296da0963..584dbd0dd 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/wire.rs @@ -57,7 +57,7 @@ impl Chunk { } } - pub fn append(self, receiver: &mut CellGroupCheckpointReceiver) -> Result<()> { + pub fn append(self, receiver: &mut impl super::receive::WireReceiver) -> Result<()> { let cell = match &self.part { Part::Materialized { cell } | Part::ExtentManifest { cell } @@ -83,7 +83,7 @@ impl Chunk { id: receiver.wire_extent(cell.as_ref().unwrap(), *ordinal, *digest, self.length)?, }, }; - receiver.append(CellGroupCheckpointChunk { + receiver.append_chunk(CellGroupCheckpointChunk { part, offset: self.offset, length: self.length, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 98b5941a1..a305b5abf 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -17,9 +17,10 @@ pub use crate::raft::transport::{ mod apply; mod archive; pub use archive::{ - CellGroupArchive, CellGroupArchiveError, CellGroupArchiveLimits, CellGroupArchivePool, - CellGroupArchiveReference, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, - CellGroupArchiveTailReference, CellGroupArchiveUsage, VerifiedCellGroupArchiveCheckpoint, + CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveError, + CellGroupArchiveLimits, CellGroupArchivePool, CellGroupArchiveReference, CellGroupArchiveStore, + CellGroupArchiveTailLimits, CellGroupArchiveTailPool, CellGroupArchiveTailReference, + CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveRecovery, }; pub(crate) mod checkpoint; diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index ae4808c0b..3a48ba535 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -93,6 +93,38 @@ pub struct PreparedGroupManifest { retention: Arc, } +/// Private storage identity for archive preparation, with no Raft engine, +/// member, disk lineage or publication permission. Never accept a caller's token. +pub(super) struct ArchiveManifestOwner { + identity: Arc<()>, + closed: std::sync::atomic::AtomicBool, +} + +impl ArchiveManifestOwner { + pub(super) fn new() -> Self { + Self { + identity: Arc::new(()), + closed: std::sync::atomic::AtomicBool::new(false), + } + } + + pub(super) fn require_live(&self) -> Result<(), ManifestError> { + if self.closed.load(std::sync::atomic::Ordering::Acquire) { + return Err(ManifestError::Invalid("closed archive store")); + } + Ok(()) + } + + pub(super) fn close(&self) { + self.closed + .store(true, std::sync::atomic::Ordering::Release); + } + + pub(super) fn owns(&self, root: &PreparedGroupManifest) -> bool { + Arc::ptr_eq(&self.identity, &root.engine_owner) + } +} + impl PreparedGroupManifest { pub fn marker(&self) -> &GroupManifest { &self.marker @@ -224,9 +256,8 @@ impl ManifestTree { if !marker.matches_group(&group.identity().scope, group.birth_digest) { return Err(ManifestError::Invalid("root/engine identity")); } - self.verify(store, root, position, |_| Ok(()))?; - let pins = ManifestPins::complete(self, store, root, &mut image)?; - store.sync_directory()?; + let prepared = + self.prepare_physical(store, marker.clone(), group.owner.clone(), &mut image)?; // Recover physical ownership only from the engine's selected marker. // A prospective checkpoint or mutation remains a preparation capability. if group @@ -235,7 +266,7 @@ impl ManifestTree { .as_ref() == Some(&marker) { - store.retention.recover(group, &marker, &pins)?; + store.retention.recover(group, &marker, &prepared.images)?; } if group .checkpoint_manifest() @@ -243,12 +274,43 @@ impl ManifestTree { .as_ref() == Some(&marker) { - store.retention.recover_checkpoint(group, &marker, &pins)?; + store + .retention + .recover_checkpoint(group, &marker, &prepared.images)?; } + Ok(prepared) + } + + pub(super) fn prepare_archive_complete( + &self, + owner: &ArchiveManifestOwner, + store: &mut ManifestPageStore, + marker: GroupManifest, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + owner.require_live()?; + if marker.route() != self.route() { + return Err(ManifestError::Invalid("archive root route")); + } + self.prepare_physical(store, marker, owner.identity.clone(), image) + } + + fn prepare_physical( + &self, + store: &mut ManifestPageStore, + marker: GroupManifest, + owner: Arc<()>, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result { + marker.check()?; + store.require_live()?; + self.verify(store, marker.root(), marker.position(), |_| Ok(()))?; + let pins = ManifestPins::complete(self, store, marker.root(), &mut |leaf| image(leaf))?; + store.sync_directory()?; Ok(PreparedGroupManifest { marker, images: pins, - engine_owner: group.owner.clone(), + engine_owner: owner, base: None, page_owner: store.owner.clone(), retention: store.retention.clone(), diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 25c647274..6533c6426 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -18,6 +18,8 @@ mod fault; #[path = "support/raft_ltx_fixture.rs"] #[allow(dead_code)] mod images; +#[path = "support/cell_group_archive_isolation.rs"] +mod isolation_fixture; #[path = "support/cell_group_archive_tail.rs"] mod tail_fixture; @@ -60,6 +62,22 @@ async fn archive_tail_extension_survives_local_prefix_compaction() -> Result<()> Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_data_recovers_after_all_voter_disks_erased() -> Result<()> { + for rooted in [false, true] { + isolation_fixture::erased_voters(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_verification_isolated_from_serving_cut() -> Result<()> { + for rooted in [false, true] { + isolation_fixture::moving_cut(rooted).await?; + } + Ok(()) +} + use cellar_ltx::{ generation::ReadServices, internal::{CacheLimits, EXTENT_BYTES}, @@ -69,51 +87,97 @@ use futures_util::TryStreamExt; use object_store::{path::Path, ObjectStore}; use std::sync::atomic::Ordering; -fn destination(cluster: &mut Cluster) -> Result { - let voter = &mut cluster.voters[2]; - drop(voter.owner.take()); - drop(voter.held.take()); - let services = ReadServices { - now_unix_millis: Arc::new(|| 10), - fill_random: Arc::new(|bytes| bytes.fill(17)), - }; - let images = match voter.extents.clone() { - Some(images) => CellGroupApplyImages::Extents { - images, - services, - limits: Default::default(), - cache_directory: cluster._root.path().join("cache"), - cache_pool: ScratchPool::new(ScratchLimits { - max_bytes: 32 * 1024 * 1024, - max_files: 1, - })?, - cache: CacheLimits { - max_cached_bytes: EXTENT_BYTES, - max_cached_extents: 1, - }, - }, - None => CellGroupApplyImages::Materialized { - images: voter.images.clone(), - services, - }, - }; - Ok(CellGroupApplier::new( - voter.log.cell_group(&cluster.birth.route().history)?, - voter.pages.take().context("destination pages")?, - images, - ManifestLimits::default(), - GroupEntryLimits::default(), - 100, - )?) +struct ArchiveDestination { + store: CellGroupArchiveStore, + images: GenerationStore, +} + +async fn destination(cluster: &Cluster) -> Result { + destination_with_filesystem(cluster, Arc::new(DirectFileSystem)).await +} + +async fn destination_with_filesystem( + cluster: &Cluster, + filesystem: Arc, +) -> Result { + let rooted = cluster + .voters + .first() + .is_some_and(|voter| voter.extents.is_some()); + destination_mode(cluster, rooted, filesystem).await +} + +async fn destination_mode( + cluster: &Cluster, + rooted: bool, + filesystem: Arc, +) -> Result { + let base = cluster._root.path().join("archive-data"); + let route = cluster.birth.route().clone(); + cluster + .domain + .blocking(move || { + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let generations = + GenerationStore::open(base.join("images"), filesystem.clone(), Default::default())?; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: cellar_ltx::internal::NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: generations.clone(), + services, + } + }; + let store = CellGroupArchiveStore::new( + route, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?; + Ok(ArchiveDestination { + store, + images: generations, + }) + })? + .await? } fn receiver( pool: &CellGroupCheckpointPool, - applier: &CellGroupApplier, + destination: &ArchiveDestination, cluster: &Cluster, -) -> Result { - Ok(pool.receive( - applier.clone(), +) -> Result { + Ok(pool.receive_archive( + destination.store.clone(), ScratchPool::new(ScratchLimits { max_bytes: 32 * 1024 * 1024, max_files: 1, @@ -235,7 +299,7 @@ async fn checkpoint_archive_roundtrip_and_reuse() -> Result<()> { .manifest() .marker() .clone(); - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let receive_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; // Prove the independent receiver uses remote data after its source engine // has closed; no borrowed source pin can satisfy this recovery. @@ -292,7 +356,7 @@ async fn checkpoint_archive_roundtrip_and_reuse() -> Result<()> { #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { for rooted in [false, true] { - let (mut cluster, _) = + let (cluster, _) = checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let history = cluster.birth.route().history; @@ -347,7 +411,7 @@ async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { .into_iter() .last() .context("held beta")?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let gate = store.hold(dependency); let mut pending = Box::pin(archive.verify_checkpoint( @@ -467,7 +531,7 @@ async fn checkpoint_archive_missing_or_corrupt_cell_refuses() -> Result<()> { .last() .context("beta dependency")?; let bytes = store.get(&dependency).await?.bytes().await?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; store.delete(&dependency).await?; assert!(matches!( @@ -543,7 +607,7 @@ async fn forged_index( #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { for rooted in [false, true] { - let (mut cluster, _) = + let (cluster, _) = checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( @@ -561,7 +625,7 @@ async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { .log .cell_group(&cluster.birth.route().history)?; let base = group.group_manifest()?.context("destination base")?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; for attack in 0..4 { let mut value = original.clone(); diff --git a/crates/cellar/tests/support/cell_group_archive_isolation.rs b/crates/cellar/tests/support/cell_group_archive_isolation.rs new file mode 100644 index 000000000..98683c76b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_isolation.rs @@ -0,0 +1,233 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Data recovery deliberately has neither an engine nor serving authority. +use super::*; +use cellar_logic::consensus::OperationId; + +pub(super) async fn erased_voters(rooted: bool) -> Result<()> { + let (mut cluster, mut expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let remote = archive_store::Store::new(); + let archive = new_archive( + remote, + Path::from("erased-voters"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let base = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + expected.insert( + "alpha".into(), + images::image(Some(&expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); + let marker = current.marker().clone(); + let membership = group.membership()?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let reference = archive + .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) + .await?; + // No source physical pin, open GroupLog or voter survives the disk erasure. + drop(current); + drop(group); + for voter in &mut cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + drop(std::mem::take(&mut cluster.voters)); + cluster.network.clear(); + let directory = cluster._root.path().to_owned(); + cluster + .domain + .blocking(move || -> Result<()> { + for n in 1..=3 { + let path = directory.join(format!("node-{n}")); + assert!(path.join("log").exists()); + assert!(path.join("images").exists()); + std::fs::remove_dir_all(&path)?; + assert!(!path.exists()); + let extents = directory.join(format!("extents-{}", n - 1)); + if rooted { + assert!(extents.exists()); + std::fs::remove_dir_all(&extents)?; + assert!(!extents.exists()); + } + } + assert!(!directory.join("archive-data").exists()); + Ok(()) + })? + .await??; + // Open only physical data stores, after erasure. No bootstrap permit, local + // member/disk identity, Raft metadata or replacement genesis is fabricated. + let destination = destination_mode(&cluster, rooted, Arc::new(DirectFileSystem)).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(recovered.manifest().marker(), &marker); + assert_eq!(recovered.membership(), &membership); + tail_fixture::check_images(&cluster, recovered.manifest(), &expected).await?; + assert!(cluster.voters.is_empty()); + // Pins protect all currently returned materialized images; releasing the + // proof must also release private image retention and both admission pools. + if !rooted { + let images = destination.images.clone(); + assert_eq!( + cluster + .domain + .blocking(move || images.reclaim(4096))? + .await?? + .files, + 0 + ); + } + drop(recovered); + assert_eq!(receivers.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + if !rooted { + let images = destination.images.clone(); + assert!( + cluster + .domain + .blocking(move || images.reclaim(4096))? + .await?? + .files + > 0 + ); + // Re-fetching the whole closure also works after private cache eviction. + let recovered = archive + .verify_tail( + &cluster.domain, + &tails, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + tail_fixture::check_images(&cluster, recovered.manifest(), &expected).await?; + } + let pending = receiver(&receivers, &destination, &cluster)?; + destination.store.close(); + assert!(archive + .verify_checkpoint(&cluster.domain, &base, pending) + .await + .is_err()); + assert!(receiver(&receivers, &destination, &cluster).is_err()); + assert_eq!(receivers.usage().sources, 0); + cluster.close().await?; + Ok(()) +} + +pub(super) async fn moving_cut(rooted: bool) -> Result<()> { + let (mut cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let remote = archive_store::Store::new(); + let archive = new_archive( + remote.clone(), + Path::from("moving-cut"), + cluster.birth.route().history, + Default::default(), + )?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let original = source(&sources, &cluster)?; + let marker = original.marker().clone(); + let reference = archive.upload_checkpoint(&cluster.domain, original).await?; + let (path, index) = index(remote.as_ref(), &reference).await?; + let gate = remote.hold(cell_objects(&path, &index, "beta")[0].clone()); + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let archive_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut pending = Box::pin(archive.verify_checkpoint( + &archive_domain, + &reference, + receiver(&receivers, &destination, &cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("archive ended before held dependency") + } + } + checkpoint_fixture::advance_partition( + &mut cluster, + "alpha", + &expected["alpha"], + rooted, + Some(2), + ) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let advanced = group.group_manifest()?.context("advanced serving cut")?; + assert!(advanced.position().index > marker.position().index); + remote.clear(); + gate.release.notify_one(); + let recovered = pending.await?; + assert_eq!(recovered.checkpoint().marker(), &marker); + tail_fixture::check_images(&cluster, recovered.checkpoint().manifest(), &expected).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&advanced)); + assert!(matches!( + sources.capture(&group, recovered.checkpoint().manifest()), + Err(CellGroupCheckpointError::Invalid("source root/owner")) + )); + drop(recovered); + // Verifying a newer checkpoint does not prevent selecting an older archive + // cut later, and neither result may mutate or publish a live engine root. + let latest = archive + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let newer = archive + .verify_checkpoint( + &cluster.domain, + &latest, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(newer.checkpoint().marker(), &advanced); + drop(cluster.voters[0].owner.take()); + let mut write = GroupWrite::new(&group); + write.group_manifest = Some(newer.checkpoint().manifest().clone()); + assert!( + matches!(cluster.voters[0].log.persist_cells(OperationId::new(777), vec![write]).await, + Err(LogError::Invalid(message)) if message == "invalid cell-group apply binding or prefix") + ); + assert_eq!(group.group_manifest()?.as_ref(), Some(&advanced)); + drop(newer); + let older = archive + .verify_checkpoint( + &cluster.domain, + &reference, + receiver(&receivers, &destination, &cluster)?, + ) + .await?; + assert_eq!(older.checkpoint().marker(), &marker); + tail_fixture::check_images(&cluster, older.checkpoint().manifest(), &expected).await?; + drop(older); + assert_eq!(receivers.usage().sources, 0); + archive_domain.shutdown().await; + cluster.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_tail.rs b/crates/cellar/tests/support/cell_group_archive_tail.rs index 0add7a1cd..4d6b4209a 100644 --- a/crates/cellar/tests/support/cell_group_archive_tail.rs +++ b/crates/cellar/tests/support/cell_group_archive_tail.rs @@ -3,7 +3,7 @@ use super::*; use raft::prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}; -async fn tail_index( +pub(super) async fn tail_index( store: &dyn ObjectStore, reference: &CellGroupArchiveTailReference, ) -> Result<(Path, serde_json::Value)> { @@ -63,7 +63,7 @@ async fn write_tail( )?) } -async fn check_images( +pub(super) async fn check_images( cluster: &Cluster, root: &PreparedGroupManifest, expected: &std::collections::BTreeMap>, @@ -173,7 +173,7 @@ pub(super) async fn enrollment_controls(rooted: bool) -> Result<()> { let (_, encoded) = tail_index(store.as_ref(), &reference).await?; assert_eq!(encoded["entries"].as_array().unwrap().len(), 9); assert_eq!(tails.usage().operations, 0); - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let receiver_group = cluster.voters[2] .log @@ -269,7 +269,7 @@ pub(super) async fn dependencies(rooted: bool) -> Result<()> { .last() .context("base beta")?; let dependency_bytes = store.get(&dependency).await?.bytes().await?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; for missing in [&dependency, &entry] { store.inner.delete(missing).await?; @@ -388,7 +388,7 @@ pub(super) async fn warm_replay(rooted: bool) -> Result<()> { let (path, original) = tail_index(store.as_ref(), &reference).await?; let entry_location = entry_path(&path, &original["entries"][0])?; let entry = store.get(&entry_location).await?.bytes().await?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let recovered = archive .verify_tail( @@ -414,7 +414,7 @@ pub(super) async fn warm_replay(rooted: bool) -> Result<()> { if let (CellImagePin::Materialized(base_pin), CellImagePin::Materialized(final_pin)) = (&base_pin, &final_pin) { - let store = cluster.voters[2].images.clone(); + let store = applier.images.clone(); let base_pin = base_pin.clone(); let final_pin = final_pin.clone(); let wrong = wrong.bytes.clone(); @@ -492,15 +492,15 @@ pub(super) async fn warm_replay(rooted: bool) -> Result<()> { CellImagePin::Materialized(pin) => cluster ._root .path() - .join("node-3/images") + .join("archive-data/images") .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))), CellImagePin::Extents(pin) => { let id = pin.manifest().extent(0).context("result extent")?; - cluster._root.path().join("extents-2").join(format!( - "e-{:08x}-{}", - id.length(), - hex::encode(id.bytes()) - )) + cluster + ._root + .path() + .join("archive-data/extents") + .join(format!("e-{:08x}-{}", id.length(), hex::encode(id.bytes()))) } CellImagePin::Empty => anyhow::bail!("expected populated result"), }; @@ -571,7 +571,11 @@ pub(super) async fn physical_cancellation() -> Result<()> { let reference = archive .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) .await?; - let applier = destination(&mut cluster)?; + let applier = destination_with_filesystem( + &cluster, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + ) + .await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let recovered = archive .verify_tail( @@ -591,7 +595,7 @@ pub(super) async fn physical_cancellation() -> Result<()> { let path = cluster ._root .path() - .join("node-3/images") + .join("archive-data/images") .join(format!("{}.sqlite", hex::encode(pin.id().bytes()))); drop(recovered); // Only the replayed result is selected; checkpoint/base reads cannot enter @@ -736,7 +740,7 @@ pub(super) async fn extend_compacted(rooted: bool) -> Result<()> { // The flattened extension names the original checkpoint and all immutable // entry objects directly; the old tail index is not a recovery dependency. store.inner.delete(&prefix_path).await?; - let applier = destination(&mut cluster)?; + let applier = destination(&cluster).await?; let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; for voter in &mut cluster.voters[..2] { drop(voter.owner.take()); diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 2a2e22add..ad915a01c 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -480,12 +480,19 @@ including an ambiguous PUT response. Corrupt existing objects refuse overwrite. `CellGroupArchiveReference` is an untrusted persistent locator. Only `verify_checkpoint` can produce `VerifiedCellGroupArchiveCheckpoint`: it downloads -and hashes every indexed object, checks continuity and limits, runs the ordinary -physical checkpoint receiver, and binds its complete recovered marker to the +and hashes every indexed object, checks continuity and limits, runs the shared +physical checkpoint decoder, and binds its complete recovered marker to the index. Missing/corrupt cells or a substituted cut refuse the whole preparation. -The receiver does not change the engine-selected current/checkpoint root. Its -ordinary installation and Raft admission contracts remain required. The source -engine and its pins are unnecessary for this independent recovery. +`CellGroupArchiveStore` owns private pages/images for an explicit expected route; +open those stores on the execution domain, separately from serving storage. +`receive_archive` needs no GroupLog, member claim, selected applied cut or surviving +voter disk. Its checkpoint type is distinct from the engine-bound receiver output +accepted by SnapshotChange. Archive roots carry a privately minted owner token +that no Raft engine accepts for publication. No replacement genesis or voter +identity is fabricated. Serving still requires fencing, fresh-incarnation seeding +and a new quorum. Verifying older and newer archive cuts never changes an engine's +selected root or the eligibility of a later archive verification. The engine-bound +receiver preserves its selected-base and membership/install checks. Source/receiver pools separately charge physical closures and scratch. Archive admission charges bounded encoded/decoded metadata and I/O windows. Complete @@ -495,13 +502,11 @@ bounded channels and retains permits until those workers exit. Domain shutdown joins entered work. The pool must be shared by node orchestration when it is wired in; constructing separate pools per group would defeat the node-wide bound. -This checkpoint-only capability has no committed tail, conditionally published -head or remote GC lease. It grants no group S3 settlement, archive coverage, -reclamation or disaster-recovery authority. CG-06 still requires those owners, -complete administrative/cell tail replay, retention admission and fenced -fresh-incarnation seeding. Its executable scope now records the component's three -production files and the native archive target/store instrumentation, replacing -the originally planned leaf output without changing any required case or oracle. +The checkpoint capability has no conditionally published head or remote GC lease. +It grants no group S3 settlement, archive coverage, reclamation or disaster-recovery +authority. CG-06 still requires those owners, retention admission and fenced +fresh-incarnation seeding. Its executable scope records the production modules +and native archive qualification without changing any required case or oracle. Quorum writes may proceed during an S3 outage within explicit disk/unarchived-byte budgets. S3-mode outputs wait for confirmed coverage. Log reclamation requires the @@ -537,7 +542,10 @@ Materialized replay uses `GenerationStore::prepare_verified`: an existing cached result cannot replace validation of the incoming LTX. The delta is recomputed in admitted anonymous scratch before the destination is independently verified and synced. `GenerationPin::verify_and_sync` also preserves the receiver's existing -independent-destination checks. Extent replay applies every delta and rechecks +independent-destination checks. New private replay images become reclaimable after +their returned pins close; verifying an existing image preserves its retention. +Private-store quotas remain enforced and live proofs retain their physical data. +Extent replay applies every delta and rechecks the complete final image. A valid but wrong LTX delta or warm destination damage must refuse, even when metadata and object hashes are well formed. @@ -548,10 +556,8 @@ returned `VerifiedCellGroupArchiveRecovery` retains this admission with physical pins. Cancellation stops further replay work and retains entered physical owners; domain shutdown joins them. Typed contention and poisoned-owner failures differ. -This remains data-only recovery through an existing destination engine/applier. -Production integration needs an isolated data-verification storage owner, which -must not depend on a serving engine's advancing selected cut or surviving voter -disks. Published/GC-protected archive coverage, S3 settlement/outage admission, +This remains data-only recovery through an independent physical storage owner. +Published/GC-protected archive coverage, S3 settlement/outage admission, archive-aware file reclamation and fenced fresh-incarnation seeding remain CG-06 work. The task scope now explicitly includes the shared checkpoint/engine fixture changes and new transition/tail modules; all required cases and oracles are kept. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 78d401ee4..041b36fdc 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1737,7 +1737,10 @@ "crates/cellar/durability/raft/cell_groups/archive/tail_codec.rs", "crates/cellar/durability/raft/cell_groups/archive/replay.rs", "crates/cellar/durability/raft/cell_groups/apply/transition.rs", - "crates/cellar/tests/support/cell_group_archive_tail.rs" + "crates/cellar/tests/support/cell_group_archive_tail.rs", + "crates/cellar/durability/raft/cell_groups/archive/data_store.rs", + "crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs", + "crates/cellar/tests/support/cell_group_archive_isolation.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From af0b61ebf5f9ae3bd1db926f37d1b45ba38a8e33 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 06:44:06 -0700 Subject: [PATCH 168/568] docs: record isolated archive recovery gate --- docs/design/multi-raft/cell-groups/README.md | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 71 ++++++++++--------- .../design/multi-raft/cell-groups/progress.md | 52 ++++++++++++++ .../multi-raft/cell-groups/protocols.md | 4 ++ 4 files changed, 96 insertions(+), 35 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 285df4d25..8fa319ad3 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -7,7 +7,9 @@ CG-01 through CG-05 have passed their scoped native evidence gates. CG-05's complete checkpoint, authenticated learner/rebuild, membership, restart and resource-ownership requirements pass at `e0550acd`. Its current gate includes all five exact cases and 205 Rust regressions across eighteen commands. -CG-06 verified S3 checkpoint/tail coverage and safe log reclamation is next. +CG-06 is IN_PROGRESS: independent checkpoint/tail data recovery passes after all +voter directories are erased. Protected S3 coverage, settlement and safe log +reclamation remain the next priorities. The production SDK/fleet has not yet cut over; topology, hibernation, complete S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 80e2e672b..0ccb3f818 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,42 +15,45 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial source is `7a29c2a9bff5c2daddf37f53c50bc5d575e0b069`. -CG-06 is IN_PROGRESS: complete immutable checkpoints plus contiguous committed -tails now recover every cell and configuration through independent LTX/data -replay. `extend_tail` reuses archived entry references after actual local prefix -compaction, with only a new suffix entry/index PUT in the tested two-entry case. -The obsolete tail index is deleted before recovery; actual physical shared-engine -file purge is not qualified. No callback, synthetic cell or engine install occurs. - -All 22 commands pass: nine exact archive cases, 156 execution cases, five snapshot -cases and four generation regressions (174 distinct Rust cases), all-target -build/Clippy, formatting/layout, dependencies/docs/preflight and 37 validator tests. -The record under -`.cellar/cell-groups-evidence/CG-06-committed-tail/qualification/` -binds 323 source hashes, four executables, frozen inputs, both fixture failures and -the preceding checkpoint. Independent hashes/source/exact-selection verification -passes. See [progress](progress.md) and the +The latest partial source is `14955f2dc2b01a6f6ab6b75b0122c812e5de6765`. +CG-06 is IN_PROGRESS. Complete checkpoints and contiguous tails now recover through +an independent physical archive store, with no destination Raft engine, member +claim, replacement genesis or surviving voter disk. Both image forms recover after +all three source voter directories and extent directories are deleted. A held +remote dependency permits source quorum progress in another execution domain; +newer and older cuts verify independently. Private archive roots are refused by +actual exact-cut engine publication and source capture. Closed stores refuse new +and outstanding receives. Newly replayed private images become reclaimable after +proof pins close; full remote replay succeeds after their physical eviction. + +All 24 commands pass: eleven exact archive cases, 156 execution cases, five +snapshot cases and four generation regressions (176 distinct Rust cases), +all-target build/Clippy, formatting/layout, dependencies/docs/preflight and 37 +validator tests. The record under +`.cellar/cell-groups-evidence/CG-06-isolated-verifier/qualification/` +binds 326 source hashes, four executables, frozen inputs and the previous tail +checkpoint. Independent hashes/source/exact-selection verification passes. +See [progress](progress.md) and the [archive contract](protocols.md#apply-checkpoint-and-archive). -The original `archive_tails_cover_enrollment_and_controls` case passes with an -empty checkpoint, four real enrollments, later mutations, explicit joint demotion/ -leave and production leadership-transfer no-op. The other four original CG-06 -cases and its complete task gate remain pending. A cached result cannot replace -incoming LTX replay; wrong but correctly hashed deltas and warm image corruption -refuse. Physical-read cancellation retains admission while another cell commits; -domain shutdown joins the held worker before refunding. Source/receiver/archive -limits, readiness assertions and the exact nine-entry control trace are unchanged. - -Uploaded locators and verified physical state grant no archive coverage, serving -right, S3 settlement or disaster-recovery authorization. Next isolate the data -verifier from a serving engine's advancing cut and surviving voter disks. Then -build published/GC-protected coverage, bounded outage admission/S3 settlement, -archive-aware file reclamation and fenced unmanaged seeding under fresh group/cell -incarnations. Shared dispatch still refuses S3 durability. Qualify framing/ -coalescing and large-group limits before cost/throughput claims. Actual RustFS, -zero-quorum erasure/fencing, managed recovery and the public hard cutover remain -later required work. The previous checkpoint-only record at `650fc638` is retained. +The original `archive_tails_cover_enrollment_and_controls` case and all prior +missing/corrupt/warm-LTX, physical cancellation/join and post-compaction extension +cases remain passing on the new neutral stores. The other four original CG-06 +cases and full task gate remain pending. No required oracle or bound was weakened. + +Next implement conditionally published, GC-protected complete-cut coverage, +bounded outage admission/S3 settlement, archive-aware physical log reclamation +and fenced unmanaged seeding under fresh group/cell incarnations. Keep source +birth/provenance and the authoritative inventory of every possible old voting +lineage, including joins after the chosen recovery cut. Selected-cut membership +alone is insufficient to fence every old host. Physical data verification grants +no serving, coverage, S3 settlement or fencing authority. Shared dispatch still +refuses S3 durability. The native erasure drill uses the actual in-memory store; +RustFS, Docker fencing, fresh-quorum serving, load and public hard cutover remain +later required work. Retain the preceding committed-tail record at `7a29c2a9`. +Before managed integration, retain the verified archive result alongside derived +manifest roots: raw root clones do not preserve the parent archive quota. Qualify +that ownership propagation and private-cache maintenance in CG-08. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 07929062a..da4a9920d 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,58 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 archive verification without voter metadata + +Source: `14955f2dc2b01a6f6ab6b75b0122c812e5de6765`. CG-06 remains IN_PROGRESS. +The archive verifier owns dedicated physical stores and needs no GroupLog, bootstrap/member claim, +replacement genesis or surviving voter metadata. Its typed checkpoint is distinct +from the engine-bound snapshot output. Shared physical decoding preserves the +live receiver's selected-base, membership, owner and installation checks; private +archive roots have a freshly minted owner token that live engines refuse. + +Both image forms recover the exact checkpoint/tail cut, all four cells including +empty enrollment, and permanent membership after closing all three source engines, +dropping their physical owners and deleting every voter log/page/image directory +and source extent directory. Destination stores are opened only after erasure. +A separate native scenario holds a remote dependency while another execution +domain advances the healthy source quorum, then verifies newer and older archive +cuts independently. Exact-cut engine publication and source capture refuse the +private archive root; engine-selected state stays unchanged. Closed stores refuse +both outstanding and new receives. New private materialized replay results release +retention after proof pins close, and complete remote recovery succeeds after +actual private-image eviction. Existing host retention is preserved. + +All 24 commands pass: eleven exact archive cases, 156 execution cases, five snapshot +cases and four LTX generation regressions (176 distinct Rust cases across four +executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, CG-06 preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-isolated-verifier/qualification/checkpoint.json` +binds 326 source hashes, copied executables, frozen plan inputs and command +outputs. Independent source/artifact/exact-selection verification passes. Transfer +this ignored evidence with its committed-tail predecessor to reproduce the audit. +No required task case, oracle, quota or readiness assertion was weakened. + +Reflection and next priorities: data integrity alone cannot authorize S3 settlement +or truncation. Implement conditionally published, GC-protected coverage of the +exact complete committed cut, then bounded outage admission/S3 output settlement, +archive-aware shared-engine file purge and fenced unmanaged recovery into fresh +group/cell incarnations. Preserve source birth/provenance and an authoritative +inventory of all possible old voting lineages, including voters joined after the +selected archive cut; fencing only selected-cut members is insufficient. Follow +with CG-08 effects/cache lifetimes and CG-07 public SDK/fleet cutover, then topology, +placement, hibernation, managed recovery and final legacy removal. Controller +composition must retain the verified archive result alongside derived manifest +roots: a raw `PreparedGroupManifest` clone does not carry its parent archive quota. +Qualify that ownership propagation in CG-08 before managed integration. + +The erasure drill uses real local Raft Engine/SQLite/LTX and the actual in-memory +object-store adapter. It does not qualify RustFS, Docker fencing, a new serving +incarnation, process/machine failure isolation, measured load or the production +hard cutover. The other four original CG-06 acceptance cases and full task gate +remain pending. Shared dispatch still refuses S3 durability. The preceding +committed-tail source/evidence is preserved below. + ## CG-06 committed-tail replay and extension after compaction Partial source checkpoint: `7a29c2a9bff5c2daddf37f53c50bc5d575e0b069`. diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index ad915a01c..fa79ef1f1 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -555,6 +555,10 @@ are explicit and do not change the ordinary archive/checkpoint pool limits. The returned `VerifiedCellGroupArchiveRecovery` retains this admission with physical pins. Cancellation stops further replay work and retains entered physical owners; domain shutdown joins them. Typed contention and poisoned-owner failures differ. +Orchestration must keep the verified archive result alongside any derived raw +manifest roots: cloning `PreparedGroupManifest` does not clone its parent archive +admission. Controller ownership propagation and private-cache maintenance require +CG-08 qualification before managed integration. This remains data-only recovery through an independent physical storage owner. Published/GC-protected archive coverage, S3 settlement/outage admission, From a5d506a6af76a72e464f27915611ada87a698282 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 08:03:35 -0700 Subject: [PATCH 169/568] feat: protect cell-group archive publication Bind publication to committed-source and physical data proofs. Persist remote ownership and never reuse collected registry/generation keys. BREAKING CHANGE: group archive locators include registry/generation; producers require pending upload ownership. Marker 1 uses a hard cutover. --- .../raft/cell_groups/archive/checkpoint.rs | 57 +- .../raft/cell_groups/archive/collection.rs | 189 +++++ .../raft/cell_groups/archive/graph.rs | 167 ++++ .../raft/cell_groups/archive/mod.rs | 39 +- .../raft/cell_groups/archive/objects.rs | 89 ++- .../raft/cell_groups/archive/ownership/mod.rs | 619 +++++++++++++++ .../cell_groups/archive/ownership/state.rs | 260 +++++++ .../cell_groups/archive/ownership/wire.rs | 80 ++ .../raft/cell_groups/archive/publication.rs | 267 +++++++ .../raft/cell_groups/archive/tail.rs | 139 +++- .../raft/cell_groups/archive/upload.rs | 216 ++++++ .../cellar/durability/raft/cell_groups/mod.rs | 13 +- .../durability/raft/cell_groups/root.rs | 6 +- .../tests/native_cell_groups_archive.rs | 208 ++++- .../support/cell_group_archive_isolation.rs | 13 +- .../support/cell_group_archive_publication.rs | 712 ++++++++++++++++++ .../tests/support/cell_group_archive_store.rs | 45 +- .../tests/support/cell_group_archive_tail.rs | 47 +- .../multi-raft/cell-groups/protocols.md | 98 ++- docs/design/multi-raft/cell-groups/tasks.json | 10 +- 20 files changed, 3155 insertions(+), 119 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/collection.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/graph.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/publication.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/upload.rs create mode 100644 crates/cellar/tests/support/cell_group_archive_publication.rs diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs index b15435dac..582c59622 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -5,20 +5,20 @@ use std::io::Write; #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct Index { +pub(super) struct Index { domain: String, - marker: GroupManifest, + pub marker: GroupManifest, bytes: u64, - chunks: Vec, + pub chunks: Vec, } #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct Chunk { +pub(super) struct Chunk { part: wire::Part, offset: u64, length: u64, - object: Object, + pub object: Object, } impl Index { @@ -102,7 +102,7 @@ impl Write for Bounded { } } -impl CellGroupArchive { +impl CellGroupArchiveUpload { /// Capture is supplied by the caller's node checkpoint pool. The bounded /// producer keeps the full source pinned through physical reads. Losing the /// async caller closes its channel; domain shutdown joins the physical owner. @@ -127,6 +127,7 @@ impl CellGroupArchive { return Err(CellGroupArchiveError::Invalid("source history")); } let permit = self.admit()?; + self.registry().pending(&self.ticket()).await?; let retained = permit.clone(); let mut index = Index { domain: DOMAIN.into(), @@ -195,11 +196,15 @@ impl CellGroupArchive { .await?; drop(permit); Ok(CellGroupArchiveReference { + registry: object.registry, + generation: object.generation, digest: object.digest, length: object.length, }) } +} +impl CellGroupArchive { /// Independently GET every object into the ordinary physical receiver. No /// source pin or upload receipt substitutes for dependency verification. /// A missing/corrupt cell refuses the whole cut. Ordinary errors join the @@ -233,10 +238,24 @@ impl CellGroupArchive { reference: &CellGroupArchiveReference, permit: Arc, ) -> Result { + Ok(self + .checkpoint_index(domain, reference, permit) + .await? + .marker) + } + + pub(super) async fn checkpoint_index( + &self, + domain: &ExecutionDomain, + reference: &CellGroupArchiveReference, + permit: Arc, + ) -> Result { let bytes = self .get( "indexes", &Object { + registry: reference.registry, + generation: reference.generation, digest: reference.digest, length: reference.length, }, @@ -250,7 +269,7 @@ impl CellGroupArchive { let index: Index = serde_json::from_slice(&bytes) .map_err(|_| CellGroupArchiveError::Invalid("checkpoint index"))?; index.validate(&archive)?; - Ok::<_, CellGroupArchiveError>(index.marker) + Ok::<_, CellGroupArchiveError>(index) }) .map_err(|_| CellGroupArchiveError::Stopped)? .await @@ -264,29 +283,9 @@ impl CellGroupArchive { mut receiver: CellGroupArchiveCheckpointReceiver, permit: Arc, ) -> Result { - let bytes = self - .get( - "indexes", - &Object { - digest: reference.digest, - length: reference.length, - }, - self.limits.max_index_bytes, - ) + let index = self + .checkpoint_index(domain, reference, permit.clone()) .await?; - let archive = self.clone(); - let retained = permit.clone(); - let index: Index = domain - .blocking(move || { - let _retained = retained; - let index: Index = serde_json::from_slice(&bytes) - .map_err(|_| CellGroupArchiveError::Invalid("checkpoint index"))?; - index.validate(&archive)?; - Ok::<_, CellGroupArchiveError>(index) - }) - .map_err(|_| CellGroupArchiveError::Stopped)? - .await - .map_err(CellGroupArchiveError::Worker)??; let marker = index.marker; let (sender, mut incoming) = tokio::sync::mpsc::channel::(1); let retained = permit.clone(); diff --git a/crates/cellar/durability/raft/cell_groups/archive/collection.rs b/crates/cellar/durability/raft/cell_groups/archive/collection.rs new file mode 100644 index 000000000..49748c057 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/collection.rs @@ -0,0 +1,189 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete marks precede deletion. Revision watermarks, protected parents and +//! never-reused generation keys make concurrent publication/delayed DELETE safe. +use super::{ + graph::{Graph, Key, Kind}, + *, +}; +use futures_util::TryStreamExt; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveCollectionLimits { + pub max_scan_objects: usize, + pub max_delete_objects: usize, +} + +impl Default for CellGroupArchiveCollectionLimits { + fn default() -> Self { + Self { + max_scan_objects: 4096, + max_delete_objects: 256, + } + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupArchiveCollectionReport { + pub revision: u64, + pub protected_objects: usize, + pub scanned_objects: usize, + pub deleted_objects: usize, + pub deleted_bytes: u64, +} + +fn parse(path: &str, prefix: &Path, registry: [u8; 32]) -> Result { + let suffix = path + .strip_prefix(prefix.as_ref()) + .and_then(|s| s.strip_prefix('/')) + .ok_or(CellGroupArchiveError::Invalid( + "collection object namespace", + ))?; + let mut parts = suffix.split('/'); + let generation = parts + .next() + .ok_or(CellGroupArchiveError::Invalid("collection generation"))?; + let parsed: u64 = generation + .parse() + .map_err(|_| CellGroupArchiveError::Invalid("collection generation"))?; + if parsed < 2 || parsed == u64::MAX || parsed.to_string() != generation { + return Err(CellGroupArchiveError::Invalid("collection generation")); + } + let kind = parts + .next() + .and_then(Kind::parse) + .ok_or(CellGroupArchiveError::Invalid("collection object kind"))?; + let (length, digest) = parts + .next() + .and_then(|p| p.split_once('-')) + .ok_or(CellGroupArchiveError::Invalid("collection object identity"))?; + let bytes: u64 = length + .parse() + .map_err(|_| CellGroupArchiveError::Invalid("collection object length"))?; + if bytes == 0 || bytes.to_string() != length || digest.len() != 64 || parts.next().is_some() { + return Err(CellGroupArchiveError::Invalid("collection object identity")); + } + let mut hash = [0; 32]; + hex::decode_to_slice(digest, &mut hash) + .map_err(|_| CellGroupArchiveError::Invalid("collection digest"))?; + if hex::encode(hash) != digest { + return Err(CellGroupArchiveError::Invalid("collection digest encoding")); + } + Ok(Key { + kind, + object: super::objects::Object { + registry, + generation: parsed, + length: bytes, + digest: hash, + }, + }) +} + +impl CellGroupArchiveRegistry { + /// Mark selected roots, explicit holds and pending parents; retain entire + /// pending/new generations. Verify every marked object before any DELETE. + /// No unknown owner expires by time. Cancellation may leave partial garbage, + /// but cannot permit a deleted key to be rebound by a later allocation. + pub async fn collect( + &self, + domain: &ExecutionDomain, + owners: &CellGroupArchiveUploadPool, + tails: &CellGroupArchiveTailPool, + limits: CellGroupArchiveCollectionLimits, + ) -> Result { + if !(1..=65_536).contains(&limits.max_scan_objects) + || limits.max_delete_objects == 0 + || limits.max_delete_objects > limits.max_scan_objects + || limits.max_delete_objects > owners.limits.max_objects + { + return Err(CellGroupArchiveError::Limit); + } + domain + .run(async { + let retained = owners.admit()?; + // Hashing entry objects needs the same bounded byte windows as tail + // verification. This lease does not construct a physical replay. + let _tail = tails.admit()?; + let io = self.archive.admit()?; + let protection = self.protection().await?; + let mut marked = Graph::new(owners.limits.max_objects); + for root in &protection.roots { + let closure = Graph::load( + &self.archive, + domain, + root, + tails.limits, + io.clone(), + retained.clone(), + owners.limits.max_objects, + ) + .await?; + for key in closure.objects { + marked.insert(key.kind, key.object, u64::MAX - 1)?; + } + } + marked.verify(&self.archive, tails.limits).await?; + let mut report = CellGroupArchiveCollectionReport { + revision: protection.revision, + protected_objects: marked.objects.len(), + ..Default::default() + }; + let prefix = self + .archive + .prefix + .child("registries") + .child(hex::encode(self.id())) + .child("generations"); + let mut objects = self.archive.store.list(Some(&prefix)); + let mut candidates = Vec::new(); + self.archive + .io(async { + while report.scanned_objects < limits.max_scan_objects { + let Some(object) = objects + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + else { + break; + }; + report.scanned_objects += 1; + let key = parse(object.location.as_ref(), &prefix, self.id())?; + if key.object.generation <= protection.revision + && !protection + .pending_generations + .contains(&key.object.generation) + && !marked.objects.contains(&key) + && candidates.len() < limits.max_delete_objects + { + candidates.push((key, object.size)); + } + } + Ok(()) + }) + .await?; + // New reservations can inherit only roots protected by this mark; + // their own generation is newer than its watermark. Released or + // aborted allocations cannot resurrect an unmarked old key. + for (key, bytes) in candidates { + let path = self.archive.object_path(key.kind.name(), &key.object); + self.archive + .io(async { + match self.archive.store.delete(&path).await { + Ok(()) | Err(object_store::Error::NotFound { .. }) => Ok(()), + Err(error) => Err(CellGroupArchiveError::Storage(error)), + } + }) + .await?; + report.deleted_objects += 1; + report.deleted_bytes = report + .deleted_bytes + .checked_add(bytes) + .ok_or(CellGroupArchiveError::Limit)?; + } + Ok(report) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/graph.rs b/crates/cellar/durability/raft/cell_groups/archive/graph.rs new file mode 100644 index 000000000..8b9fec6f3 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/graph.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact dependency-key expansion. No collector may delete from a partial mark. +use super::{objects::Object, ownership::Descriptor, *}; +use std::collections::BTreeSet; + +pub(super) const ROOT_DOMAIN: &str = "cellar-cell-group-archive-root-1"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(super) enum Kind { + Chunks, + Indexes, + Entries, + Tails, + Roots, +} + +impl Kind { + pub fn parse(name: &str) -> Option { + Some(match name { + "chunks" => Self::Chunks, + "indexes" => Self::Indexes, + "entries" => Self::Entries, + "tails" => Self::Tails, + "roots" => Self::Roots, + _ => return None, + }) + } + + pub fn name(self) -> &'static str { + match self { + Self::Chunks => "chunks", + Self::Indexes => "indexes", + Self::Entries => "entries", + Self::Tails => "tails", + Self::Roots => "roots", + } + } + + pub fn limit(self, archive: &CellGroupArchive, limits: CellGroupArchiveTailLimits) -> usize { + match self { + Self::Chunks => CHUNK_BYTES, + Self::Indexes | Self::Tails => archive.limits.max_index_bytes, + Self::Entries => limits.max_tail_bytes, + Self::Roots => 8192, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub(super) struct Key { + pub kind: Kind, + pub object: Object, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Root { + pub domain: String, + pub marker: GroupManifest, + pub tail: CellGroupArchiveTailReference, +} + +pub(super) struct Graph { + pub objects: BTreeSet, + limit: usize, +} + +impl Graph { + pub fn new(limit: usize) -> Self { + Self { + objects: BTreeSet::new(), + limit, + } + } + + pub fn limit(&self) -> usize { + self.limit + } + + pub fn insert(&mut self, kind: Kind, object: Object, generation: u64) -> Result<()> { + if object.registry == [0; 32] + || object.generation < 2 + || object.generation > generation + || object.length == 0 + { + return Err(CellGroupArchiveError::Invalid( + "archive dependency generation", + )); + } + let key = Key { kind, object }; + if !self.objects.contains(&key) && self.objects.len() == self.limit { + return Err(CellGroupArchiveError::Limit); + } + self.objects.insert(key); + Ok(()) + } + + pub async fn load( + archive: &CellGroupArchive, + domain: &ExecutionDomain, + descriptor: &Descriptor, + limits: CellGroupArchiveTailLimits, + io: Arc, + retained: Arc, + max_objects: usize, + ) -> Result { + let root = archive.get("roots", &descriptor.object, 8192).await?; + let root: Root = serde_json::from_slice(&root) + .map_err(|_| CellGroupArchiveError::Invalid("archive root encoding"))?; + if root.domain != ROOT_DOMAIN || root.marker != descriptor.marker { + return Err(CellGroupArchiveError::Invalid("archive root binding")); + } + let tail = archive + .read_tail_index(domain, &root.tail, limits, io.clone(), retained.clone()) + .await?; + if tail.cut.marker != root.marker { + return Err(CellGroupArchiveError::Invalid("archive root/tail cut")); + } + let checkpoint = archive + .checkpoint_index(domain, &tail.checkpoint, io) + .await?; + if checkpoint.marker != tail.base { + return Err(CellGroupArchiveError::Invalid( + "archive root/checkpoint cut", + )); + } + let generation = descriptor.object.generation; + let mut graph = Self::new(max_objects); + graph.insert(Kind::Roots, descriptor.object.clone(), generation)?; + graph.insert(Kind::Tails, root.tail.object(), generation)?; + graph.insert(Kind::Indexes, tail.checkpoint.object(), generation)?; + for entry in tail.entries { + graph.insert(Kind::Entries, entry.object, generation)?; + } + for chunk in checkpoint.chunks { + graph.insert(Kind::Chunks, chunk.object, generation)?; + } + if graph + .objects + .iter() + .any(|key| key.object.registry != descriptor.object.registry) + { + return Err(CellGroupArchiveError::Invalid( + "archive dependency registry", + )); + } + Ok(graph) + } + + pub async fn verify( + &self, + archive: &CellGroupArchive, + limits: CellGroupArchiveTailLimits, + ) -> Result<()> { + for key in &self.objects { + archive + .get( + key.kind.name(), + &key.object, + key.kind.limit(archive, limits), + ) + .await?; + } + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 302948a0b..7c14c1559 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -21,13 +21,27 @@ use std::{fmt, sync::Arc}; mod checkpoint; mod data_store; pub use data_store::CellGroupArchiveStore; +mod collection; +mod graph; mod objects; +mod publication; +pub use collection::{CellGroupArchiveCollectionLimits, CellGroupArchiveCollectionReport}; +pub use publication::{CellGroupArchiveCoverage, PreparedCellGroupArchivePublication}; +mod upload; +pub use upload::{ + CellGroupArchiveUpload, CellGroupArchiveUploadLimits, CellGroupArchiveUploadPool, +}; +mod ownership; +pub use ownership::{ + CellGroupArchiveCreation, CellGroupArchiveHold, CellGroupArchiveIntent, + CellGroupArchiveRegistry, CellGroupArchiveUploadTicket, PreparedCellGroupArchiveCreation, +}; mod replay; mod tail; mod tail_admission; mod tail_codec; pub use replay::VerifiedCellGroupArchiveRecovery; -pub use tail::CellGroupArchiveTailReference; +pub use tail::{CellGroupArchiveTailReference, CellGroupArchiveUploadedTail}; pub use tail_admission::{CellGroupArchiveTailLimits, CellGroupArchiveTailPool}; const DOMAIN: &str = "cellar-cell-group-archive-checkpoint-1"; @@ -40,6 +54,10 @@ pub enum CellGroupArchiveError { Busy, Stopped, Timeout, + Conflict, + Expired, + Unconfirmed, + MissingRegistry, Checkpoint(CellGroupCheckpointError), Replay(crate::raft::execution::ExecutionError), Storage(object_store::Error), @@ -129,11 +147,30 @@ pub struct CellGroupArchiveUsage { #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct CellGroupArchiveReference { + registry: [u8; 32], + generation: u64, digest: [u8; 32], length: u64, } impl CellGroupArchiveReference { + pub fn registry(&self) -> [u8; 32] { + self.registry + } + + pub fn generation(&self) -> u64 { + self.generation + } + + fn object(&self) -> objects::Object { + objects::Object { + registry: self.registry, + generation: self.generation, + digest: self.digest, + length: self.length, + } + } + pub fn digest(&self) -> [u8; 32] { self.digest } diff --git a/crates/cellar/durability/raft/cell_groups/archive/objects.rs b/crates/cellar/durability/raft/cell_groups/archive/objects.rs index 57c4f1075..443f1eb46 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/objects.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/objects.rs @@ -5,16 +5,20 @@ use futures_util::TryStreamExt; use object_store::{PutMode, PutOptions}; use std::time::Duration; -#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct Object { + pub registry: [u8; 32], + pub generation: u64, pub digest: [u8; 32], pub length: u64, } impl Object { - pub fn of(bytes: &[u8]) -> Self { + pub fn of(registry: [u8; 32], generation: u64, bytes: &[u8]) -> Self { Self { + registry, + generation, digest: Sha256::digest(bytes).into(), length: bytes.len() as u64, } @@ -31,13 +35,20 @@ impl Object { } impl CellGroupArchive { - fn object_path(&self, kind: &str, object: &Object) -> Path { + pub(super) fn object_path(&self, kind: &str, object: &Object) -> Path { self.prefix + .child("registries") + .child(hex::encode(object.registry)) + .child("generations") + .child(object.generation.to_string()) .child(kind) .child(format!("{}-{}", object.length, hex::encode(object.digest))) } - async fn io(&self, future: impl std::future::Future>) -> Result { + pub(super) async fn io( + &self, + future: impl std::future::Future>, + ) -> Result { crate::asyncrt::timeout(Duration::from_millis(self.limits.io_timeout_ms), future) .await .map_err(|_| CellGroupArchiveError::Timeout)? @@ -46,7 +57,12 @@ impl CellGroupArchive { // One deadline includes headers, every body chunk and EOF. Never allocate // from unbounded remote metadata or accept a partial/ranged object response. async fn read(&self, kind: &str, object: &Object, limit: usize) -> Result>> { - if object.length == 0 || object.length > limit as u64 { + if object.registry == [0; 32] + || object.generation < 2 + || object.generation == u64::MAX + || object.length == 0 + || object.length > limit as u64 + { return Err(CellGroupArchiveError::Limit); } let path = self.object_path(kind, object); @@ -95,11 +111,18 @@ impl CellGroupArchive { .ok_or(CellGroupArchiveError::Invalid("missing archive object")) } - pub(super) async fn put(&self, kind: &str, bytes: Vec, limit: usize) -> Result { + pub(super) async fn put_generation( + &self, + registry: [u8; 32], + generation: u64, + kind: &str, + bytes: Vec, + limit: usize, + ) -> Result { if bytes.is_empty() || bytes.len() > limit { return Err(CellGroupArchiveError::Limit); } - let object = Object::of(&bytes); + let object = Object::of(registry, generation, &bytes); // Reuse verified immutable objects without another PUT. A corrupt // existing object refuses; overwriting could erase evidence of damage. if self.read(kind, &object, limit).await?.is_some() { @@ -124,4 +147,56 @@ impl CellGroupArchive { }, } } + + pub(super) async fn read_versioned( + &self, + path: &Path, + limit: usize, + ) -> Result, object_store::UpdateVersion)>> { + self.io(async { + let result = match self.store.get(path).await { + Ok(result) => result, + Err(object_store::Error::NotFound { .. }) => return Ok(None), + Err(error) => return Err(CellGroupArchiveError::Storage(error)), + }; + let length = result.meta.size; + if length == 0 + || length > limit as u64 + || result.range.start != 0 + || result.range.end != length + { + return Err(CellGroupArchiveError::Invalid( + "control response length/range", + )); + } + let version = object_store::UpdateVersion { + e_tag: result.meta.e_tag.clone(), + version: result.meta.version.clone(), + }; + let mut bytes = Vec::with_capacity(length as usize); + let mut stream = result.into_stream(); + let mut chunks = 0; + while let Some(chunk) = stream + .try_next() + .await + .map_err(CellGroupArchiveError::Storage)? + { + chunks += 1; + if chunks > 16_384 + || bytes + .len() + .checked_add(chunk.len()) + .is_none_or(|n| n > length as usize) + { + return Err(CellGroupArchiveError::Limit); + } + bytes.extend_from_slice(&chunk); + } + if bytes.len() != length as usize { + return Err(CellGroupArchiveError::Invalid("control body length")); + } + Ok(Some((bytes, version))) + }) + .await + } } diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs new file mode 100644 index 000000000..e7e87c6f1 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -0,0 +1,619 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistent ownership of group archive generations. Dropping local handles +//! never retires a remote owner. Allocating intents must be persisted before CAS. +mod state; +mod wire; +use wire::WireCreation; + +use super::{objects::Object, *}; +use cellar_logic::cell_group::GroupRoute; +use object_store::{PutMode, PutOptions, UpdateVersion}; +use rand::RngCore; +use state::State; + +const IDENTITY_DOMAIN: &str = "cellar-cell-group-archive-identity-1"; +const STATE_DOMAIN: &str = "cellar-cell-group-archive-ownership-1"; +const INTENT_DOMAIN: &str = "cellar-cell-group-archive-intent-1"; +const MAX_CONTROL_BYTES: usize = 64 * 1024; +const MAX_INTENT_BYTES: usize = 8192; + +/// Persist before creation. Retry reconciles an already complete creation. +/// An incomplete identity requires authority recovery; counters never reset. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(try_from = "WireCreation", into = "WireCreation")] +pub struct CellGroupArchiveCreation { + domain: String, + id: [u8; 32], + route: GroupRoute, +} + +/// Fresh, one-use initialization authority. Serialized creation metadata can +/// reopen established state but cannot recreate counters after metadata loss. +pub struct PreparedCellGroupArchiveCreation { + archive: CellGroupArchive, + identity: CellGroupArchiveCreation, +} + +impl PreparedCellGroupArchiveCreation { + pub fn intent(&self) -> &CellGroupArchiveCreation { + &self.identity + } + + pub async fn commit(self, domain: &ExecutionDomain) -> Result { + self.archive.create_registry(domain, &self.identity).await + } +} + +impl CellGroupArchiveCreation { + fn check(&self, history: GroupHistory) -> Result<()> { + if self.domain != IDENTITY_DOMAIN || self.id == [0; 32] || self.route.history != history { + return Err(CellGroupArchiveError::Invalid("archive registry identity")); + } + super::super::ManifestTree::new(self.route.clone(), Default::default()) + .map_err(CellGroupCheckpointError::from)?; + Ok(()) + } +} + +/// A never-reused upload generation. It is protected until explicitly aborted +/// or consumed by publication, including after its local handle is dropped. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveUploadTicket { + registry: [u8; 32], + generation: u64, + allocation: [u8; 32], +} + +impl CellGroupArchiveUploadTicket { + pub fn generation(&self) -> u64 { + self.generation + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Descriptor { + pub object: Object, + pub marker: GroupManifest, +} + +impl Descriptor { + fn check(&self, identity: &CellGroupArchiveCreation, revision: u64) -> Result<()> { + self.marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.marker.route() != &identity.route + || self.object.registry != identity.id + || self.object.generation < 2 + || self.object.generation >= revision + || self.object.length == 0 + || self.object.length > 8192 + { + return Err(CellGroupArchiveError::Invalid( + "archive retained descriptor", + )); + } + Ok(()) + } +} + +/// Explicit remote hold, distinct from integrity, quorum and serving proofs. +/// Persist it with its owner; release only after that owner retires/fences all +/// users and joins entered work. Never release merely because a Rust value drops. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveHold { + registry: [u8; 32], + id: u64, + allocation: [u8; 32], + root: Descriptor, +} + +impl CellGroupArchiveHold { + pub fn marker(&self) -> &GroupManifest { + &self.root.marker + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) enum Action { + Reserve { + parent: Option, + }, + Acquire { + root: Descriptor, + }, + Publish { + ticket: CellGroupArchiveUploadTicket, + root: Descriptor, + }, +} + +/// One exact allocation at one registry revision. Persist the entire intent +/// before sending its mutation. A lost response never justifies a new intent. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct CellGroupArchiveIntent { + domain: String, + registry: [u8; 32], + revision: u64, + nonce: [u8; 32], + action: Action, +} + +impl CellGroupArchiveIntent { + fn digest(&self) -> Result<[u8; 32]> { + if self.domain != INTENT_DOMAIN + || self.registry == [0; 32] + || self.nonce == [0; 32] + || self.revision < 2 + || self.revision == u64::MAX + { + return Err(CellGroupArchiveError::Invalid("archive allocation intent")); + } + Ok(Sha256::digest(encode(self, MAX_INTENT_BYTES)?).into()) + } +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Pending { + pub ticket: CellGroupArchiveUploadTicket, + pub parent: Option, +} + +pub(super) enum Allocation { + Upload(CellGroupArchiveUploadTicket), + Hold(Box), +} + +pub(super) struct Protection { + pub revision: u64, + pub roots: Vec, + pub pending_generations: Vec, +} + +/// Conditional storage adapter for one exact group route and registry identity. +/// Owns no Raft engine and cannot grant membership or application durability. +#[derive(Clone)] +pub struct CellGroupArchiveRegistry { + pub(super) archive: CellGroupArchive, + identity: CellGroupArchiveCreation, +} + +fn encode(value: &impl Serialize, limit: usize) -> Result> { + let mut output = super::checkpoint::Bounded { + bytes: Vec::new(), + limit, + }; + serde_json::to_writer(&mut output, value).map_err(|_| CellGroupArchiveError::Limit)?; + Ok(output.bytes) +} + +impl CellGroupArchive { + /// Preparation performs no storage mutation. The execution domain supplies + /// randomness; the caller durably records the complete result before create. + pub async fn prepare_registry_creation( + &self, + domain: &ExecutionDomain, + route: GroupRoute, + ) -> Result { + domain + .run(async { + let mut id = [0; 32]; + crate::asyncrt::rng("cell-group-archive-registry").fill_bytes(&mut id); + let creation = CellGroupArchiveCreation { + domain: IDENTITY_DOMAIN.into(), + id, + route, + }; + creation.check(self.history)?; + Ok(PreparedCellGroupArchiveCreation { + archive: self.clone(), + identity: creation, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn create_registry( + &self, + domain: &ExecutionDomain, + creation: &CellGroupArchiveCreation, + ) -> Result { + domain + .run(async { + let _io = self.admit()?; + creation.check(self.history)?; + let registry = CellGroupArchiveRegistry { + archive: self.clone(), + identity: creation.clone(), + }; + if self + .read_versioned(®istry.path("registry"), MAX_CONTROL_BYTES) + .await? + .is_some() + { + registry.load().await?; + return Ok(registry); + } + let bytes = encode(creation, MAX_CONTROL_BYTES)?; + // Only a confirmed exclusive identity creation can initialize the + // counter. An existing identity may be reconciled only by opening + // valid established state, never by recreating a missing registry. + match registry.write("identity", bytes, PutMode::Create).await { + Ok(()) => (), + Err(_) => { + registry + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)?; + return Ok(registry); + } + } + let desired = encode(&State::empty(creation.clone()), MAX_CONTROL_BYTES)?; + let outcome = registry.write("registry", desired, PutMode::Create).await; + match registry.load().await { + Ok(_) => Ok(registry), + Err(_) => outcome.and(Err(CellGroupArchiveError::Unconfirmed)), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Open established state. Missing identity/state and malformed conditional + /// versions refuse; opening never allocates or repairs a missing record. + pub async fn open_registry( + &self, + domain: &ExecutionDomain, + expected: &CellGroupArchiveCreation, + ) -> Result { + domain + .run(async { + let _io = self.admit()?; + expected.check(self.history)?; + let registry = CellGroupArchiveRegistry { + archive: self.clone(), + identity: expected.clone(), + }; + registry.load().await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveRegistry { + pub fn archive(&self) -> &CellGroupArchive { + &self.archive + } + + pub fn route(&self) -> &GroupRoute { + &self.identity.route + } + + pub fn creation(&self) -> &CellGroupArchiveCreation { + &self.identity + } + + pub(super) fn id(&self) -> [u8; 32] { + self.identity.id + } + + fn path(&self, name: &str) -> Path { + self.archive.prefix.child("ownership").child(name) + } + + async fn write(&self, name: &str, bytes: Vec, mode: PutMode) -> Result<()> { + self.archive + .io(async { + self.archive + .store + .put_opts(&self.path(name), bytes.into(), PutOptions::from(mode)) + .await + .map_err(CellGroupArchiveError::Storage)?; + Ok(()) + }) + .await + } + + async fn load(&self) -> Result<(State, UpdateVersion)> { + let identity = self + .archive + .read_versioned(&self.path("identity"), MAX_CONTROL_BYTES) + .await? + .ok_or(CellGroupArchiveError::MissingRegistry)? + .0; + if identity != encode(&self.identity, MAX_CONTROL_BYTES)? { + return Err(CellGroupArchiveError::Invalid( + "archive registry identity changed", + )); + } + let (bytes, version) = self + .archive + .read_versioned(&self.path("registry"), MAX_CONTROL_BYTES) + .await? + .ok_or(CellGroupArchiveError::Invalid( + "missing established archive registry", + ))?; + if version.e_tag.as_ref().is_some_and(String::is_empty) + || version.version.as_ref().is_some_and(String::is_empty) + || version.e_tag.as_ref().map_or(0, String::len) + + version.version.as_ref().map_or(0, String::len) + > 8192 + || (version.e_tag.is_none() && version.version.is_none()) + { + return Err(CellGroupArchiveError::Invalid( + "archive registry conditional version", + )); + } + let state: State = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("archive registry encoding"))?; + state.check(&self.identity)?; + Ok((state, version)) + } + + pub async fn prepare_upload(&self, domain: &ExecutionDomain) -> Result { + domain + .run(async { + let _io = self.archive.admit()?; + let state = self.load().await?.0; + self.intent( + &state, + Action::Reserve { + parent: state.selected.clone(), + }, + ) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn reserve( + &self, + domain: &ExecutionDomain, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Reserve { .. }) { + return Err(CellGroupArchiveError::Invalid("upload intent purpose")); + } + domain + .run(async { + let _io = self.archive.admit()?; + match self.commit(intent).await? { + Allocation::Upload(ticket) => Ok(ticket), + _ => Err(CellGroupArchiveError::Invalid("upload allocation purpose")), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn prepare_acquire( + &self, + domain: &ExecutionDomain, + ) -> Result { + domain + .run(async { + let _io = self.archive.admit()?; + let state = self.load().await?.0; + let root = state + .selected + .clone() + .ok_or(CellGroupArchiveError::Invalid("no selected archive"))?; + self.intent(&state, Action::Acquire { root }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn acquire( + &self, + domain: &ExecutionDomain, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Acquire { .. }) { + return Err(CellGroupArchiveError::Invalid("acquire intent purpose")); + } + domain + .run(async { + let _io = self.archive.admit()?; + match self.commit(intent).await? { + Allocation::Hold(hold) => Ok(*hold), + _ => Err(CellGroupArchiveError::Invalid("hold allocation purpose")), + } + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + fn intent(&self, state: &State, action: Action) -> Result { + let mut nonce = [0; 32]; + crate::asyncrt::rng("cell-group-archive-intent").fill_bytes(&mut nonce); + let intent = CellGroupArchiveIntent { + domain: INTENT_DOMAIN.into(), + registry: self.identity.id, + revision: state + .revision + .checked_add(1) + .ok_or(CellGroupArchiveError::Limit)?, + nonce, + action, + }; + let mut proposed = state.clone(); + proposed.allocate(&intent)?; + proposed.check(&self.identity)?; + Ok(intent) + } + + pub(super) async fn commit(&self, intent: &CellGroupArchiveIntent) -> Result { + if intent.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + let (mut state, version) = self.load().await?; + if let Some(result) = state.reconcile(intent)? { + return Ok(result); + } + state.allocate(intent)?; + state.check(&self.identity)?; + let outcome = self + .write( + "registry", + encode(&state, MAX_CONTROL_BYTES)?, + PutMode::Update(version), + ) + .await; + let observed = self + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)? + .0; + if let Some(result) = observed.reconcile(intent)? { + return Ok(result); + } + if matches!( + outcome, + Err(CellGroupArchiveError::Storage( + object_store::Error::Precondition { .. } + | object_store::Error::AlreadyExists { .. } + )) + ) { + return Err(CellGroupArchiveError::Conflict); + } + Err(CellGroupArchiveError::Unconfirmed) + } + + pub(super) async fn prepare_publish( + &self, + ticket: CellGroupArchiveUploadTicket, + root: Descriptor, + ) -> Result { + let state = self.load().await?.0; + self.intent(&state, Action::Publish { ticket, root }) + } + + pub(super) async fn publication_hold( + &self, + intent: &CellGroupArchiveIntent, + ) -> Result { + if !matches!(intent.action, Action::Publish { .. }) { + return Err(CellGroupArchiveError::Invalid("publication intent purpose")); + } + match self.load().await?.0.reconcile(intent)? { + Some(Allocation::Hold(hold)) => Ok(hold), + _ => Err(CellGroupArchiveError::Expired), + } + } + + pub(super) async fn pending(&self, ticket: &CellGroupArchiveUploadTicket) -> Result { + if ticket.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + self.load() + .await? + .0 + .pending + .iter() + .find(|p| &p.ticket == ticket) + .cloned() + .ok_or(CellGroupArchiveError::Expired) + } + + pub(super) async fn protection(&self) -> Result { + let state = self.load().await?.0; + Ok(Protection { + revision: state.revision, + roots: state.roots(), + pending_generations: state.pending.iter().map(|p| p.ticket.generation).collect(), + }) + } + + pub(super) async fn referenced(&self, hold: &CellGroupArchiveHold) -> Result { + if hold.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + self.load() + .await? + .0 + .holds + .iter() + .find(|h| *h == hold) + .map(|h| h.root.clone()) + .ok_or(CellGroupArchiveError::Expired) + } + + pub async fn release( + &self, + domain: &ExecutionDomain, + hold: &CellGroupArchiveHold, + ) -> Result<()> { + if hold.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + domain + .run(self.retire(Some(hold), None)) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + pub async fn abort( + &self, + domain: &ExecutionDomain, + ticket: &CellGroupArchiveUploadTicket, + ) -> Result<()> { + if ticket.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + domain + .run(self.retire(None, Some(ticket))) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + async fn retire( + &self, + hold: Option<&CellGroupArchiveHold>, + ticket: Option<&CellGroupArchiveUploadTicket>, + ) -> Result<()> { + let _io = self.archive.admit()?; + for _ in 0..8 { + let (mut state, version) = self.load().await?; + if !state.retire(hold, ticket)? { + return Ok(()); + } + state.check(&self.identity)?; + let outcome = self + .write( + "registry", + encode(&state, MAX_CONTROL_BYTES)?, + PutMode::Update(version), + ) + .await; + let mut observed = self + .load() + .await + .map_err(|_| CellGroupArchiveError::Unconfirmed)? + .0; + if !observed.retire(hold, ticket)? { + return Ok(()); + } + if matches!( + outcome, + Err(CellGroupArchiveError::Storage( + object_store::Error::Precondition { .. } + | object_store::Error::AlreadyExists { .. } + )) + ) { + continue; + } + return Err(CellGroupArchiveError::Unconfirmed); + } + Err(CellGroupArchiveError::Conflict) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs new file mode 100644 index 000000000..cca6ef83a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs @@ -0,0 +1,260 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Pure bounded transitions. All allocating mutations reconcile their exact +//! intent digest; retired revisions never reappear and generations never repeat. +use super::*; +use std::collections::{BTreeMap, BTreeSet}; + +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct State { + domain: String, + identity: CellGroupArchiveCreation, + pub revision: u64, + pub pending: Vec, + pub holds: Vec, + pub selected: Option, +} + +impl State { + pub fn empty(identity: CellGroupArchiveCreation) -> Self { + Self { + domain: STATE_DOMAIN.into(), + identity, + revision: 1, + pending: Vec::new(), + holds: Vec::new(), + selected: None, + } + } + + pub fn check(&self, identity: &CellGroupArchiveCreation) -> Result<()> { + if self.domain != STATE_DOMAIN + || &self.identity != identity + || self.revision == 0 + || self.revision == u64::MAX + { + return Err(CellGroupArchiveError::Invalid( + "archive registry domain/revision", + )); + } + if self.pending.len() > 8 || self.holds.len() > 64 { + return Err(CellGroupArchiveError::Limit); + } + let mut ids = BTreeSet::new(); + let mut previous = 0; + for pending in &self.pending { + let t = pending.ticket; + if t.registry != identity.id + || t.generation < 2 + || t.generation <= previous + || t.generation > self.revision + || t.allocation == [0; 32] + || !ids.insert(t.generation) + { + return Err(CellGroupArchiveError::Invalid("archive pending generation")); + } + previous = t.generation; + if let Some(parent) = &pending.parent { + parent.check(identity, t.generation)?; + } + } + previous = 0; + for hold in &self.holds { + if hold.registry != identity.id + || hold.id <= previous + || hold.id > self.revision + || hold.allocation == [0; 32] + || !ids.insert(hold.id) + { + return Err(CellGroupArchiveError::Invalid( + "archive hold identity/order", + )); + } + previous = hold.id; + hold.root.check(identity, hold.id)?; + } + let mut generations = BTreeMap::new(); + for root in self.roots() { + root.check(identity, self.revision)?; + if ids.contains(&root.object.generation) { + return Err(CellGroupArchiveError::Invalid( + "pending generation already bound", + )); + } + if generations + .insert(root.object.generation, root.clone()) + .is_some_and(|before| before != root) + { + return Err(CellGroupArchiveError::Invalid( + "archive generation descriptor conflict", + )); + } + if self.selected.as_ref().is_none_or(|selected| { + root.marker.position().index > selected.marker.position().index + }) { + return Err(CellGroupArchiveError::Invalid( + "archive retained root exceeds head", + )); + } + } + Ok(()) + } + + fn advance(&mut self) -> Result { + self.revision = self + .revision + .checked_add(1) + .filter(|n| *n != u64::MAX) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(self.revision) + } + + pub fn reconcile(&self, intent: &CellGroupArchiveIntent) -> Result> { + let allocation = intent.digest()?; + if intent.registry != self.identity.id { + return Err(CellGroupArchiveError::Conflict); + } + if let Some(p) = self + .pending + .iter() + .find(|p| p.ticket.generation == intent.revision && p.ticket.allocation == allocation) + { + return Ok(Some(Allocation::Upload(p.ticket))); + } + if let Some(h) = self + .holds + .iter() + .find(|h| h.id == intent.revision && h.allocation == allocation) + { + return Ok(Some(Allocation::Hold(Box::new(h.clone())))); + } + Ok(None) + } + + pub fn allocate(&mut self, intent: &CellGroupArchiveIntent) -> Result { + if let Some(result) = self.reconcile(intent)? { + return Ok(result); + } + if intent.revision <= self.revision { + return Err(CellGroupArchiveError::Expired); + } + if self.revision.checked_add(1) != Some(intent.revision) { + return Err(CellGroupArchiveError::Conflict); + } + let allocation = intent.digest()?; + match &intent.action { + Action::Reserve { parent } => { + if parent != &self.selected { + return Err(CellGroupArchiveError::Conflict); + } + if self.pending.len() == 8 { + return Err(CellGroupArchiveError::Limit); + } + let ticket = CellGroupArchiveUploadTicket { + registry: self.identity.id, + generation: self.advance()?, + allocation, + }; + self.pending.push(Pending { + ticket, + parent: parent.clone(), + }); + Ok(Allocation::Upload(ticket)) + } + Action::Acquire { root } => { + if self.selected.as_ref() != Some(root) { + return Err(CellGroupArchiveError::Expired); + } + self.hold(root.clone(), allocation) + } + Action::Publish { ticket, root } => { + if !self.pending.iter().any(|p| &p.ticket == ticket) + || root.object.generation != ticket.generation + { + return Err(CellGroupArchiveError::Expired); + } + root.check(&self.identity, intent.revision)?; + if let Some(selected) = &self.selected { + let before = selected.marker.position(); + let after = root.marker.position(); + if after.index < before.index + || after.term < before.term + || (after.index == before.index && root.marker != selected.marker) + { + return Err(CellGroupArchiveError::Conflict); + } + } + let hold = self.hold(root.clone(), allocation)?; + self.pending.retain(|p| &p.ticket != ticket); + self.selected = Some(root.clone()); + Ok(hold) + } + } + } + + fn hold(&mut self, root: Descriptor, allocation: [u8; 32]) -> Result { + if self.holds.len() == 64 { + return Err(CellGroupArchiveError::Limit); + } + let hold = CellGroupArchiveHold { + registry: self.identity.id, + id: self.advance()?, + allocation, + root, + }; + self.holds.push(hold.clone()); + Ok(Allocation::Hold(Box::new(hold))) + } + + pub fn retire( + &mut self, + hold: Option<&CellGroupArchiveHold>, + ticket: Option<&CellGroupArchiveUploadTicket>, + ) -> Result { + let id = match (hold, ticket) { + (Some(h), None) => h.id, + (None, Some(t)) => t.generation, + _ => return Err(CellGroupArchiveError::Invalid("archive retirement purpose")), + }; + if id < 2 || id > self.revision { + return Err(CellGroupArchiveError::Conflict); + } + if hold.is_some_and(|h| self.holds.iter().any(|p| p.id == h.id && p != h)) + || ticket.is_some_and(|t| { + self.pending + .iter() + .any(|p| p.ticket.generation == t.generation && &p.ticket != t) + }) + { + return Err(CellGroupArchiveError::Conflict); + } + let before = self.pending.len() + self.holds.len(); + if let Some(hold) = hold { + self.holds.retain(|h| h != hold); + } + if let Some(ticket) = ticket { + self.pending.retain(|p| &p.ticket != ticket); + } + if before == self.pending.len() + self.holds.len() { + return Ok(false); + } + self.advance()?; + Ok(true) + } + + pub fn roots(&self) -> Vec { + let mut roots = Vec::new(); + for root in self + .selected + .iter() + .chain(self.holds.iter().map(|h| &h.root)) + .chain(self.pending.iter().filter_map(|p| p.parent.as_ref())) + { + if !roots.contains(root) { + roots.push(root.clone()); + } + } + roots + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs new file mode 100644 index 000000000..ba0b75d35 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs @@ -0,0 +1,80 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit host-side route codec; the dependency-free decision types do not +//! acquire serde or I/O dependencies. Shares canonical range-bound validation. +use super::*; +use crate::raft::cell_groups::root::Bound; +use cellar_logic::{cell_group::*, consensus::*}; + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct WireCreation { + domain: String, + id: [u8; 32], + cluster: [u8; 16], + namespace: [u8; 16], + group: [u8; 16], + incarnation: [u8; 16], + lower: Bound, + upper: Bound, + epoch: u64, + birth: [u8; 32], + schema: [u8; 32], +} + +impl From for WireCreation { + fn from(value: CellGroupArchiveCreation) -> Self { + let route = value.route; + Self { + domain: value.domain, + id: value.id, + cluster: route.history.cluster.bytes(), + namespace: route.history.namespace.bytes(), + group: route.history.group.bytes(), + incarnation: route.history.incarnation.bytes(), + lower: Bound::encode(route.range.lower()), + upper: Bound::encode(route.range.upper()), + epoch: route.epoch.get(), + birth: route.birth_digest, + schema: route.schema_digest, + } + } +} + +impl TryFrom for CellGroupArchiveCreation { + type Error = CellGroupArchiveError; + + fn try_from(value: WireCreation) -> Result { + let invalid = |_| CellGroupArchiveError::Invalid("archive registry route identity"); + let namespace = NamespaceId::new(value.namespace).map_err(invalid)?; + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new(value.cluster).map_err(invalid)?, + namespace, + group: GroupId::new(value.group).map_err(invalid)?, + incarnation: IncarnationId::new(value.incarnation).map_err(invalid)?, + }, + range: CellRange::new( + value + .lower + .decode(namespace) + .map_err(CellGroupCheckpointError::from)?, + value + .upper + .decode(namespace) + .map_err(CellGroupCheckpointError::from)?, + ) + .map_err(invalid)?, + epoch: RouteEpoch::new(value.epoch).map_err(invalid)?, + birth_digest: value.birth, + schema_digest: value.schema, + }; + let result = Self { + domain: value.domain, + id: value.id, + route, + }; + result.check(result.route.history)?; + Ok(result) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/publication.rs b/crates/cellar/durability/raft/cell_groups/archive/publication.rs new file mode 100644 index 000000000..566835789 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/publication.rs @@ -0,0 +1,267 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Source-authorized, physically verified publication. Neither a serialized +//! locator nor a registry allocation can substitute for these two proofs. +use super::super::PreparedGroupManifest; +use super::{ + graph::{Graph, Root, ROOT_DOMAIN}, + ownership::{Allocation, Descriptor}, + *, +}; +use crate::raft::{cache::GroupLog, member_state::GroupMembership}; +use raft::prelude::ConfState; + +pub(super) struct SourceCut { + group: GroupLog, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + _reservation: Arc, +} + +impl SourceCut { + pub(super) fn from_capture( + group: GroupLog, + root: PreparedGroupManifest, + configuration: ConfState, + membership: GroupMembership, + reservation: Arc, + ) -> Self { + Self { + group, + root, + configuration, + membership, + _reservation: reservation, + } + } + + fn check(&self) -> Result<()> { + let cache = self.group.read().map_err(CellGroupCheckpointError::from)?; + let target = self.root.marker().position(); + let current = cache + .group_manifest + .as_ref() + .ok_or(CellGroupArchiveError::Invalid( + "publication source root missing", + ))?; + if cache.applied.position.index < target.index + || cache.hard_state.commit < target.index.get() + || current.route() != self.root.marker().route() + || (current.position().index == target.index && current != self.root.marker()) + || cache + .term(target.index.get()) + .is_some_and(|term| term != target.term.get()) + { + return Err(CellGroupArchiveError::Invalid( + "archive publication source changed", + )); + } + // A later local checkpoint may compact the captured prefix. Its initial + // committed-source observation remains sealed to this live engine. + Ok(()) + } +} + +/// Persist `intent()` before commit. Holds the exact source and independently +/// recovered closure through CAS/readback; it cannot be deserialized or forged. +pub struct PreparedCellGroupArchivePublication { + upload: CellGroupArchiveUpload, + intent: CellGroupArchiveIntent, + verified: VerifiedCellGroupArchiveRecovery, + source: Arc, +} + +impl PreparedCellGroupArchivePublication { + pub fn intent(&self) -> &CellGroupArchiveIntent { + &self.intent + } + + pub async fn commit(self, domain: &ExecutionDomain) -> Result { + domain + .run(async { + let _io = self.upload.admit()?; + self.source.check()?; + let hold = match self.upload.registry().commit(&self.intent).await? { + Allocation::Hold(hold) => *hold, + _ => { + return Err(CellGroupArchiveError::Invalid( + "publication allocation purpose", + )) + } + }; + self.source.check()?; + Ok(CellGroupArchiveCoverage(Arc::new(Coverage { + hold, + verified: self.verified, + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +struct Coverage { + hold: CellGroupArchiveHold, + verified: VerifiedCellGroupArchiveRecovery, +} + +/// Historical complete data coverage with an explicit remote hold. Clones keep +/// local verification admission/pins. The remote hold is retired explicitly by +/// its durable owner after joined use; this grants no quorum or serving right. +#[derive(Clone)] +pub struct CellGroupArchiveCoverage(Arc); + +impl CellGroupArchiveCoverage { + pub fn hold(&self) -> &CellGroupArchiveHold { + &self.0.hold + } + + pub fn recovery(&self) -> &VerifiedCellGroupArchiveRecovery { + &self.0.verified + } + + pub fn marker(&self) -> &GroupManifest { + self.0.verified.manifest().marker() + } +} + +impl CellGroupArchiveUpload { + /// Recover every dependency and replay every entry, then prepare one exact + /// conditional publication intent. Missing B prevents coverage of later A. + /// Callbacks, engine installation and S3 command settlement do not run here. + pub async fn prepare_publication( + &self, + domain: &ExecutionDomain, + tails: &CellGroupArchiveTailPool, + uploaded: &CellGroupArchiveUploadedTail, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(async { + let source = uploaded.source.clone(); + source.check()?; + if source.root.marker().route() != self.registry().route() { + return Err(CellGroupArchiveError::Invalid( + "publication registry/source route", + )); + } + { + let _io = self.admit()?; + self.registry().pending(&self.ticket()).await?; + } + let reference = uploaded.reference(); + let verified = self.verify_tail(domain, tails, reference, receiver).await?; + if verified.manifest().marker() != source.root.marker() + || verified.configuration() != &source.configuration + || verified.membership() != &source.membership + { + return Err(CellGroupArchiveError::Invalid( + "archive publication exact cut", + )); + } + source.check()?; + let io = self.admit()?; + let root = Root { + domain: ROOT_DOMAIN.into(), + marker: source.root.marker().clone(), + tail: reference.clone(), + }; + let object = self + .put_generation( + self.registry().id(), + self.ticket().generation(), + "roots", + serde_json::to_vec(&root).map_err(|_| CellGroupArchiveError::Limit)?, + 8192, + ) + .await?; + let descriptor = Descriptor { + object, + marker: root.marker, + }; + let graph = Graph::load( + self, + domain, + &descriptor, + tails.limits, + io.clone(), + self.retained(), + self.max_objects(), + ) + .await?; + self.allows(&graph)?; + let intent = self + .registry() + .prepare_publish(self.ticket(), descriptor) + .await?; + Ok(PreparedCellGroupArchivePublication { + upload: self.clone(), + intent, + verified, + source, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveRegistry { + /// Reconcile only a publication already present under its exact persisted + /// intent. This never sends a publishing CAS or substitutes a newer head. + pub async fn reconcile_publication( + &self, + domain: &ExecutionDomain, + intent: &CellGroupArchiveIntent, + tails: &CellGroupArchiveTailPool, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + let hold = domain + .run(async { + let _io = self.archive.admit()?; + self.publication_hold(intent).await + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)??; + self.verify_hold(domain, &hold, tails, receiver).await + } + + /// Read only the exact held immutable root, independently recover all data, + /// and recheck the persistent hold before returning historical coverage. + pub async fn verify_hold( + &self, + domain: &ExecutionDomain, + hold: &CellGroupArchiveHold, + tails: &CellGroupArchiveTailPool, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + domain + .run(async { + let io = self.archive.admit()?; + let descriptor = self.referenced(hold).await?; + let bytes = self.archive.get("roots", &descriptor.object, 8192).await?; + let root: Root = serde_json::from_slice(&bytes) + .map_err(|_| CellGroupArchiveError::Invalid("held root encoding"))?; + if root.domain != ROOT_DOMAIN || root.marker != descriptor.marker { + return Err(CellGroupArchiveError::Invalid("held root binding")); + } + drop(io); + let verified = self + .archive + .verify_tail(domain, tails, &root.tail, receiver) + .await?; + if verified.manifest().marker() != &root.marker { + return Err(CellGroupArchiveError::Invalid("held recovery cut")); + } + let _io = self.archive.admit()?; + self.referenced(hold).await?; + Ok(CellGroupArchiveCoverage(Arc::new(Coverage { + hold: hold.clone(), + verified, + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs index 703f87ad4..891d2cabd 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/tail.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -12,11 +12,30 @@ const TAIL_DOMAIN: &str = "cellar-cell-group-archive-tail-1"; #[serde(deny_unknown_fields)] /// An untrusted immutable-tail locator; recovery must verify its entire closure. pub struct CellGroupArchiveTailReference { + registry: [u8; 32], + generation: u64, digest: [u8; 32], length: u64, } impl CellGroupArchiveTailReference { + pub fn registry(&self) -> [u8; 32] { + self.registry + } + + pub fn generation(&self) -> u64 { + self.generation + } + + pub(super) fn object(&self) -> Object { + Object { + registry: self.registry, + generation: self.generation, + digest: self.digest, + length: self.length, + } + } + pub fn digest(&self) -> [u8; 32] { self.digest } @@ -26,6 +45,33 @@ impl CellGroupArchiveTailReference { } } +/// An upload's exact committed-source witness, captured before asynchronous I/O. +/// This value cannot be deserialized. Persist its untrusted reference separately; +/// publication needs this witness or reconciliation of an existing publication. +#[derive(Clone)] +pub struct CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference, + pub(super) source: Arc, +} + +impl CellGroupArchiveUploadedTail { + pub fn reference(&self) -> &CellGroupArchiveTailReference { + &self.reference + } + + pub fn into_reference(self) -> CellGroupArchiveTailReference { + self.reference + } +} + +impl std::ops::Deref for CellGroupArchiveUploadedTail { + type Target = CellGroupArchiveTailReference; + + fn deref(&self) -> &Self::Target { + &self.reference + } +} + #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct Cut { @@ -60,21 +106,21 @@ impl Cut { #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct Tail { +pub(super) struct Tail { domain: String, - checkpoint: CellGroupArchiveReference, - base: GroupManifest, - cut: Cut, + pub checkpoint: CellGroupArchiveReference, + pub base: GroupManifest, + pub cut: Cut, bytes: u64, - entries: Vec, + pub entries: Vec, } #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] -struct EntryObject { +pub(super) struct EntryObject { index: u64, term: u64, - object: Object, + pub object: Object, } impl Tail { @@ -127,9 +173,9 @@ impl Tail { } struct Capture { - cut: Cut, + pub cut: Cut, entries: Vec<(u64, u64, Vec)>, - _root: PreparedGroupManifest, + authority: Arc, } fn capture( @@ -137,6 +183,7 @@ fn capture( base: &GroupManifest, current: PreparedGroupManifest, limits: CellGroupArchiveTailLimits, + source_budget: Arc, ) -> Result { if !Arc::ptr_eq(&group.owner, ¤t.engine_owner) || base.route() != current.marker().route() @@ -184,6 +231,8 @@ fn capture( .map_err(|_| CellGroupArchiveError::Invalid("tail configuration encoding"))?, membership: cache.membership.clone(), }; + let configuration = cache.conf_state.clone(); + let membership = cache.membership.clone(); drop(cache); // No engine/cache lock across encoding or physical/S3 work. cut.validate(group.identity().scope, limits)?; current.images.visit(&mut |_, _, pin, _, _| -> Result<()> { @@ -192,6 +241,41 @@ fn capture( } Ok(()) })?; + let mut source_bytes = current + .reserved_bytes() + .checked_add( + cut.conf + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .and_then(|n| n.checked_add(64 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + source_bytes = source_bytes + .checked_add( + serde_json::to_vec(&cut.membership) + .map_err(|_| CellGroupArchiveError::Limit)? + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + current.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if let super::super::CellImagePin::Extents(pin) = pin { + source_bytes = source_bytes + .checked_add( + (pin.reference().encoded_length() as usize) + .checked_mul(2) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + } + Ok(()) + })?; + let source_reservation = source_budget + .acquire(source_bytes) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy)?; let entries = entries .iter() .map(|entry| { @@ -205,7 +289,13 @@ fn capture( Ok(Capture { cut, entries, - _root: current, + authority: Arc::new(super::publication::SourceCut::from_capture( + group, + current, + configuration, + membership, + source_reservation, + )), }) } @@ -222,7 +312,7 @@ impl Drop for Stop { } } -impl CellGroupArchive { +impl CellGroupArchiveUpload { /// Copy an exact contiguous committed suffix under the engine cache lock, /// then upload immutable entry objects without holding that lock. This /// returns an untrusted locator; all base bytes and replay must still verify. @@ -233,7 +323,7 @@ impl CellGroupArchive { checkpoint: &CellGroupArchiveReference, group: &GroupLog, current: &PreparedGroupManifest, - ) -> Result { + ) -> Result { self.upload_from(domain, pool, Origin::Checkpoint(checkpoint), group, current) .await } @@ -249,7 +339,7 @@ impl CellGroupArchive { previous: &CellGroupArchiveTailReference, group: &GroupLog, current: &PreparedGroupManifest, - ) -> Result { + ) -> Result { self.upload_from(domain, pool, Origin::Tail(previous), group, current) .await } @@ -261,10 +351,11 @@ impl CellGroupArchive { origin: Origin<'_>, group: &GroupLog, current: &PreparedGroupManifest, - ) -> Result { + ) -> Result { domain .run(async { let io = self.admit()?; + self.registry().pending(&self.ticket()).await?; let retention = pool.admit()?; let limits = pool.limits; let (checkpoint, base, worker_base, prefix, prefix_bytes) = match origin { @@ -297,10 +388,11 @@ impl CellGroupArchive { let root = current.clone(); let retained = retention.clone(); let retained_io = io.clone(); + let source_budget = self.source_budget(); let captured = domain .blocking(move || { let (_retained, _io) = (retained, retained_io); - capture(group, &worker_base, root, limits) + capture(group, &worker_base, root, limits, source_budget) }) .map_err(|_| CellGroupArchiveError::Stopped)? .await @@ -362,16 +454,23 @@ impl CellGroupArchive { let object = self .put("tails", encoded, self.limits.max_index_bytes) .await?; - Ok(CellGroupArchiveTailReference { - digest: object.digest, - length: object.length, + Ok(CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference { + registry: object.registry, + generation: object.generation, + digest: object.digest, + length: object.length, + }, + source: captured.authority, }) }) .await .map_err(|_| CellGroupArchiveError::Stopped)? } +} - async fn read_tail_index( +impl CellGroupArchive { + pub(super) async fn read_tail_index( &self, domain: &ExecutionDomain, reference: &CellGroupArchiveTailReference, @@ -383,6 +482,8 @@ impl CellGroupArchive { .get( "tails", &Object { + registry: reference.registry, + generation: reference.generation, digest: reference.digest, length: reference.length, }, diff --git a/crates/cellar/durability/raft/cell_groups/archive/upload.rs b/crates/cellar/durability/raft/cell_groups/archive/upload.rs new file mode 100644 index 000000000..a66c7c05a --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/upload.rs @@ -0,0 +1,216 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node-admitted upload owners. Persistent reservations, rather than local +//! handles or clocks, protect upload generations and inherited dependencies. +use super::{ + graph::{Graph, Kind}, + ownership::*, + *, +}; +use std::{collections::BTreeMap, ops::Deref}; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupArchiveUploadLimits { + /// Counts upload maps, captured source witnesses and collector marks. + /// At least two owners are needed for a map and its publication witness. + pub max_owners: usize, + pub max_retained_bytes: usize, + pub max_objects: usize, +} + +impl Default for CellGroupArchiveUploadLimits { + fn default() -> Self { + Self { + max_owners: 8, + max_retained_bytes: 64 * 1024 * 1024, + max_objects: 16_384, + } + } +} + +/// Share this admission across every group on the node. Upload clones retain +/// their map and permit; dropping a clone never aborts its persistent reservation. +#[derive(Clone)] +pub struct CellGroupArchiveUploadPool { + pub(super) limits: CellGroupArchiveUploadLimits, + budget: Arc, + workspace: usize, +} + +impl CellGroupArchiveUploadPool { + pub fn new(limits: CellGroupArchiveUploadLimits) -> Result { + if !(2..=64).contains(&limits.max_owners) || !(1..=65_536).contains(&limits.max_objects) { + return Err(CellGroupArchiveError::Limit); + } + // Compact object marks plus the content lookup index, allocation/tree + // overhead and bounded control metadata. Archive I/O has its own pool. + let workspace = limits + .max_objects + .checked_mul(512) + .and_then(|n| n.checked_add(128 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_owners), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + operations, + retained_bytes, + } + } + + pub(super) fn admit(&self) -> Result> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} + +struct Owner { + registry: CellGroupArchiveRegistry, + ticket: CellGroupArchiveUploadTicket, + inherited: Graph, + reuse: BTreeMap<(Kind, [u8; 32], u64), super::objects::Object>, + _reservation: Arc, + source_budget: Arc, +} + +/// The only producer of immutable group archive objects. All new keys include a +/// never-reused reservation generation. Reuse is restricted to its protected +/// parent closure; an arbitrary old content address cannot be resurrected. +#[derive(Clone)] +pub struct CellGroupArchiveUpload(Arc); + +impl Deref for CellGroupArchiveUpload { + type Target = CellGroupArchive; + + fn deref(&self) -> &Self::Target { + self.0.registry.archive() + } +} + +impl CellGroupArchiveRegistry { + /// Resume only the exact still-pending persisted ticket. A replacement + /// process must fence its predecessor before sharing/retiring that owner. + pub async fn open_upload( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveUploadPool, + ticket: &CellGroupArchiveUploadTicket, + tails: &CellGroupArchiveTailPool, + ) -> Result { + domain + .run(async { + let reservation = pool.admit()?; + let io = self.archive.admit()?; + let pending = self.pending(ticket).await?; + let inherited = match pending.parent { + Some(root) => { + Graph::load( + &self.archive, + domain, + &root, + tails.limits, + io.clone(), + reservation.clone(), + pool.limits.max_objects, + ) + .await? + } + None => Graph::new(pool.limits.max_objects), + }; + // Recheck after metadata I/O; an abort cannot produce a new owner. + self.pending(ticket).await?; + let reuse = inherited + .objects + .iter() + .map(|key| { + ( + (key.kind, key.object.digest, key.object.length), + key.object.clone(), + ) + }) + .collect(); + Ok(CellGroupArchiveUpload(Arc::new(Owner { + registry: self.clone(), + ticket: *ticket, + inherited, + reuse, + _reservation: reservation, + source_budget: pool.budget.clone(), + }))) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl CellGroupArchiveUpload { + pub fn registry(&self) -> &CellGroupArchiveRegistry { + &self.0.registry + } + + pub fn ticket(&self) -> CellGroupArchiveUploadTicket { + self.0.ticket + } + + pub(super) fn retained(&self) -> Arc { + self.0._reservation.clone() + } + + pub(super) fn max_objects(&self) -> usize { + self.0.inherited.limit() + } + + pub(super) fn source_budget(&self) -> Arc { + self.0.source_budget.clone() + } + + pub(super) fn allows(&self, graph: &Graph) -> Result<()> { + if graph.objects.iter().any(|key| { + key.object.generation != self.0.ticket.generation() + && !self.0.inherited.objects.contains(key) + }) { + return Err(CellGroupArchiveError::Invalid( + "unprotected inherited archive object", + )); + } + Ok(()) + } + + pub(super) async fn put( + &self, + kind: &str, + bytes: Vec, + limit: usize, + ) -> Result { + if bytes.is_empty() || bytes.len() > limit { + return Err(CellGroupArchiveError::Limit); + } + let kind = + Kind::parse(kind).ok_or(CellGroupArchiveError::Invalid("archive object kind"))?; + let own = + super::objects::Object::of(self.registry().id(), self.ticket().generation(), &bytes); + if let Some(inherited) = self.0.reuse.get(&(kind, own.digest, own.length)) { + self.get(kind.name(), inherited, limit).await?; + return Ok(inherited.clone()); + } + self.put_generation( + self.registry().id(), + self.ticket().generation(), + kind.name(), + bytes, + limit, + ) + .await + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index a305b5abf..34a4a3552 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -17,10 +17,15 @@ pub use crate::raft::transport::{ mod apply; mod archive; pub use archive::{ - CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveError, - CellGroupArchiveLimits, CellGroupArchivePool, CellGroupArchiveReference, CellGroupArchiveStore, - CellGroupArchiveTailLimits, CellGroupArchiveTailPool, CellGroupArchiveTailReference, - CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveCheckpoint, + CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCollectionLimits, + CellGroupArchiveCollectionReport, CellGroupArchiveCoverage, CellGroupArchiveCreation, + CellGroupArchiveError, CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchiveLimits, + CellGroupArchivePool, CellGroupArchiveReference, CellGroupArchiveRegistry, + CellGroupArchiveStore, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, + CellGroupArchiveTailReference, CellGroupArchiveUpload, CellGroupArchiveUploadLimits, + CellGroupArchiveUploadPool, CellGroupArchiveUploadTicket, CellGroupArchiveUploadedTail, + CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, PreparedCellGroupArchiveCreation, + PreparedCellGroupArchivePublication, VerifiedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveRecovery, }; pub(crate) mod checkpoint; diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 3a48ba535..0735aeab8 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -391,14 +391,14 @@ fn required_root<'de, D: serde::Deserializer<'de>>( #[derive(Clone, Serialize, Deserialize)] #[serde(tag = "bound", content = "cell", deny_unknown_fields)] -enum Bound { +pub(super) enum Bound { NegativeInfinity, Key(String), PositiveInfinity, } impl Bound { - fn encode(value: &CellBound) -> Self { + pub(super) fn encode(value: &CellBound) -> Self { match value { CellBound::NegativeInfinity => Self::NegativeInfinity, CellBound::Key(k) => Self::Key(k.cell().as_str().into()), @@ -406,7 +406,7 @@ impl Bound { } } - fn decode(self, namespace: NamespaceId) -> Result { + pub(super) fn decode(self, namespace: NamespaceId) -> Result { Ok(match self { Self::NegativeInfinity => CellBound::NegativeInfinity, Self::PositiveInfinity => CellBound::PositiveInfinity, diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 6533c6426..c4662e3d5 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -20,6 +20,8 @@ mod fault; mod images; #[path = "support/cell_group_archive_isolation.rs"] mod isolation_fixture; +#[path = "support/cell_group_archive_publication.rs"] +mod publication_fixture; #[path = "support/cell_group_archive_tail.rs"] mod tail_fixture; @@ -78,6 +80,54 @@ async fn archive_verification_isolated_from_serving_cut() -> Result<()> { Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_head_requires_complete_committed_source() -> Result<()> { + for rooted in [false, true] { + publication_fixture::complete_source_cut(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_gc_retains_holds_and_pending_parents() -> Result<()> { + for rooted in [false, true] { + publication_fixture::holds_and_pending_parents(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_publication_reconciles_exact_uncertain_intent() -> Result<()> { + for rooted in [false, true] { + publication_fixture::uncertain_publication(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_delayed_delete_cannot_erase_new_generation() -> Result<()> { + for rooted in [false, true] { + publication_fixture::delayed_delete(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_registry_refuses_reset_and_expired_intents() -> Result<()> { + for rooted in [false, true] { + publication_fixture::registry_identity_and_intents(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_upload_owners_bound_source_witness_lifetimes() -> Result<()> { + for rooted in [false, true] { + publication_fixture::source_owner_lifetimes(rooted).await?; + } + Ok(()) +} + use cellar_ltx::{ generation::ReadServices, internal::{CacheLimits, EXTENT_BYTES}, @@ -195,13 +245,123 @@ fn source(pool: &CellGroupCheckpointPool, cluster: &Cluster) -> Result, prefix: Path, - history: GroupHistory, limits: CellGroupArchiveLimits, -) -> Result { - Ok(CellGroupArchivePool::new(limits)?.for_group(store, prefix, history)?) +) -> Result { + let label = prefix.as_ref().to_owned(); + let archive = CellGroupArchivePool::new(limits)?.for_group( + store, + prefix, + cluster.birth.route().history, + )?; + start_upload(cluster, &archive, &label).await +} + +async fn persist_archive_intent( + cluster: &Cluster, + intent: &T, +) -> Result { + let bytes = serde_json::to_vec(intent)?; + let name = hex::encode(Sha256::digest(&bytes)); + let directory = cluster._root.path().join("archive-intents"); + let loaded = cluster + .domain + .blocking(move || -> Result> { + use std::io::Write; + std::fs::create_dir_all(&directory)?; + let path = directory.join(name); + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(&directory)?.sync_all()?; + Ok(std::fs::read(path)?) + })? + .await??; + Ok(serde_json::from_slice(&loaded)?) +} + +async fn start_upload( + cluster: &Cluster, + archive: &CellGroupArchive, + label: &str, +) -> Result { + let binding = cluster._root.path().join(format!( + "archive-binding-{}.json", + hex::encode(Sha256::digest(label.as_bytes())) + )); + let existing = cluster + .domain + .blocking({ + let binding = binding.clone(); + move || std::fs::read(binding) + })? + .await?; + let registry = match existing { + Ok(bytes) => { + let creation: CellGroupArchiveCreation = serde_json::from_slice(&bytes)?; + archive.open_registry(&cluster.domain, &creation).await? + } + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + let prepared = archive + .prepare_registry_creation(&cluster.domain, cluster.birth.route().clone()) + .await?; + let creation = persist_archive_intent(cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &creation); + let bytes = serde_json::to_vec(&creation)?; + cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&binding)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(binding.parent().context("binding directory")?)? + .sync_all()?; + Ok(()) + })? + .await??; + prepared.commit(&cluster.domain).await? + } + Err(error) => return Err(error.into()), + }; + let intent = registry.prepare_upload(&cluster.domain).await?; + let intent = persist_archive_intent(cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &intent).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + Ok(registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?) +} + +fn registry_id(path: &Path) -> Result<[u8; 32]> { + let relative = path + .as_ref() + .split_once("/registries/") + .context("registry path")? + .1; + let id = relative.split('/').next().context("registry ID")?; + let mut registry = [0; 32]; + hex::decode_to_slice(id, &mut registry)?; + Ok(registry) +} + +fn generation_prefix(path: &Path) -> Result<(&str, u64)> { + let (prefix, relative) = path + .as_ref() + .split_once("/generations/") + .context("generation prefix")?; + let generation = relative.split('/').next().context("generation")?.parse()?; + Ok((prefix, generation)) } async fn index( @@ -216,7 +376,12 @@ async fn index( let objects: Vec<_> = store.list(None).try_collect().await?; let path = objects .into_iter() - .find(|m| m.location.as_ref().ends_with(&suffix)) + .find(|m| { + m.location.as_ref().ends_with(&suffix) + && generation_prefix(&m.location) + .is_ok_and(|(_, generation)| generation == reference.generation()) + && registry_id(&m.location).is_ok_and(|id| id == reference.registry()) + }) .context("checkpoint index object")? .location; let bytes = store.get(&path).await?.bytes().await?; @@ -224,7 +389,7 @@ async fn index( } fn cell_objects(index_path: &Path, index: &serde_json::Value, cell: &str) -> Vec { - let prefix = index_path.as_ref().split("/indexes/").next().unwrap(); + let (prefix, _) = generation_prefix(index_path).unwrap(); index["chunks"] .as_array() .unwrap() @@ -234,7 +399,8 @@ fn cell_objects(index_path: &Path, index: &serde_json::Value, cell: &str) -> Vec let digest: Vec = serde_json::from_value(chunk["object"]["digest"].clone()).unwrap(); Path::from(format!( - "{prefix}/chunks/{}-{}", + "{prefix}/generations/{}/chunks/{}-{}", + chunk["object"]["generation"], chunk["object"]["length"], hex::encode(digest) )) @@ -249,11 +415,12 @@ async fn checkpoint_archive_roundtrip_and_reuse() -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("qualification"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; store.lose_put_response.store(true, Ordering::Release); let original = archive @@ -373,7 +540,8 @@ async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { ..Default::default() }, ] { - let limited = new_archive(store.clone(), Path::from("limited"), history, limits)?; + let limited = + new_archive(&cluster, store.clone(), Path::from("limited"), limits).await?; assert!(matches!( limited .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -403,6 +571,7 @@ async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { ..history }, )?; + let archive = start_upload(&cluster, &archive, "qualification").await?; let reference = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) .await?; @@ -474,14 +643,15 @@ async fn checkpoint_archive_limits_and_cancelled_receiver() -> Result<()> { // A blocked GET is bounded by the configured complete-object deadline. let gate = store.hold(path); let timed = new_archive( + &cluster, store.clone(), Path::from("qualification"), - history, CellGroupArchiveLimits { io_timeout_ms: 20, ..Default::default() }, - )?; + ) + .await?; assert!(matches!( timed .verify_checkpoint( @@ -516,11 +686,12 @@ async fn checkpoint_archive_missing_or_corrupt_cell_refuses() -> Result<()> { .await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("qualification"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let reference = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -588,7 +759,7 @@ async fn forged_index( let bytes = serde_json::to_vec(value)?; let digest: [u8; 32] = Sha256::digest(&bytes).into(); let reference = serde_json::from_value(serde_json::json!({ - "digest": digest, "length": bytes.len(), + "registry": registry_id(original_path)?, "generation": generation_prefix(original_path)?.1, "digest": digest, "length": bytes.len(), }))?; let prefix = original_path .as_ref() @@ -611,11 +782,12 @@ async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("qualification"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let reference = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -676,7 +848,7 @@ async fn checkpoint_archive_untrusted_index_and_cut_binding() -> Result<()> { } for length in [0, u64::MAX] { let invalid = serde_json::from_value( - serde_json::json!({"digest": reference.digest(), "length": length}), + serde_json::json!({"registry": reference.registry(), "generation": reference.generation(), "digest": reference.digest(), "length": length}), )?; assert!(matches!( archive diff --git a/crates/cellar/tests/support/cell_group_archive_isolation.rs b/crates/cellar/tests/support/cell_group_archive_isolation.rs index 98683c76b..a05da40c3 100644 --- a/crates/cellar/tests/support/cell_group_archive_isolation.rs +++ b/crates/cellar/tests/support/cell_group_archive_isolation.rs @@ -9,11 +9,12 @@ pub(super) async fn erased_voters(rooted: bool) -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let remote = archive_store::Store::new(); let archive = new_archive( + &cluster, remote, Path::from("erased-voters"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -39,7 +40,8 @@ pub(super) async fn erased_voters(rooted: bool) -> Result<()> { let tails = CellGroupArchiveTailPool::new(Default::default())?; let reference = archive .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) - .await?; + .await? + .into_reference(); // No source physical pin, open GroupLog or voter survives the disk erasure. drop(current); drop(group); @@ -141,11 +143,12 @@ pub(super) async fn moving_cut(rooted: bool) -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let remote = archive_store::Store::new(); let archive = new_archive( + &cluster, remote.clone(), Path::from("moving-cut"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let original = source(&sources, &cluster)?; let marker = original.marker().clone(); diff --git a/crates/cellar/tests/support/cell_group_archive_publication.rs b/crates/cellar/tests/support/cell_group_archive_publication.rs new file mode 100644 index 000000000..f8a99d4e4 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_publication.rs @@ -0,0 +1,712 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual conditional object-store operations, three-voter engines and physical +//! SQLite/LTX verification. Publication is historical coverage, never serving. +use super::*; +use std::collections::BTreeMap; + +struct Scenario { + cluster: Cluster, + expected: BTreeMap>, + remote: Arc, + registry: CellGroupArchiveRegistry, + upload: CellGroupArchiveUpload, + owners: CellGroupArchiveUploadPool, + tails: CellGroupArchiveTailPool, + destination: ArchiveDestination, + receivers: CellGroupCheckpointPool, + archive_domain: ExecutionDomain, + gc_domain: ExecutionDomain, + gc_registry: CellGroupArchiveRegistry, + gc_owners: CellGroupArchiveUploadPool, + gc_tails: CellGroupArchiveTailPool, +} + +impl Scenario { + async fn new(rooted: bool) -> Result { + Self::with_owner_limits(rooted, Default::default()).await + } + + async fn with_owner_limits(rooted: bool, limits: CellGroupArchiveUploadLimits) -> Result { + let (cluster, expected) = + checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; + let remote = archive_store::Store::new(); + let archive = CellGroupArchivePool::new(Default::default())?.for_group( + remote.clone(), + Path::from("protected-head"), + cluster.birth.route().history, + )?; + let creation = archive + .prepare_registry_creation(&cluster.domain, cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&cluster, creation.intent()).await?; + assert_eq!(creation.intent(), &saved); + let registry = creation.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(limits)?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let upload = next_upload(&cluster, ®istry, &owners, &tails).await?; + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let archive_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + // The collector is a separate node/domain with its own admission pools. + // Both nodes coordinate exclusively through real conditional registry I/O. + let gc_domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gc_archive = CellGroupArchivePool::new(Default::default())?.for_group( + remote.clone(), + Path::from("protected-head"), + cluster.birth.route().history, + )?; + let gc_registry = gc_archive.open_registry(&gc_domain, &saved).await?; + Ok(Self { + cluster, + expected, + remote, + registry, + upload, + owners, + tails, + destination, + receivers, + archive_domain, + gc_domain, + gc_registry, + gc_owners: CellGroupArchiveUploadPool::new(Default::default())?, + gc_tails: CellGroupArchiveTailPool::new(Default::default())?, + }) + } + + async fn checkpoint( + &self, + upload: &CellGroupArchiveUpload, + ) -> Result { + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + Ok(upload + .upload_checkpoint(&self.archive_domain, source(&sources, &self.cluster)?) + .await?) + } + + async fn tail( + &self, + upload: &CellGroupArchiveUpload, + checkpoint: &CellGroupArchiveReference, + ) -> Result { + let group = self.cluster.voters[0] + .log + .cell_group(&self.cluster.birth.route().history)?; + let root = self.cluster.voters[0].owner.as_ref().unwrap().manifest(); + Ok(upload + .upload_tail(&self.archive_domain, &self.tails, checkpoint, &group, root) + .await?) + } + + async fn publish( + &self, + upload: &CellGroupArchiveUpload, + uploaded: &CellGroupArchiveUploadedTail, + ) -> Result { + let prepared = upload + .prepare_publication( + &self.archive_domain, + &self.tails, + uploaded, + receiver(&self.receivers, &self.destination, &self.cluster)?, + ) + .await?; + let saved = persist_archive_intent(&self.cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &saved); + Ok(prepared.commit(&self.archive_domain).await?) + } + + async fn collect(&self) -> Result { + Ok(self + .gc_registry + .collect( + &self.gc_domain, + &self.gc_owners, + &self.gc_tails, + Default::default(), + ) + .await?) + } + + async fn finish(self) -> Result<()> { + self.archive_domain.shutdown().await; + self.gc_domain.shutdown().await; + self.cluster.close().await + } +} + +async fn next_upload( + cluster: &Cluster, + registry: &CellGroupArchiveRegistry, + owners: &CellGroupArchiveUploadPool, + tails: &CellGroupArchiveTailPool, +) -> Result { + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + Ok(registry + .open_upload(&cluster.domain, owners, &ticket, tails) + .await?) +} + +async fn control_path(remote: &dyn ObjectStore) -> Result { + Ok(remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|m| m.location.as_ref().ends_with("/ownership/registry")) + .context("registry path")? + .location) +} + +fn object_path(prefix: &str, kind: &str, object: &serde_json::Value) -> Result { + let digest: Vec = serde_json::from_value(object["digest"].clone())?; + Ok(Path::from(format!( + "{prefix}/registries/{}/generations/{}/{kind}/{}-{}", + hex::encode(serde_json::from_value::>( + object["registry"].clone() + )?), + object["generation"], + object["length"], + hex::encode(digest) + ))) +} + +async fn held_root_path(remote: &dyn ObjectStore, hold: &CellGroupArchiveHold) -> Result { + let path = control_path(remote).await?; + let prefix = path + .as_ref() + .strip_suffix("/ownership/registry") + .context("registry namespace")?; + object_path( + prefix, + "roots", + &serde_json::to_value(hold)?["root"]["object"], + ) +} + +pub(super) async fn complete_source_cut(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let (_, original) = index(s.remote.as_ref(), &checkpoint).await?; + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let uploaded = s.tail(&s.upload, &checkpoint).await?; + let (cp_path, _) = index(s.remote.as_ref(), &checkpoint).await?; + let dependency = cell_objects(&cp_path, &original, "beta")[0].clone(); + // A reservation protects the whole in-progress generation before any head + // exists. It cannot justify publishing a missing dependency. + assert_eq!(s.collect().await?.deleted_objects, 0); + let gate = s.remote.hold(dependency.clone()); + let mut pending = Box::pin(s.upload.prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&s.receivers, &s.destination, &s.cluster)?, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("publication ended before delayed B") + } + } + assert!(matches!( + s.registry.prepare_acquire(&s.cluster.domain).await, + Err(CellGroupArchiveError::Invalid("no selected archive")) + )); + // Source authority was captured with the tail, before the source advanced. + // Another cell can commit without invalidating that immutable committed cut. + checkpoint_fixture::advance_partition( + &mut s.cluster, + "beta", + &s.expected["beta"], + rooted, + Some(2), + ) + .await?; + let later = s.cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .clone(); + s.remote.clear(); + gate.release.notify_one(); + let prepared = pending.await?; + let intent = persist_archive_intent(&s.cluster, prepared.intent()).await?; + assert_eq!(prepared.intent(), &intent); + let coverage = prepared.commit(&s.archive_domain).await?; + assert!(coverage.marker().position().index < later.position().index); + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + let hold = coverage.hold().clone(); + drop(coverage); + // A later A cut still includes unchanged B. Corrupt B on the actual store, + // and no new head or remote reference may be created. + let next = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let extended = next + .extend_tail( + &s.archive_domain, + &s.tails, + uploaded.reference(), + &s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)?, + s.cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + let before = s + .remote + .get(&control_path(s.remote.as_ref()).await?) + .await? + .bytes() + .await?; + let bytes = s.remote.inner.get(&dependency).await?.bytes().await?; + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 0x80; + s.remote.inner.put(&dependency, corrupt.into()).await?; + assert!(next + .prepare_publication( + &s.archive_domain, + &s.tails, + &extended, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await + .is_err()); + let after = s + .remote + .get(&control_path(s.remote.as_ref()).await?) + .await? + .bytes() + .await?; + assert_eq!(before, after); + let deletes = s.remote.deletes.load(Ordering::Acquire); + assert!(s.collect().await.is_err()); + assert_eq!( + s.remote.deletes.load(Ordering::Acquire), + deletes, + "a partial mark must issue no DELETE" + ); + s.remote.inner.put(&dependency, bytes.into()).await?; + let restored = s + .registry + .verify_hold( + &s.archive_domain, + &hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, restored.recovery().manifest(), &s.expected).await?; + drop(restored); + s.registry.abort(&s.archive_domain, &next.ticket()).await?; + s.finish().await +} + +pub(super) async fn holds_and_pending_parents(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let first = s.checkpoint(&s.upload).await?; + let first_tail = s.tail(&s.upload, &first).await?; + let first_coverage = s.publish(&s.upload, &first_tail).await?; + let first_hold = first_coverage.hold().clone(); + let first_root = held_root_path(s.remote.as_ref(), &first_hold).await?; + drop(first_coverage); + let intent = s.registry.prepare_acquire(&s.archive_domain).await?; + let intent = persist_archive_intent(&s.cluster, &intent).await?; + let reader = s.registry.acquire(&s.archive_domain, &intent).await?; + // Keep a child reservation on the old head while a different publisher wins. + let pending = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let newer = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let second = s.checkpoint(&newer).await?; + let second_tail = s.tail(&newer, &second).await?; + let second_coverage = s.publish(&newer, &second_tail).await?; + let second_hold = second_coverage.hold().clone(); + drop(second_coverage); + s.registry.release(&s.archive_domain, &first_hold).await?; + s.collect().await?; + assert!( + s.remote.get(&first_root).await.is_ok(), + "an independent reader hold protects the old root" + ); + s.registry.release(&s.archive_domain, &reader).await?; + s.collect().await?; + assert!( + s.remote.get(&first_root).await.is_ok(), + "pending child retains its exact parent" + ); + s.registry + .abort(&s.archive_domain, &pending.ticket()) + .await?; + // All old bytes still exist, but the current head's complete checkpoint + // no longer owns the old checkpoint index. A new publisher cannot resurrect + // that arbitrary old reference, even after successful physical replay. + let unprotected = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let candidate = unprotected + .extend_tail( + &s.archive_domain, + &s.tails, + first_tail.reference(), + &s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)?, + s.cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + let control = control_path(s.remote.as_ref()).await?; + let before = s.remote.inner.get(&control).await?.bytes().await?; + assert!(matches!( + unprotected + .prepare_publication( + &s.archive_domain, + &s.tails, + &candidate, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await, + Err(CellGroupArchiveError::Invalid( + "unprotected inherited archive object" + )) + )); + assert_eq!(s.remote.inner.get(&control).await?.bytes().await?, before); + s.registry + .abort(&s.archive_domain, &unprotected.ticket()) + .await?; + drop(candidate); + drop(unprotected); + let report = s.collect().await?; + assert!(report.deleted_objects > 0); + assert!(matches!( + s.remote.get(&first_root).await, + Err(object_store::Error::NotFound { .. }) + )); + let recovered = s + .registry + .verify_hold( + &s.archive_domain, + &second_hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, recovered.recovery().manifest(), &s.expected).await?; + drop(recovered); + // Retired tokens and allocations do not resurrect an owner on retry. + assert!(s + .registry + .acquire(&s.archive_domain, &intent) + .await + .is_err()); + assert!(s + .registry + .open_upload(&s.archive_domain, &s.owners, &pending.ticket(), &s.tails) + .await + .is_err()); + s.finish().await +} + +pub(super) async fn uncertain_publication(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let uploaded = s.tail(&s.upload, &checkpoint).await?; + let prepared = s + .upload + .prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + let saved = persist_archive_intent(&s.cluster, prepared.intent()).await?; + let path = control_path(s.remote.as_ref()).await?; + let gate = s + .remote + .hold_after(path, s.remote.puts.load(Ordering::Acquire) + 1); + s.remote.lose_put_response.store(true, Ordering::Release); + let mut pending = Box::pin(prepared.commit(&s.archive_domain)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("commit ended before readback cut"), + } + drop(pending); // The conditional write landed, but the producer has no reply. + s.remote.clear(); + gate.release.notify_one(); + assert_eq!(s.tails.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + for voter in &mut s.cluster.voters { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + let puts = s.remote.puts.load(Ordering::Acquire); + let coverage = s + .registry + .reconcile_publication( + &s.archive_domain, + &saved, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + assert_eq!( + s.remote.puts.load(Ordering::Acquire), + puts, + "reconciliation must not reallocate or republish" + ); + assert!( + s.registry.reserve(&s.archive_domain, &saved).await.is_err(), + "serialized publication intents are not publishing authority" + ); + let hold = coverage.hold().clone(); + drop(coverage); + s.registry.release(&s.archive_domain, &hold).await?; + assert!(s + .registry + .reconcile_publication( + &s.archive_domain, + &saved, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)? + ) + .await + .is_err()); + s.finish().await +} + +pub(super) async fn delayed_delete(rooted: bool) -> Result<()> { + let s = Scenario::new(rooted).await?; + let first = s.checkpoint(&s.upload).await?; + let (old_index, old_metadata) = index(s.remote.as_ref(), &first).await?; + s.registry + .abort(&s.archive_domain, &s.upload.ticket()) + .await?; + let gate = s.remote.hold_delete(old_index.clone()); + let mut collection = Box::pin(s.gc_registry.collect( + &s.gc_domain, + &s.gc_owners, + &s.gc_tails, + Default::default(), + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, collection.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("collector ended before held DELETE") + } + } + let next = next_upload(&s.cluster, &s.registry, &s.owners, &s.tails).await?; + let second = s.checkpoint(&next).await?; + let (new_index, new_metadata) = index(s.remote.as_ref(), &second).await?; + assert_ne!(first.generation(), second.generation()); + assert_ne!(old_index, new_index); + assert_eq!( + old_metadata["chunks"][0]["object"]["digest"], + new_metadata["chunks"][0]["object"]["digest"] + ); + assert_ne!( + old_metadata["chunks"][0]["object"]["generation"], + new_metadata["chunks"][0]["object"]["generation"] + ); + let uploaded = s.tail(&next, &second).await?; + let coverage = s.publish(&next, &uploaded).await?; + let hold = coverage.hold().clone(); + drop(coverage); + gate.release.notify_one(); + assert!(collection.await?.deleted_objects > 0); + assert!(s.remote.get(&old_index).await.is_err()); + assert!(s.remote.get(&new_index).await.is_ok()); + let recovered = s + .registry + .verify_hold( + &s.archive_domain, + &hold, + &s.tails, + receiver(&s.receivers, &s.destination, &s.cluster)?, + ) + .await?; + tail_fixture::check_images(&s.cluster, recovered.recovery().manifest(), &s.expected).await?; + drop(recovered); + s.finish().await +} + +pub(super) async fn registry_identity_and_intents(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let first = s.registry.prepare_upload(&s.archive_domain).await?; + let stale = s.registry.prepare_upload(&s.archive_domain).await?; + assert_ne!( + first, stale, + "distinct callers must not share an allocation" + ); + let saved = persist_archive_intent(&s.cluster, &first).await?; + let ticket = s.registry.reserve(&s.archive_domain, &saved).await?; + let puts = s.remote.puts.load(Ordering::Acquire); + assert_eq!(s.registry.reserve(&s.archive_domain, &saved).await?, ticket); + assert!(s.registry.reserve(&s.archive_domain, &stale).await.is_err()); + assert_eq!(s.remote.puts.load(Ordering::Acquire), puts); + s.registry.abort(&s.archive_domain, &ticket).await?; + let puts = s.remote.puts.load(Ordering::Acquire); + assert!(s.registry.reserve(&s.archive_domain, &saved).await.is_err()); + assert_eq!( + s.remote.puts.load(Ordering::Acquire), + puts, + "an aborted allocation never reappears" + ); + let next = s.registry.prepare_upload(&s.archive_domain).await?; + let saved = persist_archive_intent(&s.cluster, &next).await?; + let newer = s.registry.reserve(&s.archive_domain, &saved).await?; + assert!(newer.generation() > ticket.generation()); + + // Opening requires the owner's persisted identity. Counter loss refuses and + // cannot be repaired by creating a new identity in the same namespace. + let path = control_path(s.remote.as_ref()).await?; + let identity_path = Path::from( + path.as_ref() + .replace("/ownership/registry", "/ownership/identity"), + ); + let identity = s.remote.inner.get(&identity_path).await?.bytes().await?; + s.remote.inner.delete(&path).await?; + assert!(s + .registry + .archive() + .open_registry(&s.archive_domain, s.registry.creation()) + .await + .is_err()); + let fresh = s + .registry + .archive() + .prepare_registry_creation(&s.archive_domain, s.cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&s.cluster, fresh.intent()).await?; + assert_ne!(&saved, s.registry.creation()); + assert!(fresh.commit(&s.archive_domain).await.is_err()); + assert!(s.remote.inner.get(&path).await.is_err()); + assert_eq!( + s.remote.inner.get(&identity_path).await?.bytes().await?, + identity + ); + // The unavailable S3 metadata does not stop the independent local quorum. + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + + // Losing the identity-create response leaves no permission to initialize or + // reset a registry on restart. A serialized intent is opening metadata only. + let uncertain = CellGroupArchivePool::new(Default::default())?.for_group( + s.remote.clone(), + Path::from("uncertain-initialization"), + s.cluster.birth.route().history, + )?; + let creation = uncertain + .prepare_registry_creation(&s.archive_domain, s.cluster.birth.route().clone()) + .await?; + let saved = persist_archive_intent(&s.cluster, creation.intent()).await?; + s.remote.lose_put_response.store(true, Ordering::Release); + assert!(matches!( + creation.commit(&s.archive_domain).await, + Err(CellGroupArchiveError::Unconfirmed) + )); + assert!(uncertain + .open_registry(&s.archive_domain, &saved) + .await + .is_err()); + let controls: Vec<_> = s + .remote + .list(Some(&Path::from("uncertain-initialization"))) + .try_collect() + .await?; + assert_eq!(controls.len(), 1); + assert!(controls[0] + .location + .as_ref() + .ends_with("/ownership/identity")); + s.finish().await +} + +pub(super) async fn source_owner_lifetimes(rooted: bool) -> Result<()> { + let mut s = Scenario::with_owner_limits( + rooted, + CellGroupArchiveUploadLimits { + max_owners: 2, + ..Default::default() + }, + ) + .await?; + assert_eq!(s.owners.usage().operations, 1); + let checkpoint = s.checkpoint(&s.upload).await?; + let first = s.tail(&s.upload, &checkpoint).await?; + assert_eq!(s.owners.usage().operations, 2); + let retained = first.clone(); + drop(first); + assert_eq!( + s.owners.usage().operations, + 2, + "clones retain the source witness owner" + ); + assert!(matches!(s.tail(&s.upload, &checkpoint).await, + Err(error) if error.downcast_ref::().is_some_and(|e| matches!(e, CellGroupArchiveError::Busy)))); + assert_eq!(s.tails.usage().operations, 0); + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + s.expected.insert( + "alpha".into(), + images::image(Some(&s.expected["alpha"]), "UPDATE value SET n=n+1")?, + ); + let _persistent_locator = retained.into_reference(); + assert_eq!(s.owners.usage().operations, 1); + let latest = s.tail(&s.upload, &checkpoint).await?; + assert_eq!(s.owners.usage().operations, 2); + let coverage = s.publish(&s.upload, &latest).await?; + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + drop(coverage); + drop(latest); + assert_eq!(s.owners.usage().operations, 1); + s.finish().await +} diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index 7f7cdcd6b..161e3a3d1 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -21,7 +21,10 @@ pub struct Store { pub inner: Arc, pub puts: AtomicUsize, pub lose_put_response: AtomicBool, - held: Mutex)>>, + pub deletes: AtomicUsize, + pub lose_delete_response: AtomicBool, + held: Mutex)>>, + held_delete: Mutex)>>, } impl Store { @@ -30,13 +33,26 @@ impl Store { inner: Arc::new(memory::InMemory::new()), puts: AtomicUsize::new(0), lose_put_response: AtomicBool::new(false), + deletes: AtomicUsize::new(0), + lose_delete_response: AtomicBool::new(false), held: Mutex::new(None), + held_delete: Mutex::new(None), }) } pub fn hold(&self, path: Path) -> Arc { + self.hold_after(path, 0) + } + + pub fn hold_after(&self, path: Path, puts: usize) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held.lock().unwrap() = Some((path, puts, gate.clone())); + gate + } + + pub fn hold_delete(&self, path: Path) -> Arc { let gate = Arc::new(Gate::default()); - *self.held.lock().unwrap() = Some((path, gate.clone())); + *self.held_delete.lock().unwrap() = Some((path, gate.clone())); gate } @@ -84,8 +100,8 @@ impl ObjectStore for Store { .lock() .unwrap() .as_ref() - .filter(|(held, _)| held == path) - .map(|(_, gate)| gate.clone()); + .filter(|(held, puts, _)| held == path && self.puts.load(Ordering::Acquire) >= *puts) + .map(|(_, _, gate)| gate.clone()); if let Some(gate) = gate { gate.entered.notify_one(); gate.release.notified().await; @@ -94,7 +110,26 @@ impl ObjectStore for Store { } async fn delete(&self, path: &Path) -> Result<()> { - self.inner.delete(path).await + self.deletes.fetch_add(1, Ordering::AcqRel); + let gate = self + .held_delete + .lock() + .unwrap() + .as_ref() + .filter(|(held, _)| held == path) + .map(|(_, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + self.inner.delete(path).await?; + if self.lose_delete_response.swap(false, Ordering::AcqRel) { + return Err(Error::Generic { + store: "lost DELETE response", + source: "injected".into(), + }); + } + Ok(()) } fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result> { diff --git a/crates/cellar/tests/support/cell_group_archive_tail.rs b/crates/cellar/tests/support/cell_group_archive_tail.rs index 4d6b4209a..0d535aac8 100644 --- a/crates/cellar/tests/support/cell_group_archive_tail.rs +++ b/crates/cellar/tests/support/cell_group_archive_tail.rs @@ -15,7 +15,12 @@ pub(super) async fn tail_index( let objects: Vec<_> = store.list(None).try_collect().await?; let path = objects .into_iter() - .find(|m| m.location.as_ref().ends_with(&suffix)) + .find(|m| { + m.location.as_ref().ends_with(&suffix) + && generation_prefix(&m.location) + .is_ok_and(|(_, generation)| generation == reference.generation()) + && registry_id(&m.location).is_ok_and(|id| id == reference.registry()) + }) .context("tail index")? .location; let body = store.get(&path).await?.bytes().await?; @@ -24,13 +29,10 @@ pub(super) async fn tail_index( fn entry_path(path: &Path, entry: &serde_json::Value) -> Result { let digest: Vec = serde_json::from_value(entry["object"]["digest"].clone())?; - let prefix = path - .as_ref() - .split("/tails/") - .next() - .context("tail prefix")?; + let (prefix, _) = generation_prefix(path)?; Ok(Path::from(format!( - "{prefix}/entries/{}-{}", + "{prefix}/generations/{}/entries/{}-{}", + entry["object"]["generation"], entry["object"]["length"], hex::encode(digest) ))) @@ -59,7 +61,7 @@ async fn write_tail( ) .await?; Ok(serde_json::from_value( - serde_json::json!({"digest": digest, "length": bytes.len()}), + serde_json::json!({"registry": registry_id(original)?, "generation": generation_prefix(original)?.1, "digest": digest, "length": bytes.len()}), )?) } @@ -105,11 +107,12 @@ pub(super) async fn enrollment_controls(rooted: bool) -> Result<()> { cluster.voters[2].paused = true; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("tail-qualification"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -192,7 +195,7 @@ pub(super) async fn enrollment_controls(rooted: bool) -> Result<()> { ) .await?; assert_eq!(recovered.manifest().marker(), current.marker()); - assert_eq!(recovered.reference(), &reference); + assert_eq!(recovered.reference(), reference.reference()); assert!(recovered.configuration().voters_outgoing.is_empty()); assert_eq!(recovered.configuration().learners, vec![3]); assert_eq!(recovered.membership(), &expected_membership); @@ -231,11 +234,12 @@ pub(super) async fn dependencies(rooted: bool) -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("tail-qualification"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -366,11 +370,12 @@ pub(super) async fn warm_replay(rooted: bool) -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("tail-warm"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -469,7 +474,7 @@ pub(super) async fn warm_replay(rooted: bool) -> Result<()> { body.extend(command); let digest: [u8; 32] = Sha256::digest(&body).into(); let mut forged = original.clone(); - forged["entries"][0]["object"] = serde_json::json!({"digest": digest, "length": body.len()}); + forged["entries"][0]["object"] = serde_json::json!({"registry": original["entries"][0]["object"]["registry"], "generation": original["entries"][0]["object"]["generation"], "digest": digest, "length": body.len()}); forged["bytes"] = serde_json::json!(body.len()); let object = entry_path(&path, &forged["entries"][0])?; store.inner.put(&object, body.into()).await?; @@ -546,11 +551,12 @@ pub(super) async fn physical_cancellation() -> Result<()> { let expected = checkpoint_fixture::enroll_images(&mut cluster, false, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store, Path::from("tail-cancellation"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -666,11 +672,12 @@ pub(super) async fn extend_compacted(rooted: bool) -> Result<()> { checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; let store = archive_store::Store::new(); let archive = new_archive( + &cluster, store.clone(), Path::from("tail-extension"), - cluster.birth.route().history, Default::default(), - )?; + ) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index fa79ef1f1..0cb3d89cf 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -520,8 +520,10 @@ root under the cache read lock, then encodes/uploads without that lock. Entries have bounded canonical framing, including context and sync metadata. The complete tail index names the original checkpoint, every immutable entry object, and the final root/configuration/permanent membership. Missing ranges, unsupported types, -invalid cuts and configured byte/entry/cell limits refuse. Upload still returns -an untrusted locator rather than coverage authority. +invalid cuts and configured byte/entry/cell limits refuse. Upload returns a typed +`CellGroupArchiveUploadedTail` retaining the exact committed-source observation. +Its persistent reference remains an untrusted locator. `into_reference` discards +the local source witness; deserializing a locator cannot recreate that witness. `extend_tail` reuses an existing archived prefix and captures only the local suffix after its cut. Its flattened index directly names the original checkpoint @@ -560,11 +562,93 @@ manifest roots: cloning `PreparedGroupManifest` does not clone its parent archiv admission. Controller ownership propagation and private-cache maintenance require CG-08 qualification before managed integration. -This remains data-only recovery through an independent physical storage owner. -Published/GC-protected archive coverage, S3 settlement/outage admission, -archive-aware file reclamation and fenced fresh-incarnation seeding remain CG-06 -work. The task scope now explicitly includes the shared checkpoint/engine fixture -changes and new transition/tail modules; all required cases and oracles are kept. +The physical verifier remains data-only. CG-06's separate publication owner now +combines that proof with the exact committed source and persistent remote ownership. +S3 command settlement/outage admission, local Raft file reclamation and fenced +fresh-incarnation seeding remain unfinished; no required task oracle is changed. + +### Protected publication and remote collection + +A node shares one `CellGroupArchiveUploadPool` for upload maps, source witnesses +and collector marks, alongside the existing I/O, checkpoint and tail pools. Its +explicit byte/object/owner bounds include inherited-key lookup maps and compact +collection candidates. Clones retain their admission. Source-witness admission is +held until its final clone closes, independently of the asynchronous upload job. + +Fresh registry initialization uses a one-use, non-deserializable +`PreparedCellGroupArchiveCreation`. Persist its entire `intent()` in the durable +authoritative owner before commit. Opening requires that exact expected creation +identity; never discover and adopt a replacement mutable identity on restart. +Established identity/state loss refuses admission. A lost identity-create reply +with no complete state requires authority recovery; it cannot reset counters. +Controller/catalog integration must durably bind this expected identity before +archive use and make it recoverable without the voter disks. That integration is +still required before production activation or disaster recovery. + +Persist each allocation intent before CAS and retry only that exact intent after +an uncertain outcome. Its registry, next revision, nonce and complete operation +form its identity. Definite conflicts can expire it. Live exact results reconcile +a lost reply even after unrelated registry changes; retired/aborted results never +reappear. Revision exhaustion refuses. Registry records are bounded to 64 KiB, +eight pending generations and 64 explicit holds. These are simultaneous bounds; +retirement permits continued lifetime churn without growing tombstone lists. + +A reservation protects its never-reused upload generation and its exact selected +parent, atomically at allocation. New object keys include both the registry ID +and generation. Reuse is restricted to exact objects in that protected parent +closure, with GET verification; an arbitrary old content address cannot be +resurrected. Binding checks every inherited key even after successful physical +replay. Dropping a local handle never releases a remote reservation or hold. No +unknown owner expires by wall time. A replaced process must be fenced before its +persisted allocation can be shared or retired. + +Tail capture seals a local committed-source witness with the real engine owner, +root, configuration and permanent membership before asynchronous upload. Source +writes and local checkpoint compaction may advance afterward. Publication +rechecks that sealed prefix against the live source, independently recovers the +entire immutable checkpoint/tail and matches its complete cut to the witness. +Neither a raw locator, a private archive root nor a serialized publication intent +can replace the source witness for a new publication. + +`prepare_publication` writes an immutable root descriptor in the pending generation +and prepares one exact publication intent. Persist it before `commit`. One CAS +consumes the reservation, selects the non-regressing complete root and allocates +an explicit remote hold; independent readback confirms the exact result. Equal +indices require the same complete marker, and terms cannot regress. A later peer +mutation does not invalidate a still-live exact result. Lost readback/cancellation +leaves its intent for explicit reconciliation; it never allocates another hold +or silently substitutes a newer mutable head. Reconciliation only finds an +already-published hold and independently recovers that exact immutable root. + +Readers acquire and persist their own independent hold before downloading a +selected root. `verify_hold` reads that exact closure and rechecks its hold before +returning `CellGroupArchiveCoverage`. Clones keep local pins/admission; remote +retirement is explicit, after the durable owner retires/fences all uses and joins +entered work. A predecessor's late release cannot revoke a replacement's distinct +hold. Coverage is historical data durability, without current quorum, serving, +membership or disaster-recovery authorization. S3 command output settlement +still needs its operation-specific receipt/version and current routing checks. + +Remote collection snapshots one conditional registry revision, selected roots, +explicit holds, pending parents and pending generations. It expands every root's +exact dependency keys and downloads/hashes every marked object before issuing +any DELETE. Missing/corrupt protected data or metadata refuses the entire mark. +Collection excludes marked keys, complete pending generations and generations +newer than its watermark. New reservations can inherit only already-protected +parents; retired generations cannot be rebound. Thus concurrent publication +cannot resurrect an unmarked object, and a delayed DELETE cannot erase a new +copy of identical content under another generation. Unknown DELETE outcomes may +be retried against the same permanently obsolete key. Control identities, +registries, other groups and legacy namespaces are never collection candidates. +Scans/deletions and compact candidate inventories have explicit limits. + +This remote object collection does not authorize local shared-engine Raft file +purge. Controller ownership reconciliation, private-cache maintenance, S3 +settlement with bounded unarchived bytes, immutable source birth/fence inventory, +fresh group/cell incarnations and fresh-quorum serving remain required. The +fencing inventory must cover every possible old voting lineage, including joins +after the selected archived cut. Real RustFS, Docker fencing and measured-load +qualification remain later gates. ## Split and merge diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 041b36fdc..9edbf03eb 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1740,7 +1740,15 @@ "crates/cellar/tests/support/cell_group_archive_tail.rs", "crates/cellar/durability/raft/cell_groups/archive/data_store.rs", "crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs", - "crates/cellar/tests/support/cell_group_archive_isolation.rs" + "crates/cellar/tests/support/cell_group_archive_isolation.rs", + "crates/cellar/durability/raft/cell_groups/archive/collection.rs", + "crates/cellar/durability/raft/cell_groups/archive/graph.rs", + "crates/cellar/durability/raft/cell_groups/archive/publication.rs", + "crates/cellar/durability/raft/cell_groups/archive/upload.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs", + "crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs", + "crates/cellar/tests/support/cell_group_archive_publication.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 873484774c1ad5fb261257e4890291f8ad956f6e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 08:05:02 -0700 Subject: [PATCH 170/568] fix: unwrap reconciled archive hold --- .../cellar/durability/raft/cell_groups/archive/ownership/mod.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs index e7e87c6f1..787e63a56 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -506,7 +506,7 @@ impl CellGroupArchiveRegistry { return Err(CellGroupArchiveError::Invalid("publication intent purpose")); } match self.load().await?.0.reconcile(intent)? { - Some(Allocation::Hold(hold)) => Ok(hold), + Some(Allocation::Hold(hold)) => Ok(*hold), _ => Err(CellGroupArchiveError::Expired), } } From 6e9e05bdd90f2a498eb528170b33bbafea590cf2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 08:32:19 -0700 Subject: [PATCH 171/568] docs: record protected archive publication gate --- docs/design/multi-raft/cell-groups/README.md | 7 +- docs/design/multi-raft/cell-groups/handoff.md | 84 ++++++++++--------- .../design/multi-raft/cell-groups/progress.md | 67 +++++++++++++++ 3 files changed, 117 insertions(+), 41 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 8fa319ad3..b32fe011b 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -7,9 +7,10 @@ CG-01 through CG-05 have passed their scoped native evidence gates. CG-05's complete checkpoint, authenticated learner/rebuild, membership, restart and resource-ownership requirements pass at `e0550acd`. Its current gate includes all five exact cases and 205 Rust regressions across eighteen commands. -CG-06 is IN_PROGRESS: independent checkpoint/tail data recovery passes after all -voter directories are erased. Protected S3 coverage, settlement and safe log -reclamation remain the next priorities. +CG-06 is IN_PROGRESS: independent data recovery after voter-directory erasure and +protected complete-cut publication/remote collection pass native gates. S3 output +settlement, bounded outage admission, local Raft file purge and fenced seeding +remain the next priorities. The production SDK/fleet has not yet cut over; topology, hibernation, complete S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 0ccb3f818..4d2fd0478 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,45 +15,53 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial source is `14955f2dc2b01a6f6ab6b75b0122c812e5de6765`. -CG-06 is IN_PROGRESS. Complete checkpoints and contiguous tails now recover through -an independent physical archive store, with no destination Raft engine, member -claim, replacement genesis or surviving voter disk. Both image forms recover after -all three source voter directories and extent directories are deleted. A held -remote dependency permits source quorum progress in another execution domain; -newer and older cuts verify independently. Private archive roots are refused by -actual exact-cut engine publication and source capture. Closed stores refuse new -and outstanding receives. Newly replayed private images become reclaimable after -proof pins close; full remote replay succeeds after their physical eviction. - -All 24 commands pass: eleven exact archive cases, 156 execution cases, five -snapshot cases and four generation regressions (176 distinct Rust cases), -all-target build/Clippy, formatting/layout, dependencies/docs/preflight and 37 -validator tests. The record under -`.cellar/cell-groups-evidence/CG-06-isolated-verifier/qualification/` -binds 326 source hashes, four executables, frozen inputs and the previous tail -checkpoint. Independent hashes/source/exact-selection verification passes. +The latest partial source is `873484774c1ad5fb261257e4890291f8ad956f6e`. +CG-06 is IN_PROGRESS. Protected publication now combines a sealed committed-source +witness with independently recovered checkpoint/tail data. One conditional registry +update selects a non-regressing complete root and allocates an explicit remote hold. +Exact persisted intents reconcile unknown replies without republishing or allocating +again; retired/aborted tokens cannot resurrect. Source writes can advance while a +delayed dependency blocks publication of the earlier sealed cut. + +Keys include a registry ID and never-reused generation. Pending owners retain their +exact parent closure, and arbitrary unprotected old dependencies refuse even after +successful physical replay. Remote collection verifies every protected object before +DELETE and retains selected roots, reader holds, pending parents/generations and +newer generations. Native races prove late old-generation DELETE cannot erase an +identical new copy and actual remote reclamation preserves the current cut. Missing +counters/uncertain initial creation refuse; source-witness clones retain node quota. + +All 30 commands pass: seventeen exact archive cases, 156 execution cases, five +snapshot cases and four generation regressions (182 distinct Rust cases), all-target +build/Clippy, formatting/layout, dependencies/docs/preflight and 37 Python tests. +The record under +`.cellar/cell-groups-evidence/CG-06-protected-head/qualification/` +binds 334 source hashes, four executables, frozen inputs and the previous neutral- +verifier checkpoint. Independent hashes/source/exact-selection verification passes. +The preceding preflight compiler failure at `a5d506a6` is retained in `build-failure/`. See [progress](progress.md) and the -[archive contract](protocols.md#apply-checkpoint-and-archive). - -The original `archive_tails_cover_enrollment_and_controls` case and all prior -missing/corrupt/warm-LTX, physical cancellation/join and post-compaction extension -cases remain passing on the new neutral stores. The other four original CG-06 -cases and full task gate remain pending. No required oracle or bound was weakened. - -Next implement conditionally published, GC-protected complete-cut coverage, -bounded outage admission/S3 settlement, archive-aware physical log reclamation -and fenced unmanaged seeding under fresh group/cell incarnations. Keep source -birth/provenance and the authoritative inventory of every possible old voting -lineage, including joins after the chosen recovery cut. Selected-cut membership -alone is insufficient to fence every old host. Physical data verification grants -no serving, coverage, S3 settlement or fencing authority. Shared dispatch still -refuses S3 durability. The native erasure drill uses the actual in-memory store; -RustFS, Docker fencing, fresh-quorum serving, load and public hard cutover remain -later required work. Retain the preceding committed-tail record at `7a29c2a9`. -Before managed integration, retain the verified archive result alongside derived -manifest roots: raw root clones do not preserve the parent archive quota. Qualify -that ownership propagation and private-cache maintenance in CG-08. +[archive contract](protocols.md#protected-publication-and-remote-collection). + +Next persist the exact expected registry identity in authoritative namespace/owner +state and wire the publisher into node-owned work. Implement real S3 output settlement +with receipt/version/current-route proofs, captured identity and no callback replay; +add bounded unarchived-byte/disk admission independent of Quorum. Shared dispatch +still refuses S3. Then qualify actual local shared-engine Raft file purge and fenced +unmanaged seeding with fresh group/cell incarnations. Source birth/provenance and +all possible old voter lineages, including joins after the recovery cut, must survive +voter loss. Selected-cut membership alone cannot establish the complete fence. + +Fresh registry creation is a non-deserializable one-use authority. Persist its intent +before commit; restart only opens the exact expected identity. An incomplete identity +requires authority recovery, never automatic counter recreation. Coverage is historical +data durability; it supplies no current quorum, serving or DR authorization. Controller +composition must retain verified result admission alongside derived raw roots; CG-08 +still needs that propagation and private-cache maintenance before managed integration. + +The actual in-memory adapter supplies native evidence. Real RustFS, Docker fencing, +fresh-quorum serving, load and public hard cutover remain later gates. The other four +original CG-06 cases and full task gate remain pending. Preserve the preceding independent- +verifier record at `14955f2d` and committed-tail record at `7a29c2a9`. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index da4a9920d..69887c736 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,73 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 protected publication and remote object collection + +Source: `873484774c1ad5fb261257e4890291f8ad956f6e`. CG-06 remains IN_PROGRESS. +A complete immutable checkpoint/tail can now be published with a persistent remote hold, using a sealed +committed-source witness and independent physical replay. This is historical +complete-cut coverage; command S3 settlement and local Raft file purge remain +separate unfinished boundaries. + +Immutable keys now contain a registry ID and never-reused allocation generation. +Marker 1 makes a hard cutover, with no legacy reader or deletion of old namespaces. +Pending generations retain their exact selected parent, and uploads reuse only +objects in that protected closure. Fresh creation is a one-use capability, with +its intent durably recorded before mutation. Opening requires the owner's exact +persisted creation identity. Missing/corrupt established state and uncertain +identity creation without complete state refuse; neither resets counters. + +Tail capture retains a typed committed-source witness before upload. New source +writes may advance while B is delayed, and the older sealed cut can still publish +after every dependency verifies. Publication atomically consumes the reservation, +selects a non-regressing descriptor and creates an explicit remote hold. An exact +persisted intent reconciles a lost response/readback cancellation without another +allocation or publication, including after the source engines close. Serialized +locators/intents cannot create source authority. Retired/aborted intents never +resurrect; reader holds and pending parents remain independent of newer heads. + +Collection fully expands and hashes every protected closure before any DELETE. +It excludes marked keys, complete pending generations and newer generations. +Actual remote deletions preserve current recovered data; an old-generation DELETE +held while identical bytes are reuploaded and published cannot erase the new copy. +Corrupt B blocks both new coverage and complete collection. A physically valid +old checkpoint outside the new upload's protected parent refuses publication. +Compact marks/candidates and source witnesses have explicit node admission. +A two-owner test retains a witness through clones, refuses another capture while +quorum writes continue, and refunds only after the final witness drops. + +All 30 commands pass: seventeen exact archive cases, 156 execution cases, five +snapshot cases and four LTX generation regressions (182 distinct Rust cases across +four executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-protected-head/qualification/checkpoint.json` +binds 334 source hashes, four copied executables, frozen inputs, hashed outputs +and the preceding independent-verifier record. Independent source/artifact/exact- +selection verification passes. The first qualification build at `a5d506a6` caught one +missed unboxing branch after the Clippy enum-layout fix; its source-bound failed +output is retained under `build-failure/`. The correction and full gate use the +source above. No required case/oracle or pre-existing quota was weakened. + +Reflection and next priorities: wire protected publication into the node controller +and persist the expected registry identity in authoritative namespace/owner state. +Then settle actual S3 command/enrollment/receipt outputs against exact cell versions, +results and current routing authority; preserve captured identities/inputs and +never rerun callbacks. Replace the Quorum-only fingerprint/guards only when those +proofs exist. Add explicit unarchived-byte/disk admission while Quorum can continue +without S3. Follow with local shared-engine file purge and fenced unmanaged seeding +under fresh group/cell incarnations. Persist source birth/provenance and fence all +possible old voting lineages, including joins after the selected archived cut. + +CG-08 still owns controller/cache lifetime composition and private-cache maintenance; +retain verified result admission alongside derived raw manifest roots. Then CG-07 +cuts public SDK/fleet routing over, followed by topology, placement, hibernation, +managed recovery, RustFS/load/domain qualification and final legacy removal. +The native fixture uses the actual in-memory object-store adapter; no real RustFS, +Docker fence, fresh-incarnation serving or performance claim follows. The other +four original CG-06 cases and full task gate remain pending. Shared dispatch still +refuses S3 durability. Transfer the ignored evidence and its predecessors for audit. + ## CG-06 archive verification without voter metadata Source: `14955f2dc2b01a6f6ab6b75b0122c812e5de6765`. CG-06 remains IN_PROGRESS. From df0ba1f67c0253a57127a215540f08bb13135279 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 09:00:45 -0700 Subject: [PATCH 172/568] feat: bind archive identity and capture stable cuts Persist the expected group archive registry in namespace metadata so voter loss does not require adopting an untrusted replacement identity. Seal a complete checkpoint before upload to preserve its committed cut while quorum advances, using the existing runtime worker and node pools. --- crates/cellar/durability/raft/catalog.rs | 9 +- .../raft/cell_groups/archive/ownership/mod.rs | 4 + .../raft/cell_groups/archive/publication.rs | 52 ++++- .../raft/cell_groups/archive/tail.rs | 110 +++++---- .../raft/cell_groups/checkpoint/mod.rs | 22 ++ .../raft/cell_groups/directory/archive.rs | 211 ++++++++++++++++++ .../raft/cell_groups/directory/mod.rs | 2 + .../raft/cell_groups/directory/wire.rs | 1 + .../cellar/durability/raft/cell_groups/mod.rs | 9 +- .../durability/raft/cell_groups/owner.rs | 9 + .../raft/cell_groups/runtime/mod.rs | 27 +++ .../raft/cell_groups/runtime/worker.rs | 5 + .../tests/native_cell_groups_archive.rs | 15 ++ .../tests/native_cell_groups_execution.rs | 5 + .../support/cell_group_archive_binding.rs | 157 +++++++++++++ .../support/cell_group_archive_isolation.rs | 36 ++- .../support/cell_group_archive_publication.rs | 67 ++++++ .../tests/support/cell_group_runtime.rs | 73 ++++++ .../multi-raft/cell-groups/protocols.md | 27 ++- docs/design/multi-raft/cell-groups/tasks.json | 4 +- 20 files changed, 776 insertions(+), 69 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/archive.rs create mode 100644 crates/cellar/tests/support/cell_group_archive_binding.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 4b0846445..0bf9e85ec 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -26,10 +26,11 @@ pub use births::BirthDisposition; mod range_directory; pub(crate) use range_directory::RouteCache; pub use range_directory::{ - CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, - CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, - DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, + CellGroupBootstrapPermit, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, + CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, + PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs index 787e63a56..2347ddab0 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -46,6 +46,10 @@ impl PreparedCellGroupArchiveCreation { } impl CellGroupArchiveCreation { + pub fn route(&self) -> &GroupRoute { + &self.route + } + fn check(&self, history: GroupHistory) -> Result<()> { if self.domain != IDENTITY_DOMAIN || self.id == [0; 32] || self.route.history != history { return Err(CellGroupArchiveError::Invalid("archive registry identity")); diff --git a/crates/cellar/durability/raft/cell_groups/archive/publication.rs b/crates/cellar/durability/raft/cell_groups/archive/publication.rs index 566835789..6d09c8237 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/publication.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/publication.rs @@ -25,15 +25,57 @@ impl SourceCut { root: PreparedGroupManifest, configuration: ConfState, membership: GroupMembership, - reservation: Arc, - ) -> Self { - Self { + source_budget: Arc, + ) -> Result { + root.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if matches!(pin, super::super::CellImagePin::Extents(pin) if pin.reference().encoded_length() > 1024 * 1024) { + return Err(CellGroupArchiveError::Limit); + } + Ok(()) + })?; + let mut source_bytes = root + .reserved_bytes() + .checked_add( + crate::raft::manifest::encode_conf(&configuration) + .map_err(|_| CellGroupArchiveError::Invalid("tail configuration encoding"))? + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .and_then(|n| n.checked_add(64 * 1024)) + .ok_or(CellGroupArchiveError::Limit)?; + source_bytes = source_bytes + .checked_add( + serde_json::to_vec(&membership) + .map_err(|_| CellGroupArchiveError::Limit)? + .len() + .checked_mul(4) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + root.images.visit(&mut |_, _, pin, _, _| -> Result<()> { + if let super::super::CellImagePin::Extents(pin) = pin { + source_bytes = source_bytes + .checked_add( + (pin.reference().encoded_length() as usize) + .checked_mul(2) + .ok_or(CellGroupArchiveError::Limit)?, + ) + .ok_or(CellGroupArchiveError::Limit)?; + } + Ok(()) + })?; + let source_reservation = source_budget + .acquire(source_bytes) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy)?; + Ok(Self { group, root, configuration, membership, - _reservation: reservation, - } + _reservation: source_reservation, + }) } fn check(&self) -> Result<()> { diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs index 891d2cabd..f1e195ea0 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/tail.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -235,47 +235,6 @@ fn capture( let membership = cache.membership.clone(); drop(cache); // No engine/cache lock across encoding or physical/S3 work. cut.validate(group.identity().scope, limits)?; - current.images.visit(&mut |_, _, pin, _, _| -> Result<()> { - if matches!(pin, super::super::CellImagePin::Extents(pin) if pin.reference().encoded_length() > 1024 * 1024) { - return Err(CellGroupArchiveError::Limit); - } - Ok(()) - })?; - let mut source_bytes = current - .reserved_bytes() - .checked_add( - cut.conf - .len() - .checked_mul(4) - .ok_or(CellGroupArchiveError::Limit)?, - ) - .and_then(|n| n.checked_add(64 * 1024)) - .ok_or(CellGroupArchiveError::Limit)?; - source_bytes = source_bytes - .checked_add( - serde_json::to_vec(&cut.membership) - .map_err(|_| CellGroupArchiveError::Limit)? - .len() - .checked_mul(4) - .ok_or(CellGroupArchiveError::Limit)?, - ) - .ok_or(CellGroupArchiveError::Limit)?; - current.images.visit(&mut |_, _, pin, _, _| -> Result<()> { - if let super::super::CellImagePin::Extents(pin) = pin { - source_bytes = source_bytes - .checked_add( - (pin.reference().encoded_length() as usize) - .checked_mul(2) - .ok_or(CellGroupArchiveError::Limit)?, - ) - .ok_or(CellGroupArchiveError::Limit)?; - } - Ok(()) - })?; - let source_reservation = source_budget - .acquire(source_bytes) - .map(Arc::new) - .ok_or(CellGroupArchiveError::Busy)?; let entries = entries .iter() .map(|entry| { @@ -294,8 +253,8 @@ fn capture( current, configuration, membership, - source_reservation, - )), + source_budget, + )?), }) } @@ -313,6 +272,71 @@ impl Drop for Stop { } impl CellGroupArchiveUpload { + /// Upload one complete captured cut without recapturing the live engine + /// after object-store I/O. Quorum may continue applying throughout upload. + /// The checkpoint pool must admit the source before this call; its slot is + /// released before the caller starts independent recovery verification. + pub async fn upload_cut( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchiveTailPool, + checkpoint: CellGroupCheckpoint, + ) -> Result { + domain + .run(async { + let _retained = pool.admit()?; + let (group, root, configuration, membership) = checkpoint.archive_source()?; + let cut = Cut { + marker: root.marker().clone(), + conf: crate::raft::manifest::encode_conf(&configuration).map_err(|_| { + CellGroupArchiveError::Invalid("tail configuration encoding") + })?, + membership: membership.clone(), + }; + cut.validate(self.history, pool.limits)?; + let source = Arc::new(super::publication::SourceCut::from_capture( + group, + root, + configuration, + membership, + self.source_budget(), + )?); + let reference = self.upload_checkpoint(domain, checkpoint).await?; + let io = self.admit()?; + self.registry().pending(&self.ticket()).await?; + let tail = Tail { + domain: TAIL_DOMAIN.into(), + checkpoint: reference, + base: cut.marker.clone(), + cut, + bytes: 0, + entries: Vec::new(), + }; + tail.validate(self, pool.limits)?; + let mut output = checkpoint::Bounded { + bytes: Vec::new(), + limit: self.limits.max_index_bytes, + }; + serde_json::to_writer(&mut output, &tail) + .map_err(|_| CellGroupArchiveError::Limit)?; + let object = self + .put("tails", output.bytes, self.limits.max_index_bytes) + .await?; + drop(io); + Ok(CellGroupArchiveUploadedTail { + reference: CellGroupArchiveTailReference { + registry: object.registry, + generation: object.generation, + digest: object.digest, + length: object.length, + }, + source, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + /// Copy an exact contiguous committed suffix under the engine cache lock, /// then upload immutable entry objects without holding that lock. This /// returns an untrusted locator; all base bytes and replay must still verify. diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index acc730fcb..99bb4853c 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -354,6 +354,28 @@ pub struct CellGroupCheckpoint { } impl CellGroupCheckpoint { + /// Only a locally captured, unstreamed source can retain committed-source + /// authority. Received checkpoints use a different type and cannot mint it. + pub(super) fn archive_source( + &self, + ) -> Result<( + GroupLog, + PreparedGroupManifest, + raft::prelude::ConfState, + GroupMembership, + )> { + if self.started { + return Err(CellGroupCheckpointError::Invalid("source already streamed")); + } + let header = Header::decode(&self.header)?; + Ok(( + self.group.clone(), + self.root.clone(), + header.configuration()?, + header.membership, + )) + } + /// Metadata for this immutable source cut; it does not establish publication. pub fn snapshot(&self) -> Result { Header::decode(&self.header)?.snapshot() diff --git a/crates/cellar/durability/raft/cell_groups/directory/archive.rs b/crates/cellar/durability/raft/cell_groups/directory/archive.rs new file mode 100644 index 000000000..af186f949 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/archive.rs @@ -0,0 +1,211 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace-owned archive identity. Recovery reads the expected identity from +//! immutable catalog metadata, never by trusting a mutable registry's identity. +use super::*; +use crate::{ + raft::cell_groups::{ + CellGroupArchive, CellGroupArchiveCreation, CellGroupArchiveError, CellGroupArchivePool, + CellGroupArchiveRegistry, PreparedCellGroupArchiveCreation, + }, + sdk::ExecutionDomain, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Debug)] +pub enum CellGroupArchiveBindingError { + Catalog(CatalogError), + Archive(CellGroupArchiveError), +} + +impl std::fmt::Display for CellGroupArchiveBindingError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group archive binding: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveBindingError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Catalog(error) => error, + Self::Archive(error) => error, + }) + } +} + +impl From for CellGroupArchiveBindingError { + fn from(error: CatalogError) -> Self { + Self::Catalog(error) + } +} + +impl From for CellGroupArchiveBindingError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +type Result = std::result::Result; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Binding { + namespace: [u8; 32], + creation: CellGroupArchiveCreation, +} + +/// Persist `encode()` before commit. This one-use initializer first binds the +/// expected identity in the namespace catalog, then creates registry counters. +/// After a crash, open the existing binding; never allocate a replacement. +/// Incomplete initialization refuses admission and requires authority recovery. +pub struct PreparedCellGroupArchiveBinding { + directory: RangeDirectory, + birth: CellGroupBirth, + bytes: Vec, + creation: PreparedCellGroupArchiveCreation, +} + +impl PreparedCellGroupArchiveBinding { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub async fn commit(self, domain: &ExecutionDomain) -> Result { + domain + .run(async { + self.directory + .require_current_group_birth(&self.birth) + .await?; + self.directory + .bounded(async { + let observed = self + .directory + .namespace + .inner + .create_and_observe_bounded( + &self.directory.archive_binding_key(self.birth.route()), + &self.bytes, + wire::LIMIT, + ) + .await?; + if observed != self.bytes { + return Err(CatalogError::Conflict("group archive binding")); + } + Ok(()) + }) + .await?; + self.directory + .require_current_group_birth(&self.birth) + .await?; + let registry = self.creation.commit(domain).await?; + self.directory + .require_current_group_birth(&self.birth) + .await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +impl RangeDirectory { + fn archive_binding_key(&self, route: &GroupRoute) -> Path { + self.key(&format!( + "archives/{}/{}/binding", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + fn group_archive( + &self, + pool: &CellGroupArchivePool, + route: &GroupRoute, + ) -> Result { + Ok(pool.for_group( + self.namespace.inner.store.clone(), + Path::from(format!( + "{}cell-group-archives", + self.namespace.data_prefix() + )), + route.history, + )?) + } + + /// Prepare a fresh namespace-owned archive for a selected immutable birth. + /// Use one node-shared archive pool. No caller-selected backend or data key + /// can substitute another group's archive. This grants no serving authority. + pub async fn prepare_group_archive_binding( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + domain + .run(async { + self.require_current_group_birth(birth).await?; + let archive = self.group_archive(pool, birth.route())?; + let creation = archive + .prepare_registry_creation(domain, birth.route().clone()) + .await?; + let bytes = record::encode( + wire::GROUP_ARCHIVE, + &Binding { + namespace: self.namespace.root_digest(), + creation: creation.intent().clone(), + }, + wire::LIMIT, + )?; + Ok(PreparedCellGroupArchiveBinding { + directory: self.clone(), + birth: birth.clone(), + bytes, + creation, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + + /// Reopen only the immutable expected identity. Missing binding, identity or + /// counters fail closed, including after every voter disk has disappeared. + /// This does not recreate metadata, fence old owners or authorize new voters. + pub async fn open_group_archive( + &self, + domain: &ExecutionDomain, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + domain + .run(async { + self.require_current_group_birth(birth).await?; + let bytes = self + .bounded(async { + self.namespace + .inner + .read_versioned_bounded( + &self.archive_binding_key(birth.route()), + wire::LIMIT, + ) + .await? + .map(|(bytes, _)| bytes) + .ok_or(CatalogError::Corrupt("missing group archive binding")) + }) + .await?; + let binding: Binding = record::decode(wire::GROUP_ARCHIVE, &bytes, wire::LIMIT)?; + if binding.namespace != self.namespace.root_digest() + || binding.creation.route() != birth.route() + { + return Err(CatalogError::Corrupt("group archive binding identity").into()); + } + let registry = self + .group_archive(pool, birth.route())? + .open_registry(domain, &binding.creation) + .await?; + self.require_current_group_birth(birth).await?; + Ok(registry) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index 5ae9c87b1..c3c82c075 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -17,6 +17,8 @@ use std::{ time::Duration, }; +mod archive; +pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding}; mod birth; mod bootstrap; mod route_cache; diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 8c0ce8a0c..86f2e8dec 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -12,6 +12,7 @@ pub(super) const REQUEST: u8 = 85; // Fleet maintenance already uses 86 in the shared catalog envelope. pub(super) const GROUP_BIRTH: u8 = 186; pub(super) const GROUP_CLAIM: u8 = 187; +pub(super) const GROUP_ARCHIVE: u8 = 188; pub(super) const LIMIT: usize = 64 * 1024; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 34a4a3552..f8dd3700c 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -92,10 +92,11 @@ pub use manifest::{ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; pub use crate::raft::catalog::{ - CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, CellGroupBootstrapPermit, - CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, - CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, - DirectoryPublication, DirectoryView, RangeDirectory, ResolvedCellRoute, + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, + CellGroupBootstrapPermit, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, + CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, + DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, + PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 6c3e1cf52..047aadd4c 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -332,6 +332,15 @@ impl CellGroupOwner { &self.current } + pub(super) fn capture_checkpoint( + &self, + pool: &super::CellGroupCheckpointPool, + ) -> Result { + self.live() + .map_err(super::CellGroupCheckpointError::Authority)?; + pool.capture(&self.group, &self.current) + } + pub(super) fn select_read( &self, cell: &CellHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index c3a856b48..a0c529706 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -153,6 +153,7 @@ enum Command { SnapshotStatus(u64, bool), Read(Box), Apply(CellGroupTicket, Box), + CaptureCheckpoint(CellGroupCheckpointPool), Status, Tick, } @@ -169,6 +170,7 @@ enum Response { Status(CellGroupStatus), Read(Box), Seed(Box), + CheckpointSource(Box>), } struct Request { @@ -848,6 +850,31 @@ impl CellGroupRuntime { } } + /// Capture the owner's exact durable applied cut on the existing worker. + /// This performs bounded metadata work only; streaming and archive I/O run + /// outside the worker. Concurrent persistence may transiently invalidate a + /// cut, which the archive producer can retry after the next apply event. + pub async fn capture_checkpoint( + &self, + history: GroupHistory, + pool: &CellGroupCheckpointPool, + ) -> Result { + match self + .call( + Some(history), + Command::CaptureCheckpoint(pool.clone()), + 4096, + ) + .await + .map_err(CellGroupCheckpointError::Authority)? + { + Response::CheckpointSource(source) => *source, + _ => Err(CellGroupCheckpointError::Authority(RuntimeError::Protocol( + "checkpoint source reply".into(), + ))), + } + } + pub async fn tick(&self) -> Result<(), RuntimeError> { self.call(None, Command::Tick, 4096).await.map(|_| ()) } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index a0242729b..250410271 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -232,6 +232,11 @@ impl Worker { reply(request, group.owner.status().map(Response::Status)); return; } + if let Command::CaptureCheckpoint(pool) = &request.command { + let source = group.owner.capture_checkpoint(pool); + reply(request, Ok(Response::CheckpointSource(Box::new(source)))); + return; + } if let Command::SnapshotStatus(member, delivered) = request.command { if let Err(error) = group.owner.validate_snapshot_member(member) { reply(request, Err(error)); diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index c4662e3d5..3d678460b 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -9,6 +9,8 @@ #[path = "support/cell_group_archive_store.rs"] mod archive_store; +#[path = "support/cell_group_archive_binding.rs"] +mod binding_fixture; #[path = "support/cell_group_checkpoint.rs"] #[allow(dead_code)] mod checkpoint_fixture; @@ -27,6 +29,19 @@ mod tail_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_cut_upload_survives_quorum_progress() -> Result<()> { + for rooted in [false, true] { + publication_fixture::moving_upload_cut(rooted).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_catalog_binding_refuses_replacement_and_reset() -> Result<()> { + binding_fixture::identity().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn archive_tails_cover_enrollment_and_controls() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 62628d525..a3e354472 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -14,6 +14,11 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn runtime_archive_capture_retains_cut_without_blocking_quorum() -> Result<()> { + Box::pin(runtime_fixture::capture_archive_cut()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn seed_retry_materialized_reclaims_interrupted_disk() -> Result<()> { Box::pin(runtime_fixture::seed_retry(false)).await diff --git a/crates/cellar/tests/support/cell_group_archive_binding.rs b/crates/cellar/tests/support/cell_group_archive_binding.rs new file mode 100644 index 000000000..fc4102edf --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_binding.rs @@ -0,0 +1,157 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh catalog owners must recover the immutable archive identity remotely. +use super::*; + +pub(super) async fn reopen_directory( + cluster: &Cluster, +) -> Result<(RangeDirectory, CellGroupBirth)> { + let route = cluster.birth.route(); + let catalog = Catalog::new(cluster.store.clone(), Default::default())?; + let namespace = cluster + .domain + .run(catalog.existing_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, Default::default())?; + let birth = cluster + .domain + .run(directory.materialize_selected_group(route)) + .await??; + Ok((directory, birth)) +} + +pub(super) async fn identity() -> Result<()> { + let cluster = Cluster::new().await?; + let pool = CellGroupArchivePool::new(Default::default())?; + let before = cluster + .store + .list(None) + .try_collect::>() + .await? + .len(); + assert!(matches!( + cluster + .directory + .open_group_archive(&cluster.domain, &pool, &cluster.birth) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Corrupt("missing group archive binding") + )) + )); + assert_eq!( + cluster + .store + .list(None) + .try_collect::>() + .await? + .len(), + before + ); + let first = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &pool, &cluster.birth) + .await?; + let replacement = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &pool, &cluster.birth) + .await?; + assert_ne!(first.encode(), replacement.encode()); + let saved = persist_archive_intent(&cluster, &first.encode().to_vec()).await?; + assert_eq!(saved, first.encode()); + let registry = first.commit(&cluster.domain).await?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(&cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + drop(registry); + assert!(matches!( + replacement.commit(&cluster.domain).await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Conflict("group archive binding") + )) + )); + let (directory, birth) = reopen_directory(&cluster).await?; + // Even the original typed birth observation belongs to another catalog + // owner. Materialize and verify it again rather than transferring authority. + assert!(matches!( + directory + .open_group_archive(&cluster.domain, &pool, &cluster.birth) + .await, + Err(CellGroupArchiveBindingError::Catalog( + CatalogError::Conflict("foreign group birth observation") + )) + )); + let reopened = directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await?; + // Persisted exact allocations reconcile through the remotely bound identity. + assert_eq!(reopened.reserve(&cluster.domain, &saved).await?, ticket); + reopened.abort(&cluster.domain, &ticket).await?; + let objects = cluster.store.list(None).try_collect::>().await?; + let binding = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/binding")) + .context("remote binding")? + .location + .clone(); + let counter = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/ownership/registry")) + .context("remote counters")? + .location + .clone(); + let identity = objects + .iter() + .find(|m| m.location.as_ref().ends_with("/ownership/identity")) + .context("remote identity")? + .location + .clone(); + let expected_binding = cluster.store.get(&binding).await?.bytes().await?; + let expected_identity = cluster.store.get(&identity).await?.bytes().await?; + drop(reopened); + cluster.store.delete(&counter).await?; + let before = cluster + .store + .list(None) + .try_collect::>() + .await? + .len(); + assert!(directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await + .is_err()); + assert_eq!( + cluster + .store + .list(None) + .try_collect::>() + .await? + .len(), + before + ); + let replacement = directory + .prepare_group_archive_binding(&cluster.domain, &pool, &birth) + .await?; + assert!(replacement.commit(&cluster.domain).await.is_err()); + assert_eq!( + cluster.store.get(&binding).await?.bytes().await?, + expected_binding + ); + assert_eq!( + cluster.store.get(&identity).await?.bytes().await?, + expected_identity + ); + assert!(cluster.store.get(&counter).await.is_err()); + // A lost binding cannot be inferred from whichever mutable identity remains. + cluster.store.delete(&binding).await?; + assert!(directory + .open_group_archive(&cluster.domain, &pool, &birth) + .await + .is_err()); + assert!(cluster.store.get(&binding).await.is_err()); + cluster.close().await +} diff --git a/crates/cellar/tests/support/cell_group_archive_isolation.rs b/crates/cellar/tests/support/cell_group_archive_isolation.rs index a05da40c3..9b64d7e69 100644 --- a/crates/cellar/tests/support/cell_group_archive_isolation.rs +++ b/crates/cellar/tests/support/cell_group_archive_isolation.rs @@ -7,14 +7,22 @@ use cellar_logic::consensus::OperationId; pub(super) async fn erased_voters(rooted: bool) -> Result<()> { let (mut cluster, mut expected) = checkpoint_fixture::populate_images(rooted, true, false, 1, 8192).await?; - let remote = archive_store::Store::new(); - let archive = new_archive( - &cluster, - remote, - Path::from("erased-voters"), - Default::default(), - ) - .await?; + let archive_pool = CellGroupArchivePool::new(Default::default())?; + let binding = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &archive_pool, &cluster.birth) + .await?; + let saved = persist_archive_intent(&cluster, &binding.encode().to_vec()).await?; + assert_eq!(saved, binding.encode()); + let registry = binding.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let saved = persist_archive_intent(&cluster, &intent).await?; + let ticket = registry.reserve(&cluster.domain, &saved).await?; + let archive = registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?; let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let base = archive .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) @@ -37,7 +45,6 @@ pub(super) async fn erased_voters(rooted: bool) -> Result<()> { let current = cluster.voters[0].owner.as_ref().unwrap().manifest().clone(); let marker = current.marker().clone(); let membership = group.membership()?; - let tails = CellGroupArchiveTailPool::new(Default::default())?; let reference = archive .upload_tail(&cluster.domain, &tails, &base, &group, ¤t) .await? @@ -45,6 +52,9 @@ pub(super) async fn erased_voters(rooted: bool) -> Result<()> { // No source physical pin, open GroupLog or voter survives the disk erasure. drop(current); drop(group); + drop(archive); + drop(registry); + drop(owners); for voter in &mut cluster.voters { drop(voter.owner.take()); drop(voter.held.take()); @@ -70,9 +80,17 @@ pub(super) async fn erased_voters(rooted: bool) -> Result<()> { } } assert!(!directory.join("archive-data").exists()); + std::fs::remove_dir_all(directory.join("archive-intents"))?; Ok(()) })? .await??; + // Reconstruct the expected registry identity with a fresh catalog owner, + // using only remote metadata after all local intent journals were erased. + let (directory, birth) = binding_fixture::reopen_directory(&cluster).await?; + let registry = directory + .open_group_archive(&cluster.domain, &archive_pool, &birth) + .await?; + let archive = registry.archive(); // Open only physical data stores, after erasure. No bootstrap permit, local // member/disk identity, Raft metadata or replacement genesis is fabricated. let destination = destination_mode(&cluster, rooted, Arc::new(DirectFileSystem)).await?; diff --git a/crates/cellar/tests/support/cell_group_archive_publication.rs b/crates/cellar/tests/support/cell_group_archive_publication.rs index f8a99d4e4..b067237c0 100644 --- a/crates/cellar/tests/support/cell_group_archive_publication.rs +++ b/crates/cellar/tests/support/cell_group_archive_publication.rs @@ -5,6 +5,73 @@ use super::*; use std::collections::BTreeMap; +pub(super) async fn moving_upload_cut(rooted: bool) -> Result<()> { + let mut s = Scenario::new(rooted).await?; + let checkpoint = s.checkpoint(&s.upload).await?; + let (path, index) = index(s.remote.as_ref(), &checkpoint).await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let captured = source(&sources, &s.cluster)?; + let original = captured.marker().clone(); + let gate = s + .remote + .hold(cell_objects(&path, &index, "beta")[0].clone()); + let mut pending = Box::pin(s.upload.upload_cut(&s.archive_domain, &s.tails, captured)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )); + match futures_util::future::select(entered, pending.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("cut upload skipped held B"), + } + assert_eq!(sources.usage().sources, 1); + checkpoint_fixture::advance_partition( + &mut s.cluster, + "alpha", + &s.expected["alpha"], + rooted, + Some(2), + ) + .await?; + assert!( + s.cluster.voters[0] + .owner + .as_ref() + .unwrap() + .manifest() + .marker() + .position() + .index + > original.position().index + ); + s.remote.clear(); + gate.release.notify_one(); + let uploaded = pending.await?; + assert_eq!(sources.usage().sources, 0); + // The same single-source pool must admit the independent receiver. Upload + // cannot retain its source slot while waiting for physical verification. + let prepared = s + .upload + .prepare_publication( + &s.archive_domain, + &s.tails, + &uploaded, + receiver(&sources, &s.destination, &s.cluster)?, + ) + .await?; + let intent = persist_archive_intent(&s.cluster, prepared.intent()).await?; + assert_eq!(&intent, prepared.intent()); + let coverage = prepared.commit(&s.archive_domain).await?; + assert_eq!(coverage.marker(), &original); + tail_fixture::check_images(&s.cluster, coverage.recovery().manifest(), &s.expected).await?; + let hold = coverage.hold().clone(); + drop(coverage); + drop(uploaded); + assert_eq!(sources.usage().sources, 0); + s.registry.release(&s.cluster.domain, &hold).await?; + s.finish().await +} + struct Scenario { cluster: Cluster, expected: BTreeMap>, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index d225dde8a..4887be6c1 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -343,6 +343,79 @@ pub(super) async fn replicated_cells() -> Result<()> { fixture.close().await } +pub(super) async fn capture_archive_cut() -> Result<()> { + let (cluster, expected) = + checkpoint_fixture::populate_images(false, false, false, 0, 8192).await?; + let mut fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 64).await?; + let route = fixture.cluster.birth.route().clone(); + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + let cut = source.marker().clone(); + assert!(matches!( + fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await, + Err(CellGroupCheckpointError::Busy) + )); + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(4)")?; + let delta = images::Cut::new(&[], &image, 0, true, false); + let accepted = fixture.runtimes[0] + .propose( + route.history, + request(&route, history(&route, "later")?, 1, &delta)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + assert!( + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index + > cut.position().index + ); + let restored = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut restored = BTreeMap::>::new(); + source.stream(|chunk| { + if let CellGroupCheckpointPart::Materialized(cell) = chunk.part { + restored + .entry(cell.as_str().into()) + .or_default() + .extend_from_slice(chunk.bytes); + } + Ok(()) + })?; + Ok(restored) + })? + .await??; + for (cell, bytes) in expected { + if !bytes.is_empty() { + assert_eq!(restored[&cell], bytes); + } + } + assert!(!restored.contains_key("later")); + assert_eq!(pool.usage().sources, 0); + let next = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + assert!(next.marker().position().index > cut.position().index); + drop(next); + let runtime = fixture.runtimes[0].clone(); + fixture.close().await?; + assert!(matches!( + runtime.capture_checkpoint(route.history, &pool).await, + Err(CellGroupCheckpointError::Authority(RuntimeError::Closed)) + )); + Ok(()) +} + pub(super) async fn cancelled_close_joins_sync() -> Result<()> { let mut fixture = RuntimeCluster::new().await?; let route = fixture.cluster.birth.route().clone(); diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 0cb3d89cf..165b73ec6 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -581,9 +581,19 @@ authoritative owner before commit. Opening requires that exact expected creation identity; never discover and adopt a replacement mutable identity on restart. Established identity/state loss refuses admission. A lost identity-create reply with no complete state requires authority recovery; it cannot reset counters. -Controller/catalog integration must durably bind this expected identity before -archive use and make it recoverable without the voter disks. That integration is -still required before production activation or disaster recovery. +`RangeDirectory::prepare_group_archive_binding` supplies that immutable namespace +binding. Persist its canonical `encode()` bytes before committing the one-use +initializer. It checks the selected birth before binding, before initialization +and before return. The marker-1 catalog record binds the namespace root and exact +creation route/identity; its key is specific to group/incarnation. The namespace +chooses the object store and group-archive data prefix. A competing initializer +cannot replace the binding or initialize a different registry beneath it. +`open_group_archive` requires a freshly verified birth observation belonging to +the same catalog owner, reads the immutable binding, checks its namespace/route, +and opens only that expected registry. Missing binding/identity/counters refuse +without repair. An incomplete initialization requires authority recovery; replaying +serialized bytes does not restore initialization authority. These APIs do not +make controller ownership durable or grant serving/disaster-recovery authority. Persist each allocation intent before CAS and retry only that exact intent after an uncertain outcome. Its registry, next revision, nonce and complete operation @@ -610,6 +620,17 @@ entire immutable checkpoint/tail and matches its complete cut to the witness. Neither a raw locator, a private archive root nor a serialized publication intent can replace the source witness for a new publication. +`CellGroupRuntime::capture_checkpoint` asks the existing worker to capture its +owner's exact durable applied root. It performs bounded metadata capture only; +physical streaming and remote I/O remain outside the worker. Owner/cache mismatch +during concurrent persistence may refuse transiently, to retry after apply. +`upload_cut` consumes that sealed checkpoint, retains its original configuration +and permanent membership under the upload-owner budget, uploads the checkpoint, +and writes a zero-entry tail for that same cut. It never recaptures a live root +after object-store I/O. Healthy quorum can advance while a dependency is held. +The source checkpoint slot is released before independent recovery starts, so a +single-source checkpoint pool can serve upload and verification sequentially. + `prepare_publication` writes an immutable root descriptor in the pending generation and prepares one exact publication intent. Persist it before `commit`. One CAS consumes the reservation, selects the non-regressing complete root and allocates diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 9edbf03eb..d57aee382 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1748,7 +1748,9 @@ "crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs", "crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs", "crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs", - "crates/cellar/tests/support/cell_group_archive_publication.rs" + "crates/cellar/tests/support/cell_group_archive_publication.rs", + "crates/cellar/durability/raft/cell_groups/directory/archive.rs", + "crates/cellar/tests/support/cell_group_archive_binding.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 8bfd5b7029fe6397d5b34a8ec4a7217578babead Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 09:29:49 -0700 Subject: [PATCH 173/568] docs: record archive binding qualification --- .../multi-raft/cell-groups/baseline.json | 14 +-- docs/design/multi-raft/cell-groups/handoff.md | 93 ++++++++++--------- .../design/multi-raft/cell-groups/progress.md | 63 +++++++++++++ 3 files changed, 121 insertions(+), 49 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index c02abea07..0c16f31e4 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ab2a308a937f349d3482a825d4b669d042cb03c3", - "runtime_revision": "ab2a308a937f349d3482a825d4b669d042cb03c3", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open.", + "source_revision": "df0ba1f67c0253a57127a215540f08bb13135279", + "runtime_revision": "df0ba1f67c0253a57127a215540f08bb13135279", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "433421a645cba08f49e8e2332e1ed152034c6752eeb759b2e29e72890b877759", - "line": 114, + "sha256": "3dd15a523c1ba6bb326b84f3aefa415ba3ea0714913ee346f08702c444387718", + "line": 115, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "433421a645cba08f49e8e2332e1ed152034c6752eeb759b2e29e72890b877759", - "line": 183, + "sha256": "3dd15a523c1ba6bb326b84f3aefa415ba3ea0714913ee346f08702c444387718", + "line": 184, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 4d2fd0478..387a5e1c1 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,53 +15,62 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial source is `873484774c1ad5fb261257e4890291f8ad956f6e`. -CG-06 is IN_PROGRESS. Protected publication now combines a sealed committed-source -witness with independently recovered checkpoint/tail data. One conditional registry -update selects a non-regressing complete root and allocates an explicit remote hold. -Exact persisted intents reconcile unknown replies without republishing or allocating -again; retired/aborted tokens cannot resurrect. Source writes can advance while a -delayed dependency blocks publication of the earlier sealed cut. - -Keys include a registry ID and never-reused generation. Pending owners retain their -exact parent closure, and arbitrary unprotected old dependencies refuse even after -successful physical replay. Remote collection verifies every protected object before -DELETE and retains selected roots, reader holds, pending parents/generations and -newer generations. Native races prove late old-generation DELETE cannot erase an -identical new copy and actual remote reclamation preserves the current cut. Missing -counters/uncertain initial creation refuse; source-witness clones retain node quota. - -All 30 commands pass: seventeen exact archive cases, 156 execution cases, five -snapshot cases and four generation regressions (182 distinct Rust cases), all-target +The latest partial source is `df0ba1f67c0253a57127a215540f08bb13135279`. +CG-06 is IN_PROGRESS. Namespace-owned immutable binding now fixes the expected +archive registry, complete birth route and namespace data location. Persist the +one-use initializer's canonical bytes before commit. Fresh catalog owners reopen +only that expected identity; competing initialization and missing binding/counters +refuse. Interrupted initialization requires authority recovery, never repair/reset. + +The existing runtime worker can capture its exact durable applied root with the +node checkpoint pool. Physical reads and remote I/O run outside the worker. +`upload_cut` seals that checkpoint's root/configuration/permanent membership before +upload and creates a zero-entry tail without recapturing the moving live source. +Both image modes qualify held B upload plus advancing A quorum, physical verification +and publication of the exact earlier cut. A single-source pool admits upload and +later verification sequentially. Runtime capture respects quotas and joined close. + +All-three-voter disk erasure now removes local archive intent journals and drops +uploader/registry owners, then reconstructs expected identity from remote namespace +metadata through a fresh catalog owner before physical data recovery. Untrusted +checkpoint/tail locators remain externally retained. This is not fenced fresh serving. + +All 32 gate commands pass: nineteen exact archive cases, 157 execution cases, five +snapshot cases and four LTX generation regressions (185 distinct Rust cases), all-target build/Clippy, formatting/layout, dependencies/docs/preflight and 37 Python tests. -The record under -`.cellar/cell-groups-evidence/CG-06-protected-head/qualification/` -binds 334 source hashes, four executables, frozen inputs and the previous neutral- -verifier checkpoint. Independent hashes/source/exact-selection verification passes. -The preceding preflight compiler failure at `a5d506a6` is retained in `build-failure/`. -See [progress](progress.md) and the +The partial record under +`.cellar/cell-groups-evidence/CG-06-catalog-cut/qualification/` +binds 336 source hashes, four copied executables, frozen inputs, hashed outputs +and its protected-publication predecessor. Independent hashes/source/exact-selection +verification passes. See [progress](progress.md) and the [archive contract](protocols.md#protected-publication-and-remote-collection). -Next persist the exact expected registry identity in authoritative namespace/owner -state and wire the publisher into node-owned work. Implement real S3 output settlement -with receipt/version/current-route proofs, captured identity and no callback replay; -add bounded unarchived-byte/disk admission independent of Quorum. Shared dispatch -still refuses S3. Then qualify actual local shared-engine Raft file purge and fenced -unmanaged seeding with fresh group/cell incarnations. Source birth/provenance and -all possible old voter lineages, including joins after the recovery cut, must survive -voter loss. Selected-cut membership alone cannot establish the complete fence. - -Fresh registry creation is a non-deserializable one-use authority. Persist its intent -before commit; restart only opens the exact expected identity. An incomplete identity -requires authority recovery, never automatic counter recreation. Coverage is historical -data durability; it supplies no current quorum, serving or DR authorization. Controller -composition must retain verified result admission alongside derived raw roots; CG-08 -still needs that propagation and private-cache maintenance before managed integration. +Next compose controller-owned archive work with durable operation journals, bounded +retry, joined cancellation/shutdown and retained verification admission. Settle +actual S3 command/enrollment/receipt outputs using their exact cell version/result +and current routing authority, preserving captured inputs and never rerunning callbacks. +Shared dispatch still refuses S3. Add explicit unarchived-byte/disk admission so +Quorum can progress independently of S3 within its bound. Qualify physical local +shared-engine Raft file purge and fenced unmanaged recovery with fresh group/cell +incarnations. Source birth/provenance and every possible old voter lineage, including +joins after the recovery cut, must survive voter loss; selected-cut membership is +not the complete fence. The fencing-authority API plus Docker qualification remains +the accepted direction. + +Protected publication/remote collection retains the earlier source-witness, +independent replay, explicit reader/pending-parent holds, never-reused generation +and delayed-DELETE guarantees. Persist every exact intent before mutation. Unknown +outcomes reconcile only the original intent; retired/aborted tokens cannot resurrect. +Coverage is historical data durability, without current serving/quorum/DR authority. +CG-08 still owns cache/effect/result lifetime composition and private-cache maintenance, +then CG-07 cuts the public SDK/fleet over. Topology, placement, hibernation, managed +recovery, RustFS/load/fault qualification and final legacy removal remain. The actual in-memory adapter supplies native evidence. Real RustFS, Docker fencing, -fresh-quorum serving, load and public hard cutover remain later gates. The other four -original CG-06 cases and full task gate remain pending. Preserve the preceding independent- -verifier record at `14955f2d` and committed-tail record at `7a29c2a9`. +fresh-quorum serving, measured load and public hard cutover remain unqualified. The +other four original CG-06 cases and full task gate are pending. Preserve the protected- +publication source `87348477`, isolated-verifier source `14955f2d`, committed-tail +source `7a29c2a9` and their ignored evidence for audit. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 69887c736..478ad87e7 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,69 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 namespace archive identity and stable cut capture + +Source: `df0ba1f67c0253a57127a215540f08bb13135279`. CG-06 remains IN_PROGRESS. +The namespace catalog now +owns the expected archive registry identity and its object-store/data location. +A one-use initializer binds the namespace root and exact group birth route before +registry creation; persist its canonical bytes before mutation. A fresh catalog +owner can reopen only that immutable identity. Competing initializers, missing +bindings and missing counters refuse rather than replacing or repairing state. +Interrupted initialization remains fail-closed and requires authority recovery. + +`CellGroupRuntime::capture_checkpoint` performs bounded metadata capture on the +existing worker, using its owner and node checkpoint pool. Physical I/O remains +outside that worker. A held source consumes its quota, permits subsequent quorum +writes, and refunds on final release. New capture refuses after joined shutdown. +`upload_cut` retains the captured root/configuration/permanent membership before +upload starts and writes a zero-entry tail for that same cut. It never recaptures +a moving source after remote I/O. A held B upload allows A to commit while the old +cut still uploads, independently verifies and publishes in both image forms. +The same single-source checkpoint pool can admit the later independent receiver. + +The all-three-voter erasure scenario now also deletes every local archive intent +journal and drops the uploader/registry owners. It reconstructs the expected +registry identity through a fresh catalog owner from remote namespace metadata +before physically recovering the exact cut. No local intent journal or voter +metadata supplies that identity. This remains data-only recovery with externally +retained untrusted checkpoint/tail locators, without a fence or replacement voters. + +All 32 gate commands pass: nineteen exact archive cases, 157 execution cases, five +snapshot cases and four LTX generation regressions (185 distinct Rust cases across +four executables), all-target build/Clippy for Cellar/logic/LTX, formatting/layout, +dependency isolation, docs, preflight, plan validation and 37 Python tests. +The partial record at +`.cellar/cell-groups-evidence/CG-06-catalog-cut/qualification/checkpoint.json` +binds 336 source hashes, four copied executables, frozen plan inputs, hashed +outputs and the preceding protected-publication checkpoint. Independent source, +artifact and exact-selection verification passes. No original required case, +oracle or pre-existing quota was weakened. The first final Python run caught the +unrefreshed catalog source anchor: only reexports changed, moving the unchanged +NamespaceConfig/BirthProposal excerpts by one line. That failed source-bound output +and its old baseline are retained under `plan-baseline-failure/`. After reviewing +and pinning all anchors to the qualified source, docs, preflight, plan validation +and all 37 Python tests pass. Rust source stayed fixed; its passing checks were +retained. This record is not full task evidence. + +Reflection and next priorities: compose controller-owned publication with durable +journaling and exact unknown-outcome reconciliation, joined cancellation/shutdown, +verified-result lifetime ownership and fair bounded retry. Then implement actual +S3 command/enrollment/receipt output settlement against exact cell versions/results +and current routing authority, preserving captured inputs without callback replay. +Add explicit unarchived-byte/disk admission so bounded Quorum progress does not +require S3. Follow with physical local shared-engine Raft file purge and fenced +unmanaged recovery under fresh group/cell incarnations. The permanent fence inventory +must cover every possible old voting lineage, including joins after the archived cut. + +CG-08 effect/cache lifetime composition precedes CG-07 public SDK/fleet cutover; +then topology, placement, hibernation, managed recovery, RustFS/load/fault qualification +and final legacy removal remain. Shared dispatch still refuses S3 durability. The +other four original CG-06 acceptance cases and full task gate remain pending. +These native scenarios use the actual in-memory object-store adapter. No real +RustFS, Docker fence, fresh-quorum serving, measured load or full DR is claimed. +Preserve and transfer ignored evidence and its predecessors for audit. + ## CG-06 protected publication and remote object collection Source: `873484774c1ad5fb261257e4890291f8ad956f6e`. CG-06 remains IN_PROGRESS. From 0247fb8b951a9366bbabef0b0257280d358834e3 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 10:34:54 -0700 Subject: [PATCH 174/568] feat: own cell-group archive publication Journal exact reservation and publication intents before mutation. Reconcile or fence uncertain publications before allocating again, keep idle applied tails dirty, and join orphaned physical archive work before releasing node resources. Published metadata grants no S3 settlement. --- .../raft/cell_groups/archive/checkpoint.rs | 14 +- .../raft/cell_groups/archive/collection.rs | 2 +- .../raft/cell_groups/archive/execution.rs | 141 ++++++ .../cell_groups/archive/execution_tests.rs | 32 ++ .../raft/cell_groups/archive/graph.rs | 2 +- .../raft/cell_groups/archive/mod.rs | 9 +- .../raft/cell_groups/archive/ownership/mod.rs | 26 +- .../raft/cell_groups/archive/publication.rs | 11 +- .../raft/cell_groups/archive/tail.rs | 12 +- .../raft/cell_groups/archive/upload.rs | 2 +- .../raft/cell_groups/checkpoint/mod.rs | 5 +- .../cell_groups/controller/archive/job.rs | 249 ++++++++++ .../cell_groups/controller/archive/journal.rs | 169 +++++++ .../controller/archive/journal_tests.rs | 100 ++++ .../cell_groups/controller/archive/mod.rs | 456 ++++++++++++++++++ .../cell_groups/controller/checkpoints.rs | 2 +- .../raft/cell_groups/controller/mod.rs | 7 + .../raft/cell_groups/controller/requests.rs | 3 +- .../raft/cell_groups/controller/worker.rs | 13 +- .../raft/cell_groups/directory/archive.rs | 18 +- .../cellar/durability/raft/cell_groups/mod.rs | 25 +- .../raft/group_snapshot_transport/mod.rs | 1 + .../native_cell_groups_archive_controller.rs | 38 ++ .../tests/support/cell_group_archive_store.rs | 63 ++- .../support/cell_group_controller_archive.rs | 385 +++++++++++++++ .../tests/support/cell_group_runtime.rs | 2 + crates/cellar/tests/support/cell_group_sql.rs | 5 + .../multi-raft/cell-groups/protocols.md | 49 +- docs/design/multi-raft/cell-groups/tasks.json | 10 +- 29 files changed, 1789 insertions(+), 62 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/execution.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/archive/job.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs create mode 100644 crates/cellar/tests/native_cell_groups_archive_controller.rs create mode 100644 crates/cellar/tests/support/cell_group_controller_archive.rs diff --git a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs index 582c59622..24ae2890c 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/checkpoint.rs @@ -109,7 +109,7 @@ impl CellGroupArchiveUpload { /// Success confirms immutable bytes, without granting group coverage. pub async fn upload_checkpoint( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, source: CellGroupCheckpoint, ) -> Result { domain @@ -120,7 +120,7 @@ impl CellGroupArchiveUpload { async fn upload( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, mut source: CellGroupCheckpoint, ) -> Result { if source.marker().route().history != self.history { @@ -211,7 +211,7 @@ impl CellGroupArchive { /// receiver; cancellation retains its permits until physical work finishes. pub async fn verify_checkpoint( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveReference, receiver: CellGroupArchiveCheckpointReceiver, ) -> Result { @@ -223,7 +223,7 @@ impl CellGroupArchive { async fn verify( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveReference, receiver: CellGroupArchiveCheckpointReceiver, ) -> Result { @@ -234,7 +234,7 @@ impl CellGroupArchive { pub(super) async fn checkpoint_marker( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveReference, permit: Arc, ) -> Result { @@ -246,7 +246,7 @@ impl CellGroupArchive { pub(super) async fn checkpoint_index( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveReference, permit: Arc, ) -> Result { @@ -278,7 +278,7 @@ impl CellGroupArchive { pub(super) async fn verify_admitted( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveReference, mut receiver: CellGroupArchiveCheckpointReceiver, permit: Arc, diff --git a/crates/cellar/durability/raft/cell_groups/archive/collection.rs b/crates/cellar/durability/raft/cell_groups/archive/collection.rs index 49748c057..e6ab75f15 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/collection.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/collection.rs @@ -87,7 +87,7 @@ impl CellGroupArchiveRegistry { /// but cannot permit a deleted key to be rebound by a later allocation. pub async fn collect( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, owners: &CellGroupArchiveUploadPool, tails: &CellGroupArchiveTailPool, limits: CellGroupArchiveCollectionLimits, diff --git a/crates/cellar/durability/raft/cell_groups/archive/execution.rs b/crates/cellar/durability/raft/cell_groups/archive/execution.rs new file mode 100644 index 000000000..2f6a2f6ee --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/execution.rs @@ -0,0 +1,141 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Scheduling capability without ownership of execution-domain shutdown. +use crate::asyncrt::{DomainClosed, DomainToken, ExecutionDomain, TaskHandle}; +use std::future::Future; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Arc, +}; +use tokio::sync::Notify; + +mod sealed { + pub trait Sealed {} + + impl Sealed for super::ExecutionDomain {} + + impl Sealed for super::DomainToken {} + + impl Sealed for super::ControllerExecution {} +} + +#[derive(Default)] +struct Physical { + active: AtomicUsize, + finished: Notify, +} + +struct Entered(Arc); + +impl Drop for Entered { + fn drop(&mut self) { + self.0.active.fetch_sub(1, Ordering::AcqRel); + self.0.finished.notify_waiters(); + } +} + +/// One controller's physical joins, including closures whose async waiter has +/// already failed or disappeared. Does not join unrelated execution-domain work. +#[derive(Clone)] +pub(in crate::raft::cell_groups) struct ControllerExecution { + domain: DomainToken, + physical: Arc, +} + +impl ControllerExecution { + pub(in crate::raft::cell_groups) fn new(domain: DomainToken) -> Self { + Self { + domain, + physical: Arc::new(Physical::default()), + } + } + + pub(in crate::raft::cell_groups) async fn join(&self) { + loop { + let finished = self.physical.finished.notified(); + tokio::pin!(finished); + finished.as_mut().enable(); + if self.physical.active.load(Ordering::Acquire) == 0 { + return; + } + finished.await; + } + } +} + +impl CellGroupArchiveExecution for ControllerExecution { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + self.domain.run(future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + self.physical.active.fetch_add(1, Ordering::AcqRel); + let entered = Entered(self.physical.clone()); + self.domain.blocking(move || { + let _entered = entered; + operation() + }) + } +} + +/// Archive I/O runs on its owning execution domain. Sealed implementations let +/// node controllers use their captured token without acquiring shutdown rights. +pub trait CellGroupArchiveExecution: sealed::Sealed + Send + Sync { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send; + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static; +} + +impl CellGroupArchiveExecution for ExecutionDomain { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + ExecutionDomain::run(self, future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + ExecutionDomain::blocking(self, operation) + } +} + +impl CellGroupArchiveExecution for DomainToken { + fn run(&self, future: F) -> impl Future> + Send + where + F: Future + Send, + F::Output: Send, + { + DomainToken::run(self, future) + } + + fn blocking(&self, operation: F) -> Result, DomainClosed> + where + T: Send + 'static, + F: FnOnce() -> T + Send + 'static, + { + DomainToken::blocking(self, operation) + } +} + +#[cfg(test)] +#[path = "execution_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs b/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs new file mode 100644 index 000000000..910a77aca --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs @@ -0,0 +1,32 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[allow(clippy::disallowed_methods)] +async fn dropped_async_waiter_still_requires_physical_join() { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let execution = ControllerExecution::new(domain.services().domain_token().unwrap()); + let (entered, observed) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + let task = execution + .blocking(move || { + entered.send(()).unwrap(); + blocked.recv().unwrap(); + }) + .unwrap(); + observed.await.unwrap(); + drop(task); + let join = execution.join(); + tokio::pin!(join); + assert!( + tokio::time::timeout(std::time::Duration::from_millis(25), join.as_mut()) + .await + .is_err() + ); + release.send(()).unwrap(); + tokio::time::timeout(std::time::Duration::from_secs(5), join) + .await + .unwrap(); + domain.shutdown().await; +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/graph.rs b/crates/cellar/durability/raft/cell_groups/archive/graph.rs index 8b9fec6f3..04e02d7f9 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/graph.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/graph.rs @@ -98,7 +98,7 @@ impl Graph { pub async fn load( archive: &CellGroupArchive, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, descriptor: &Descriptor, limits: CellGroupArchiveTailLimits, io: Arc, diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 7c14c1559..7f28ac97f 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -8,16 +8,17 @@ pub use super::checkpoint::{ CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint, }; use super::{checkpoint::wire, CellGroupCheckpoint, CellGroupCheckpointError, GroupManifest}; -use crate::{ - asyncrt::ExecutionDomain, - raft::budget::{Budget, Permit}, -}; +use crate::raft::budget::{Budget, Permit}; use cellar_logic::cell_group::GroupHistory; use object_store::{path::Path, ObjectStore}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::{fmt, sync::Arc}; +mod execution; +pub use execution::CellGroupArchiveExecution; +pub(super) use execution::ControllerExecution; + mod checkpoint; mod data_store; pub use data_store::CellGroupArchiveStore; diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs index 2347ddab0..135ab3ae7 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -40,7 +40,10 @@ impl PreparedCellGroupArchiveCreation { &self.identity } - pub async fn commit(self, domain: &ExecutionDomain) -> Result { + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { self.archive.create_registry(domain, &self.identity).await } } @@ -202,7 +205,7 @@ impl CellGroupArchive { /// randomness; the caller durably records the complete result before create. pub async fn prepare_registry_creation( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, route: GroupRoute, ) -> Result { domain @@ -226,7 +229,7 @@ impl CellGroupArchive { async fn create_registry( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, creation: &CellGroupArchiveCreation, ) -> Result { domain @@ -274,7 +277,7 @@ impl CellGroupArchive { /// versions refuse; opening never allocates or repairs a missing record. pub async fn open_registry( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, expected: &CellGroupArchiveCreation, ) -> Result { domain @@ -363,7 +366,10 @@ impl CellGroupArchiveRegistry { Ok((state, version)) } - pub async fn prepare_upload(&self, domain: &ExecutionDomain) -> Result { + pub async fn prepare_upload( + &self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { domain .run(async { let _io = self.archive.admit()?; @@ -381,7 +387,7 @@ impl CellGroupArchiveRegistry { pub async fn reserve( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, intent: &CellGroupArchiveIntent, ) -> Result { if !matches!(intent.action, Action::Reserve { .. }) { @@ -401,7 +407,7 @@ impl CellGroupArchiveRegistry { pub async fn prepare_acquire( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, ) -> Result { domain .run(async { @@ -419,7 +425,7 @@ impl CellGroupArchiveRegistry { pub async fn acquire( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, intent: &CellGroupArchiveIntent, ) -> Result { if !matches!(intent.action, Action::Acquire { .. }) { @@ -554,7 +560,7 @@ impl CellGroupArchiveRegistry { pub async fn release( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, hold: &CellGroupArchiveHold, ) -> Result<()> { if hold.registry != self.identity.id { @@ -568,7 +574,7 @@ impl CellGroupArchiveRegistry { pub async fn abort( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, ticket: &CellGroupArchiveUploadTicket, ) -> Result<()> { if ticket.registry != self.identity.id { diff --git a/crates/cellar/durability/raft/cell_groups/archive/publication.rs b/crates/cellar/durability/raft/cell_groups/archive/publication.rs index 6d09c8237..661fc9011 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/publication.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/publication.rs @@ -119,7 +119,10 @@ impl PreparedCellGroupArchivePublication { &self.intent } - pub async fn commit(self, domain: &ExecutionDomain) -> Result { + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { domain .run(async { let _io = self.upload.admit()?; @@ -174,7 +177,7 @@ impl CellGroupArchiveUpload { /// Callbacks, engine installation and S3 command settlement do not run here. pub async fn prepare_publication( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, tails: &CellGroupArchiveTailPool, uploaded: &CellGroupArchiveUploadedTail, receiver: CellGroupArchiveCheckpointReceiver, @@ -254,7 +257,7 @@ impl CellGroupArchiveRegistry { /// intent. This never sends a publishing CAS or substitutes a newer head. pub async fn reconcile_publication( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, intent: &CellGroupArchiveIntent, tails: &CellGroupArchiveTailPool, receiver: CellGroupArchiveCheckpointReceiver, @@ -273,7 +276,7 @@ impl CellGroupArchiveRegistry { /// and recheck the persistent hold before returning historical coverage. pub async fn verify_hold( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, hold: &CellGroupArchiveHold, tails: &CellGroupArchiveTailPool, receiver: CellGroupArchiveCheckpointReceiver, diff --git a/crates/cellar/durability/raft/cell_groups/archive/tail.rs b/crates/cellar/durability/raft/cell_groups/archive/tail.rs index f1e195ea0..aa789c3ae 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/tail.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/tail.rs @@ -278,7 +278,7 @@ impl CellGroupArchiveUpload { /// released before the caller starts independent recovery verification. pub async fn upload_cut( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveTailPool, checkpoint: CellGroupCheckpoint, ) -> Result { @@ -342,7 +342,7 @@ impl CellGroupArchiveUpload { /// returns an untrusted locator; all base bytes and replay must still verify. pub async fn upload_tail( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveTailPool, checkpoint: &CellGroupArchiveReference, group: &GroupLog, @@ -358,7 +358,7 @@ impl CellGroupArchiveUpload { /// required, and the caller's publication owner must protect the base/prefix. pub async fn extend_tail( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveTailPool, previous: &CellGroupArchiveTailReference, group: &GroupLog, @@ -370,7 +370,7 @@ impl CellGroupArchiveUpload { async fn upload_from( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveTailPool, origin: Origin<'_>, group: &GroupLog, @@ -496,7 +496,7 @@ impl CellGroupArchiveUpload { impl CellGroupArchive { pub(super) async fn read_tail_index( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, reference: &CellGroupArchiveTailReference, limits: CellGroupArchiveTailLimits, io: Arc, @@ -535,7 +535,7 @@ impl CellGroupArchive { /// head/GC protection, serving permission or S3 settlement is supplied here. pub async fn verify_tail( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveTailPool, reference: &CellGroupArchiveTailReference, receiver: CellGroupArchiveCheckpointReceiver, diff --git a/crates/cellar/durability/raft/cell_groups/archive/upload.rs b/crates/cellar/durability/raft/cell_groups/archive/upload.rs index a66c7c05a..edf16bc1d 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/upload.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/upload.rs @@ -103,7 +103,7 @@ impl CellGroupArchiveRegistry { /// process must fence its predecessor before sharing/retiring that owner. pub async fn open_upload( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchiveUploadPool, ticket: &CellGroupArchiveUploadTicket, tails: &CellGroupArchiveTailPool, diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 99bb4853c..738e796a0 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -83,6 +83,9 @@ pub enum CellGroupCheckpointError { Invalid(&'static str), Limit, Busy, + /// The requested current cut raced durable apply. Retry after progress; + /// unlike malformed source metadata, this does not establish corruption. + NotReady, Manifest(super::ManifestError), Log(crate::raft::storage::LogError), Image(cellar_ltx::generation::GenerationError), @@ -296,7 +299,7 @@ impl CellGroupCheckpointPool { .as_ref() .is_some_and(|pins| pins.root() != marker.root()) { - return Err(CellGroupCheckpointError::Invalid("source durable cut")); + return Err(CellGroupCheckpointError::NotReady); } cache .membership diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs new file mode 100644 index 000000000..4bafb4c6c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -0,0 +1,249 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Each remote mutation follows its exact durable intent. Restart either finds +//! that publication or fences its pending ticket before allocating again. +use super::*; +use journal::{Journal, Pending, Record}; + +#[derive(Clone)] +pub(super) struct Context { + pub journal: Arc, + pub registry: CellGroupArchiveRegistry, +} + +pub(super) async fn open( + registration: &CellGroupArchiveRegistration, + runtime: &CellGroupRuntime, + execution: &ControllerExecution, +) -> Result { + let registry = registration + .directory + .open_group_archive(execution, ®istration.io, ®istration.birth) + .await?; + let filesystem = execution + .run(async { crate::asyncrt::fs() }) + .await + .map_err(|_| Error::Stopped)?; + let history = registration.birth.route().history; + let mut hash = sha2::Sha256::new(); + use sha2::Digest; + hash.update(b"cellar-node-group-archive-journal-1"); + for bytes in [ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + ] { + hash.update(bytes); + } + let directory = runtime + .log() + .limits() + .directory + .join("cell-group-archive-journals") + .join(hex::encode(hash.finalize())); + let engine = runtime + .log() + .storage_instance() + .map_err(|error| Error::Runtime(RuntimeError::Storage(error)))? + .bytes(); + let creation = registry.creation().clone(); + let journal = execution + .blocking(move || Journal::open(filesystem, directory, engine, creation)) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)??; + Ok(Context { + journal: Arc::new(journal), + registry, + }) +} + +async fn load(context: &Context, execution: &ControllerExecution) -> Result { + let journal = context.journal.clone(); + execution + .blocking(move || journal.load()) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)? +} + +async fn store(context: &Context, execution: &ControllerExecution, record: &Record) -> Result<()> { + let journal = context.journal.clone(); + let record = record.clone(); + execution + .blocking(move || journal.store(&record)) + .map_err(|_| Error::Stopped)? + .await + .map_err(Error::Worker)? +} + +fn receiver( + registration: &CellGroupArchiveRegistration, +) -> Result { + Ok(registration + .checkpoints + .receive_archive( + registration.store.clone(), + registration.scratch.clone(), + registration.scratch_directory.clone(), + ) + .map_err(CellGroupArchiveError::from)?) +} + +async fn select( + context: &Context, + execution: &ControllerExecution, + record: &mut Record, + coverage: CellGroupArchiveCoverage, +) -> Result { + let head = coverage.hold().clone(); + if record.retire.is_some() { + return Err(Error::Invalid("unfinished archive hold retirement")); + } + record.retire = record.head.replace(head); + record.pending = Pending::None; + store(context, execution, record).await?; + // No raw root escapes this proof's admission. The journal keeps the remote + // hold; future S3 outputs must obtain their own physical verification proof. + let marker = coverage.marker().clone(); + drop(coverage); + retire(context, execution, record).await?; + Ok(marker) +} + +async fn retire( + context: &Context, + execution: &ControllerExecution, + record: &mut Record, +) -> Result<()> { + if let Some(previous) = &record.retire { + context.registry.release(execution, previous).await?; + record.retire = None; + store(context, execution, record).await?; + } + Ok(()) +} + +pub(super) async fn drive( + context: &Context, + registration: &CellGroupArchiveRegistration, + runtime: &CellGroupRuntime, + execution: &ControllerExecution, +) -> Result { + registration + .directory + .require_current_group_birth(®istration.birth) + .await + .map_err(CellGroupArchiveBindingError::from)?; + let mut record = load(context, execution).await?; + retire(context, execution, &mut record).await?; + if let Pending::Publish { ticket, intent } = record.pending.clone() { + match context + .registry + .reconcile_publication( + execution, + &intent, + ®istration.tails, + receiver(registration)?, + ) + .await + { + Ok(coverage) => return select(context, execution, &mut record, coverage).await, + Err(CellGroupArchiveError::Expired) => (), + Err(error) => return Err(error.into()), + } + // Absence alone does not disprove a delayed publishing CAS. Abort the + // exact ticket to invalidate its old conditional version, then reconcile + // once more in case publication won the race with that abort. + context.registry.abort(execution, &ticket).await?; + match context + .registry + .reconcile_publication( + execution, + &intent, + ®istration.tails, + receiver(registration)?, + ) + .await + { + Ok(coverage) => return select(context, execution, &mut record, coverage).await, + Err(CellGroupArchiveError::Expired) => (), + Err(error) => return Err(error.into()), + } + record.pending = Pending::None; + store(context, execution, &record).await?; + } + if matches!(record.pending, Pending::None) { + let current = runtime + .status(registration.birth.route().history) + .await + .map_err(Error::Runtime)?; + if let Some(head) = &record.head { + if head.marker().position().index >= current.applied.index { + let coverage = context + .registry + .verify_hold( + execution, + head, + ®istration.tails, + receiver(registration)?, + ) + .await?; + return Ok(coverage.marker().clone()); + } + } + let intent = context.registry.prepare_upload(execution).await?; + record.pending = Pending::Reserve(intent); + store(context, execution, &record).await?; + } + if let Pending::Reserve(intent) = &record.pending { + match context.registry.reserve(execution, intent).await { + Ok(ticket) => record.pending = Pending::Upload(ticket), + Err(CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict) => { + record.pending = Pending::None; + store(context, execution, &record).await?; + return Err(CellGroupArchiveError::Expired.into()); + } + Err(error) => return Err(error.into()), + } + store(context, execution, &record).await?; + } + let Pending::Upload(ticket) = record.pending else { + return Err(Error::Invalid("archive upload stage")); + }; + let upload = context + .registry + .open_upload( + execution, + ®istration.owners, + &ticket, + ®istration.tails, + ) + .await?; + let source = runtime + .capture_checkpoint( + registration.birth.route().history, + ®istration.checkpoints, + ) + .await + .map_err(CellGroupArchiveError::from)?; + let uploaded = upload + .upload_cut(execution, ®istration.tails, source) + .await?; + let publication = upload + .prepare_publication( + execution, + ®istration.tails, + &uploaded, + receiver(registration)?, + ) + .await?; + record.pending = Pending::Publish { + ticket, + intent: publication.intent().clone(), + }; + store(context, execution, &record).await?; + let coverage = publication.commit(execution).await?; + select(context, execution, &mut record, coverage).await +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs new file mode 100644 index 000000000..ee9f6f338 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs @@ -0,0 +1,169 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One atomically replaced, fsynced record under an exclusive node-disk lock. +use super::*; +use cellar_ltx::{FileSystem, HostFileLock}; +use serde::{Deserialize, Serialize}; + +const LIMIT: usize = 32 * 1024; + +#[derive(Clone, Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct Identity { + domain: String, + engine: [u8; 16], + creation: CellGroupArchiveCreation, +} + +#[derive(Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) enum Pending { + #[default] + None, + Reserve(CellGroupArchiveIntent), + Upload(CellGroupArchiveUploadTicket), + Publish { + ticket: CellGroupArchiveUploadTicket, + intent: CellGroupArchiveIntent, + }, +} + +#[derive(Clone, Default, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Record { + pub head: Option, + pub retire: Option, + pub pending: Pending, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Envelope { + identity: Identity, + record: Record, +} + +pub(super) struct Journal { + filesystem: Arc, + directory: std::path::PathBuf, + identity: Identity, + _lock: HostFileLock, +} + +impl Journal { + pub(super) fn open( + filesystem: Arc, + directory: std::path::PathBuf, + engine: [u8; 16], + creation: CellGroupArchiveCreation, + ) -> Result { + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &directory)?; + let lock = filesystem.lock_file(&directory.join("owner.lock"))?; + let identity = Identity { + domain: "cellar-cell-group-archive-journal-1".into(), + engine, + creation, + }; + let journal = Self { + filesystem, + directory, + identity, + _lock: lock, + }; + let guard = journal.read("identity")?; + let record = journal.read("state")?; + match (guard, record) { + (None, None) => { + let bytes = encode(&journal.identity)?; + let mut file = journal + .filesystem + .create_new(&journal.directory.join("identity"))?; + file.write_all(&bytes)?; + file.sync_all()?; + journal.filesystem.sync_all(&journal.directory)?; + // Only this fresh, confirmed exclusive creation initializes a + // journal. An interrupted initializer is never reconstructed. + journal.store(&Record::default())?; + } + (Some(bytes), Some(_)) if bytes == encode(&journal.identity)? => { + journal.load()?; + } + _ => { + return Err(Error::Invalid( + "archive journal identity/state lost or changed", + )) + } + } + Ok(journal) + } + + fn read(&self, name: &str) -> Result>> { + let path = self.directory.join(name); + let metadata = match self.filesystem.symlink_metadata(&path) { + Ok(metadata) => metadata, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None), + Err(error) => return Err(error.into()), + }; + if !metadata.is_file || metadata.len == 0 || metadata.len > LIMIT as u64 { + return Err(Error::Invalid("archive journal file bounds/type")); + } + let mut file = self.filesystem.open(&path)?; + let length = file.file_len()?; + if length != metadata.len { + return Err(Error::Invalid("archive journal changed during read")); + } + Ok(Some(file.read_exact_at(0, length as usize)?)) + } + + pub(super) fn load(&self) -> Result { + if self.read("identity")?.as_deref() != Some(encode(&self.identity)?.as_slice()) { + return Err(Error::Invalid("archive journal identity changed")); + } + let bytes = self + .read("state")? + .ok_or(Error::Invalid("archive journal state lost"))?; + let envelope: Envelope = serde_json::from_slice(&bytes) + .map_err(|_| Error::Invalid("archive journal encoding"))?; + if envelope.identity != self.identity || encode(&envelope)? != bytes { + return Err(Error::Invalid("archive journal binding/canonical bytes")); + } + Ok(envelope.record) + } + + pub(super) fn store(&self, record: &Record) -> Result<()> { + let bytes = encode(&Envelope { + identity: self.identity.clone(), + record: record.clone(), + })?; + let temporary = self.directory.join("next"); + // Only the locked owner removes an unselected interrupted temporary. + match self.filesystem.remove_file(&temporary) { + Ok(()) => (), + Err(error) if error.kind() == std::io::ErrorKind::NotFound => (), + Err(error) => return Err(error.into()), + } + let mut file = self.filesystem.create_new(&temporary)?; + file.write_all(&bytes)?; + file.sync_all()?; + self.filesystem + .rename(&temporary, &self.directory.join("state"))?; + self.filesystem.sync_all(&self.directory)?; + if self.read("state")?.as_deref() != Some(bytes.as_slice()) { + return Err(Error::Invalid("archive journal write readback")); + } + Ok(()) + } +} + +fn encode(value: &impl Serialize) -> Result> { + let bytes = + serde_json::to_vec(value).map_err(|_| Error::Invalid("archive journal encoding"))?; + if bytes.len() > LIMIT { + return Err(Error::Invalid("archive journal record limit")); + } + Ok(bytes) +} + +#[cfg(test)] +#[path = "journal_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs new file mode 100644 index 000000000..1d7950a0b --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs @@ -0,0 +1,100 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::{cell_group::*, consensus::*}; + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +#[allow(clippy::disallowed_methods)] +async fn archive_journal_reopens_exact_intent_and_refuses_lost_state() -> anyhow::Result<()> { + let domain = crate::sdk::ExecutionDomain::new(tokio::runtime::Handle::current()); + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let archive = CellGroupArchivePool::new(Default::default())?.for_group( + Arc::new(object_store::memory::InMemory::new()), + object_store::path::Path::from("journal"), + history, + )?; + let creation = archive + .prepare_registry_creation( + &domain, + GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [5; 32], + schema_digest: [6; 32], + }, + ) + .await?; + let expected = creation.intent().clone(); + let registry = creation.commit(&domain).await?; + let intent = registry.prepare_upload(&domain).await?; + let disk = tempfile::tempdir()?; + let directory = disk.path().join("owner"); + let path = directory.clone(); + let identity = expected.clone(); + let journal = domain + .blocking(move || { + Journal::open( + Arc::new(cellar_ltx::DirectFileSystem), + path, + [7; 16], + identity, + ) + })? + .await??; + let saved = intent.clone(); + domain + .blocking(move || -> Result<()> { + journal.store(&Record { + pending: Pending::Reserve(saved), + ..Default::default() + })?; + Ok(()) + })? + .await??; + let path = directory.clone(); + let identity = expected.clone(); + let reopened = domain + .blocking(move || { + Journal::open( + Arc::new(cellar_ltx::DirectFileSystem), + path, + [7; 16], + identity, + ) + })? + .await??; + let recovered = domain.blocking(move || reopened.load())?.await??; + let Pending::Reserve(recovered) = recovered.pending else { + anyhow::bail!("lost reserve intent") + }; + assert_eq!(recovered, intent); + let ticket = registry.reserve(&domain, &recovered).await?; + assert_eq!(registry.reserve(&domain, &intent).await?, ticket); + let path = directory.clone(); + let identity = expected.clone(); + domain + .blocking(move || -> Result<()> { + let filesystem = cellar_ltx::DirectFileSystem; + filesystem.remove_file(&path.join("state"))?; + let before = filesystem.read(&path.join("identity"))?; + assert!(Journal::open(Arc::new(filesystem), path.clone(), [7; 16], identity).is_err()); + assert_eq!( + cellar_ltx::DirectFileSystem.read(&path.join("identity"))?, + before + ); + assert!(cellar_ltx::DirectFileSystem + .metadata(&path.join("state")) + .is_err()); + Ok(()) + })? + .await??; + registry.abort(&domain, &ticket).await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs new file mode 100644 index 000000000..f8d43db45 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -0,0 +1,456 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded, fair archive lane on the existing node controller. Published +//! markers are observations, never S3 command settlement or serving authority. +mod job; +mod journal; + +use super::calls::Delivery; +use super::*; +use crate::raft::cell_groups::{archive::ControllerExecution, *}; +use cellar_logic::{cell_group::GroupHistory, consensus::LogIndex}; +use futures_util::future::BoxFuture; +use std::collections::VecDeque; +use tokio::sync::oneshot; + +#[derive(Debug)] +pub enum CellGroupArchiveControlError { + Invalid(&'static str), + Runtime(RuntimeError), + Archive(CellGroupArchiveError), + Binding(CellGroupArchiveBindingError), + Journal(std::io::Error), + Worker(crate::asyncrt::TaskPanic), + Stopped, +} + +impl std::fmt::Display for CellGroupArchiveControlError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "group archive controller: {self:?}") + } +} + +impl std::error::Error for CellGroupArchiveControlError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::Runtime(error) => Some(error), + Self::Archive(error) => Some(error), + Self::Binding(error) => Some(error), + Self::Journal(error) => Some(error), + Self::Worker(error) => Some(error), + _ => None, + } + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: CellGroupArchiveBindingError) -> Self { + Self::Binding(error) + } +} + +impl From for CellGroupArchiveControlError { + fn from(error: std::io::Error) -> Self { + Self::Journal(error) + } +} + +type Error = CellGroupArchiveControlError; +type Result = std::result::Result; + +/// Share all admission pools across this node's registrations. The binding must +/// already be initialized in the namespace catalog. The controller owns and +/// closes the private verifier store after joining archive work. Use fresh stores +/// on restart. No application callback is needed for background publication. +#[derive(Clone)] +pub struct CellGroupArchiveRegistration { + pub directory: RangeDirectory, + pub birth: CellGroupBirth, + pub io: CellGroupArchivePool, + pub owners: CellGroupArchiveUploadPool, + pub tails: CellGroupArchiveTailPool, + pub checkpoints: CellGroupCheckpointPool, + pub store: CellGroupArchiveStore, + pub scratch: cellar_ltx::stream_snapshot::ScratchPool, + pub scratch_directory: std::path::PathBuf, +} + +#[derive(Clone, Default)] +pub struct CellGroupArchiveStatus { + /// Last complete cut verified/published by this owner. Advisory historical + /// metadata only; a consumer still needs a retained physical coverage proof. + pub published: Option, + pub attempts: u64, + pub error: Option>, + pub stopped: bool, +} + +pub struct CellGroupArchiveCall { + reply: oneshot::Receiver>, +} + +impl CellGroupArchiveCall { + /// Caller cancellation does not cancel the admitted archive owner. + pub async fn response(self) -> std::result::Result<(), Arc> { + self.reply + .await + .map_err(|_| Arc::new(Error::Stopped))? + .result + } +} + +pub(super) struct Request { + registration: CellGroupArchiveRegistration, + reply: RegistrationReply, +} + +struct RegistrationReply { + sender: oneshot::Sender>, + permit: Arc, +} + +impl RegistrationReply { + fn send(self, result: std::result::Result<(), Arc>) { + let _ = self.sender.send(Delivery { + result, + _permit: self.permit, + }); + } +} + +impl CellGroupController { + /// Install background archival for an already attached group. Admission is + /// node-bounded and survives response cancellation. This never enables S3 + /// acknowledgements or authorizes local log reclamation. + pub fn archive( + &self, + registration: CellGroupArchiveRegistration, + ) -> Result { + let shared = &self.handle.0; + if shared.closing.load(Ordering::Acquire) { + return Err(Error::Stopped); + } + let group = registration.birth.route().history; + if !shared.groups.contains(&group).map_err(Error::Runtime)? { + return Err(Error::Runtime(RuntimeError::NotFound)); + } + if registration.store.route() != registration.birth.route() + || !registration.scratch_directory.is_absolute() + || registration.scratch_directory.as_os_str().len() > 4096 + { + return Err(Error::Invalid("archive registration binding/path")); + } + let permit = shared + .commands + .acquire(8192) + .ok_or(Error::Runtime(RuntimeError::Overloaded))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Archive(Box::new(Request { + registration, + reply: RegistrationReply { + sender: reply, + permit: Arc::new(permit), + }, + }))) + .map_err(|error| match error { + tokio::sync::mpsc::error::TrySendError::Closed(_) => Error::Stopped, + tokio::sync::mpsc::error::TrySendError::Full(_) => { + Error::Runtime(RuntimeError::Overloaded) + } + })?; + Ok(CellGroupArchiveCall { reply: receiver }) + } + + pub fn archive_status( + &self, + group: GroupHistory, + ) -> std::result::Result, RuntimeError> { + Ok(self + .handle + .0 + .archives + .lock() + .map_err(|_| RuntimeError::Closed)? + .get(&group) + .cloned()) + } +} + +struct Entry { + registration: CellGroupArchiveRegistration, + context: Option, + reply: Option, + queued: bool, + latest_applied: Option, + dirty: bool, + fatal: bool, + retry_at: u64, +} + +pub(super) struct Completed { + group: GroupHistory, + context: Option, + result: Result, + observation: std::result::Result, +} + +pub(super) struct Manager { + entries: BTreeMap, + queue: VecDeque, + job: Option>, + execution: ControllerExecution, +} + +impl Manager { + pub fn new(domain: crate::asyncrt::DomainToken) -> Self { + Self { + entries: BTreeMap::new(), + queue: VecDeque::new(), + job: None, + execution: ControllerExecution::new(domain), + } + } + + pub fn request(&mut self, request: Request, shared: &Shared) { + let group = request.registration.birth.route().history; + if self.entries.contains_key(&group) || self.entries.len() >= shared.limits.max_groups { + request.reply.send(Err(Arc::new(Error::Invalid( + "duplicate/full archive roster", + )))); + return; + } + shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .insert(group, CellGroupArchiveStatus::default()); + self.entries.insert( + group, + Entry { + registration: request.registration, + context: None, + reply: Some(request.reply), + queued: true, + latest_applied: None, + dirty: true, + fatal: false, + retry_at: 0, + }, + ); + self.queue.push_back(group); + } + + pub fn applied(&mut self, group: GroupHistory, index: LogIndex) { + if let Some(entry) = self.entries.get_mut(&group) { + entry.latest_applied = Some( + entry + .latest_applied + .map_or(index, |previous| previous.max(index)), + ); + if !entry.fatal { + entry.dirty = true; + if !entry.queued { + entry.queued = true; + self.queue.push_back(group); + } + } + } + } + + pub fn running(&self) -> bool { + self.job.is_some() + } + + pub async fn next(&mut self) -> Completed { + self.job.as_mut().expect("active archive job").await + } + + pub fn launch(&mut self, runtime: &CellGroupRuntime, shared: &Shared, now: u64) { + if self.running() { + return; + } + for _ in 0..self.queue.len() { + let group = self.queue.pop_front().expect("archive roster queue"); + let entry = self.entries.get_mut(&group).expect("archive roster entry"); + entry.queued = false; + if !entry.dirty || entry.fatal { + continue; + } + if entry.retry_at > now { + entry.queued = true; + self.queue.push_back(group); + continue; + } + entry.dirty = false; + if let Some(status) = shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .get_mut(&group) + { + status.attempts = status.attempts.saturating_add(1); + } + let registration = entry.registration.clone(); + let mut context = entry.context.clone(); + let execution = self.execution.clone(); + let runtime = runtime.clone(); + self.job = Some(Box::pin(async move { + let result = std::panic::AssertUnwindSafe(async { + if context.is_none() { + context = Some(job::open(®istration, &runtime, &execution).await?); + } + job::drive( + context.as_ref().expect("opened archive context"), + ®istration, + &runtime, + &execution, + ) + .await + }) + .catch_unwind() + .await + .unwrap_or(Err(Error::Invalid("archive job panicked"))); + // Reconciliation may have recovered an older head while all + // newer apply events were already consumed before this job. + // Observe the owner after the attempt so an idle final tail + // cannot disappear when the dirty flag was cleared at launch. + let observation = runtime + .status(group) + .await + .map(|status| status.applied.index); + Completed { + group, + context, + result, + observation, + } + })); + break; + } + } + + pub fn complete(&mut self, completed: Completed, shared: &Shared, now: u64, closing: bool) { + self.job = None; + let entry = self + .entries + .get_mut(&completed.group) + .expect("completed archive owner"); + entry.context = completed.context; + let mut statuses = shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()); + let status = statuses.get_mut(&completed.group).expect("archive status"); + match completed.result { + Ok(marker) => { + let covered = marker.position().index; + status.published = Some(marker); + status.error = None; + match completed.observation { + Ok(applied) => { + let latest = entry + .latest_applied + .map_or(applied, |seen| seen.max(applied)); + entry.dirty = latest > covered; + } + Err(error) => { + let error = Error::Runtime(error); + entry.fatal = !retryable(&error); + status.stopped = entry.fatal; + entry.dirty = true; + entry.retry_at = now.saturating_add(shared.limits.retry_interval_ms); + status.error = Some(Arc::new(error)); + } + } + } + Err(error) => { + entry.fatal = !retryable(&error); + status.stopped = entry.fatal; + entry.dirty = true; + entry.retry_at = now.saturating_add(shared.limits.retry_interval_ms); + status.error = Some(Arc::new(error)); + } + } + if entry.dirty && !entry.fatal && !entry.queued { + entry.queued = true; + self.queue.push_back(completed.group); + } + if closing || entry.context.is_some() || entry.fatal { + if let Some(reply) = entry.reply.take() { + let result = if closing { + Err(Arc::new(Error::Stopped)) + } else if entry.context.is_some() { + Ok(()) + } else { + Err(status.error.clone().expect("failed archive initialization")) + }; + reply.send(result); + } + } + } + + pub async fn finish(&mut self, shared: &Shared) { + if let Some(job) = self.job.take() { + let completed = job.await; + self.complete(completed, shared, 0, true); + } + self.execution.join().await; + for entry in self.entries.values_mut() { + entry.registration.store.close(); + if let Some(reply) = entry.reply.take() { + reply.send(Err(Arc::new(Error::Stopped))); + } + } + self.entries.clear(); + for status in shared + .archives + .lock() + .unwrap_or_else(|error| error.into_inner()) + .values_mut() + { + status.stopped = true; + } + } +} + +fn archive_retryable(error: &CellGroupArchiveError) -> bool { + matches!( + error, + CellGroupArchiveError::Busy + | CellGroupArchiveError::Timeout + | CellGroupArchiveError::Conflict + | CellGroupArchiveError::Expired + | CellGroupArchiveError::Unconfirmed + | CellGroupArchiveError::Storage(_) + | CellGroupArchiveError::Checkpoint( + CellGroupCheckpointError::NotReady + | CellGroupCheckpointError::Busy + | CellGroupCheckpointError::Authority( + RuntimeError::NotReady | RuntimeError::Overloaded + ) + ) + ) +} + +fn retryable(error: &Error) -> bool { + match error { + Error::Archive(error) | Error::Binding(CellGroupArchiveBindingError::Archive(error)) => { + archive_retryable(error) + } + Error::Binding(CellGroupArchiveBindingError::Catalog(error)) => matches!( + error, + crate::raft::catalog::CatalogError::Storage(_) + | crate::raft::catalog::CatalogError::Unconfirmed + ), + Error::Runtime(RuntimeError::Overloaded | RuntimeError::NotReady) => true, + _ => false, + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs index 67e59b3b5..5a6a0796c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs @@ -243,7 +243,7 @@ impl Manager { } pub(super) fn retryable(error: &CellGroupControllerError) -> bool { - matches!(error, CellGroupControllerError::Checkpoint(error) if matches!(error.as_ref(), CellGroupCheckpointError::Busy)) + matches!(error, CellGroupControllerError::Checkpoint(error) if matches!(error.as_ref(), CellGroupCheckpointError::Busy | CellGroupCheckpointError::NotReady)) || matches!( error, CellGroupControllerError::Runtime(RuntimeError::Overloaded) diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 7d7e9e515..94969f927 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -3,7 +3,12 @@ //! Owns registered groups' consensus events, transport dispatch and physical apply. //! Admitted command ownership survives response cancellation and expiry. +mod archive; mod calls; +pub use archive::{ + CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, + CellGroupArchiveStatus, +}; mod checkpoints; pub use checkpoints::CellGroupCheckpointPolicy; mod commands; @@ -111,6 +116,7 @@ struct Shared { commands: Arc, provisioning: Mutex>>, requests: tokio::sync::mpsc::Sender, + archives: Mutex>, active_apply: AtomicUsize, pending_apply: AtomicUsize, applied: AtomicU64, @@ -235,6 +241,7 @@ impl CellGroupController { commands: commands::budget(limits), provisioning: Mutex::new(None), requests, + archives: Mutex::new(BTreeMap::new()), active_apply: AtomicUsize::new(0), pending_apply: AtomicUsize::new(0), applied: AtomicU64::new(0), diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index ea8fb60b0..7a72767b6 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -5,6 +5,7 @@ use super::*; pub(super) enum Request { + Archive(Box), Registration(Box), Provision(Box), Command(Box), @@ -49,7 +50,7 @@ pub(super) async fn drive( progress: tokio::sync::watch::Receiver<()>, ) -> Completed { match request { - Request::Registration(_) | Request::Provision(_) => { + Request::Registration(_) | Request::Provision(_) | Request::Archive(_) => { unreachable!("registration has its own physical join owner") } Request::Command(request) => Completed::Command(Box::new( diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 5efe4d70c..794d4bb88 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -46,6 +46,7 @@ pub(super) struct Worker { routes: BTreeMap, progress: tokio::sync::watch::Sender<()>, checkpoints: checkpoints::Manager, + archives: archive::Manager, _checkpoint_routes: Vec, } @@ -69,6 +70,7 @@ impl Worker { ); } } + let archives = archive::Manager::new(shared.domain.clone()); Ok(Self { runtime, events, @@ -91,6 +93,7 @@ impl Worker { routes: BTreeMap::new(), progress: tokio::sync::watch::channel(()).0, checkpoints: checkpoints::Manager::default(), + archives, _checkpoint_routes: checkpoint_routes, }) } @@ -181,6 +184,7 @@ impl Worker { .mono_ms() .map_err(|_| CellGroupControllerError::Closed)?; self.launch(now)?; + self.archives.launch(&self.runtime, &self.shared, now); let timer = domain .sleep_until(wake_at) .map_err(|_| CellGroupControllerError::Closed)?; @@ -194,6 +198,7 @@ impl Worker { }, Some(request) = self.requests.recv() => { match request { + requests::Request::Archive(request) => self.archives.request(*request, &self.shared), requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared), requests::Request::Registration(request) => { let group = request.registration.applier.history(); @@ -264,6 +269,10 @@ impl Worker { } } Some(()) = self.hints.next(), if !self.hints.is_empty() => (), + completed = self.archives.next(), if self.archives.running() => { + self.archives.complete(completed, &self.shared, now, false); + self.progress.send_replace(()); + } event = self.events.recv() => { let event = event.ok_or(CellGroupControllerError::Closed)?; let group = event.group(); @@ -319,7 +328,8 @@ impl Worker { let work = event.retain_apply().ok_or(CellGroupControllerError::Invalid("apply ticket"))?; self.pending.push_back(Pending { group, work: Work::Prepare(Box::new(work)), retry_at: now }); } - CellGroupEffect::Applied { .. } | CellGroupEffect::CheckpointInstalled { .. } => { + CellGroupEffect::Applied { manifest, .. } | CellGroupEffect::CheckpointInstalled { manifest } => { + self.archives.applied(group, manifest.position().index); // Only this post-sync event advances the published count. self.shared.applied.fetch_add(1, Ordering::Relaxed); self.progress.send_replace(()); @@ -381,6 +391,7 @@ impl Worker { } self.provisioning.join().await; self.checkpoints.join().await; + self.archives.finish(&self.shared).await; self.transport.close().await; let sql = self .sql diff --git a/crates/cellar/durability/raft/cell_groups/directory/archive.rs b/crates/cellar/durability/raft/cell_groups/directory/archive.rs index af186f949..801c68cda 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/archive.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/archive.rs @@ -3,12 +3,9 @@ //! Namespace-owned archive identity. Recovery reads the expected identity from //! immutable catalog metadata, never by trusting a mutable registry's identity. use super::*; -use crate::{ - raft::cell_groups::{ - CellGroupArchive, CellGroupArchiveCreation, CellGroupArchiveError, CellGroupArchivePool, - CellGroupArchiveRegistry, PreparedCellGroupArchiveCreation, - }, - sdk::ExecutionDomain, +use crate::raft::cell_groups::{ + CellGroupArchive, CellGroupArchiveCreation, CellGroupArchiveError, CellGroupArchiveExecution, + CellGroupArchivePool, CellGroupArchiveRegistry, PreparedCellGroupArchiveCreation, }; use serde::{Deserialize, Serialize}; @@ -70,7 +67,10 @@ impl PreparedCellGroupArchiveBinding { &self.bytes } - pub async fn commit(self, domain: &ExecutionDomain) -> Result { + pub async fn commit( + self, + domain: &impl CellGroupArchiveExecution, + ) -> Result { domain .run(async { self.directory @@ -137,7 +137,7 @@ impl RangeDirectory { /// can substitute another group's archive. This grants no serving authority. pub async fn prepare_group_archive_binding( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchivePool, birth: &CellGroupBirth, ) -> Result { @@ -172,7 +172,7 @@ impl RangeDirectory { /// This does not recreate metadata, fence old owners or authorize new voters. pub async fn open_group_archive( &self, - domain: &ExecutionDomain, + domain: &impl CellGroupArchiveExecution, pool: &CellGroupArchivePool, birth: &CellGroupBirth, ) -> Result { diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index f8dd3700c..477a4d83e 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -19,14 +19,14 @@ mod archive; pub use archive::{ CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCollectionLimits, CellGroupArchiveCollectionReport, CellGroupArchiveCoverage, CellGroupArchiveCreation, - CellGroupArchiveError, CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchiveLimits, - CellGroupArchivePool, CellGroupArchiveReference, CellGroupArchiveRegistry, - CellGroupArchiveStore, CellGroupArchiveTailLimits, CellGroupArchiveTailPool, - CellGroupArchiveTailReference, CellGroupArchiveUpload, CellGroupArchiveUploadLimits, - CellGroupArchiveUploadPool, CellGroupArchiveUploadTicket, CellGroupArchiveUploadedTail, - CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, PreparedCellGroupArchiveCreation, - PreparedCellGroupArchivePublication, VerifiedCellGroupArchiveCheckpoint, - VerifiedCellGroupArchiveRecovery, + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchiveHold, CellGroupArchiveIntent, + CellGroupArchiveLimits, CellGroupArchivePool, CellGroupArchiveReference, + CellGroupArchiveRegistry, CellGroupArchiveStore, CellGroupArchiveTailLimits, + CellGroupArchiveTailPool, CellGroupArchiveTailReference, CellGroupArchiveUpload, + CellGroupArchiveUploadLimits, CellGroupArchiveUploadPool, CellGroupArchiveUploadTicket, + CellGroupArchiveUploadedTail, CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, + PreparedCellGroupArchiveCreation, PreparedCellGroupArchivePublication, + VerifiedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveRecovery, }; pub(crate) mod checkpoint; pub use checkpoint::{ @@ -38,10 +38,11 @@ pub use checkpoint::{ mod controller; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, - CellGroupCheckpointPolicy, CellGroupController, CellGroupControllerError, - CellGroupControllerLimits, CellGroupControllerUsage, CellGroupProvisionCall, - CellGroupProvisioning, CellGroupRegistration, CellGroupRegistrationCall, - CellGroupRegistrationError, + CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, + CellGroupArchiveStatus, CellGroupCheckpointPolicy, CellGroupController, + CellGroupControllerError, CellGroupControllerLimits, CellGroupControllerUsage, + CellGroupProvisionCall, CellGroupProvisioning, CellGroupRegistration, + CellGroupRegistrationCall, CellGroupRegistrationError, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs index baa223aee..0c486fb3e 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -163,6 +163,7 @@ impl State { fn checkpoint_error(error: CellGroupCheckpointError) -> TransportError { match error { CellGroupCheckpointError::Busy => TransportError::Overloaded, + CellGroupCheckpointError::NotReady => TransportError::Runtime(RuntimeError::NotReady), CellGroupCheckpointError::Authority(error) => TransportError::Runtime(error), CellGroupCheckpointError::Limit => TransportError::Limit, other => TransportError::Invalid(other.to_string()), diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs new file mode 100644 index 000000000..d08e70f35 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -0,0 +1,38 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Controller-owned archive publication; no fixture event pumping after startup. +#![allow(clippy::disallowed_methods)] + +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +#[allow(dead_code)] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +#[allow(dead_code)] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +#[allow(dead_code)] +mod images; +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_archive_publishes_stable_cuts_while_quorum_advances() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_publish(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_archive_reconciles_exact_intent_and_joins_close() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_reconcile(rooted)).await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index 161e3a3d1..ad845c39a 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -25,21 +25,49 @@ pub struct Store { pub lose_delete_response: AtomicBool, held: Mutex)>>, held_delete: Mutex)>>, + publication: Mutex>>, + pub publication_cuts: Mutex>, + pub outage: AtomicBool, + pub lose_publication_and_reads: AtomicBool, } impl Store { pub fn new() -> Arc { + Self::around(Arc::new(memory::InMemory::new())) + } + + pub fn around(inner: Arc) -> Arc { Arc::new(Self { - inner: Arc::new(memory::InMemory::new()), + inner, puts: AtomicUsize::new(0), lose_put_response: AtomicBool::new(false), deletes: AtomicUsize::new(0), lose_delete_response: AtomicBool::new(false), held: Mutex::new(None), held_delete: Mutex::new(None), + publication: Mutex::new(None), + publication_cuts: Mutex::new(Vec::new()), + outage: AtomicBool::new(false), + lose_publication_and_reads: AtomicBool::new(false), }) } + pub fn hold_publication(&self) -> Arc { + let gate = Arc::new(Gate::default()); + *self.publication.lock().unwrap() = Some(gate.clone()); + gate + } + + fn available(&self) -> Result<()> { + if self.outage.load(Ordering::Acquire) { + return Err(Error::Generic { + store: "archive outage", + source: "injected".into(), + }); + } + Ok(()) + } + pub fn hold(&self, path: Path) -> Arc { self.hold_after(path, 0) } @@ -75,8 +103,40 @@ impl ObjectStore for Store { payload: PutPayload, opts: PutOptions, ) -> Result { + self.available()?; self.puts.fetch_add(1, Ordering::AcqRel); + let publication = if path.as_ref().ends_with("/ownership/registry") { + let bytes: Vec = payload + .iter() + .flat_map(|bytes| bytes.iter().copied()) + .collect(); + let value: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + value["selected"]["marker"]["index"].as_u64() + } else { + None + }; + if publication.is_some() { + let gate = self.publication.lock().unwrap().take(); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } + } let result = self.inner.put_opts(path, payload, opts).await?; + if let Some(index) = publication { + self.publication_cuts.lock().unwrap().push(index); + } + if publication.is_some() + && self + .lose_publication_and_reads + .swap(false, Ordering::AcqRel) + { + self.outage.store(true, Ordering::Release); + return Err(Error::Generic { + store: "lost archive publication", + source: "injected".into(), + }); + } if self.lose_put_response.swap(false, Ordering::AcqRel) { return Err(Error::Generic { store: "lost PUT response", @@ -95,6 +155,7 @@ impl ObjectStore for Store { } async fn get_opts(&self, path: &Path, opts: GetOptions) -> Result { + self.available()?; let gate = self .held .lock() diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs new file mode 100644 index 000000000..2e2c852ef --- /dev/null +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -0,0 +1,385 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real three-voter controllers, conditional object-store I/O and private replay. +use super::*; +use futures_util::TryStreamExt; +use object_store::ObjectStore; +use std::sync::atomic::Ordering; + +#[path = "cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; + +async fn verifier( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + name: &str, +) -> Result { + let route = route.clone(); + let path = fixture.cluster._root.path().join(name); + fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let filesystem = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &path.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let services = cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|b| b.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + path.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: path.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open( + path.join("images"), + filesystem, + Default::default(), + )?, + services, + } + }; + Ok(CellGroupArchiveStore::new( + route, + pages, + images, + Default::default(), + Default::default(), + 100, + )?) + })? + .await? +} + +async fn registration( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, +) -> Result<(CellGroupArchiveRegistration, Arc)> { + let io = CellGroupArchivePool::new(Default::default())?; + let binding = fixture + .cluster + .directory + .prepare_group_archive_binding(&fixture.cluster.domain, &io, &fixture.cluster.birth) + .await?; + let bytes = binding.encode().to_vec(); + let path = fixture.cluster._root.path().join("archive-binding-intent"); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&bytes)?; + file.sync_all()?; + std::fs::File::open(path.parent().unwrap())?.sync_all()?; + Ok(()) + })? + .await??; + binding.commit(&fixture.cluster.domain).await?; + let remote = archive_store::Store::around(fixture.cluster.store.clone()); + let catalog = Catalog::new(remote.clone(), Default::default())?; + let namespace = fixture + .cluster + .domain + .run(catalog.existing_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + let directory = RangeDirectory::new(namespace, Default::default())?; + let birth = fixture + .cluster + .domain + .run(directory.materialize_selected_group(route)) + .await??; + let store = verifier(fixture, route, rooted, "controller-archive").await?; + Ok(( + CellGroupArchiveRegistration { + directory, + birth, + io, + owners: CellGroupArchiveUploadPool::new(Default::default())?, + tails: CellGroupArchiveTailPool::new(Default::default())?, + checkpoints: CellGroupCheckpointPool::new(checkpoint_fixture::limits())?, + store, + scratch: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + scratch_directory: fixture.cluster._root.path().into(), + }, + remote, + )) +} + +async fn published( + controller: &CellGroupController, + group: GroupHistory, + index: LogIndex, +) -> Result<()> { + controller::wait_for(async || { + let status = controller + .archive_status(group)? + .context("archive status")?; + if let Some(error) = &status.error { + if status.stopped { + anyhow::bail!("archive stopped: {error}"); + } + } + Ok(status + .published + .is_some_and(|marker| marker.position().index >= index)) + }) + .await +} + +async fn add( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + ordinal: u8, +) -> Result { + let image = images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({ordinal})"), + )?; + let cell = history(route, &format!("later-{ordinal}"))?; + let proposal = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(route, &cell, &image, rooted)?, + ) + .await?; + controller::applied(fixture, route.history, proposal.position.index).await?; + Ok(proposal.position.index) +} + +async fn setup(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + Ok((fixture, route)) +} + +async fn audit( + fixture: &RuntimeCluster, + config: &CellGroupArchiveRegistration, + rooted: bool, + last: u8, +) -> Result<()> { + let route = config.birth.route(); + let registry = config + .directory + .open_group_archive(&fixture.cluster.domain, &config.io, &config.birth) + .await?; + let directory = fixture + .cluster + ._root + .path() + .join("node-1/log/cell-group-archive-journals"); + let envelope = fixture + .cluster + .domain + .blocking(move || -> Result { + let entry = std::fs::read_dir(directory)? + .next() + .context("journal directory")??; + Ok(serde_json::from_slice(&std::fs::read( + entry.path().join("state"), + )?)?) + })? + .await??; + let hold: CellGroupArchiveHold = serde_json::from_value(envelope["record"]["head"].clone())?; + assert!(envelope["record"]["retire"].is_null()); + assert_eq!(envelope["record"]["pending"], "None"); + let destination = verifier(fixture, route, rooted, "controller-archive-audit").await?; + let coverage = registry + .verify_hold( + &fixture.cluster.domain, + &hold, + &config.tails, + config.checkpoints.receive_archive( + destination, + config.scratch.clone(), + config.scratch_directory.clone(), + )?, + ) + .await?; + let root = coverage.recovery().manifest(); + assert_eq!(root.pinned_images(), usize::from(last) + 2); + let names = ["registered-a".to_owned(), "registered-b".to_owned()] + .into_iter() + .chain((1..=last).map(|n| format!("later-{n}"))); + for name in names { + let expected = if let Some(n) = name.strip_prefix("later-") { + images::image( + None, + &format!("CREATE TABLE value(n); INSERT INTO value VALUES({n})"), + )? + } else { + images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")? + }; + let (_, pin) = root + .pin_cell(&CellId::new(&name)?) + .context("archived cell")?; + let restored = fixture + .cluster + .domain + .blocking(move || -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await??; + assert_eq!(restored, expected); + } + Ok(()) +} + +pub(crate) async fn archive_controller_publish(rooted: bool) -> Result<()> { + let (mut fixture, route) = setup(rooted).await?; + let (config, remote) = registration(&fixture, &route, rooted).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + let gate = remote.hold_publication(); + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let call = controllers[0].archive(config.clone())?; + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + // A held archive publication does not stop real quorum/apply progress. + let later = add(&fixture, &route, rooted, 1).await?; + assert!(later > initial); + assert!(controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .is_none()); + gate.release.notify_one(); + call.response().await?; + published(&controllers[0], route.history, later).await?; + assert!(remote + .publication_cuts + .lock() + .unwrap() + .contains(&initial.get())); + for controller in &controllers { + controller.close().await?; + } + assert!( + controllers[0] + .archive_status(route.history)? + .unwrap() + .stopped + ); + audit(&fixture, &config, rooted, 1).await?; + drop(config); + fixture.close().await +} + +pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { + let (mut fixture, route) = setup(rooted).await?; + let (config, remote) = registration(&fixture, &route, rooted).await?; + let initial = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + remote + .lose_publication_and_reads + .store(true, Ordering::Release); + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + // Cancelling a registration response does not cancel its admitted owner. + drop(controllers[0].archive(config.clone())?); + controller::wait_for(async || Ok(remote.outage.load(Ordering::Acquire))).await?; + let mut latest = initial; + for n in 1..=4 { + latest = add(&fixture, &route, rooted, n).await?; + } + let status = controllers[0].archive_status(route.history)?.unwrap(); + assert!(status.published.is_none()); + assert!(status.error.is_some()); + // Reconciliation must recover the exact earlier publication without a new + // allocation/CAS. Hold the first subsequent registry write to inspect that. + let gate = remote.hold_publication(); + remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + assert_eq!( + controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .unwrap() + .position() + .index, + initial + ); + let mut closing = Box::pin(controllers[0].close()); + assert!( + tokio::time::timeout(Duration::from_millis(50), closing.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + closing.await?; + // Closing joins the already admitted archive job, including its new cut. + let selected = controllers[0].archive_status(route.history)?.unwrap(); + assert!(selected.published.unwrap().position().index >= latest); + for controller in controllers.iter().skip(1) { + controller.close().await?; + } + audit(&fixture, &config, rooted, 4).await?; + let objects = remote.list(None).try_collect::>().await?; + let key = objects + .iter() + .find(|o| o.location.as_ref().ends_with("/ownership/registry")) + .context("registry")?; + let bytes = remote.get(&key.location).await?.bytes().await?; + let registry: serde_json::Value = serde_json::from_slice(&bytes)?; + assert_eq!(registry["holds"].as_array().context("holds")?.len(), 1); + drop(config); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 4887be6c1..6c68c2c7a 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -803,6 +803,8 @@ pub(super) async fn attachment_reserves_before_owner() -> Result<()> { #[path = "cell_group_sql.rs"] mod sql; +#[allow(unused_imports)] +pub(super) use sql::{archive_controller_publish, archive_controller_reconcile}; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index f014233b6..c1afae55d 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -526,8 +526,13 @@ pub(crate) use dispatch::routed as routed_queries; mod command_dispatch; pub(crate) use command_dispatch::commands as routed_commands; +#[path = "cell_group_controller_archive.rs"] +#[allow(dead_code, unused_imports)] +mod archive_controller; #[path = "cell_group_controller.rs"] mod controller; +#[allow(unused_imports)] +pub(crate) use archive_controller::{archive_controller_publish, archive_controller_reconcile}; #[path = "cell_group_controller_commands.rs"] mod controller_commands; pub(crate) use controller::{ diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 165b73ec6..642cfe9dd 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -623,7 +623,7 @@ can replace the source witness for a new publication. `CellGroupRuntime::capture_checkpoint` asks the existing worker to capture its owner's exact durable applied root. It performs bounded metadata capture only; physical streaming and remote I/O remain outside the worker. Owner/cache mismatch -during concurrent persistence may refuse transiently, to retry after apply. +during concurrent persistence returns typed `NotReady`, to retry after apply. `upload_cut` consumes that sealed checkpoint, retains its original configuration and permanent membership under the upload-owner budget, uploads the checkpoint, and writes a zero-entry tail for that same cut. It never recaptures a live root @@ -671,6 +671,53 @@ fencing inventory must cover every possible old voting lineage, including joins after the selected archived cut. Real RustFS, Docker fencing and measured-load qualification remain later gates. +### Controller-owned archive publication + +`CellGroupController::archive` installs archival for an already attached group. +The namespace binding must already be initialized. Registration checks the exact +birth/store route, then opens only the namespace's expected registry. One fair +archive lane on the existing node worker rotates a deduplicated dirty-group +queue, independently of SQL apply capacity. Clean groups require no roster scan. Apply/checkpoint-install events mark a group dirty. Transient +I/O/admission failures use the existing node timer; malformed metadata and lost +journal state refuse archival. Completions compare the covered cut with both the +owner applied index and consumed apply events; recovery of an older publication +cannot lose an idle final tail. This does not enable S3 command acknowledgements +or authorize local log reclamation. + +Each group has an exclusively locked node journal bound to the engine creation +identity and expected archive creation. Its immutable identity guard precedes its +initial state; missing identity/state never reconstructs an interrupted journal. +Updates write an exclusive temporary, sync it, rename, sync the directory and +read back the exact bytes. Every attempt reloads the selected state, so an error +after rename cannot roll back to stale in-memory intent. Record size is bounded +to 32 KiB; the roster and single active lane bound retained control metadata. + +| Selected journal phase | Permitted next action | +| --- | --- | +| None | Verify an already sufficient held cut, or prepare and persist one reservation intent. | +| Reserve | Reconcile/reserve that exact intent, then persist its ticket before any object upload. Definite expiry permits a newly persisted intent. | +| Upload | Capture and upload a sealed complete cut using that pending generation; physically verify it and persist the exact publication intent before CAS. | +| Publish | Reconcile only the exact existing publication. If absent, abort its pending ticket to fence a delayed CAS, then reconcile again before allocating anything new. | + +Selecting a new head first persists its explicit hold and the predecessor to +retire. Only afterward may the old hold be released; both release and the journal +clear are idempotent. Source callbacks never run in this workflow. Physical +coverage/root admission stays scoped to the job, rather than retaining a replay +workspace for every registered group. `archive_status` exposes historical metadata +and errors only; it is neither a physical coverage capability nor an S3 receipt. + +The sealed archive execution capability accepts either the full execution domain +or a controller's captured token, without acquiring shutdown rights. A controller +tracks its admitted blocking closures independently of their async waiters. +Shutdown joins logical archive work, drains those physical closures, closes private +verifier stores and releases journal locks before closing runtime/SQL ownership. +Fresh verifier stores are required on restart. No unrelated domain work is joined +through this archive-specific drain. + +Actual S3 command/enrollment/receipt settlement, explicit unarchived-byte/disk +admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, +RustFS/load and process-kill qualification remain required gates. + ## Split and merge Use the same transition state machine for one parent to multiple child ranges diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index d57aee382..1e2bb6f8c 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1750,7 +1750,15 @@ "crates/cellar/durability/raft/cell_groups/archive/ownership/wire.rs", "crates/cellar/tests/support/cell_group_archive_publication.rs", "crates/cellar/durability/raft/cell_groups/directory/archive.rs", - "crates/cellar/tests/support/cell_group_archive_binding.rs" + "crates/cellar/tests/support/cell_group_archive_binding.rs", + "crates/cellar/durability/raft/cell_groups/archive/execution.rs", + "crates/cellar/durability/raft/cell_groups/archive/execution_tests.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/job.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs", + "crates/cellar/tests/native_cell_groups_archive_controller.rs", + "crates/cellar/tests/support/cell_group_controller_archive.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From a4f37541cdc979252ea422d3dd5e224eb8b0ac2a Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 10:46:31 -0700 Subject: [PATCH 175/568] test: load archive fixture once --- crates/cellar/tests/native_cell_groups_archive_controller.rs | 3 --- 1 file changed, 3 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index d08e70f35..299efba1f 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -3,9 +3,6 @@ //! Controller-owned archive publication; no fixture event pumping after startup. #![allow(clippy::disallowed_methods)] -#[path = "support/cell_group_archive_store.rs"] -#[allow(dead_code)] -mod archive_store; #[path = "support/cell_group_checkpoint.rs"] #[allow(dead_code)] mod checkpoint_fixture; From 7914177d6f17e3f6d647a39dc496122157283591 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 11:11:48 -0700 Subject: [PATCH 176/568] fix: keep checkpoint source refusals strict --- .../raft/cell_groups/checkpoint/mod.rs | 50 ++++++++++++++++++- .../durability/raft/cell_groups/owner.rs | 2 +- 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 738e796a0..757c2f685 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -163,6 +163,13 @@ pub struct CellGroupCheckpointPool { limits: CellGroupCheckpointLimits, } +#[derive(Clone, Copy)] +enum SourceRoot { + Published, + Retained, + OwnerCurrent, +} + impl CellGroupCheckpointPool { pub fn new(limits: CellGroupCheckpointLimits) -> Result { if !(1..=4096).contains(&limits.max_sources) @@ -223,6 +230,31 @@ impl CellGroupCheckpointPool { group: &GroupLog, root: &PreparedGroupManifest, is_retained: bool, + ) -> Result { + self.capture_checked( + group, + root, + if is_retained { + SourceRoot::Retained + } else { + SourceRoot::Published + }, + ) + } + + pub(super) fn capture_owner( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + ) -> Result { + self.capture_checked(group, root, SourceRoot::OwnerCurrent) + } + + fn capture_checked( + &self, + group: &GroupLog, + root: &PreparedGroupManifest, + source: SourceRoot, ) -> Result { if !Arc::ptr_eq(&root.engine_owner, &group.owner) || !root @@ -273,7 +305,7 @@ impl CellGroupCheckpointPool { let header = { let cache = group.read()?; let marker = root.marker(); - if is_retained { + if matches!(source, SourceRoot::Retained) { if cache.checkpoint_manifest.as_ref() != Some(marker) { return Err(CellGroupCheckpointError::Invalid( "retained checkpoint changed", @@ -299,7 +331,21 @@ impl CellGroupCheckpointPool { .as_ref() .is_some_and(|pins| pins.root() != marker.root()) { - return Err(CellGroupCheckpointError::NotReady); + // Only the owner's sealed prior applied root may race a + // newer engine publication. Public callers supplying stale + // or prospective preparations retain strict refusal. + if matches!(source, SourceRoot::OwnerCurrent) + && cache.applied.position.index > marker.position().index + && cache.applied.image.is_none() + && cache.hard_state.commit >= marker.position().index.get() + && cache.group_manifest.as_ref().is_some_and(|published| { + published.position() == cache.applied.position + && published.route() == marker.route() + }) + { + return Err(CellGroupCheckpointError::NotReady); + } + return Err(CellGroupCheckpointError::Invalid("source durable cut")); } cache .membership diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 047aadd4c..953bece9c 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -338,7 +338,7 @@ impl CellGroupOwner { ) -> Result { self.live() .map_err(super::CellGroupCheckpointError::Authority)?; - pool.capture(&self.group, &self.current) + pool.capture_owner(&self.group, &self.current) } pub(super) fn select_read( From b8b1c7cb94eeaa1e0633d289175e69b3fcc36e3c Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 11:51:54 -0700 Subject: [PATCH 177/568] docs: record controller archive qualification --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 111 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 70 +++++++++++ 3 files changed, 128 insertions(+), 59 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 0c16f31e4..eea9ed310 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "df0ba1f67c0253a57127a215540f08bb13135279", - "runtime_revision": "df0ba1f67c0253a57127a215540f08bb13135279", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified.", + "source_revision": "7914177d6f17e3f6d647a39dc496122157283591", + "runtime_revision": "7914177d6f17e3f6d647a39dc496122157283591", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 387a5e1c1..c38df8196 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,62 +15,61 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial source is `df0ba1f67c0253a57127a215540f08bb13135279`. -CG-06 is IN_PROGRESS. Namespace-owned immutable binding now fixes the expected -archive registry, complete birth route and namespace data location. Persist the -one-use initializer's canonical bytes before commit. Fresh catalog owners reopen -only that expected identity; competing initialization and missing binding/counters -refuse. Interrupted initialization requires authority recovery, never repair/reset. - -The existing runtime worker can capture its exact durable applied root with the -node checkpoint pool. Physical reads and remote I/O run outside the worker. -`upload_cut` seals that checkpoint's root/configuration/permanent membership before -upload and creates a zero-entry tail without recapturing the moving live source. -Both image modes qualify held B upload plus advancing A quorum, physical verification -and publication of the exact earlier cut. A single-source pool admits upload and -later verification sequentially. Runtime capture respects quotas and joined close. - -All-three-voter disk erasure now removes local archive intent journals and drops -uploader/registry owners, then reconstructs expected identity from remote namespace -metadata through a fresh catalog owner before physical data recovery. Untrusted -checkpoint/tail locators remain externally retained. This is not fenced fresh serving. - -All 32 gate commands pass: nineteen exact archive cases, 157 execution cases, five -snapshot cases and four LTX generation regressions (185 distinct Rust cases), all-target -build/Clippy, formatting/layout, dependencies/docs/preflight and 37 Python tests. -The partial record under -`.cellar/cell-groups-evidence/CG-06-catalog-cut/qualification/` -binds 336 source hashes, four copied executables, frozen inputs, hashed outputs -and its protected-publication predecessor. Independent hashes/source/exact-selection -verification passes. See [progress](progress.md) and the -[archive contract](protocols.md#protected-publication-and-remote-collection). - -Next compose controller-owned archive work with durable operation journals, bounded -retry, joined cancellation/shutdown and retained verification admission. Settle -actual S3 command/enrollment/receipt outputs using their exact cell version/result -and current routing authority, preserving captured inputs and never rerunning callbacks. -Shared dispatch still refuses S3. Add explicit unarchived-byte/disk admission so -Quorum can progress independently of S3 within its bound. Qualify physical local -shared-engine Raft file purge and fenced unmanaged recovery with fresh group/cell -incarnations. Source birth/provenance and every possible old voter lineage, including -joins after the recovery cut, must survive voter loss; selected-cut membership is -not the complete fence. The fencing-authority API plus Docker qualification remains -the accepted direction. - -Protected publication/remote collection retains the earlier source-witness, -independent replay, explicit reader/pending-parent holds, never-reused generation -and delayed-DELETE guarantees. Persist every exact intent before mutation. Unknown -outcomes reconcile only the original intent; retired/aborted tokens cannot resurrect. -Coverage is historical data durability, without current serving/quorum/DR authority. -CG-08 still owns cache/effect/result lifetime composition and private-cache maintenance, -then CG-07 cuts the public SDK/fleet over. Topology, placement, hibernation, managed -recovery, RustFS/load/fault qualification and final legacy removal remain. - -The actual in-memory adapter supplies native evidence. Real RustFS, Docker fencing, -fresh-quorum serving, measured load and public hard cutover remain unqualified. The -other four original CG-06 cases and full task gate are pending. Preserve the protected- -publication source `87348477`, isolated-verifier source `14955f2d`, committed-tail -source `7a29c2a9` and their ignored evidence for audit. +The latest partial source is `7914177d6f17e3f6d647a39dc496122157283591`. +CG-06 is IN_PROGRESS. One fair archive lane now runs on the existing node controller, +using a deduplicated dirty-group queue, existing timer and shared admission pools. +Registration requires the initialized namespace binding and a private verifier store. +An engine/creation-bound locked journal fsyncs exact intents before remote mutations. +Missing established metadata refuses. Uncertain publication reconciles its exact +hold, or abort-fences the ticket and reconciles again before allocating anything new. +New head plus predecessor retirement are journaled before release. Physical coverage +stays job-scoped; archive status is historical metadata without S3 settlement rights. + +Completion compares covered position with runtime applied state and consumed apply +events. This fixes the reproduced idle-tail gap after recovery of an older publication. +Both image forms pass held publication/quorum progress, lost CAS reply/readback outage, +exact-intent recovery, cancelled registration response, joined close, single current +hold and independent recovery of every cell. Journal reopen/missing-state and orphaned +physical waiter tests pass. A sealed scheduling capability uses captured domain tokens +without shutdown rights; archive-specific physical joins precede private-store, +journal-lock and runtime/SQL teardown. New verifier stores are required on restart. + +All 40 gate commands pass: nineteen exact archive, two exact controller and two exact +lifecycle/journal unit cases, 157 execution, five snapshot and four LTX regressions +(189 distinct Rust cases across six copied executables), all-target build/Clippy, +formatting/layout, dependencies/docs/preflight and all 37 Python tests. The record at +`.cellar/cell-groups-evidence/CG-06-controller-archive/qualification/checkpoint.json` +binds 344 source hashes, frozen inputs, hashed logs and its catalog-cut predecessor. +Independent source/artifact/exact-selection audit passes. Initial compile, native +idle-progress, strict source-refusal and Clippy failures are source-bound and retained. See [progress](progress.md) +and the [controller archive contract](protocols.md#controller-owned-archive-publication). + +Next implement actual S3 command/enrollment/receipt output settlement using retained +independently verified coverage with exact cell version/result/current-route checks. +Preserve captured identities/inputs and deadlines; never rerun callbacks. Shared +dispatch still refuses S3. Add explicit unarchived-byte/disk admission for bounded +Quorum availability without S3. Then qualify physical local Raft Engine file purge, +fenced unmanaged recovery into fresh group/cell incarnations and complete old-voter +lineage inventory, including joins after the recovery cut. Fencing-authority API plus +Docker qualification remains the accepted direction. Automatic remote/private-cache +maintenance and process-kill controller restart are still unqualified. This initial +controller coalesces complete cuts; measured incremental scheduling remains future work. + +Namespace binding fixes expected registry identity and canonical storage location. +Runtime source capture/upload_cut keeps root/configuration/permanent membership +stable across I/O. All-voter disk/journal erasure still recovers data via a fresh catalog +owner and externally retained untrusted locators, without fencing/replacement voters. +Protected publication/collection retains exact-intent, never-reused generation, +pending-parent/reader hold and delayed-DELETE guarantees. Persist every intent before +mutation; unknown outcomes never justify another identity or counter reset. + +CG-08 effect/cache/result lifetime and maintenance precede CG-07 public SDK/fleet +cutover. Topology, placement, hibernation, managed recovery, RustFS/load/fault gates +and final legacy removal remain. Actual in-memory adapters supply the current native +evidence. Real RustFS, Docker fencing, fresh-quorum serving, measured load and full +hard cutover remain unqualified. Four original CG-06 cases and its full task gate are +pending. Preserve catalog-cut source `df0ba1f6`, protected publication `87348477`, +isolated verifier `14955f2d`, committed tail `7a29c2a9` and their ignored evidence. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 478ad87e7..136fe5bde 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,76 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 controller-owned archive publication + +Source: `7914177d6f17e3f6d647a39dc496122157283591`. CG-06 remains IN_PROGRESS. +The existing node controller now owns one fair archive lane, using a deduplicated +dirty-group queue and its existing timer. Clean groups need no roster scan. Apply +and checkpoint-install progress coalesce while object-store work runs outside the +SQL lane. Registration opens only the namespace-bound expected registry, with a +private verifier store and node-shared source/I/O/upload/replay admission. + +An exclusively locked journal binds the node engine creation and archive identity. +It fsyncs canonical reservation/publication intents before remote mutation, using +an atomic file replacement plus directory sync/readback. Missing established +identity/state refuses rather than recreating metadata. Attempts reload selected +state after uncertain local writes. Restart/retry reconciles an exact publication; +if absent, it abort-fences the ticket and reconciles again before reallocation. +A new head and predecessor retirement are journaled before the old hold is released. +Published markers remain advisory metadata; physical root/replay admission does +not escape the job or remain reserved for every registered group. + +The sealed archive execution capability accepts the controller's captured domain +token without ownership of domain shutdown. Scoped blocking-work accounting spans +abandoned async waiters; close joins logical jobs and their physical closures, +then closes private stores and releases journal locks before runtime/SQL teardown. +Typed checkpoint `NotReady` and admission overload retry after progress; corrupt +metadata and journal loss stop archival. They do not become S3 acknowledgements. + +Both image modes qualify held publication while the real quorum continues, lost +publication response plus failed readback, persisted exact-intent reconciliation, +cancelled registration response, joined close, a single current remote hold and +independent recovery of every cell afterward. The first uncertain-publication +scenario found a lost idle-tail wake: recovery of M could clear dirty progress for +already-applied N. Completion now observes both runtime applied state and consumed +apply events, and the unchanged regression proves N archives without another request. +Unit cases qualify actual journal reopen/missing-state refusal and physical drain +after dropping a blocking waiter. This does not qualify a process-kill controller +restart or a bounded S3 outage ledger. + +All 40 gate commands pass: nineteen exact archive cases, two exact controller cases, +two exact lifecycle/journal unit cases, 157 execution cases, five snapshot cases and +four LTX generation cases (189 distinct Rust cases across six copied executables). +Locked all-target build/Clippy for Cellar/logic/LTX, formatting/layout, dependencies, +docs, preflight, plan validation and all 37 Python tests pass. The partial record at +`.cellar/cell-groups-evidence/CG-06-controller-archive/qualification/checkpoint.json` +binds 344 source hashes, frozen plan inputs, hashed logs/binaries and the preceding +catalog-cut checkpoint. Independent source/artifact/exact-selection verification +passes. Source snapshots retain the initial error-wrapper compile failure, native +idle-tail regression, prospective-root refusal regression and Clippy reply-type +and duplicate-module failures. Prospective/stale public roots still refuse as +invalid; only trusted owner-current capture can return typed retryable NotReady. No acceptance case, assertion, +oracle or existing quota was weakened. This is not full CG-06 task evidence. + +Reflection and next priorities: connect pending S3 outputs to retained independent +coverage proofs, verifying exact command/enrollment/receipt identity, cell version, +result and current routing authority. Preserve captured inputs and original deadlines; +never rerun callbacks. Add explicit unarchived-byte/disk admission before claiming +bounded Quorum progress through S3 outages. Then qualify local shared-engine Raft +file purge and fenced recovery under fresh group/cell incarnations. The source fence +inventory must cover every possible old voter lineage, including later joins. +Automatic remote/private-cache maintenance and controller process-kill restart still +need qualification. The controller currently uploads complete coalesced cuts; +incremental tail scheduling and performance require the measured-load gates. + +CG-08 effects/cache lifetime work precedes CG-07 public SDK/fleet cutover, followed +by topology, placement, hibernation, managed recovery, RustFS/load/fault qualification +and final legacy removal. Shared dispatch still refuses S3 durability. The other +four original CG-06 cases and its full task gate remain pending. These scenarios +use the actual in-memory object-store adapter; no real RustFS, Docker fence, +fresh-quorum serving, load or full hard cutover is claimed. Transfer ignored evidence +and its predecessors for audit. + ## CG-06 namespace archive identity and stable cut capture Source: `df0ba1f67c0253a57127a215540f08bb13135279`. CG-06 remains IN_PROGRESS. From 08d5a1072673cc0a1d4fb430b86e52195b4fb2a3 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:01:40 -0700 Subject: [PATCH 178/568] test: qualify cell-group archives with RustFS --- .../native_cell_groups_archive_controller.rs | 41 ++++ .../tests/support/cell_group_archive_store.rs | 6 +- .../support/cell_group_controller_archive.rs | 57 ++++-- .../support/cell_group_engine_fixture.rs | 20 +- .../tests/support/cell_group_runtime.rs | 4 + .../cellar/tests/support/cell_group_rustfs.rs | 30 +++ .../tests/support/cell_group_s3_response.rs | 132 +++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 8 + .../support/cell_group_sql_registered.rs | 25 +++ docs/design/multi-raft/cell-groups/tasks.json | 5 +- tools/multi-raft/README.md | 39 +++- tools/multi-raft/cell_groups.py | 185 ++++++++++++++++++ tools/multi-raft/local.py | 13 +- 13 files changed, 545 insertions(+), 20 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_rustfs.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_response.rs create mode 100644 tools/multi-raft/cell_groups.py diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index 299efba1f..c595d4ad3 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -15,6 +15,8 @@ mod images; #[path = "support/cell_group_runtime.rs"] #[allow(dead_code, unused_imports)] mod runtime_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; include!("support/cell_group_engine_fixture.rs"); @@ -33,3 +35,42 @@ async fn controller_archive_reconciles_exact_intent_and_joins_close() -> Result< } Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_controller_archive_publishes_stable_cuts() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_publish_store( + rooted, + rustfs::store("publish", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_controller_archive_reconciles_exact_intent_and_joins_close() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_controller_reconcile_store( + rooted, + rustfs::store("reconcile", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_unproved_s3_responses_refuse() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::unproved_s3_responses( + rooted, + rustfs::store("responses", rooted)?, + )) + .await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index ad845c39a..544131645 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -1,6 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Instrument the actual in-memory object-store adapter, without replacing its +//! Instrument the selected object-store adapter, without replacing its //! conditional-create semantics, bytes or physical checkpoint verification. use futures_util::stream::BoxStream; use object_store::{path::Path, *}; @@ -18,7 +18,7 @@ pub struct Gate { #[derive(Debug)] pub struct Store { - pub inner: Arc, + pub inner: Arc, pub puts: AtomicUsize, pub lose_put_response: AtomicBool, pub deletes: AtomicUsize, @@ -36,7 +36,7 @@ impl Store { Self::around(Arc::new(memory::InMemory::new())) } - pub fn around(inner: Arc) -> Arc { + pub fn around(inner: Arc) -> Arc { Arc::new(Self { inner, puts: AtomicUsize::new(0), diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 2e2c852ef..95701934b 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -185,8 +185,15 @@ async fn add( } async fn setup(rooted: bool) -> Result<(RuntimeCluster, GroupRoute)> { + setup_store(rooted, Arc::new(object_store::memory::InMemory::new())).await +} + +async fn setup_store( + rooted: bool, + store: Arc, +) -> Result<(RuntimeCluster, GroupRoute)> { let class = RaftClass::new("Counter", 1, [81; 32])?; - let (fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + let (fixture, route, _, _) = Box::pin(registered::setup_store(rooted, &class, store)).await?; Ok((fixture, route)) } @@ -280,18 +287,32 @@ async fn audit( pub(crate) async fn archive_controller_publish(rooted: bool) -> Result<()> { let (mut fixture, route) = setup(rooted).await?; - let (config, remote) = registration(&fixture, &route, rooted).await?; + publish(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +pub(crate) async fn archive_controller_publish_store( + rooted: bool, + store: Arc, +) -> Result<()> { + let (mut fixture, route) = setup_store(rooted, store).await?; + publish(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +async fn publish(fixture: &mut RuntimeCluster, route: &GroupRoute, rooted: bool) -> Result<()> { + let (config, remote) = registration(fixture, route, rooted).await?; let initial = fixture.runtimes[0] .status(route.history) .await? .applied .index; let gate = remote.hold_publication(); - let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let controllers = controller::start_with_limits(fixture, controller::limits()).await?; let call = controllers[0].archive(config.clone())?; tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; // A held archive publication does not stop real quorum/apply progress. - let later = add(&fixture, &route, rooted, 1).await?; + let later = add(fixture, route, rooted, 1).await?; assert!(later > initial); assert!(controllers[0] .archive_status(route.history)? @@ -315,14 +336,28 @@ pub(crate) async fn archive_controller_publish(rooted: bool) -> Result<()> { .unwrap() .stopped ); - audit(&fixture, &config, rooted, 1).await?; + audit(fixture, &config, rooted, 1).await?; drop(config); - fixture.close().await + Ok(()) } pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { let (mut fixture, route) = setup(rooted).await?; - let (config, remote) = registration(&fixture, &route, rooted).await?; + reconcile(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +pub(crate) async fn archive_controller_reconcile_store( + rooted: bool, + store: Arc, +) -> Result<()> { + let (mut fixture, route) = setup_store(rooted, store).await?; + reconcile(&mut fixture, &route, rooted).await?; + fixture.close().await +} + +async fn reconcile(fixture: &mut RuntimeCluster, route: &GroupRoute, rooted: bool) -> Result<()> { + let (config, remote) = registration(fixture, route, rooted).await?; let initial = fixture.runtimes[0] .status(route.history) .await? @@ -331,13 +366,13 @@ pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { remote .lose_publication_and_reads .store(true, Ordering::Release); - let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let controllers = controller::start_with_limits(fixture, controller::limits()).await?; // Cancelling a registration response does not cancel its admitted owner. drop(controllers[0].archive(config.clone())?); controller::wait_for(async || Ok(remote.outage.load(Ordering::Acquire))).await?; let mut latest = initial; for n in 1..=4 { - latest = add(&fixture, &route, rooted, n).await?; + latest = add(fixture, route, rooted, n).await?; } let status = controllers[0].archive_status(route.history)?.unwrap(); assert!(status.published.is_none()); @@ -371,7 +406,7 @@ pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { for controller in controllers.iter().skip(1) { controller.close().await?; } - audit(&fixture, &config, rooted, 4).await?; + audit(fixture, &config, rooted, 4).await?; let objects = remote.list(None).try_collect::>().await?; let key = objects .iter() @@ -381,5 +416,5 @@ pub(crate) async fn archive_controller_reconcile(rooted: bool) -> Result<()> { let registry: serde_json::Value = serde_json::from_slice(&bytes)?; assert_eq!(registry["holds"].as_array().context("holds")?.len(), 1); drop(config); - fixture.close().await + Ok(()) } diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 7cdb2e132..563f4b483 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -33,7 +33,7 @@ struct Cluster { _root: tempfile::TempDir, domain: ExecutionDomain, directory: RangeDirectory, - store: Arc, + store: Arc, birth: CellGroupBirth, voters: Vec, network: VecDeque, @@ -71,6 +71,23 @@ impl Cluster { root: tempfile::TempDir, filesystem: Arc, schema: [u8; 32], + ) -> Result { + Self::with_store( + options, + root, + filesystem, + schema, + Arc::new(object_store::memory::InMemory::new()), + ) + .await + } + + async fn with_store( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { @@ -79,7 +96,6 @@ impl Cluster { group: GroupId::new([3; 16])?, incarnation: IncarnationId::new([4; 16])?, }; - let store = Arc::new(object_store::memory::InMemory::new()); let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; let namespace = domain .run(catalog.open_namespace(NamespaceConfig::new( diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 6c68c2c7a..37acf4868 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -804,7 +804,11 @@ pub(super) async fn attachment_reserves_before_owner() -> Result<()> { #[path = "cell_group_sql.rs"] mod sql; #[allow(unused_imports)] +pub(super) use sql::unproved_s3_responses; +#[allow(unused_imports)] pub(super) use sql::{archive_controller_publish, archive_controller_reconcile}; +#[allow(unused_imports)] +pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; diff --git a/crates/cellar/tests/support/cell_group_rustfs.rs b/crates/cellar/tests/support/cell_group_rustfs.rs new file mode 100644 index 000000000..518f78498 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_rustfs.rs @@ -0,0 +1,30 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit local RustFS configuration; never fall back to memory or ambient IAM. +use anyhow::{ensure, Result}; +use object_store::{aws::AmazonS3Builder, path::Path, prefix::PrefixStore, ObjectStore}; +use std::{env, sync::Arc}; + +pub fn store(case: &str, rooted: bool) -> Result> { + let endpoint = env::var("CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT")?; + ensure!( + endpoint.starts_with("http://127.0.0.1:"), + "local RustFS endpoint" + ); + let prefix = env::var("CELLAR_CELL_GROUPS_RUSTFS_PREFIX")?; + ensure!(!prefix.is_empty(), "isolated RustFS prefix"); + let prefix = Path::parse(format!( + "{prefix}/{case}/{}", + if rooted { "extents" } else { "materialized" } + ))?; + let store = AmazonS3Builder::new() + .with_bucket_name(env::var("CELLAR_CELL_GROUPS_RUSTFS_BUCKET")?) + .with_endpoint(endpoint) + .with_access_key_id(env::var("AWS_ACCESS_KEY_ID")?) + .with_secret_access_key(env::var("AWS_SECRET_ACCESS_KEY")?) + .with_region("us-east-1") + .with_allow_http(true) + .with_conditional_put(object_store::aws::S3ConditionalPut::ETagMatch) + .build()?; + Ok(Arc::new(PrefixStore::new(store, prefix))) +} diff --git a/crates/cellar/tests/support/cell_group_s3_response.rs b/crates/cellar/tests/support/cell_group_s3_response.rs new file mode 100644 index 000000000..14d0a08f4 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_response.rs @@ -0,0 +1,132 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A reachable S3 backend cannot substitute for an exact output durability proof. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn unproved_s3_responses( + rooted: bool, + store: Arc, +) -> Result<()> { + let callbacks = Arc::new(AtomicUsize::new(0)); + let initializations = callbacks.clone(); + let commands = callbacks.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([82; 32], move |_, _| { + initializations.fetch_add(1, Ordering::SeqCst); + Ok(()) + })? + .command("Add", [83; 32], move |_, _| { + commands.fetch_add(1, Ordering::SeqCst); + Ok(b"unproved".to_vec()) + })?, + ); + let (fixture, _, cell, _) = + Box::pin(registered::setup_store(rooted, &class, store.clone())).await?; + let mut minimum = NamespaceConfig::new( + cell.cluster, + cell.namespace, + cellar_logic::NamespaceMode::Raft, + class.schema_digest(), + ); + minimum.minimum_durability = Durability::S3; + let catalog = Catalog::new( + Arc::new(object_store::prefix::PrefixStore::new(store, "s3-minimum")), + Default::default(), + )?; + let namespace = fixture + .cluster + .domain + .run(catalog.open_namespace(minimum)) + .await??; + let required = RangeDirectory::new(namespace, Default::default())?; + for (directory, durability) in [ + (fixture.cluster.directory.clone(), Durability::S3), + (required, Durability::Quorum), + ] { + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + directory, + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let bounds = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 2, + }; + let id = CommandId::new(ClientId::new([188; 16])?, 1)?; + let contract = class.contract("Add").context("Add")?; + let error = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id, + contract: contract.clone(), + input: vec![], + durability, + retry_until_ms: u64::MAX, + }, + bounds, + None, + ) + .await + .err() + .context("unproved S3 command refused")?; + assert!(!error.ambiguous); + refusal(&error.cause); + let error = dispatcher + .lookup_receipt( + cell.clone(), + CellReceiptRequest { + id, + contract, + input: vec![], + durability, + }, + bounds, + None, + ) + .err() + .context("unproved S3 receipt refused")?; + refusal(&error); + let error = dispatcher + .enroll( + cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([188; 16])?, + durability, + }, + bounds, + None, + ) + .await + .err() + .context("unproved S3 enrollment refused")?; + assert!(!error.ambiguous); + refusal(&error.cause); + assert_eq!(dispatcher.usage(), (0, 0)); + } + assert_eq!(callbacks.load(Ordering::SeqCst), 0); + fixture.close().await +} + +fn refusal(error: &CellDispatchError) { + assert!(matches!( + error, + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c1afae55d..0f964c3a0 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -529,10 +529,18 @@ pub(crate) use command_dispatch::commands as routed_commands; #[path = "cell_group_controller_archive.rs"] #[allow(dead_code, unused_imports)] mod archive_controller; +#[path = "cell_group_s3_response.rs"] +mod s3_response; +#[allow(unused_imports)] +pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] mod controller; #[allow(unused_imports)] pub(crate) use archive_controller::{archive_controller_publish, archive_controller_reconcile}; +#[allow(unused_imports)] +pub(crate) use archive_controller::{ + archive_controller_publish_store, archive_controller_reconcile_store, +}; #[path = "cell_group_controller_commands.rs"] mod controller_commands; pub(crate) use controller::{ diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index 1bc14b8af..652a6d966 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -177,6 +177,31 @@ pub(super) async fn setup_in( class.schema_digest(), ) .await?; + setup_cluster(rooted, cluster, filesystem).await +} + +pub(super) async fn setup_store( + rooted: bool, + class: &RaftClass, + store: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Cluster::with_store( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + ) + .await?; + setup_cluster(rooted, cluster, filesystem).await +} + +async fn setup_cluster( + rooted: bool, + cluster: Cluster, + filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; let route = fixture.cluster.birth.route().clone(); if rooted { diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 1e2bb6f8c..3c5dd8887 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1758,7 +1758,10 @@ "crates/cellar/durability/raft/cell_groups/controller/archive/journal.rs", "crates/cellar/durability/raft/cell_groups/controller/archive/journal_tests.rs", "crates/cellar/tests/native_cell_groups_archive_controller.rs", - "crates/cellar/tests/support/cell_group_controller_archive.rs" + "crates/cellar/tests/support/cell_group_controller_archive.rs", + "crates/cellar/tests/support/cell_group_rustfs.rs", + "crates/cellar/tests/support/cell_group_s3_response.rs", + "tools/multi-raft/cell_groups.py" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 0352612c6..faa34b787 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -26,7 +26,10 @@ The helper uses `docker --context colima-cellar-multi-raft` explicitly. It never stops another Colima profile or changes the user's global Docker context. Preparation requires a fresh run directory and digest-pinned RustFS image. A randomly named private network, object volume, credentials and loopback-only -S3 port belong to that run. RustFS has one CPU and 2 GiB memory. The readiness +S3 port belong to that run. The helper chooses a free host port explicitly: +Docker's guest-selected ephemeral port can collide with another Colima profile's +host forward and reach the wrong service. The pinned image uses the same +20-character access-key profile as SDK CI. RustFS has one CPU and 2 GiB memory. The readiness probe checks its actual captured cgroup settings. The S3 probe checks seven real HTTP operations: bucket creation, conditional creation, matching read, refused overwrite, ETag compare-and-swap, refused stale ETag and read after replacement. @@ -53,6 +56,40 @@ preparations retain their environment record for diagnosis. A separate cleanup step can remove the exact recorded resources after their data is no longer needed; never use a global Docker prune or delete a Colima profile as cleanup. +## Shared cell-group archive checks + +Commit the test and harness sources, then run the native three-voter TLS fixture +against the prepared RustFS backend. Use an absolute Cargo target directory unique +to this checkout and a fresh output directory: + +```sh +python3 tools/multi-raft/cell_groups.py \ + --backend /tmp/cellar-raft-local-001 \ + --output /tmp/cellar-cell-groups-rustfs-001 \ + --target-directory "$HOME/Workspace/crabbuild-target/cellar-my-checkout" +``` + +The runner verifies the backend's ownership label, image, running state, endpoint +and actual resource limits, then makes seven signed S3 probes in a new bucket. +It runs three exact ignored RustFS cases, each in materialized and extent modes: +stable publication with continuing quorum, uncertain-publication reconciliation +with joined close, and refusal of unproved S3 responses. Catalog setup, archive +objects, conditional registry updates and independent recovery use the actual +S3 client. Fault injection surrounds that client; lost publication responses are +injected after the real conditional write commits. No memory fallback exists. + +The response case currently verifies that command, receipt and enrollment +requests cannot claim S3 durability, including when required by the namespace. +Successful S3 response settlement still needs operation-specific proofs. The +runner does not qualify local log file purge, fenced recovery or performance. +The voters run as native host tasks, so this is not the container resource gate. + +`record.json` retains source and executable hashes, exact selections, output +hashes, S3 observations and captured backend identity without credentials. +Failed runs retain their record and logs. Every run owns a new remote prefix; +remote objects and the backend remain available for inspection until explicitly +stopped using the command above. No cleanup deletes another run's data. + ## Native node runs The repository's `tools/perf/raft.py` harness now provides three native SDK node diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py new file mode 100644 index 000000000..6e4318395 --- /dev/null +++ b/tools/multi-raft/cell_groups.py @@ -0,0 +1,185 @@ +#!/usr/bin/env python3 +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Qualify shared-group archive setup and response refusals against local RustFS. + +Uses only the recorded owned backend. Each invocation owns a new evidence +directory and remote prefix. Credentials are passed through the child environment, +never arguments or evidence. This does not qualify successful S3 acknowledgements. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import secrets +import shutil +import subprocess +import time + +from local import command +from store_probe import probe + +ROOT = Path(__file__).resolve().parents[2] +TARGET = "native_cell_groups_archive_controller" +CASES = ( + "rustfs_controller_archive_publishes_stable_cuts", + "rustfs_controller_archive_reconciles_exact_intent_and_joins_close", + "rustfs_unproved_s3_responses_refuse", +) + + +def sha(path): + with path.open("rb") as source: + return hashlib.file_digest(source, "sha256").hexdigest() + + +def sources(): + result = subprocess.check_output(["git", "ls-files", "-z"], cwd=ROOT) + return [ + {"path": name, "sha256": sha(ROOT / name)} + for name in result.decode().split("\0") + if name and not name.startswith(("docs/", "plans/", "packages/", "examples/")) + and (ROOT / name).is_file() + ] + + +def backend(root): + record = json.loads((root / "environment.json").read_text()) + if (record["phase"] != "backend-ready" + or not re.fullmatch(r"cellar-multi-raft-[0-9a-f]{16}", record["owner"]) + or not re.fullmatch(r"colima-cellar-multi-raft(?:-[a-z0-9-]+)?", record["context"]) + or not re.fullmatch(r"http://127\.0\.0\.1:[0-9]+", record["endpoint"])): + raise ValueError("requires an owned, running loopback RustFS backend") + docker = ["docker", "--context", record["context"]] + actual = json.loads(command(docker + [ + "inspect", "--format", + '{"owner":{{json (index .Config.Labels "cellar.multi-raft.run")}},' + '"image":{{json .Image}},"running":{{json .State.Running}},' + '"limits":{{json .HostConfig}},"ports":{{json .NetworkSettings.Ports}}}', + record["container"], + ])) + limits = actual["limits"] + if (actual["owner"] != record["owner"] or actual["image"] != record["image_id"] + or not actual["running"] + or (limits["NanoCpus"], limits["Memory"], limits["MemorySwap"]) + != (10**9, 2 * 1024**3, 2 * 1024**3)): + raise ValueError("backend identity/running state/resource limits changed") + ports = actual["ports"]["9000/tcp"] + if ports != [{"HostIp": "127.0.0.1", "HostPort": record["endpoint"].rsplit(":", 1)[1]}]: + raise ValueError("backend endpoint differs from the captured container") + credentials = dict(line.split("=", 1) for line in (root / "rustfs.env").read_text().splitlines()) + record["observed_rustfs"] = { + "owner": actual["owner"], "image": actual["image"], + "nano_cpus": limits["NanoCpus"], "memory_bytes": limits["Memory"], + "memory_swap_bytes": limits["MemorySwap"], "ports": ports, + } + return record, credentials + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--backend", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--target-directory", type=Path, required=True, + help="absolute checkout-private Cargo target on the workspace volume") + args = parser.parse_args() + if not args.target_directory.is_absolute(): + parser.error("--target-directory must be absolute and unique to this checkout") + subprocess.run(["git", "diff", "--exit-code", "HEAD", "--", ".", + ":(exclude)docs/**", ":(exclude)plans/**"], cwd=ROOT, check=True, + stdout=subprocess.DEVNULL) + untracked = subprocess.check_output(["git", "ls-files", "--others", "--exclude-standard", "-z"], cwd=ROOT) + if any(name.startswith(("crates/", "tools/")) for name in untracked.decode().split("\0")): + parser.error("commit qualification sources before running") + remote, credentials = backend(args.backend.resolve()) + out = args.output.resolve() + out.mkdir(mode=0o700, parents=True, exist_ok=False) + prefix = "cell-groups/" + secrets.token_hex(16) + record = { + "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and unproved-S3 refusal", + "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), + "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], + "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " + "No successful S3 command/receipt/enrollment acknowledgements, byte/disk outage bound, " + "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", + } + + def save(): + (out / "record.json").write_text(json.dumps(record, indent=2) + "\n") + + save() + try: + bucket = remote["owner"] + "-cg-" + secrets.token_hex(4) + record["s3_probe"] = probe(remote["endpoint"], bucket, + credentials["RUSTFS_ACCESS_KEY"], credentials["RUSTFS_SECRET_KEY"]) + record["bucket"] = bucket + env = dict(os.environ) + # Avoid inheriting session credentials or arbitrary AWS endpoint overrides. + for key in list(env): + if key.startswith(("AWS_", "CELLAR_CELL_GROUPS_RUSTFS_")): + del env[key] + env.update( + AWS_ACCESS_KEY_ID=credentials["RUSTFS_ACCESS_KEY"], + AWS_SECRET_ACCESS_KEY=credentials["RUSTFS_SECRET_KEY"], + AWS_EC2_METADATA_DISABLED="true", + CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT=remote["endpoint"], + CELLAR_CELL_GROUPS_RUSTFS_BUCKET=bucket, + CELLAR_CELL_GROUPS_RUSTFS_PREFIX=prefix, + CARGO_TARGET_DIR=str(args.target_directory), + CARGO_PROFILE_DEV_DEBUG="0", CARGO_PROFILE_TEST_DEBUG="0", + ) + cargo = ["cargo", "+1.98.1", "test", "-p", "cellar", "--locked", "--test", TARGET, "--"] + for number, selection in enumerate(("--list", *CASES)): + argv = cargo + ([selection] if number == 0 else [selection, "--exact", "--ignored", "--test-threads=1"]) + print(f"START {number + 1}/4: {selection}", flush=True) + started = time.monotonic() + result = subprocess.run(argv, cwd=ROOT, env=env, text=True, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + output = result.stdout + for secret in credentials.values(): + output = output.replace(secret, "[REDACTED]") + log = out / f"{number:02}.log" + log.write_text(output) + record["commands"].append({"argv": argv, "exit_code": result.returncode, + "elapsed_seconds": round(time.monotonic() - started, 3), + "output": log.name, "sha256": sha(log)}) + save() + if result.returncode: + raise RuntimeError(f"{selection} failed; see {log}") + if number == 0: + for case in CASES: + if not re.search(rf"^{re.escape(case)}: test$", output, re.M): + raise RuntimeError("missing exact RustFS test in executable inventory") + elif not (re.search(rf"^test {re.escape(selection)} \.\.\. ok$", output, re.M) + and "test result: ok. 1 passed; 0 failed; 0 ignored;" in output): + raise RuntimeError("exact RustFS test was not executed successfully") + match = re.search(r"Running [^\n]*\(([^\n)]+/native_cell_groups_archive_controller-[a-f0-9]+)\)", output) + if not match: + raise RuntimeError("missing executable provenance") + executable = Path(match[1]) + selected_hash = sha(executable) + if "executable" in record and record["executable"]["sha256"] != selected_hash: + raise RuntimeError("executable changed during qualification") + destination = out / executable.name + shutil.copy2(executable, destination) + record["executable"] = {"path": destination.name, "sha256": selected_hash} + print(f"PASS {selection}", flush=True) + if sources() != record["source_inputs"]: + raise RuntimeError("source changed during qualification") + backend(args.backend.resolve()) + record["status"] = "PASSED" + record["passed_cases"] = list(CASES) + except Exception: + record["status"] = "FAILED" + raise + finally: + save() + print(json.dumps({"record": str(out / "record.json"), "status": record["status"], + "rustfs_cases": len(CASES), "image_modes_per_case": 2}), flush=True) + + +if __name__ == "__main__": + main() diff --git a/tools/multi-raft/local.py b/tools/multi-raft/local.py index 61074227c..cd08e837f 100644 --- a/tools/multi-raft/local.py +++ b/tools/multi-raft/local.py @@ -14,6 +14,7 @@ import pathlib import re import secrets +import socket import subprocess import time import urllib.error @@ -83,9 +84,17 @@ def prepare(args): rustfs_image=args.rustfs_image, network=owner, volume=owner + "-objects", container=owner + "-rustfs") save(root, record) + # Docker allocates ephemeral ports inside the guest. Another Colima profile + # may already forward that port on the host, silently reaching its service. + # Choose on the host instead; the signed probe still verifies connectivity. + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + host_port = listener.getsockname()[1] credential_file = root / "rustfs.env" with open(credential_file, "x", opener=lambda path, flags: os.open(path, flags, 0o600)) as output: - output.write("RUSTFS_ACCESS_KEY=" + secrets.token_hex(16) + "\n") + # The pinned image rejects longer access keys during signed S3 requests, + # even when its unauthenticated health endpoint reports readiness. + output.write("RUSTFS_ACCESS_KEY=" + secrets.token_hex(10) + "\n") output.write("RUSTFS_SECRET_KEY=" + secrets.token_hex(32) + "\n") command(docker + ["pull", args.rustfs_image]) command(docker + ["network", "create", "--label", label, record["network"]]) @@ -95,7 +104,7 @@ def prepare(args): "--network", record["network"], "--network-alias", "rustfs", "--cpus", "1", "--memory", "2g", "--memory-swap", "2g", "--ulimit", "nofile=65536:65536", "--restart", "no", - "--publish", "127.0.0.1::9000", "--env-file", str(credential_file), + "--publish", f"127.0.0.1:{host_port}:9000", "--env-file", str(credential_file), "--env", "RUSTFS_CONSOLE_ENABLE=false", "--env", "RUSTFS_OBS_LOGGER_LEVEL=error", "--mount", f"type=volume,src={record['volume']},dst=/data", args.rustfs_image, "/data", From 3158dc535a693e9bfebdc6b9638a7c6cf4f9c44b Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:02:01 -0700 Subject: [PATCH 179/568] fix: hash evidence with system Python --- tools/multi-raft/cell_groups.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 6e4318395..035568ecd 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -32,8 +32,11 @@ def sha(path): + digest = hashlib.sha256() with path.open("rb") as source: - return hashlib.file_digest(source, "sha256").hexdigest() + for chunk in iter(lambda: source.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() def sources(): From d739bf795f7e58c1c27b2c6c496f9f57355e3c02 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:04:06 -0700 Subject: [PATCH 180/568] test: scope shared S3 fixture lint allowance --- crates/cellar/tests/support/cell_group_sql.rs | 2 ++ 1 file changed, 2 insertions(+) diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 0f964c3a0..83cbb1139 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -530,6 +530,8 @@ pub(crate) use command_dispatch::commands as routed_commands; #[allow(dead_code, unused_imports)] mod archive_controller; #[path = "cell_group_s3_response.rs"] +// Selected by the archive-controller target; sibling targets share this SQL fixture. +#[allow(dead_code)] mod s3_response; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; From 261ccaf80384190c557ff212f6dfcbc77d2fb6e6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:18:37 -0700 Subject: [PATCH 181/568] docs: record local RustFS qualification --- docs/design/multi-raft/cell-groups/handoff.md | 18 ++++++++ .../design/multi-raft/cell-groups/progress.md | 42 +++++++++++++++++++ 2 files changed, 60 insertions(+) diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index c38df8196..3cf79e11d 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,6 +15,24 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). +The latest local RustFS qualification source is `d739bf795f7e58c1c27b2c6c496f9f57355e3c02`; production runtime +bytes remain unchanged from `7914177d`. Use +[the RustFS runner](../../../../tools/multi-raft/cell_groups.py) and +[setup instructions](../../../../tools/multi-raft/README.md#shared-cell-group-archive-checks) +for local S3 verification. Three exact cases pass in both image modes: controller +publication, uncertain publication/reconciliation with joined close, and refusal +of unproved S3 command/receipt/enrollment responses and namespace minimums. +Seven signed S3 probes, 183 native regressions and all ten check commands pass. +The source/executable/output-bound record and independent audit are under +`.cellar/cell-groups-evidence/CG-06-rustfs-controller/`. Credentials stay outside +source/evidence. Port allocation now accounts for other Colima profiles' host +forwards. The backend is real RustFS; voters are host-native tasks. + +Next extend this runner with positive S3 output cases while implementing exact +operation proof ownership. Existing refusal cases are not successful S3 settlement. +Unarchived-byte/disk limits, physical Raft log purge, fencing/fresh serving, measured +load and public cutover remain open. See the newest [progress](progress.md) entry. + The latest partial source is `7914177d6f17e3f6d647a39dc496122157283591`. CG-06 is IN_PROGRESS. One fair archive lane now runs on the existing node controller, using a deduplicated dirty-group queue, existing timer and shared admission pools. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 136fe5bde..07c48dde2 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,48 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 local RustFS archive and response-boundary qualification + +Source: `d739bf795f7e58c1c27b2c6c496f9f57355e3c02`. Production runtime bytes are unchanged from `7914177d`. +CG-06 remains IN_PROGRESS. The local qualification backend is real RustFS, as +requested. [The runner](../../../../tools/multi-raft/cell_groups.py) verifies the +owned backend's image, running state, endpoint and actual cgroup limits, then +creates a fresh bucket and isolated scenario/image prefixes. Explicit S3 clients +use ETag conditional writes without a memory or ambient-credential fallback. + +All three exact ignored cases pass in both materialized and extent modes: +controller publication while quorum advances, lost publication response/readback +with exact-intent reconciliation and joined close, and refusal of unproved S3 +command/receipt/enrollment responses, including a namespace S3 minimum. Catalog +setup and archive recovery use RustFS. Lost-response injection follows a real +successful conditional write. The refusal case does not qualify successful S3 +acknowledgements; shared dispatch still refuses those until exact operation proofs +are connected. Existing assertions and required CG-06 cases remain unchanged. + +Seven signed S3 protocol operations and all ten regression commands pass, including +183 native execution/archive/controller/snapshot cases, all-target Cellar Clippy, +format/layout, dependency/docs/preflight/plan checks and 37 Python tests. System +Python 3.9 evidence hashing passes. The record under +`.cellar/cell-groups-evidence/CG-06-rustfs-controller/` binds 2561 +source hashes, copied executables, exact selections and output hashes, plus prior +qualification and retained setup/runner/Clippy failures. Independent source/hash, +selection and credential-exclusion audit passes. No credentials enter evidence. + +The initial health check reached an unrelated VM's port forward because Docker +allocated its port inside the guest. The helper now selects a free host port +explicitly; signed requests then verify the credential-bound service. The access +key uses SDK CI's 20-character profile. Source-bound corrections retain the system +Python hashing failure and the target-specific shared-fixture dead-code failure. + +Reflection: extend this executable RustFS path with positive S3 command, receipt +and enrollment cases while adding exact cell-version/result/current-route proof +ownership. Preserve captured identity, original deadlines and callback-once +behavior. Then bound unarchived-byte/disk admission, physically reclaim Raft files +and qualify fenced fresh-incarnation serving. These host-native voters and a +single local RustFS VM do not qualify container node limits, load, process-kill +restart, independent hosts or disaster recovery. Four original CG-06 cases and +its full gate remain pending; CG-08 lifetimes precede CG-07 public cutover. + ## CG-06 controller-owned archive publication Source: `7914177d6f17e3f6d647a39dc496122157283591`. CG-06 remains IN_PROGRESS. From dba41bda8d405264af2632c53c2d0d32894bdb8c Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:53:35 -0700 Subject: [PATCH 182/568] feat: prove local command S3 durability --- .../raft/cell_groups/archive/mod.rs | 2 + .../raft/cell_groups/archive/publication.rs | 22 + .../raft/cell_groups/archive/receipt.rs | 112 +++++ .../cell_groups/controller/archive/job.rs | 25 +- .../cell_groups/controller/archive/mod.rs | 59 ++- .../controller/archive/settlement.rs | 306 ++++++++++++++ .../raft/cell_groups/controller/mod.rs | 10 + .../raft/cell_groups/controller/requests.rs | 3 +- .../raft/cell_groups/controller/worker.rs | 1 + .../raft/cell_groups/dispatch/command/mod.rs | 101 ++++- .../raft/cell_groups/dispatch/mod.rs | 15 + .../raft/host_sql/cell_groups/mod.rs | 21 +- .../raft/host_sql/cell_groups/worker.rs | 10 +- .../native_cell_groups_archive_controller.rs | 81 ++++ .../support/cell_group_controller_archive.rs | 6 +- .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_s3_commands.rs | 396 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 5 + .../multi-raft/cell-groups/protocols.md | 46 +- docs/design/multi-raft/cell-groups/tasks.json | 5 +- tools/multi-raft/README.md | 14 +- tools/multi-raft/cell_groups.py | 13 +- 22 files changed, 1208 insertions(+), 47 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/receipt.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_commands.rs diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 7f28ac97f..0e8b2ef35 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -27,6 +27,7 @@ mod graph; mod objects; mod publication; pub use collection::{CellGroupArchiveCollectionLimits, CellGroupArchiveCollectionReport}; +pub(in crate::raft::cell_groups) use publication::CoverageUse; pub use publication::{CellGroupArchiveCoverage, PreparedCellGroupArchivePublication}; mod upload; pub use upload::{ @@ -37,6 +38,7 @@ pub use ownership::{ CellGroupArchiveCreation, CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchiveRegistry, CellGroupArchiveUploadTicket, PreparedCellGroupArchiveCreation, }; +pub(in crate::raft::cell_groups) mod receipt; mod replay; mod tail; mod tail_admission; diff --git a/crates/cellar/durability/raft/cell_groups/archive/publication.rs b/crates/cellar/durability/raft/cell_groups/archive/publication.rs index 661fc9011..12f7705aa 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/publication.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/publication.rs @@ -151,6 +151,21 @@ struct Coverage { verified: VerifiedCellGroupArchiveRecovery, } +pub(in crate::raft::cell_groups) struct CoverageUse { + hold: CellGroupArchiveHold, + coverage: std::sync::Weak, +} + +impl CoverageUse { + pub(in crate::raft::cell_groups) fn retains(&self, hold: &CellGroupArchiveHold) -> bool { + &self.hold == hold && self.coverage.strong_count() != 0 + } + + pub(in crate::raft::cell_groups) fn live(&self) -> bool { + self.coverage.strong_count() != 0 + } +} + /// Historical complete data coverage with an explicit remote hold. Clones keep /// local verification admission/pins. The remote hold is retired explicitly by /// its durable owner after joined use; this grants no quorum or serving right. @@ -158,6 +173,13 @@ struct Coverage { pub struct CellGroupArchiveCoverage(Arc); impl CellGroupArchiveCoverage { + pub(in crate::raft::cell_groups) fn retain_use(&self) -> CoverageUse { + CoverageUse { + hold: self.0.hold.clone(), + coverage: Arc::downgrade(&self.0), + } + } + pub fn hold(&self) -> &CellGroupArchiveHold { &self.0.hold } diff --git a/crates/cellar/durability/raft/cell_groups/archive/receipt.rs b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs new file mode 100644 index 000000000..38fb4a125 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs @@ -0,0 +1,112 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read the exact receipt from independently recovered, immutable archive bytes. +use super::*; +use crate::raft::cell_groups::{CellGroupApplyImages, CellImagePin}; +use crate::raft::execution::{CellCommittedReceipt, CommandRequest, ExecutionLimits}; +use std::sync::atomic::{AtomicBool, Ordering}; + +pub(in crate::raft::cell_groups) fn command( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + expected: &CellCommittedReceipt, + mut limits: ExecutionLimits, + deadline: u64, + cancelled: Arc, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 receipt verifier owner/route", + )); + } + if root.marker().position().index < expected.position.index { + return Ok(false); + } + let (leaf, pin) = root + .pin_cell(&expected.version.history.cell) + .ok_or(CellGroupArchiveError::Invalid("S3 receipt cell missing"))?; + if leaf.version.history != expected.version.history { + return Err(CellGroupArchiveError::Invalid("S3 receipt history")); + } + if leaf.version.sequence < expected.version.sequence { + return Ok(false); + } + if leaf.version == expected.version && leaf.witness != expected.witness { + return Err(CellGroupArchiveError::Invalid("S3 receipt decision")); + } + let images = store + .storage + .try_lock() + .map_err(|_| CellGroupArchiveError::Busy)? + .images + .clone(); + let database = match (images, pin) { + (CellGroupApplyImages::Materialized { services, .. }, CellImagePin::Materialized(pin)) => { + crate::storage::open_immutable_generation( + pin, + crate::storage::ImmutableQueryOptions { + services, + sqlite_vec: false, + }, + ) + } + ( + CellGroupApplyImages::Extents { + services, + cache_directory, + cache_pool, + cache, + .. + }, + CellImagePin::Extents(pin), + ) => { + let image = pin + .view() + .image(cache_directory, cache_pool, cache) + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + crate::storage::open_immutable_extent_image( + image, + crate::storage::ImmutableQueryOptions { + services, + sqlite_vec: false, + }, + ) + } + _ => return Err(CellGroupArchiveError::Invalid("S3 receipt image")), + } + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + database + .connection() + .progress_handler( + 1000, + Some(move || { + cancelled.load(Ordering::Acquire) || crate::asyncrt::mono_ms() >= deadline + }), + ) + .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; + // The pending output already owns the original bytes. Bound the additional + // row allocation before lookup; identity and the entire retained row match. + limits.max_result_bytes = expected.receipt.result().len(); + let request = CommandRequest { + id: expected.receipt.id, + operation: expected.receipt.operation.clone(), + contract_digest: expected.receipt.contract_digest, + request_digest: expected.receipt.request_digest, + durability: expected.receipt.minimum_durability, + retry_until_ms: expected.receipt.retry_until_ms, + }; + let receipt = crate::raft::receipts::lookup_cell_identity( + database.connection(), + &request, + &limits, + leaf.version.sequence.get(), + ) + .map_err(CellGroupArchiveError::Replay)?; + if receipt.as_ref() != Some(&expected.receipt) { + return Err(CellGroupArchiveError::Invalid("S3 receipt identity/result")); + } + store.require_live()?; + Ok(true) +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs index 4bafb4c6c..1b9da927e 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -9,6 +9,7 @@ use journal::{Journal, Pending, Record}; pub(super) struct Context { pub journal: Arc, pub registry: CellGroupArchiveRegistry, + pub uses: Arc>>, } pub(super) async fn open( @@ -56,6 +57,7 @@ pub(super) async fn open( Ok(Context { journal: Arc::new(journal), registry, + uses: Arc::new(Mutex::new(Vec::new())), }) } @@ -96,7 +98,7 @@ async fn select( execution: &ControllerExecution, record: &mut Record, coverage: CellGroupArchiveCoverage, -) -> Result { +) -> Result { let head = coverage.hold().clone(); if record.retire.is_some() { return Err(Error::Invalid("unfinished archive hold retirement")); @@ -104,12 +106,8 @@ async fn select( record.retire = record.head.replace(head); record.pending = Pending::None; store(context, execution, record).await?; - // No raw root escapes this proof's admission. The journal keeps the remote - // hold; future S3 outputs must obtain their own physical verification proof. - let marker = coverage.marker().clone(); - drop(coverage); retire(context, execution, record).await?; - Ok(marker) + Ok(coverage) } async fn retire( @@ -118,6 +116,17 @@ async fn retire( record: &mut Record, ) -> Result<()> { if let Some(previous) = &record.retire { + let retained = { + let mut uses = context + .uses + .lock() + .map_err(|_| Error::Invalid("archive use lock"))?; + uses.retain(|usage| usage.live()); + uses.iter().any(|usage| usage.retains(previous)) + }; + if retained { + return Err(CellGroupArchiveError::Busy.into()); + } context.registry.release(execution, previous).await?; record.retire = None; store(context, execution, record).await?; @@ -130,7 +139,7 @@ pub(super) async fn drive( registration: &CellGroupArchiveRegistration, runtime: &CellGroupRuntime, execution: &ControllerExecution, -) -> Result { +) -> Result { registration .directory .require_current_group_birth(®istration.birth) @@ -190,7 +199,7 @@ pub(super) async fn drive( receiver(registration)?, ) .await?; - return Ok(coverage.marker().clone()); + return Ok(coverage); } } let intent = context.registry.prepare_upload(execution).await?; diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs index f8d43db45..d14d85717 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -4,6 +4,7 @@ //! markers are observations, never S3 command settlement or serving authority. mod job; mod journal; +pub(in crate::raft::cell_groups::controller) mod settlement; use super::calls::Delivery; use super::*; @@ -193,6 +194,7 @@ struct Entry { dirty: bool, fatal: bool, retry_at: u64, + waiters: Vec, } pub(super) struct Completed { @@ -200,6 +202,7 @@ pub(super) struct Completed { context: Option, result: Result, observation: std::result::Result, + waiters: Vec, } pub(super) struct Manager { @@ -243,6 +246,7 @@ impl Manager { dirty: true, fatal: false, retry_at: 0, + waiters: Vec::new(), }, ); self.queue.push_back(group); @@ -265,6 +269,26 @@ impl Manager { } } + pub fn settle(&mut self, request: settlement::Request) { + let group = request.route.history; + let Some(entry) = self.entries.get_mut(&group) else { + request.fail(Arc::new(Error::Runtime(RuntimeError::NotReady))); + return; + }; + if entry.fatal || entry.registration.birth.route() != &request.route { + request.fail(Arc::new(Error::Invalid( + "S3 archive registration unavailable/changed", + ))); + return; + } + entry.waiters.push(request); + entry.dirty = true; + if !entry.queued { + entry.queued = true; + self.queue.push_back(group); + } + } + pub fn running(&self) -> bool { self.job.is_some() } @@ -302,18 +326,32 @@ impl Manager { let mut context = entry.context.clone(); let execution = self.execution.clone(); let runtime = runtime.clone(); + // Bound receipt SQL work per fair group turn, independently of the + // node's potentially larger total command admission. + let count = entry.waiters.len().min(32); + let mut waiters = entry.waiters.drain(..count).collect(); + settlement::prune(&mut waiters, now); self.job = Some(Box::pin(async move { let result = std::panic::AssertUnwindSafe(async { if context.is_none() { context = Some(job::open(®istration, &runtime, &execution).await?); } - job::drive( + let coverage = job::drive( context.as_ref().expect("opened archive context"), ®istration, &runtime, &execution, ) - .await + .await?; + waiters = settlement::prove( + std::mem::take(&mut waiters), + context.as_ref().expect("opened archive context"), + ®istration, + &coverage, + &execution, + ) + .await; + Ok(coverage.marker().clone()) }) .catch_unwind() .await @@ -331,6 +369,7 @@ impl Manager { context, result, observation, + waiters, } })); break; @@ -344,6 +383,8 @@ impl Manager { .get_mut(&completed.group) .expect("completed archive owner"); entry.context = completed.context; + entry.waiters.extend(completed.waiters); + settlement::prune(&mut entry.waiters, now); let mut statuses = shared .archives .lock() @@ -379,6 +420,17 @@ impl Manager { status.error = Some(Arc::new(error)); } } + if entry.fatal { + let error = status + .error + .clone() + .unwrap_or_else(|| Arc::new(Error::Stopped)); + for request in entry.waiters.drain(..) { + request.fail(error.clone()); + } + } else if !entry.waiters.is_empty() { + entry.dirty = true; + } if entry.dirty && !entry.fatal && !entry.queued { entry.queued = true; self.queue.push_back(completed.group); @@ -404,6 +456,9 @@ impl Manager { } self.execution.join().await; for entry in self.entries.values_mut() { + for request in entry.waiters.drain(..) { + request.fail(Arc::new(Error::Stopped)); + } entry.registration.store.close(); if let Some(reply) = entry.reply.take() { reply.send(Err(Arc::new(Error::Stopped))); diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs new file mode 100644 index 000000000..54da02781 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -0,0 +1,306 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Operation proofs share the archive lane; waiting never owns a SQL producer. +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{CellDispatchError, CellSqlReceipt}; +use crate::raft::execution::ExecutionLimits; +use cellar_logic::cell_group::GroupRoute; +use std::sync::{atomic::AtomicBool, Weak}; + +#[derive(Clone)] +pub(crate) struct Client(Weak); + +pub(crate) struct CommandProof { + _coverage: CellGroupArchiveCoverage, + // The exclusive journal lock survives response buffering and node close. + // A replacement owner cannot retire this hold while a consumer still uses it. + _journal: Arc, + _permit: Arc, + directory: RangeDirectory, + birth: CellGroupBirth, +} + +impl CommandProof { + pub(crate) async fn require_current(&self) -> std::result::Result<(), CellDispatchError> { + self.directory + .require_current_group_birth(&self.birth) + .await + .map_err(|error| CellDispatchError::Archive(Arc::new(Error::Binding(error.into())))) + } +} + +pub(crate) struct Settled { + pub receipt: Box, + pub proof: CommandProof, +} + +pub(in crate::raft::cell_groups::controller) struct Request { + pub route: GroupRoute, + receipt: Box, + limits: ExecutionLimits, + deadline: u64, + cancelled: Arc, + reply: oneshot::Sender>, + permit: Arc, +} + +struct Cancel(Arc); + +impl Drop for Cancel { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupController { + pub(crate) fn s3_client( + &self, + domain: &crate::asyncrt::DomainToken, + ) -> std::result::Result { + if !self.handle.0.domain.same_owner(domain) { + return Err(RuntimeError::Invalid("S3 controller domain".into())); + } + Ok(Client(Arc::downgrade(&self.handle.0))) + } +} + +impl Client { + pub(crate) fn require_runtime( + &self, + runtime: &CellGroupRuntime, + ) -> std::result::Result<(), CellDispatchError> { + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + if !shared.runtime.same_owner(runtime) { + return Err(CellDispatchError::Runtime(RuntimeError::Invalid( + "S3 runtime owner".into(), + ))); + } + Ok(()) + } + + pub(crate) fn require_group( + &self, + route: &GroupRoute, + ) -> std::result::Result<(), CellDispatchError> { + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let states = shared + .archives + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let state = states + .get(&route.history) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotReady))?; + if state.stopped { + return Err(state.error.clone().map_or( + CellDispatchError::Runtime(RuntimeError::Closed), + CellDispatchError::Archive, + )); + } + Ok(()) + } + + pub(crate) async fn command( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + receipt: Box, + limits: ExecutionLimits, + deadline: u64, + ) -> std::result::Result { + self.require_runtime(runtime)?; + self.require_group(&route)?; + let shared = self + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + let size = receipt + .receipt() + .receipt + .result() + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let permit = shared + .s3 + .acquire(size) + .map(Arc::new) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let receipt = receipt.retain_for_archive(permit.clone()); + let (reply, receiver) = oneshot::channel(); + let cancelled = Arc::new(AtomicBool::new(false)); + let _cancel = Cancel(cancelled.clone()); + let now = shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + shared + .requests + .try_send(requests::Request::S3(Box::new(Request { + route, + receipt, + limits, + deadline, + cancelled, + reply, + permit, + }))) + .map_err(|e| { + CellDispatchError::Runtime(match e { + tokio::sync::mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }) + })?; + let timer = shared + .domain + .sleep_until(deadline) + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let result = crate::asyncrt::select! { + result = receiver => result.map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + if shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + >= deadline + { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + result + } +} + +impl Request { + pub(super) fn fail(self, error: Arc) { + let _ = self.reply.send(Err(CellDispatchError::Archive(error))); + } + + fn expired(&self, now: u64) -> bool { + self.cancelled.load(Ordering::Acquire) || self.reply.is_closed() || now >= self.deadline + } +} + +pub(super) fn prune(requests: &mut Vec, now: u64) { + let mut live = Vec::new(); + for request in requests.drain(..) { + if now >= request.deadline { + let _ = request.reply.send(Err(CellDispatchError::Runtime( + RuntimeError::DeadlineExceeded, + ))); + } else if !request.expired(now) { + live.push(request); + } + } + *requests = live; +} + +pub(super) async fn prove( + requests: Vec, + context: &job::Context, + registration: &CellGroupArchiveRegistration, + coverage: &CellGroupArchiveCoverage, + execution: &ControllerExecution, +) -> Vec { + let mut waiting = Vec::new(); + { + let mut uses = context.uses.lock().unwrap_or_else(|e| e.into_inner()); + uses.retain(|usage| usage.live()); + uses.push(coverage.retain_use()); + } + for request in requests { + let now = execution + .run(async { crate::asyncrt::mono_ms() }) + .await + .unwrap_or(u64::MAX); + if request.expired(now) { + continue; + } + if request.route != *coverage.marker().route() { + request.fail(Arc::new(Error::Invalid("S3 command route"))); + continue; + } + let store = registration.store.clone(); + let physical = coverage.clone(); + let read = execution.blocking(move || { + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + crate::raft::cell_groups::archive::receipt::command( + &physical, + &store, + request.receipt.receipt(), + request.limits, + request.deadline, + request.cancelled.clone(), + ) + })) + .unwrap_or_else(|panic| { + Err(CellGroupArchiveError::Replay( + crate::raft::execution::storage(crate::native_worker::panic_error(panic)), + )) + }); + (request, result) + }); + // A refused submission drops the closure and its owned request. Entered + // closures retain both until the controller-specific physical join. + let Ok(read) = read else { + continue; + }; + let Ok((request, result)) = read.await else { + continue; + }; + let now = execution + .run(async { crate::asyncrt::mono_ms() }) + .await + .unwrap_or(u64::MAX); + if request.expired(now) { + continue; + } + match result { + Ok(false) | Err(CellGroupArchiveError::Busy) => waiting.push(request), + Err(error) => request.fail(Arc::new(Error::Archive(error))), + Ok(true) => { + if let Err(error) = registration + .directory + .require_current_group_birth(®istration.birth) + .await + { + request.fail(Arc::new(Error::Binding(error.into()))); + continue; + } + let Request { + receipt, + reply, + permit, + .. + } = request; + let _ = reply.send(Ok(Settled { + receipt, + proof: CommandProof { + _coverage: coverage.clone(), + _journal: context.journal.clone(), + _permit: permit, + directory: registration.directory.clone(), + birth: registration.birth.clone(), + }, + })); + } + } + } + waiting +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 94969f927..44fd6f95c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -4,6 +4,7 @@ //! Admitted command ownership survives response cancellation and expiry. mod archive; +pub(crate) use archive::settlement::{Client as S3Client, CommandProof as S3CommandProof}; mod calls; pub use archive::{ CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, @@ -105,6 +106,10 @@ pub struct CellGroupControllerUsage { pub retained_preparations: usize, pub preparation_bytes: usize, pub provision_waiters: usize, + /// Independent archive-settlement admission, bounded by max_commands and + /// max_command_bytes. This lane cannot recursively consume command slots. + pub retained_s3: usize, + pub s3_bytes: usize, } struct Shared { @@ -117,6 +122,7 @@ struct Shared { provisioning: Mutex>>, requests: tokio::sync::mpsc::Sender, archives: Mutex>, + s3: Arc, active_apply: AtomicUsize, pending_apply: AtomicUsize, applied: AtomicU64, @@ -242,6 +248,7 @@ impl CellGroupController { provisioning: Mutex::new(None), requests, archives: Mutex::new(BTreeMap::new()), + s3: commands::budget(limits), active_apply: AtomicUsize::new(0), pending_apply: AtomicUsize::new(0), applied: AtomicU64::new(0), @@ -308,6 +315,7 @@ impl CellGroupController { pub fn usage(&self) -> CellGroupControllerUsage { let shared = &self.handle.0; let (command_bytes, retained_commands) = shared.commands.usage(); + let (s3_bytes, retained_s3) = shared.s3.usage(); let (preparation_bytes, retained_preparations, provision_waiters) = shared .provisioning .lock() @@ -328,6 +336,8 @@ impl CellGroupController { quarantined_groups: shared.groups.quarantined(), command_bytes, retained_commands, + s3_bytes, + retained_s3, active_apply: shared.active_apply.load(Ordering::Acquire), pending_apply: shared.pending_apply.load(Ordering::Acquire), applied: shared.applied.load(Ordering::Relaxed), diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index 7a72767b6..5919024e4 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -6,6 +6,7 @@ use super::*; pub(super) enum Request { Archive(Box), + S3(Box), Registration(Box), Provision(Box), Command(Box), @@ -50,7 +51,7 @@ pub(super) async fn drive( progress: tokio::sync::watch::Receiver<()>, ) -> Completed { match request { - Request::Registration(_) | Request::Provision(_) | Request::Archive(_) => { + Request::Registration(_) | Request::Provision(_) | Request::Archive(_) | Request::S3(_) => { unreachable!("registration has its own physical join owner") } Request::Command(request) => Completed::Command(Box::new( diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 794d4bb88..f1870cf3a 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -199,6 +199,7 @@ impl Worker { Some(request) = self.requests.recv() => { match request { requests::Request::Archive(request) => self.archives.request(*request, &self.shared), + requests::Request::S3(request) => self.archives.settle(*request), requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared), requests::Request::Registration(request) => { let group = request.registration.applier.history(); diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index d0b22cc2e..b2af0cbbb 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -62,6 +62,7 @@ pub enum CellCommandSettlement { pub struct CellCommandResponse { receipt: Box, token: CellVersionToken, + s3: Option, } impl CellCommandResponse { @@ -74,7 +75,11 @@ impl CellCommandResponse { } pub fn durability(&self) -> Durability { - Durability::Quorum + if self.s3.is_some() { + Durability::S3 + } else { + Durability::Quorum + } } } @@ -131,11 +136,10 @@ impl CellGroupDispatcher { cause: Box::new(cause), }; let started = self.domain.mono_ms().map_err(|_| error(closed()))?; - // Shared-group archival coverage is not integrated yet. Never silently - // lower either the request or the immutable namespace minimum. - if request.durability != Durability::Quorum - || self.directory.namespace_config().minimum_durability != Durability::Quorum - { + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + if request.durability == Durability::S3 && self.s3.is_none() { return Err(error(CellDispatchError::Execution( ExecutionError::Refused(CommandFailure::UnsupportedCapability), ))); @@ -361,6 +365,18 @@ impl CellCommandInvocation { if authority.route() != attempt.route() { return Err(CellDispatchError::Invalid("destination route binding")); } + if self.request.durability == Durability::S3 { + self.dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 command owner"))? + .require_runtime(runtime)?; + self.dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 command owner"))? + .require_group(attempt.route())?; + } let (proposal, preview) = self .dispatcher .domain @@ -433,7 +449,6 @@ impl CellCommandInvocation { } CellSqlOutcome::Receipt(receipt) => { let response = self.response(receipt).await?; - self.origin = None; Ok(CellCommandPreparation::Committed(response)) } _ => Err(CellDispatchError::Invalid("command preparation response")), @@ -597,7 +612,6 @@ impl CellCommandInvocation { match origin.sql.reconcile(origin.proposal, authority).await { Ok(CellSqlReconciliation::Committed(receipt)) => { let response = self.response(receipt).await?; - self.origin = None; Ok(CellCommandSettlement::Committed(response)) } Ok(CellSqlReconciliation::NotCommitted) => { @@ -628,19 +642,68 @@ impl CellCommandInvocation { } async fn response( - &self, + &mut self, receipt: Box, ) -> Result { if receipt.receipt().receipt.command() != self.request.id + || receipt.receipt().receipt.operation != self.request.contract.operation + || receipt.receipt().receipt.contract_digest != self.request.contract.digest + || receipt.receipt().receipt.request_digest + != command_request_digest(&self.request.input) || receipt.receipt().version.history != self.cell { return Err(CellDispatchError::Invalid("command receipt binding")); } - if receipt.receipt().receipt.minimum_durability() != Durability::Quorum { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } + // SQL has resolved the producer. Archive delay, failure or cancellation + // must not retain an unresolved origin or quarantine this healthy group. + let mut origin = self + .origin + .take() + .ok_or(CellDispatchError::Invalid("resolved command origin"))?; + let required = self + .request + .durability + .max(receipt.receipt().receipt.minimum_durability()); + let (receipt, s3, authority) = if required == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + let limits = match &origin.options { + CellSqlOptions::Materialized(o) => o.limits, + CellSqlOptions::Extents(o) => o.limits, + }; + // The worker has closed this producer. Its preparation must not + // keep a duplicate SQL admission lease during the archive wait. + origin.prepared = None; + let settled = owner + .command( + &origin.runtime, + origin.route.clone(), + receipt, + limits, + self.session.deadline_ms(), + ) + .await?; + // Archive coverage is historical data durability. Current catalog and + // leader authority remain independent requirements for this output. + let authority = origin + .runtime + .execution_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &origin.route { + return Err(CellDispatchError::Invalid("S3 response current route")); + } + settled.proof.require_current().await?; + authority.check().map_err(CellDispatchError::Runtime)?; + (settled.receipt, Some(settled.proof), Some(authority)) + } else { + (receipt, None, None) + }; let now = self .dispatcher .domain @@ -656,7 +719,10 @@ impl CellCommandInvocation { now, ) .map_err(CellDispatchError::Version)?; - Ok(CellCommandResponse { receipt, token }) + if let Some(authority) = authority { + authority.check().map_err(CellDispatchError::Runtime)?; + } + Ok(CellCommandResponse { receipt, token, s3 }) } } @@ -675,7 +741,10 @@ fn fingerprint( hash.update(request.contract.operation.as_bytes()); hash.update(request.contract.digest); hash.update(command_request_digest(&request.input)); - hash.update([1]); // Quorum is the only supported request/minimum until CG-06. + hash.update([match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }]); hash.update(request.retry_until_ms.to_be_bytes()); hash.update(inputs.now_unix_millis.to_be_bytes()); hash.update(inputs.random); diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 1742c4875..b6d51ac5e 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -64,6 +64,7 @@ pub enum CellDispatchError { Execution(ExecutionError), Version(VersionError), Transport(crate::raft::transport::TransportError), + Archive(Arc), } impl fmt::Display for CellDispatchError { @@ -85,6 +86,7 @@ pub struct CellGroupDispatcher { limits: CellDispatchLimits, budget: Arc, routes: Arc, + s3: Option, } impl CellGroupDispatcher { @@ -141,9 +143,22 @@ impl CellGroupDispatcher { domain, limits, budget: Budget::new(limits.retained_bytes, limits.max_invocations), + s3: None, }) } + /// Bind local command S3 settlement to this node's bounded archive owner. + /// The destination group must already have an archive registration. Receipt + /// lookup, enrollment and remote response settlement have separate gates. + pub fn with_s3(mut self, controller: &CellGroupController) -> Result { + self.s3 = Some( + controller + .s3_client(&self.domain) + .map_err(CellDispatchError::Runtime)?, + ); + Ok(self) + } + /// Retained namespace range hints, never per-cell execution or authority. pub fn route_cache_usage(&self) -> Result { self.routes.usage().map_err(catalog_failure) diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 2f68b9931..ce2d9cf93 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -238,10 +238,29 @@ pub struct CellSqlRejection { pub struct CellSqlReceipt { receipt: CellCommittedReceipt, - _bytes: Arc, + _retention: ReceiptRetention, +} + +enum ReceiptRetention { + Sql { + _bytes: Arc, + }, + Archive { + _permit: Arc, + }, } impl CellSqlReceipt { + /// Only a resolved output can transfer its bytes to an independently admitted + /// archive owner. Acquire that reservation before releasing SQL retention. + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc, + ) -> Box { + self._retention = ReceiptRetention::Archive { _permit: permit }; + self + } + pub fn receipt(&self) -> &CellCommittedReceipt { &self.receipt } diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index c136f7712..b9692ee1b 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -227,7 +227,7 @@ impl Cells { Bytes::shrink(&mut bytes, retained)?; Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { receipt: *receipt, - _bytes: bytes, + _retention: ReceiptRetention::Sql { _bytes: bytes }, }))) } Err(error) => { @@ -375,7 +375,9 @@ impl Cells { pending.producer.close()?; Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { receipt, - _bytes: pending.prepared._bytes, + _retention: ReceiptRetention::Sql { + _bytes: pending.prepared._bytes, + }, }))) } Action::Reconcile(proposal, authority) => { @@ -396,7 +398,9 @@ impl Cells { CellReconciliation::Committed(receipt) => { CellSqlReconciliation::Committed(Box::new(CellSqlReceipt { receipt: *receipt, - _bytes: pending.prepared._bytes, + _retention: ReceiptRetention::Sql { + _bytes: pending.prepared._bytes, + }, })) } CellReconciliation::NotCommitted => CellSqlReconciliation::NotCommitted, diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index c595d4ad3..f02613af8 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -20,6 +20,87 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_command_requires_the_sibling_archive_dependencies() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_missing_sibling( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_command_requires_the_sibling_archive_dependencies() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_missing_sibling( + rooted, + rustfs::store("s3-sibling", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_commands_require_exact_archived_results() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_commands( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_command_deadlines_do_not_quarantine_committed_sql() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_commands_require_exact_archived_results() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_commands( + rooted, + rustfs::store("s3-commands", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_command_deadlines_do_not_quarantine_committed_sql() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_deadline( + rooted, + rustfs::store(if cancel { "s3-cancel" } else { "s3-deadline" }, rooted)?, + cancel, + )) + .await?; + } + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn controller_archive_publishes_stable_cuts_while_quorum_advances() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 95701934b..86b1b22ea 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -8,7 +8,7 @@ use std::sync::atomic::Ordering; #[path = "cell_group_archive_store.rs"] #[allow(dead_code)] -mod archive_store; +pub(super) mod archive_store; async fn verifier( fixture: &RuntimeCluster, @@ -75,7 +75,7 @@ async fn verifier( .await? } -async fn registration( +pub(super) async fn registration( fixture: &RuntimeCluster, route: &GroupRoute, rooted: bool, @@ -142,7 +142,7 @@ async fn registration( )) } -async fn published( +pub(super) async fn published( controller: &CellGroupController, group: GroupHistory, index: LogIndex, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 37acf4868..ddd528748 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -813,6 +813,8 @@ pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; pub(super) use sql::{owned_command_failure, owned_commands}; +#[allow(unused_imports)] +pub(super) use sql::{s3_commands, s3_deadline, s3_missing_sibling}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs new file mode 100644 index 000000000..37a09876c --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -0,0 +1,396 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Positive command output proofs on the actual controller and archive lane. +use super::*; +use futures_util::TryStreamExt; +use object_store::{path::Path as ObjectPath, ObjectStore}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +struct Harness { + fixture: RuntimeCluster, + route: GroupRoute, + cell: CellHistory, + controllers: Vec, + dispatcher: CellGroupDispatcher, + class: Arc, + archive: CellGroupArchiveRegistration, + remote: Arc, + calls: Arc, + retry_until: u64, +} + +impl Harness { + async fn new( + rooted: bool, + store: Arc, + max_commands: usize, + ) -> Result { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + }, + )?); + let (mut fixture, route, cell, _) = + Box::pin(registered::setup_store(rooted, &class, store)).await?; + let (archive, remote) = archive_controller::registration(&fixture, &route, rooted).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_commands, + ..controller::limits() + }, + ) + .await?; + controllers[0].archive(archive.clone())?.response().await?; + archive_controller::published( + &controllers[0], + route.history, + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index, + ) + .await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&controllers[0])?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms() as u64 + 60_000 }) + .await?; + Ok(Self { + fixture, + route, + cell, + controllers, + dispatcher, + class, + archive, + remote, + calls, + retry_until, + }) + } + + async fn command( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + timeout: Duration, + ) -> Result { + let invocation = self + .dispatcher + .command( + self.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + retry_until_ms: self.retry_until, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?; + Ok( + self.controllers[0] + .command(invocation, self.fixture.cell_options(&self.cell).await?)?, + ) + } + + async fn waiting(&self) -> Result<()> { + controller::wait_for(async || { + Ok(self.controllers[0].usage().retained_s3 == 1 + && self.fixture.appliers[&(0, self.route.history)] + .0 + .usage()? + .active_cells + == 0) + }) + .await + } + + async fn close(self) -> Result<()> { + for controller in &self.controllers { + controller.close().await?; + } + drop(self.dispatcher); + drop(self.archive); + self.fixture.close().await + } +} + +pub(crate) async fn s3_commands( + rooted: bool, + store: Arc, +) -> Result<()> { + let h = Harness::new(rooted, store, 2).await?; + // A controller in the same execution domain still cannot lend another + // runtime's archive authority to this producer. + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let invocation = foreign + .command( + h.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"first".to_vec(), + durability: Durability::S3, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let error = h.controllers[0] + .command(invocation, h.fixture.cell_options(&h.cell).await?)? + .response() + .await + .err() + .context("foreign S3 runtime refused")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + assert_eq!(h.calls.load(Ordering::SeqCst), 0); + h.remote.outage.store(true, Ordering::Release); + let call = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await?; + h.waiting().await?; + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + // The same cell commits its successor during the first S3 wait. The proof + // must read the original receipt from that later independently replayed image. + let next = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + assert_eq!(next.receipt().receipt.result(), b"second"); + drop(next); + let gate = h.remote.hold_publication(); + h.remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut waiting = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = waiting.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.receipt().receipt.result(), b"first"); + assert_eq!(response.receipt().version.sequence.get(), 2); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + // A Quorum duplicate still honors the stored S3 minimum. A retained response + // keeps its physical proof and remote hold until the final consumer drops it. + let mut duplicate = Box::pin( + h.command(1, b"first", Durability::Quorum, Duration::from_secs(15)) + .await? + .response(), + ); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 2)).await?; + assert!( + tokio::time::timeout(Duration::from_millis(30), duplicate.as_mut()) + .await + .is_err() + ); + drop(response); + let response = duplicate.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.receipt().receipt.result(), b"first"); + assert_eq!(response.receipt().version.sequence.get(), 3); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(response); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + h.close().await +} + +pub(crate) async fn s3_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + h.remote.outage.store(true, Ordering::Release); + let call = h + .command(1, b"first", Durability::S3, Duration::from_millis(800)) + .await?; + h.waiting().await?; + if cancel { + drop(call); + } else { + let error = call.response().await.err().context("S3 deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_commands == 0 + && h.controllers[0].usage().retained_s3 == 0) + }) + .await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .active_cells, + 0 + ); + let next = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(next.receipt().receipt.result(), b"second"); + drop(next); + h.remote.outage.store(false, Ordering::Release); + let response = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.receipt().receipt.result(), b"first"); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(response); + h.close().await +} + +pub(crate) async fn s3_missing_sibling(rooted: bool, store: Arc) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let result = h + .command(1, b"first", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + let index = result.receipt().position.index; + drop(result); + archive_controller::published(&h.controllers[0], h.route.history, index).await?; + let marker = h.controllers[0] + .archive_status(h.route.history)? + .context("published archive")? + .published + .context("archive marker")?; + let objects = h.remote.list(None).try_collect::>().await?; + let mut missing = None; + for object in objects + .iter() + .filter(|object| object.location.as_ref().contains("/indexes/")) + { + let value: serde_json::Value = + serde_json::from_slice(&h.remote.get(&object.location).await?.bytes().await?)?; + let Some(chunks) = value.get("chunks").and_then(|v| v.as_array()) else { + continue; + }; + let selected: GroupManifest = serde_json::from_value(value["marker"].clone())?; + if selected != marker { + continue; + } + let prefix = object + .location + .as_ref() + .split_once("/generations/") + .context("generation prefix")? + .0; + let path = |chunk: &serde_json::Value| -> Result { + let object = &chunk["object"]; + let digest: Vec = serde_json::from_value(object["digest"].clone())?; + Ok(ObjectPath::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + object["generation"], + object["length"], + hex::encode(digest) + ))) + }; + let a = chunks + .iter() + .filter(|chunk| chunk["part"]["cell"] == "registered-a") + .map(&path) + .collect::>>()?; + for chunk in chunks + .iter() + .filter(|chunk| chunk["part"]["cell"] == "registered-b") + { + let candidate = path(chunk)?; + if !a.contains(&candidate) { + missing = Some(candidate); + break; + } + } + if missing.is_some() { + break; + } + } + let missing = missing.context("B dependency outside A closure")?; + h.remote.inner.delete(&missing).await?; + // A still has its committed receipt. A cached published marker must not + // acknowledge it while any other cell's selected dependency is missing. + let error = h + .command(1, b"first", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("incomplete group must refuse S3")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Archive(_) + )); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let result = h + .command(2, b"second", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(result.receipt().receipt.result(), b"second"); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + drop(result); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 83cbb1139..aeb290a46 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -529,11 +529,16 @@ pub(crate) use command_dispatch::commands as routed_commands; #[path = "cell_group_controller_archive.rs"] #[allow(dead_code, unused_imports)] mod archive_controller; +#[path = "cell_group_s3_commands.rs"] +#[allow(dead_code)] +mod s3_commands; #[path = "cell_group_s3_response.rs"] // Selected by the archive-controller target; sibling targets share this SQL fixture. #[allow(dead_code)] mod s3_response; #[allow(unused_imports)] +pub(crate) use s3_commands::{s3_commands, s3_deadline, s3_missing_sibling}; +#[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] mod controller; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 642cfe9dd..60f5f1542 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -714,9 +714,51 @@ verifier stores and releases journal locks before closing runtime/SQL ownership. Fresh verifier stores are required on restart. No unrelated domain work is joined through this archive-specific drain. -Actual S3 command/enrollment/receipt settlement, explicit unarchived-byte/disk +### Local command S3 response gate + +`CellGroupDispatcher::with_s3` binds local command settlement to a controller in +the same execution domain. Execution also requires its exact runtime owner and +an active archive registration for the selected group/route. Effective durability +is the maximum of the request, namespace minimum and existing receipt minimum. +Receipt lookup, enrollment and remote S3 response paths retain their refusal gates +until their own output proofs are integrated. + +A committed SQL receipt resolves its producer before any archive wait. The owner +acquires independent S3 byte/item admission before transferring output retention +from SQL; it releases the resolved preparation's duplicate SQL permit. The S3 +lane uses the node's `max_commands` and `max_command_bytes` as separate bounds, +including fixed metadata and additional receipt-read bytes. It never recursively +acquires ordinary command slots. Held or uncertain S3 work cannot keep the SQL +producer unresolved or quarantine an otherwise healthy group. + +The existing fair archive lane takes at most 32 pending command proofs per group +turn. It publishes or reopens a protected complete cut, then reads each exact +receipt from that independently recovered immutable image. Coverage must contain +the same cell history at or beyond the selected committed version/position. An +equal version must carry the same decision witness. All stored receipt fields, +including operation, contract/input digests, result, durability minimum and retry +expiry, must match. A later image can prove an earlier retained receipt; a group +index or sibling progress cannot replace that read. Missing/corrupt dependencies +anywhere in the selected group refuse the output. No callback runs during proof. + +The response rechecks the selected catalog birth/route and live leader authority +before issuing its version token. Original deadlines and captured inputs remain +unchanged. Deadline/caller cancellation does not release an entered physical read; +its cancelled flag interrupts SQL, and the archive-specific owner joins the closure. +Pending requests and unconsumed responses retain bounded S3 admission. + +A successful response retains the complete physical coverage, its pool admission +and the exclusive journal lock. The journal owner tracks weak uses and refuses +predecessor-hold retirement while a response still retains that coverage. A +replacement owner cannot acquire that journal lock before the response drops. +Idle registrations do not retain replay workspaces. Active consumers can occupy +those shared workspaces and backpressure subsequent archival until consumption; +this is explicit pool admission, without an unaccounted metadata-only proof cache. + +S3 receipt-lookup/enrollment/remote responses, explicit unarchived-byte/disk admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, -RustFS/load and process-kill qualification remain required gates. +load and process-kill qualification remain required gates. This local command +boundary is not the complete CG-06 or public SDK/fleet cutover gate. ## Split and merge diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 3c5dd8887..f4b3adf13 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1761,7 +1761,10 @@ "crates/cellar/tests/support/cell_group_controller_archive.rs", "crates/cellar/tests/support/cell_group_rustfs.rs", "crates/cellar/tests/support/cell_group_s3_response.rs", - "tools/multi-raft/cell_groups.py" + "tools/multi-raft/cell_groups.py", + "crates/cellar/durability/raft/cell_groups/archive/receipt.rs", + "crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs", + "crates/cellar/tests/support/cell_group_s3_commands.rs" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index faa34b787..220d2e356 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,16 +71,20 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs three exact ignored RustFS cases, each in materialized and extent modes: +It runs six exact ignored RustFS cases, each in materialized and extent modes: stable publication with continuing quorum, uncertain-publication reconciliation -with joined close, and refusal of unproved S3 responses. Catalog setup, archive +with joined close, refusal of unproved S3 responses, positive local command S3 +settlement, deadline/cancellation isolation, and refusal when a sibling dependency +is missing. Catalog setup, archive objects, conditional registry updates and independent recovery use the actual S3 client. Fault injection surrounds that client; lost publication responses are injected after the real conditional write commits. No memory fallback exists. -The response case currently verifies that command, receipt and enrollment -requests cannot claim S3 durability, including when required by the namespace. -Successful S3 response settlement still needs operation-specific proofs. The +The refusal case verifies that an unbound dispatcher cannot claim S3 durability, +including a namespace minimum. The positive command cases use an explicit +controller binding, exact receipts from later archived images, retained proof +admission and fresh routing/leader checks. Receipt lookup, enrollment and remote +S3 responses remain unqualified. The runner does not qualify local log file purge, fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 035568ecd..9488ca50e 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 # Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -"""Qualify shared-group archive setup and response refusals against local RustFS. +"""Qualify shared-group archive setup and local command responses against local RustFS. Uses only the recorded owned backend. Each invocation owns a new evidence directory and remote prefix. Credentials are passed through the child environment, -never arguments or evidence. This does not qualify successful S3 acknowledgements. +never arguments or evidence. Positive command acknowledgements require the explicit controller binding. """ import argparse @@ -28,6 +28,9 @@ "rustfs_controller_archive_publishes_stable_cuts", "rustfs_controller_archive_reconciles_exact_intent_and_joins_close", "rustfs_unproved_s3_responses_refuse", + "rustfs_s3_commands_require_exact_archived_results", + "rustfs_s3_command_deadlines_do_not_quarantine_committed_sql", + "rustfs_s3_command_requires_the_sibling_archive_dependencies", ) @@ -102,11 +105,11 @@ def main(): out.mkdir(mode=0o700, parents=True, exist_ok=False) prefix = "cell-groups/" + secrets.token_hex(16) record = { - "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and unproved-S3 refusal", + "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command S3 settlement", "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " - "No successful S3 command/receipt/enrollment acknowledgements, byte/disk outage bound, " + "No S3 receipt-lookup/enrollment or remote command acknowledgements, byte/disk outage bound, " "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", } @@ -137,7 +140,7 @@ def save(): cargo = ["cargo", "+1.98.1", "test", "-p", "cellar", "--locked", "--test", TARGET, "--"] for number, selection in enumerate(("--list", *CASES)): argv = cargo + ([selection] if number == 0 else [selection, "--exact", "--ignored", "--test-threads=1"]) - print(f"START {number + 1}/4: {selection}", flush=True) + print(f"START {number + 1}/{len(CASES) + 1}: {selection}", flush=True) started = time.monotonic() result = subprocess.run(argv, cwd=ROOT, env=env, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) From 2b524f2a49978c3ac32acfbc4e08792a69f297fb Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 12:55:35 -0700 Subject: [PATCH 183/568] fix: box retained S3 response proofs --- .../raft/cell_groups/controller/archive/settlement.rs | 6 +++--- .../durability/raft/cell_groups/dispatch/command/mod.rs | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs index 54da02781..8395fcdce 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -32,7 +32,7 @@ impl CommandProof { pub(crate) struct Settled { pub receipt: Box, - pub proof: CommandProof, + pub proof: Box, } pub(in crate::raft::cell_groups::controller) struct Request { @@ -291,13 +291,13 @@ pub(super) async fn prove( } = request; let _ = reply.send(Ok(Settled { receipt, - proof: CommandProof { + proof: Box::new(CommandProof { _coverage: coverage.clone(), _journal: context.journal.clone(), _permit: permit, directory: registration.directory.clone(), birth: registration.birth.clone(), - }, + }), })); } } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index b2af0cbbb..01b8e86d2 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -62,7 +62,7 @@ pub enum CellCommandSettlement { pub struct CellCommandResponse { receipt: Box, token: CellVersionToken, - s3: Option, + s3: Option>, } impl CellCommandResponse { From db78d771b5f3336bdebff4b5336e0d94067f08d4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 13:12:28 -0700 Subject: [PATCH 184/568] fix: share S3 route authority retention --- .../cell_groups/controller/archive/job.rs | 19 ++++++++++++++++++ .../controller/archive/settlement.rs | 20 +++++++------------ .../multi-raft/cell-groups/protocols.md | 3 ++- 3 files changed, 28 insertions(+), 14 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs index 1b9da927e..1935e64ac 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -10,6 +10,21 @@ pub(super) struct Context { pub journal: Arc, pub registry: CellGroupArchiveRegistry, pub uses: Arc>>, + pub authority: Arc, +} + +pub(super) struct Authority { + directory: RangeDirectory, + birth: CellGroupBirth, +} + +impl Authority { + pub(super) async fn require_current(&self) -> Result<()> { + self.directory + .require_current_group_birth(&self.birth) + .await + .map_err(|error| Error::Binding(error.into())) + } } pub(super) async fn open( @@ -58,6 +73,10 @@ pub(super) async fn open( journal: Arc::new(journal), registry, uses: Arc::new(Mutex::new(Vec::new())), + authority: Arc::new(Authority { + directory: registration.directory.clone(), + birth: registration.birth.clone(), + }), }) } diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs index 8395fcdce..5b0ecd999 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -17,16 +17,15 @@ pub(crate) struct CommandProof { // A replacement owner cannot retire this hold while a consumer still uses it. _journal: Arc, _permit: Arc, - directory: RangeDirectory, - birth: CellGroupBirth, + authority: Arc, } impl CommandProof { pub(crate) async fn require_current(&self) -> std::result::Result<(), CellDispatchError> { - self.directory - .require_current_group_birth(&self.birth) + self.authority + .require_current() .await - .map_err(|error| CellDispatchError::Archive(Arc::new(Error::Binding(error.into())))) + .map_err(|error| CellDispatchError::Archive(Arc::new(error))) } } @@ -275,12 +274,8 @@ pub(super) async fn prove( Ok(false) | Err(CellGroupArchiveError::Busy) => waiting.push(request), Err(error) => request.fail(Arc::new(Error::Archive(error))), Ok(true) => { - if let Err(error) = registration - .directory - .require_current_group_birth(®istration.birth) - .await - { - request.fail(Arc::new(Error::Binding(error.into()))); + if let Err(error) = context.authority.require_current().await { + request.fail(Arc::new(error)); continue; } let Request { @@ -295,8 +290,7 @@ pub(super) async fn prove( _coverage: coverage.clone(), _journal: context.journal.clone(), _permit: permit, - directory: registration.directory.clone(), - birth: registration.birth.clone(), + authority: context.authority.clone(), }), })); } diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 60f5f1542..78c3a0068 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -748,7 +748,8 @@ its cancelled flag interrupts SQL, and the archive-specific owner joins the clos Pending requests and unconsumed responses retain bounded S3 admission. A successful response retains the complete physical coverage, its pool admission -and the exclusive journal lock. The journal owner tracks weak uses and refuses +and the exclusive journal lock. Immutable route authority is shared per group, +so membership records are not copied into every response. The journal owner tracks weak uses and refuses predecessor-hold retirement while a response still retains that coverage. A replacement owner cannot acquire that journal lock before the response drops. Idle registrations do not retain replay workspaces. Active consumers can occupy From 2e04d7509e2f9436d573adb656dfb911e07c20e2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 13:27:24 -0700 Subject: [PATCH 185/568] test: enforce namespace S3 response minimums --- .../native_cell_groups_archive_controller.rs | 25 +++++++++++ .../support/cell_group_controller_archive.rs | 7 +--- .../support/cell_group_engine_fixture.rs | 27 ++++++++---- .../tests/support/cell_group_runtime.rs | 2 +- .../tests/support/cell_group_s3_commands.rs | 41 ++++++++++++++++++- crates/cellar/tests/support/cell_group_sql.rs | 2 +- .../support/cell_group_sql_registered.rs | 12 +++++- tools/multi-raft/README.md | 10 ++--- tools/multi-raft/cell_groups.py | 1 + 9 files changed, 103 insertions(+), 24 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index f02613af8..2435d67a8 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -20,6 +20,31 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_namespace_minimum_cannot_be_downgraded() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_namespace_minimum( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_namespace_minimum_cannot_be_downgraded() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_namespace_minimum( + rooted, + rustfs::store("s3-minimum", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn s3_command_requires_the_sibling_archive_dependencies() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 86b1b22ea..00f41ce19 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -109,12 +109,7 @@ pub(super) async fn registration( let namespace = fixture .cluster .domain - .run(catalog.existing_namespace(NamespaceConfig::new( - route.history.cluster, - route.history.namespace, - cellar_logic::NamespaceMode::Raft, - route.schema_digest, - ))) + .run(catalog.existing_namespace(fixture.cluster.directory.namespace_config().clone())) .await??; let directory = RangeDirectory::new(namespace, Default::default())?; let birth = fixture diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 563f4b483..e4d7d510f 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -88,6 +88,17 @@ impl Cluster { filesystem: Arc, schema: [u8; 32], store: Arc, + ) -> Result { + Self::with_store_minimum(options, root, filesystem, schema, store, Durability::Quorum).await + } + + async fn with_store_minimum( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { @@ -97,14 +108,14 @@ impl Cluster { incarnation: IncarnationId::new([4; 16])?, }; let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; - let namespace = domain - .run(catalog.open_namespace(NamespaceConfig::new( - history.cluster, - history.namespace, - cellar_logic::NamespaceMode::Raft, - schema, - ))) - .await??; + let mut config = NamespaceConfig::new( + history.cluster, + history.namespace, + cellar_logic::NamespaceMode::Raft, + schema, + ); + config.minimum_durability = minimum; + let namespace = domain.run(catalog.open_namespace(config)).await??; let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; let request = directory.prepare_group_birth(CellGroupBirthSpec { history, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index ddd528748..068fed243 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -814,7 +814,7 @@ pub(super) use sql::{ }; pub(super) use sql::{owned_command_failure, owned_commands}; #[allow(unused_imports)] -pub(super) use sql::{s3_commands, s3_deadline, s3_missing_sibling}; +pub(super) use sql::{s3_commands, s3_deadline, s3_missing_sibling, s3_namespace_minimum}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 37a09876c..cc9f6485f 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -24,6 +24,15 @@ impl Harness { rooted: bool, store: Arc, max_commands: usize, + ) -> Result { + Self::with_minimum(rooted, store, max_commands, Durability::Quorum).await + } + + async fn with_minimum( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -36,8 +45,10 @@ impl Harness { Ok(input.to_vec()) }, )?); - let (mut fixture, route, cell, _) = - Box::pin(registered::setup_store(rooted, &class, store)).await?; + let (mut fixture, route, cell, _) = Box::pin(registered::setup_store_minimum( + rooted, &class, store, minimum, + )) + .await?; let (archive, remote) = archive_controller::registration(&fixture, &route, rooted).await?; let controllers = controller::start_with_limits( &mut fixture, @@ -148,6 +159,32 @@ impl Harness { } } +pub(crate) async fn s3_namespace_minimum(rooted: bool, store: Arc) -> Result<()> { + let h = Harness::with_minimum(rooted, store, 1, Durability::S3).await?; + let gate = h.remote.hold_publication(); + let call = h + .command(1, b"minimum", Durability::Quorum, Duration::from_secs(15)) + .await?; + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut waiting = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), waiting.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = waiting.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response.receipt().receipt.minimum_durability(), + Durability::S3 + ); + assert_eq!(response.receipt().receipt.result(), b"minimum"); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(response); + h.close().await +} + pub(crate) async fn s3_commands( rooted: bool, store: Arc, diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index aeb290a46..235e86be6 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -537,7 +537,7 @@ mod s3_commands; #[allow(dead_code)] mod s3_response; #[allow(unused_imports)] -pub(crate) use s3_commands::{s3_commands, s3_deadline, s3_missing_sibling}; +pub(crate) use s3_commands::{s3_commands, s3_deadline, s3_missing_sibling, s3_namespace_minimum}; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index 652a6d966..e9bfe6b17 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -184,14 +184,24 @@ pub(super) async fn setup_store( rooted: bool, class: &RaftClass, store: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + setup_store_minimum(rooted, class, store, Durability::Quorum).await +} + +pub(super) async fn setup_store_minimum( + rooted: bool, + class: &RaftClass, + store: Arc, + minimum: Durability, ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let filesystem = Arc::new(DirectFileSystem); - let cluster = Cluster::with_store( + let cluster = Cluster::with_store_minimum( RaftRuntimeOptions::default(), tempfile::tempdir()?, filesystem.clone(), class.schema_digest(), store, + minimum, ) .await?; setup_cluster(rooted, cluster, filesystem).await diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 220d2e356..3abe1b580 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,11 +71,11 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs six exact ignored RustFS cases, each in materialized and extent modes: +It runs seven exact ignored RustFS cases, each in materialized and extent modes: stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 -settlement, deadline/cancellation isolation, and refusal when a sibling dependency -is missing. Catalog setup, archive +settlement, deadline/cancellation isolation, refusal when a sibling dependency +is missing, and namespace minimum enforcement. Catalog setup, archive objects, conditional registry updates and independent recovery use the actual S3 client. Fault injection surrounds that client; lost publication responses are injected after the real conditional write commits. No memory fallback exists. @@ -84,8 +84,8 @@ The refusal case verifies that an unbound dispatcher cannot claim S3 durability, including a namespace minimum. The positive command cases use an explicit controller binding, exact receipts from later archived images, retained proof admission and fresh routing/leader checks. Receipt lookup, enrollment and remote -S3 responses remain unqualified. The -runner does not qualify local log file purge, fenced recovery or performance. +S3 responses remain unqualified. The runner does not qualify local log file purge, +fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. `record.json` retains source and executable hashes, exact selections, output diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 9488ca50e..c65d8b104 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -31,6 +31,7 @@ "rustfs_s3_commands_require_exact_archived_results", "rustfs_s3_command_deadlines_do_not_quarantine_committed_sql", "rustfs_s3_command_requires_the_sibling_archive_dependencies", + "rustfs_s3_namespace_minimum_cannot_be_downgraded", ) From 2bf0dd2a6992954af41ff68722536064e4465a88 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 13:28:43 -0700 Subject: [PATCH 186/568] test: pass namespace policy through fixtures --- .../support/cell_group_controller_archive.rs | 18 +++++++++++++++++- .../tests/support/cell_group_s3_commands.rs | 2 +- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 00f41ce19..33036d3b9 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -79,6 +79,15 @@ pub(super) async fn registration( fixture: &RuntimeCluster, route: &GroupRoute, rooted: bool, +) -> Result<(CellGroupArchiveRegistration, Arc)> { + registration_minimum(fixture, route, rooted, Durability::Quorum).await +} + +pub(super) async fn registration_minimum( + fixture: &RuntimeCluster, + route: &GroupRoute, + rooted: bool, + minimum: Durability, ) -> Result<(CellGroupArchiveRegistration, Arc)> { let io = CellGroupArchivePool::new(Default::default())?; let binding = fixture @@ -106,10 +115,17 @@ pub(super) async fn registration( binding.commit(&fixture.cluster.domain).await?; let remote = archive_store::Store::around(fixture.cluster.store.clone()); let catalog = Catalog::new(remote.clone(), Default::default())?; + let mut config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ); + config.minimum_durability = minimum; let namespace = fixture .cluster .domain - .run(catalog.existing_namespace(fixture.cluster.directory.namespace_config().clone())) + .run(catalog.existing_namespace(config)) .await??; let directory = RangeDirectory::new(namespace, Default::default())?; let birth = fixture diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index cc9f6485f..b01d739d8 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -49,7 +49,7 @@ impl Harness { rooted, &class, store, minimum, )) .await?; - let (archive, remote) = archive_controller::registration(&fixture, &route, rooted).await?; + let (archive, remote) = archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; let controllers = controller::start_with_limits( &mut fixture, CellGroupControllerLimits { From 3d564f3db2c2eefc87119e7b3395f1bca17b99fa Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 13:31:55 -0700 Subject: [PATCH 187/568] style: format namespace archive fixture --- crates/cellar/tests/support/cell_group_s3_commands.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index b01d739d8..6d7ab7d19 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -49,7 +49,8 @@ impl Harness { rooted, &class, store, minimum, )) .await?; - let (archive, remote) = archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; + let (archive, remote) = + archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; let controllers = controller::start_with_limits( &mut fixture, CellGroupControllerLimits { From ab1f02284c2f035aa08d220a93e1bdf6564c74d7 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 13:58:26 -0700 Subject: [PATCH 188/568] docs: record S3 command qualification --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 121 +++++++----------- .../design/multi-raft/cell-groups/progress.md | 56 ++++++++ 3 files changed, 107 insertions(+), 76 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index eea9ed310..b2b53cbdc 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "7914177d6f17e3f6d647a39dc496122157283591", - "runtime_revision": "7914177d6f17e3f6d647a39dc496122157283591", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged.", + "source_revision": "3d564f3db2c2eefc87119e7b3395f1bca17b99fa", + "runtime_revision": "3d564f3db2c2eefc87119e7b3395f1bca17b99fa", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 3cf79e11d..c176338c1 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,79 +15,54 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest local RustFS qualification source is `d739bf795f7e58c1c27b2c6c496f9f57355e3c02`; production runtime -bytes remain unchanged from `7914177d`. Use -[the RustFS runner](../../../../tools/multi-raft/cell_groups.py) and -[setup instructions](../../../../tools/multi-raft/README.md#shared-cell-group-archive-checks) -for local S3 verification. Three exact cases pass in both image modes: controller -publication, uncertain publication/reconciliation with joined close, and refusal -of unproved S3 command/receipt/enrollment responses and namespace minimums. -Seven signed S3 probes, 183 native regressions and all ten check commands pass. -The source/executable/output-bound record and independent audit are under -`.cellar/cell-groups-evidence/CG-06-rustfs-controller/`. Credentials stay outside -source/evidence. Port allocation now accounts for other Colima profiles' host -forwards. The backend is real RustFS; voters are host-native tasks. - -Next extend this runner with positive S3 output cases while implementing exact -operation proof ownership. Existing refusal cases are not successful S3 settlement. -Unarchived-byte/disk limits, physical Raft log purge, fencing/fresh serving, measured -load and public cutover remain open. See the newest [progress](progress.md) entry. - -The latest partial source is `7914177d6f17e3f6d647a39dc496122157283591`. -CG-06 is IN_PROGRESS. One fair archive lane now runs on the existing node controller, -using a deduplicated dirty-group queue, existing timer and shared admission pools. -Registration requires the initialized namespace binding and a private verifier store. -An engine/creation-bound locked journal fsyncs exact intents before remote mutations. -Missing established metadata refuses. Uncertain publication reconciles its exact -hold, or abort-fences the ticket and reconciles again before allocating anything new. -New head plus predecessor retirement are journaled before release. Physical coverage -stays job-scoped; archive status is historical metadata without S3 settlement rights. - -Completion compares covered position with runtime applied state and consumed apply -events. This fixes the reproduced idle-tail gap after recovery of an older publication. -Both image forms pass held publication/quorum progress, lost CAS reply/readback outage, -exact-intent recovery, cancelled registration response, joined close, single current -hold and independent recovery of every cell. Journal reopen/missing-state and orphaned -physical waiter tests pass. A sealed scheduling capability uses captured domain tokens -without shutdown rights; archive-specific physical joins precede private-store, -journal-lock and runtime/SQL teardown. New verifier stores are required on restart. - -All 40 gate commands pass: nineteen exact archive, two exact controller and two exact -lifecycle/journal unit cases, 157 execution, five snapshot and four LTX regressions -(189 distinct Rust cases across six copied executables), all-target build/Clippy, -formatting/layout, dependencies/docs/preflight and all 37 Python tests. The record at -`.cellar/cell-groups-evidence/CG-06-controller-archive/qualification/checkpoint.json` -binds 344 source hashes, frozen inputs, hashed logs and its catalog-cut predecessor. -Independent source/artifact/exact-selection audit passes. Initial compile, native -idle-progress, strict source-refusal and Clippy failures are source-bound and retained. See [progress](progress.md) -and the [controller archive contract](protocols.md#controller-owned-archive-publication). - -Next implement actual S3 command/enrollment/receipt output settlement using retained -independently verified coverage with exact cell version/result/current-route checks. -Preserve captured identities/inputs and deadlines; never rerun callbacks. Shared -dispatch still refuses S3. Add explicit unarchived-byte/disk admission for bounded -Quorum availability without S3. Then qualify physical local Raft Engine file purge, -fenced unmanaged recovery into fresh group/cell incarnations and complete old-voter -lineage inventory, including joins after the recovery cut. Fencing-authority API plus -Docker qualification remains the accepted direction. Automatic remote/private-cache -maintenance and process-kill controller restart are still unqualified. This initial -controller coalesces complete cuts; measured incremental scheduling remains future work. - -Namespace binding fixes expected registry identity and canonical storage location. -Runtime source capture/upload_cut keeps root/configuration/permanent membership -stable across I/O. All-voter disk/journal erasure still recovers data via a fresh catalog -owner and externally retained untrusted locators, without fencing/replacement voters. -Protected publication/collection retains exact-intent, never-reused generation, -pending-parent/reader hold and delayed-DELETE guarantees. Persist every intent before -mutation; unknown outcomes never justify another identity or counter reset. - -CG-08 effect/cache/result lifetime and maintenance precede CG-07 public SDK/fleet -cutover. Topology, placement, hibernation, managed recovery, RustFS/load/fault gates -and final legacy removal remain. Actual in-memory adapters supply the current native -evidence. Real RustFS, Docker fencing, fresh-quorum serving, measured load and full -hard cutover remain unqualified. Four original CG-06 cases and its full task gate are -pending. Preserve catalog-cut source `df0ba1f6`, protected publication `87348477`, -isolated verifier `14955f2d`, committed tail `7a29c2a9` and their ignored evidence. +The latest partial runtime/qualification source is `3d564f3db2c2eefc87119e7b3395f1bca17b99fa`. +CG-06 remains IN_PROGRESS. Local command S3 responses are enabled only through +`CellGroupDispatcher::with_s3` and that exact controller/runtime's archive owner. +A complete published cut is physically replayed, then its immutable SQL receipt +must match the exact command/history/version/result. Later images can prove older +receipts. Catalog birth and live leader authority are rechecked before the token. +Request, namespace and stored durability minima are never lowered. + +A committed producer resolves before its S3 wait. Independent S3 admission takes +result retention before SQL releases its operation/byte permit; existing quotas +remain unchanged. One fair archive lane batches at most 32 proofs per group turn. +Successful outputs retain complete physical coverage, S3 admission and the journal +lock, and weak owner uses prevent premature predecessor-hold retirement. Those +active consumers can backpressure the shared replay pools; idle groups retain no +workspace. Cancellation/deadlines keep their original identity and cannot rerun +callbacks or quarantine already-resolved SQL ownership. + +Seven exact RustFS cases pass in both image forms, plus seven signed backend probes. +All fourteen check/regression commands pass, covering 193 Rust cases across six +copied executables, all-target Clippy, format/layout, dependency/docs/preflight/plan +checks, 37 Python tests and system-Python hashing. Source/artifact/exact-selection +and credential-exclusion audits pass. Retained records, frozen inputs and the +initial unchanged-quota retention failure are under +`.cellar/cell-groups-evidence/CG-06-s3-commands/`. See the newest +[progress](progress.md), [S3 response contract](protocols.md#local-command-s3-response-gate) +and [RustFS runner](../../../../tools/multi-raft/cell_groups.py). + +Next integrate S3 enrollment, receipt lookup and remote response settlement; +those paths still refuse unsupported S3 capabilities. Add explicit unarchived-byte/ +disk limits, then physical local Raft Engine file purge and fenced fresh-incarnation +serving. Retention across multiple heads, controller process-kill restart and +physical-read shutdown cuts need qualification. The fence inventory must cover +every possible old voting lineage, including joins after the recovery cut. +Fencing-authority API plus Docker qualification remains the accepted direction. +CG-08 effect/cache/result lifetime and maintenance precedes CG-07 public SDK/fleet +cutover. Topology, placement, hibernation, managed recovery, measured load and final +legacy removal remain. The source/runtime goal is not complete. + +Namespace bindings, stable runtime cut capture/upload, exact-intent journals, +protected publication/remote collection and source-independent complete recovery +retain their earlier contracts. Persist every exact intent before mutation; +unknown outcomes never allocate another identity or reset counters. The prior +RustFS qualification at `d739bf79` and runtime source `7914177d`, catalog-cut +`df0ba1f6`, protected publication `87348477`, isolated verifier `14955f2d` and +committed tail `7a29c2a9` remain retained. Current voters are host-native tasks with +one local RustFS VM; no node-container resource gate, independent hosts, measured +load, fenced fresh-quorum disaster serving or complete hard cutover is claimed. +Four original CG-06 acceptance cases and its full task gate remain pending. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 07c48dde2..396fa4114 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,62 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 local command S3 response settlement + +Source: `3d564f3db2c2eefc87119e7b3395f1bca17b99fa`. CG-06 remains IN_PROGRESS. An explicit +`CellGroupDispatcher::with_s3` binding now settles local commands through the +existing node archive lane. It checks the exact runtime owner and active group +registration. Required durability is the maximum of request, namespace and stored +receipt minimums. Unbound dispatchers retain their refusal; receipt lookup, +enrollment and remote S3 response gates remain unfinished. + +SQL resolution clears producer ownership before archival. A separate bounded S3 +stage acquires admission before taking over the committed result bytes; resolved +preparations no longer retain duplicate SQL admission. Existing SQL/command quotas +are unchanged. The first same-cell successor test reproduced Overloaded because +the waiting receipt still held the SQL pool's only operation permit. Its source +snapshot, executable and output are retained. The fixed test passes with that same +one-slot SQL quota, including an explicit zero retained-SQL-operation assertion. + +The fair archive lane takes at most 32 receipt proofs per group turn. A published +complete cut is independently replayed and its exact immutable receipt is read +without callbacks. History/version/position, equal-version decision and every +stored receipt field/result must match. A later image can prove an older receipt; +an advisory index or sibling progress cannot. The response rechecks catalog birth +and live leader authority before its token. It retains physical coverage/admission, +S3 byte/item retention and the exclusive journal lock. Weak owner uses block +predecessor-hold retirement until consumers release the coverage. Idle groups keep +no replay workspace; held outputs intentionally backpressure those shared pools. + +Seven exact RustFS cases pass in both image modes, including positive S3 responses, +a later same-cell quorum mutation during archive outage, publication-held output, +Quorum retries honoring a stored S3 minimum, proof-slot retention, callback-once +retries, one-command-slot expiry/cancellation isolation and missing-B dependency +refusal while A's receipt and quorum remain healthy. Foreign runtimes refuse +before SQL. A namespace S3 minimum overrides a Quorum request, persists that +minimum and keeps its response pending until publication. The three earlier setup/publication/refusal scenarios still pass. +The actual S3 client performs catalog/archive I/O and conditional writes. + +All fourteen regression/check commands pass: 193 distinct Rust cases across six +executables, locked all-target Clippy for Cellar/logic/LTX, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +The records under `.cellar/cell-groups-evidence/CG-06-s3-commands/` bind +2564 source hashes, frozen inputs, copied executables, exact selections, +output hashes and predecessors. Independent source/hash/selection/credential +exclusion audit passes. The source-bound Clippy large-enum failure is retained; +boxing the private proof bundle fixes it without public variant or contract changes. +No original required case/oracle or pre-existing quota was weakened. Four original +CG-06 cases and its full task gate remain pending; this is partial evidence. + +Reflection: integrate S3 enrollment, receipt lookup and remote output contracts +using this retained owner, then explicit unarchived-byte/disk admission. Qualify +multi-head retention, controller process-kill restart and physical-read shutdown +cuts, physical shared-engine Raft log purge and fenced recovery into fresh group/ +cell incarnations. Keep the complete old-voter lineage inventory, including later +joins. CG-08 cache/effect/result lifetime work precedes CG-07 public SDK/fleet +cutover. Native voters share one host and RustFS one local VM; no independent +failure-domain, load, complete disaster recovery or full hard cutover is claimed. + ## CG-06 local RustFS archive and response-boundary qualification Source: `d739bf795f7e58c1c27b2c6c496f9f57355e3c02`. Production runtime bytes are unchanged from `7914177d`. From cc7171d39bd1309c5b9bb8122a1a99a71bb1657e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 14:07:31 -0700 Subject: [PATCH 189/568] feat: prove S3 enrollment durability --- .../raft/cell_groups/archive/enrollment.rs | 33 +++ .../raft/cell_groups/archive/mod.rs | 1 + .../controller/archive/settlement.rs | 144 +++++++++--- .../raft/cell_groups/controller/mod.rs | 2 +- .../raft/cell_groups/dispatch/command/mod.rs | 2 +- .../raft/cell_groups/dispatch/enrollment.rs | 77 +++++- .../cell_groups/dispatch/enrollment_remote.rs | 8 + .../raft/cell_groups/dispatch/mod.rs | 4 +- .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../raft/host_sql/cell_groups/mod.rs | 34 ++- .../raft/host_sql/cell_groups/worker.rs | 14 +- .../native_cell_groups_archive_controller.rs | 63 +++++ .../tests/support/cell_group_runtime.rs | 5 +- .../tests/support/cell_group_s3_commands.rs | 32 ++- .../tests/support/cell_group_s3_enrollment.rs | 219 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 5 +- .../multi-raft/cell-groups/protocols.md | 30 ++- docs/design/multi-raft/cell-groups/tasks.json | 4 +- tools/multi-raft/README.md | 11 +- tools/multi-raft/cell_groups.py | 10 +- 20 files changed, 625 insertions(+), 74 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive/enrollment.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_enrollment.rs diff --git a/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs b/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs new file mode 100644 index 000000000..289fce993 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/enrollment.rs @@ -0,0 +1,33 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Original creation provenance in a complete independently recovered group. +use super::*; +use crate::raft::cell_groups::EnrollmentRecord; + +pub(in crate::raft::cell_groups) fn verify( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + expected: &EnrollmentRecord, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 enrollment verifier owner/route", + )); + } + let Some((leaf, _)) = root.pin_cell(&expected.cell().cell) else { + // An older complete cut can precede this committed creation. + return Ok(false); + }; + if &leaf.enrollment != expected { + return Err(CellGroupArchiveError::Invalid( + "S3 enrollment creation record", + )); + } + // The full coverage already verified every cell image. The immutable + // creation record, including its initial commitment, survives later images; + // retaining superseded initial image bytes is not required. + store.require_live()?; + Ok(true) +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 0e8b2ef35..d5b420586 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -38,6 +38,7 @@ pub use ownership::{ CellGroupArchiveCreation, CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchiveRegistry, CellGroupArchiveUploadTicket, PreparedCellGroupArchiveCreation, }; +pub(in crate::raft::cell_groups) mod enrollment; pub(in crate::raft::cell_groups) mod receipt; mod replay; mod tail; diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs index 5b0ecd999..f0c282541 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -3,7 +3,7 @@ //! Operation proofs share the archive lane; waiting never owns a SQL producer. use super::*; use crate::raft::budget::Permit; -use crate::raft::cell_groups::{CellDispatchError, CellSqlReceipt}; +use crate::raft::cell_groups::{CellDispatchError, CellSqlEnrollment, CellSqlReceipt}; use crate::raft::execution::ExecutionLimits; use cellar_logic::cell_group::GroupRoute; use std::sync::{atomic::AtomicBool, Weak}; @@ -11,7 +11,7 @@ use std::sync::{atomic::AtomicBool, Weak}; #[derive(Clone)] pub(crate) struct Client(Weak); -pub(crate) struct CommandProof { +pub(crate) struct OutputProof { _coverage: CellGroupArchiveCoverage, // The exclusive journal lock survives response buffering and node close. // A replacement owner cannot retire this hold while a consumer still uses it. @@ -20,7 +20,7 @@ pub(crate) struct CommandProof { authority: Arc, } -impl CommandProof { +impl OutputProof { pub(crate) async fn require_current(&self) -> std::result::Result<(), CellDispatchError> { self.authority .require_current() @@ -29,15 +29,51 @@ impl CommandProof { } } -pub(crate) struct Settled { +pub(crate) struct CommandSettled { pub receipt: Box, - pub proof: Box, + pub proof: Box, +} + +pub(crate) struct EnrollmentSettled { + pub enrollment: Box, + pub proof: Box, +} + +enum Output { + Command(Box), + Enrollment(Box), +} + +impl Output { + fn retained_bytes(&self) -> std::result::Result { + match self { + Self::Command(receipt) => receipt.receipt().receipt.result().len(), + // Creation records are bounded by the protocol to 4096 bytes. Their + // variable identity bytes and the archive comparison stay admitted. + Self::Enrollment(_) => 4096, + } + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded)) + } + + fn retain(self, permit: Arc) -> Self { + match self { + Self::Command(receipt) => Self::Command(receipt.retain_for_archive(permit)), + Self::Enrollment(enrollment) => Self::Enrollment(enrollment.retain_for_archive(permit)), + } + } +} + +struct Settled { + output: Output, + proof: Box, } pub(in crate::raft::cell_groups::controller) struct Request { pub route: GroupRoute, - receipt: Box, - limits: ExecutionLimits, + output: Output, + limits: Option, deadline: u64, cancelled: Arc, reply: oneshot::Sender>, @@ -115,6 +151,57 @@ impl Client { receipt: Box, limits: ExecutionLimits, deadline: u64, + ) -> std::result::Result { + let settled = self + .settle( + runtime, + route, + Output::Command(receipt), + Some(limits), + deadline, + ) + .await?; + let Output::Command(receipt) = settled.output else { + return Err(CellDispatchError::Invalid("S3 command output")); + }; + Ok(CommandSettled { + receipt, + proof: settled.proof, + }) + } + + pub(crate) async fn enrollment( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + enrollment: Box, + deadline: u64, + ) -> std::result::Result { + let settled = self + .settle( + runtime, + route, + Output::Enrollment(enrollment), + None, + deadline, + ) + .await?; + let Output::Enrollment(enrollment) = settled.output else { + return Err(CellDispatchError::Invalid("S3 enrollment output")); + }; + Ok(EnrollmentSettled { + enrollment, + proof: settled.proof, + }) + } + + async fn settle( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + output: Output, + limits: Option, + deadline: u64, ) -> std::result::Result { self.require_runtime(runtime)?; self.require_group(&route)?; @@ -122,20 +209,12 @@ impl Client { .0 .upgrade() .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; - let size = receipt - .receipt() - .receipt - .result() - .len() - .checked_mul(2) - .and_then(|n| n.checked_add(8192)) - .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; let permit = shared .s3 - .acquire(size) + .acquire(output.retained_bytes()?) .map(Arc::new) .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; - let receipt = receipt.retain_for_archive(permit.clone()); + let output = output.retain(permit.clone()); let (reply, receiver) = oneshot::channel(); let cancelled = Arc::new(AtomicBool::new(false)); let _cancel = Cancel(cancelled.clone()); @@ -150,7 +229,7 @@ impl Client { .requests .try_send(requests::Request::S3(Box::new(Request { route, - receipt, + output, limits, deadline, cancelled, @@ -232,21 +311,32 @@ pub(super) async fn prove( continue; } if request.route != *coverage.marker().route() { - request.fail(Arc::new(Error::Invalid("S3 command route"))); + request.fail(Arc::new(Error::Invalid("S3 output route"))); continue; } let store = registration.store.clone(); let physical = coverage.clone(); let read = execution.blocking(move || { - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { - crate::raft::cell_groups::archive::receipt::command( + let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| match &request + .output + { + Output::Command(receipt) => crate::raft::cell_groups::archive::receipt::command( &physical, &store, - request.receipt.receipt(), - request.limits, + receipt.receipt(), + request + .limits + .ok_or(CellGroupArchiveError::Invalid("S3 command limits"))?, request.deadline, request.cancelled.clone(), - ) + ), + Output::Enrollment(enrollment) => { + crate::raft::cell_groups::archive::enrollment::verify( + &physical, + &store, + enrollment.record(), + ) + } })) .unwrap_or_else(|panic| { Err(CellGroupArchiveError::Replay( @@ -279,14 +369,14 @@ pub(super) async fn prove( continue; } let Request { - receipt, + output, reply, permit, .. } = request; let _ = reply.send(Ok(Settled { - receipt, - proof: Box::new(CommandProof { + output, + proof: Box::new(OutputProof { _coverage: coverage.clone(), _journal: context.journal.clone(), _permit: permit, diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 44fd6f95c..3dbfb93e6 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -4,7 +4,7 @@ //! Admitted command ownership survives response cancellation and expiry. mod archive; -pub(crate) use archive::settlement::{Client as S3Client, CommandProof as S3CommandProof}; +pub(crate) use archive::settlement::{Client as S3Client, OutputProof as S3OutputProof}; mod calls; pub use archive::{ CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 01b8e86d2..e8c6045d6 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -62,7 +62,7 @@ pub enum CellCommandSettlement { pub struct CellCommandResponse { receipt: Box, token: CellVersionToken, - s3: Option>, + s3: Option>, } impl CellCommandResponse { diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index 6be3134da..57f79eada 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -49,6 +49,7 @@ pub struct CellEnrollmentInvocation { dispatcher: CellGroupDispatcher, cell: CellHistory, creation: ProposalId, + durability: Durability, preview: ImageId, inputs: CommandInputs, session: CellRoutingSession, @@ -76,7 +77,7 @@ impl CellGroupDispatcher { pub(crate) async fn enroll_with_inputs( &self, cell: CellHistory, - request: CellEnrollmentRequest, + mut request: CellEnrollmentRequest, limits: CellRoutingLimits, cached: Option, captured: Option<(CommandInputs, ImageId)>, @@ -87,9 +88,10 @@ impl CellGroupDispatcher { cause: Box::new(cause), }; let started = self.domain.mono_ms().map_err(|_| error(closed()))?; - if request.durability != Durability::Quorum - || self.directory.namespace_config().minimum_durability != Durability::Quorum - { + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + if request.durability == Durability::S3 && self.s3.is_none() { return Err(error(CellDispatchError::Execution( ExecutionError::Refused(CommandFailure::UnsupportedCapability), ))); @@ -135,7 +137,13 @@ impl CellGroupDispatcher { hash.update(inputs.now_unix_millis.to_be_bytes()); hash.update(inputs.random); hash.update(limits.timeout.as_nanos().to_be_bytes()); - hash.update([limits.max_attempts, 1]); + hash.update([ + limits.max_attempts, + match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }, + ]); let session = self .directory .routing_session_in( @@ -155,6 +163,7 @@ impl CellGroupDispatcher { dispatcher: self.clone(), cell, creation: request.creation, + durability: request.durability, preview, inputs, session, @@ -255,6 +264,15 @@ impl CellEnrollmentInvocation { if authority.state().route() != attempt.route() { return Err(CellDispatchError::Invalid("enrollment destination route")); } + if self.durability == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 enrollment owner"))?; + owner.require_runtime(runtime)?; + owner.require_group(attempt.route())?; + } self.session .mark_submitted(&attempt) .map_err(CellDispatchError::Routing)?; @@ -280,7 +298,7 @@ impl CellEnrollmentInvocation { ) .await; match staged { - Ok(CellSqlOutcome::Enrollment(enrolled)) => return self.committed(*enrolled), + Ok(CellSqlOutcome::Enrollment(enrolled)) => return self.committed(*enrolled).await, Ok(CellSqlOutcome::Prepared(prepared)) => self.origin_mut()?.prepared = Some(prepared), Ok(_) => { return Err(CellDispatchError::Invalid( @@ -372,7 +390,7 @@ impl CellEnrollmentInvocation { .await; match resolved { Ok(CellSqlEnrollmentResolution::Committed(enrolled)) => { - return self.committed(*enrolled) + return self.committed(*enrolled).await } Ok(result) => { self.origin = None; @@ -416,7 +434,7 @@ impl CellEnrollmentInvocation { .ok_or(CellDispatchError::Invalid("missing enrollment preparation")) } - fn committed( + async fn committed( &mut self, enrolled: CellSqlEnrollment, ) -> Result { @@ -427,8 +445,47 @@ impl CellEnrollmentInvocation { { return Err(CellDispatchError::Invalid("enrollment receipt binding")); } - self.origin = None; - Ok(enrolled) + // SQL is resolved. Archive failures and deadlines must not quarantine + // this group or keep the initializer's producer/preparation admitted. + let mut origin = self + .origin + .take() + .ok_or(CellDispatchError::Invalid("resolved enrollment origin"))?; + if self.durability == Durability::Quorum { + return Ok(enrolled); + } + origin.prepared = None; + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 enrollment owner"))?; + let settled = owner + .enrollment( + &origin.runtime, + origin.route.clone(), + Box::new(enrolled), + self.session.deadline_ms(), + ) + .await?; + let authority = origin + .runtime + .enrollment_authority(origin.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.state().route() != &origin.route + || authority.state().record() != Some(settled.enrollment.record()) + { + return Err(CellDispatchError::Invalid( + "S3 enrollment current creation/route", + )); + } + settled.proof.require_current().await?; + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + authority.check().map_err(CellDispatchError::Runtime)?; + Ok((*settled.enrollment).with_archive_proof(settled.proof)) } async fn reject( diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs index b959e9b48..a82261d3f 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs @@ -81,6 +81,14 @@ impl CellEnrollmentInvocation { "consumed or local enrollment attempt", ))); } + // The remote response schema has no retained archive proof contract yet. + if self.durability == Durability::S3 { + return Err( + self.error(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))), + ); + } if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(self.error(CellDispatchError::Invalid("enrollment transport domain"))); } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index b6d51ac5e..49d6d375f 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -147,9 +147,9 @@ impl CellGroupDispatcher { }) } - /// Bind local command S3 settlement to this node's bounded archive owner. + /// Bind local command/enrollment S3 settlement to this node's bounded archive owner. /// The destination group must already have an archive registration. Receipt - /// lookup, enrollment and remote response settlement have separate gates. + /// lookup and remote response settlement have separate gates. pub fn with_s3(mut self, controller: &CellGroupController) -> Result { self.s3 = Some( controller diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 477a4d83e..fb08b4406 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -36,6 +36,7 @@ pub use checkpoint::{ PreparedLocalCellGroupCheckpoint, }; mod controller; +pub(crate) use controller::S3OutputProof; pub use controller::{ CellCommandAdmissionError, CellCommandCall, CellEnrollmentAdmissionError, CellEnrollmentCall, CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index ce2d9cf93..49fef4179 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -151,13 +151,39 @@ impl CellSqlRootOptions { pub struct CellSqlEnrollment { record: EnrollmentRecord, - _bytes: Arc, + _retention: OutputRetention, + s3: Option>, } impl CellSqlEnrollment { pub fn record(&self) -> &EnrollmentRecord { &self.record } + + /// Achieved output durability, never inferred from creation metadata alone. + pub fn durability(&self) -> cellar_logic::consensus::Durability { + if self.s3.is_some() { + cellar_logic::consensus::Durability::S3 + } else { + cellar_logic::consensus::Durability::Quorum + } + } + + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc, + ) -> Box { + self._retention = OutputRetention::Archive { _permit: permit }; + self + } + + pub(crate) fn with_archive_proof( + mut self, + proof: Box, + ) -> Self { + self.s3 = Some(proof); + self + } } pub enum CellSqlEnrollmentResolution { @@ -238,10 +264,10 @@ pub struct CellSqlRejection { pub struct CellSqlReceipt { receipt: CellCommittedReceipt, - _retention: ReceiptRetention, + _retention: OutputRetention, } -enum ReceiptRetention { +enum OutputRetention { Sql { _bytes: Arc, }, @@ -257,7 +283,7 @@ impl CellSqlReceipt { mut self: Box, permit: Arc, ) -> Box { - self._retention = ReceiptRetention::Archive { _permit: permit }; + self._retention = OutputRetention::Archive { _permit: permit }; self } diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index b9692ee1b..0dd6334aa 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -227,7 +227,7 @@ impl Cells { Bytes::shrink(&mut bytes, retained)?; Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { receipt: *receipt, - _retention: ReceiptRetention::Sql { _bytes: bytes }, + _retention: OutputRetention::Sql { _bytes: bytes }, }))) } Err(error) => { @@ -263,7 +263,8 @@ impl Cells { } return Ok(CellSqlOutcome::Enrollment(Box::new(CellSqlEnrollment { record: record.clone(), - _bytes: bytes, + _retention: OutputRetention::Sql { _bytes: bytes }, + s3: None, }))); } let mut executor = options.open_enrollment(authority)?; @@ -313,7 +314,10 @@ impl Cells { CellEnrollmentResolution::Committed(record) => { CellSqlEnrollmentResolution::Committed(Box::new(CellSqlEnrollment { record: *record, - _bytes: pending.prepared._bytes, + _retention: OutputRetention::Sql { + _bytes: pending.prepared._bytes, + }, + s3: None, })) } CellEnrollmentResolution::NotCommitted => { @@ -375,7 +379,7 @@ impl Cells { pending.producer.close()?; Ok(CellSqlOutcome::Receipt(Box::new(CellSqlReceipt { receipt, - _retention: ReceiptRetention::Sql { + _retention: OutputRetention::Sql { _bytes: pending.prepared._bytes, }, }))) @@ -398,7 +402,7 @@ impl Cells { CellReconciliation::Committed(receipt) => { CellSqlReconciliation::Committed(Box::new(CellSqlReceipt { receipt: *receipt, - _retention: ReceiptRetention::Sql { + _retention: OutputRetention::Sql { _bytes: pending.prepared._bytes, }, })) diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index 2435d67a8..36b8cd89c 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -180,3 +180,66 @@ async fn rustfs_unproved_s3_responses_refuse() -> Result<()> { } Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_enrollment_requires_the_original_archived_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_enrollment( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_enrollment_deadlines_preserve_resolved_ownership() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_enrollment_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_enrollment_requires_the_original_archived_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_enrollment( + rooted, + rustfs::store("s3-enrollment", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_enrollment_deadlines_preserve_resolved_ownership() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_enrollment_deadline( + rooted, + rustfs::store( + if cancel { + "s3-enrollment-cancel" + } else { + "s3-enrollment-deadline" + }, + rooted, + )?, + cancel, + )) + .await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 068fed243..501a0fb69 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -814,7 +814,10 @@ pub(super) use sql::{ }; pub(super) use sql::{owned_command_failure, owned_commands}; #[allow(unused_imports)] -pub(super) use sql::{s3_commands, s3_deadline, s3_missing_sibling, s3_namespace_minimum}; +pub(super) use sql::{ + s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, + s3_namespace_minimum, +}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 6d7ab7d19..fffc30d71 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -1,7 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Positive command output proofs on the actual controller and archive lane. +//! SQL output proofs on the actual controller and archive lane. +#[path = "cell_group_s3_enrollment.rs"] +mod enrollment; use super::*; +pub(crate) use enrollment::{s3_enrollment, s3_enrollment_deadline}; use futures_util::TryStreamExt; use object_store::{path::Path as ObjectPath, ObjectStore}; use std::sync::atomic::{AtomicUsize, Ordering}; @@ -16,6 +19,7 @@ struct Harness { archive: CellGroupArchiveRegistration, remote: Arc, calls: Arc, + initializations: Arc, retry_until: u64, } @@ -36,15 +40,22 @@ impl Harness { ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); - let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( - "Add", - [82; 32], - move |context, input| { - count.fetch_add(1, Ordering::SeqCst); - context.sql().execute("UPDATE value SET n=n+1", [])?; - Ok(input.to_vec()) - }, - )?); + let initializations = Arc::new(AtomicUsize::new(0)); + let init_count = initializations.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, transaction| { + init_count.fetch_add(1, Ordering::SeqCst); + transaction + .execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(input.to_vec()) + })?, + ); let (mut fixture, route, cell, _) = Box::pin(registered::setup_store_minimum( rooted, &class, store, minimum, )) @@ -103,6 +114,7 @@ impl Harness { archive, remote, calls, + initializations, retry_until, }) } diff --git a/crates/cellar/tests/support/cell_group_s3_enrollment.rs b/crates/cellar/tests/support/cell_group_s3_enrollment.rs new file mode 100644 index 000000000..140fa7891 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_enrollment.rs @@ -0,0 +1,219 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Original creation records survive delayed archival and later cell mutations. +use super::*; + +impl Harness { + async fn enroll( + &self, + dispatcher: &CellGroupDispatcher, + creation: u8, + durability: Durability, + timeout: Duration, + ) -> Result { + let invocation = dispatcher + .enroll( + self.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([creation; 16])?, + durability, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + ) + .await?; + Ok(self.controllers[0].enroll(invocation, self.fixture.cell_options(&self.cell).await?)?) + } + + async fn creation(&self) -> Result { + Ok(self.fixture.runtimes[0] + .enrollment_state(self.route.history, self.cell.clone()) + .await? + .record() + .context("committed creation")? + .clone()) + } +} + +pub(crate) async fn s3_enrollment(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "namespace-s3" + } else { + "namespace-quorum" + }), + )); + let mut h = Harness::with_minimum(rooted, backend, 2, minimum).await?; + h.cell.cell = CellId::new("new-s3-enrollment".to_owned())?; + let requested = if minimum == Durability::S3 { + Durability::Quorum + } else { + Durability::S3 + }; + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let error = h + .enroll(&foreign, 210, requested, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("foreign enrollment runtime")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Invalid(_)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + h.remote.outage.store(true, Ordering::Release); + let call = h + .enroll(&h.dispatcher, 210, requested, Duration::from_secs(15)) + .await?; + h.waiting().await?; + let original = h.creation().await?; + assert_eq!(original.creation(), ProposalId::new([210; 16])?); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + if minimum == Durability::Quorum { + let changed = h + .command( + 1, + b"after-initialization", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(changed.receipt().version.sequence.get(), 2); + drop(changed); + } + let gate = h.remote.hold_publication(); + h.remote.outage.store(false, Ordering::Release); + tokio::time::timeout(Duration::from_secs(10), gate.entered.notified()).await?; + let mut response = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), response.as_mut()) + .await + .is_err() + ); + gate.release.notify_one(); + let response = response.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.record(), &original); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + let duplicate = h + .enroll(&h.dispatcher, 210, Durability::S3, Duration::from_secs(15)) + .await?; + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 2)).await?; + let mut duplicate = Box::pin(duplicate.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), duplicate.as_mut()) + .await + .is_err() + ); + drop(response); + let response = duplicate.await?; + assert_eq!(response.record(), &original); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + drop(response); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + let error = h + .enroll(&h.dispatcher, 211, Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("conflicting creation")?; + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::CommandConflict)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_enrollment_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let mut h = Harness::new(rooted, store, 1).await?; + h.cell.cell = CellId::new("expired-s3-enrollment".to_owned())?; + h.remote.outage.store(true, Ordering::Release); + let call = h + .enroll( + &h.dispatcher, + 212, + Durability::S3, + Duration::from_millis(800), + ) + .await?; + h.waiting().await?; + let original = h.creation().await?; + if cancel { + drop(call); + } else { + let error = call + .response() + .await + .err() + .context("S3 enrollment deadline")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_commands == 0 + && h.controllers[0].usage().retained_s3 == 0) + }) + .await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + let changed = h + .command( + 1, + b"after-deadline", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(changed.receipt().version.sequence.get(), 2); + drop(changed); + h.remote.outage.store(false, Ordering::Release); + let response = h + .enroll(&h.dispatcher, 212, Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(response.record(), &original); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + drop(response); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 235e86be6..42f80ebb5 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -537,7 +537,10 @@ mod s3_commands; #[allow(dead_code)] mod s3_response; #[allow(unused_imports)] -pub(crate) use s3_commands::{s3_commands, s3_deadline, s3_missing_sibling, s3_namespace_minimum}; +pub(crate) use s3_commands::{ + s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, + s3_namespace_minimum, +}; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 78c3a0068..bf82e8cee 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -720,8 +720,8 @@ through this archive-specific drain. the same execution domain. Execution also requires its exact runtime owner and an active archive registration for the selected group/route. Effective durability is the maximum of the request, namespace minimum and existing receipt minimum. -Receipt lookup, enrollment and remote S3 response paths retain their refusal gates -until their own output proofs are integrated. +Receipt lookup and remote S3 response paths retain their refusal gates until +their own output proofs are integrated. A committed SQL receipt resolves its producer before any archive wait. The owner acquires independent S3 byte/item admission before transferring output retention @@ -756,11 +756,35 @@ Idle registrations do not retain replay workspaces. Active consumers can occupy those shared workspaces and backpressure subsequent archival until consumption; this is explicit pool admission, without an unaccounted metadata-only proof cache. -S3 receipt-lookup/enrollment/remote responses, explicit unarchived-byte/disk +S3 receipt-lookup/remote responses, explicit unarchived-byte/disk admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, load and process-kill qualification remain required gates. This local command boundary is not the complete CG-06 or public SDK/fleet cutover gate. +### Local enrollment S3 response gate + +The same explicit dispatcher binding and bounded archive lane also settle local +enrollments. Required durability is the maximum of the request and immutable +namespace minimum, captured in the invocation fingerprint. The exact runtime +owner and active archive registration are checked before entering initialization. +Remote enrollment refuses S3 before transmission until its wire proof contract +is integrated; it cannot silently send a Quorum request. + +SQL resolves and closes the initializer before archival. A separately admitted +S3 output takes retention of the bounded original creation record before SQL +releases its bytes and operation slot. The complete independently recovered root +must contain that exact history, creation ID, schema and initial image commitment. +A cut preceding creation waits; a different creation refuses. Later mutations do +not replace the original record or require retaining superseded initial bytes. +No initializer runs during proof or exact retry. + +Before returning, recheck the current catalog birth, actual leader authority and +its exact committed creation record under the original deadline. The returned +`CellSqlEnrollment::durability` derives S3 only from its private retained proof. +That output retains the same physical coverage, pool admission and journal lock +as a command response. A deadline or dropped reply keeps commitment ambiguous, +without retaining an unresolved SQL producer or quarantining its healthy group. + ## Split and merge Use the same transition state machine for one parent to multiple child ranges diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index f4b3adf13..0fe6c17db 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1764,7 +1764,9 @@ "tools/multi-raft/cell_groups.py", "crates/cellar/durability/raft/cell_groups/archive/receipt.rs", "crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs", - "crates/cellar/tests/support/cell_group_s3_commands.rs" + "crates/cellar/tests/support/cell_group_s3_commands.rs", + "crates/cellar/durability/raft/cell_groups/archive/enrollment.rs", + "crates/cellar/tests/support/cell_group_s3_enrollment.rs" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 3abe1b580..1a0672969 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,11 +71,12 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs seven exact ignored RustFS cases, each in materialized and extent modes: +It runs nine exact ignored RustFS cases, each in materialized and extent modes: stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 settlement, deadline/cancellation isolation, refusal when a sibling dependency -is missing, and namespace minimum enforcement. Catalog setup, archive +is missing, namespace minimum enforcement, original enrollment creation proofs +and enrollment deadline/cancellation isolation. Catalog setup, archive objects, conditional registry updates and independent recovery use the actual S3 client. Fault injection surrounds that client; lost publication responses are injected after the real conditional write commits. No memory fallback exists. @@ -83,8 +84,10 @@ injected after the real conditional write commits. No memory fallback exists. The refusal case verifies that an unbound dispatcher cannot claim S3 durability, including a namespace minimum. The positive command cases use an explicit controller binding, exact receipts from later archived images, retained proof -admission and fresh routing/leader checks. Receipt lookup, enrollment and remote -S3 responses remain unqualified. The runner does not qualify local log file purge, +admission and fresh routing/leader checks. Enrollment cases preserve the original +creation after later mutations, honor namespace minimums, retain output proofs and +retry without rerunning initialization. Receipt lookup and remote S3 responses +remain unqualified. The runner does not qualify local log file purge, fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index c65d8b104..c584fb167 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 # Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -"""Qualify shared-group archive setup and local command responses against local RustFS. +"""Qualify shared-group archive setup and local command/enrollment responses against local RustFS. Uses only the recorded owned backend. Each invocation owns a new evidence directory and remote prefix. Credentials are passed through the child environment, -never arguments or evidence. Positive command acknowledgements require the explicit controller binding. +never arguments or evidence. Positive acknowledgements require the explicit controller binding. """ import argparse @@ -32,6 +32,8 @@ "rustfs_s3_command_deadlines_do_not_quarantine_committed_sql", "rustfs_s3_command_requires_the_sibling_archive_dependencies", "rustfs_s3_namespace_minimum_cannot_be_downgraded", + "rustfs_s3_enrollment_requires_the_original_archived_creation", + "rustfs_s3_enrollment_deadlines_preserve_resolved_ownership", ) @@ -106,11 +108,11 @@ def main(): out.mkdir(mode=0o700, parents=True, exist_ok=False) prefix = "cell-groups/" + secrets.token_hex(16) record = { - "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command S3 settlement", + "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command/enrollment S3 settlement", "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " - "No S3 receipt-lookup/enrollment or remote command acknowledgements, byte/disk outage bound, " + "No S3 receipt-lookup or remote acknowledgements, byte/disk outage bound, " "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", } From c2801b6586ffc8b3b765312e0042986e881d773a Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 14:19:13 -0700 Subject: [PATCH 190/568] test: avoid enrollment fixture allocations --- crates/cellar/tests/support/cell_group_s3_enrollment.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_enrollment.rs b/crates/cellar/tests/support/cell_group_s3_enrollment.rs index 140fa7891..42e9947d5 100644 --- a/crates/cellar/tests/support/cell_group_s3_enrollment.rs +++ b/crates/cellar/tests/support/cell_group_s3_enrollment.rs @@ -49,7 +49,7 @@ pub(crate) async fn s3_enrollment(rooted: bool, store: Arc) -> }), )); let mut h = Harness::with_minimum(rooted, backend, 2, minimum).await?; - h.cell.cell = CellId::new("new-s3-enrollment".to_owned())?; + h.cell.cell = CellId::new("new-s3-enrollment")?; let requested = if minimum == Durability::S3 { Durability::Quorum } else { @@ -154,7 +154,7 @@ pub(crate) async fn s3_enrollment_deadline( cancel: bool, ) -> Result<()> { let mut h = Harness::new(rooted, store, 1).await?; - h.cell.cell = CellId::new("expired-s3-enrollment".to_owned())?; + h.cell.cell = CellId::new("expired-s3-enrollment")?; h.remote.outage.store(true, Ordering::Release); let call = h .enroll( From f3353202c402841047d8f9976012bf61f6fba84f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 14:47:47 -0700 Subject: [PATCH 191/568] docs: record S3 enrollment qualification --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 95 ++++++++++--------- .../design/multi-raft/cell-groups/progress.md | 58 +++++++++++ 3 files changed, 111 insertions(+), 48 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index b2b53cbdc..2c7b0e5bc 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "3d564f3db2c2eefc87119e7b3395f1bca17b99fa", - "runtime_revision": "3d564f3db2c2eefc87119e7b3395f1bca17b99fa", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged.", + "source_revision": "c2801b6586ffc8b3b765312e0042986e881d773a", + "runtime_revision": "c2801b6586ffc8b3b765312e0042986e881d773a", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index c176338c1..669047da8 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,54 +15,59 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial runtime/qualification source is `3d564f3db2c2eefc87119e7b3395f1bca17b99fa`. -CG-06 remains IN_PROGRESS. Local command S3 responses are enabled only through -`CellGroupDispatcher::with_s3` and that exact controller/runtime's archive owner. -A complete published cut is physically replayed, then its immutable SQL receipt -must match the exact command/history/version/result. Later images can prove older -receipts. Catalog birth and live leader authority are rechecked before the token. -Request, namespace and stored durability minima are never lowered. - -A committed producer resolves before its S3 wait. Independent S3 admission takes -result retention before SQL releases its operation/byte permit; existing quotas -remain unchanged. One fair archive lane batches at most 32 proofs per group turn. -Successful outputs retain complete physical coverage, S3 admission and the journal -lock, and weak owner uses prevent premature predecessor-hold retirement. Those -active consumers can backpressure the shared replay pools; idle groups retain no -workspace. Cancellation/deadlines keep their original identity and cannot rerun -callbacks or quarantine already-resolved SQL ownership. - -Seven exact RustFS cases pass in both image forms, plus seven signed backend probes. -All fourteen check/regression commands pass, covering 193 Rust cases across six +The latest partial runtime/qualification source is `c2801b6586ffc8b3b765312e0042986e881d773a`. +CG-06 remains IN_PROGRESS. Local command and enrollment S3 responses require an +explicit `CellGroupDispatcher::with_s3` binding to the exact controller/runtime +archive owner. A complete published cut is independently recovered before proving +the exact immutable command receipt or original enrollment creation record. +Later images can prove older receipts/creation records. Current catalog and actual +leader authority are rechecked under the original deadline; namespace minima are +never lowered. Enrollment returns original provenance, not a current-version token. + +Resolved SQL ownership clears before archival. Separate bounded S3 admission takes +output retention before SQL releases its bytes/operation permit. One fair node +archive lane handles both output kinds, at most 32 proofs per group turn. +Successful outputs retain complete physical coverage, shared pool admission and +the exclusive journal lock; weak owner uses prevent premature hold retirement. +Active consumers backpressure those pools; idle groups retain no replay workspace. +Archive failure/deadline cannot quarantine a resolved initializer or command. +Exact creation retries return the original record without initialization replay. + +Nine exact RustFS cases pass in both image forms, plus seven signed backend probes. +Fourteen check/regression commands pass with 195 distinct Rust cases across six copied executables, all-target Clippy, format/layout, dependency/docs/preflight/plan -checks, 37 Python tests and system-Python hashing. Source/artifact/exact-selection -and credential-exclusion audits pass. Retained records, frozen inputs and the -initial unchanged-quota retention failure are under -`.cellar/cell-groups-evidence/CG-06-s3-commands/`. See the newest -[progress](progress.md), [S3 response contract](protocols.md#local-command-s3-response-gate) +checks, 37 Python tests and system-Python hashing. Independent source/artifact/ +selection and credential-exclusion audits pass. Records, frozen inputs and exact +executables are under `.cellar/cell-groups-evidence/CG-06-s3-enrollment/`. +See the newest [progress](progress.md), [enrollment proof contract](protocols.md#local-enrollment-s3-response-gate) and [RustFS runner](../../../../tools/multi-raft/cell_groups.py). -Next integrate S3 enrollment, receipt lookup and remote response settlement; -those paths still refuse unsupported S3 capabilities. Add explicit unarchived-byte/ -disk limits, then physical local Raft Engine file purge and fenced fresh-incarnation -serving. Retention across multiple heads, controller process-kill restart and -physical-read shutdown cuts need qualification. The fence inventory must cover -every possible old voting lineage, including joins after the recovery cut. -Fencing-authority API plus Docker qualification remains the accepted direction. -CG-08 effect/cache/result lifetime and maintenance precedes CG-07 public SDK/fleet -cutover. Topology, placement, hibernation, managed recovery, measured load and final -legacy removal remain. The source/runtime goal is not complete. - -Namespace bindings, stable runtime cut capture/upload, exact-intent journals, -protected publication/remote collection and source-independent complete recovery -retain their earlier contracts. Persist every exact intent before mutation; -unknown outcomes never allocate another identity or reset counters. The prior -RustFS qualification at `d739bf79` and runtime source `7914177d`, catalog-cut -`df0ba1f6`, protected publication `87348477`, isolated verifier `14955f2d` and -committed tail `7a29c2a9` remain retained. Current voters are host-native tasks with -one local RustFS VM; no node-container resource gate, independent hosts, measured -load, fenced fresh-quorum disaster serving or complete hard cutover is claimed. -Four original CG-06 acceptance cases and its full task gate remain pending. +Next implement local S3 receipt lookup: no absent row establishes non-commit, and +later archived observations must not falsely attest an older absence. Preserve +exact identity, typed expiry/conflict and output admission. Then integrate remote +output proofs with lifetime through physical delivery. Those S3 wire paths still +refuse unsupported capabilities; remote enrollment cannot downgrade to Quorum. +Add explicit unarchived-byte/disk admission, physical shared-engine Raft file purge +and fenced fresh-incarnation serving with complete old-voter lineage inventory, +including joins after the recovery cut. Fencing-authority API plus Docker +qualification remains the accepted direction. Multi-head proof retention, +controller process-kill restart and physical-read shutdown cuts need qualification. + +CG-08 effect/cache/result lifetimes and maintenance precede CG-07 public SDK/fleet +cutover, followed by topology, placement, hibernation, managed recovery, measured +load and final legacy removal. Four original CG-06 cases and its full task gate +remain pending. Voters are host-native tasks with one local RustFS VM; independent +hosts, container node limits, fenced replacement-quorum serving, measured load +and full hard cutover remain unqualified. The end-to-end goal is not complete. + +Preserve the preceding local command source `3d564f3d`, RustFS source `d739bf79`, +controller source `7914177d`, catalog-cut `df0ba1f6`, protected publication +`87348477`, isolated verifier `14955f2d`, committed tail `7a29c2a9` and their +source-bound evidence/failures. Exact intents precede remote mutation; unknown +outcomes never allocate a new identity or reset counters. Namespace binding, +stable cut capture, protected publication and source-independent recovery retain +their existing contracts. The controller initially coalesces complete cuts; +incremental scheduling remains future work. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 396fa4114..fb8c6b77e 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,64 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 local enrollment S3 response settlement + +Source: `c2801b6586ffc8b3b765312e0042986e881d773a`. CG-06 remains IN_PROGRESS. The explicit +`CellGroupDispatcher::with_s3` binding now also settles local enrollment responses. +The existing fair archive lane owns both output kinds under independent bounded +S3 admission. Enrollment reserves its bounded original creation record before +releasing resolved SQL output retention; no original SQL/command quota changes. + +Required durability is the maximum of request and immutable namespace minimum, +captured in the invocation fingerprint. Exact runtime ownership and active archive +registration precede initialization. After commitment, producer/preparation +ownership clears before the archive wait. A complete independently recovered group +must contain the exact history, creation ID, schema and initial image commitment. +An older cut waits; conflicting creation refuses. Subsequent mutations preserve +that original provenance without retaining superseded initial bytes. Proof and +exact retry never rerun initialization. + +The response rechecks catalog birth, actual leader authority and its exact current +creation record under the original deadline. `CellSqlEnrollment::durability` +reports S3 only while retaining its private complete-coverage proof, pool admission +and journal lock. Active outputs backpressure the shared proof pools until release; +idle groups retain no replay workspace. Remote enrollment explicitly refuses S3 +before transmission until its wire contract is integrated. + +Nine exact RustFS cases pass in both image formats. The two new cases also exercise +request and namespace minimums, a foreign runtime refusing before initialization, +publication-held output, a later same-cell Quorum mutation during the S3 wait, +retained proof slots, original-record duplicates and conflicting creation IDs. +Deadline and dropped-reply branches keep the one-command/one-SQL-slot quotas, +refund admission, allow a Quorum successor during archive outage and retry the +same creation with exactly one initializer invocation. Seven signed S3 probes +and all earlier local command/archive/refusal cases pass. + +All fourteen regression/check commands pass: 195 distinct Rust cases across six +copied executables, locked all-target Clippy for Cellar/logic/LTX, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +The retained records under `.cellar/cell-groups-evidence/CG-06-s3-enrollment/` +bind 2566 source hashes, frozen plan inputs, exact selections, executable +and output hashes, orchestration sources and prior qualification. Independent +source/hash/selection and credential-exclusion audit passes. No original task +case, oracle or pre-existing quota was weakened. The initial fixture-allocation +Clippy refusal and its exact source/log remain retained; the correction changes +only two literals. Four original CG-06 cases and its full gate remain pending; +this is partial task evidence. + +Reflection: finish local receipt lookup, preserving typed expiry/conflict and the +ambiguity of an absent row, then remote command/enrollment/receipt output proofs +with retention through physical delivery. Add explicit unarchived-byte/disk +admission, physical shared-engine Raft file purge and fenced new-incarnation +serving with complete old-voter lineage inventory. Qualify multi-head proof +retention, controller process-kill restart and physical-read shutdown cuts. +A performance opportunity is incremental committed-tail scheduling and smaller +sealed output attestations backed by real hold leases, qualified under load. +CG-08 lifetimes/maintenance precede CG-07 public SDK/fleet cutover; topology, +placement, hibernation, managed recovery, load and final legacy removal remain. +Voters share one host and RustFS one local VM; independent hosts, measured load, +fenced replacement-quorum serving and the full hard cutover remain unqualified. + ## CG-06 local command S3 response settlement Source: `3d564f3db2c2eefc87119e7b3395f1bca17b99fa`. CG-06 remains IN_PROGRESS. An explicit From 7e1d93a00d68038fb6aa40a59dc29885f26c62b0 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 15:20:35 -0700 Subject: [PATCH 192/568] feat: prove S3 receipt lookup results --- crates/cellar/durability/raft/budget.rs | 9 + .../raft/cell_groups/archive/receipt.rs | 119 +++- .../controller/archive/settlement.rs | 231 ++++++-- .../raft/cell_groups/controller/mod.rs | 9 +- .../raft/cell_groups/dispatch/command/mod.rs | 10 + .../cell_groups/dispatch/command/remote.rs | 7 + .../raft/cell_groups/dispatch/enrollment.rs | 5 + .../raft/cell_groups/dispatch/mod.rs | 10 +- .../raft/cell_groups/dispatch/receipt.rs | 205 ++++++- .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../raft/host_sql/cell_groups/mod.rs | 8 +- .../raft/host_sql/cell_groups/query.rs | 23 +- crates/cellar/durability/raft/receipts.rs | 27 +- .../native_cell_groups_archive_controller.rs | 113 ++++ .../tests/support/cell_group_runtime.rs | 3 +- .../tests/support/cell_group_s3_commands.rs | 56 +- .../tests/support/cell_group_s3_receipts.rs | 510 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 3 +- .../multi-raft/cell-groups/protocols.md | 45 +- docs/design/multi-raft/cell-groups/tasks.json | 3 +- tools/multi-raft/README.md | 12 +- tools/multi-raft/cell_groups.py | 10 +- 22 files changed, 1296 insertions(+), 123 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_s3_receipts.rs diff --git a/crates/cellar/durability/raft/budget.rs b/crates/cellar/durability/raft/budget.rs index afe737fd4..a750b8971 100644 --- a/crates/cellar/durability/raft/budget.rs +++ b/crates/cellar/durability/raft/budget.rs @@ -79,6 +79,12 @@ pub(crate) struct Permit { } impl Permit { + /// Ensure an admitted read can retain its payload before allocating it. + /// Repeated or smaller requests keep the same bytes and item slot. + pub(crate) fn grow_at_least(&mut self, bytes: usize) -> bool { + bytes <= self.bytes || self.grow_to(bytes) + } + /// Extend an admitted operation without consuming another item slot. /// Failure leaves its existing reservation unchanged. pub(crate) fn grow_to(&mut self, bytes: usize) -> bool { @@ -218,6 +224,9 @@ mod tests { assert_eq!(budget.usage(), (100, 2)); assert!(!pin.grow_to(31)); assert!(!pin.grow_to(29)); + assert!(pin.grow_at_least(29)); + assert!(pin.grow_at_least(30)); + assert!(!pin.grow_at_least(31)); assert_eq!(budget.usage(), (100, 2)); drop(other); assert!(pin.grow_to(100)); diff --git a/crates/cellar/durability/raft/cell_groups/archive/receipt.rs b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs index 38fb4a125..859cea6b2 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/receipt.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/receipt.rs @@ -2,8 +2,10 @@ //! Read the exact receipt from independently recovered, immutable archive bytes. use super::*; -use crate::raft::cell_groups::{CellGroupApplyImages, CellImagePin}; -use crate::raft::execution::{CellCommittedReceipt, CommandRequest, ExecutionLimits}; +use crate::raft::cell_groups::{CellGroupApplyImages, CellImagePin, CellSqlReceiptLookup}; +use crate::raft::execution::{ + CellCommittedReceipt, CommandRequest, ExecutionError, ExecutionLimits, +}; use std::sync::atomic::{AtomicBool, Ordering}; pub(in crate::raft::cell_groups) fn command( @@ -36,6 +38,95 @@ pub(in crate::raft::cell_groups) fn command( if leaf.version == expected.version && leaf.witness != expected.witness { return Err(CellGroupArchiveError::Invalid("S3 receipt decision")); } + let database = open(store, pin, deadline, cancelled)?; + // The pending output already owns the original bytes. Bound the additional + // row allocation before lookup; identity and the entire retained row match. + limits.max_result_bytes = expected.receipt.result().len(); + let request = CommandRequest { + id: expected.receipt.id, + operation: expected.receipt.operation.clone(), + contract_digest: expected.receipt.contract_digest, + request_digest: expected.receipt.request_digest, + durability: expected.receipt.minimum_durability, + retry_until_ms: expected.receipt.retry_until_ms, + }; + let receipt = crate::raft::receipts::lookup_cell_identity( + database.connection(), + &request, + &limits, + leaf.version.sequence.get(), + ) + .map_err(CellGroupArchiveError::Replay)?; + if receipt.as_ref() != Some(&expected.receipt) { + return Err(CellGroupArchiveError::Invalid("S3 receipt identity/result")); + } + store.require_live()?; + Ok(true) +} + +pub(in crate::raft::cell_groups) struct LookupRead<'a> { + pub request: &'a CommandRequest, + pub limits: ExecutionLimits, + pub deadline: u64, + pub cancelled: Arc, +} + +/// Return the actual archived observation at or beyond the quorum selection. +/// Absence remains ambiguous; it never resolves a submitted producer. +pub(in crate::raft::cell_groups) fn lookup( + coverage: &CellGroupArchiveCoverage, + store: &CellGroupArchiveStore, + selected: &mut CellSqlReceiptLookup, + read: LookupRead<'_>, + reserve_result: impl FnMut(usize) -> std::result::Result<(), ExecutionError>, +) -> Result { + store.require_live()?; + let root = coverage.recovery().manifest(); + if root.marker().route() != store.route() || !store.owner.owns(root) { + return Err(CellGroupArchiveError::Invalid( + "S3 lookup verifier owner/route", + )); + } + let floor = selected.observation(); + let Some((leaf, pin)) = root.pin_cell(&floor.version.history.cell) else { + return Ok(false); + }; + if leaf.version.history != floor.version.history { + return Err(CellGroupArchiveError::Invalid("S3 lookup history")); + } + if leaf.version.sequence < floor.version.sequence { + return Ok(false); + } + if leaf.version == floor.version && leaf.witness != floor.witness { + return Err(CellGroupArchiveError::Invalid("S3 lookup decision")); + } + let database = open(store, pin, read.deadline, read.cancelled)?; + let receipt = crate::raft::receipts::lookup_cell_identity_admitted( + database.connection(), + read.request, + &read.limits, + leaf.version.sequence.get(), + reserve_result, + ) + .map_err(CellGroupArchiveError::Replay)?; + // An equal logical version cannot disagree with the quorum read. A known + // retained result also remains exact across later images; expiry stays typed. + if (leaf.version == floor.version || selected.receipt().is_some()) + && receipt.as_ref() != selected.receipt() + { + return Err(CellGroupArchiveError::Invalid("S3 lookup identity/result")); + } + store.require_live()?; + selected.select_archived(receipt, leaf.observation()); + Ok(true) +} + +fn open( + store: &CellGroupArchiveStore, + pin: CellImagePin, + deadline: u64, + cancelled: Arc, +) -> Result { let images = store .storage .try_lock() @@ -86,27 +177,5 @@ pub(in crate::raft::cell_groups) fn command( }), ) .map_err(|e| CellGroupArchiveError::Replay(crate::raft::execution::storage(e)))?; - // The pending output already owns the original bytes. Bound the additional - // row allocation before lookup; identity and the entire retained row match. - limits.max_result_bytes = expected.receipt.result().len(); - let request = CommandRequest { - id: expected.receipt.id, - operation: expected.receipt.operation.clone(), - contract_digest: expected.receipt.contract_digest, - request_digest: expected.receipt.request_digest, - durability: expected.receipt.minimum_durability, - retry_until_ms: expected.receipt.retry_until_ms, - }; - let receipt = crate::raft::receipts::lookup_cell_identity( - database.connection(), - &request, - &limits, - leaf.version.sequence.get(), - ) - .map_err(CellGroupArchiveError::Replay)?; - if receipt.as_ref() != Some(&expected.receipt) { - return Err(CellGroupArchiveError::Invalid("S3 receipt identity/result")); - } - store.require_live()?; - Ok(true) + Ok(database) } diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs index f0c282541..4c6e9ffb5 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -3,10 +3,12 @@ //! Operation proofs share the archive lane; waiting never owns a SQL producer. use super::*; use crate::raft::budget::Permit; -use crate::raft::cell_groups::{CellDispatchError, CellSqlEnrollment, CellSqlReceipt}; -use crate::raft::execution::ExecutionLimits; +use crate::raft::cell_groups::{ + CellDispatchError, CellSqlEnrollment, CellSqlReceipt, CellSqlReceiptLookup, +}; +use crate::raft::execution::{CommandRequest, ExecutionError, ExecutionLimits}; use cellar_logic::cell_group::GroupRoute; -use std::sync::{atomic::AtomicBool, Weak}; +use std::sync::{atomic::AtomicBool, Mutex, Weak}; #[derive(Clone)] pub(crate) struct Client(Weak); @@ -16,7 +18,7 @@ pub(crate) struct OutputProof { // The exclusive journal lock survives response buffering and node close. // A replacement owner cannot retire this hold while a consumer still uses it. _journal: Arc, - _permit: Arc, + _permit: Arc>, authority: Arc, } @@ -39,9 +41,36 @@ pub(crate) struct EnrollmentSettled { pub proof: Box, } +pub(crate) struct LookupSettled { + pub selected: Box, + pub proof: Box, +} + +struct Lookup { + selected: Box, + request: CommandRequest, +} + +/// Admitted immutable observation; no node work is queued until response. +pub(crate) struct LookupPrepared(Prepared); + +impl LookupPrepared { + pub(crate) async fn response(self) -> std::result::Result { + let settled = self.0.response().await?; + let Output::Lookup(lookup) = settled.output else { + return Err(CellDispatchError::Invalid("S3 lookup output")); + }; + Ok(LookupSettled { + selected: lookup.selected, + proof: settled.proof, + }) + } +} + enum Output { Command(Box), Enrollment(Box), + Lookup(Box), } impl Output { @@ -51,16 +80,24 @@ impl Output { // Creation records are bounded by the protocol to 4096 bytes. Their // variable identity bytes and the archive comparison stay admitted. Self::Enrollment(_) => 4096, + Self::Lookup(lookup) => lookup + .selected + .receipt() + .map_or(0, |row| row.result().len()), } .checked_mul(2) .and_then(|n| n.checked_add(8192)) .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded)) } - fn retain(self, permit: Arc) -> Self { + fn retain(self, permit: Arc>) -> Self { match self { Self::Command(receipt) => Self::Command(receipt.retain_for_archive(permit)), Self::Enrollment(enrollment) => Self::Enrollment(enrollment.retain_for_archive(permit)), + Self::Lookup(lookup) => Self::Lookup(Box::new(Lookup { + selected: lookup.selected.retain_for_archive(permit), + request: lookup.request, + })), } } } @@ -77,7 +114,7 @@ pub(in crate::raft::cell_groups::controller) struct Request { deadline: u64, cancelled: Arc, reply: oneshot::Sender>, - permit: Arc, + permit: Arc>, } struct Cancel(Arc); @@ -195,6 +232,25 @@ impl Client { }) } + pub(crate) fn prepare_lookup( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + selected: Box, + request: CommandRequest, + limits: ExecutionLimits, + deadline: u64, + ) -> std::result::Result { + self.prepare( + runtime, + route, + Output::Lookup(Box::new(Lookup { selected, request })), + Some(limits), + deadline, + ) + .map(LookupPrepared) + } + async fn settle( &self, runtime: &CellGroupRuntime, @@ -203,6 +259,19 @@ impl Client { limits: Option, deadline: u64, ) -> std::result::Result { + self.prepare(runtime, route, output, limits, deadline)? + .response() + .await + } + + fn prepare( + &self, + runtime: &CellGroupRuntime, + route: GroupRoute, + output: Output, + limits: Option, + deadline: u64, + ) -> std::result::Result { self.require_runtime(runtime)?; self.require_group(&route)?; let shared = self @@ -212,12 +281,11 @@ impl Client { let permit = shared .s3 .acquire(output.retained_bytes()?) - .map(Arc::new) + .map(|permit| Arc::new(Mutex::new(permit))) .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; let output = output.retain(permit.clone()); let (reply, receiver) = oneshot::channel(); let cancelled = Arc::new(AtomicBool::new(false)); - let _cancel = Cancel(cancelled.clone()); let now = shared .domain .mono_ms() @@ -225,17 +293,56 @@ impl Client { if now >= deadline { return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); } - shared - .requests - .try_send(requests::Request::S3(Box::new(Request { + Ok(Prepared { + client: self.clone(), + request: Request { route, output, limits, deadline, - cancelled, + cancelled: cancelled.clone(), reply, permit, - }))) + }, + receiver, + _cancel: Cancel(cancelled), + }) + } +} + +struct Prepared { + client: Client, + request: Request, + receiver: oneshot::Receiver>, + _cancel: Cancel, +} + +impl Prepared { + async fn response(self) -> std::result::Result { + let Self { + client, + request, + receiver, + _cancel, + } = self; + let shared = client + .0 + .upgrade() + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed))?; + let deadline = request.deadline; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let now = shared + .domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + shared + .requests + .try_send(requests::Request::S3(Box::new(request))) .map_err(|e| { CellDispatchError::Runtime(match e { tokio::sync::mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, @@ -317,32 +424,73 @@ pub(super) async fn prove( let store = registration.store.clone(); let physical = coverage.clone(); let read = execution.blocking(move || { - let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| match &request - .output - { - Output::Command(receipt) => crate::raft::cell_groups::archive::receipt::command( - &physical, - &store, - receipt.receipt(), - request - .limits - .ok_or(CellGroupArchiveError::Invalid("S3 command limits"))?, - request.deadline, - request.cancelled.clone(), - ), - Output::Enrollment(enrollment) => { - crate::raft::cell_groups::archive::enrollment::verify( - &physical, - &store, - enrollment.record(), - ) - } - })) - .unwrap_or_else(|panic| { - Err(CellGroupArchiveError::Replay( - crate::raft::execution::storage(crate::native_worker::panic_error(panic)), - )) - }); + let mut request = request; + let result = + std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + match &mut request.output { + Output::Command(receipt) => { + crate::raft::cell_groups::archive::receipt::command( + &physical, + &store, + receipt.receipt(), + request + .limits + .ok_or(CellGroupArchiveError::Invalid("S3 command limits"))?, + request.deadline, + request.cancelled.clone(), + ) + } + Output::Enrollment(enrollment) => { + crate::raft::cell_groups::archive::enrollment::verify( + &physical, + &store, + enrollment.record(), + ) + } + Output::Lookup(lookup) => { + let original = lookup + .selected + .receipt() + .map_or(0, |row| row.result().len()); + crate::raft::cell_groups::archive::receipt::lookup( + &physical, + &store, + &mut lookup.selected, + crate::raft::cell_groups::archive::receipt::LookupRead { + request: &lookup.request, + limits: request.limits.ok_or( + CellGroupArchiveError::Invalid("S3 lookup limits"), + )?, + deadline: request.deadline, + cancelled: request.cancelled.clone(), + }, + |length| { + let bytes = length + .max(original) + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded, + ))?; + let mut permit = request.permit.lock().map_err(|_| { + ExecutionError::Invalid("S3 result admission") + })?; + if !permit.grow_at_least(bytes) { + return Err(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::Overloaded, + )); + } + Ok(()) + }, + ) + } + } + })) + .unwrap_or_else(|panic| { + Err(CellGroupArchiveError::Replay( + crate::raft::execution::storage(crate::native_worker::panic_error(panic)), + )) + }); (request, result) }); // A refused submission drops the closure and its owned request. Entered @@ -362,6 +510,11 @@ pub(super) async fn prove( } match result { Ok(false) | Err(CellGroupArchiveError::Busy) => waiting.push(request), + Err(CellGroupArchiveError::Replay(error)) + if matches!(&request.output, Output::Lookup(_)) => + { + let _ = request.reply.send(Err(CellDispatchError::Execution(error))); + } Err(error) => request.fail(Arc::new(Error::Archive(error))), Ok(true) => { if let Err(error) = context.authority.require_current().await { diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 3dbfb93e6..91d4ad6c4 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -4,7 +4,9 @@ //! Admitted command ownership survives response cancellation and expiry. mod archive; -pub(crate) use archive::settlement::{Client as S3Client, OutputProof as S3OutputProof}; +pub(crate) use archive::settlement::{ + Client as S3Client, LookupPrepared as S3LookupPrepared, OutputProof as S3OutputProof, +}; mod calls; pub use archive::{ CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, @@ -50,8 +52,9 @@ pub struct CellGroupControllerLimits { /// Shared by commands, enrollments and registrations, including queued work, resolution /// and unconsumed responses. pub max_commands: usize, - /// Invocation/input/options retention. SQL capture, apply and result bytes - /// retain the SQL pool's independent admission as well. + /// Invocation/input/options retention. The independent S3 output stage has + /// the same byte ceiling; configure their combined envelope as twice this + /// value. SQL capture/apply/results retain separate SQL pool admission. pub max_command_bytes: usize, /// Automatic compaction at an apply boundary. Requires the transport's /// shared checkpoint pool. None leaves local publication to its caller. diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index e8c6045d6..9abf83a99 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -366,6 +366,11 @@ impl CellCommandInvocation { return Err(CellDispatchError::Invalid("destination route binding")); } if self.request.durability == Durability::S3 { + if self.remote_route.is_some() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } self.dispatcher .s3 .as_ref() @@ -665,6 +670,11 @@ impl CellCommandInvocation { .durability .max(receipt.receipt().receipt.minimum_durability()); let (receipt, s3, authority) = if required == Durability::S3 { + if self.remote_route.is_some() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } let owner = self .dispatcher .s3 diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs index 640090b88..dd6c127e9 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -28,6 +28,13 @@ impl CellCommandInvocation { "consumed or local command attempt", ))); } + if self.request.durability == Durability::S3 { + return Err( + self.error(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))), + ); + } if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(self.error(CellDispatchError::Invalid("command transport domain"))); } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index 57f79eada..7b399af98 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -265,6 +265,11 @@ impl CellEnrollmentInvocation { return Err(CellDispatchError::Invalid("enrollment destination route")); } if self.durability == Durability::S3 { + if self.remote_route.is_some() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } let owner = self .dispatcher .s3 diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 49d6d375f..37d589531 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -4,7 +4,9 @@ //! Routing observations never replace destination authority or ingress grants. mod receipt; -pub use receipt::{CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse}; +pub use receipt::{ + CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse, CellReceiptSelection, +}; mod enrollment; pub use enrollment::{CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest}; @@ -147,9 +149,9 @@ impl CellGroupDispatcher { }) } - /// Bind local command/enrollment S3 settlement to this node's bounded archive owner. - /// The destination group must already have an archive registration. Receipt - /// lookup and remote response settlement have separate gates. + /// Bind local command, enrollment and receipt S3 settlement to the node archive owner. + /// The destination group must already have an archive registration. Remote + /// response settlement has a separate gate. pub fn with_s3(mut self, controller: &CellGroupController) -> Result { self.s3 = Some( controller diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs index 90387284e..f7828b9f0 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -3,7 +3,7 @@ //! Receipt-only dispatch never stages SQL or turns absence into non-commit. use super::*; use crate::application::consensus::command_request_digest; -use crate::raft::execution::{CommandRequest, StoredReceipt}; +use crate::raft::execution::{CommandRequest, ExecutionLimits, StoredReceipt}; use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; #[derive(Clone)] @@ -27,16 +27,49 @@ pub struct CellReceiptInvocation { pub struct CellReceiptResponse { command: CommandId, - selected: CellSqlReceiptLookup, + selected: Box, token: CellVersionToken, + s3: Option>, +} + +/// One coherent quorum selection, bound to its original operation and deadline. +/// No result escapes before `response` satisfies the required durability. S3 +/// admission takes SQL retention here; dropping this phase releases admission. +pub struct CellReceiptSelection<'a> { + invocation: &'a mut CellReceiptInvocation, + runtime: CellGroupRuntime, + attempt: CellRouteAttempt, + selected: Selected, +} + +enum Selected { + Quorum(Box), + S3(Box), +} + +impl CellReceiptSelection<'_> { + pub async fn response(self) -> Result { + self.invocation + .response(&self.runtime, self.attempt, self.selected) + .await + } } impl CellReceiptResponse { + /// Achieved durability of this observation, including an ambiguous absence. + pub fn durability(&self) -> Durability { + if self.s3.is_some() { + Durability::S3 + } else { + Durability::Quorum + } + } + pub fn command(&self) -> CommandId { self.command } - /// None is ambiguous, even after a quorum read: an unresolved or delayed + /// None is ambiguous, even after a quorum or S3 read: an unresolved or delayed /// original command may still commit. This response releases no producer. pub fn receipt(&self) -> Option<&StoredReceipt> { self.selected.receipt() @@ -59,9 +92,10 @@ impl CellGroupDispatcher { limits: CellRoutingLimits, cached: Option, ) -> Result { - if request.durability != Durability::Quorum - || self.directory.namespace_config().minimum_durability != Durability::Quorum - { + request.durability = request + .durability + .max(self.directory.namespace_config().minimum_durability); + if request.durability == Durability::S3 && self.s3.is_none() { return Err(CellDispatchError::Execution(ExecutionError::Refused( CommandFailure::UnsupportedCapability, ))); @@ -101,7 +135,13 @@ impl CellGroupDispatcher { hash.update(request.contract.operation.as_bytes()); hash.update(request.contract.digest); hash.update(command_request_digest(&request.input)); - hash.update([1, limits.max_attempts]); + hash.update([ + match request.durability { + Durability::Quorum => 1, + Durability::S3 => 2, + }, + limits.max_attempts, + ]); hash.update(limits.timeout.as_nanos().to_be_bytes()); let session = self .directory @@ -198,6 +238,18 @@ impl CellReceiptInvocation { sql: &CellGroupSqlPool, options: CellSqlOptions, ) -> Result { + self.select(runtime, sql, options).await?.response().await + } + + /// Select one immutable quorum image under the original attempt/deadline. + /// S3 selection transfers result retention before returning this opaque phase; + /// only its response can expose the required-durability result. + pub async fn select( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + ) -> Result, CellDispatchError> { let attempt = self.current()?.clone(); self.session .validate_attempt(&attempt) @@ -211,6 +263,32 @@ impl CellReceiptInvocation { { return Err(CellDispatchError::Invalid("receipt execution domain")); } + if self.request.durability == Durability::S3 { + if self.remote_route.is_some() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Invalid("S3 receipt owner"))?; + owner.require_runtime(runtime)?; + owner.require_group(attempt.route())?; + } + let limits = match &options { + CellSqlOptions::Materialized(o) => o.limits, + CellSqlOptions::Extents(o) => o.limits, + }; + let request = CommandRequest { + id: self.request.id, + operation: self.request.contract.operation.clone(), + contract_digest: self.request.contract.digest, + request_digest: command_request_digest(&self.request.input), + durability: self.request.durability, + retry_until_ms: 0, + }; let authority = runtime .read_authority( attempt.route().history, @@ -228,26 +306,96 @@ impl CellReceiptInvocation { options, authority, self.dispatcher.class.clone(), - CommandRequest { - id: self.request.id, - operation: self.request.contract.operation.clone(), - contract_digest: self.request.contract.digest, - request_digest: command_request_digest(&self.request.input), - durability: self.request.durability, - retry_until_ms: 0, - }, + request.clone(), self.request.input.clone(), ) .await .map_err(CellDispatchError::Execution)?; - if selected - .receipt() - .is_some_and(|r| r.minimum_durability() != Durability::Quorum) - { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + let required = self.request.durability.max( + selected + .receipt() + .map_or(Durability::Quorum, |row| row.minimum_durability()), + ); + let selected = if required == Durability::S3 { + if self.remote_route.is_some() { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + Selected::S3(Box::new(owner.prepare_lookup( + runtime, + attempt.route().clone(), + Box::new(selected), + request, + limits, + attempt.deadline_ms(), + )?)) + } else { + Selected::Quorum(Box::new(selected)) + }; + Ok(CellReceiptSelection { + invocation: self, + runtime: runtime.clone(), + attempt, + selected, + }) + } + + async fn response( + &mut self, + runtime: &CellGroupRuntime, + attempt: CellRouteAttempt, + selected: Selected, + ) -> Result { + self.session + .validate_attempt(&attempt) + .map_err(CellDispatchError::Routing)?; + let (selected, s3, authority) = match selected { + Selected::Quorum(selected) => (selected, None, None), + Selected::S3(prepared) => { + let settled = prepared.response().await?; + settled.proof.require_current().await?; + // Use a fresh read authority, whose leader fence tolerates later + // cell writes, rather than retaining a private writer's base. + let authority = runtime + .read_authority( + attempt.route().history, + self.cell.clone(), + CellReadConsistency::Linearizable, + attempt.deadline_ms(), + ) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != attempt.route() + || authority.cell().version.history != self.cell + { + return Err(CellDispatchError::Invalid( + "S3 receipt current route/history", + )); + } + if authority.cell().version.sequence + < settled.selected.observation().version.sequence + { + return Err(CellDispatchError::Runtime(RuntimeError::NotReady)); + } + if authority.cell().version == settled.selected.observation().version + && authority.cell().witness != settled.selected.observation().witness + { + return Err(CellDispatchError::Invalid("S3 receipt current decision")); + } + (settled.selected, Some(settled.proof), Some(authority)) + } + }; let now = self .dispatcher .domain @@ -262,10 +410,16 @@ impl CellReceiptInvocation { self.session .validate_attempt(&attempt) .map_err(CellDispatchError::Routing)?; + if let Some(authority) = authority { + authority + .check(self.dispatcher.domain.mono_ms().map_err(|_| closed())?) + .map_err(CellDispatchError::Runtime)?; + } Ok(CellReceiptResponse { command: self.request.id, selected, token, + s3, }) } @@ -281,6 +435,11 @@ impl CellReceiptInvocation { if self.dispatched || self.remote_route.is_some() { return Err(CellDispatchError::Invalid("consumed receipt attempt")); } + if self.request.durability == Durability::S3 { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + ))); + } self.dispatched = true; if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(CellDispatchError::Invalid("receipt transport domain")); diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index fb08b4406..6a5b9392d 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -53,6 +53,7 @@ pub use dispatch::{ CellDispatchLimits, CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest, CellGroupDispatcher, CellQueryConsistency, CellQueryInvocation, CellQueryRequest, CellQueryResponse, CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse, + CellReceiptSelection, }; mod execution_authority; diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 49fef4179..a967a9ab6 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -30,7 +30,7 @@ use cellar_logic::cell_group::{CellHistory, GroupHistory}; use cellar_logic::consensus::{CommandFailure, ProposalId}; use std::sync::{ atomic::{AtomicBool, Ordering}, - Arc, + Arc, Mutex, }; use tokio::sync::{oneshot, Notify}; pub(super) use worker::{Cells, Request}; @@ -171,7 +171,7 @@ impl CellSqlEnrollment { pub(crate) fn retain_for_archive( mut self: Box, - permit: Arc, + permit: Arc>, ) -> Box { self._retention = OutputRetention::Archive { _permit: permit }; self @@ -272,7 +272,7 @@ enum OutputRetention { _bytes: Arc, }, Archive { - _permit: Arc, + _permit: Arc>, }, } @@ -281,7 +281,7 @@ impl CellSqlReceipt { /// archive owner. Acquire that reservation before releasing SQL retention. pub(crate) fn retain_for_archive( mut self: Box, - permit: Arc, + permit: Arc>, ) -> Box { self._retention = OutputRetention::Archive { _permit: permit }; self diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index 883a85a34..ca70b54a8 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -58,10 +58,27 @@ impl CellSqlQueryResult { pub struct CellSqlReceiptLookup { receipt: Option, observation: crate::raft::version::CellVersionObservation, - _bytes: Arc, + _retention: OutputRetention, } impl CellSqlReceiptLookup { + pub(crate) fn retain_for_archive( + mut self: Box, + permit: Arc>, + ) -> Box { + self._retention = OutputRetention::Archive { _permit: permit }; + self + } + + pub(crate) fn select_archived( + &mut self, + receipt: Option, + observation: crate::raft::version::CellVersionObservation, + ) { + self.receipt = receipt; + self.observation = observation; + } + pub fn receipt(&self) -> Option<&crate::raft::execution::StoredReceipt> { self.receipt.as_ref() } @@ -403,7 +420,9 @@ pub(super) fn run( Output::Receipt(receipt) => CellSqlOutcome::ReceiptLookedUp(CellSqlReceiptLookup { receipt, observation, - _bytes: query.bytes, + _retention: OutputRetention::Sql { + _bytes: query.bytes, + }, }), }) } diff --git a/crates/cellar/durability/raft/receipts.rs b/crates/cellar/durability/raft/receipts.rs index 6a2df6c97..713b3a287 100644 --- a/crates/cellar/durability/raft/receipts.rs +++ b/crates/cellar/durability/raft/receipts.rs @@ -128,6 +128,18 @@ pub(super) fn lookup_cell_identity( request: &CommandRequest, limits: &ExecutionLimits, cell_sequence: u64, +) -> Result, ExecutionError> { + lookup_cell_identity_admitted(connection, request, limits, cell_sequence, |_| Ok(())) +} + +/// Reserve a retained result after inspecting its length, before fetching bytes. +/// Admission failure is not absence and consumes no additional operation slot. +pub(super) fn lookup_cell_identity_admitted( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + cell_sequence: u64, + mut reserve_result: impl FnMut(usize) -> Result<(), ExecutionError>, ) -> Result, ExecutionError> { let tables: u64 = connection.query_row( "SELECT count(*) FROM sqlite_schema WHERE type='table' AND name IN ('__sys_raft_clients','__sys_raft_receipts')", @@ -143,7 +155,10 @@ pub(super) fn lookup_cell_identity( })) } } - 2 => lookup_identity(connection, request, limits), + 2 => match inspect_identity_admitted(connection, request, limits, &mut reserve_result)? { + Identity::Retained(receipt) => Ok(Some(receipt)), + Identity::Vacant { .. } => Ok(None), + }, _ => Err(corrupt()), } } @@ -157,6 +172,15 @@ fn inspect_identity( connection: &Connection, request: &CommandRequest, limits: &ExecutionLimits, +) -> Result { + inspect_identity_admitted(connection, request, limits, &mut |_| Ok(())) +} + +fn inspect_identity_admitted( + connection: &Connection, + request: &CommandRequest, + limits: &ExecutionLimits, + reserve_result: &mut dyn FnMut(usize) -> Result<(), ExecutionError>, ) -> Result { let state = client(connection, request.id.client)?; let (last, floor) = state.unwrap_or((0, 0)); @@ -181,6 +205,7 @@ fn inspect_identity( if length.is_none_or(|n| n < 0 || n as u64 > limits.max_result_bytes as u64) { return Err(corrupt()); } + reserve_result(length.ok_or_else(corrupt)? as usize)?; let row=connection.query_row( "SELECT operation,contract_digest,request_digest,result,minimum_durability,retry_until FROM __sys_raft_receipts WHERE client_id=?1 AND sequence=?2", diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index 36b8cd89c..a5815f163 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -243,3 +243,116 @@ async fn rustfs_s3_enrollment_deadlines_preserve_resolved_ownership() -> Result< } Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_tracks_archived_observations() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_observations( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_tracks_archived_observations() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_observations( + rooted, + rustfs::store("s3-receipt-observations", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_honors_namespace_and_stored_minimums() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_minimums( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_honors_namespace_and_stored_minimums() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_minimums( + rooted, + rustfs::store("s3-receipt-minimums", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_preserves_typed_failures_and_result_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_failures( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_preserves_typed_failures_and_result_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_receipt_failures( + rooted, + rustfs::store("s3-receipt-failures", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_receipt_lookup_deadlines_refund_read_admission() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_receipt_deadline( + rooted, + Arc::new(object_store::memory::InMemory::new()), + cancel, + )) + .await?; + } + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_receipt_lookup_deadlines_refund_read_admission() -> Result<()> { + for rooted in [false, true] { + for cancel in [false, true] { + Box::pin(runtime_fixture::s3_receipt_deadline( + rooted, + rustfs::store( + if cancel { + "s3-receipt-cancel" + } else { + "s3-receipt-deadline" + }, + rooted, + )?, + cancel, + )) + .await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 501a0fb69..67b4e6c6c 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -816,7 +816,8 @@ pub(super) use sql::{owned_command_failure, owned_commands}; #[allow(unused_imports)] pub(super) use sql::{ s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, - s3_namespace_minimum, + s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, + s3_receipt_observations, }; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index fffc30d71..51a41e6e5 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -3,10 +3,15 @@ //! SQL output proofs on the actual controller and archive lane. #[path = "cell_group_s3_enrollment.rs"] mod enrollment; +#[path = "cell_group_s3_receipts.rs"] +mod receipts; use super::*; pub(crate) use enrollment::{s3_enrollment, s3_enrollment_deadline}; use futures_util::TryStreamExt; use object_store::{path::Path as ObjectPath, ObjectStore}; +pub(crate) use receipts::{ + s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, +}; use std::sync::atomic::{AtomicUsize, Ordering}; struct Harness { @@ -37,6 +42,16 @@ impl Harness { store: Arc, max_commands: usize, minimum: Durability, + ) -> Result { + Self::configured(rooted, store, max_commands, minimum, None).await + } + + async fn configured( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -50,11 +65,19 @@ impl Harness { .execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; Ok(()) })? - .command("Add", [82; 32], move |context, input| { - count.fetch_add(1, Ordering::SeqCst); - context.sql().execute("UPDATE value SET n=n+1", [])?; - Ok(input.to_vec()) - })?, + .command( + "Add", + if result_size.is_some() { + [84; 32] + } else { + [82; 32] + }, + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(result_size.map_or_else(|| input.to_vec(), |length| vec![7; length])) + }, + )?, ); let (mut fixture, route, cell, _) = Box::pin(registered::setup_store_minimum( rooted, &class, store, minimum, @@ -125,6 +148,24 @@ impl Harness { input: &[u8], durability: Durability, timeout: Duration, + ) -> Result { + self.command_using( + sequence, + input, + durability, + timeout, + self.fixture.cell_options(&self.cell).await?, + ) + .await + } + + async fn command_using( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + timeout: Duration, + options: CellSqlOptions, ) -> Result { let invocation = self .dispatcher @@ -144,10 +185,7 @@ impl Harness { None, ) .await?; - Ok( - self.controllers[0] - .command(invocation, self.fixture.cell_options(&self.cell).await?)?, - ) + Ok(self.controllers[0].command(invocation, options)?) } async fn waiting(&self) -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_s3_receipts.rs b/crates/cellar/tests/support/cell_group_s3_receipts.rs new file mode 100644 index 000000000..15ab27ff5 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_receipts.rs @@ -0,0 +1,510 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read-only S3 results at a later cut, with bounded payload allocation. +use super::*; + +impl Harness { + fn receipt_request( + &self, + sequence: u64, + input: &[u8], + durability: Durability, + ) -> Result { + Ok(CellReceiptRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + }) + } + + async fn lookup( + &self, + request: CellReceiptRequest, + timeout: Duration, + ) -> Result { + let mut invocation = self.dispatcher.lookup_receipt( + self.cell.clone(), + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + None, + )?; + invocation.begin().await?; + Ok(invocation) + } + + async fn now(&self) -> Result { + Ok(self + .fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?) + } + + async fn token(&self, response: &CellReceiptResponse) -> Result<()> { + let keys = VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + assert_eq!( + keys.verify_cell( + response.token(), + &self.cell, + self.class.schema_digest(), + self.now().await? + )?, + *response.observation() + ); + Ok(()) + } + + async fn published_command(&self, sequence: u64, input: &[u8]) -> Result<()> { + self.published_with_ceiling(sequence, input, None).await + } + + async fn published_with_ceiling( + &self, + sequence: u64, + input: &[u8], + ceiling: Option, + ) -> Result<()> { + let mut options = self.fixture.cell_options(&self.cell).await?; + if let Some(ceiling) = ceiling { + match &mut options { + CellSqlOptions::Materialized(o) => o.limits.max_retained_receipts = ceiling, + CellSqlOptions::Extents(o) => o.limits.max_retained_receipts = ceiling, + } + } + let response = self + .command_using( + sequence, + input, + Durability::Quorum, + Duration::from_secs(15), + options, + ) + .await? + .response() + .await?; + let index = response.receipt().position.index; + drop(response); + archive_controller::published(&self.controllers[0], self.route.history, index).await + } +} + +pub(crate) async fn s3_receipt_observations( + rooted: bool, + store: Arc, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let requested = h.receipt_request(1, b"late", Durability::S3)?; + let mut unknown = h.lookup(requested.clone(), Duration::from_secs(15)).await?; + let response = unknown + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert!(response.receipt().is_none()); + assert_eq!(response.observation().version.sequence.get(), 1); + h.token(&response).await?; + drop(response); + drop(unknown); + let mut lookup = h.lookup(requested.clone(), Duration::from_secs(15)).await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(h.controllers[0].usage().retained_s3, 1); + assert_eq!(h.controllers[0].usage().s3_bytes, 8192); + assert_eq!(pool.usage()?.retained_operations, 0); + // The original SQL read observed None at version 1. The same one-slot node + // can now commit and archive the command before proof submission. + h.published_command(1, b"late").await?; + let response = selection.response().await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .receipt() + .context("later archived receipt")? + .result(), + b"late" + ); + assert_eq!(response.observation().version.sequence.get(), 2); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!(h.controllers[0].usage().s3_bytes, 8192 + 2 * b"late".len()); + h.token(&response).await?; + drop(response); + drop(lookup); + // A retained result is exact across newer images, and its token identifies + // the actually inspected archived image rather than the older SQL selection. + let mut known = h.lookup(requested, Duration::from_secs(15)).await?; + let selection = known + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + h.published_command(2, b"").await?; + let response = selection.response().await?; + assert_eq!( + response.receipt().context("retained result")?.result(), + b"late" + ); + assert_eq!(response.observation().version.sequence.get(), 3); + h.token(&response).await?; + drop(response); + drop(known); + let mut empty = h + .lookup( + h.receipt_request(2, b"", Durability::S3)?, + Duration::from_secs(15), + ) + .await?; + let response = empty + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!( + response.receipt().map(|row| row.result()), + Some(b"".as_slice()) + ); + drop(response); + drop(empty); + let mut missing = h.receipt_request(1, b"missing", Durability::S3)?; + missing.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut missing = h.lookup(missing, Duration::from_secs(15)).await?; + let response = missing + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert!(response.receipt().is_none()); + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.observation().version.sequence.get(), 3); + drop(response); + drop(missing); + // A provider in the same domain cannot lend a different runtime's proof. + let foreign = h.dispatcher.clone().with_s3(&h.controllers[1])?; + let mut foreign = foreign.lookup_receipt( + h.cell.clone(), + h.receipt_request(1, b"late", Durability::S3)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + )?; + foreign.begin().await?; + assert!(matches!( + foreign + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await + .err(), + Some(CellDispatchError::Runtime(RuntimeError::Invalid(_))) + )); + drop(foreign); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + assert_eq!(pool.usage()?.retained_operations, 0); + h.close().await +} + +pub(crate) async fn s3_receipt_minimums(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "namespace-s3" + } else { + "namespace-quorum" + }), + )); + let h = Harness::with_minimum(rooted, backend, 1, minimum).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let result = h + .command(1, b"once", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(result); + let request = h.receipt_request(1, b"once", Durability::Quorum)?; + let mut lookup = h.lookup(request, Duration::from_secs(15)).await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(h.controllers[0].usage().retained_s3, 1); + assert_eq!(pool.usage()?.retained_operations, 0); + drop(selection); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + drop(lookup); + let mut lookup = h + .lookup( + h.receipt_request(1, b"once", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + let response = lookup + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response + .receipt() + .context("stored S3 minimum")? + .minimum_durability(), + Durability::S3 + ); + assert_eq!(response.receipt().unwrap().result(), b"once"); + h.token(&response).await?; + drop(response); + drop(lookup); + if minimum == Durability::Quorum { + let gateway = h.fixture.transports[0].install_cell_command_gateway( + h.dispatcher.clone(), + h.controllers[0].clone(), + h.fixture.cell_options(&h.cell).await?, + )?; + let router = super::super::query_origins::Router::start(&h.fixture).await?; + router + .transport + .install_cell_query_group(h.fixture.tcp_route(h.route.history)?)?; + router.grant(&h.fixture, 1)?; + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 1, + peer: router.peer.clone(), + }, + )?; + // The gateway has a valid S3 provider, but remote proof delivery is + // still guarded when a Quorum request discovers a stored S3 minimum. + let mut remote = h + .lookup( + h.receipt_request(1, b"once", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + assert!(matches!( + remote + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err(), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + ))) + )); + drop(remote); + let mut duplicate = h + .dispatcher + .command( + h.cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"once".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + duplicate.begin().await?; + let error = duplicate + .execute_remote(&router.transport, MemberId::new(1)?) + .await + .err() + .context("remote stored S3 command refused")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); + drop(duplicate); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + drop(gateway); + router.close().await?; + } + let mut missing = h.receipt_request(1, b"missing", Durability::Quorum)?; + missing.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut missing = h.lookup(missing, Duration::from_secs(15)).await?; + let response = missing + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert!(response.receipt().is_none()); + assert_eq!(response.durability(), minimum); + drop(response); + drop(missing); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_receipt_failures(rooted: bool, store: Arc) -> Result<()> { + for profile in ["conflict", "expiry", "allocation"] { + let backend = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(profile), + )); + let mut h = Harness::configured( + rooted, + backend, + 1, + Durability::Quorum, + (profile == "allocation").then_some(40 * 1024), + ) + .await?; + let mut expiry = None; + if profile == "expiry" { + h.retry_until = h.now().await? + 2000; + expiry = Some(h.retry_until); + h.published_command(1, b"wanted").await?; + h.retry_until = h.now().await? + 60_000; + h.published_command(2, b"protected-latest").await?; + } + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let mut lookup = h + .lookup( + h.receipt_request(1, b"wanted", Durability::S3)?, + Duration::from_secs(15), + ) + .await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(pool.usage()?.retained_operations, 0); + if profile == "expiry" { + let expiry = expiry.context("original retry horizon")?; + controller::wait_for(async || Ok(h.now().await? > expiry)).await?; + // Only the retry horizon changes; the selected original operation, + // inputs and deadline stay sealed in the lookup phase. + h.retry_until = h.now().await? + 60_000; + // Two retained rows fill this new scenario's ceiling. Maintenance + // prunes only the expired prefix; the latest receipt stays protected. + h.published_with_ceiling(3, b"next", Some(2)).await?; + } else { + h.published_command( + 1, + if profile == "conflict" { + b"other" + } else { + b"wanted" + }, + ) + .await?; + } + let error = selection + .response() + .await + .err() + .context("typed archived refusal")?; + assert!( + matches!(error, CellDispatchError::Execution(ExecutionError::Refused(failure)) if match profile { + "conflict" => failure == CommandFailure::CommandConflict, + "expiry" => failure == CommandFailure::CommandExpired, + _ => failure == CommandFailure::Overloaded, + }) + ); + drop(lookup); + assert_eq!(h.controllers[0].usage().retained_s3, 0); + assert_eq!(h.controllers[0].usage().s3_bytes, 0); + assert_eq!(pool.usage()?.retained_operations, 0); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + if profile == "allocation" { + // The legitimate 40 KiB result remains readable by Quorum. It must + // be refused before payload allocation when the 64 KiB S3 budget + // cannot retain the original and extra read bytes. No quota grows. + let mut read = h + .lookup( + h.receipt_request(1, b"wanted", Durability::Quorum)?, + Duration::from_secs(15), + ) + .await?; + let result = read + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!( + result + .receipt() + .context("intact large result")? + .result() + .len(), + 40 * 1024 + ); + assert_eq!(result.durability(), Durability::Quorum); + drop(result); + drop(read); + } + assert_eq!( + h.calls.load(Ordering::SeqCst), + if profile == "expiry" { 3 } else { 1 } + ); + h.close().await?; + } + Ok(()) +} + +pub(crate) async fn s3_receipt_deadline( + rooted: bool, + store: Arc, + cancel: bool, +) -> Result<()> { + let h = Harness::new(rooted, store, 1).await?; + let pool = &h.fixture.appliers[&(0, h.route.history)].0; + let runtime = &h.fixture.runtimes[0]; + let request = h.receipt_request(1, b"after", Durability::S3)?; + let mut lookup = h + .lookup(request.clone(), Duration::from_millis(800)) + .await?; + let selection = lookup + .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + h.remote.outage.store(true, Ordering::Release); + let mut pending = Box::pin(selection.response()); + tokio::select! { + result = pending.as_mut() => { let _ = result?; anyhow::bail!("lookup completed during archive outage"); } + result = h.waiting() => result?, + } + assert_eq!(pool.usage()?.retained_operations, 0); + if !cancel { + assert!(matches!( + pending.as_mut().await.err(), + Some(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)) + )); + } + drop(pending); + drop(lookup); + controller::wait_for(async || Ok(h.controllers[0].usage().retained_s3 == 0)).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let response = h + .command(1, b"after", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(response); + h.remote.outage.store(false, Ordering::Release); + let mut retry = h.lookup(request, Duration::from_secs(15)).await?; + let response = retry + .execute(runtime, pool, h.fixture.cell_options(&h.cell).await?) + .await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!( + response.receipt().context("original command")?.result(), + b"after" + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(response); + drop(retry); + assert_eq!(pool.usage()?.retained_operations, 0); + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 42f80ebb5..93cbd7b1a 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -539,7 +539,8 @@ mod s3_response; #[allow(unused_imports)] pub(crate) use s3_commands::{ s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, - s3_namespace_minimum, + s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, + s3_receipt_observations, }; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index bf82e8cee..5357fdf34 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -720,8 +720,8 @@ through this archive-specific drain. the same execution domain. Execution also requires its exact runtime owner and an active archive registration for the selected group/route. Effective durability is the maximum of the request, namespace minimum and existing receipt minimum. -Receipt lookup and remote S3 response paths retain their refusal gates until -their own output proofs are integrated. +Remote S3 response paths retain refusal gates until their own output proofs +are integrated, including when the receiver has a configured archive provider. A committed SQL receipt resolves its producer before any archive wait. The owner acquires independent S3 byte/item admission before transferring output retention @@ -756,7 +756,7 @@ Idle registrations do not retain replay workspaces. Active consumers can occupy those shared workspaces and backpressure subsequent archival until consumption; this is explicit pool admission, without an unaccounted metadata-only proof cache. -S3 receipt-lookup/remote responses, explicit unarchived-byte/disk +Remote S3 responses, explicit unarchived-byte/disk admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, load and process-kill qualification remain required gates. This local command boundary is not the complete CG-06 or public SDK/fleet cutover gate. @@ -785,6 +785,45 @@ That output retains the same physical coverage, pool admission and journal lock as a command response. A deadline or dropped reply keeps commitment ambiguous, without retaining an unresolved SQL producer or quarantining its healthy group. +### Local receipt lookup S3 response gate + +`CellReceiptInvocation::select` binds an opaque quorum selection to the original +request, route and deadline. Required durability is the maximum of request, +namespace and stored receipt minimum. A configured S3 provider must own that exact +runtime and active group registration. S3 admission takes the selected bytes +before SQL read admission releases; no producer is created or retired. Selection +queues no archive work until its consuming `response` call. The existing `execute` +method runs both phases. Dropping an unsubmitted selection refunds admission. + +A complete published cut must be independently recovered. Its cell history and +logical version must cover the quorum selection; an equal version must retain the +same decision and receipt outcome. A known receipt must match every stored field +and result, even in a later image. An initially absent receipt is read again from +the actual later archive: it may return a newly committed receipt or an ambiguous +absence, with that archived image's observation and signed token. No absence, +including an S3-protected one, proves non-commit or releases original work. + +The shared receipt reader checks row length before fetching payload bytes. The +existing S3 reservation grows atomically for actual result size without another +item slot, within the unchanged node byte/item ceilings. Failure returns typed +Overloaded before payload allocation; expiry, conflict and sequence gaps remain +typed rather than becoming None. Receipt table loss remains corruption except +for legitimate initial sequence-1 images. Empty results remain distinct from +absence. No application callback or receipt maintenance runs during lookup. + +Before output, recheck current catalog birth and a fresh correlated quorum read's +leader/route fence under the original deadline. The current cell cannot precede +the archived observation; equal versions must retain the same decision. This read +fence permits later concurrent mutations without retaining a private writer's +base. A successful response retains full physical coverage, S3 admission and the +journal lock, and reports achieved durability from that private proof. + +Remote senders refuse explicit/effective S3 requests. Receivers also refuse S3 +before SQL when already required, or after read-only duplicate inspection when a +stored minimum requires it. Configuring an S3 provider cannot enable an unqualified +wire response. Command duplicate refusal clears resolved producer ownership first. +Remote proof ownership through physical delivery remains a separate required gate. + ## Split and merge Use the same transition state machine for one parent to multiple child ranges diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 0fe6c17db..11fe694bd 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1766,7 +1766,8 @@ "crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs", "crates/cellar/tests/support/cell_group_s3_commands.rs", "crates/cellar/durability/raft/cell_groups/archive/enrollment.rs", - "crates/cellar/tests/support/cell_group_s3_enrollment.rs" + "crates/cellar/tests/support/cell_group_s3_enrollment.rs", + "crates/cellar/tests/support/cell_group_s3_receipts.rs" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 1a0672969..591597ca7 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,12 +71,14 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs nine exact ignored RustFS cases, each in materialized and extent modes: +It runs thirteen exact ignored RustFS cases, each in materialized and extent modes: stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 settlement, deadline/cancellation isolation, refusal when a sibling dependency is missing, namespace minimum enforcement, original enrollment creation proofs -and enrollment deadline/cancellation isolation. Catalog setup, archive +and enrollment deadline/cancellation isolation. Receipt cases cover later archived +observations, namespace/stored minima, typed failures and payload admission, and +read cancellation/deadlines. Catalog setup, archive objects, conditional registry updates and independent recovery use the actual S3 client. Fault injection surrounds that client; lost publication responses are injected after the real conditional write commits. No memory fallback exists. @@ -86,8 +88,10 @@ including a namespace minimum. The positive command cases use an explicit controller binding, exact receipts from later archived images, retained proof admission and fresh routing/leader checks. Enrollment cases preserve the original creation after later mutations, honor namespace minimums, retain output proofs and -retry without rerunning initialization. Receipt lookup and remote S3 responses -remain unqualified. The runner does not qualify local log file purge, +retry without rerunning initialization. Receipt lookup preserves ambiguous absence, +returns the actual later archived observation, reserves result bytes before fetch, +and tests remote refusal with a provider-configured TLS gateway. Remote S3 +responses remain unqualified. The runner does not qualify local log file purge, fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index c584fb167..aaa1dc9b8 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -"""Qualify shared-group archive setup and local command/enrollment responses against local RustFS. +"""Qualify shared-group archive setup and local command/enrollment/receipt responses against local RustFS. Uses only the recorded owned backend. Each invocation owns a new evidence directory and remote prefix. Credentials are passed through the child environment, @@ -34,6 +34,10 @@ "rustfs_s3_namespace_minimum_cannot_be_downgraded", "rustfs_s3_enrollment_requires_the_original_archived_creation", "rustfs_s3_enrollment_deadlines_preserve_resolved_ownership", + "rustfs_s3_receipt_lookup_tracks_archived_observations", + "rustfs_s3_receipt_lookup_honors_namespace_and_stored_minimums", + "rustfs_s3_receipt_lookup_preserves_typed_failures_and_result_admission", + "rustfs_s3_receipt_lookup_deadlines_refund_read_admission", ) @@ -108,11 +112,11 @@ def main(): out.mkdir(mode=0o700, parents=True, exist_ok=False) prefix = "cell-groups/" + secrets.token_hex(16) record = { - "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command/enrollment S3 settlement", + "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command/enrollment/receipt S3 settlement", "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " - "No S3 receipt-lookup or remote acknowledgements, byte/disk outage bound, " + "No remote S3 acknowledgements, byte/disk outage bound, " "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", } From 214a1a13883e45f67c501d06e9a94c0933d3c19f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 15:32:30 -0700 Subject: [PATCH 193/568] fix: scope receipt proof catalog checks --- .../durability/raft/cell_groups/dispatch/receipt.rs | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs index f7828b9f0..5e0cc0465 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -3,7 +3,7 @@ //! Receipt-only dispatch never stages SQL or turns absence into non-commit. use super::*; use crate::application::consensus::command_request_digest; -use crate::raft::execution::{CommandRequest, ExecutionLimits, StoredReceipt}; +use crate::raft::execution::{CommandRequest, StoredReceipt}; use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; #[derive(Clone)] @@ -364,7 +364,11 @@ impl CellReceiptInvocation { Selected::Quorum(selected) => (selected, None, None), Selected::S3(prepared) => { let settled = prepared.response().await?; - settled.proof.require_current().await?; + self.dispatcher + .domain + .run(settled.proof.require_current()) + .await + .map_err(|_| closed())??; // Use a fresh read authority, whose leader fence tolerates later // cell writes, rather than retaining a private writer's base. let authority = runtime From 59619e0a98e84e2968fec62d0647b034c7c757bb Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 15:41:18 -0700 Subject: [PATCH 194/568] fix: bound S3 proof checks by caller deadline --- .../controller/archive/settlement.rs | 31 ++++++++++++++++--- .../raft/cell_groups/dispatch/command/mod.rs | 5 ++- .../raft/cell_groups/dispatch/enrollment.rs | 5 ++- .../raft/cell_groups/dispatch/receipt.rs | 9 +++--- 4 files changed, 38 insertions(+), 12 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs index 4c6e9ffb5..47928213a 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/settlement.rs @@ -23,11 +23,32 @@ pub(crate) struct OutputProof { } impl OutputProof { - pub(crate) async fn require_current(&self) -> std::result::Result<(), CellDispatchError> { - self.authority - .require_current() - .await - .map_err(|error| CellDispatchError::Archive(Arc::new(error))) + pub(crate) async fn require_current( + &self, + domain: &crate::asyncrt::DomainToken, + deadline: u64, + ) -> std::result::Result<(), CellDispatchError> { + let now = domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if now >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + let result = crate::asyncrt::select! { + result = domain.run(self.authority.require_current()) => result.map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + if domain + .mono_ms() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + >= deadline + { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + result.map_err(|error| CellDispatchError::Archive(Arc::new(error))) } } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 9abf83a99..9e40eb50c 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -708,7 +708,10 @@ impl CellCommandInvocation { if authority.route() != &origin.route { return Err(CellDispatchError::Invalid("S3 response current route")); } - settled.proof.require_current().await?; + settled + .proof + .require_current(&self.dispatcher.domain, self.session.deadline_ms()) + .await?; authority.check().map_err(CellDispatchError::Runtime)?; (settled.receipt, Some(settled.proof), Some(authority)) } else { diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index 7b399af98..81b6302bb 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -485,7 +485,10 @@ impl CellEnrollmentInvocation { "S3 enrollment current creation/route", )); } - settled.proof.require_current().await?; + settled + .proof + .require_current(&self.dispatcher.domain, self.session.deadline_ms()) + .await?; if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() { return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs index 5e0cc0465..d0e579631 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -364,11 +364,10 @@ impl CellReceiptInvocation { Selected::Quorum(selected) => (selected, None, None), Selected::S3(prepared) => { let settled = prepared.response().await?; - self.dispatcher - .domain - .run(settled.proof.require_current()) - .await - .map_err(|_| closed())??; + settled + .proof + .require_current(&self.dispatcher.domain, attempt.deadline_ms()) + .await?; // Use a fresh read authority, whose leader fence tolerates later // cell writes, rather than retaining a private writer's base. let authority = runtime From cde0eea2f516486578fdee7a4cc0b8be9ba8f72a Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 15:48:07 -0700 Subject: [PATCH 195/568] test: observe queued S3 lookup cancellation --- .../tests/support/cell_group_s3_receipts.rs | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_receipts.rs b/crates/cellar/tests/support/cell_group_s3_receipts.rs index 15ab27ff5..98a4c648d 100644 --- a/crates/cellar/tests/support/cell_group_s3_receipts.rs +++ b/crates/cellar/tests/support/cell_group_s3_receipts.rs @@ -470,11 +470,21 @@ pub(crate) async fn s3_receipt_deadline( .select(runtime, pool, h.fixture.cell_options(&h.cell).await?) .await?; h.remote.outage.store(true, Ordering::Release); + let attempts = h.controllers[0] + .archive_status(h.route.history)? + .context("archive registration")? + .attempts; let mut pending = Box::pin(selection.response()); - tokio::select! { - result = pending.as_mut() => { let _ = result?; anyhow::bail!("lookup completed during archive outage"); } - result = h.waiting() => result?, - } + assert!(futures_util::poll!(pending.as_mut()).is_pending()); + controller::wait_for(async || { + Ok(h.controllers[0] + .archive_status(h.route.history)? + .context("archive status")? + .attempts + > attempts + && h.controllers[0].usage().retained_s3 == 1) + }) + .await?; assert_eq!(pool.usage()?.retained_operations, 0); if !cancel { assert!(matches!( From f540dd349a53e51a346ecd197adab0622ffad924 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 16:38:11 -0700 Subject: [PATCH 196/568] docs: record S3 receipt qualification --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 108 ++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 67 +++++++++++ 3 files changed, 131 insertions(+), 50 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 2c7b0e5bc..24b569e5f 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "c2801b6586ffc8b3b765312e0042986e881d773a", - "runtime_revision": "c2801b6586ffc8b3b765312e0042986e881d773a", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged.", + "source_revision": "cde0eea2f516486578fdee7a4cc0b8be9ba8f72a", + "runtime_revision": "cde0eea2f516486578fdee7a4cc0b8be9ba8f72a", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 669047da8..997390d2b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,59 +15,73 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial runtime/qualification source is `c2801b6586ffc8b3b765312e0042986e881d773a`. -CG-06 remains IN_PROGRESS. Local command and enrollment S3 responses require an -explicit `CellGroupDispatcher::with_s3` binding to the exact controller/runtime -archive owner. A complete published cut is independently recovered before proving -the exact immutable command receipt or original enrollment creation record. -Later images can prove older receipts/creation records. Current catalog and actual -leader authority are rechecked under the original deadline; namespace minima are -never lowered. Enrollment returns original provenance, not a current-version token. - -Resolved SQL ownership clears before archival. Separate bounded S3 admission takes -output retention before SQL releases its bytes/operation permit. One fair node -archive lane handles both output kinds, at most 32 proofs per group turn. -Successful outputs retain complete physical coverage, shared pool admission and -the exclusive journal lock; weak owner uses prevent premature hold retirement. -Active consumers backpressure those pools; idle groups retain no replay workspace. -Archive failure/deadline cannot quarantine a resolved initializer or command. -Exact creation retries return the original record without initialization replay. - -Nine exact RustFS cases pass in both image forms, plus seven signed backend probes. -Fourteen check/regression commands pass with 195 distinct Rust cases across six +The latest partial runtime/qualification source is `cde0eea2f516486578fdee7a4cc0b8be9ba8f72a`. +CG-06 remains IN_PROGRESS. Local command, enrollment and receipt S3 responses +require explicit `CellGroupDispatcher::with_s3` binding to the exact controller/ +runtime archive owner. Complete published cuts are independently recovered before +proving an immutable command receipt, original creation record or receipt lookup. +All request/namespace/stored receipt minima and current catalog/leader checks hold. + +Receipt `select` seals one quorum observation under its original operation and +deadline, transfers SQL retention to bounded S3 admission and queues nothing until +consuming `response`. Existing `execute` runs both. Known receipts match exactly; +an initial absence may become a receipt at a later archive. Return the actually +read archived observation/token. No absence proves non-commit or retires a producer. +Payload length is checked before fetching, and the same S3 permit grows within +unchanged node byte/item ceilings. Typed expiry/conflict/gaps and corruption remain +failures. Read admission drops before waiting; no application callback runs. + +All three output kinds scope catalog proof rechecks in the captured domain and +bound them by the original deadline. Receipt responses also use a fresh correlated +quorum read's leader/route fence and version floor; later writes need not invalidate +that read. Successful outputs retain full physical coverage/admission and journal +lock; active consumers backpressure the shared pools, while idle groups keep no +replay workspace. Remote S3 requests and stored-minimum upgrades are still refused +on both ends, even with a provider-configured gateway. Command duplicate refusal +clears resolved producer ownership without granting non-commit evidence. + +Thirteen exact RustFS cases pass in both image forms, plus seven signed probes. +Sixteen check/regression commands pass with 205 distinct Rust cases across six copied executables, all-target Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. Independent source/artifact/ -selection and credential-exclusion audits pass. Records, frozen inputs and exact -executables are under `.cellar/cell-groups-evidence/CG-06-s3-enrollment/`. -See the newest [progress](progress.md), [enrollment proof contract](protocols.md#local-enrollment-s3-response-gate) +selection and credential-exclusion audits pass. Records, frozen inputs, executables +and the initial direct-caller domain panic/backtrace are under +`.cellar/cell-groups-evidence/CG-06-s3-receipts/`. See the newest +[progress](progress.md), [receipt proof contract](protocols.md#local-receipt-lookup-s3-response-gate) and [RustFS runner](../../../../tools/multi-raft/cell_groups.py). -Next implement local S3 receipt lookup: no absent row establishes non-commit, and -later archived observations must not falsely attest an older absence. Preserve -exact identity, typed expiry/conflict and output admission. Then integrate remote -output proofs with lifetime through physical delivery. Those S3 wire paths still -refuse unsupported capabilities; remote enrollment cannot downgrade to Quorum. -Add explicit unarchived-byte/disk admission, physical shared-engine Raft file purge -and fenced fresh-incarnation serving with complete old-voter lineage inventory, -including joins after the recovery cut. Fencing-authority API plus Docker -qualification remains the accepted direction. Multi-head proof retention, -controller process-kill restart and physical-read shutdown cuts need qualification. +Next implement remote S3 command/enrollment/receipt output ownership through the +actual physical send, with achieved durability and authenticated operation/route/ +version checks under original deadlines. Do not merely remove refusal guards or +serialize a proof-free success. The current `command_cell_transport.rs` handlers +return encoded bytes, dropping the native response before `query_transport.rs` +writes and flushes them; enrollment clones only its creation record. Replace this +bytes-only boundary with a bounded owned reply that retains the proof through +physical write/drain. Authenticate achieved durability in all three payloads and +validate the required minimum on receipt. Qualify held writes, disconnects, +post-commit deadlines and route/leader changes without callback replay or false +non-commit. Add explicit unarchived-byte/disk admission, +physical shared-engine Raft file purge and fenced new-incarnation serving with +complete old-voter lineage inventory, including joins after the recovery cut. +Fencing-authority API plus Docker qualification remains the accepted direction. +Multi-head proof retention, process-kill restart and physical-read shutdown cuts +still need qualification. Incremental tail scheduling and smaller sealed output +attestations require real retained remote holds and load evidence. CG-08 effect/cache/result lifetimes and maintenance precede CG-07 public SDK/fleet -cutover, followed by topology, placement, hibernation, managed recovery, measured -load and final legacy removal. Four original CG-06 cases and its full task gate -remain pending. Voters are host-native tasks with one local RustFS VM; independent -hosts, container node limits, fenced replacement-quorum serving, measured load -and full hard cutover remain unqualified. The end-to-end goal is not complete. - -Preserve the preceding local command source `3d564f3d`, RustFS source `d739bf79`, -controller source `7914177d`, catalog-cut `df0ba1f6`, protected publication -`87348477`, isolated verifier `14955f2d`, committed tail `7a29c2a9` and their -source-bound evidence/failures. Exact intents precede remote mutation; unknown -outcomes never allocate a new identity or reset counters. Namespace binding, -stable cut capture, protected publication and source-independent recovery retain -their existing contracts. The controller initially coalesces complete cuts; -incremental scheduling remains future work. +cutover, followed by topology, placement, hibernation, managed recovery, load and +final legacy removal. Four original CG-06 cases/full gate remain pending. Voters +are host-native tasks with one local RustFS VM; independent hosts, node-container +limits, measured load, fenced replacement-quorum serving and full hard cutover +remain unqualified. The end-to-end goal remains active and incomplete. + +Preserve enrollment source `c2801b65`, command source `3d564f3d`, RustFS source +`d739bf79`, controller source `7914177d`, catalog-cut `df0ba1f6`, protected +publication `87348477`, isolated verifier `14955f2d`, committed tail `7a29c2a9` +and their source-bound evidence/failures. Exact intents precede remote mutation; +unknown outcomes never allocate a new identity or reset counters. Namespace +binding, stable cut capture, protected publication and complete source-independent +recovery retain their contracts. The controller initially coalesces complete cuts. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index fb8c6b77e..bde61b7f5 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,73 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 local receipt S3 observations and bounded result reads + +Source: `cde0eea2f516486578fdee7a4cc0b8be9ba8f72a`. CG-06 remains IN_PROGRESS. Local command, +enrollment and receipt outputs now share the explicitly bound controller archive +owner. `CellReceiptInvocation::select` returns an opaque quorum selection; its +consuming `response` settles required durability. Existing `execute` runs both. +S3 admission takes the selected bytes before SQL read admission releases. The +selection queues no archive work until response; dropping it refunds admission. +Request, namespace and stored receipt minima are never lowered. + +The complete independently recovered group must cover the quorum cell history/ +version and equal-version decision. Known receipts remain exact across later +images. An initially absent receipt is read again from the actual archive, which +may now contain the command. Return that image's actual observation and token; +never attest an old absence against a changed image. None remains ambiguous and +never releases a producer. Empty results remain distinct. Lookup runs no command, +initializer or receipt maintenance; expiry, conflict and sequence gaps stay typed. + +Result length is inspected before payload fetch. The same S3 permit grows for +actual bytes without another item slot or increased node ceilings. A legitimate +40 KiB result exceeds the unchanged 64 KiB fixture S3 envelope with duplicate-read +headroom: S3 returns typed Overloaded before allocation, then Quorum still reads +the intact result. Original SQL/S3 node quotas remain unchanged. The new expiry +scenario deliberately fills a two-receipt ceiling so normal replicated maintenance +prunes an expired prefix while retaining the latest receipt and permanent floor. + +S3 outputs retain complete physical coverage, shared pool admission and the journal +lock. Catalog proof rechecks enter the captured execution domain and remain bounded +by the original deadline for all three output kinds. Receipt output additionally +checks a fresh correlated quorum read's current leader/route and version floor. +That read fence tolerates later cell mutations without retaining a writer's base. +Remote S3 delivery remains guarded on both ends, including provider-configured +receivers that discover a stored S3 minimum. A rejected committed command duplicate +clears SQL ownership first and remains ambiguous to its remote caller. + +All thirteen exact RustFS cases pass in both image formats, plus seven signed S3 +probes. The four new cases cover later receipts replacing initial absence, known +results at newer cuts, signed observations, empty/absent results, namespace/stored +minima, unsubmitted-phase drops, foreign providers, typed conflict/expiry, payload +admission, read deadlines/cancellation and actual TLS receiver refusal without +callback replay or quarantine. The earlier archive/command/enrollment cases pass. + +All sixteen check/regression commands pass: 205 distinct Rust cases across six +copied executables, locked all-target Cellar/logic/LTX Clippy, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing. +Records under `.cellar/cell-groups-evidence/CG-06-s3-receipts/` bind +2567 source hashes, exact selections, frozen plan inputs, executable/output +hashes and predecessors. Independent source/hash/selection and credential-exclusion +audit passes. The initial direct-caller domain panic retains its exact source, +executable, output and minimized backtrace. Production scoping fixes it; no global +host handle or assertion workaround was added. The intermediate four-case success +and final deadline-bound source remain distinguishable. The fixture macro-lint +refusal is retained; cancellation now polls the response and observes an actual +archive attempt before dropping it. No original task oracle +or existing quota was weakened. Four original CG-06 cases/full gate remain pending. + +Reflection: implement remote S3 output proofs through physical delivery, preserving +authenticated purpose, achieved durability, route/version and original deadlines. +Then explicit unarchived-byte/disk admission, physical shared-engine Raft purge +and fenced fresh-incarnation serving with every possible old-voter lineage. +Qualify multi-head leases, process-kill restart and physical-read shutdown cuts. +Incremental committed tails and smaller sealed attestations backed by real remote +hold leases remain performance opportunities. CG-08 lifetimes/maintenance precede +CG-07 SDK/fleet cutover; topology, placement, hibernation, managed recovery, load +and final legacy removal remain. Voters share one host and RustFS one local VM; +independent hosts, measured load and fenced replacement serving remain unqualified. + ## CG-06 local enrollment S3 response settlement Source: `c2801b6586ffc8b3b765312e0042986e881d773a`. CG-06 remains IN_PROGRESS. The explicit From 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 16:59:06 -0700 Subject: [PATCH 197/568] feat: retain remote S3 output proofs --- .../raft/cell_groups/dispatch/command/mod.rs | 10 - .../cell_groups/dispatch/command/remote.rs | 7 - .../raft/cell_groups/dispatch/enrollment.rs | 5 - .../cell_groups/dispatch/enrollment_remote.rs | 10 +- .../raft/cell_groups/dispatch/receipt.rs | 15 - .../durability/raft/command_cell_transport.rs | 42 +- .../durability/raft/command_cell_wire.rs | 18 +- .../durability/raft/enrollment_cell_wire.rs | 36 +- .../durability/raft/query_cell_transport.rs | 4 +- crates/cellar/durability/raft/query_reply.rs | 95 +++++ .../cellar/durability/raft/query_transport.rs | 20 +- crates/cellar/durability/raft/query_wire.rs | 17 +- .../durability/raft/query_wire_tests.rs | 219 ++++++++-- .../durability/raft/receipt_cell_wire.rs | 17 +- crates/cellar/durability/raft/transport.rs | 2 + .../native_cell_groups_archive_controller.rs | 75 ++++ .../support/cell_group_network_probes.rs | 65 ++- .../tests/support/cell_group_runtime.rs | 2 +- .../tests/support/cell_group_s3_commands.rs | 17 + .../tests/support/cell_group_s3_receipts.rs | 40 +- .../tests/support/cell_group_s3_remote.rs | 396 ++++++++++++++++++ .../support/cell_group_s3_remote_wire.rs | 271 ++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 2 +- crates/cellar/tests/support/cell_group_tcp.rs | 29 +- .../multi-raft/cell-groups/protocols.md | 51 ++- docs/design/multi-raft/cell-groups/tasks.json | 5 +- tools/multi-raft/README.md | 12 +- tools/multi-raft/cell_groups.py | 9 +- 28 files changed, 1303 insertions(+), 188 deletions(-) create mode 100644 crates/cellar/durability/raft/query_reply.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_remote.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_remote_wire.rs diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 9e40eb50c..39cb9d1e7 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -366,11 +366,6 @@ impl CellCommandInvocation { return Err(CellDispatchError::Invalid("destination route binding")); } if self.request.durability == Durability::S3 { - if self.remote_route.is_some() { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } self.dispatcher .s3 .as_ref() @@ -670,11 +665,6 @@ impl CellCommandInvocation { .durability .max(receipt.receipt().receipt.minimum_durability()); let (receipt, s3, authority) = if required == Durability::S3 { - if self.remote_route.is_some() { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } let owner = self .dispatcher .s3 diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs index dd6c127e9..640090b88 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -28,13 +28,6 @@ impl CellCommandInvocation { "consumed or local command attempt", ))); } - if self.request.durability == Durability::S3 { - return Err( - self.error(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))), - ); - } if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(self.error(CellDispatchError::Invalid("command transport domain"))); } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index 81b6302bb..df9e3c568 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -265,11 +265,6 @@ impl CellEnrollmentInvocation { return Err(CellDispatchError::Invalid("enrollment destination route")); } if self.durability == Durability::S3 { - if self.remote_route.is_some() { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } let owner = self .dispatcher .s3 diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs index a82261d3f..133fbe5c4 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment_remote.rs @@ -81,14 +81,6 @@ impl CellEnrollmentInvocation { "consumed or local enrollment attempt", ))); } - // The remote response schema has no retained archive proof contract yet. - if self.durability == Durability::S3 { - return Err( - self.error(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))), - ); - } if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(self.error(CellDispatchError::Invalid("enrollment transport domain"))); } @@ -108,7 +100,7 @@ impl CellEnrollmentInvocation { let call = CellEnrollmentCall { request: CellEnrollmentRequest { creation: self.creation, - durability: Durability::Quorum, + durability: self.durability, }, inputs: self.inputs, preview: self.preview, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs index d0e579631..2e8b85bac 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/receipt.rs @@ -264,11 +264,6 @@ impl CellReceiptInvocation { return Err(CellDispatchError::Invalid("receipt execution domain")); } if self.request.durability == Durability::S3 { - if self.remote_route.is_some() { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } let owner = self .dispatcher .s3 @@ -320,11 +315,6 @@ impl CellReceiptInvocation { .map_or(Durability::Quorum, |row| row.minimum_durability()), ); let selected = if required == Durability::S3 { - if self.remote_route.is_some() { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } let owner = self .dispatcher .s3 @@ -438,11 +428,6 @@ impl CellReceiptInvocation { if self.dispatched || self.remote_route.is_some() { return Err(CellDispatchError::Invalid("consumed receipt attempt")); } - if self.request.durability == Durability::S3 { - return Err(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - ))); - } self.dispatched = true; if !transport.uses_command_domain(&self.dispatcher.domain) { return Err(CellDispatchError::Invalid("receipt transport domain")); diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs index 7972c4fcd..3332b7d79 100644 --- a/crates/cellar/durability/raft/command_cell_transport.rs +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -11,7 +11,7 @@ use crate::raft::query_identity::QueryRoute; use crate::raft::query_wire::{ self as wire, command::{CellCommandCall, CellCommandValue}, - enrollment::CellEnrollmentCall, + enrollment::{CellEnrollmentCall, CellEnrollmentValue}, receipt::{CellReceiptCall, CellReceiptValue}, }; use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; @@ -65,7 +65,7 @@ impl CellRemoteCommandResponse { } pub fn durability(&self) -> cellar_logic::consensus::Durability { - cellar_logic::consensus::Durability::Quorum + self.value.durability } pub fn result(&self) -> &[u8] { @@ -89,6 +89,7 @@ impl CellRemoteCommandResponse { /// The original enrollment image is not a claim about the latest cell version. pub struct CellRemoteEnrollmentResponse { record: EnrollmentRecord, + durability: cellar_logic::consensus::Durability, served_by: MemberId, _node: Permit, _peer: Permit, @@ -104,7 +105,7 @@ impl CellRemoteEnrollmentResponse { } pub fn durability(&self) -> cellar_logic::consensus::Durability { - cellar_logic::consensus::Durability::Quorum + self.durability } } @@ -116,6 +117,10 @@ pub struct CellRemoteReceiptResponse { } impl CellRemoteReceiptResponse { + pub fn durability(&self) -> cellar_logic::consensus::Durability { + self.value.value.durability + } + pub fn command(&self) -> cellar_logic::consensus::CommandId { self.value.value.command } @@ -181,7 +186,7 @@ impl WriteCall { enum WriteValue { Command(CellCommandValue), - Enrollment(EnrollmentRecord), + Enrollment(CellEnrollmentValue), Receipt(CellReceiptValue), } @@ -309,11 +314,12 @@ impl RaftTransport { deadline, ) .await?; - let WriteValue::Enrollment(record) = response.value else { + let WriteValue::Enrollment(value) = response.value else { return Err(TransportError::Protocol); }; Ok(CellRemoteEnrollmentResponse { - record, + record: value.record, + durability: value.durability, served_by: response.served_by, _node: response.node, _peer: response.peer, @@ -542,7 +548,7 @@ impl State { .map_err(|_| TransportError::Authentication)?; wire::command::decode_result( &body, - (call.request.id, &request.cell), + (call.request.id, &request.cell, call.request.durability), request.route.schema_digest, &request.keys, now, @@ -556,7 +562,7 @@ impl State { .map_err(|_| TransportError::Authentication)?; wire::receipt::decode_result( &body, - (call.request.id, &request.cell), + (call.request.id, &request.cell, call.request.durability), request.route.schema_digest, &request.keys, now, @@ -567,7 +573,7 @@ impl State { } WriteCall::Enrollment(call) => wire::enrollment::decode_result( &body, - call.request.creation, + (call.request.creation, call.request.durability), &request.cell, request.route.schema_digest, self.options.max_query_response_bytes, @@ -596,7 +602,7 @@ impl State { &self, route: &QueryRoute, call: CellReceiptCall, - ) -> Result, TransportError> { + ) -> Result { let (_, target, cell) = route.decode_shared()?; let gateway = self .cell_command_gateways @@ -637,14 +643,14 @@ impl State { .map_err(|e| command_failure(&e)) } .await; - wire::receipt::encode_result(&result, self.options.max_query_response_bytes) + super::query_replies::Reply::receipt(result, self.options.max_query_response_bytes) } pub(super) async fn execute_cell_enrollment( &self, route: &QueryRoute, call: CellEnrollmentCall, - ) -> Result, TransportError> { + ) -> Result { let (_, target, cell) = route.decode_shared()?; let gateway = self .cell_command_gateways @@ -687,21 +693,19 @@ impl State { .controller .enroll(invocation, gateway.options.clone()) .map_err(|error| command_failure(&error.error.cause))?; - let response = call - .response() + call.response() .await - .map_err(|error| command_failure(&error.cause))?; - Ok(response.record().clone()) + .map_err(|error| command_failure(&error.cause)) } .await; - wire::enrollment::encode_result(&result, self.options.max_query_response_bytes) + super::query_replies::Reply::enrollment(result, self.options.max_query_response_bytes) } pub(super) async fn execute_cell_command( &self, route: &QueryRoute, call: CellCommandCall, - ) -> Result, TransportError> { + ) -> Result { let (_, target, cell) = route.decode_shared()?; let gateway = self .cell_command_gateways @@ -751,7 +755,7 @@ impl State { .map_err(|error| command_failure(&error.cause)) } .await; - wire::command::encode_result(&result, self.options.max_query_response_bytes) + super::query_replies::Reply::command(result, self.options.max_query_response_bytes) } } diff --git a/crates/cellar/durability/raft/command_cell_wire.rs b/crates/cellar/durability/raft/command_cell_wire.rs index 0bbbfcb09..dc3413499 100644 --- a/crates/cellar/durability/raft/command_cell_wire.rs +++ b/crates/cellar/durability/raft/command_cell_wire.rs @@ -19,6 +19,7 @@ pub(in crate::raft) struct CellCommandCall { pub(in crate::raft) struct CellCommandValue { pub command: CommandId, + pub durability: Durability, pub retry_until_ms: u64, pub value: cell::CellQueryValue, } @@ -34,7 +35,6 @@ impl CellCommandCall { || r.contract.operation.len() > 128 || r.contract.operation.chars().any(char::is_control) || !(1..=3_600_000).contains(&self.timeout_ms) - || r.durability != Durability::Quorum { return Err(TransportError::Protocol); } @@ -45,7 +45,7 @@ impl CellCommandCall { body.extend_from_slice(&self.birth_digest); string(&mut body, &r.contract.operation)?; body.extend_from_slice(&r.contract.digest); - body.push(1); // Quorum; never silently lower an S3 request. + body.push(durability_code(r.durability)); body.extend_from_slice(&r.retry_until_ms.to_be_bytes()); body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); body.extend_from_slice(&self.inputs.random); @@ -87,9 +87,7 @@ impl CellCommandCall { kind: OperationKind::Command, digest: body.array()?, }; - if body.byte()? != 1 { - return Err(TransportError::Protocol); - } + let durability = read_durability(&mut body)?; let retry_until_ms = u64::from_be_bytes(body.array()?); let inputs = CommandInputs { now_unix_millis: u64::from_be_bytes(body.array()?), @@ -109,7 +107,7 @@ impl CellCommandCall { id, contract, input, - durability: Durability::Quorum, + durability, retry_until_ms, }, inputs, @@ -134,6 +132,7 @@ pub(in crate::raft) fn encode_result( Ok(response) => { let receipt = &response.receipt().receipt; body.push(0); + body.push(durability_code(response.durability())); body.extend_from_slice(&receipt.command().client.bytes()); body.extend_from_slice(&receipt.command().sequence().to_be_bytes()); body.extend_from_slice(&receipt.retry_until_ms().to_be_bytes()); @@ -171,7 +170,7 @@ pub(in crate::raft) fn encode_result( pub(in crate::raft) fn decode_result( bytes: &[u8], - expected: (CommandId, &CellHistory), + expected: (CommandId, &CellHistory, Durability), schema: [u8; 32], keys: &VersionKeys, now: u64, @@ -187,6 +186,10 @@ pub(in crate::raft) fn decode_result( } let result = match body.byte()? { 0 => { + let durability = read_durability(&mut body)?; + if durability < expected.2 { + return Err(TransportError::Authentication); + } let command = command_id(&mut body)?; if command != expected.0 { return Err(TransportError::Authentication); @@ -200,6 +203,7 @@ pub(in crate::raft) fn decode_result( let length = u32::from_be_bytes(body.array()?) as usize; Ok(CellCommandValue { command, + durability, retry_until_ms, value: cell::CellQueryValue { result: body.take(length)?.to_vec(), diff --git a/crates/cellar/durability/raft/enrollment_cell_wire.rs b/crates/cellar/durability/raft/enrollment_cell_wire.rs index 8ad707819..ac823bf75 100644 --- a/crates/cellar/durability/raft/enrollment_cell_wire.rs +++ b/crates/cellar/durability/raft/enrollment_cell_wire.rs @@ -19,21 +19,25 @@ pub(in crate::raft) struct CellEnrollmentCall { pub birth_digest: [u8; 32], } +#[derive(Debug, PartialEq, Eq)] +pub(in crate::raft) struct CellEnrollmentValue { + pub record: EnrollmentRecord, + pub durability: Durability, +} + impl CellEnrollmentCall { pub fn encode(&self, limit: usize) -> Result, TransportError> { if self.route_epoch == 0 || self.birth_digest == [0; 32] { return Err(TransportError::Identity); } - if !(1..=3_600_000).contains(&self.timeout_ms) - || self.request.durability != Durability::Quorum - { + if !(1..=3_600_000).contains(&self.timeout_ms) { return Err(TransportError::Protocol); } let mut body = b"CLGCEQ01".to_vec(); body.extend_from_slice(&self.request.creation.bytes()); body.extend_from_slice(&self.route_epoch.to_be_bytes()); body.extend_from_slice(&self.birth_digest); - body.push(1); + body.push(durability_code(self.request.durability)); body.extend_from_slice(&self.inputs.now_unix_millis.to_be_bytes()); body.extend_from_slice(&self.inputs.random); body.extend_from_slice(&self.preview.bytes()); @@ -55,9 +59,7 @@ impl CellEnrollmentCall { let creation = ProposalId::new(body.array()?).map_err(|_| TransportError::Identity)?; let route_epoch = u64::from_be_bytes(body.array()?); let birth_digest = body.array()?; - if body.byte()? != 1 { - return Err(TransportError::Protocol); - } + let durability = read_durability(&mut body)?; let inputs = CommandInputs { now_unix_millis: u64::from_be_bytes(body.array()?), random: body.array()?, @@ -66,7 +68,7 @@ impl CellEnrollmentCall { let call = Self { request: CellEnrollmentRequest { creation, - durability: Durability::Quorum, + durability, }, inputs, preview, @@ -84,14 +86,15 @@ impl CellEnrollmentCall { } pub(in crate::raft) fn encode_result( - result: &Result, + result: &Result<(&EnrollmentRecord, Durability), CommandFailure>, limit: usize, ) -> Result, TransportError> { let mut body = b"CLGCES01".to_vec(); match result { - Ok(record) => { + Ok((record, durability)) => { let record = record.encode().map_err(|_| TransportError::Protocol)?; body.push(0); + body.push(durability_code(*durability)); body.extend_from_slice(&(record.len() as u32).to_be_bytes()); body.extend_from_slice(&record); } @@ -112,11 +115,11 @@ pub(in crate::raft) fn encode_result( /// creation data, never a signed version token or proof of the current cell image. pub(in crate::raft) fn decode_result( bytes: &[u8], - creation: ProposalId, + expected: (ProposalId, Durability), cell: &CellHistory, schema: [u8; 32], limit: usize, -) -> Result, TransportError> { +) -> Result, TransportError> { if bytes.len() > limit { return Err(TransportError::Limit); } @@ -126,13 +129,18 @@ pub(in crate::raft) fn decode_result( } let result = match body.byte()? { 0 => { + let durability = read_durability(&mut body)?; + if durability < expected.1 { + return Err(TransportError::Authentication); + } let length = u32::from_be_bytes(body.array()?) as usize; let record = EnrollmentRecord::decode(body.take(length)?) .map_err(|_| TransportError::Protocol)?; - if record.creation() != creation || record.cell() != cell || record.schema() != schema { + if record.creation() != expected.0 || record.cell() != cell || record.schema() != schema + { return Err(TransportError::Authentication); } - Ok(record) + Ok(CellEnrollmentValue { record, durability }) } 9 => Err(CommandFailure::CommandConflict), 12 => { diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index ab8e344a1..02500bda7 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -404,7 +404,7 @@ impl IncomingCall { self, state: &State, route: &QueryRoute, - ) -> Result, TransportError> { + ) -> Result { match self { Self::Command(call) => state.execute_cell_command(route, call).await, Self::Enrollment(call) => state.execute_cell_enrollment(route, call).await, @@ -431,6 +431,7 @@ impl IncomingCall { return Err(TransportError::Identity); } wire::encode_result(&result, state.options.max_query_response_bytes) + .map(super::query_replies::Reply::plain) } Self::Shared(call) => { let (_, target, cell) = route.decode_shared()?; @@ -449,6 +450,7 @@ impl IncomingCall { .await .map_err(query_failure); wire::cell::encode_cell_result(&result, state.options.max_query_response_bytes) + .map(super::query_replies::Reply::plain) } } } diff --git a/crates/cellar/durability/raft/query_reply.rs b/crates/cellar/durability/raft/query_reply.rs new file mode 100644 index 000000000..ebb497633 --- /dev/null +++ b/crates/cellar/durability/raft/query_reply.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An authenticated reply owns its S3 output until the physical writer finishes. +use super::query_wire as wire; +use super::transport::TransportError; +use crate::application::consensus::CommandFailure; +use crate::raft::cell_groups::{CellCommandResponse, CellReceiptResponse, CellSqlEnrollment}; +use cellar_logic::consensus::Durability; + +enum Owner { + Command { _response: Box }, + Enrollment { _response: Box }, + Receipt { _response: Box }, +} + +pub(super) struct Reply { + bytes: Vec, + _owner: Option, +} + +impl Reply { + pub(super) fn plain(bytes: Vec) -> Self { + Self { + bytes, + _owner: None, + } + } + + pub(super) fn bytes(&self) -> &[u8] { + &self.bytes + } + + pub(super) fn command( + result: Result, + limit: usize, + ) -> Result { + let bytes = wire::command::encode_result(&result, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Command { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } + + pub(super) fn enrollment( + result: Result, + limit: usize, + ) -> Result { + let encoded = result + .as_ref() + .map(|r| (r.record(), r.durability())) + .map_err(Clone::clone); + let bytes = wire::enrollment::encode_result(&encoded, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Enrollment { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } + + pub(super) fn receipt( + result: Result, + limit: usize, + ) -> Result { + let bytes = wire::receipt::encode_result(&result, limit)?; + let owner = result + .ok() + .filter(|r| r.durability() == Durability::S3) + .map(|response| Owner::Receipt { + _response: Box::new(response), + }); + Ok(Self { + bytes, + _owner: owner, + }) + } +} + +/// Fields drop in declaration order. On cancellation or task destruction, close +/// the real TLS/socket stream before refunding the proof/coverage/hold. On success +/// the caller takes the stream only after flush, then drops the consumed reply. +pub(super) struct Write { + pub stream: S, + pub reply: Reply, +} diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index b5ed91a88..ada418df5 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -675,16 +675,28 @@ impl State { &exporter, &header, &head.tag, - &body, + body.bytes(), &self.options, )?; + // This aggregate closes the stream before dropping the output proof + // on every exit, including destruction of the entire serving task. + let mut writing = super::query_replies::Write { + stream, + reply: body, + }; self.query_until(&pool, until, cancel, permission, async { - write_head(&mut stream, &response).await?; - stream.write_all(&body).await?; - stream.flush().await?; + write_head(&mut writing.stream, &response).await?; + writing.stream.write_all(writing.reply.bytes()).await?; + writing.stream.flush().await?; Ok(()) }) .await?; + let super::query_replies::Write { + stream: resumed, + reply, + } = writing; + stream = resumed; + drop(reply); self.accepted.fetch_add(1, Ordering::Relaxed); } } diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index dff07bee8..ec84c03bf 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -12,7 +12,7 @@ use crate::application::consensus::{ QueryRouting, StaleReadPolicy, VersionKeys, VersionToken, }; use crate::peer_auth::PeerAuth; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::consensus::{Durability, GroupScope, MemberId}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::time::Duration; @@ -26,6 +26,21 @@ pub(super) mod enrollment; #[path = "receipt_cell_wire.rs"] pub(super) mod receipt; +fn durability_code(value: Durability) -> u8 { + match value { + Durability::Quorum => 1, + Durability::S3 => 2, + } +} + +fn read_durability(body: &mut Reader<'_>) -> Result { + match body.byte()? { + 1 => Ok(Durability::Quorum), + 2 => Ok(Durability::S3), + _ => Err(TransportError::Protocol), + } +} + pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; const RESPONSE: &[u8] = b"cellar-raft-query-response-1"; diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index 930563974..c9353392f 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -681,7 +681,14 @@ fn shared_command_wire_preserves_inputs_and_refuses_query_payloads() { assert!(command::CellCommandCall::decode(&bytes, bytes.len() - 1).is_err()); let mut call = call; call.request.durability = Durability::S3; - assert!(call.encode(4096).is_err()); + let bytes = call.encode(4096).unwrap(); + assert_eq!( + command::CellCommandCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); } #[test] @@ -756,6 +763,7 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); let mut body = b"CLGCCS01".to_vec(); body.push(0); + body.push(1); body.extend_from_slice(&call.request.id.client.bytes()); body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); body.extend_from_slice(&9000u64.to_be_bytes()); @@ -764,7 +772,7 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { body.extend_from_slice(b"ok"); let decoded = command::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1001, @@ -778,7 +786,7 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { let wrong = CommandId::new(call.request.id.client, 4).unwrap(); assert!(command::decode_result( &body, - (wrong, cell), + (wrong, cell, Durability::Quorum), [9; 32], &keys(), 1001, @@ -790,7 +798,7 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { other.incarnation = IncarnationId::new([78; 16]).unwrap(); assert!(command::decode_result( &body, - (call.request.id, &other), + (call.request.id, &other, Durability::Quorum), [9; 32], &keys(), 1001, @@ -809,7 +817,7 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { let body = command::encode_result(&Err(failure.clone()), 4096).unwrap(); let decoded = command::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1001, @@ -821,6 +829,105 @@ fn shared_command_results_bind_identity_history_and_typed_refusals() { } } +#[test] +fn shared_s3_results_refuse_durability_downgrades() { + let call = command_call(); + let observation = cell_observation(); + let cell = &observation.version.history; + let token = keys().issue_cell(&observation, [9; 32], 1000).unwrap(); + let creation = enrollment_call().request.creation; + let record = crate::raft::cell_groups::EnrollmentRecord::new( + cell.clone(), + creation, + [9; 32], + observation.version.image, + ) + .unwrap(); + for achieved in [Durability::Quorum, Durability::S3] { + let mut command = b"CLGCCS01".to_vec(); + command.extend_from_slice(&[0, durability_code(achieved)]); + command.extend_from_slice(&call.request.id.client.bytes()); + command.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + command.extend_from_slice(&9000_u64.to_be_bytes()); + string(&mut command, token.as_str()).unwrap(); + command.extend_from_slice(&0_u32.to_be_bytes()); + let mut receipt = b"CLGCRS01".to_vec(); + receipt.extend_from_slice(&[13, durability_code(achieved)]); + receipt.extend_from_slice(&call.request.id.client.bytes()); + receipt.extend_from_slice(&call.request.id.sequence().to_be_bytes()); + string(&mut receipt, token.as_str()).unwrap(); + let enrollment = enrollment::encode_result(&Ok((&record, achieved)), 4096).unwrap(); + for required in [Durability::Quorum, Durability::S3] { + let a = command::decode_result( + &command, + (call.request.id, cell, required), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ); + let b = receipt::decode_result( + &receipt, + (call.request.id, cell, required), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap(), + ); + let c = + enrollment::decode_result(&enrollment, (creation, required), cell, [9; 32], 4096); + if achieved < required { + assert!(matches!(a, Err(TransportError::Authentication))); + assert!(matches!(b, Err(TransportError::Authentication))); + assert!(matches!(c, Err(TransportError::Authentication))); + } else { + assert_eq!(a.unwrap().unwrap().durability, achieved); + assert_eq!(b.unwrap().unwrap().value.durability, achieved); + let value = c.unwrap().unwrap(); + assert_eq!(value.record, record); + assert_eq!(value.durability, achieved); + } + } + // Unknown durability codes must not become either successful contract. + for invalid in [0, 3, 255] { + command[9] = invalid; + receipt[9] = invalid; + let mut enrollment = enrollment.clone(); + enrollment[9] = invalid; + assert!(command::decode_result( + &command, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + assert!(receipt::decode_result( + &receipt, + (call.request.id, cell, Durability::Quorum), + [9; 32], + &keys(), + 1001, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); + assert!(enrollment::decode_result( + &enrollment, + (creation, Durability::Quorum), + cell, + [9; 32], + 4096 + ) + .is_err()); + } + } +} + fn enrollment_call() -> enrollment::CellEnrollmentCall { enrollment::CellEnrollmentCall { request: crate::raft::cell_groups::CellEnrollmentRequest { @@ -863,7 +970,14 @@ fn shared_enrollment_wire_preserves_captured_identity_and_inputs() { assert!(cell::CellQueryCall::decode(&bytes, 4096).is_err()); let mut call = call; call.request.durability = Durability::S3; - assert!(call.encode(4096).is_err()); + let bytes = call.encode(4096).unwrap(); + assert_eq!( + enrollment::CellEnrollmentCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); call.request.durability = Durability::Quorum; call.timeout_ms = 0; assert!(call.encode(4096).is_err()); @@ -928,24 +1042,44 @@ fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { image, ) .unwrap(); - let bytes = enrollment::encode_result(&Ok(record.clone()), 4096).unwrap(); + let bytes = enrollment::encode_result(&Ok((&record, Durability::Quorum)), 4096).unwrap(); assert_eq!( - enrollment::decode_result(&bytes, call.request.creation, &cell, [9; 32], 4096) - .unwrap() - .unwrap(), + enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .unwrap() + .unwrap() + .record, record ); let mut other = cell.clone(); other.incarnation = cellar_logic::consensus::IncarnationId::new([85; 16]).unwrap(); - assert!( - enrollment::decode_result(&bytes, call.request.creation, &other, [9; 32], 4096).is_err() - ); - assert!( - enrollment::decode_result(&bytes, call.request.creation, &cell, [8; 32], 4096).is_err() - ); assert!(enrollment::decode_result( &bytes, - cellar_logic::consensus::ProposalId::new([86; 16]).unwrap(), + (call.request.creation, Durability::Quorum), + &other, + [9; 32], + 4096 + ) + .is_err()); + assert!(enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &cell, + [8; 32], + 4096 + ) + .is_err()); + assert!(enrollment::decode_result( + &bytes, + ( + cellar_logic::consensus::ProposalId::new([86; 16]).unwrap(), + Durability::Quorum + ), &cell, [9; 32], 4096 @@ -954,7 +1088,7 @@ fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { for end in 0..bytes.len() { assert!(enrollment::decode_result( &bytes[..end], - call.request.creation, + (call.request.creation, Durability::Quorum), &cell, [9; 32], 4096 @@ -963,9 +1097,14 @@ fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { } let mut extra = bytes.clone(); extra.push(0); - assert!( - enrollment::decode_result(&extra, call.request.creation, &cell, [9; 32], 4096).is_err() - ); + assert!(enrollment::decode_result( + &extra, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .is_err()); for failure in [ CommandFailure::CommandConflict, CommandFailure::NotLeader { @@ -975,9 +1114,15 @@ fn shared_enrollment_reply_binds_creation_cell_schema_and_initial_image() { ] { let bytes = enrollment::encode_result(&Err(failure.clone()), 4096).unwrap(); assert_eq!( - enrollment::decode_result(&bytes, call.request.creation, &cell, [9; 32], 4096) - .unwrap() - .unwrap_err(), + enrollment::decode_result( + &bytes, + (call.request.creation, Durability::Quorum), + &cell, + [9; 32], + 4096 + ) + .unwrap() + .unwrap_err(), failure ); } @@ -1018,7 +1163,14 @@ fn shared_receipt_wire_refuses_command_payloads_and_preserves_identity() { assert!(receipt::CellReceiptCall::decode(&command_call().encode(4096).unwrap(), 4096).is_err()); let mut call = call; call.request.durability = Durability::S3; - assert!(call.encode(4096).is_err()); + let bytes = call.encode(4096).unwrap(); + assert_eq!( + receipt::CellReceiptCall::decode(&bytes, 4096) + .unwrap() + .request + .durability, + Durability::S3 + ); } #[test] @@ -1031,6 +1183,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { for retained in [false, true] { let mut body = b"CLGCRS01".to_vec(); body.push(if retained { 0 } else { 13 }); + body.push(1); body.extend_from_slice(&call.request.id.client.bytes()); body.extend_from_slice(&call.request.id.sequence().to_be_bytes()); string(&mut body, token.as_str()).unwrap(); @@ -1040,7 +1193,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { } let value = receipt::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1056,7 +1209,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { for end in 0..body.len() { assert!(receipt::decode_result( &body[..end], - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1068,7 +1221,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { let other_id = CommandId::new(call.request.id.client, 99).unwrap(); assert!(receipt::decode_result( &body, - (other_id, cell), + (other_id, cell, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1078,7 +1231,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { .is_err()); assert!(receipt::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [8; 32], &keys(), 1000, @@ -1090,7 +1243,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { other.incarnation = IncarnationId::new([99; 16]).unwrap(); assert!(receipt::decode_result( &body, - (call.request.id, &other), + (call.request.id, &other, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1100,7 +1253,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { .is_err()); assert!(command::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1111,7 +1264,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { body.push(0); assert!(receipt::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, @@ -1129,7 +1282,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { assert_eq!( receipt::decode_result( &body, - (call.request.id, cell), + (call.request.id, cell, Durability::Quorum), [9; 32], &keys(), 1000, diff --git a/crates/cellar/durability/raft/receipt_cell_wire.rs b/crates/cellar/durability/raft/receipt_cell_wire.rs index ddb5be98d..49ecf9e6d 100644 --- a/crates/cellar/durability/raft/receipt_cell_wire.rs +++ b/crates/cellar/durability/raft/receipt_cell_wire.rs @@ -31,7 +31,6 @@ impl CellReceiptCall { || r.contract.operation.len() > 128 || r.contract.operation.chars().any(char::is_control) || !(1..=3_600_000).contains(&self.timeout_ms) - || r.durability != Durability::Quorum { return Err(TransportError::Protocol); } @@ -42,7 +41,7 @@ impl CellReceiptCall { body.extend_from_slice(&self.birth_digest); string(&mut body, &r.contract.operation)?; body.extend_from_slice(&r.contract.digest); - body.push(1); // Quorum; never silently lower an S3 request. + body.push(durability_code(r.durability)); body.extend_from_slice(&self.timeout_ms.to_be_bytes()); let length = u32::try_from(r.input.len()).map_err(|_| TransportError::Limit)?; if body @@ -81,9 +80,7 @@ impl CellReceiptCall { kind: OperationKind::Command, digest: body.array()?, }; - if body.byte()? != 1 { - return Err(TransportError::Protocol); - } + let durability = read_durability(&mut body)?; let timeout_ms = u64::from_be_bytes(body.array()?); if !(1..=3_600_000).contains(&timeout_ms) { return Err(TransportError::Protocol); @@ -98,7 +95,7 @@ impl CellReceiptCall { id, contract, input, - durability: Durability::Quorum, + durability, }, timeout_ms, route_epoch, @@ -115,6 +112,7 @@ pub(in crate::raft) fn encode_result( match result { Ok(response) => { body.push(if response.receipt().is_some() { 0 } else { 13 }); + body.push(durability_code(response.durability())); body.extend_from_slice(&response.command().client.bytes()); body.extend_from_slice(&response.command().sequence().to_be_bytes()); string(&mut body, response.token().as_str())?; @@ -154,7 +152,7 @@ pub(in crate::raft) fn encode_result( pub(in crate::raft) fn decode_result( bytes: &[u8], - expected: (CommandId, &CellHistory), + expected: (CommandId, &CellHistory, Durability), schema: [u8; 32], keys: &VersionKeys, now: u64, @@ -170,6 +168,10 @@ pub(in crate::raft) fn decode_result( } let result = match body.byte()? { status @ (0 | 13) => { + let durability = read_durability(&mut body)?; + if durability < expected.2 { + return Err(TransportError::Authentication); + } let command = command::command_id(&mut body)?; if command != expected.0 { return Err(TransportError::Authentication); @@ -191,6 +193,7 @@ pub(in crate::raft) fn decode_result( retained, value: command::CellCommandValue { command, + durability, retry_until_ms, value: cell::CellQueryValue { result, diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index e7496a2b3..d7a18f245 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -48,6 +48,8 @@ mod cell_commands; mod cell_queries; #[path = "command_origins.rs"] mod command_origins; +#[path = "query_reply.rs"] +mod query_replies; pub use cell_commands::{ CellCommandGateway, CellRemoteCommandResponse, CellRemoteEnrollmentResponse, CellRemoteReceiptResponse, diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index a5815f163..230f9fc34 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -20,6 +20,81 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_outputs_preserve_durability_and_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_outputs( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_deadlines_preserve_ambiguity_and_callbacks() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_deadlines( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_remote_slow_delivery_retains_proofs_and_bounds() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_slow_delivery( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_outputs_preserve_durability_and_creation() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_outputs( + rooted, + rustfs::store("s3-remote-outputs", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_deadlines_preserve_ambiguity_and_callbacks() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_deadlines( + rooted, + rustfs::store("s3-remote-deadlines", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_s3_remote_slow_delivery_retains_proofs_and_bounds() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::s3_remote_slow_delivery( + rooted, + rustfs::store("s3-remote-delivery", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn s3_namespace_minimum_cannot_be_downgraded() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_network_probes.rs b/crates/cellar/tests/support/cell_group_network_probes.rs index d88dc046f..5f77721d6 100644 --- a/crates/cellar/tests/support/cell_group_network_probes.rs +++ b/crates/cellar/tests/support/cell_group_network_probes.rs @@ -22,6 +22,14 @@ impl Connection { pub(super) async fn open_with_exporter( address: std::net::SocketAddr, label: &[u8], + ) -> Result { + Self::open_with_receive_buffer(address, label, None).await + } + + pub(super) async fn open_with_receive_buffer( + address: std::net::SocketAddr, + label: &[u8], + receive_buffer: Option, ) -> Result { let mut roots = rustls::RootCertStore::empty(); roots.add( @@ -36,11 +44,15 @@ impl Connection { .with_root_certificates(roots) .with_no_client_auth(); config.alpn_protocols = vec![b"cellar-raft-1".to_vec()]; + let tcp = if let Some(bytes) = receive_buffer { + let socket = tokio::net::TcpSocket::new_v4()?; + socket.set_recv_buffer_size(bytes)?; + socket.connect(address).await? + } else { + TcpStream::connect(address).await? + }; let stream = tokio_rustls::TlsConnector::from(Arc::new(config)) - .connect( - rustls::pki_types::ServerName::try_from("localhost")?, - TcpStream::connect(address).await?, - ) + .connect(rustls::pki_types::ServerName::try_from("localhost")?, tcp) .await?; let exporter = stream .get_ref() @@ -70,19 +82,35 @@ impl Connection { body: &[u8], ) -> Result<[u8; 32]> { let header = serde_json::to_vec(&header)?; + self.frame_bytes(magic, purpose, Some(binding), &header, body) + .await + } + + pub(super) async fn frame_bytes( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: Option<&[u8]>, + header: &[u8], + body: &[u8], + ) -> Result<[u8; 32]> { let mut prefix = [0; 20]; prefix[..4].copy_from_slice(magic); prefix[4..8].copy_from_slice(&1u32.to_be_bytes()); prefix[8..12].copy_from_slice(&(header.len() as u32).to_be_bytes()); prefix[12..].copy_from_slice(&(body.len() as u64).to_be_bytes()); let mut mac = Hmac::::new_from_slice(&[17; 32])?; - for part in [purpose, &self.exporter, binding, &prefix, &header] { + for part in std::iter::once(purpose) + .chain(std::iter::once(self.exporter.as_slice())) + .chain(binding) + .chain([prefix.as_slice(), header]) + { mac.update(&(part.len() as u64).to_be_bytes()); mac.update(part); } let tag: [u8; 32] = mac.finalize().into_bytes().into(); self.stream.write_all(&prefix).await?; - self.stream.write_all(&header).await?; + self.stream.write_all(header).await?; self.stream.write_all(&tag).await?; self.stream.write_all(body).await?; self.stream.flush().await?; @@ -97,6 +125,20 @@ impl Connection { purpose: &[u8], binding: &[u8], ) -> Result<(Value, Vec, [u8; 32])> { + tokio::time::timeout(Duration::from_secs(5), async { + let (header, length, tag) = self.receive_head(magic, purpose, binding).await?; + let body = self.receive_body(length).await?; + Ok((header, body, tag)) + }) + .await? + } + + pub(super) async fn receive_head( + &mut self, + magic: &[u8; 4], + purpose: &[u8], + binding: &[u8], + ) -> Result<(Value, usize, [u8; 32])> { tokio::time::timeout(Duration::from_secs(5), async { let mut prefix = [0; 20]; self.stream.read_exact(&mut prefix).await?; @@ -115,13 +157,18 @@ impl Connection { mac.update(part); } mac.verify_slice(&tag)?; - let mut body = vec![0; body_len as usize]; - self.stream.read_exact(&mut body).await?; - Ok((serde_json::from_slice(&header)?, body, tag)) + Ok((serde_json::from_slice(&header)?, body_len as usize, tag)) }) .await? } + pub(super) async fn receive_body(&mut self, length: usize) -> Result> { + assert!(length <= 64 * 1024); + let mut body = vec![0; length]; + tokio::time::timeout(Duration::from_secs(5), self.stream.read_exact(&mut body)).await??; + Ok(body) + } + pub(super) async fn refused(mut self) -> Result<()> { let mut byte = [0]; let result = diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 67b4e6c6c..d51ff72b2 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -817,7 +817,7 @@ pub(super) use sql::{owned_command_failure, owned_commands}; pub(super) use sql::{ s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, - s3_receipt_observations, + s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, }; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 51a41e6e5..ce88b5f28 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -5,6 +5,8 @@ mod enrollment; #[path = "cell_group_s3_receipts.rs"] mod receipts; +#[path = "cell_group_s3_remote.rs"] +mod remote; use super::*; pub(crate) use enrollment::{s3_enrollment, s3_enrollment_deadline}; use futures_util::TryStreamExt; @@ -12,6 +14,7 @@ use object_store::{path::Path as ObjectPath, ObjectStore}; pub(crate) use receipts::{ s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, }; +pub(crate) use remote::{s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery}; use std::sync::atomic::{AtomicUsize, Ordering}; struct Harness { @@ -52,6 +55,17 @@ impl Harness { max_commands: usize, minimum: Durability, result_size: Option, + ) -> Result { + Self::configured_transport(rooted, store, max_commands, minimum, result_size, None).await + } + + async fn configured_transport( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -85,6 +99,9 @@ impl Harness { .await?; let (archive, remote) = archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; + if send_buffer.is_some() { + fixture.connect_tcp_with_query_buffer(send_buffer).await?; + } let controllers = controller::start_with_limits( &mut fixture, CellGroupControllerLimits { diff --git a/crates/cellar/tests/support/cell_group_s3_receipts.rs b/crates/cellar/tests/support/cell_group_s3_receipts.rs index 98a4c648d..cf70c5c74 100644 --- a/crates/cellar/tests/support/cell_group_s3_receipts.rs +++ b/crates/cellar/tests/support/cell_group_s3_receipts.rs @@ -4,7 +4,7 @@ use super::*; impl Harness { - fn receipt_request( + pub(super) fn receipt_request( &self, sequence: u64, input: &[u8], @@ -278,24 +278,22 @@ pub(crate) async fn s3_receipt_minimums(rooted: bool, store: Arc Result { + let gateway = h.fixture.transports[0].install_cell_command_gateway( + dispatcher, + h.controllers[0].clone(), + h.fixture.cell_options(&h.cell).await?, + )?; + let router = super::super::super::query_origins::Router::start(&h.fixture).await?; + router + .transport + .install_cell_query_group(h.fixture.tcp_route(h.route.history)?)?; + router.grant(&h.fixture, 1)?; + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 1, + peer: router.peer.clone(), + }, + )?; + Ok(Self { gateway, router }) + } + + async fn close(self) -> Result<()> { + drop(self.gateway); + self.router.close().await + } +} + +fn command_request( + h: &Harness, + sequence: u64, + input: &[u8], + durability: Durability, +) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: h.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability, + retry_until_ms: h.retry_until, + }) +} + +async fn command( + h: &Harness, + remote: &Remote, + sequence: u64, + input: &[u8], + durability: Durability, +) -> Result { + let mut invocation = h + .dispatcher + .command( + h.cell.clone(), + command_request(h, sequence, input, durability)?, + bounds(), + None, + ) + .await?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +async fn receipt( + h: &Harness, + remote: &Remote, + known: bool, + durability: Durability, +) -> Result { + let mut request = h.receipt_request(1, b"remote", durability)?; + if !known { + request.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + } + let mut invocation = h + .dispatcher + .lookup_receipt(h.cell.clone(), request, bounds(), None)?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +async fn enrollment( + h: &Harness, + remote: &Remote, + durability: Durability, +) -> Result { + let mut invocation = h + .dispatcher + .enroll( + h.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([210; 16])?, + durability, + }, + bounds(), + None, + ) + .await?; + invocation.begin().await?; + Ok(invocation + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await?) +} + +fn bounds() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +async fn refunded(h: &Harness) -> Result<()> { + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_s3 == 0 + && h.controllers[0].usage().retained_commands == 0 + && h.fixture.transports[0].usage().query_items == 0) + }) + .await +} + +pub(crate) async fn s3_remote_outputs(rooted: bool, store: Arc) -> Result<()> { + for minimum in [Durability::Quorum, Durability::S3] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(if minimum == Durability::S3 { + "minimum-s3" + } else { + "minimum-quorum" + }), + )); + let mut h = Harness::with_minimum(rooted, store, 1, minimum).await?; + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + let requested = if minimum == Durability::S3 { + Durability::Quorum + } else { + Durability::S3 + }; + let response = command(&h, &remote, 1, b"remote", requested).await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.result(), b"remote"); + assert_eq!(response.observation().version.sequence.get(), 2); + let expected = response.observation().clone(); + drop(response); + refunded(&h).await?; + let found = receipt(&h, &remote, true, Durability::Quorum).await?; + assert_eq!(found.durability(), Durability::S3); + assert_eq!(found.result(), Some(b"remote".as_slice())); + assert_eq!(*found.observation(), expected); + drop(found); + refunded(&h).await?; + let empty = command(&h, &remote, 2, b"", requested).await?; + assert_eq!(empty.result(), b""); + assert_eq!(empty.durability(), Durability::S3); + drop(empty); + refunded(&h).await?; + let unknown = receipt(&h, &remote, false, requested).await?; + assert_eq!(unknown.durability(), Durability::S3); + assert_eq!(unknown.result(), None); + assert_eq!(unknown.observation().version.sequence.get(), 3); + drop(unknown); + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + h.cell.cell = CellId::new("remote-created")?; + let created = enrollment(&h, &remote, requested).await?; + assert_eq!(created.durability(), Durability::S3); + assert_eq!(created.record().cell(), &h.cell); + let original = created.record().clone(); + drop(created); + refunded(&h).await?; + let changed = command(&h, &remote, 1, b"changed", requested).await?; + assert_eq!(changed.durability(), Durability::S3); + drop(changed); + refunded(&h).await?; + let retry = enrollment(&h, &remote, requested).await?; + assert_eq!(retry.record(), &original); + assert_eq!(retry.durability(), Durability::S3); + drop(retry); + refunded(&h).await?; + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + remote.close().await?; + h.close().await?; + } + // Reachable S3 and a bound sender cannot lend an unbound gateway a proof. + let h = Harness::new( + rooted, + Arc::new(object_store::prefix::PrefixStore::new(store, "unbound")), + 1, + ) + .await?; + let unbound = CellGroupDispatcher::new( + &h.fixture.cluster.domain, + h.fixture.cluster.directory.clone(), + h.class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let remote = Remote::start(&h, unbound).await?; + let mut attempt = h + .dispatcher + .command( + h.cell.clone(), + command_request(&h, 1, b"remote", Durability::S3)?, + bounds(), + None, + ) + .await?; + attempt.begin().await?; + let error = attempt + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("unbound command")?; + assert!(error.ambiguous); + assert!(matches!( + *error.cause, + CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability + )) + )); + drop(attempt); + assert!(receipt(&h, &remote, false, Durability::S3).await.is_err()); + assert!(enrollment(&h, &remote, Durability::S3).await.is_err()); + assert_eq!(h.calls.load(Ordering::SeqCst), 0); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + refunded(&h).await?; + remote.close().await?; + h.close().await +} + +async fn deadline_attempt(h: &Harness, remote: &Remote, kind: &str) -> Result<()> { + let bounds = CellRoutingLimits { + timeout: Duration::from_millis(800), + max_attempts: 4, + }; + match kind { + "command" => { + let mut i = h + .dispatcher + .command( + h.cell.clone(), + command_request(h, 1, b"deadline", Durability::S3)?, + bounds, + None, + ) + .await?; + i.begin().await?; + let error = i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("S3 command deadline")?; + assert!(error.ambiguous); + } + "enrollment" => { + let mut i = h + .dispatcher + .enroll( + h.cell.clone(), + CellEnrollmentRequest { + creation: ProposalId::new([210; 16])?, + durability: Durability::S3, + }, + bounds, + None, + ) + .await?; + i.begin().await?; + let error = i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .err() + .context("S3 enrollment deadline")?; + assert!(error.ambiguous); + } + "receipt" => { + let mut request = h.receipt_request(1, b"remote", Durability::S3)?; + request.id = CommandId::new(ClientId::new([189; 16])?, 1)?; + let mut i = h + .dispatcher + .lookup_receipt(h.cell.clone(), request, bounds, None)?; + i.begin().await?; + assert!(i + .execute_remote(&remote.router.transport, MemberId::new(1)?) + .await + .is_err()); + } + _ => anyhow::bail!("deadline kind"), + } + Ok(()) +} + +pub(crate) async fn s3_remote_deadlines(rooted: bool, store: Arc) -> Result<()> { + for kind in ["command", "enrollment", "receipt"] { + for cancel in [false, true] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(format!("{kind}-{cancel}")), + )); + let mut h = Harness::new(rooted, store, 1).await?; + if kind == "enrollment" { + h.cell.cell = CellId::new("deadline-creation")?; + } + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + h.remote.outage.store(true, Ordering::Release); + let mut pending = Box::pin(deadline_attempt(&h, &remote, kind)); + match futures_util::future::select(pending.as_mut(), Box::pin(h.waiting())).await { + futures_util::future::Either::Left((result, _)) => { + anyhow::bail!("{kind} completed before S3 wait: {result:?}") + } + futures_util::future::Either::Right((result, _)) => result?, + } + if cancel { + drop(pending); + } else { + pending.await?; + } + refunded(&h).await?; + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let next = if kind == "command" { 2 } else { 1 }; + let successor = h + .command( + next, + b"successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + drop(successor); + h.remote.outage.store(false, Ordering::Release); + match kind { + "command" => { + let r = command(&h, &remote, 1, b"deadline", Durability::S3).await?; + assert_eq!(r.result(), b"deadline"); + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + } + "enrollment" => { + let r = enrollment(&h, &remote, Durability::S3).await?; + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.initializations.load(Ordering::SeqCst), 1); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + } + "receipt" => { + let r = receipt(&h, &remote, false, Durability::S3).await?; + assert_eq!(r.result(), None); + assert_eq!(r.durability(), Durability::S3); + drop(r); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + } + _ => unreachable!(), + } + refunded(&h).await?; + remote.close().await?; + h.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs new file mode 100644 index 000000000..72bc25e77 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -0,0 +1,271 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A deliberately slow independent TLS reader holds an actual response write. +use super::super::super::super::network_probes::Connection; +use super::*; +use serde::Serialize; + +#[derive(Serialize)] +struct Endpoint<'a> { + node: [u8; 16], + disk: [u8; 16], + process: [u8; 16], + label: &'a str, + member: Option, +} + +#[derive(Serialize)] +#[serde(tag = "kind", content = "identity")] +enum Scope<'a> { + Shared { + group: [[u8; 16]; 4], + cell: &'a str, + incarnation: [u8; 16], + }, +} + +#[derive(Serialize)] +struct Route<'a> { + purpose: &'a str, + scope: Scope<'a>, + schema: [u8; 32], + source: Endpoint<'a>, + target: Endpoint<'a>, +} + +#[derive(Serialize)] +struct Header<'a> { + route: Route<'a>, + sequence: &'a str, + digest: [u8; 32], +} + +fn header(h: &Harness, remote: &Remote, kind: &str, body: &[u8]) -> Result> { + let g = h.route.history; + let target = h.fixture.cluster.voters[0] + .log + .cell_group(&g)? + .identity() + .clone(); + let source = &remote.router.peer; + let route = Route { + purpose: if kind == "command" { + "Command" + } else { + "Receipt" + }, + scope: Scope::Shared { + group: [ + g.cluster.bytes(), + g.namespace.bytes(), + g.group.bytes(), + g.incarnation.bytes(), + ], + cell: h.cell.cell.as_str(), + incarnation: h.cell.incarnation.bytes(), + }, + schema: h.class.schema_digest(), + source: Endpoint { + node: source.node.bytes(), + disk: source.disk.bytes(), + process: source.process.bytes(), + label: &source.label, + member: None, + }, + target: Endpoint { + node: target.node.bytes(), + disk: target.disk.bytes(), + process: target.process.bytes(), + label: "node-1", + member: Some(target.member.get().to_string()), + }, + }; + Ok(serde_json::to_vec(&Header { + route, + sequence: "1", + digest: Sha256::digest(body).into(), + })?) +} + +fn request(h: &Harness, kind: &str) -> Result> { + let request = command_request(h, 1, b"held", Durability::S3)?; + let mut body = if kind == "command" { + b"CLGCCQ01".to_vec() + } else { + b"CLGCRQ01".to_vec() + }; + body.extend_from_slice(&request.id.client.bytes()); + body.extend_from_slice(&request.id.sequence().to_be_bytes()); + body.extend_from_slice(&h.route.epoch.get().to_be_bytes()); + body.extend_from_slice(&h.route.birth_digest); + body.extend_from_slice(&(request.contract.operation.len() as u32).to_be_bytes()); + body.extend_from_slice(request.contract.operation.as_bytes()); + body.extend_from_slice(&request.contract.digest); + body.push(2); + if kind == "command" { + body.extend_from_slice(&h.retry_until.to_be_bytes()); + body.extend_from_slice(&(h.retry_until - 60_000).to_be_bytes()); + body.extend_from_slice(&[23; 32]); + } + body.extend_from_slice(&15_000_u64.to_be_bytes()); + body.extend_from_slice(&(request.input.len() as u32).to_be_bytes()); + body.extend_from_slice(&request.input); + Ok(body) +} + +pub(crate) async fn s3_remote_slow_delivery( + rooted: bool, + store: Arc, +) -> Result<()> { + for kind in ["command", "receipt"] { + for fault in ["drain", "disconnect", "revoke", "timeout"] { + let store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from(format!("{kind}-{fault}")), + )); + // Exactly fills the original 64 KiB S3 envelope (8 KiB + 2*28 KiB). + // Only the real socket buffers are reduced; no node quota is raised. + let h = Harness::configured_transport( + rooted, + store, + 1, + Durability::Quorum, + Some(28 * 1024), + Some(4096), + ) + .await?; + let remote = Remote::start(&h, h.dispatcher.clone()).await?; + if kind == "receipt" { + let r = h + .command(1, b"held", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(r); + } + let mut connection = Connection::open_with_receive_buffer( + h.fixture.transports[0].query_address(), + b"EXPORTER-cellar-raft-query-1", + Some(1024), + ) + .await?; + let body = request(&h, kind)?; + let request_header = header(&h, &remote, kind, &body)?; + let binding = connection + .frame_bytes( + b"CLQP", + b"cellar-raft-query-request-1", + None, + &request_header, + &body, + ) + .await?; + let (reply, length, _) = connection + .receive_head(b"CLQA", b"cellar-raft-query-response-1", &binding) + .await?; + assert_eq!(reply["request"], serde_json::json!(binding)); + assert_eq!(reply["sequence"], "1"); + assert!(length > 28 * 1024); + // The header has physically arrived, but the result body is blocked. + // The native output must still own one proof, full coverage and bytes. + assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); + assert_eq!(h.controllers[0].usage().s3_bytes, 64 * 1024); + assert_eq!(h.archive.tails.usage().operations, 1); + assert_eq!( + h.fixture.appliers[&(0, h.route.history)] + .0 + .usage()? + .retained_operations, + 0 + ); + let successor = h + .command(2, b"successor", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + drop(successor); + assert_eq!(h.controllers[0].usage().retained_s3, 1); + let mut overloaded = h.dispatcher.lookup_receipt( + h.cell.clone(), + h.receipt_request(1, b"held", Durability::S3)?, + bounds(), + None, + )?; + overloaded.begin().await?; + let refused = overloaded + .execute( + &h.fixture.runtimes[0], + &h.fixture.appliers[&(0, h.route.history)].0, + h.fixture.cell_options(&h.cell).await?, + ) + .await + .err() + .context("held wire proof bounds S3 admission")?; + assert!(matches!( + refused, + CellDispatchError::Runtime(RuntimeError::Overloaded) + | CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::Overloaded + )) + )); + drop(overloaded); + match fault { + "drain" => { + let bytes = connection.receive_body(length).await?; + assert_eq!( + &bytes[..8], + if kind == "command" { + b"CLGCCS01" + } else { + b"CLGCRS01" + } + ); + assert_eq!(&bytes[8..10], &[0, 2]); + assert_eq!( + reply["digest"], + serde_json::json!(<[u8; 32]>::from(Sha256::digest(&bytes))) + ); + drop(connection); + } + "disconnect" => drop(connection), + "revoke" => { + h.fixture.transports[0].revoke_command_origin( + h.route.history.cluster, + h.route.history.namespace, + remote.router.peer.node, + 2, + )?; + refunded(&h).await?; + assert!(connection.receive_body(length).await.is_err()); + drop(connection); + h.fixture.transports[0].grant_command_origin( + cellar::application::consensus::NamespaceCommandOrigin { + cluster: h.route.history.cluster, + namespace: h.route.history.namespace, + revision: 3, + peer: remote.router.peer.clone(), + }, + )?; + } + "timeout" => { + refunded(&h).await?; + assert!(connection.receive_body(length).await.is_err()); + drop(connection); + } + _ => unreachable!(), + } + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + let retry = command(&h, &remote, 1, b"held", Durability::S3).await?; + assert_eq!(retry.result(), vec![7; 28 * 1024]); + assert_eq!(retry.durability(), Durability::S3); + drop(retry); + refunded(&h).await?; + assert_eq!(h.calls.load(Ordering::SeqCst), 2); + remote.close().await?; + h.close().await?; + } + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 93cbd7b1a..6ca08564b 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -540,7 +540,7 @@ mod s3_response; pub(crate) use s3_commands::{ s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, - s3_receipt_observations, + s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, }; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs index d7d2a57b3..5f2648cc2 100644 --- a/crates/cellar/tests/support/cell_group_tcp.rs +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -25,6 +25,26 @@ pub(super) async fn start( runtime: &CellGroupRuntime, n: usize, ) -> Result { + start_with_query_buffer(runtime, n, None).await +} + +async fn start_with_query_buffer( + runtime: &CellGroupRuntime, + n: usize, + send_buffer: Option, +) -> Result { + let query = if let Some(bytes) = send_buffer { + let socket = tokio::net::TcpSocket::new_v4()?; + socket.set_send_buffer_size(bytes)?; + socket.bind( + "127.0.0.1:0" + .parse() + .map_err(|_| TransportError::Identity)?, + )?; + socket.listen(128)? + } else { + TcpListener::bind("127.0.0.1:0").await? + }; RaftTransport::start_shared( runtime.clone(), Arc::new( @@ -34,7 +54,7 @@ pub(super) async fn start( tls().map_err(|e| TransportError::Invalid(e.to_string()))?, TcpListener::bind("127.0.0.1:0").await?, TcpListener::bind("127.0.0.1:0").await?, - TcpListener::bind("127.0.0.1:0").await?, + query, TransportOptions::default(), ) .await @@ -42,10 +62,15 @@ pub(super) async fn start( impl RuntimeCluster { pub(super) async fn connect_tcp(&mut self) -> Result<()> { + self.connect_tcp_with_query_buffer(None).await + } + + pub(super) async fn connect_tcp_with_query_buffer(&mut self, bytes: Option) -> Result<()> { assert!(self.transports.is_empty()); assert!(self.network.is_empty()); for (n, runtime) in self.runtimes.iter().enumerate() { - self.transports.push(start(runtime, n).await?); + self.transports + .push(start_with_query_buffer(runtime, n, bytes).await?); } for history in self.applied.keys() { let route = self.tcp_route(*history)?; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 5357fdf34..0c3d6c44b 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -720,8 +720,8 @@ through this archive-specific drain. the same execution domain. Execution also requires its exact runtime owner and an active archive registration for the selected group/route. Effective durability is the maximum of the request, namespace minimum and existing receipt minimum. -Remote S3 response paths retain refusal gates until their own output proofs -are integrated, including when the receiver has a configured archive provider. +Remote execution uses the same receiver-side proof gate and retains its output +through physical delivery as specified below. An unbound provider still refuses. A committed SQL receipt resolves its producer before any archive wait. The owner acquires independent S3 byte/item admission before transferring output retention @@ -756,8 +756,7 @@ Idle registrations do not retain replay workspaces. Active consumers can occupy those shared workspaces and backpressure subsequent archival until consumption; this is explicit pool admission, without an unaccounted metadata-only proof cache. -Remote S3 responses, explicit unarchived-byte/disk -admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, +Explicit unarchived-byte/disk admission, physical shared-engine Raft file purge, fenced fresh-incarnation serving, load and process-kill qualification remain required gates. This local command boundary is not the complete CG-06 or public SDK/fleet cutover gate. @@ -767,8 +766,8 @@ The same explicit dispatcher binding and bounded archive lane also settle local enrollments. Required durability is the maximum of the request and immutable namespace minimum, captured in the invocation fingerprint. The exact runtime owner and active archive registration are checked before entering initialization. -Remote enrollment refuses S3 before transmission until its wire proof contract -is integrated; it cannot silently send a Quorum request. +Remote enrollment carries the effective request requirement unchanged and returns +authenticated achieved durability with the original creation record. SQL resolves and closes the initializer before archival. A separately admitted S3 output takes retention of the bounded original creation record before SQL @@ -818,11 +817,41 @@ fence permits later concurrent mutations without retaining a private writer's base. A successful response retains full physical coverage, S3 admission and the journal lock, and reports achieved durability from that private proof. -Remote senders refuse explicit/effective S3 requests. Receivers also refuse S3 -before SQL when already required, or after read-only duplicate inspection when a -stored minimum requires it. Configuring an S3 provider cannot enable an unqualified -wire response. Command duplicate refusal clears resolved producer ownership first. -Remote proof ownership through physical delivery remains a separate required gate. +### Remote S3 response delivery gate + +Command, enrollment and receipt RPCs carry the effective request durability. Each +purpose retains its separate authenticated route and marker-1 payload. Successful +payloads also carry achieved durability, including an ambiguous receipt absence. +The exporter-bound MAC covers the entire response digest and the exact outstanding +request tag, purpose, node/disk/process, group/cell history and schema. Decoders +reject unknown durability codes and any achieved level below the original request. +Version tokens bind command/receipt observations; enrollment binds the original +creation/history/schema/image record and does not become a current-version token. + +The receiver applies the same local proof gates above, including namespace and +stored receipt minima. A configured provider must own its exact runtime and active +archive registration. An unbound receiver refuses before mutation when S3 is +already required; a stored-minimum duplicate resolves SQL before refusing. The +sender retains its original invocation, captured inputs and deadline. Possible +transmission keeps mutation failures ambiguous; a lost or rejected reply never +proves non-commit and no transport retry invokes callbacks automatically. + +Encoding returns a bounded owned reply. S3 replies retain the native output, +complete physical coverage, S3 admission and journal/remote-hold ownership alongside +the encoded bytes; the existing transport reservation admits those bytes. Quorum +replies release their resolved SQL output as before. The physical write stage owns +the TLS stream and reply together, with stream destruction ordered before proof +release. Flush completes before normal consumption; cancellation, revocation, +deadline or task destruction closes the stream before refunding its proof. No new +per-cell task, timer or proof cache is introduced. Client outputs retain their +existing node/peer byte admission through consumption. + +Qualification must observe an actual blocked TLS body, retained S3/coverage slots, +SQL release, bounded overload, a same-cell Quorum successor and physical refund on +drain, disconnect, timeout and grant revocation. Exact retries must return the +original result or creation record without callback replay. This delivery gate +does not establish outage disk bounds, local Raft file purge, fresh-incarnation +serving, process-kill recovery, independent hosts or load performance. ## Split and merge diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 11fe694bd..f5c22b358 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1767,7 +1767,10 @@ "crates/cellar/tests/support/cell_group_s3_commands.rs", "crates/cellar/durability/raft/cell_groups/archive/enrollment.rs", "crates/cellar/tests/support/cell_group_s3_enrollment.rs", - "crates/cellar/tests/support/cell_group_s3_receipts.rs" + "crates/cellar/tests/support/cell_group_s3_receipts.rs", + "crates/cellar/durability/raft/query_reply.rs", + "crates/cellar/tests/support/cell_group_s3_remote.rs", + "crates/cellar/tests/support/cell_group_s3_remote_wire.rs" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 591597ca7..9e8b53faa 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,14 +71,17 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs thirteen exact ignored RustFS cases, each in materialized and extent modes: +It runs sixteen exact ignored RustFS cases, each in materialized and extent modes: stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 settlement, deadline/cancellation isolation, refusal when a sibling dependency is missing, namespace minimum enforcement, original enrollment creation proofs and enrollment deadline/cancellation isolation. Receipt cases cover later archived observations, namespace/stored minima, typed failures and payload admission, and -read cancellation/deadlines. Catalog setup, archive +read cancellation/deadlines. Remote cases cover all three authenticated output +kinds, namespace/stored minima, original creation provenance, archive-wait +deadlines/cancellation and real TLS backpressure through drain, disconnect, +revocation and timeout. Catalog setup, archive objects, conditional registry updates and independent recovery use the actual S3 client. Fault injection surrounds that client; lost publication responses are injected after the real conditional write commits. No memory fallback exists. @@ -90,8 +93,9 @@ admission and fresh routing/leader checks. Enrollment cases preserve the origina creation after later mutations, honor namespace minimums, retain output proofs and retry without rerunning initialization. Receipt lookup preserves ambiguous absence, returns the actual later archived observation, reserves result bytes before fetch, -and tests remote refusal with a provider-configured TLS gateway. Remote S3 -responses remain unqualified. The runner does not qualify local log file purge, +and tests stored-minimum S3 delivery with a provider-configured TLS gateway. +Reduced real socket buffers hold 28 KiB replies inside the original 64 KiB S3 +envelope while a Quorum successor progresses. The runner does not qualify local log file purge, fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index aaa1dc9b8..485cc7dc0 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -1,7 +1,7 @@ #!/usr/bin/env python3 # Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -"""Qualify shared-group archive setup and local command/enrollment/receipt responses against local RustFS. +"""Qualify shared-group archive setup and local/remote command/enrollment/receipt responses against local RustFS. Uses only the recorded owned backend. Each invocation owns a new evidence directory and remote prefix. Credentials are passed through the child environment, @@ -25,6 +25,9 @@ ROOT = Path(__file__).resolve().parents[2] TARGET = "native_cell_groups_archive_controller" CASES = ( + "rustfs_s3_remote_outputs_preserve_durability_and_creation", + "rustfs_s3_remote_deadlines_preserve_ambiguity_and_callbacks", + "rustfs_s3_remote_slow_delivery_retains_proofs_and_bounds", "rustfs_controller_archive_publishes_stable_cuts", "rustfs_controller_archive_reconciles_exact_intent_and_joins_close", "rustfs_unproved_s3_responses_refuse", @@ -112,11 +115,11 @@ def main(): out.mkdir(mode=0o700, parents=True, exist_ok=False) prefix = "cell-groups/" + secrets.token_hex(16) record = { - "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local command/enrollment/receipt S3 settlement", + "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local/remote command/enrollment/receipt S3 settlement", "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " - "No remote S3 acknowledgements, byte/disk outage bound, " + "No byte/disk outage bound, " "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", } From 000e737ebc2a7517ee94722a532ea9159ceefd86 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 17:04:50 -0700 Subject: [PATCH 198/568] fix: scope retained reply ownership --- crates/cellar/durability/raft/query_reply.rs | 4 ++-- crates/cellar/tests/support/cell_group_s3_remote_wire.rs | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/crates/cellar/durability/raft/query_reply.rs b/crates/cellar/durability/raft/query_reply.rs index ebb497633..c341661dc 100644 --- a/crates/cellar/durability/raft/query_reply.rs +++ b/crates/cellar/durability/raft/query_reply.rs @@ -1,10 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! An authenticated reply owns its S3 output until the physical writer finishes. -use super::query_wire as wire; -use super::transport::TransportError; +use super::TransportError; use crate::application::consensus::CommandFailure; use crate::raft::cell_groups::{CellCommandResponse, CellReceiptResponse, CellSqlEnrollment}; +use crate::raft::query_wire as wire; use cellar_logic::consensus::Durability; enum Owner { diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index 72bc25e77..8384fd484 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -178,6 +178,8 @@ pub(crate) async fn s3_remote_slow_delivery( .retained_operations, 0 ); + let node = &h.controllers[0]; + controller::wait_for(async || Ok(node.usage().retained_commands == 0)).await?; let successor = h .command(2, b"successor", Durability::Quorum, Duration::from_secs(15)) .await? From 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 17:24:20 -0700 Subject: [PATCH 199/568] fix: resolve remote fixture router --- crates/cellar/tests/support/cell_group_s3_remote.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_remote.rs b/crates/cellar/tests/support/cell_group_s3_remote.rs index c2590f5f8..e3252dd15 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote.rs @@ -12,7 +12,7 @@ pub(crate) use wire::s3_remote_slow_delivery; struct Remote { gateway: CellCommandGateway, - router: super::super::super::query_origins::Router, + router: super::super::query_origins::Router, } impl Remote { @@ -22,7 +22,7 @@ impl Remote { h.controllers[0].clone(), h.fixture.cell_options(&h.cell).await?, )?; - let router = super::super::super::query_origins::Router::start(&h.fixture).await?; + let router = super::super::query_origins::Router::start(&h.fixture).await?; router .transport .install_cell_query_group(h.fixture.tcp_route(h.route.history)?)?; From 566e2e06884dfb44b3878f39a277e341800d113e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 17:41:24 -0700 Subject: [PATCH 200/568] fix: encode the probe operation length --- crates/cellar/tests/support/cell_group_s3_remote_wire.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index 8384fd484..7b28e5f68 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -98,7 +98,7 @@ fn request(h: &Harness, kind: &str) -> Result> { body.extend_from_slice(&request.id.sequence().to_be_bytes()); body.extend_from_slice(&h.route.epoch.get().to_be_bytes()); body.extend_from_slice(&h.route.birth_digest); - body.extend_from_slice(&(request.contract.operation.len() as u32).to_be_bytes()); + body.extend_from_slice(&(request.contract.operation.len() as u16).to_be_bytes()); body.extend_from_slice(request.contract.operation.as_bytes()); body.extend_from_slice(&request.contract.digest); body.push(2); From 79d6d1ae02a0504353bd22ef92296874487fb28f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 17:53:18 -0700 Subject: [PATCH 201/568] test: observe TLS reader buffering --- crates/cellar/tests/support/cell_group_s3_remote_wire.rs | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index 7b28e5f68..22cf59fd6 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -166,6 +166,12 @@ pub(crate) async fn s3_remote_slow_delivery( assert_eq!(reply["request"], serde_json::json!(binding)); assert_eq!(reply["sequence"], "1"); assert!(length > 28 * 1024); + if h.controllers[0].usage().retained_s3 == 0 { + let usage = h.fixture.transports[0].usage(); + let started = std::time::Instant::now(); + let bytes = connection.receive_body(length).await; + eprintln!("[DEBUG-remote-write] before_body={usage:?}; body={:?}; elapsed={:?}", bytes.as_ref().map(|b| (b.len(), b.get(9).copied(), reply["digest"] == serde_json::json!(<[u8;32]>::from(Sha256::digest(b))))), started.elapsed()); + } // The header has physically arrived, but the result body is blocked. // The native output must still own one proof, full coverage and bytes. assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); From 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 18:00:01 -0700 Subject: [PATCH 202/568] test: hold TLS reads during physical writes --- .../support/cell_group_s3_remote_wire.rs | 91 +++++++++++++------ 1 file changed, 61 insertions(+), 30 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index 22cf59fd6..d1111319d 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -113,6 +113,44 @@ fn request(h: &Harness, kind: &str) -> Result> { Ok(body) } +fn written(h: &Harness) -> Result { + let snapshot = h.fixture.cluster.voters[0].log.performance_snapshot(); + anyhow::ensure!( + snapshot.available && !snapshot.saturated, + "socket counter available" + ); + snapshot + .counters + .iter() + .find(|c| c.name == "transport_query_write_bytes") + .map(|c| c.value) + .context("query socket byte counter") +} + +async fn complete(connection: &mut Connection, binding: &[u8; 32], kind: &str) -> Result<()> { + let (reply, length, _) = connection + .receive_head(b"CLQA", b"cellar-raft-query-response-1", binding) + .await?; + assert_eq!(reply["request"], serde_json::json!(binding)); + assert_eq!(reply["sequence"], "1"); + assert!(length > 28 * 1024); + let bytes = connection.receive_body(length).await?; + assert_eq!( + &bytes[..8], + if kind == "command" { + b"CLGCCS01" + } else { + b"CLGCRS01" + } + ); + assert_eq!(&bytes[8..10], &[0, 2]); + assert_eq!( + reply["digest"], + serde_json::json!(<[u8; 32]>::from(Sha256::digest(&bytes))) + ); + Ok(()) +} + pub(crate) async fn s3_remote_slow_delivery( rooted: bool, store: Arc, @@ -151,6 +189,7 @@ pub(crate) async fn s3_remote_slow_delivery( .await?; let body = request(&h, kind)?; let request_header = header(&h, &remote, kind, &body)?; + let before = written(&h)?; let binding = connection .frame_bytes( b"CLQP", @@ -160,20 +199,19 @@ pub(crate) async fn s3_remote_slow_delivery( &body, ) .await?; - let (reply, length, _) = connection - .receive_head(b"CLQA", b"cellar-raft-query-response-1", &binding) - .await?; - assert_eq!(reply["request"], serde_json::json!(binding)); - assert_eq!(reply["sequence"], "1"); - assert!(length > 28 * 1024); - if h.controllers[0].usage().retained_s3 == 0 { - let usage = h.fixture.transports[0].usage(); - let started = std::time::Instant::now(); - let bytes = connection.receive_body(length).await; - eprintln!("[DEBUG-remote-write] before_body={usage:?}; body={:?}; elapsed={:?}", bytes.as_ref().map(|b| (b.len(), b.get(9).copied(), reply["digest"] == serde_json::json!(<[u8;32]>::from(Sha256::digest(b))))), started.elapsed()); - } - // The header has physically arrived, but the result body is blocked. - // The native output must still own one proof, full coverage and bytes. + // Reading even the header lets TLS prefetch the whole body and + // releases backpressure. Observe successful socket writes below TLS + // before any client read. Fewer than 28 KiB cannot carry this result. + controller::wait_for(async || { + let bytes = written(&h)? + .checked_sub(before) + .context("socket counter monotonic")?; + Ok(bytes >= 4096 + && bytes < 28 * 1024 + && h.controllers[0].usage().retained_s3 == 1 + && h.fixture.transports[0].usage().query_items == 1) + }) + .await?; assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); assert_eq!(h.controllers[0].usage().s3_bytes, 64 * 1024); assert_eq!(h.archive.tails.usage().operations, 1); @@ -217,22 +255,15 @@ pub(crate) async fn s3_remote_slow_delivery( )) )); drop(overloaded); + assert!( + written(&h)? + .checked_sub(before) + .context("socket counter monotonic")? + < 28 * 1024 + ); match fault { "drain" => { - let bytes = connection.receive_body(length).await?; - assert_eq!( - &bytes[..8], - if kind == "command" { - b"CLGCCS01" - } else { - b"CLGCRS01" - } - ); - assert_eq!(&bytes[8..10], &[0, 2]); - assert_eq!( - reply["digest"], - serde_json::json!(<[u8; 32]>::from(Sha256::digest(&bytes))) - ); + complete(&mut connection, &binding, kind).await?; drop(connection); } "disconnect" => drop(connection), @@ -244,7 +275,7 @@ pub(crate) async fn s3_remote_slow_delivery( 2, )?; refunded(&h).await?; - assert!(connection.receive_body(length).await.is_err()); + assert!(complete(&mut connection, &binding, kind).await.is_err()); drop(connection); h.fixture.transports[0].grant_command_origin( cellar::application::consensus::NamespaceCommandOrigin { @@ -257,7 +288,7 @@ pub(crate) async fn s3_remote_slow_delivery( } "timeout" => { refunded(&h).await?; - assert!(connection.receive_body(length).await.is_err()); + assert!(complete(&mut connection, &binding, kind).await.is_err()); drop(connection); } _ => unreachable!(), From 7b69250f993c4e5af58846c98c5e89e11520da62 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 18:05:38 -0700 Subject: [PATCH 203/568] test: measure partial reply observations --- .../tests/support/cell_group_s3_remote_wire.rs | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index d1111319d..6f7dab1f1 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -202,16 +202,25 @@ pub(crate) async fn s3_remote_slow_delivery( // Reading even the header lets TLS prefetch the whole body and // releases backpressure. Observe successful socket writes below TLS // before any client read. Fewer than 28 KiB cannot carry this result. - controller::wait_for(async || { + let mut observed = (0u64, u64::MAX, 0u64, 0usize, 0usize); + let reached = controller::wait_for(async || { let bytes = written(&h)? .checked_sub(before) .context("socket counter monotonic")?; + let s3 = h.controllers[0].usage().retained_s3; + let query = h.fixture.transports[0].usage().query_items; + observed.0 = observed.0.max(bytes); + if s3 > 0 { observed.1 = observed.1.min(bytes); observed.2 = observed.2.max(bytes); } + observed.3 = observed.3.max(s3); + observed.4 = observed.4.max(query); Ok(bytes >= 4096 && bytes < 28 * 1024 - && h.controllers[0].usage().retained_s3 == 1 - && h.fixture.transports[0].usage().query_items == 1) + && s3 == 1 + && query == 1) }) - .await?; + .await; + eprintln!("[DEBUG-remote-write] {kind}-{fault} overall/active_min/active_max/s3/query={observed:?}; now_s3={}; now_query={}", h.controllers[0].usage().retained_s3, h.fixture.transports[0].usage().query_items); + reached?; assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); assert_eq!(h.controllers[0].usage().s3_bytes, 64 * 1024); assert_eq!(h.archive.tails.usage().operations, 1); From 25938fd86d13a2755066660797b07aff955ec1ad Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 18:18:45 -0700 Subject: [PATCH 204/568] test: fill the physical reply socket window One undrained reply fits kernel buffers, and a TLS header read prefetches the body. Fill a bounded window with sequential exact retries and observe the fixed physical-write phase before reading. Preserve original payload, admission and deadline limits. --- crates/cellar/durability/raft/performance.rs | 5 +- .../cellar/durability/raft/query_transport.rs | 24 ++- .../support/cell_group_s3_remote_wire.rs | 160 +++++++++++++----- .../tests/support/raft_metrics_native.rs | 5 +- .../multi-raft/cell-groups/protocols.md | 8 +- tools/multi-raft/README.md | 5 +- 6 files changed, 148 insertions(+), 59 deletions(-) diff --git a/crates/cellar/durability/raft/performance.rs b/crates/cellar/durability/raft/performance.rs index 8d26260af..d1b5fd7b9 100644 --- a/crates/cellar/durability/raft/performance.rs +++ b/crates/cellar/durability/raft/performance.rs @@ -58,6 +58,7 @@ pub(crate) enum Operation { DataQueue, ControlExchange, DataExchange, + QueryReplyWrite, ControlAdmit, DataAdmit, ArchiveAdmissionCheck, @@ -83,7 +84,7 @@ pub(crate) enum Operation { EnginePurge, } -const OPERATIONS: [Operation; 44] = [ +const OPERATIONS: [Operation; 45] = [ Operation::EngineQueue, Operation::EngineFileSync, Operation::EngineDirectorySync, @@ -105,6 +106,7 @@ const OPERATIONS: [Operation; 44] = [ Operation::DataQueue, Operation::ControlExchange, Operation::DataExchange, + Operation::QueryReplyWrite, Operation::ControlAdmit, Operation::DataAdmit, Operation::ArchiveAdmissionCheck, @@ -154,6 +156,7 @@ impl Operation { Self::DataQueue => "transport_data_queue", Self::ControlExchange => "transport_control_exchange", Self::DataExchange => "transport_data_exchange", + Self::QueryReplyWrite => "transport_query_reply_write", Self::ControlAdmit => "transport_control_admit", Self::DataAdmit => "transport_data_admit", Self::ArchiveAdmissionCheck => "archive_admission_check", diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index ada418df5..dc6ada81b 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -684,19 +684,29 @@ impl State { stream, reply: body, }; - self.query_until(&pool, until, cancel, permission, async { - write_head(&mut writing.stream, &response).await?; - writing.stream.write_all(writing.reply.bytes()).await?; - writing.stream.flush().await?; - Ok(()) - }) - .await?; + let observation = self + .performance + .timer(crate::raft::performance::Operation::QueryReplyWrite); + let result = self + .query_until(&pool, until, cancel, permission, async { + write_head(&mut writing.stream, &response).await?; + writing.stream.write_all(writing.reply.bytes()).await?; + writing.stream.flush().await?; + Ok(()) + }) + .await; + if let Err(error) = result { + drop(writing); + observation.finish(false); + return Err(error); + } let super::query_replies::Write { stream: resumed, reply, } = writing; stream = resumed; drop(reply); + observation.finish(true); self.accepted.fetch_add(1, Ordering::Relaxed); } } diff --git a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs index 6f7dab1f1..7f4537e91 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote_wire.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote_wire.rs @@ -40,7 +40,7 @@ struct Header<'a> { digest: [u8; 32], } -fn header(h: &Harness, remote: &Remote, kind: &str, body: &[u8]) -> Result> { +fn header(h: &Harness, remote: &Remote, kind: &str, body: &[u8], sequence: u64) -> Result> { let g = h.route.history; let target = h.fixture.cluster.voters[0] .log @@ -82,7 +82,7 @@ fn header(h: &Harness, remote: &Remote, kind: &str, body: &[u8]) -> Result Result { .context("query socket byte counter") } -async fn complete(connection: &mut Connection, binding: &[u8; 32], kind: &str) -> Result<()> { +fn writes(h: &Harness) -> Result<[u64; 4]> { + let snapshot = h.fixture.cluster.voters[0].log.performance_snapshot(); + anyhow::ensure!( + snapshot.available && !snapshot.saturated, + "write observations available" + ); + let sample = snapshot + .operations + .iter() + .find(|sample| sample.name == "transport_query_reply_write") + .context("physical reply write observation")?; + Ok([ + sample.started, + sample.succeeded, + sample.failed, + sample.abandoned, + ]) +} + +async fn fill( + h: &Harness, + remote: &Remote, + connection: &mut Connection, + kind: &str, +) -> Result> { + let body = request(h, kind)?; + let mut bindings = Vec::new(); + // One exact retry at a time: the next request is sent only after the + // preceding physical flush and resource refund. No callbacks are replayed. + // Bound the undrained window independently of all original node quotas. + for sequence in 1..=128 { + let before = writes(h)?; + let request_header = header(h, remote, kind, &body, sequence)?; + bindings.push( + connection + .frame_bytes( + b"CLQP", + b"cellar-raft-query-request-1", + None, + &request_header, + &body, + ) + .await?, + ); + controller::wait_for(async || { + let state = writes(h)?; + anyhow::ensure!( + state[2..] == before[2..], + "physical write failed before pressure" + ); + Ok(state[0] == before[0] + 1) + }) + .await?; + // A TLS reader would prefetch and empty the kernel window. Observe + // only the server's fixed physical-write phase without client reads. + for _ in 0..3 { + if writes(h)?[1] != before[1] { + break; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } + let state = writes(h)?; + anyhow::ensure!( + state[2..] == before[2..], + "physical write failed while filling window" + ); + if state[1] == before[1] { + anyhow::ensure!( + h.controllers[0].usage().retained_s3 == 1 + && h.fixture.transports[0].usage().query_items == 1, + "incomplete physical write retains its owners" + ); + return Ok(bindings); + } + assert_eq!(state[1], before[1] + 1); + refunded(h).await?; + } + anyhow::bail!("bounded undrained window did not create physical backpressure") +} + +async fn complete( + connection: &mut Connection, + binding: &[u8; 32], + kind: &str, + sequence: usize, +) -> Result<()> { let (reply, length, _) = connection .receive_head(b"CLQA", b"cellar-raft-query-response-1", binding) .await?; assert_eq!(reply["request"], serde_json::json!(binding)); - assert_eq!(reply["sequence"], "1"); + assert_eq!(reply["sequence"], sequence.to_string()); assert!(length > 28 * 1024); let bytes = connection.receive_body(length).await?; assert_eq!( @@ -148,6 +233,18 @@ async fn complete(connection: &mut Connection, binding: &[u8; 32], kind: &str) - reply["digest"], serde_json::json!(<[u8; 32]>::from(Sha256::digest(&bytes))) ); + assert_eq!(&bytes[bytes.len() - 28 * 1024..], vec![7; 28 * 1024]); + Ok(()) +} + +async fn completed_prefix( + connection: &mut Connection, + bindings: &[[u8; 32]], + kind: &str, +) -> Result<()> { + for (i, binding) in bindings.iter().enumerate().take(bindings.len() - 1) { + complete(connection, binding, kind, i + 1).await?; + } Ok(()) } @@ -187,40 +284,9 @@ pub(crate) async fn s3_remote_slow_delivery( Some(1024), ) .await?; - let body = request(&h, kind)?; - let request_header = header(&h, &remote, kind, &body)?; let before = written(&h)?; - let binding = connection - .frame_bytes( - b"CLQP", - b"cellar-raft-query-request-1", - None, - &request_header, - &body, - ) - .await?; - // Reading even the header lets TLS prefetch the whole body and - // releases backpressure. Observe successful socket writes below TLS - // before any client read. Fewer than 28 KiB cannot carry this result. - let mut observed = (0u64, u64::MAX, 0u64, 0usize, 0usize); - let reached = controller::wait_for(async || { - let bytes = written(&h)? - .checked_sub(before) - .context("socket counter monotonic")?; - let s3 = h.controllers[0].usage().retained_s3; - let query = h.fixture.transports[0].usage().query_items; - observed.0 = observed.0.max(bytes); - if s3 > 0 { observed.1 = observed.1.min(bytes); observed.2 = observed.2.max(bytes); } - observed.3 = observed.3.max(s3); - observed.4 = observed.4.max(query); - Ok(bytes >= 4096 - && bytes < 28 * 1024 - && s3 == 1 - && query == 1) - }) - .await; - eprintln!("[DEBUG-remote-write] {kind}-{fault} overall/active_min/active_max/s3/query={observed:?}; now_s3={}; now_query={}", h.controllers[0].usage().retained_s3, h.fixture.transports[0].usage().query_items); - reached?; + let bindings = fill(&h, &remote, &mut connection, kind).await?; + let binding = bindings.last().context("held reply")?; assert_eq!(h.controllers[0].usage().retained_s3, 1, "{kind}-{fault}"); assert_eq!(h.controllers[0].usage().s3_bytes, 64 * 1024); assert_eq!(h.archive.tails.usage().operations, 1); @@ -264,15 +330,11 @@ pub(crate) async fn s3_remote_slow_delivery( )) )); drop(overloaded); - assert!( - written(&h)? - .checked_sub(before) - .context("socket counter monotonic")? - < 28 * 1024 - ); + assert!(written(&h)? > before, "real socket output entered"); match fault { "drain" => { - complete(&mut connection, &binding, kind).await?; + completed_prefix(&mut connection, &bindings, kind).await?; + complete(&mut connection, binding, kind, bindings.len()).await?; drop(connection); } "disconnect" => drop(connection), @@ -284,7 +346,10 @@ pub(crate) async fn s3_remote_slow_delivery( 2, )?; refunded(&h).await?; - assert!(complete(&mut connection, &binding, kind).await.is_err()); + completed_prefix(&mut connection, &bindings, kind).await?; + assert!(complete(&mut connection, binding, kind, bindings.len()) + .await + .is_err()); drop(connection); h.fixture.transports[0].grant_command_origin( cellar::application::consensus::NamespaceCommandOrigin { @@ -297,7 +362,10 @@ pub(crate) async fn s3_remote_slow_delivery( } "timeout" => { refunded(&h).await?; - assert!(complete(&mut connection, &binding, kind).await.is_err()); + completed_prefix(&mut connection, &bindings, kind).await?; + assert!(complete(&mut connection, binding, kind, bindings.len()) + .await + .is_err()); drop(connection); } _ => unreachable!(), diff --git a/crates/cellar/tests/support/raft_metrics_native.rs b/crates/cellar/tests/support/raft_metrics_native.rs index b3f273525..56492abad 100644 --- a/crates/cellar/tests/support/raft_metrics_native.rs +++ b/crates/cellar/tests/support/raft_metrics_native.rs @@ -240,11 +240,12 @@ async fn cluster(groups: usize) -> Result> { let observed = service.performance(); ensure!(observed.available && !observed.saturated && observed.source.len() == 32); ensure!( - observed.operations.len() == 44 && observed.counters.len() == 42, + observed.operations.len() == 45 && observed.counters.len() == 42, "performance inventory changed: {} operations, {} counters", observed.operations.len(), observed.counters.len() ); ensure!(observed.operations.iter().any(|operation| operation.name == "engine_purge")); + ensure!(observed.operations.iter().any(|operation| operation.name == "transport_query_reply_write")); let snapshot_operations = observed.operations.iter().map(|operation| operation.name) .filter(|name| name.starts_with("snapshot_")).collect::>(); ensure!(snapshot_operations == BTreeSet::from(["snapshot_encode", "snapshot_encode_queue", "snapshot_seed_encode", "snapshot_checkpoint_encode", "snapshot_transfer_encode"])); @@ -296,7 +297,7 @@ async fn cluster(groups: usize) -> Result> { // Two gauges, 42 counters, and 29 samples per operation (five outcome // counters, 22 buckets, a sum and a count). This remains independent // of the observed inventory and of the group population. - ensure!(perf_ids.len() == 1320); + ensure!(perf_ids.len() == 1349); let expected = observed.prometheus().lines().filter(|line| !line.starts_with('#') && !line.is_empty()).map(|line| line.split_whitespace().next().unwrap().to_owned()).collect::>(); ensure!(perf_ids == expected && perf_ids.is_disjoint(&identities)); identities.extend(perf_ids); diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 0c3d6c44b..e5dfd0d2a 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -846,7 +846,13 @@ deadline or task destruction closes the stream before refunding its proof. No ne per-cell task, timer or proof cache is introduced. Client outputs retain their existing node/peer byte admission through consumption. -Qualification must observe an actual blocked TLS body, retained S3/coverage slots, +The fixed `transport_query_reply_write` observation measures only response +header/body/flush, independently of archive settlement. Its bounded host-lifetime +histogram is advisory and grants no durability authority. Qualification fills a +bounded socket window with exact retries, sending each only after the preceding +physical flush/refund, without reading replies or changing node quotas. + +Qualification must observe an actual blocked TLS write, retained S3/coverage slots, SQL release, bounded overload, a same-cell Quorum successor and physical refund on drain, disconnect, timeout and grant revocation. Exact retries must return the original result or creation record without callback replay. This delivery gate diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 9e8b53faa..a0496ef3a 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -94,8 +94,9 @@ creation after later mutations, honor namespace minimums, retain output proofs a retry without rerunning initialization. Receipt lookup preserves ambiguous absence, returns the actual later archived observation, reserves result bytes before fetch, and tests stored-minimum S3 delivery with a provider-configured TLS gateway. -Reduced real socket buffers hold 28 KiB replies inside the original 64 KiB S3 -envelope while a Quorum successor progresses. The runner does not qualify local log file purge, +A bounded sequence of exact retries fills reduced real socket buffers without +reading replies. The fixed physical-write observation identifies a held 28 KiB +reply inside the original 64 KiB S3 envelope while a Quorum successor progresses. The runner does not qualify local log file purge, fenced recovery or performance. The voters run as native host tasks, so this is not the container resource gate. From debdc30c8327aad35757c6b849e029c1bbdab691 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 19:01:14 -0700 Subject: [PATCH 205/568] test: account for archive readiness gauges PR #159 added two fixed archive readiness gauges while the resource inventory still expected 111. Require both named gauges and 113 total, preserving one-versus-32-group identity checks. Correct the API count without changing runtime behavior. --- .../cellar/application/consensus/resource_metrics.rs | 2 +- crates/cellar/tests/support/raft_metrics_native.rs | 11 ++++++++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/crates/cellar/application/consensus/resource_metrics.rs b/crates/cellar/application/consensus/resource_metrics.rs index 3946bedcd..e4336803a 100644 --- a/crates/cellar/application/consensus/resource_metrics.rs +++ b/crates/cellar/application/consensus/resource_metrics.rs @@ -55,7 +55,7 @@ fn budget(text: &mut Text, prefix: &str, labels: &str, usage: RaftBudgetUsage) { } impl RaftResourceUsage { - /// Bounded label vocabularies, with 111 samples when SQL, image, group and + /// Bounded label vocabularies, with 113 samples when SQL, image, group and /// transport sources are live, plus 13 for a configured controller. /// A live extent inventory adds 13 samples. /// Unavailable sources expose only their availability gauge, diff --git a/crates/cellar/tests/support/raft_metrics_native.rs b/crates/cellar/tests/support/raft_metrics_native.rs index 56492abad..1a4e008be 100644 --- a/crates/cellar/tests/support/raft_metrics_native.rs +++ b/crates/cellar/tests/support/raft_metrics_native.rs @@ -123,7 +123,16 @@ fn samples(text: &str) -> Result> { "duplicate or malformed sample" ); } - ensure!(identities.len() == 111); + ensure!(identities.len() == 113); + for name in [ + "cellar_raft_groups_archive_current", + "cellar_raft_groups_archive_unconfirmed", + ] { + ensure!( + identities.contains(name), + "missing archive readiness gauge: {name}" + ); + } ensure!(text.contains("cellar_raft_extents_sample_available 0\n")); for identity in &identities { ensure!( From ff216de875ab1f02ab5dce02060c347f5309fa9d Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 19:09:37 -0700 Subject: [PATCH 206/568] docs: record remote S3 delivery qualification Record real RustFS results, physical writer ownership and the next CG-06 gates. Preserve runtime and validation-repair provenance, failed probes and original acceptance requirements. --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 136 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 88 ++++++++++++ 3 files changed, 162 insertions(+), 68 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 24b569e5f..8f2779647 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "cde0eea2f516486578fdee7a4cc0b8be9ba8f72a", - "runtime_revision": "cde0eea2f516486578fdee7a4cc0b8be9ba8f72a", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions.", + "source_revision": "debdc30c8327aad35757c6b849e029c1bbdab691", + "runtime_revision": "25938fd86d13a2755066660797b07aff955ec1ad", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 997390d2b..1555328aa 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,73 +15,79 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial runtime/qualification source is `cde0eea2f516486578fdee7a4cc0b8be9ba8f72a`. -CG-06 remains IN_PROGRESS. Local command, enrollment and receipt S3 responses -require explicit `CellGroupDispatcher::with_s3` binding to the exact controller/ -runtime archive owner. Complete published cuts are independently recovered before -proving an immutable command receipt, original creation record or receipt lookup. -All request/namespace/stored receipt minima and current catalog/leader checks hold. - -Receipt `select` seals one quorum observation under its original operation and -deadline, transfers SQL retention to bounded S3 admission and queues nothing until -consuming `response`. Existing `execute` runs both. Known receipts match exactly; -an initial absence may become a receipt at a later archive. Return the actually -read archived observation/token. No absence proves non-commit or retires a producer. -Payload length is checked before fetching, and the same S3 permit grows within -unchanged node byte/item ceilings. Typed expiry/conflict/gaps and corruption remain -failures. Read admission drops before waiting; no application callback runs. - -All three output kinds scope catalog proof rechecks in the captured domain and -bound them by the original deadline. Receipt responses also use a fresh correlated -quorum read's leader/route fence and version floor; later writes need not invalidate -that read. Successful outputs retain full physical coverage/admission and journal -lock; active consumers backpressure the shared pools, while idle groups keep no -replay workspace. Remote S3 requests and stored-minimum upgrades are still refused -on both ends, even with a provider-configured gateway. Command duplicate refusal -clears resolved producer ownership without granting non-commit evidence. - -Thirteen exact RustFS cases pass in both image forms, plus seven signed probes. -Sixteen check/regression commands pass with 205 distinct Rust cases across six -copied executables, all-target Clippy, format/layout, dependency/docs/preflight/plan -checks, 37 Python tests and system-Python hashing. Independent source/artifact/ -selection and credential-exclusion audits pass. Records, frozen inputs, executables -and the initial direct-caller domain panic/backtrace are under -`.cellar/cell-groups-evidence/CG-06-s3-receipts/`. See the newest -[progress](progress.md), [receipt proof contract](protocols.md#local-receipt-lookup-s3-response-gate) -and [RustFS runner](../../../../tools/multi-raft/cell_groups.py). - -Next implement remote S3 command/enrollment/receipt output ownership through the -actual physical send, with achieved durability and authenticated operation/route/ -version checks under original deadlines. Do not merely remove refusal guards or -serialize a proof-free success. The current `command_cell_transport.rs` handlers -return encoded bytes, dropping the native response before `query_transport.rs` -writes and flushes them; enrollment clones only its creation record. Replace this -bytes-only boundary with a bounded owned reply that retains the proof through -physical write/drain. Authenticate achieved durability in all three payloads and -validate the required minimum on receipt. Qualify held writes, disconnects, -post-commit deadlines and route/leader changes without callback replay or false -non-commit. Add explicit unarchived-byte/disk admission, -physical shared-engine Raft file purge and fenced new-incarnation serving with -complete old-voter lineage inventory, including joins after the recovery cut. -Fencing-authority API plus Docker qualification remains the accepted direction. -Multi-head proof retention, process-kill restart and physical-read shutdown cuts -still need qualification. Incremental tail scheduling and smaller sealed output +The latest partial runtime source is `25938fd86d13a2755066660797b07aff955ec1ad`; +validation-only metric inventory correction is +`debdc30c8327aad35757c6b849e029c1bbdab691`. CG-06 remains IN_PROGRESS. Local and +remote command/enrollment/receipt S3 outputs use the exact receiver +controller/runtime archive owner and complete independently recovered cuts. All +request, namespace and stored receipt minima, original deadlines and current +catalog/leader checks remain enforced. + +Remote requests preserve effective durability. Successful replies authenticate +achieved durability against the original requirement and exact request/purpose/ +peer/group/cell history. Enrollment retains original creation provenance, not a +current-version token. Receipt absence stays ambiguous. Possible transmission +never grants non-commit evidence, and retries do not rerun callbacks. + +`query_reply.rs` retains a native S3 output alongside encoded bytes. Its writer +owns the real TLS stream before the reply, ordering socket destruction before +proof/coverage/admission/journal release on cancellation or task destruction. +Normal flush precedes consumption. Quorum output release and existing node/peer +transport admission are unchanged. Unbound receivers still refuse; configuring +S3 cannot bypass exact runtime/group ownership or complete dependency +verification. + +Sixteen exact RustFS cases pass in both image forms, plus seven signed probes. +The new cases cover all remote output kinds, unbound receivers, minima, creation +retries, archive-wait cancellation/deadlines and actual stalled command/receipt +TLS writes through drain, disconnect, timeout and grant revocation. Bounded +exact retries fill small kernel buffers until the fixed physical-write +observation identifies a held 28 KiB body inside the original 64 KiB S3 +envelope. SQL releases and a same-cell Quorum successor commits while the wire +proof remains held; another S3 operation overloads. Exact retries preserve +results without callback replay. + +Twenty regression/check commands pass with 232 distinct Rust cases across seven +copied executables, eighteen wire codec/authentication cases, runtime all-target Clippy, focused inventory-fixture Clippy, format/layout, +dependency/docs/preflight/plan checks, 37 Python tests and system Python hashing. +The runtime and 231 passing cases retain their original source and output hashes; +the metrics repair and remaining source checks run at the validation successor. +Independent byte checks limit the successor to stricter inventory assertions for +the two readiness gauges already added on main and the matching API comment. +Independent source/hash/selection and credential-exclusion audits pass. Records, +frozen inputs, executables and initial import failures are under +`.cellar/cell-groups-evidence/CG-06-s3-remote/`. See [progress](progress.md), the +[remote response contract](protocols.md#remote-s3-response-delivery-gate) and +[RustFS runner](../../../../tools/multi-raft/cell_groups.py). + +Next add explicit unarchived-byte/disk admission, including a full-backend S3 +outage gate; the archive-wrapper faults above do not qualify that bound. Fresh +destination catalog binding still depends on S3 until replicated transfer seals +are integrated. Resolve that dependency without weakening ownership admission. +Then physical shared-engine Raft file purge must respect every checkpoint, +archive, protected reader, transfer and sibling-group constraint. Fenced +new-incarnation serving must inventory every possible old-voter lineage, +including joins after the selected recovery cut. Fencing-authority API plus +Docker qualification remains accepted. Qualify multi-head leases, process-kill +restart and physical-read shutdown cuts. Incremental tail scheduling and smaller attestations require real retained remote holds and load evidence. -CG-08 effect/cache/result lifetimes and maintenance precede CG-07 public SDK/fleet -cutover, followed by topology, placement, hibernation, managed recovery, load and -final legacy removal. Four original CG-06 cases/full gate remain pending. Voters -are host-native tasks with one local RustFS VM; independent hosts, node-container -limits, measured load, fenced replacement-quorum serving and full hard cutover -remain unqualified. The end-to-end goal remains active and incomplete. - -Preserve enrollment source `c2801b65`, command source `3d564f3d`, RustFS source -`d739bf79`, controller source `7914177d`, catalog-cut `df0ba1f6`, protected -publication `87348477`, isolated verifier `14955f2d`, committed tail `7a29c2a9` -and their source-bound evidence/failures. Exact intents precede remote mutation; -unknown outcomes never allocate a new identity or reset counters. Namespace -binding, stable cut capture, protected publication and complete source-independent -recovery retain their contracts. The controller initially coalesces complete cuts. +CG-08 effect/cache/result lifetimes and maintenance precede CG-07 public +SDK/fleet cutover, followed by topology, placement, hibernation, managed +recovery, measured load and final legacy removal. Four original CG-06 cases/full +gate remain pending. Voters are host-native tasks with one RustFS VM; +independent hosts, container node limits, measured load, fenced replacement +serving and the hard cutover remain unqualified. The end-to-end goal remains +active and incomplete. + +Preserve receipt source `cde0eea2`, enrollment source `c2801b65`, command source +`3d564f3d`, RustFS source `d739bf79`, controller source `7914177d`, catalog-cut +`df0ba1f6`, protected publication `87348477`, isolated verifier `14955f2d`, +committed tail `7a29c2a9` and their source-bound evidence/failures. Exact +intents precede remote mutation; unknown outcomes never allocate a new identity +or reset counters. Namespace binding, stable cut capture, protected publication +and complete source-independent recovery retain their contracts. The controller +initially coalesces complete cuts. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index bde61b7f5..eede47d08 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,94 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 remote S3 output ownership through physical delivery + +Runtime source: `25938fd86d13a2755066660797b07aff955ec1ad`. Validation-only +metric inventory correction: `debdc30c8327aad35757c6b849e029c1bbdab691`. CG-06 +remains IN_PROGRESS. Commands, enrollment and receipt lookups now transmit their +effective S3 requirement and return authenticated achieved durability. Unknown +durability codes and replies below the original requirement refuse before result +use. Existing exporter-bound MACs bind the body digest to the exact request, +purpose, peer process and complete group/cell history. Command/receipt tokens +retain their signed observation checks; enrollment returns the original creation +record, including after later mutations. Marker-1 hard cutover adds no legacy +reader or runtime mode flag. + +The receiver still needs an explicit provider binding to its exact runtime and +active archive group. It uses the independently recovered complete-cut proofs, +namespace/stored minima, current catalog/leader checks and original deadlines +already required locally. An unbound receiver refuses before mutation when S3 is +required. A stored-minimum duplicate resolves SQL before refusing an unbound +owner. Possible transmission keeps mutation failures ambiguous. No transport +retry replays application callbacks, and receipt absence remains ambiguous. + +A bounded owned reply now retains the native S3 output with its encoded bytes. +That output owns physical coverage, S3 admission and journal/remote-hold +lifetime. The existing transport reservation accounts for encoded bytes. Quorum +outputs release resolved SQL as before. During writing, an aggregate owns both +the TLS stream and reply: stream destruction precedes proof release, even if the +entire serving future is destroyed. Successful flush precedes consumption. +Cancellation, revocation and timeout cannot leave an acknowledged proof-free +response in the writer. No per-cell task, timer or metadata-only proof cache is +introduced. + +All sixteen exact RustFS cases pass in both image forms, plus seven signed S3 +protocol probes. Three new cases cover all remote output kinds and namespace/ +stored minima, original creation retries, unbound receiver refusal, archive-wait +deadlines/cancellation and actual slow command/receipt TLS delivery. Independent +wire clients use a bounded sequence of exact retries to fill small real kernel +buffers without reading replies, holding a 28 KiB body within the unchanged 64 +KiB S3 envelope. TLS header reads prefetch and a single reply fits the buffers. +The new fixed physical-write observation identifies the held phase without +client reads. Proof/coverage and byte admission remain retained while SQL is +free and a same-cell Quorum successor commits. Another S3 operation overloads +within the original one-slot limit. Drain succeeds; disconnect, grant revocation +and timeout join physical release. Truncated bodies cannot become successful +replies; exact retries return the original result without callback replay. The +earlier thirteen archive/local-output scenarios also pass. + +Twenty regression/check commands pass with 232 distinct Rust cases across seven +copied executables, including all eighteen wire codec/authentication cases, +receipt/ budget primitives, fixed performance outcomes/cardinality, the four +native shared-group suites and LTX clones. Runtime all-target Clippy, +format/layout, dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Runtime Clippy and 231 passing regressions retain +their original source and hashes. The validation successor fixes the two +archive-readiness gauge names/count already added on main, plus the matching API +comment. Its exact metrics fixture, focused Clippy, format and repository checks +pass. An independently checked two-file delta excludes runtime behavior or quota +changes. Records under `.cellar/cell-groups-evidence/CG-06-s3-remote/` bind 2570 +source hashes, exact selections, logs/executables, frozen plan inputs and +predecessors. Independent source/artifact/selection and credential-exclusion +audit passes. The initial import-path failures retain their exact sources and +outputs. The failed raw probe and copied-executable reproductions also remain +retained: its operation length needed u16, and reading a TLS header prefetched +the full body, while a single undrained reply still fit the socket buffers. The +final probe fills a bounded retry window and observes the physical write phase. +The initial native metrics inventory failure and copied-executable reproduction +remain retained. PR #159 added two fixed readiness gauges while its inventory +expected 111. The correction requires both names and exactly 113 resource +samples, preserving duplicate/type/size checks and one-versus-32-group identity +equality. An explicit released-command admission barrier identifies the +successor phase. No original case or existing node quota was weakened. Four +original CG-06 cases and its full gate remain open. + +Reflection: next implement explicit unarchived-byte/disk admission and qualify +bounded operation during a full S3 outage; the archive-wrapper faults above are +not that gate. Fresh destination catalog binding can still depend on S3 until +replicated transfer seals are integrated; resolve that dependency without +weakening ownership admission. Add safe physical shared-engine Raft file purge +with complete checkpoint/archive/reader/transfer constraints, then fenced +fresh-incarnation serving with every possible old-voter lineage, including joins +after the cut. Fencing-authority API plus Docker qualification remains accepted. +Multi-head hold retention, process-kill restart and physical-read shutdown cuts +still need broader qualification. Incremental committed tails and smaller sealed +attestations backed by actual remote holds remain performance opportunities. +CG-08 lifetimes precede CG-07 SDK/fleet cutover; topology, placement, +hibernation, managed recovery, load and final legacy removal remain. Voters +share one host and RustFS one local VM; independent hosts, measured load and +fenced replacement serving remain unqualified. + ## CG-06 local receipt S3 observations and bounded result reads Source: `cde0eea2f516486578fdee7a4cc0b8be9ba8f72a`. CG-06 remains IN_PROGRESS. Local command, From 66aa95e38fc30b0010bee70e7f6539745f99face Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 19:29:59 -0700 Subject: [PATCH 207/568] feat: gate local compaction on archive coverage Persist the local archive requirement before remote I/O and recover it without a trusted frontier. Independently verified complete coverage opens the exact checkpoint boundary; owner and storage checks prevent direct bypass. Skip blocked automatic captures so apply and sibling groups continue. --- crates/cellar/durability/raft/cache.rs | 16 +++ .../durability/raft/cell_groups/README.md | 13 ++ .../raft/cell_groups/archive_compaction.rs | 94 ++++++++++++++ .../cell_groups/controller/archive/job.rs | 5 + .../cell_groups/controller/archive/mod.rs | 8 ++ .../cell_groups/controller/checkpoints.rs | 9 ++ .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../durability/raft/cell_groups/owner.rs | 3 + crates/cellar/durability/raft/records.rs | 7 ++ .../cellar/durability/raft/snapshot_seed.rs | 1 + crates/cellar/durability/raft/storage.rs | 31 +++++ .../raft/storage_cell_genesis_tests.rs | 49 ++++++++ .../native_cell_groups_archive_controller.rs | 25 ++++ .../support/cell_group_controller_archive.rs | 118 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 4 +- crates/cellar/tests/support/cell_group_sql.rs | 4 +- .../multi-raft/cell-groups/protocols.md | 28 +++++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- tools/multi-raft/cell_groups.py | 1 + 19 files changed, 417 insertions(+), 3 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/archive_compaction.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 5b02f6604..bdcc74e52 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -105,6 +105,7 @@ pub enum SnapshotChange { #[derive(Clone)] pub struct GroupWrite { + pub(crate) require_archive: bool, pub(crate) consensus_pin: Option>, pub(crate) seed_grant: Option>, pub group: GroupLog, @@ -120,6 +121,7 @@ pub struct GroupWrite { impl GroupWrite { pub fn new(group: &GroupLog) -> Self { Self { + require_archive: false, consensus_pin: None, seed_grant: None, group: group.clone(), @@ -190,6 +192,7 @@ impl GroupWrite { /// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] pub struct GroupLog { + pub(crate) archive_compaction: Arc, pub(crate) consensus_claimed: Arc, pub(crate) scratch_directory: std::path::PathBuf, pub(crate) snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, @@ -442,6 +445,7 @@ impl Storage for ConsensusStorage { #[derive(Clone)] pub(crate) struct CachedGroup { + pub archive_required: bool, pub hard_state: HardState, pub conf_state: ConfState, pub membership: GroupMembership, @@ -510,6 +514,11 @@ impl CachedGroup { } pub fn validate(&self, max_entries: usize) -> Result<(), LogError> { + if self.archive_required && self.group_manifest.is_none() { + return Err(LogError::Corrupt( + "archive requirement without cell group".into(), + )); + } if self.group_seed.is_some_and(|index| { index <= 1 || index > self.applied.position.index.get() || self.group_manifest.is_none() }) { @@ -588,6 +597,12 @@ impl CachedGroup { history: GroupHistory, ) -> Result { let mut next = self.clone(); + if write.require_archive { + if self.group_manifest.is_none() { + return Err(LogError::WrongIdentity); + } + next.archive_required = true; + } if let Some(grant) = &write.seed_grant { grant.check(&write.group.owner, history, write.group.identity())?; let Some(SnapshotChange::CellGroup(prepared)) = &write.snapshot else { @@ -949,6 +964,7 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { ..Default::default() }; CachedGroup { + archive_required: false, hard_state, membership: GroupMembership::initial(&conf_state), conf_state, diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 53c736c2e..f4583d042 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -176,6 +176,19 @@ compacts only the prefix represented by the complete checkpoint. Mixed writes and stale cuts refuse. `CheckpointPublished` is emitted after the matching engine completion. Source admission and pins survive accepted publication and shutdown. +Archive registration now fsyncs a required local retention bit before object-store +I/O. That bit is part of the required marker-1 group application record, remains +set through ordinary writes and snapshot installation, and is recovered on open. +The in-memory verified archive frontier is deliberately absent after restart. +Local checkpoint publication and the storage preflight refuse a cut beyond that +frontier. Only the controller's independently verified complete archive coverage +advances it; the public archive status cannot authorize compaction. Its journal +continues to own the current remote hold. The frontier retains bounded marker +metadata, without keeping a replay workspace per idle group. A behind-archive +checkpoint is skipped before capture so quorum apply and sibling checkpoint work +continue. This is a prerequisite for retained-log admission, not its byte/disk +bound or the physical Raft file reclamation gate. + `capture_snapshot` pins the retained checkpoint named by the exact Raft snapshot, including its original membership/configuration, after ordinary apply has advanced the current root. A concurrent checkpoint replacement may refuse capture; a diff --git a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs new file mode 100644 index 000000000..2e8948b21 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs @@ -0,0 +1,94 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local checkpoint compaction follows independently verified archive coverage. +use super::{CellGroupArchiveCoverage, GroupManifest}; +use crate::raft::{cache::GroupLog, storage::LogError}; +use cellar_logic::cell_group::GroupHistory; +use std::sync::Mutex; + +struct State { + required: bool, + covered: Option, +} + +/// Shared by every clone of one engine-local group. The requirement survives +/// restart in the group record; the frontier never survives without rechecking +/// the actual remote hold. No replay workspace or per-cell state is retained. +pub(crate) struct ArchiveCompaction { + state: Mutex, +} + +impl ArchiveCompaction { + pub(crate) fn new(required: bool) -> Self { + Self { + state: Mutex::new(State { + required, + covered: None, + }), + } + } + + pub(crate) fn require(&self) -> Result<(), LogError> { + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + state.required = true; + state.covered = None; + Ok(()) + } + + /// Called only while the controller owns its journaled current remote hold. + /// This historical frontier grants neither an S3 response nor serving rights. + pub(in crate::raft::cell_groups) fn confirm( + &self, + group: &GroupLog, + coverage: &CellGroupArchiveCoverage, + ) -> Result<(), LogError> { + let cache = group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + let marker = coverage.marker(); + let position = marker.position(); + if !cache.archive_required + || marker.route() != current.route() + || position.index > current.position().index + || (position.index == current.position().index && marker != current) + { + return Err(LogError::WrongIdentity); + } + // An old pending intent can reconcile behind a checkpoint selected + // before archival was enabled. Keep compaction closed until a newer + // verified cut reaches the retained local history; do not quarantine a + // valid historical publication or invent its missing local term. + if position.index.get() < cache.snapshot.get_metadata().index { + return Ok(()); + } + if cache.term(position.index.get()) != Some(position.term.get()) { + return Err(LogError::WrongIdentity); + } + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + if !state.required { + return Err(LogError::WrongIdentity); + } + if let Some(previous) = &state.covered { + if previous.position().index > position.index { + return Ok(()); + } + if previous.position().index == position.index && previous != marker { + return Err(LogError::WrongIdentity); + } + } + state.covered = Some(marker.clone()); + Ok(()) + } + + pub(crate) fn permits(&self, marker: &GroupManifest) -> Result { + let state = self.state.lock().map_err(|_| LogError::Closed)?; + Ok(!state.required + || state.covered.as_ref().is_some_and(|covered| { + covered.route() == marker.route() + && covered.position().index >= marker.position().index + && (covered.position().index != marker.position().index || covered == marker) + })) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs index 1935e64ac..2a611240d 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -32,6 +32,11 @@ pub(super) async fn open( runtime: &CellGroupRuntime, execution: &ControllerExecution, ) -> Result { + runtime + .log() + .require_cell_group_archive(®istration.birth.route().history) + .await + .map_err(|error| Error::Runtime(error.into()))?; let registry = registration .directory .open_group_archive(execution, ®istration.io, ®istration.birth) diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs index d14d85717..464c70a3c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -343,6 +343,14 @@ impl Manager { &execution, ) .await?; + let local = runtime + .log() + .cell_group(&group) + .map_err(|error| Error::Runtime(error.into()))?; + local + .archive_compaction + .confirm(&local, &coverage) + .map_err(|error| Error::Runtime(error.into()))?; waiters = settlement::prove( std::mem::take(&mut waiters), context.as_ref().expect("opened archive context"), diff --git a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs index 5a6a0796c..45e4b28b0 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/checkpoints.rs @@ -112,6 +112,15 @@ impl Manager { .checkpoint_manifest() .map_err(RuntimeError::from)? .map_or(1, |root| root.position().index.get()); + // An archive outage must not turn a due checkpoint into an apply stall. + // Recheck the exact captured cut again at owner and storage admission. + if !group + .archive_compaction + .permits(work.base.marker()) + .map_err(RuntimeError::from)? + { + return Ok(false); + } let index = work.base.marker().position().index.get(); if index.saturating_sub(previous) < config.policy.interval_entries { return Ok(false); diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 6a5b9392d..a9bf05dd2 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -16,6 +16,7 @@ pub use crate::raft::transport::{ mod apply; mod archive; +pub(crate) mod archive_compaction; pub use archive::{ CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCollectionLimits, CellGroupArchiveCollectionReport, CellGroupArchiveCoverage, CellGroupArchiveCreation, diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 953bece9c..c717bb1d9 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -641,6 +641,9 @@ impl CellGroupOwner { { return Err(RuntimeError::Overloaded); } + if !self.group.archive_compaction.permits(root.marker())? { + return Err(RuntimeError::Overloaded); + } self.phase = Some(Phase::Checkpoint(checkpoint, false)); Ok(()) } diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index 11f4f4515..c111f317d 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -242,6 +242,8 @@ pub(crate) enum ApplicationRecord { #[serde(deny_unknown_fields)] pub(crate) struct GroupApplication { manifest: super::cell_groups::GroupManifest, + // Required and local. A peer snapshot never changes this retention policy. + archive_required: bool, // Required, local to this mapping/engine, and never snapshot payload. seed: LocalGroupSeed, } @@ -277,6 +279,7 @@ impl GroupRecord { application: match &cache.group_manifest { Some(marker) => ApplicationRecord::Group(Box::new(GroupApplication { manifest: marker.clone(), + archive_required: cache.archive_required, seed: cache.group_seed.map_or(LocalGroupSeed::Absent, |index| { LocalGroupSeed::Installed { index } }), @@ -314,6 +317,10 @@ impl GroupRecord { .transpose()? .flatten(); let cache = CachedGroup { + archive_required: match &self.application { + ApplicationRecord::Group(application) => application.archive_required, + ApplicationRecord::Image(_) => false, + }, hard_state: HardState::parse_from_bytes(&self.hard_state).map_err(corrupt)?, conf_state: ConfState::parse_from_bytes(&self.conf_state).map_err(corrupt)?, membership: self.membership.clone(), diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index f4de5847e..b410c00aa 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -288,6 +288,7 @@ impl PreparedSeed { ..Default::default() }; Ok(crate::raft::cache::CachedGroup { + archive_required: false, hard_state, conf_state: snapshot.get_metadata().get_conf_state().clone(), membership: self.descriptor.membership.clone(), diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 93d6ed6bf..2fab63d53 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -729,6 +729,24 @@ impl ReplicaLog { .await } + /// Enable a local irreversible retention requirement before archive I/O. + /// Operation zero correlates this private idempotent metadata write only; + /// it is never a business proposal or a durability response. + pub(crate) async fn require_cell_group_archive( + &self, + history: &GroupHistory, + ) -> Result<(), LogError> { + let group = self.cell_group(history)?; + group.archive_compaction.require()?; + let required = group.read()?.archive_required; + if !required { + let mut write = GroupWrite::new(&group); + write.require_archive = true; + self.persist_cells(OperationId::new(0), vec![write]).await?; + } + Ok(()) + } + pub async fn persist( &self, operation: OperationId, @@ -1160,6 +1178,11 @@ impl Worker { scope: S, ) -> Result, LogError> { Ok(GroupLog { + archive_compaction: Arc::new( + super::cell_groups::archive_compaction::ArchiveCompaction::new( + cache.archive_required, + ), + ), consensus_claimed: Arc::new(AtomicBool::new(false)), scratch_directory: self.options.directory.clone(), snapshot_scratch: self.snapshot_scratch.clone(), @@ -1451,6 +1474,14 @@ impl Worker { .get(&write.group.key) .ok_or(LogError::WrongIdentity)?; let old = write.group.read()?; + if write.require_archive { + write.group.archive_compaction.require()?; + } + if let Some(SnapshotChange::CellGroupCheckpoint(prepared)) = &write.snapshot { + if !write.group.archive_compaction.permits(prepared.marker())? { + return Err(LogError::Overloaded); + } + } let next = old.changed( write, self.options.max_entries_per_group, diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index d8cb23558..cd9230c16 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -221,3 +221,52 @@ async fn cell_group_genesis_stays_invisible_until_actual_sync() -> anyhow::Resul domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut opts = options(root.path()); + let initial = bootstrap(5); + let log = domain + .run(ReplicaLog::open( + opts.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + assert!(group.archive_compaction.permits(&marker)?); + log.require_cell_group_archive(&initial.route.history) + .await?; + assert!(group.read()?.archive_required); + assert!(!group.clone().archive_compaction.permits(&marker)?); + let cache = group.read()?; + let mapping = MappingRecord::new_cells(group.numeric_key(), &initial, &cache.conf_state)?; + let record = GroupRecord::from_cache(mapping, &cache)?; + drop(cache); + let mut encoded = serde_json::to_value(record)?; + encoded["application"]["state"] + .as_object_mut() + .unwrap() + .remove("archive_required"); + assert!(records::decode::(&serde_json::to_vec(&encoded)?).is_err()); + log.persist_cells(OperationId::new(17), vec![GroupWrite::new(&group)]) + .await?; + assert!(group.read()?.archive_required); + log.close().await?; + opts.mode = LogOpenMode::Resume; + opts.process = ProcessGeneration::new([6; 16])?; + let resumed = domain + .run(ReplicaLog::open( + opts, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let restored = resumed.cell_group(&initial.route.history)?; + assert!(restored.read()?.archive_required); + assert!(!restored.archive_compaction.permits(&marker)?); + resumed.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index 230f9fc34..e11a2e197 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -20,6 +20,31 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn archive_coverage_gates_local_compaction() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_checkpoint_frontier( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_archive_coverage_gates_local_compaction() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::archive_checkpoint_frontier( + rooted, + rustfs::store("archive-compaction", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn s3_remote_outputs_preserve_durability_and_creation() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 33036d3b9..dbe403d34 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -429,3 +429,121 @@ async fn reconcile(fixture: &mut RuntimeCluster, route: &GroupRoute, rooted: boo drop(config); Ok(()) } + +/// A configured archive blocks local compaction, not quorum apply or a sibling's +/// checkpoint lane. Only actual complete coverage can reopen that boundary. +pub(crate) async fn archive_checkpoint_frontier( + rooted: bool, + store: Arc, +) -> Result<()> { + use raft::Storage as _; + let (mut fixture, route) = setup_store(rooted, store).await?; + let second = controller::add_second(&mut fixture, route.history).await?; + fixture.connect_tcp().await?; + let pools: Vec<_> = (0..3) + .map(|_| CellGroupCheckpointPool::new(checkpoint_fixture::limits())) + .collect::>()?; + for (transport, pool) in fixture.transports.iter().zip(&pools) { + transport.enable_group_checkpoints(pool.clone())?; + } + let (config, remote) = registration(&fixture, &route, rooted).await?; + let publication = remote.hold_publication(); + let mut limits = controller::limits(); + limits.checkpoints = Some(CellGroupCheckpointPolicy { + interval_entries: 1, + max_parallel: 1, + }); + let controllers = controller::start_with_limits(&mut fixture, limits).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.checkpoint_manifest()?; + let first = group.first_index()?; + let call = controllers[0].archive(config.clone())?; + tokio::time::timeout(Duration::from_secs(10), publication.entered.notified()).await?; + let later = add(&fixture, &route, rooted, 1).await?; + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(group.first_index()?, first); + assert!(controllers[0] + .archive_status(route.history)? + .unwrap() + .published + .is_none()); + let marker = group.group_manifest()?.context("held archive cut")?; + let captured_group = group.clone(); + let captured_marker = marker.clone(); + let pool = pools[0].clone(); + let prepared = fixture + .cluster + .domain + .blocking(move || { + pool.capture(&captured_group, &captured_marker)? + .prepare_publication() + })? + .await??; + controller::wait_for(async || { + match fixture.runtimes[0] + .publish_checkpoint(route.history, prepared.clone()) + .await + { + Err(RuntimeError::Overloaded) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + other => anyhow::bail!("unverified checkpoint admitted: {other:?}"), + } + }) + .await?; + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(prepared.clone())); + assert!(matches!( + fixture.cluster.voters[0] + .log + .persist_cells(OperationId::new(911), vec![write]) + .await, + Err(LogError::Overloaded) + )); + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(group.first_index()?, first); + + let sibling = fixture.cluster.voters[0].log.cell_group(&second)?; + let sibling_route = sibling + .group_manifest()? + .context("sibling")? + .route() + .clone(); + add(&fixture, &sibling_route, false, 1).await?; + let sibling_base = sibling.group_manifest()?.context("sibling base")?; + add(&fixture, &sibling_route, false, 2).await?; + controller::wait_for(async || { + Ok(sibling + .checkpoint_manifest()? + .as_ref() + .is_some_and(|cut| cut.position().index >= sibling_base.position().index)) + }) + .await?; + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + publication.release.notify_one(); + call.response().await?; + published(&controllers[0], route.history, later).await?; + fixture.runtimes[0] + .publish_checkpoint(route.history, prepared) + .await?; + controller::wait_for(async || Ok(group.checkpoint_manifest()?.as_ref() == Some(&marker))) + .await?; + assert_eq!(group.first_index()?, later.get() + 1); + remote.outage.store(true, Ordering::Release); + add(&fixture, &route, rooted, 2).await?; + let last = add(&fixture, &route, rooted, 3).await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&marker)); + assert_eq!(group.first_index()?, later.get() + 1); + assert!(group.last_index()? >= last.get()); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + remote.outage.store(false, Ordering::Release); + published(&controllers[0], route.history, last).await?; + audit(&fixture, &config, rooted, 3).await?; + for controller in &controllers { + controller.close().await?; + } + assert_eq!(config.tails.usage().operations, 0); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index d51ff72b2..5f6755713 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -806,7 +806,9 @@ mod sql; #[allow(unused_imports)] pub(super) use sql::unproved_s3_responses; #[allow(unused_imports)] -pub(super) use sql::{archive_controller_publish, archive_controller_reconcile}; +pub(super) use sql::{ + archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, +}; #[allow(unused_imports)] pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; pub(super) use sql::{ diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 6ca08564b..979d11d44 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -547,7 +547,9 @@ pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] mod controller; #[allow(unused_imports)] -pub(crate) use archive_controller::{archive_controller_publish, archive_controller_reconcile}; +pub(crate) use archive_controller::{ + archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, +}; #[allow(unused_imports)] pub(crate) use archive_controller::{ archive_controller_publish_store, archive_controller_reconcile_store, diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index e5dfd0d2a..b6864f20b 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -999,3 +999,31 @@ checking the ledger and archive cut. It verifies every cell, old-token rejection receipt retries, new writes and retained-disk restart. Repeat during split/merge crash cuts and after physical log reclamation. This remains group/component recovery, not a globally consistent snapshot of unrelated groups or external effects. + +## Archive-aware local checkpoint compaction + +An archive registration first makes the local group's retention requirement +irreversible through an engine-synced metadata write. The required marker-1 +`archive_required` field belongs to the local group application record; a peer +snapshot neither grants a frontier nor clears this policy. Missing-field records +refuse; no legacy decoder, migration or marker increment is introduced. + +After opening a retained disk, that requirement blocks local checkpoint +compaction until the journaled remote hold has been independently verified again. +The controller advances a private engine-local frontier only from complete archive +coverage. Public `archive_status` remains advisory. Equal-index coverage must +match the exact local root and term; newer coverage cannot be replaced with older +or conflicting evidence. An intent reconciled behind the retained checkpoint +keeps compaction closed until the next verified cut reaches local history. + +The automatic checkpoint scheduler checks the frontier before capture, letting +quorum apply and another group's checkpoint lane progress during archive delay. +The actual owner checks the exact captured cut again; the storage worker repeats +the check before any snapshot compaction batch. An already prepared local +checkpoint cannot bypass the configured requirement through direct persistence. +The frontier retains only bounded marker metadata; the durable archive journal +owns the current remote hold, and no idle per-group replay workspace is kept. + +This gate covers local checkpoint publication. Full-backend outage retention +admission, Raft file purge, peer-install integration of deployment archive policy, +fresh-quorum recovery and the public hard cutover remain separate requirements. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index f5c22b358..951943fb2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1770,7 +1770,8 @@ "crates/cellar/tests/support/cell_group_s3_receipts.rs", "crates/cellar/durability/raft/query_reply.rs", "crates/cellar/tests/support/cell_group_s3_remote.rs", - "crates/cellar/tests/support/cell_group_s3_remote_wire.rs" + "crates/cellar/tests/support/cell_group_s3_remote_wire.rs", + "crates/cellar/durability/raft/cell_groups/archive_compaction.rs" ], "test_target": "native_cell_groups_archive", "steps": [ diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 485cc7dc0..532a8976a 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -25,6 +25,7 @@ ROOT = Path(__file__).resolve().parents[2] TARGET = "native_cell_groups_archive_controller" CASES = ( + "rustfs_archive_coverage_gates_local_compaction", "rustfs_s3_remote_outputs_preserve_durability_and_creation", "rustfs_s3_remote_deadlines_preserve_ambiguity_and_callbacks", "rustfs_s3_remote_slow_delivery_retains_proofs_and_bounds", From ea995319971d023be37e4446c7b87eb467bff2c8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 19:46:20 -0700 Subject: [PATCH 208/568] fix: bind archive policy and capture exact roots Check local birth before changing retention policy. Use the runtime checkpoint capability in the fixture, import its explicit operation identity and finish sibling work before retaining the sole source slot. Preserve original quotas. --- .../cell_groups/controller/archive/mod.rs | 13 +++++ .../support/cell_group_controller_archive.rs | 50 +++++++++---------- tools/multi-raft/README.md | 3 +- 3 files changed, 40 insertions(+), 26 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs index 464c70a3c..237a61479 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -141,6 +141,19 @@ impl CellGroupController { if !shared.groups.contains(&group).map_err(Error::Runtime)? { return Err(Error::Runtime(RuntimeError::NotFound)); } + let local = shared + .runtime + .log() + .cell_group(&group) + .map_err(|error| Error::Runtime(error.into()))?; + if local + .group_manifest() + .map_err(|error| Error::Runtime(error.into()))? + .as_ref() + .is_none_or(|marker| marker.route() != registration.birth.route()) + { + return Err(Error::Invalid("archive registration local group birth")); + } if registration.store.route() != registration.birth.route() || !registration.scratch_directory.is_absolute() || registration.scratch_directory.as_os_str().len() > 4096 diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index dbe403d34..1239c1194 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -2,6 +2,7 @@ //! Real three-voter controllers, conditional object-store I/O and private replay. use super::*; +use cellar_logic::consensus::OperationId; use futures_util::TryStreamExt; use object_store::ObjectStore; use std::sync::atomic::Ordering; @@ -469,18 +470,35 @@ pub(crate) async fn archive_checkpoint_frontier( .unwrap() .published .is_none()); + let sibling = fixture.cluster.voters[0].log.cell_group(&second)?; + let sibling_route = sibling + .group_manifest()? + .context("sibling")? + .route() + .clone(); + add(&fixture, &sibling_route, false, 1).await?; + let sibling_base = sibling.group_manifest()?.context("sibling base")?; + add(&fixture, &sibling_route, false, 2).await?; + controller::wait_for(async || { + Ok(sibling + .checkpoint_manifest()? + .as_ref() + .is_some_and(|cut| cut.position().index >= sibling_base.position().index)) + }) + .await?; + assert_eq!(group.checkpoint_manifest()?, before); + assert_eq!(controllers[0].usage().quarantined_groups, 0); + assert_eq!(pools[0].usage().sources, 0); let marker = group.group_manifest()?.context("held archive cut")?; - let captured_group = group.clone(); - let captured_marker = marker.clone(); - let pool = pools[0].clone(); + let source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pools[0]) + .await?; let prepared = fixture .cluster .domain - .blocking(move || { - pool.capture(&captured_group, &captured_marker)? - .prepare_publication() - })? + .blocking(move || source.prepare_publication())? .await??; + assert_eq!(prepared.marker(), &marker); controller::wait_for(async || { match fixture.runtimes[0] .publish_checkpoint(route.history, prepared.clone()) @@ -504,24 +522,6 @@ pub(crate) async fn archive_checkpoint_frontier( assert_eq!(group.checkpoint_manifest()?, before); assert_eq!(group.first_index()?, first); - let sibling = fixture.cluster.voters[0].log.cell_group(&second)?; - let sibling_route = sibling - .group_manifest()? - .context("sibling")? - .route() - .clone(); - add(&fixture, &sibling_route, false, 1).await?; - let sibling_base = sibling.group_manifest()?.context("sibling base")?; - add(&fixture, &sibling_route, false, 2).await?; - controller::wait_for(async || { - Ok(sibling - .checkpoint_manifest()? - .as_ref() - .is_some_and(|cut| cut.position().index >= sibling_base.position().index)) - }) - .await?; - assert_eq!(group.checkpoint_manifest()?, before); - assert_eq!(controllers[0].usage().quarantined_groups, 0); publication.release.notify_one(); call.response().await?; published(&controllers[0], route.history, later).await?; diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index a0496ef3a..d766c3ff0 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,7 +71,8 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs sixteen exact ignored RustFS cases, each in materialized and extent modes: +It runs seventeen exact ignored RustFS cases, each in materialized and extent modes: +archive-gated local compaction with independent sibling checkpoint progress, stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 settlement, deadline/cancellation isolation, refusal when a sibling dependency From 93b51065410952389f9345a4afa795d2d1971151 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 20:18:44 -0700 Subject: [PATCH 209/568] fix: drain checkpoint admission before archive policy Retain owner and storage admissions through checkpoint completion. Archive activation retries their drain before syncing its irreversible policy, avoiding a source-slot cycle. Scope the recovery test read guard before awaiting physical work. --- .../durability/raft/cell_groups/README.md | 4 +- .../raft/cell_groups/archive_compaction.rs | 50 ++++++++++++++++--- .../cell_groups/controller/archive/job.rs | 5 +- .../durability/raft/cell_groups/owner.rs | 22 +++++--- crates/cellar/durability/raft/storage.rs | 11 ++-- .../raft/storage_cell_genesis_tests.rs | 16 ++++-- .../multi-raft/cell-groups/protocols.md | 7 +++ 7 files changed, 91 insertions(+), 24 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index f4583d042..7a008005e 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -177,7 +177,9 @@ and stale cuts refuse. `CheckpointPublished` is emitted after the matching engin completion. Source admission and pins survive accepted publication and shutdown. Archive registration now fsyncs a required local retention bit before object-store -I/O. That bit is part of the required marker-1 group application record, remains +I/O. Already admitted checkpoints retain owner/storage guards; activation retries +until those guards drain before syncing the requirement. That bit is part of the +required marker-1 group application record, remains set through ordinary writes and snapshot installation, and is recovered on open. The in-memory verified archive frontier is deliberately absent after restart. Local checkpoint publication and the storage preflight refuse a cut beyond that diff --git a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs index 2e8948b21..a286da4a0 100644 --- a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs +++ b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs @@ -4,11 +4,36 @@ use super::{CellGroupArchiveCoverage, GroupManifest}; use crate::raft::{cache::GroupLog, storage::LogError}; use cellar_logic::cell_group::GroupHistory; -use std::sync::Mutex; +use std::sync::{Arc, Mutex}; struct State { required: bool, covered: Option, + admitted: usize, +} + +impl State { + fn permits(&self, marker: &GroupManifest) -> bool { + !self.required + || self.covered.as_ref().is_some_and(|covered| { + covered.route() == marker.route() + && covered.position().index >= marker.position().index + && (covered.position().index != marker.position().index || covered == marker) + }) + } +} + +/// Admission already entered under the preceding policy. Archive activation +/// retries until both the owner and entered storage work have released these +/// guards; it cannot retract a write while retaining its sole source slot. +pub(crate) struct ArchiveCompactionAdmission(Arc); + +impl Drop for ArchiveCompactionAdmission { + fn drop(&mut self) { + if let Ok(mut state) = self.0.state.lock() { + state.admitted -= 1; + } + } } /// Shared by every clone of one engine-local group. The requirement survives @@ -24,12 +49,16 @@ impl ArchiveCompaction { state: Mutex::new(State { required, covered: None, + admitted: 0, }), } } pub(crate) fn require(&self) -> Result<(), LogError> { let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + if state.admitted != 0 { + return Err(LogError::Overloaded); + } state.required = true; state.covered = None; Ok(()) @@ -84,11 +113,18 @@ impl ArchiveCompaction { pub(crate) fn permits(&self, marker: &GroupManifest) -> Result { let state = self.state.lock().map_err(|_| LogError::Closed)?; - Ok(!state.required - || state.covered.as_ref().is_some_and(|covered| { - covered.route() == marker.route() - && covered.position().index >= marker.position().index - && (covered.position().index != marker.position().index || covered == marker) - })) + Ok(state.permits(marker)) + } + + pub(crate) fn admit( + self: &Arc, + marker: &GroupManifest, + ) -> Result, LogError> { + let mut state = self.state.lock().map_err(|_| LogError::Closed)?; + if !state.permits(marker) { + return Ok(None); + } + state.admitted = state.admitted.checked_add(1).ok_or(LogError::Overloaded)?; + Ok(Some(ArchiveCompactionAdmission(self.clone()))) } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs index 2a611240d..56830a845 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -36,7 +36,10 @@ pub(super) async fn open( .log() .require_cell_group_archive(®istration.birth.route().history) .await - .map_err(|error| Error::Runtime(error.into()))?; + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => Error::Runtime(RuntimeError::Overloaded), + error => Error::Runtime(error.into()), + })?; let registry = registration .directory .open_group_archive(execution, ®istration.io, ®istration.birth) diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index c717bb1d9..89d0f1aa5 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -164,7 +164,11 @@ enum Phase { Ready(Box, Term), Light(LightReady), Apply(CellGroupTicket, Box, bool), - Checkpoint(PreparedLocalCellGroupCheckpoint, bool), + Checkpoint( + PreparedLocalCellGroupCheckpoint, + bool, + super::archive_compaction::ArchiveCompactionAdmission, + ), } pub struct CellGroupOwner { @@ -641,10 +645,12 @@ impl CellGroupOwner { { return Err(RuntimeError::Overloaded); } - if !self.group.archive_compaction.permits(root.marker())? { - return Err(RuntimeError::Overloaded); - } - self.phase = Some(Phase::Checkpoint(checkpoint, false)); + let admission = self + .group + .archive_compaction + .admit(root.marker())? + .ok_or(RuntimeError::Overloaded)?; + self.phase = Some(Phase::Checkpoint(checkpoint, false, admission)); Ok(()) } @@ -1066,7 +1072,7 @@ impl CellGroupOwner { entries(light.committed_entries()) + messages(light.messages()) } Some(Phase::Apply(_, next, _)) => next.reserved_bytes() + 4096, - Some(Phase::Checkpoint(checkpoint, _)) => { + Some(Phase::Checkpoint(checkpoint, _, _)) => { checkpoint.root().reserved_bytes() + checkpoint.snapshot().compute_size() as usize + 4096 @@ -1284,12 +1290,12 @@ impl CellGroupOwner { self.messages(light.take_messages())?; self.enqueue(light.take_committed_entries())?; } - Some(Phase::Checkpoint(checkpoint, submitted)) => { + Some(Phase::Checkpoint(checkpoint, submitted, admission)) => { if !submitted { let mut write = GroupWrite::new(&self.group); write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); - self.phase = Some(Phase::Checkpoint(checkpoint, true)); + self.phase = Some(Phase::Checkpoint(checkpoint, true, admission)); return self.persistence(write).map(Some); } let cache = self.group.read()?; diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 2fab63d53..df9f89b98 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1465,6 +1465,7 @@ impl Worker { let mut candidates = Vec::with_capacity(writes.len()); let mut next_bytes = self.cache_bytes; let mut batch = LogBatch::default(); + let mut archive_admissions = Vec::new(); for write in &writes { if !Arc::ptr_eq(&write.group.owner, &self.owner) || !unique.insert(write.group.key) { return Err(LogError::WrongIdentity); @@ -1478,9 +1479,13 @@ impl Worker { write.group.archive_compaction.require()?; } if let Some(SnapshotChange::CellGroupCheckpoint(prepared)) = &write.snapshot { - if !write.group.archive_compaction.permits(prepared.marker())? { - return Err(LogError::Overloaded); - } + archive_admissions.push( + write + .group + .archive_compaction + .admit(prepared.marker())? + .ok_or(LogError::Overloaded)?, + ); } let next = old.changed( write, diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index cd9230c16..1ea77abf9 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -237,14 +237,22 @@ async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Re let group = log.create_cell_group(initial.clone()).await?; let marker = group.group_manifest()?.unwrap(); assert!(group.archive_compaction.permits(&marker)?); + let admitted = group.archive_compaction.admit(&marker)?.unwrap(); + assert!(matches!( + log.require_cell_group_archive(&initial.route.history).await, + Err(LogError::Overloaded) + )); + assert!(!group.read()?.archive_required); + drop(admitted); log.require_cell_group_archive(&initial.route.history) .await?; assert!(group.read()?.archive_required); assert!(!group.clone().archive_compaction.permits(&marker)?); - let cache = group.read()?; - let mapping = MappingRecord::new_cells(group.numeric_key(), &initial, &cache.conf_state)?; - let record = GroupRecord::from_cache(mapping, &cache)?; - drop(cache); + let record = { + let cache = group.read()?; + let mapping = MappingRecord::new_cells(group.numeric_key(), &initial, &cache.conf_state)?; + GroupRecord::from_cache(mapping, &cache)? + }; let mut encoded = serde_json::to_value(record)?; encoded["application"]["state"] .as_object_mut() diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index b6864f20b..d092f9c78 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -1016,6 +1016,13 @@ match the exact local root and term; newer coverage cannot be replaced with olde or conflicting evidence. An intent reconciled behind the retained checkpoint keeps compaction closed until the next verified cut reaches local history. +Activation retries while an earlier checkpoint admission is still owned. Both +the Raft phase and the entered storage batch retain an admission guard through +completion. The requirement is synced only after those guards drain, so it cannot +retract an already entered checkpoint while retaining that checkpoint's sole +source slot. Capacity refusal is retried specifically at this activation boundary. +No mutex guard crosses an await or physical I/O. + The automatic checkpoint scheduler checks the frontier before capture, letting quorum apply and another group's checkpoint lane progress during archive delay. The actual owner checks the exact captured cut again; the storage worker repeats From a86f29e1626d96d0d0686a40f7c224e2d82eb003 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 22:03:05 -0700 Subject: [PATCH 210/568] test: report archive publication wait status --- .../support/cell_group_controller_archive.rs | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 1239c1194..86d938343 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -461,7 +461,23 @@ pub(crate) async fn archive_checkpoint_frontier( let before = group.checkpoint_manifest()?; let first = group.first_index()?; let call = controllers[0].archive(config.clone())?; - tokio::time::timeout(Duration::from_secs(10), publication.entered.notified()).await?; + tokio::time::timeout(Duration::from_secs(10), publication.entered.notified()) + .await + .with_context(|| { + format!( + "archive publication did not enter (rooted={rooted}): {:?}", + controllers[0].archive_status(route.history).map(|status| { + status.map(|status| { + ( + status.attempts, + status.published, + status.error, + status.stopped, + ) + }) + }) + ) + })?; let later = add(&fixture, &route, rooted, 1).await?; assert_eq!(group.checkpoint_manifest()?, before); assert_eq!(group.first_index()?, first); From 8dfe9ff3a316d0579b656af4818dc57d839a6765 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 22:06:18 -0700 Subject: [PATCH 211/568] fix: use valid archive policy operation The public persistence API rejects operation zero. Keep that validation\nand use a private nonzero correlation ID; physical sync sequencing\nremains assigned by the storage worker. --- crates/cellar/durability/raft/storage.rs | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index df9f89b98..870995d0d 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -730,8 +730,9 @@ impl ReplicaLog { } /// Enable a local irreversible retention requirement before archive I/O. - /// Operation zero correlates this private idempotent metadata write only; - /// it is never a business proposal or a durability response. + /// The fixed nonzero operation correlates only this private idempotent + /// metadata write. Its receipt is never a business or durability response; + /// the storage worker assigns the actual durable sync sequence. pub(crate) async fn require_cell_group_archive( &self, history: &GroupHistory, @@ -742,7 +743,7 @@ impl ReplicaLog { if !required { let mut write = GroupWrite::new(&group); write.require_archive = true; - self.persist_cells(OperationId::new(0), vec![write]).await?; + self.persist_cells(OperationId::new(1), vec![write]).await?; } Ok(()) } From 20d30c07b42e8c45461edb59f6a0be9c006b2b97 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 23:08:10 -0700 Subject: [PATCH 212/568] docs: record archive compaction qualification --- .../multi-raft/cell-groups/baseline.json | 12 +- docs/design/multi-raft/cell-groups/handoff.md | 152 ++++++++++-------- .../design/multi-raft/cell-groups/progress.md | 82 ++++++++++ 3 files changed, 169 insertions(+), 77 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 8f2779647..749a7a718 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "debdc30c8327aad35757c6b849e029c1bbdab691", - "runtime_revision": "25938fd86d13a2755066660797b07aff955ec1ad", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending.", + "source_revision": "8dfe9ff3a316d0579b656af4818dc57d839a6765", + "runtime_revision": "8dfe9ff3a316d0579b656af4818dc57d839a6765", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "81ca3a200c2438c7a924d25cdb1a86e17e9a71e182f31df200d02ac882da142a", - "line": 444, - "excerpt": "pub(crate) struct CachedGroup {\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" + "sha256": "476ded3a8a01b2512cd387f839434a54729e1d2aa5afaa6ca63281fbb157f65d", + "line": 447, + "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 1555328aa..e83d12f43 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,79 +15,89 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest partial runtime source is `25938fd86d13a2755066660797b07aff955ec1ad`; -validation-only metric inventory correction is -`debdc30c8327aad35757c6b849e029c1bbdab691`. CG-06 remains IN_PROGRESS. Local and -remote command/enrollment/receipt S3 outputs use the exact receiver -controller/runtime archive owner and complete independently recovered cuts. All -request, namespace and stored receipt minima, original deadlines and current -catalog/leader checks remain enforced. - -Remote requests preserve effective durability. Successful replies authenticate -achieved durability against the original requirement and exact request/purpose/ -peer/group/cell history. Enrollment retains original creation provenance, not a -current-version token. Receipt absence stays ambiguous. Possible transmission -never grants non-commit evidence, and retries do not rerun callbacks. - -`query_reply.rs` retains a native S3 output alongside encoded bytes. Its writer -owns the real TLS stream before the reply, ordering socket destruction before -proof/coverage/admission/journal release on cancellation or task destruction. -Normal flush precedes consumption. Quorum output release and existing node/peer -transport admission are unchanged. Unbound receivers still refuse; configuring -S3 cannot bypass exact runtime/group ownership or complete dependency -verification. - -Sixteen exact RustFS cases pass in both image forms, plus seven signed probes. -The new cases cover all remote output kinds, unbound receivers, minima, creation -retries, archive-wait cancellation/deadlines and actual stalled command/receipt -TLS writes through drain, disconnect, timeout and grant revocation. Bounded -exact retries fill small kernel buffers until the fixed physical-write -observation identifies a held 28 KiB body inside the original 64 KiB S3 -envelope. SQL releases and a same-cell Quorum successor commits while the wire -proof remains held; another S3 operation overloads. Exact retries preserve -results without callback replay. - -Twenty regression/check commands pass with 232 distinct Rust cases across seven -copied executables, eighteen wire codec/authentication cases, runtime all-target Clippy, focused inventory-fixture Clippy, format/layout, -dependency/docs/preflight/plan checks, 37 Python tests and system Python hashing. -The runtime and 231 passing cases retain their original source and output hashes; -the metrics repair and remaining source checks run at the validation successor. -Independent byte checks limit the successor to stricter inventory assertions for -the two readiness gauges already added on main and the matching API comment. -Independent source/hash/selection and credential-exclusion audits pass. Records, -frozen inputs, executables and initial import failures are under -`.cellar/cell-groups-evidence/CG-06-s3-remote/`. See [progress](progress.md), the -[remote response contract](protocols.md#remote-s3-response-delivery-gate) and -[RustFS runner](../../../../tools/multi-raft/cell_groups.py). - -Next add explicit unarchived-byte/disk admission, including a full-backend S3 -outage gate; the archive-wrapper faults above do not qualify that bound. Fresh -destination catalog binding still depends on S3 until replicated transfer seals -are integrated. Resolve that dependency without weakening ownership admission. -Then physical shared-engine Raft file purge must respect every checkpoint, -archive, protected reader, transfer and sibling-group constraint. Fenced -new-incarnation serving must inventory every possible old-voter lineage, -including joins after the selected recovery cut. Fencing-authority API plus -Docker qualification remains accepted. Qualify multi-head leases, process-kill -restart and physical-read shutdown cuts. Incremental tail scheduling and smaller -attestations require real retained remote holds and load evidence. - -CG-08 effect/cache/result lifetimes and maintenance precede CG-07 public -SDK/fleet cutover, followed by topology, placement, hibernation, managed -recovery, measured load and final legacy removal. Four original CG-06 cases/full -gate remain pending. Voters are host-native tasks with one RustFS VM; -independent hosts, container node limits, measured load, fenced replacement -serving and the hard cutover remain unqualified. The end-to-end goal remains -active and incomplete. - -Preserve receipt source `cde0eea2`, enrollment source `c2801b65`, command source -`3d564f3d`, RustFS source `d739bf79`, controller source `7914177d`, catalog-cut +The latest runtime/qualification source is +`8dfe9ff3a316d0579b656af4818dc57d839a6765`. CG-06 remains IN_PROGRESS. Local +checkpoint compaction for an archive-configured group now requires independently +verified complete coverage, with a durable local requirement and a fresh +frontier after reopen. Owner and storage admission guard the exact cut. +Scheduler prechecks skip blocked capture while quorum apply and sibling +checkpoints progress. Activation drains earlier owner/storage checkpoint +admissions before syncing policy, preserving the original one-source quota +without a source-slot cycle. Public archive status grants no compaction or +response authority. The frontier retains bounded metadata; the journal owns the +current remote hold. + +The required marker-1 `archive_required` field belongs to local group +application metadata. Missing-field records refuse without migration or a legacy +reader. Ordinary writes and peer snapshot installation cannot clear the local +policy or manufacture a frontier. Peer-install integration of deployment archive +policy still needs its separate qualification. + +Local and remote command/enrollment/receipt S3 outputs retain their exact +receiver controller/runtime archive owner, complete independently recovered +cuts, original request/namespace/stored minima and deadlines, fresh +catalog/leader checks and authenticated achieved durability. A native output +owns its physical proof through TLS flush or ordered stream destruction. Quorum +output releases SQL independently. No retry replays application callbacks; +missing receipts remain ambiguous. Enrollment retains original creation +provenance. These contracts remain required at the public cutover. + +Seventeen exact RustFS cases pass in both image forms plus seven signed probes. +The new scenario holds real conditional publication while quorum appends and a +sibling checkpoints, refuses direct owner/raw-storage compaction, then accepts +the same prepared checkpoint after complete coverage. Another archive outage +preserves later entries; independent recovery verifies every cell. The storage +case proves guarded activation, missing-field refusal, irreversible policy and +closed compaction after retained-disk reopen. + +Twenty-two regression/check commands pass with 260 Rust cases across seven +copied executables: four native shared-group suites, +storage/snapshot/cache/receipt/budget/journal/physical-join units, eighteen wire +codec cases and fixed performance cardinality. Locked all-target Clippy, +format/layout, dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. + +Independent source/hash/selection and credential-exclusion audit passes. +Records, frozen plan inputs, failed compile/lint/native outputs, copied +reproductions and executables are under +`.cellar/cell-groups-evidence/CG-06-archive-compaction/`. + +The earlier broad slow-reader timeout remains retained; unchanged isolated and +ordered-controller reproductions pass, and its timing cause is unresolved. + +See [progress](progress.md) and the [local compaction contract](protocols.md#archive-aware-local-checkpoint-compaction). + +Next implement explicit unarchived-byte/entry admission across node/group +storage, including inherited peer/control work, pre-callback reservations and +reserved physical/checkpoint progress capacity. Qualify archive activation +fairness under sustained checkpoint traffic before enabling managed capture and +public admission. Qualify a full-backend S3 outage and resolve fresh catalog +admission without weakening ownership or epoch fencing. Archive-wrapper faults +and the compaction prerequisite do not establish that bound. Add safe physical +shared-engine Raft file purge constrained by every local checkpoint, archive, +reader, transfer and sibling group. Fenced fresh-incarnation serving must +inventory every possible old voter, including joins after the selected archive +cut. Fencing-authority API plus Docker qualification remains accepted. +Peer-install policy, controller process-kill restart and +multi-head/physical-read shutdown cuts remain. Incremental tails and smaller +sealed attestations require actual remote holds and load evidence. + +CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover, then +topology, placement, hibernation, managed recovery, measured load and final +legacy removal. Four original CG-06 cases/full gate remain pending. Voters are +host-native tasks and RustFS one VM; independent domains, container node limits, +measured load, fresh-quorum disaster serving and the hard cutover remain +unqualified. The full end-to-end goal remains active. + +Preserve the remote response source `25938fd8` and validation-only inventory +correction `debdc30c`, receipt source `cde0eea2`, enrollment `c2801b65`, command +`3d564f3d`, prior RustFS `d739bf79`, controller `7914177d`, catalog cut `df0ba1f6`, protected publication `87348477`, isolated verifier `14955f2d`, committed tail `7a29c2a9` and their source-bound evidence/failures. Exact -intents precede remote mutation; unknown outcomes never allocate a new identity -or reset counters. Namespace binding, stable cut capture, protected publication -and complete source-independent recovery retain their contracts. The controller -initially coalesces complete cuts. +intents precede remote mutation; uncertain outcomes cannot allocate new +identities or reset counters. Namespace binding, stable cuts, protected +publication and independent complete recovery preserve their contracts. The +archive controller still coalesces complete cuts. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index eede47d08..180843ed2 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,88 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 archive-aware local checkpoint compaction + +Source: `8dfe9ff3a316d0579b656af4818dc57d839a6765`. CG-06 remains IN_PROGRESS. +Archive registration now syncs an irreversible local `archive_required` bit +before object-store I/O. The marker-1 group application record requires this +field, without a legacy decoder or migration. Ordinary writes and peer snapshot +installation preserve the local requirement. Reopen reconstructs the requirement +with no trusted in-memory frontier; the journaled current remote hold must be +independently verified again before local compaction. + +Each engine-local group shares one bounded frontier across its handles. Only +independently verified complete archive coverage advances it. The route/history +and local retained term must match; equal-index coverage must match the exact +root. Older conflicting evidence cannot replace a newer cut. A reconciled intent +behind the retained checkpoint grants no frontier and waits for a newer complete +cut. Archive status remains advisory. The frontier grants neither S3 output nor +serving authority, and retains no replay workspace per idle group. The durable +journal continues to own the current remote hold. + +The automatic scheduler checks the frontier before capture, allowing quorum +apply and a sibling group's checkpoint to continue while archival is held. The +actual Raft owner checks its exact captured cut, and the storage worker repeats +admission before any local checkpoint compaction batch. A prepared source +remains a data capability and cannot bypass either boundary. Already admitted +checkpoints retain owner and physical-storage guards until completion; archive +activation retries their drain before syncing policy. This avoids retracting a +checkpoint while retaining its sole source slot. No mutex spans an await or +physical I/O. Registration also checks the actual local birth before policy +changes. + +The new native and real RustFS scenario holds actual registry publication in +both image forms. Quorum adds another cell while the configured group's +checkpoint and log prefix stay intact. A sibling publishes through the original +one-source pool and one controller checkpoint slot. Direct owner admission and +raw persistence of the prepared local checkpoint both refuse. After actual +complete coverage, the same preparation publishes; another archive outage +preserves later committed log entries without quarantine. Independent recovery +checks every selected cell. The storage regression checks activation refusal +while an earlier admission is held, durable activation after drain, +missing-field codec refusal, inability of ordinary writes to clear policy and +recovered admission remaining closed without fresh coverage. + +All seventeen exact RustFS cases pass in both image forms, with seven signed S3 +probes. All twenty-two regression/check commands pass with 260 distinct Rust +cases across seven copied executables. These include the four native +shared-group suites, all seventeen storage tests, nine snapshot unit cases, +cache/receipt/budget/journal/physical-join primitives, eighteen wire cases and +fixed metric cardinality. Locked all-target Cellar/logic/LTX Clippy, +format/layout, dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Records under +`.cellar/cell-groups-evidence/CG-06-archive-compaction/` bind 2571 source +hashes, exact selections, frozen plan inputs and executable/output hashes. +Independent source/artifact/selection and credential-exclusion audit passes. The +initial capture/import compile refusal, recovery-test lock-scope lint and native +zero-operation refusal remain source-bound. A copied executable reproduced the +native failure; typed status identified the invalid private metadata correlation +ID. The correction uses a valid nonzero ID while preserving public zero-ID +rejection and the independent physical sync sequence. The fixture uses runtime +capture and orders sibling work before retaining its original source slot. +Original cases, oracles and quotas remain unchanged. The first broad regression +attempt timed out in the existing native slow-reader case; unchanged +copied-executable isolated and full ordered-controller reruns passed. The +passing complete rerun retains that failure and its unresolved timing limitation +without changing code, timeouts, quotas or assertions. Four original CG-06 +cases/full gate remain pending. + +Reflection: add explicit node-wide unarchived-byte/entry admission, including +inherited peer and control work, pre-callback reservations and capacity for +checkpoint/physical completion progress. Qualify a full S3 backend outage, +including the catalog-binding dependency without weakening ownership checks. +This compaction gate alone does not establish that disk bound. Follow with +physical shared-engine Raft file purge under every +checkpoint/archive/reader/transfer/sibling constraint, and fenced +fresh-incarnation serving with complete old-voter lineage inventory. +Peer-install integration of deployment archive policy, controller process-kill +and multi-head/physical-read shutdown cuts still need qualification. Archive +activation fairness under sustained checkpoint traffic also needs an adversarial +gate. CG-08 lifetimes precede CG-07 SDK/fleet cutover. Topology, placement, +hibernation, managed recovery, measured load and legacy removal remain. Voters +are host-native and RustFS is one local VM; independent failure domains and the +production hard cutover remain unqualified. + ## CG-06 remote S3 output ownership through physical delivery Runtime source: `25938fd86d13a2755066660797b07aff955ec1ad`. Validation-only From 793236007d1e7dbef14bb9a923a3fd44bb77f3a1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 23:15:26 -0700 Subject: [PATCH 213/568] test: close checkpoint admission during activation --- crates/cellar/durability/raft/storage_cell_genesis_tests.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index 1ea77abf9..d717c9efb 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -243,6 +243,10 @@ async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Re Err(LogError::Overloaded) )); assert!(!group.read()?.archive_required); + assert!( + group.archive_compaction.admit(&marker)?.is_none(), + "a pending archive activation must stop later checkpoint admissions" + ); drop(admitted); log.require_cell_group_archive(&initial.route.history) .await?; From ab0abf26dea4df436b5ced926f28a9ede4f360fd Mon Sep 17 00:00:00 2001 From: forhappy Date: Sat, 31 Oct 2026 23:45:28 -0700 Subject: [PATCH 214/568] fix: own archive activation through storage Close later checkpoint admission while older cut-bound work drains. Carry the same lease into storage so pending policy cannot strand it. Retain activation in entered metadata I/O after its waiter disappears; complete the irreversible requirement only after sync and publication. --- crates/cellar/durability/raft/cache.rs | 10 +- .../durability/raft/cell_groups/README.md | 11 +- .../raft/cell_groups/archive_compaction.rs | 152 +++++++++++-- .../cell_groups/controller/archive/job.rs | 3 +- .../cell_groups/controller/archive/mod.rs | 23 +- .../durability/raft/cell_groups/owner.rs | 3 +- crates/cellar/durability/raft/storage.rs | 40 +++- .../raft/storage_cell_genesis_tests.rs | 209 +++++++++++++++++- .../multi-raft/cell-groups/protocols.md | 22 +- 9 files changed, 428 insertions(+), 45 deletions(-) diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index bdcc74e52..1f12c1a7e 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -105,7 +105,10 @@ pub enum SnapshotChange { #[derive(Clone)] pub struct GroupWrite { - pub(crate) require_archive: bool, + pub(crate) require_archive: + Option>, + pub(crate) checkpoint_admission: + Option>, pub(crate) consensus_pin: Option>, pub(crate) seed_grant: Option>, pub group: GroupLog, @@ -121,7 +124,8 @@ pub struct GroupWrite { impl GroupWrite { pub fn new(group: &GroupLog) -> Self { Self { - require_archive: false, + require_archive: None, + checkpoint_admission: None, consensus_pin: None, seed_grant: None, group: group.clone(), @@ -597,7 +601,7 @@ impl CachedGroup { history: GroupHistory, ) -> Result { let mut next = self.clone(); - if write.require_archive { + if write.require_archive.is_some() { if self.group_manifest.is_none() { return Err(LogError::WrongIdentity); } diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 7a008005e..dbd8e9e90 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -176,10 +176,13 @@ compacts only the prefix represented by the complete checkpoint. Mixed writes and stale cuts refuse. `CheckpointPublished` is emitted after the matching engine completion. Source admission and pins survive accepted publication and shutdown. -Archive registration now fsyncs a required local retention bit before object-store -I/O. Already admitted checkpoints retain owner/storage guards; activation retries -until those guards drain before syncing the requirement. That bit is part of the -required marker-1 group application record, remains +Archive registration owns an activation intent across retries, closing later +checkpoint admission while earlier exact cut-bound leases drain. Owner and +storage share the same lease so entered work can finish under the preceding +policy. Rejected or unentered shutdown work releases its intent; entered metadata +I/O retains it after waiter cancellation and fsyncs the required retention bit +before object-store I/O. Only successful durable publication completes intent. +That bit is part of the required marker-1 group application record, remains set through ordinary writes and snapshot installation, and is recovered on open. The in-memory verified archive frontier is deliberately absent after restart. Local checkpoint publication and the storage preflight refuse a cut beyond that diff --git a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs index a286da4a0..f21ca2c10 100644 --- a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs +++ b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs @@ -4,38 +4,143 @@ use super::{CellGroupArchiveCoverage, GroupManifest}; use crate::raft::{cache::GroupLog, storage::LogError}; use cellar_logic::cell_group::GroupHistory; -use std::sync::{Arc, Mutex}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, +}; struct State { required: bool, covered: Option, admitted: usize, + activations: usize, } impl State { fn permits(&self, marker: &GroupManifest) -> bool { - !self.required - || self.covered.as_ref().is_some_and(|covered| { - covered.route() == marker.route() - && covered.position().index >= marker.position().index - && (covered.position().index != marker.position().index || covered == marker) - }) + self.activations == 0 + && (!self.required + || self.covered.as_ref().is_some_and(|covered| { + covered.route() == marker.route() + && covered.position().index >= marker.position().index + && (covered.position().index != marker.position().index + || covered == marker) + })) } } /// Admission already entered under the preceding policy. Archive activation -/// retries until both the owner and entered storage work have released these -/// guards; it cannot retract a write while retaining its sole source slot. -pub(crate) struct ArchiveCompactionAdmission(Arc); +/// retries until both the owner and entered storage work have released the same +/// cut-bound lease. Storage can finish this work after new admission is closed. +pub(crate) struct ArchiveCompactionAdmission { + gate: Arc, + cut: GroupManifest, +} + +// The existing 4 KiB checkpoint metadata charge covers this lease, including +// both bounded routing-key strings and the Arc allocation's reference counts. +const _: () = assert!( + std::mem::size_of::() + + 2 * cellar_logic::cell_group::MAX_CELL_ID_BYTES + + 2 * std::mem::size_of::() + <= 4096 +); + +impl ArchiveCompactionAdmission { + pub(crate) fn validate( + &self, + gate: &Arc, + cut: &GroupManifest, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) || &self.cut != cut { + return Err(LogError::WrongIdentity); + } + let state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if state.admitted == 0 { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} impl Drop for ArchiveCompactionAdmission { fn drop(&mut self) { - if let Ok(mut state) = self.0.state.lock() { + if let Ok(mut state) = self.gate.state.lock() { state.admitted -= 1; } } } +/// Owned by the controller's registration, and by an entered metadata write. +/// Dropping an unentered intent restores the previous policy. Successful sync +/// makes the requirement irreversible before releasing the admission barrier. +pub(crate) struct ArchiveActivation { + gate: Arc, + pending: AtomicBool, + prepared: AtomicBool, +} + +impl ArchiveActivation { + pub(crate) fn drained(&self, gate: &Arc) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) { + return Err(LogError::WrongIdentity); + } + let mut state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if !self.pending.load(Ordering::Relaxed) { + return if state.required { + Ok(()) + } else { + Err(LogError::WrongIdentity) + }; + } + if state.activations == 0 { + return Err(LogError::WrongIdentity); + } + if state.admitted != 0 { + return Err(LogError::Overloaded); + } + if !self.prepared.swap(true, Ordering::Relaxed) { + // Reset before I/O, once per intent. A delayed duplicate metadata + // completion must not discard coverage verified in the meantime. + state.covered = None; + } + Ok(()) + } + + /// Call only after selecting the durable local requirement, or rechecking + /// an existing durable bit. No failed/uncertain write may complete intent. + pub(crate) fn published(&self, gate: &Arc) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.gate, gate) { + return Err(LogError::WrongIdentity); + } + let mut state = self.gate.state.lock().map_err(|_| LogError::Closed)?; + if !self.pending.load(Ordering::Relaxed) { + return if state.required { + Ok(()) + } else { + Err(LogError::WrongIdentity) + }; + } + if state.activations == 0 || state.admitted != 0 || !self.prepared.load(Ordering::Relaxed) { + return Err(LogError::WrongIdentity); + } + state.required = true; + self.pending.store(false, Ordering::Relaxed); + state.activations -= 1; + Ok(()) + } +} + +impl Drop for ArchiveActivation { + fn drop(&mut self) { + if let Ok(mut state) = self.gate.state.lock() { + if self.pending.swap(false, Ordering::Relaxed) { + state.activations -= 1; + } + } + } +} + /// Shared by every clone of one engine-local group. The requirement survives /// restart in the group record; the frontier never survives without rechecking /// the actual remote hold. No replay workspace or per-cell state is retained. @@ -50,18 +155,22 @@ impl ArchiveCompaction { required, covered: None, admitted: 0, + activations: 0, }), } } - pub(crate) fn require(&self) -> Result<(), LogError> { + pub(crate) fn begin_activation(self: &Arc) -> Result, LogError> { let mut state = self.state.lock().map_err(|_| LogError::Closed)?; - if state.admitted != 0 { - return Err(LogError::Overloaded); - } - state.required = true; - state.covered = None; - Ok(()) + state.activations = state + .activations + .checked_add(1) + .ok_or(LogError::Overloaded)?; + Ok(Arc::new(ArchiveActivation { + gate: self.clone(), + pending: AtomicBool::new(true), + prepared: AtomicBool::new(false), + })) } /// Called only while the controller owns its journaled current remote hold. @@ -119,12 +228,15 @@ impl ArchiveCompaction { pub(crate) fn admit( self: &Arc, marker: &GroupManifest, - ) -> Result, LogError> { + ) -> Result>, LogError> { let mut state = self.state.lock().map_err(|_| LogError::Closed)?; if !state.permits(marker) { return Ok(None); } state.admitted = state.admitted.checked_add(1).ok_or(LogError::Overloaded)?; - Ok(Some(ArchiveCompactionAdmission(self.clone()))) + Ok(Some(Arc::new(ArchiveCompactionAdmission { + gate: self.clone(), + cut: marker.clone(), + }))) } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs index 56830a845..ea7b44918 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/job.rs @@ -31,10 +31,11 @@ pub(super) async fn open( registration: &CellGroupArchiveRegistration, runtime: &CellGroupRuntime, execution: &ControllerExecution, + activation: Arc, ) -> Result { runtime .log() - .require_cell_group_archive(®istration.birth.route().history) + .require_cell_group_archive(®istration.birth.route().history, activation) .await .map_err(|error| match error { crate::raft::storage::LogError::Overloaded => Error::Runtime(RuntimeError::Overloaded), diff --git a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs index 237a61479..9d510023b 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/archive/mod.rs @@ -8,6 +8,7 @@ pub(in crate::raft::cell_groups::controller) mod settlement; use super::calls::Delivery; use super::*; +use crate::raft::cell_groups::archive_compaction::ArchiveActivation; use crate::raft::cell_groups::{archive::ControllerExecution, *}; use cellar_logic::{cell_group::GroupHistory, consensus::LogIndex}; use futures_util::future::BoxFuture; @@ -108,6 +109,7 @@ impl CellGroupArchiveCall { pub(super) struct Request { registration: CellGroupArchiveRegistration, + activation: Arc, reply: RegistrationReply, } @@ -164,11 +166,16 @@ impl CellGroupController { .commands .acquire(8192) .ok_or(Error::Runtime(RuntimeError::Overloaded))?; + let activation = local + .archive_compaction + .begin_activation() + .map_err(|error| Error::Runtime(error.into()))?; let (reply, receiver) = oneshot::channel(); shared .requests .try_send(requests::Request::Archive(Box::new(Request { registration, + activation, reply: RegistrationReply { sender: reply, permit: Arc::new(permit), @@ -200,6 +207,7 @@ impl CellGroupController { struct Entry { registration: CellGroupArchiveRegistration, + activation: Option>, context: Option, reply: Option, queued: bool, @@ -252,6 +260,7 @@ impl Manager { group, Entry { registration: request.registration, + activation: Some(request.activation), context: None, reply: Some(request.reply), queued: true, @@ -336,6 +345,7 @@ impl Manager { status.attempts = status.attempts.saturating_add(1); } let registration = entry.registration.clone(); + let activation = entry.activation.clone(); let mut context = entry.context.clone(); let execution = self.execution.clone(); let runtime = runtime.clone(); @@ -347,7 +357,15 @@ impl Manager { self.job = Some(Box::pin(async move { let result = std::panic::AssertUnwindSafe(async { if context.is_none() { - context = Some(job::open(®istration, &runtime, &execution).await?); + context = Some( + job::open( + ®istration, + &runtime, + &execution, + activation.ok_or(Error::Invalid("missing archive activation"))?, + ) + .await?, + ); } let coverage = job::drive( context.as_ref().expect("opened archive context"), @@ -441,6 +459,9 @@ impl Manager { status.error = Some(Arc::new(error)); } } + if entry.context.is_some() || entry.fatal { + entry.activation.take(); + } if entry.fatal { let error = status .error diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 89d0f1aa5..3564b61c9 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -167,7 +167,7 @@ enum Phase { Checkpoint( PreparedLocalCellGroupCheckpoint, bool, - super::archive_compaction::ArchiveCompactionAdmission, + Arc, ), } @@ -1295,6 +1295,7 @@ impl CellGroupOwner { let mut write = GroupWrite::new(&self.group); write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + write.checkpoint_admission = Some(admission.clone()); self.phase = Some(Phase::Checkpoint(checkpoint, true, admission)); return self.persistence(write).map(Some); } diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 870995d0d..f8b316aeb 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -736,14 +736,17 @@ impl ReplicaLog { pub(crate) async fn require_cell_group_archive( &self, history: &GroupHistory, + activation: Arc, ) -> Result<(), LogError> { let group = self.cell_group(history)?; - group.archive_compaction.require()?; + activation.drained(&group.archive_compaction)?; let required = group.read()?.archive_required; if !required { let mut write = GroupWrite::new(&group); - write.require_archive = true; + write.require_archive = Some(activation); self.persist_cells(OperationId::new(1), vec![write]).await?; + } else { + activation.published(&group.archive_compaction)?; } Ok(()) } @@ -1476,17 +1479,31 @@ impl Worker { .get(&write.group.key) .ok_or(LogError::WrongIdentity)?; let old = write.group.read()?; - if write.require_archive { - write.group.archive_compaction.require()?; + if write.checkpoint_admission.is_some() + && (write.require_archive.is_some() + || !matches!( + &write.snapshot, + Some(SnapshotChange::CellGroupCheckpoint(_)) + )) + { + return Err(LogError::WrongIdentity); + } + if let Some(activation) = &write.require_archive { + activation.drained(&write.group.archive_compaction)?; } if let Some(SnapshotChange::CellGroupCheckpoint(prepared)) = &write.snapshot { - archive_admissions.push( - write + let admission = match &write.checkpoint_admission { + Some(admission) => { + admission.validate(&write.group.archive_compaction, prepared.marker())?; + admission.clone() + } + None => write .group .archive_compaction .admit(prepared.marker())? .ok_or(LogError::Overloaded)?, - ); + }; + archive_admissions.push(admission); } let next = old.changed( write, @@ -1592,6 +1609,15 @@ impl Worker { .commit() .map_err(|e| LogError::Corrupt(e.to_string()))?; } + for write in &writes { + if let Some(activation) = &write.require_archive { + activation + .published(&write.group.archive_compaction) + .map_err(|error| { + LogError::Corrupt(format!("archive policy publication: {error}")) + })?; + } + } self.cache_bytes = next_bytes; self.cache_usage.store(self.cache_bytes, Ordering::Release); self.performance diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index d717c9efb..aec21d36d 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -238,8 +238,10 @@ async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Re let marker = group.group_manifest()?.unwrap(); assert!(group.archive_compaction.permits(&marker)?); let admitted = group.archive_compaction.admit(&marker)?.unwrap(); + let activation = group.archive_compaction.begin_activation()?; assert!(matches!( - log.require_cell_group_archive(&initial.route.history).await, + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, Err(LogError::Overloaded) )); assert!(!group.read()?.archive_required); @@ -247,8 +249,15 @@ async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Re group.archive_compaction.admit(&marker)?.is_none(), "a pending archive activation must stop later checkpoint admissions" ); + let physical = admitted.clone(); drop(admitted); - log.require_cell_group_archive(&initial.route.history) + assert!(matches!( + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, + Err(LogError::Overloaded) + )); + drop(physical); + log.require_cell_group_archive(&initial.route.history, activation) .await?; assert!(group.read()?.archive_required); assert!(!group.clone().archive_compaction.permits(&marker)?); @@ -282,3 +291,199 @@ async fn archive_requirement_is_local_irreversible_and_recovered() -> anyhow::Re domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_activation_drains_an_admitted_checkpoint_at_storage() -> anyhow::Result<()> { + use crate::raft::cell_groups::{ + CellGroupCheckpointLimits, CellGroupCheckpointPool, ManifestLimits, ManifestPageStore, + ManifestTree, + }; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let initial = bootstrap(5); + let group = log.create_cell_group(initial.clone()).await?; + let genesis = group.group_manifest()?.unwrap(); + let earlier = group.archive_compaction.admit(&genesis)?.unwrap(); + let source = group.clone(); + let directory = root.path().join("pages"); + let (pages, next) = domain + .blocking(move || -> anyhow::Result<_> { + let mut pages = ManifestPageStore::open( + &directory, + Arc::new(cellar_ltx::DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(genesis.route().clone(), ManifestLimits::default())?; + let prepared = tree.prepare_complete( + &source, + &mut pages, + genesis.root(), + genesis.position(), + genesis.membership_revision(), + |_| unreachable!("empty group"), + )?; + let next = tree.prepare_unchanged( + &pages, + &prepared, + LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }, + )?; + Ok((pages, next)) + })? + .await??; + let mut write = GroupWrite::new(&group); + write.entries.push(Entry { + index: 2, + term: 1, + ..Default::default() + }); + let mut hard = group.initial_state()?.hard_state; + hard.commit = 2; + write.hard_state = Some(hard); + log.persist_cells(OperationId::new(20), vec![write]).await?; + let mut write = GroupWrite::new(&group); + write.group_manifest = Some(next.clone()); + log.persist_cells(OperationId::new(21), vec![write]).await?; + let pool = CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + })?; + let capture = pool.clone(); + let source = group.clone(); + let cut = next.clone(); + let checkpoint = domain + .blocking(move || capture.capture(&source, &cut)?.prepare_publication())? + .await??; + let admitted = group + .archive_compaction + .admit(checkpoint.marker())? + .unwrap(); + let activation = group.archive_compaction.begin_activation()?; + assert!(!group.archive_compaction.permits(checkpoint.marker())?); + assert!(group + .archive_compaction + .admit(checkpoint.marker())? + .is_none()); + + let mut wrong_cut = GroupWrite::new(&group); + wrong_cut.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + wrong_cut.checkpoint_admission = Some(earlier.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(22), vec![wrong_cut]) + .await, + Err(LogError::WrongIdentity) + )); + drop(earlier); + let foreign = log.create_cell_group(bootstrap(7)).await?; + let mut wrong_group = GroupWrite::new(&foreign); + wrong_group.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + wrong_group.checkpoint_admission = Some(admitted.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(23), vec![wrong_group]) + .await, + Err(LogError::WrongIdentity) + )); + let mut missing_cut = GroupWrite::new(&group); + missing_cut.checkpoint_admission = Some(admitted.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(24), vec![missing_cut]) + .await, + Err(LogError::WrongIdentity) + )); + let mut unadmitted = GroupWrite::new(&group); + unadmitted.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + assert!(matches!( + log.persist_cells(OperationId::new(25), vec![unadmitted]) + .await, + Err(LogError::Overloaded) + )); + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(checkpoint.clone())); + write.checkpoint_admission = Some(admitted); + // This unpolled physical request now owns the final checkpoint lease. + // Activation must wait, then storage must accept its inherited exact cut. + let saving = log.persist_cells(OperationId::new(26), vec![write]); + assert!(matches!( + log.require_cell_group_archive(&initial.route.history, activation.clone()) + .await, + Err(LogError::Overloaded) + )); + saving.await?; + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(next.marker())); + assert_eq!(group.first_index()?, 3); + log.require_cell_group_archive(&initial.route.history, activation) + .await?; + assert!(group.read()?.archive_required); + assert!(!group.archive_compaction.permits(next.marker())?); + assert_eq!(pool.usage().sources, 1); + drop(checkpoint); + assert_eq!(pool.usage().sources, 0); + log.close().await?; + drop(pages); + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn archive_activation_is_retained_by_entered_storage() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let initial = bootstrap(5); + let group = log.create_cell_group(initial.clone()).await?; + let marker = group.group_manifest()?.unwrap(); + let abandoned = group.archive_compaction.begin_activation()?; + assert!(!group.archive_compaction.permits(&marker)?); + drop(abandoned); + assert!(group.archive_compaction.permits(&marker)?); + assert!(!group.read()?.archive_required); + + let activation = group.archive_compaction.begin_activation()?; + let release = gate.arm(fault::HOLD_SYNC); + let mut writing = + Box::pin(log.require_cell_group_archive(&initial.route.history, activation.clone())); + let pending = domain + .run(crate::asyncrt::timeout( + Duration::from_millis(50), + writing.as_mut(), + )) + .await?; + assert!(pending.is_err(), "actual metadata sync remains held"); + domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await??; + drop(writing); + drop(activation); + assert!(!group.archive_compaction.permits(&marker)?); + assert!(!group.read()?.archive_required); + drop(release); + log.reserve_node_revision().await?; + assert!(group.read()?.archive_required); + assert!(!group.archive_compaction.permits(&marker)?); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index d092f9c78..e7f543700 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -1016,12 +1016,22 @@ match the exact local root and term; newer coverage cannot be replaced with olde or conflicting evidence. An intent reconciled behind the retained checkpoint keeps compaction closed until the next verified cut reaches local history. -Activation retries while an earlier checkpoint admission is still owned. Both -the Raft phase and the entered storage batch retain an admission guard through -completion. The requirement is synced only after those guards drain, so it cannot -retract an already entered checkpoint while retaining that checkpoint's sole -source slot. Capacity refusal is retried specifically at this activation boundary. -No mutex guard crosses an await or physical I/O. +An accepted registration owns a pending activation intent across retries. It +closes later checkpoint admission while earlier work drains. The Raft phase and +its storage batch share an exact engine/group/cut-bound lease; storage may finish +that earlier work while the intent is pending. Raw persistence cannot manufacture +or substitute the lease. The requirement is synced only after all older leases +drain, preserving the checkpoint's sole source slot without letting new captures +overtake activation. Capacity refusal is retried at this boundary. + +An unentered intent releases its barrier when registration is rejected or closed. +An entered metadata write retains the intent independently of its waiter and +publishes the irreversible requirement after physical sync and cache/root +publication. Unknown I/O closes the engine rather than completing the intent. +An existing durable bit follows the same drain and frontier-reset boundary. +Frontier reset happens once before I/O; a delayed duplicate completion cannot +discard subsequently verified coverage. No mutex guard crosses an await or +physical I/O. These owners are local capabilities and add no serialized field. The automatic checkpoint scheduler checks the frontier before capture, letting quorum apply and another group's checkpoint lane progress during archive delay. From 4bd5eb543708886efe6257c67e4428c4a75735ca Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 00:43:47 -0700 Subject: [PATCH 215/568] docs: record archive activation qualification Keep retained-log limits and fresh-quorum recovery explicit as the next blocking gates. --- .../multi-raft/cell-groups/baseline.json | 10 +- docs/design/multi-raft/cell-groups/handoff.md | 127 ++++++++---------- .../design/multi-raft/cell-groups/progress.md | 60 +++++++++ 3 files changed, 119 insertions(+), 78 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 749a7a718..c748607d7 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "8dfe9ff3a316d0579b656af4818dc57d839a6765", - "runtime_revision": "8dfe9ff3a316d0579b656af4818dc57d839a6765", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission.", + "source_revision": "ab0abf26dea4df436b5ced926f28a9ede4f360fd", + "runtime_revision": "ab0abf26dea4df436b5ced926f28a9ede4f360fd", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "476ded3a8a01b2512cd387f839434a54729e1d2aa5afaa6ca63281fbb157f65d", - "line": 447, + "sha256": "b9c7fc76a87a6ec0ae5ee28c79f24e59c6e582daecb58727d1f1a64c7b75a2e7", + "line": 451, "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index e83d12f43..07620b33b 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -16,85 +16,66 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). The latest runtime/qualification source is -`8dfe9ff3a316d0579b656af4818dc57d839a6765`. CG-06 remains IN_PROGRESS. Local -checkpoint compaction for an archive-configured group now requires independently -verified complete coverage, with a durable local requirement and a fresh -frontier after reopen. Owner and storage admission guard the exact cut. -Scheduler prechecks skip blocked capture while quorum apply and sibling -checkpoints progress. Activation drains earlier owner/storage checkpoint -admissions before syncing policy, preserving the original one-source quota -without a source-slot cycle. Public archive status grants no compaction or -response authority. The frontier retains bounded metadata; the journal owns the -current remote hold. - -The required marker-1 `archive_required` field belongs to local group -application metadata. Missing-field records refuse without migration or a legacy -reader. Ordinary writes and peer snapshot installation cannot clear the local -policy or manufacture a frontier. Peer-install integration of deployment archive -policy still needs its separate qualification. - -Local and remote command/enrollment/receipt S3 outputs retain their exact -receiver controller/runtime archive owner, complete independently recovered -cuts, original request/namespace/stored minima and deadlines, fresh -catalog/leader checks and authenticated achieved durability. A native output -owns its physical proof through TLS flush or ordered stream destruction. Quorum -output releases SQL independently. No retry replays application callbacks; -missing receipts remain ambiguous. Enrollment retains original creation -provenance. These contracts remain required at the public cutover. - -Seventeen exact RustFS cases pass in both image forms plus seven signed probes. -The new scenario holds real conditional publication while quorum appends and a -sibling checkpoints, refuses direct owner/raw-storage compaction, then accepts -the same prepared checkpoint after complete coverage. Another archive outage -preserves later entries; independent recovery verifies every cell. The storage -case proves guarded activation, missing-field refusal, irreversible policy and -closed compaction after retained-disk reopen. - -Twenty-two regression/check commands pass with 260 Rust cases across seven -copied executables: four native shared-group suites, -storage/snapshot/cache/receipt/budget/journal/physical-join units, eighteen wire -codec cases and fixed performance cardinality. Locked all-target Clippy, -format/layout, dependency/docs/preflight/plan checks, 37 Python tests and -system-Python hashing pass. - -Independent source/hash/selection and credential-exclusion audit passes. -Records, frozen plan inputs, failed compile/lint/native outputs, copied -reproductions and executables are under -`.cellar/cell-groups-evidence/CG-06-archive-compaction/`. - -The earlier broad slow-reader timeout remains retained; unchanged isolated and -ordered-controller reproductions pass, and its timing cause is unresolved. - -See [progress](progress.md) and the [local compaction contract](protocols.md#archive-aware-local-checkpoint-compaction). - -Next implement explicit unarchived-byte/entry admission across node/group -storage, including inherited peer/control work, pre-callback reservations and -reserved physical/checkpoint progress capacity. Qualify archive activation -fairness under sustained checkpoint traffic before enabling managed capture and -public admission. Qualify a full-backend S3 outage and resolve fresh catalog -admission without weakening ownership or epoch fencing. Archive-wrapper faults -and the compaction prerequisite do not establish that bound. Add safe physical -shared-engine Raft file purge constrained by every local checkpoint, archive, -reader, transfer and sibling group. Fenced fresh-incarnation serving must -inventory every possible old voter, including joins after the selected archive -cut. Fencing-authority API plus Docker qualification remains accepted. -Peer-install policy, controller process-kill restart and -multi-head/physical-read shutdown cuts remain. Incremental tails and smaller -sealed attestations require actual remote holds and load evidence. +`ab0abf26dea4df436b5ced926f28a9ede4f360fd`. CG-06 remains IN_PROGRESS. Archive +registration owns a pending activation that closes later checkpoint admission. +Earlier owner/storage work shares an exact cut-bound lease and drains before +durable policy publication. Entered metadata persistence owns activation after +waiter loss. An unentered intent releases its temporary barrier on +rejection/shutdown; only sync and cache/root publication make the archive +requirement irreversible. A once-only pre-I/O frontier reset avoids discarding +newer coverage on duplicate completion. Wrong-group, wrong-cut and +non-checkpoint lease use refuse. Marker-1 records and original quotas are +unchanged; the cut lease fits the existing 4 KiB charge. + +Archive-configured local checkpoint compaction still requires independently +verified complete coverage and a fresh frontier after retained-disk reopen. +Missing-field records refuse without migration or a legacy reader. Ordinary +writes and peer snapshot installation cannot clear policy or manufacture +coverage. Scheduler prechecks allow apply and sibling checkpoints to progress. +Public archive status grants no compaction or response authority. The frontier +retains bounded metadata; the journal owns the remote hold. Deployment policy on +peer install remains separately unqualified. + +Local and remote command/enrollment/receipt S3 outputs retain the exact receiver +controller/runtime archive owner, complete independently recovered cuts, +original request/namespace/stored minima and deadlines, fresh catalog/leader +checks and authenticated achieved durability. Native outputs own physical proof +through TLS flush or ordered stream destruction. Quorum outputs release SQL +independently. No retry replays callbacks; missing receipts remain ambiguous. +Enrollment preserves original creation provenance. These contracts remain +required at public cutover. + +All three exact storage ownership cases pass, including the unchanged overtaking regression, real checkpoint drain and a held-sync cancelled waiter. Seventeen exact RustFS cases pass in both image forms with seven signed probes. Twenty-two regression/check commands pass with 262 Rust cases across seven copied executables, including nineteen storage cases. Locked all-target Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing pass. Independent source/hash/selection and credential-exclusion audit passes. Source-bound records, frozen plans, orchestration, original red regression and copied executables are under `.cellar/cell-groups-evidence/CG-06-retention-admission/`. Earlier evidence stays retained. The `8dfe9ff3` slow-reader timeout remains unexplained despite unchanged passing reruns. Sustained activation/load fairness and zero-voter-disk-loss recovery are not established by this matrix. See [progress](progress.md) and the [local compaction contract](protocols.md#archive-aware-local-checkpoint-compaction). + +Next implement explicit retained-log byte/entry admission before new callbacks, +preserving known receipt reconciliation. Account for follower/control work, +suffix replacement, restart and uncertain physical outcomes; transfer +reservations into durable debt without release gaps or double charges. Preserve +reserved completion/apply/checkpoint capacity so a full budget cannot strand +collected Ready. Qualify a full-backend S3 outage, resolving fresh catalog +authority without weakening ownership or epoch fencing. Wrapper faults and local +compaction checks alone do not establish a disk bound. Add physical +shared-engine file purge constrained by every checkpoint, archive, reader, +transfer and sibling group. Fenced fresh-incarnation serving must inventory +every possible old voter, including joins after the selected archive cut. +Fencing-authority API plus Docker qualification remains accepted. Peer-install +policy, process-kill and multi-head/physical-read shutdown cuts still need +qualification. CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover, then -topology, placement, hibernation, managed recovery, measured load and final -legacy removal. Four original CG-06 cases/full gate remain pending. Voters are +topology, placement, hibernation, managed recovery, measured load and legacy +removal. Four original CG-06 cases/full gate remain pending. Voters are host-native tasks and RustFS one VM; independent domains, container node limits, measured load, fresh-quorum disaster serving and the hard cutover remain unqualified. The full end-to-end goal remains active. -Preserve the remote response source `25938fd8` and validation-only inventory -correction `debdc30c`, receipt source `cde0eea2`, enrollment `c2801b65`, command -`3d564f3d`, prior RustFS `d739bf79`, controller `7914177d`, catalog cut -`df0ba1f6`, protected publication `87348477`, isolated verifier `14955f2d`, -committed tail `7a29c2a9` and their source-bound evidence/failures. Exact -intents precede remote mutation; uncertain outcomes cannot allocate new +Preserve archive-compaction runtime `8dfe9ff3`, its source-bound failures and +the unresolved slow-reader timing limitation. Preserve remote response source +`25938fd8` with validation-only inventory correction `debdc30c`, receipt +`cde0eea2`, enrollment `c2801b65`, command `3d564f3d`, prior RustFS `d739bf79`, +controller `7914177d`, catalog cut `df0ba1f6`, protected publication `87348477`, +isolated verifier `14955f2d`, committed tail `7a29c2a9` and their evidence. +Exact intents precede remote mutation; uncertain outcomes cannot allocate new identities or reset counters. Namespace binding, stable cuts, protected publication and independent complete recovery preserve their contracts. The archive controller still coalesces complete cuts. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 180843ed2..b5a536148 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,66 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 owned archive activation and checkpoint drain + +Source: `ab0abf26dea4df436b5ced926f28a9ede4f360fd`. CG-06 remains IN_PROGRESS. A +deterministic regression at `79323600` reproduced a checkpoint overtaking +pending archive activation. Registration now owns an activation intent that +closes later checkpoint admission across retries. Rejected or unentered work +releases its intent. An already admitted checkpoint shares one exact +group/cut-bound lease between its owner and physical storage, so that work can +drain while new admission is closed. Storage rejects foreign-group, wrong-cut +and non-checkpoint lease use. The existing 4 KiB metadata charge bounds the +lease; quotas and serialized marker-1 records are unchanged. + +Once metadata persistence enters, physical work retains the activation even +after the waiter disappears. The irreversible local requirement becomes +effective only after sync and cache/root publication. Failed or uncertain writes +cannot publish the intent. Frontier reset happens once before I/O, so a delayed +duplicate completion cannot clear newer verified coverage. No mutex spans I/O or +an await. Cancellation before entry releases the temporary barrier; it never +clears an existing durable policy. + +Three exact storage cases pass. The original recovery case now checks refusal of +later admission and ownership across cloned physical leases. The new checkpoint +case uses actual persisted entries and a prepared checkpoint, tests invalid +lease bindings, then drains an earlier checkpoint into the real engine while +activation waits. The new held-sync case drops the waiter and client activation, +proves physical work still owns the closed barrier, releases the actual +filesystem gate and joins the storage queue before checking durable policy. +These establish the owning boundaries; sustained load fairness is still pending. + +All seventeen exact RustFS cases pass in both image forms with seven signed +protocol probes. Twenty-two regression/check commands pass with 262 distinct +Rust cases across seven copied executables, including all nineteen storage cases +and the four native shared-group suites. Locked all-target Cellar/logic/LTX +Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests +and system-Python hashing pass. Records under +`.cellar/cell-groups-evidence/CG-06-retention-admission/` bind 2571 source +hashes, selections, copied binaries/outputs, orchestration and frozen plan +inputs. Independent source/artifact/selection and credential-exclusion audit +passes. The original failing regression is retained with its source and +executable. Original CG-06 cases, oracles and quotas remain unchanged; four +original cases/full gate remain pending. The earlier `8dfe9ff3` native +slow-reader timeout remains retained and unexplained despite unchanged +successful reruns; this ownership change is not claimed to fix it. The RustFS +controller matrix does not delete voter disks. + +Reflection: implement explicit retained-log byte/entry admission across node and +group before new callbacks, preserving known receipt reconciliation. Include +inherited follower entries, control entries, suffix replacement, restart and +uncertain physical outcomes, with reserved progress capacity so collected Ready, +apply and checkpoint work can drain. Then qualify a full S3 backend outage, +including fresh catalog authority without weakening ownership checks. Physical +shared-engine file purge must honor every checkpoint, archive, reader, transfer +and sibling constraint. Fenced fresh-incarnation serving needs complete +possible-old-voter lineage and the accepted fencing-authority API plus Docker +qualification. CG-08 effect/cache/result lifetimes precede CG-07 public +SDK/fleet cutover. Topology, placement, hibernation, managed recovery, measured +load and legacy removal follow. Voters remain host-native tasks and RustFS one +local VM; independent failure domains and the full hard cutover remain +unqualified. + ## CG-06 archive-aware local checkpoint compaction Source: `8dfe9ff3a316d0579b656af4818dc57d839a6765`. CG-06 remains IN_PROGRESS. From c2759d74fe652fccff52e6f3f5878a89263f4161 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 01:11:28 -0700 Subject: [PATCH 216/568] feat: admit retained group logs before callbacks Transfer owned reservations into selected entry debt. Reserve election and automatic joint-leave capacity before Raft mutation, and preserve independent read, apply and checkpoint progress lanes. --- crates/cellar/durability/raft/cache.rs | 17 + .../durability/raft/cell_groups/README.md | 31 + .../raft/cell_groups/enrollment_authority.rs | 15 + .../raft/cell_groups/execution_authority.rs | 15 + .../raft/cell_groups/log_admission.rs | 221 +++++++ .../raft/cell_groups/log_retention.rs | 563 ++++++++++++++++++ .../cellar/durability/raft/cell_groups/mod.rs | 5 + .../durability/raft/cell_groups/owner.rs | 95 ++- .../raft/cell_groups/runtime/worker.rs | 4 + .../cellar/durability/raft/execution/cell.rs | 19 +- .../durability/raft/execution/enrollment.rs | 17 +- .../durability/raft/execution/submission.rs | 29 + crates/cellar/durability/raft/options.rs | 3 + crates/cellar/durability/raft/storage.rs | 61 ++ .../raft/storage_cell_genesis_tests.rs | 174 ++++++ .../tests/native_cell_groups_execution.rs | 10 + .../support/cell_group_engine_fixture.rs | 39 +- .../tests/support/cell_group_log_retention.rs | 239 ++++++++ .../tests/support/cell_group_runtime.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../multi-raft/cell-groups/protocols.md | 34 ++ 21 files changed, 1585 insertions(+), 14 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/log_admission.rs create mode 100644 crates/cellar/durability/raft/cell_groups/log_retention.rs create mode 100644 crates/cellar/tests/support/cell_group_log_retention.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 1f12c1a7e..f922c60d7 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -105,6 +105,8 @@ pub enum SnapshotChange { #[derive(Clone)] pub struct GroupWrite { + pub(crate) log_reservations: + BTreeMap>, pub(crate) require_archive: Option>, pub(crate) checkpoint_admission: @@ -124,6 +126,7 @@ pub struct GroupWrite { impl GroupWrite { pub fn new(group: &GroupLog) -> Self { Self { + log_reservations: BTreeMap::new(), require_archive: None, checkpoint_admission: None, consensus_pin: None, @@ -196,6 +199,7 @@ impl GroupWrite { /// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] pub struct GroupLog { + pub(crate) log_retention: Option>, pub(crate) archive_compaction: Arc, pub(crate) consensus_claimed: Arc, pub(crate) scratch_directory: std::path::PathBuf, @@ -987,3 +991,16 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { }, } } + +impl GroupLog { + /// Advisory logical entry debt and owned reservations; not physical file + /// usage, archive coverage or permission to admit a mutation. + pub fn log_retention_usage( + &self, + ) -> Result { + self.log_retention + .as_ref() + .ok_or(LogError::WrongIdentity)? + .usage() + } +} diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index dbd8e9e90..aa97aeb88 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -432,3 +432,34 @@ Shared log-engine I/O/corruption/closure and lost shared SQL worker replies stil close the node. Unknown events and controller ownership violations fail closed. Request-driven group provisioning, configuration lifecycle and public SDK integration remain unfinished. See the delivery package for qualification status. + + +### Retained entry admission + +`ReplicaLogOptions.cell_group_retention` bounds node/group logical retained log +entries plus owned reservations. Its progress sublimits reserve room for empty +Raft election and automatic joint-leave entries; ordinary payloads cannot consume +that space. The charge includes 512 bytes of fixed bookkeeping per entry. These +are logical entry and admission ceilings, not physical engine file byte limits. +`log_retention_usage` reports debt and reservations without granting admission. + +New SQL commands reserve their bounded encoded envelope after duplicate lookup +and before application callbacks. Enrollment reserves before initialization. +Prepared commands retain that lease through submission and shrink it to actual +encoded bytes. Raft owners reserve inbound appends and local proposals before +mutation, including prospective election and automatic configuration entries. +Existing completion and read-index work does not consume ordinary entry admission. +Sealed pre-append rejection or authoritative receipt reconciliation retires an +unused preparation independently of client clones. + +Entered storage retains exact engine/group/entry-bound leases after waiter loss. +After sync and cache/root publication, one ledger operation transfers them into +actual selected entry debt without a release gap or double charge. Suffix +replacement and selected checkpoints update debt from actual retained entries; +archive observations cannot refund it. Reopen reconstructs debt from the selected +log, retaining entries even under a newly lowered admission limit. Mutations +refuse until a reservation fits. No serialized field or per-cell worker is added. + +The admission implementation and new boundary cases are under qualification. +Full-backend S3 outage authority, physical file amplification/reclamation, +sustained-load fairness and the original CG-06 task gates remain pending. diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs index d9e0986e2..c77883723 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -22,6 +22,21 @@ pub struct CellEnrollmentState { } impl CellEnrollmentState { + pub(crate) fn reserve_log( + &self, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, RuntimeError> { + self.group + .log_retention + .as_ref() + .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? + .reserve_execution(envelope) + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + }) + } + pub(crate) fn engine_owner(&self) -> &Arc<()> { &self.group.owner } diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index fffde1d66..d982afee1 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -25,6 +25,21 @@ pub struct CellExecutionAuthority { } impl CellExecutionAuthority { + pub(crate) fn reserve_log( + &self, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, RuntimeError> { + self.group + .log_retention + .as_ref() + .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? + .reserve_execution(envelope) + .map_err(|error| match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + }) + } + pub(crate) fn engine_owner(&self) -> &Arc<()> { &self.group.owner } diff --git a/crates/cellar/durability/raft/cell_groups/log_admission.rs b/crates/cellar/durability/raft/cell_groups/log_admission.rs new file mode 100644 index 000000000..6ebbc20da --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/log_admission.rs @@ -0,0 +1,221 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Reserve before RawNode can append; retain exact entries through persistence. +use super::log_retention::LogReservation; +use crate::raft::{ + cache::{ConsensusStorage, GroupLog}, + completion::{entry_digest, RuntimeError}, +}; +use ::raft::{prelude::*, RawNode, StateRole}; +use cellar_logic::cell_group::GroupHistory; +use protobuf::Message as _; +use std::{collections::BTreeMap, sync::Arc}; + +#[derive(Default)] +pub(super) struct LogAdmission { + entries: BTreeMap)>, + automatic_leave: Option>, +} + +#[derive(Default)] +pub(super) struct Prospective { + entries: BTreeMap)>, + control: Option>, + automatic_leave: Option>, +} + +fn retention( + group: &GroupLog, +) -> Result<&Arc, RuntimeError> { + group + .log_retention + .as_ref() + .ok_or_else(|| RuntimeError::Protocol("missing group log admission".into())) +} + +impl LogAdmission { + pub(super) fn prepare( + &self, + group: &GroupLog, + entries: &[Entry], + supplied: Option<&Arc>, + ) -> Result { + let mut next = Prospective::default(); + if supplied.is_some() && entries.len() != 1 { + return Err(RuntimeError::Invalid( + "prepared log reservation shape".into(), + )); + } + let cache = group.read()?; + for entry in entries { + let digest = entry_digest(entry)?; + if self + .entries + .get(&entry.index) + .is_some_and(|(old, _)| old == &digest) + { + continue; + } + if cache.entries.get(&entry.index) == Some(entry) { + continue; + } + let lease = if let Some(lease) = supplied { + lease.bind(entry)?; + lease.validate(retention(group)?, entry)?; + lease.clone() + } else { + retention(group)? + .reserve_entry(entry) + .map_err(admission_error)? + }; + next.entries.insert(entry.index, (digest, lease)); + } + Ok(next) + } + + pub(super) fn election( + &self, + group: &GroupLog, + node: &RawNode, + mut next: Prospective, + ) -> Result { + next.control = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + if self.automatic_leave.is_none() && needs_automatic_leave(group, node)? { + next.automatic_leave = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + } + Ok(next) + } + + pub(super) fn configuration( + &self, + group: &GroupLog, + mut next: Prospective, + auto_leave: bool, + ) -> Result { + if auto_leave && self.automatic_leave.is_none() { + next.automatic_leave = Some( + retention(group)? + .reserve(1024, true) + .map_err(admission_error)?, + ); + } + Ok(next) + } + + /// A promised automatic entry must never seek admission after collected + /// apply work has already selected its new durable configuration. + pub(super) fn applying(&mut self, automatic: bool) -> Result { + let mut next = Prospective::default(); + if automatic { + next.control = Some(self.automatic_leave.take().ok_or_else(|| { + RuntimeError::Protocol("unreserved automatic joint leave".into()) + })?); + } + Ok(next) + } + + pub(super) fn finish( + &mut self, + group: &GroupLog, + node: &RawNode, + mut next: Prospective, + ) -> Result<(), RuntimeError> { + let mut entries = BTreeMap::new(); + let cache = group.read()?; + for entry in node.raft.raft_log.unstable_entries() { + let digest = entry_digest(entry)?; + if cache.entries.get(&entry.index) == Some(entry) { + continue; + } + let old = self + .entries + .get(&entry.index) + .filter(|(old, _)| old == &digest); + let fresh = next + .entries + .remove(&entry.index) + .filter(|(new, _)| new == &digest); + let lease = if let Some((_, lease)) = old { + lease.clone() + } else if let Some((_, lease)) = fresh { + lease + } else if entry.data.is_empty() && entry.context.is_empty() { + let lease = next.control.take().ok_or_else(|| { + RuntimeError::Protocol("unreserved Raft control entry".into()) + })?; + lease.bind(entry)?; + lease + } else { + return Err(RuntimeError::Protocol("unreserved Raft entry".into())); + }; + entries.insert(entry.index, (digest, lease)); + } + drop(cache); + self.entries = entries; + if node.raft.state == StateRole::Leader && needs_automatic_leave(group, node)? { + if let Some(lease) = next.automatic_leave { + self.automatic_leave = Some(lease); + } + } else { + self.automatic_leave = None; + } + Ok(()) + } + + pub(super) fn persistence( + &self, + entries: &[Entry], + ) -> Result>, RuntimeError> { + let mut leases = BTreeMap::new(); + for entry in entries { + if let Some((digest, lease)) = self.entries.get(&entry.index) { + if *digest != entry_digest(entry)? { + return Err(RuntimeError::Protocol("changed reserved entry".into())); + } + leases.insert(entry.index, lease.clone()); + } + } + Ok(leases) + } +} + +fn needs_automatic_leave( + group: &GroupLog, + node: &RawNode, +) -> Result { + if node.raft.prs().conf().auto_leave { + return Ok(true); + } + let cache = group.read()?; + for entry in cache + .entries + .values() + .chain(node.raft.raft_log.unstable_entries()) + { + if entry.index > node.raft.raft_log.applied + && entry.get_entry_type() == EntryType::EntryConfChangeV2 + { + let change = ConfChangeV2::parse_from_bytes(&entry.data) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?; + if change.enter_joint() == Some(true) { + return Ok(true); + } + } + } + Ok(false) +} + +fn admission_error(error: crate::raft::storage::LogError) -> RuntimeError { + match error { + crate::raft::storage::LogError::Overloaded => RuntimeError::Overloaded, + other => RuntimeError::Storage(other), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/log_retention.rs b/crates/cellar/durability/raft/cell_groups/log_retention.rs new file mode 100644 index 000000000..e0f1f0789 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/log_retention.rs @@ -0,0 +1,563 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Engine-local retained entry debt and owned reservations. Remote archive +//! observations never release this accounting; only selected storage state can. +use crate::raft::storage::LogError; +use ::raft::prelude::{Entry, EntryType}; +use protobuf::Message as _; +use sha2::{Digest, Sha256}; +use std::collections::{BTreeMap, BTreeSet}; +use std::sync::{ + atomic::{AtomicBool, Ordering}, + Arc, Mutex, MutexGuard, +}; + +// Includes the cached entry's fixed metadata and an outstanding lease/map node. +const ENTRY_OVERHEAD: usize = 512; + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupLogCapacity { + pub bytes: usize, + pub entries: usize, +} + +impl CellGroupLogCapacity { + fn add(self, other: Self) -> Result { + Ok(Self { + bytes: self + .bytes + .checked_add(other.bytes) + .ok_or(LogError::Overloaded)?, + entries: self + .entries + .checked_add(other.entries) + .ok_or(LogError::Overloaded)?, + }) + } + + fn sub(self, other: Self) -> Result { + Ok(Self { + bytes: self + .bytes + .checked_sub(other.bytes) + .ok_or_else(accounting_error)?, + entries: self + .entries + .checked_sub(other.entries) + .ok_or_else(accounting_error)?, + }) + } + + fn fits(self, limit: Self) -> bool { + self.bytes <= limit.bytes && self.entries <= limit.entries + } +} + +/// Hard node/group entry admission, including separately reserved capacity for +/// Raft's empty election and automatic joint-leave entries. This bounds logical +/// retained entries and reservations, not physical engine file amplification. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupLogRetentionLimits { + pub node: CellGroupLogCapacity, + pub group: CellGroupLogCapacity, + pub progress_node: CellGroupLogCapacity, + pub progress_group: CellGroupLogCapacity, +} + +impl Default for CellGroupLogRetentionLimits { + fn default() -> Self { + Self { + node: CellGroupLogCapacity { + bytes: 192 * 1024 * 1024, + entries: 32_768, + }, + group: CellGroupLogCapacity { + bytes: 64 * 1024 * 1024, + entries: 8192, + }, + progress_node: CellGroupLogCapacity { + bytes: 2 * 1024 * 1024, + entries: 1024, + }, + progress_group: CellGroupLogCapacity { + bytes: 128 * 1024, + entries: 64, + }, + } + } +} + +impl CellGroupLogRetentionLimits { + pub(crate) fn validate(self) -> Result<(), LogError> { + if self.node.bytes > 2 * 1024 * 1024 * 1024 + || self.node.entries > 1_048_576 + || !self.group.fits(self.node) + || !self.progress_group.fits(self.progress_node) + || [ + self.node, + self.group, + self.progress_node, + self.progress_group, + ] + .iter() + .any(|x| x.bytes == 0 || x.entries == 0) + || self.progress_node.bytes >= self.node.bytes + || self.progress_node.entries >= self.node.entries + || self.progress_group.bytes >= self.group.bytes + || self.progress_group.entries >= self.group.entries + { + return Err(LogError::Invalid("cell-group log retention limits".into())); + } + Ok(()) + } +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CellGroupLogRetentionUsage { + pub retained_bytes: usize, + pub retained_entries: usize, + pub reserved_bytes: usize, + pub reserved_entries: usize, +} + +#[derive(Clone, Copy, Debug, Default)] +struct Charge { + data: CellGroupLogCapacity, + progress: CellGroupLogCapacity, +} + +impl Charge { + fn entry(entry: &Entry) -> Result { + let capacity = CellGroupLogCapacity { + bytes: (entry.compute_size() as usize) + .checked_add(ENTRY_OVERHEAD) + .ok_or(LogError::Overloaded)?, + entries: 1, + }; + Ok(Self::new(capacity, progress_entry(entry))) + } + + fn new(capacity: CellGroupLogCapacity, progress: bool) -> Self { + if progress { + Self { + progress: capacity, + ..Self::default() + } + } else { + Self { + data: capacity, + ..Self::default() + } + } + } + + fn add(self, other: Self) -> Result { + Ok(Self { + data: self.data.add(other.data)?, + progress: self.progress.add(other.progress)?, + }) + } + + fn sub(self, other: Self) -> Result { + Ok(Self { + data: self.data.sub(other.data)?, + progress: self.progress.sub(other.progress)?, + }) + } + + fn fits( + self, + limit: CellGroupLogCapacity, + progress: CellGroupLogCapacity, + ) -> Result { + Ok(self.data.fits(limit.sub(progress)?) + && self.progress.fits(progress) + && self.data.add(self.progress)?.fits(limit)) + } + + fn total(self) -> Result { + self.data.add(self.progress) + } +} + +fn accounting_error() -> LogError { + LogError::Corrupt("cell-group retained log accounting".into()) +} + +fn progress_entry(entry: &Entry) -> bool { + entry.data.is_empty() + && entry.context.is_empty() + && matches!( + entry.get_entry_type(), + EntryType::EntryNormal | EntryType::EntryConfChangeV2 + ) +} + +fn binding(entry: &Entry) -> Result<[u8; 32], LogError> { + Ok(Sha256::digest( + entry + .write_to_bytes() + .map_err(|e| LogError::Invalid(e.to_string()))?, + ) + .into()) +} + +fn retained<'a>(mut entries: impl Iterator) -> Result { + entries.try_fold(Charge::default(), |sum, entry| { + sum.add(Charge::entry(entry)?) + }) +} + +#[derive(Default)] +struct Group { + durable: Charge, + usage: Charge, +} + +struct Hold { + group: u64, + charge: Charge, + binding: Option<[u8; 32]>, +} + +#[derive(Default)] +struct State { + groups: BTreeMap, + holds: BTreeMap, + next: u64, + durable: Charge, + usage: Charge, +} + +pub(crate) struct NodeLogRetention { + state: Mutex, + limits: CellGroupLogRetentionLimits, + closed: Arc, +} + +pub(crate) struct GroupLogRetention { + node: Arc, + key: u64, +} + +/// Clones share one charge. Storage transfers it into actual durable debt in +/// the same accounting turn; losing the waiter cannot refund entered work. +pub(crate) struct LogReservation { + group: Arc, + id: u64, +} + +impl NodeLogRetention { + pub(crate) fn new(limits: CellGroupLogRetentionLimits, closed: Arc) -> Arc { + Arc::new(Self { + state: Mutex::new(State::default()), + limits, + closed, + }) + } + + fn lock(&self) -> Result, LogError> { + if self.closed.load(Ordering::Acquire) { + return Err(LogError::Closed); + } + self.state.lock().map_err(|_| { + self.closed.store(true, Ordering::Release); + LogError::Closed + }) + } + + /// Reconstruct debt from actual selected entries, including when reopening + /// under a smaller ceiling. New reservations refuse until the debt fits. + pub(crate) fn register<'a>( + self: &Arc, + key: u64, + entries: impl Iterator, + ) -> Result, LogError> { + let durable = retained(entries)?; + let mut state = self.lock()?; + if state.groups.contains_key(&key) { + return Err(LogError::WrongIdentity); + } + let usage = state.usage.add(durable)?; + let all_durable = state.durable.add(durable)?; + state.groups.insert( + key, + Group { + durable, + usage: durable, + }, + ); + state.usage = usage; + state.durable = all_durable; + Ok(Arc::new(GroupLogRetention { + node: self.clone(), + key, + })) + } + + pub(crate) fn usage(&self) -> Result { + let state = self.lock()?; + usage(state.durable, state.usage) + } +} + +fn usage(durable: Charge, all: Charge) -> Result { + let retained = durable.total()?; + let reserved = all.sub(durable)?.total()?; + Ok(CellGroupLogRetentionUsage { + retained_bytes: retained.bytes, + retained_entries: retained.entries, + reserved_bytes: reserved.bytes, + reserved_entries: reserved.entries, + }) +} + +impl GroupLogRetention { + pub(crate) fn usage(&self) -> Result { + let state = self.node.lock()?; + let group = state.groups.get(&self.key).ok_or(LogError::WrongIdentity)?; + usage(group.durable, group.usage) + } + + pub(crate) fn reserve( + self: &Arc, + bytes: usize, + progress: bool, + ) -> Result, LogError> { + if bytes < ENTRY_OVERHEAD { + return Err(LogError::Invalid("entry reservation overhead".into())); + } + let charge = Charge::new(CellGroupLogCapacity { bytes, entries: 1 }, progress); + let mut state = self.node.lock()?; + let group = state.groups.get(&self.key).ok_or(LogError::WrongIdentity)?; + let next_group = group.usage.add(charge)?; + let next_node = state.usage.add(charge)?; + let limits = self.node.limits; + if !next_group.fits(limits.group, limits.progress_group)? + || !next_node.fits(limits.node, limits.progress_node)? + { + return Err(LogError::Overloaded); + } + let id = state.next.checked_add(1).ok_or(LogError::Overloaded)?; + state.next = id; + state.holds.insert( + id, + Hold { + group: self.key, + charge, + binding: None, + }, + ); + state + .groups + .get_mut(&self.key) + .ok_or(LogError::WrongIdentity)? + .usage = next_group; + state.usage = next_node; + Ok(Arc::new(LogReservation { + group: self.clone(), + id, + })) + } + + pub(crate) fn reserve_execution( + self: &Arc, + envelope: crate::application::consensus::EnvelopeLimits, + ) -> Result, LogError> { + let bytes = envelope + .max_header_bytes + .checked_add(envelope.max_payload_bytes) + .and_then(|n| n.checked_add(1024)) + .ok_or(LogError::Overloaded)?; + self.reserve(bytes, false) + } + + pub(crate) fn reserve_entry( + self: &Arc, + entry: &Entry, + ) -> Result, LogError> { + let lease = self.reserve(Charge::entry(entry)?.total()?.bytes, progress_entry(entry))?; + lease.bind(entry)?; + Ok(lease) + } + + /// Call only after the exact cache/root state is durably selected. Every new + /// entry must have an entered lease; accounting can only stay equal or fall. + pub(crate) fn publish<'a>( + self: &Arc, + entries: impl Iterator, + leases: &[Arc], + ) -> Result<(), LogError> { + let next = retained(entries)?; + let mut state = self.node.lock()?; + let old = state + .groups + .get(&self.key) + .ok_or(LogError::WrongIdentity)? + .durable; + let mut consumed = Charge::default(); + let mut ids = BTreeSet::new(); + for lease in leases { + if !Arc::ptr_eq(self, &lease.group) || !ids.insert(lease.id) { + return Err(LogError::WrongIdentity); + } + let hold = state.holds.get(&lease.id).ok_or(LogError::WrongIdentity)?; + if hold.group != self.key || hold.binding.is_none() { + return Err(LogError::WrongIdentity); + } + consumed = consumed.add(hold.charge)?; + } + let admitted = old.add(consumed)?; + if !next.data.fits(admitted.data) || !next.progress.fits(admitted.progress) { + return Err(accounting_error()); + } + let next_group = state + .groups + .get(&self.key) + .ok_or(LogError::WrongIdentity)? + .usage + .sub(admitted)? + .add(next)?; + let next_node = state.usage.sub(admitted)?.add(next)?; + let next_durable = state.durable.sub(old)?.add(next)?; + for id in ids { + state.holds.remove(&id); + } + state.groups.insert( + self.key, + Group { + durable: next, + usage: next_group, + }, + ); + state.usage = next_node; + state.durable = next_durable; + Ok(()) + } +} + +impl LogReservation { + /// Shrink the pre-callback envelope to the actual encoded proposal, retaining + /// enough space for its later Raft term/index and fixed entry bookkeeping. + pub(crate) fn encoded(&self, bytes: usize) -> Result<(), LogError> { + let charge = Charge::new( + CellGroupLogCapacity { + bytes: bytes + .checked_add(64 + ENTRY_OVERHEAD) + .ok_or(LogError::Overloaded)?, + entries: 1, + }, + false, + ); + let mut state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.binding.is_some() + || !charge.data.fits(hold.charge.data) + || !charge.progress.fits(hold.charge.progress) + { + return Err(LogError::WrongIdentity); + } + let released = hold.charge.sub(charge)?; + let next_group = state + .groups + .get(&self.group.key) + .ok_or(LogError::WrongIdentity)? + .usage + .sub(released)?; + state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)? + .usage = next_group; + state.usage = state.usage.sub(released)?; + state + .holds + .get_mut(&self.id) + .ok_or(LogError::WrongIdentity)? + .charge = charge; + Ok(()) + } + + /// Only a sealed pre-append rejection or authoritative reconciliation may + /// release a live lease independently of its remaining client clones. + pub(crate) fn release(&self) -> Result<(), LogError> { + let mut state = self.group.node.lock()?; + if let Some(hold) = state.holds.remove(&self.id) { + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(hold.charge)?; + state.usage = state.usage.sub(hold.charge)?; + } + Ok(()) + } + + pub(crate) fn bind(&self, entry: &Entry) -> Result<(), LogError> { + let digest = binding(entry)?; + let charge = Charge::entry(entry)?; + let mut state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.group != self.group.key + || hold.binding.is_some_and(|old| old != digest) + || !charge.data.fits(hold.charge.data) + || !charge.progress.fits(hold.charge.progress) + { + return Err(LogError::WrongIdentity); + } + let released = hold.charge.sub(charge)?; + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(released)?; + state.usage = state.usage.sub(released)?; + let hold = state + .holds + .get_mut(&self.id) + .ok_or(LogError::WrongIdentity)?; + hold.charge = charge; + hold.binding = Some(digest); + Ok(()) + } + + pub(crate) fn validate( + &self, + group: &Arc, + entry: &Entry, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(&self.group, group) { + return Err(LogError::WrongIdentity); + } + let digest = binding(entry)?; + let state = self.group.node.lock()?; + let hold = state.holds.get(&self.id).ok_or(LogError::WrongIdentity)?; + if hold.group != group.key || hold.binding != Some(digest) { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl Drop for LogReservation { + fn drop(&mut self) { + // Even a closed engine may release an unselected reservation. No new + // admission is possible there; reopen reconstructs actual durable debt. + if let Ok(mut state) = self.group.node.state.lock() { + if let Some(hold) = state.holds.remove(&self.id) { + let result = (|| { + let group = state + .groups + .get_mut(&self.group.key) + .ok_or(LogError::WrongIdentity)?; + group.usage = group.usage.sub(hold.charge)?; + state.usage = state.usage.sub(hold.charge)?; + Ok::<_, LogError>(()) + })(); + if result.is_err() { + self.group.node.closed.store(true, Ordering::Release); + } + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index a9bf05dd2..38ce2de26 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -69,11 +69,16 @@ mod image_pin; mod manifest; pub(crate) mod membership; pub use membership::CellGroupMembershipChange; +mod log_admission; +pub(crate) mod log_retention; mod owner; mod page_store; mod page_wire; mod pins; mod proposal_admission; +pub use log_retention::{ + CellGroupLogCapacity, CellGroupLogRetentionLimits, CellGroupLogRetentionUsage, +}; mod read_authority; pub(crate) mod reclaim; pub use read_authority::{CellReadAuthority, CellReadConsistency}; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 3564b61c9..fc8861fb1 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -172,6 +172,7 @@ enum Phase { } pub struct CellGroupOwner { + log_admission: super::log_admission::LogAdmission, leader_term: Arc, lease: Arc, log: ReplicaLog, @@ -259,6 +260,7 @@ impl CellGroupOwner { Some(seed), )?; Ok(Self { + log_admission: Default::default(), leader_term: Arc::new(AtomicU64::new(0)), lease, log: permit.log().clone(), @@ -285,6 +287,18 @@ impl CellGroupOwner { }) } + fn finish_log_admission( + &mut self, + next: super::log_admission::Prospective, + ) -> Result<(), RuntimeError> { + if let Err(error) = self.log_admission.finish(&self.group, &self.node, next) { + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } + Ok(()) + } + pub(super) fn revoke_execution(&mut self) { self.leader_term.store(0, Ordering::Release); self.proposals.clear(); @@ -495,7 +509,11 @@ impl CellGroupOwner { if self.phase.is_some() || self.checkpoint.is_some() { return Err(RuntimeError::NotReady); } + let admission = self + .log_admission + .election(&self.group, &self.node, Default::default())?; let result = self.node.campaign(); + self.finish_log_admission(admission)?; self.refresh_execution_fence(); result?; Ok(()) @@ -543,7 +561,14 @@ impl CellGroupOwner { if self.phase.is_some() || self.node.has_ready() { return Err(RuntimeError::NotReady); } + let admission = if self.node.raft.state == StateRole::Leader { + Default::default() + } else { + self.log_admission + .election(&self.group, &self.node, Default::default())? + }; self.node.tick(); + self.finish_log_admission(admission)?; self.refresh_execution_fence(); Ok(()) } @@ -700,6 +725,7 @@ impl CellGroupOwner { return Err(RuntimeError::Overloaded); } let result = self.node.step(message); + self.finish_log_admission(Default::default())?; self.refresh_execution_fence(); result?; if let Some(snapshot) = self.node.snap() { @@ -724,7 +750,26 @@ impl CellGroupOwner { return Err(RuntimeError::NotReady); } self.validate_message(&message)?; + let entries = if message.get_msg_type() == MessageType::MsgAppend { + message.entries.as_slice() + } else { + &[] + }; + let mut admission = self.log_admission.prepare(&self.group, entries, None)?; + if self.node.raft.state != StateRole::Leader + && matches!( + message.get_msg_type(), + MessageType::MsgRequestVoteResponse + | MessageType::MsgRequestPreVoteResponse + | MessageType::MsgTimeoutNow + ) + { + admission = self + .log_admission + .election(&self.group, &self.node, admission)?; + } let result = self.node.step(message); + self.finish_log_admission(admission)?; self.refresh_execution_fence(); result?; Ok(()) @@ -788,7 +833,17 @@ impl CellGroupOwner { } drop(cache); let expected = entry.data.clone(); - if let Err(error) = self.node.propose_conf_change(Vec::new(), change.change) { + let admission = + self.log_admission + .prepare(&self.group, std::slice::from_ref(&entry), None)?; + let admission = self.log_admission.configuration( + &self.group, + admission, + change.change.enter_joint() == Some(true), + )?; + let result = self.node.propose_conf_change(Vec::new(), change.change); + self.finish_log_admission(admission)?; + if let Err(error) = result { let error = RuntimeError::from(error); if self.node.raft.raft_log.last_index() != before { self.failed = Some(error.clone()); @@ -833,7 +888,8 @@ impl CellGroupOwner { if self.prepared_only { return Err(RuntimeError::Invalid("prepared submission required".into())); } - self.try_propose(data)?.ok_or(RuntimeError::Overloaded) + self.try_propose(data, None)? + .ok_or(RuntimeError::Overloaded) } /// Seals a definitively refused preparation against every delayed clone. @@ -853,11 +909,15 @@ impl CellGroupOwner { if prepared.term().get() != self.node.raft.term { return Err(RuntimeError::StaleCompletion); } - self.try_propose(prepared.bytes().to_vec()) + self.try_propose(prepared.bytes().to_vec(), Some(prepared.log_reservation()?)) }) } - fn try_propose(&mut self, data: Vec) -> Result, RuntimeError> { + fn try_propose( + &mut self, + data: Vec, + reservation: Option<&Arc>, + ) -> Result, RuntimeError> { if !self.status()?.ready { return Err(RuntimeError::NotReady); } @@ -887,7 +947,24 @@ impl CellGroupOwner { super::proposal_admission::Admission::New(candidate, permit) => (candidate, permit), }; let before = self.node.raft.raft_log.last_index(); - if let Err(error) = self.node.propose(Vec::new(), data) { + let entry = Entry { + index: before + 1, + term: self.node.raft.term, + data: data.clone().into(), + ..Default::default() + }; + let admission = + match self + .log_admission + .prepare(&self.group, std::slice::from_ref(&entry), reservation) + { + Ok(admission) => admission, + Err(RuntimeError::Overloaded) => return Ok(None), + Err(error) => return Err(error), + }; + let result = self.node.propose(Vec::new(), data); + self.finish_log_admission(admission)?; + if let Err(error) = result { let error = RuntimeError::from(error); if self.node.raft.raft_log.last_index() != before { self.failed = Some(error.clone()); @@ -1133,6 +1210,7 @@ impl CellGroupOwner { mut write: GroupWrite, ) -> Result { let ticket = self.ticket(LogPosition::default(), [0; 32])?; + write.log_reservations = self.log_admission.persistence(&write.entries)?; write.consensus_pin = Some(self.lease.clone()); self.waiting = Some(ticket.clone()); Ok(CellGroupEffect::Persist(Box::new(CellGroupPersist { @@ -1257,6 +1335,7 @@ impl CellGroupOwner { } self.enqueue(ready.take_committed_entries())?; let light = self.node.advance_append(*ready); + self.finish_log_admission(Default::default())?; if let Some(index) = installed { self.node.advance_apply_to(index); self.proposals.clear(); @@ -1356,7 +1435,13 @@ impl CellGroupOwner { self.refresh_execution_fence(); } let last = self.node.raft.raft_log.last_index(); + let automatic = self.node.raft.state == StateRole::Leader + && self.node.raft.prs().conf().auto_leave + && self.node.raft.raft_log.applied <= self.node.raft.pending_conf_index + && ticket.position.index.get() >= self.node.raft.pending_conf_index; + let admission = self.log_admission.applying(automatic)?; self.node.advance_apply_to(ticket.position.index.get()); + self.finish_log_admission(admission)?; if self.node.raft.raft_log.last_index() > last { // Raft appended an automatic joint-leave entry. Drive // replication without waiting for a later write/tick; diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 250410271..dfdec7361 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -366,6 +366,10 @@ impl Worker { made = true; } Err(RuntimeError::NotReady) => (), + Err(RuntimeError::Overloaded) => { + group.tick = false; + made = true; + } Err(error) => group.fail(error), } } else if let Some(request) = group.deferred.pop_front() { diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index e83e55864..0985e214d 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -179,6 +179,10 @@ impl CellPrivateExecutor { } command::Admission::New(maintenance) => maintenance, }; + let log_reservation = self + .authority + .reserve_log(self.options.limits.envelope) + .map_err(authority_failure)?; let sequence = self .authority .cell() @@ -255,13 +259,17 @@ impl CellPrivateExecutor { ) .map_err(storage)?; self.authority.check().map_err(authority_failure)?; + log_reservation + .encoded(bytes.len()) + .map_err(|error| authority_failure(error.into()))?; let prepared = CellPreparedCommand::new( bytes, attempt.proposal, self.authority.term(), route.history, self.authority.engine_owner().clone(), - ); + ) + .with_log_reservation(log_reservation); self.pending = Some(PendingCell { prepared: prepared.clone(), request: attempt.request, @@ -314,6 +322,10 @@ impl CellPrivateExecutor { .check_committed_entry(pending.prepared.bytes()) .map_err(authority_failure)?; let receipt = self.read_receipt(&committed, &pending.request)?; + pending + .prepared + .release_log_reservation() + .map_err(authority_failure)?; self.pending = None; self.poisoned = true; Ok(receipt) @@ -367,6 +379,10 @@ impl CellPrivateExecutor { Ok(None) => return Err(ExecutionError::Uncertain), }; authority.check().map_err(authority_failure)?; + pending + .prepared + .release_log_reservation() + .map_err(authority_failure)?; self.pending = None; self.poisoned = true; Ok(resolution) @@ -414,6 +430,7 @@ impl CellPrivateExecutor { pub(crate) fn authority_failure(error: RuntimeError) -> ExecutionError { match error { + RuntimeError::Overloaded => ExecutionError::Refused(CommandFailure::Overloaded), RuntimeError::NotReady | RuntimeError::NotLeader | RuntimeError::Closed => { ExecutionError::Refused(CommandFailure::Unavailable) } diff --git a/crates/cellar/durability/raft/execution/enrollment.rs b/crates/cellar/durability/raft/execution/enrollment.rs index 355f7287a..76b5eed16 100644 --- a/crates/cellar/durability/raft/execution/enrollment.rs +++ b/crates/cellar/durability/raft/execution/enrollment.rs @@ -24,6 +24,7 @@ pub enum CellEnrollmentResolution { } struct Pending { + log_reservation: std::sync::Arc, record: EnrollmentRecord, _preview: Preview, submitted: bool, @@ -102,6 +103,11 @@ impl CellPrivateEnrollment { return Err(ExecutionError::Busy); } self.authority.check().map_err(authority_failure)?; + let log_reservation = self + .authority + .state() + .reserve_log(self.options.limits.envelope) + .map_err(authority_failure)?; self.staged = true; let working = self .working @@ -168,7 +174,11 @@ impl CellPrivateEnrollment { ) .map_err(storage)?; self.authority.check().map_err(authority_failure)?; + log_reservation + .encoded(bytes.len()) + .map_err(|error| authority_failure(error.into()))?; self.pending = Some(Pending { + log_reservation: log_reservation.clone(), record, _preview: preview, submitted: false, @@ -179,7 +189,8 @@ impl CellPrivateEnrollment { state.term(), state.route().history, state.engine_owner().clone(), - )) + ) + .with_log_reservation(log_reservation)) } pub fn mark_submitted(&mut self, creation: ProposalId) -> Result<(), ExecutionError> { @@ -215,6 +226,10 @@ impl CellPrivateEnrollment { Some(_) => CellEnrollmentResolution::Conflict, None => CellEnrollmentResolution::NotCommitted, }; + pending + .log_reservation + .release() + .map_err(|error| authority_failure(error.into()))?; self.pending = None; Ok(outcome) } diff --git a/crates/cellar/durability/raft/execution/submission.rs b/crates/cellar/durability/raft/execution/submission.rs index f3b32759f..4840a18b7 100644 --- a/crates/cellar/durability/raft/execution/submission.rs +++ b/crates/cellar/durability/raft/execution/submission.rs @@ -26,6 +26,7 @@ struct Submission { #[derive(Clone)] pub struct CellPreparedCommand { + log_reservation: Option>, bytes: Arc>, proposal: ProposalId, term: Term, @@ -55,6 +56,7 @@ impl CellPreparedCommand { engine: Arc<()>, ) -> Self { Self { + log_reservation: None, bytes: Arc::new(bytes), proposal, term, @@ -66,6 +68,29 @@ impl CellPreparedCommand { } } + pub(super) fn with_log_reservation( + mut self, + lease: Arc, + ) -> Self { + self.log_reservation = Some(lease); + self + } + + pub(crate) fn log_reservation( + &self, + ) -> Result<&Arc, RuntimeError> { + self.log_reservation + .as_ref() + .ok_or_else(|| RuntimeError::Invalid("missing prepared log reservation".into())) + } + + pub(super) fn release_log_reservation(&self) -> Result<(), RuntimeError> { + if let Some(lease) = &self.log_reservation { + lease.release()?; + } + Ok(()) + } + pub fn term(&self) -> Term { self.term } @@ -126,6 +151,7 @@ impl CellPreparedCommand { } Ok(None) if !uncertain => { *decision = Decision::Rejected; + self.release_log_reservation()?; Ok(CellSubmissionOutcome::Rejected(self.rejected())) } Ok(None) => { @@ -159,6 +185,9 @@ impl CellPreparedCommand { if matches!(*decision, Decision::Open) { *decision = Decision::Rejected; } + if matches!(*decision, Decision::Rejected) { + self.release_log_reservation()?; + } Ok(matches!(*decision, Decision::Rejected).then(|| self.rejected())) } diff --git a/crates/cellar/durability/raft/options.rs b/crates/cellar/durability/raft/options.rs index dca050a98..9fe3ca813 100644 --- a/crates/cellar/durability/raft/options.rs +++ b/crates/cellar/durability/raft/options.rs @@ -25,6 +25,7 @@ pub struct ReplicaLogOptions { pub max_batch_bytes: usize, pub max_cache_bytes: usize, pub max_entries_per_group: usize, + pub cell_group_retention: super::cell_groups::CellGroupLogRetentionLimits, pub target_file_bytes: u64, pub purge_threshold_bytes: u64, /// Minimum interval between dirty-engine purge passes on the storage worker. @@ -56,6 +57,7 @@ impl ReplicaLogOptions { max_batch_bytes: 32 * 1024 * 1024, max_cache_bytes: 256 * 1024 * 1024, max_entries_per_group: 16_384, + cell_group_retention: Default::default(), target_file_bytes: 16 * 1024 * 1024, purge_threshold_bytes: 64 * 1024 * 1024, purge_interval_ms: Some(1000), @@ -64,6 +66,7 @@ impl ReplicaLogOptions { } pub(crate) fn validate(&self) -> Result<(), LogError> { + self.cell_group_retention.validate()?; self.snapshot_scratch .validate() .map_err(|e| LogError::Invalid(e.to_string()))?; diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index f8b316aeb..fc73f37fd 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -47,6 +47,7 @@ const LIFECYCLE_KEY: &[u8] = b"cellar-node-lifecycle"; #[derive(Clone)] struct EngineCaches { + log_retention: Arc, usage: Arc, lifecycle: Arc>, } @@ -164,6 +165,7 @@ pub struct ReplicaLog { } struct LogHandle { + log_retention: Arc, snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, performance: Arc, instance: std::sync::OnceLock, @@ -304,6 +306,15 @@ impl RuntimeLease { } impl ReplicaLog { + /// Advisory node-wide logical retained-entry debt and reservation use. + /// This is not physical file accounting or an admission capability. + pub fn cell_group_log_retention_usage( + &self, + ) -> Result { + self.admit()?; + self.inner.log_retention.usage() + } + pub(crate) fn performance(&self) -> Arc { self.inner.performance.clone() } @@ -413,6 +424,10 @@ impl ReplicaLog { let (ready, wait) = oneshot::channel(); let groups = Arc::new(RwLock::new(BTreeMap::new())); let closed = Arc::new(AtomicBool::new(false)); + let log_retention = super::cell_groups::log_retention::NodeLogRetention::new( + options.cell_group_retention, + closed.clone(), + ); let abort = Arc::new(AtomicBool::new(false)); let terminal = Arc::new(Terminal { result: Mutex::new(None), @@ -424,6 +439,7 @@ impl ReplicaLog { let lifecycle_cache = Arc::new(Mutex::new(LifecycleRecord::empty())); let handle = Self { inner: Arc::new(LogHandle { + log_retention: log_retention.clone(), snapshot_scratch, performance: performance.clone(), instance: std::sync::OnceLock::new(), @@ -475,6 +491,7 @@ impl ReplicaLog { owner, closed.clone(), EngineCaches { + log_retention, usage: cache_usage, lifecycle: lifecycle_cache, }, @@ -850,6 +867,7 @@ impl MessageExt for EntryCodec { } struct Worker { + log_retention: Arc, filesystem: Arc, snapshot_scratch: cellar_ltx::stream_snapshot::ScratchPool, performance: Arc, @@ -1009,6 +1027,7 @@ impl Worker { } *caches.lifecycle.lock().map_err(|_| LogError::Closed)? = lifecycle.clone(); let mut worker = Self { + log_retention: caches.log_retention, filesystem: fs.clone(), snapshot_scratch, performance, @@ -1181,7 +1200,16 @@ impl Worker { cache: CachedGroup, scope: S, ) -> Result, LogError> { + let log_retention = if mapping.is_cells() { + Some( + self.log_retention + .register(mapping.key, cache.entries.values())?, + ) + } else { + None + }; Ok(GroupLog { + log_retention, archive_compaction: Arc::new( super::cell_groups::archive_compaction::ArchiveCompaction::new( cache.archive_required, @@ -1470,6 +1498,7 @@ impl Worker { let mut next_bytes = self.cache_bytes; let mut batch = LogBatch::default(); let mut archive_admissions = Vec::new(); + let mut log_admissions = Vec::new(); for write in &writes { if !Arc::ptr_eq(&write.group.owner, &self.owner) || !unique.insert(write.group.key) { return Err(LogError::WrongIdentity); @@ -1510,6 +1539,28 @@ impl Worker { self.options.max_entries_per_group, mapping.history()?, )?; + let mut leases = Vec::new(); + if let Some(retention) = &write.group.log_retention { + let indices: BTreeSet<_> = write.entries.iter().map(|e| e.index).collect(); + if !write + .log_reservations + .keys() + .all(|index| indices.contains(index)) + { + return Err(LogError::WrongIdentity); + } + for entry in &write.entries { + if let Some(lease) = write.log_reservations.get(&entry.index) { + lease.validate(retention, entry)?; + leases.push(lease.clone()); + } else if old.entries.get(&entry.index) != Some(entry) { + leases.push(retention.reserve_entry(entry)?); + } + } + } else if !write.log_reservations.is_empty() { + return Err(LogError::WrongIdentity); + } + log_admissions.push(leases); next_bytes = next_bytes .checked_sub(old.bytes()) .and_then(|size| size.checked_add(next.bytes())) @@ -1609,6 +1660,16 @@ impl Worker { .commit() .map_err(|e| LogError::Corrupt(e.to_string()))?; } + for (write, leases) in writes.iter().zip(&log_admissions) { + if let Some(retention) = &write.group.log_retention { + let cache = write.group.read()?; + retention + .publish(cache.entries.values(), leases) + .map_err(|error| { + LogError::Corrupt(format!("retained log publication: {error}")) + })?; + } + } for write in &writes { if let Some(activation) = &write.require_archive { activation diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index aec21d36d..f79530642 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -487,3 +487,177 @@ async fn archive_activation_is_retained_by_entered_storage() -> anyhow::Result<( domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_admission_accounts_selected_entries_and_reopen() -> anyhow::Result<()> { + use crate::raft::cell_groups::{CellGroupLogCapacity, CellGroupLogRetentionLimits}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut settings = options(root.path()); + settings.cell_group_retention = CellGroupLogRetentionLimits { + node: CellGroupLogCapacity { + bytes: 8192, + entries: 5, + }, + group: CellGroupLogCapacity { + bytes: 4096, + entries: 3, + }, + progress_node: CellGroupLogCapacity { + bytes: 2048, + entries: 2, + }, + progress_group: CellGroupLogCapacity { + bytes: 1024, + entries: 1, + }, + }; + let log = domain + .run(ReplicaLog::open( + settings.clone(), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let group = log.create_cell_group(bootstrap(5)).await?; + let sibling = log.create_cell_group(bootstrap(7)).await?; + let data = |index| Entry { + index, + term: 1, + data: vec![9; 256].into(), + ..Default::default() + }; + let mut first = GroupWrite::new(&group); + first.entries = vec![data(2), data(3)]; + log.persist_cells(OperationId::new(50), vec![first]).await?; + let selected = group.log_retention_usage()?; + assert_eq!(selected.retained_entries, 2); + assert_eq!(selected.reserved_entries, 0); + let mut full = GroupWrite::new(&group); + full.entries.push(data(4)); + assert!(matches!( + log.persist_cells(OperationId::new(51), vec![full]).await, + Err(LogError::Overloaded) + )); + assert_eq!(group.last_index()?, 3); + assert_eq!(group.log_retention_usage()?, selected); + // Empty Raft control entries have their own bounded space. Another data + // entry cannot consume it, and a healthy sibling still has node capacity. + let mut control = GroupWrite::new(&group); + control.entries.push(Entry { + index: 4, + term: 1, + ..Default::default() + }); + log.persist_cells(OperationId::new(52), vec![control]) + .await?; + let mut other = GroupWrite::new(&sibling); + other.entries.push(data(2)); + log.persist_cells(OperationId::new(53), vec![other]).await?; + let before = log.cell_group_log_retention_usage()?; + assert_eq!(before.retained_entries, 4); + assert_eq!(before.reserved_entries, 0); + log.close().await?; + drop(group); + drop(sibling); + drop(log); + settings.mode = LogOpenMode::Resume; + settings.process = ProcessGeneration::new([9; 16])?; + let log = domain + .run(ReplicaLog::open( + settings, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!(log.cell_group_log_retention_usage()?, before); + log.close().await?; + domain.shutdown().await; + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow::Result<()> { + use crate::raft::storage_fault as fault; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let gate = fault::Gate::new(None); + let log = domain + .run(ReplicaLog::open( + options(root.path()), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let group = log.create_cell_group(bootstrap(5)).await?; + let sibling = log.create_cell_group(bootstrap(7)).await?; + let entry = Entry { + index: 2, + term: 1, + data: vec![1; 256].into(), + ..Default::default() + }; + let lease = group + .log_retention + .as_ref() + .unwrap() + .reserve_entry(&entry)?; + let reserved = group.log_retention_usage()?; + assert_eq!( + (reserved.retained_entries, reserved.reserved_entries), + (0, 1) + ); + let mut wrong = GroupWrite::new(&sibling); + wrong.entries.push(entry.clone()); + wrong.log_reservations.insert(2, lease.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(60), vec![wrong]).await, + Err(LogError::WrongIdentity) + )); + let mut changed = GroupWrite::new(&group); + changed.entries.push(Entry { + term: 2, + ..entry.clone() + }); + changed.log_reservations.insert(2, lease.clone()); + assert!(matches!( + log.persist_cells(OperationId::new(61), vec![changed]).await, + Err(LogError::WrongIdentity) + )); + let release = gate.arm(fault::HOLD_SYNC); + let mut write = GroupWrite::new(&group); + write.entries.push(entry.clone()); + write.log_reservations.insert(2, lease.clone()); + let mut saving = Box::pin(log.persist_cells(OperationId::new(62), vec![write])); + assert!(domain + .run(crate::asyncrt::timeout( + Duration::from_millis(50), + saving.as_mut() + )) + .await? + .is_err()); + domain + .run(crate::asyncrt::timeout( + Duration::from_secs(5), + gate.entered.notified(), + )) + .await??; + drop(saving); + assert_eq!(group.log_retention_usage()?, reserved); + drop(release); + log.reserve_node_revision().await?; + let selected = group.log_retention_usage()?; + assert_eq!( + (selected.retained_entries, selected.reserved_entries), + (1, 0) + ); + assert_eq!(selected.retained_bytes, reserved.reserved_bytes); + // Keeping the original lease does not double-charge selected debt. It also + // cannot authorize another write after its exact reservation was consumed. + assert!(matches!( + lease.validate(group.log_retention.as_ref().unwrap(), &entry), + Err(LogError::WrongIdentity) + )); + drop(lease); + assert_eq!(group.log_retention_usage()?, selected); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index a3e354472..bfc6f0175 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -1347,3 +1347,13 @@ async fn provisioning_recovers_selected_roots() -> Result<()> { async fn provisioning_refusals_preserve_healthy_group() -> Result<()> { runtime_fixture::provisioning_refusals_preserve_healthy_group().await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn retained_log_materialized_refuses_new_callbacks_and_reclaims() -> Result<()> { + Box::pin(runtime_fixture::retained_log_admission(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn retained_log_extents_refuse_new_callbacks_and_reclaim() -> Result<()> { + Box::pin(runtime_fixture::retained_log_admission(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index e4d7d510f..61571c051 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -99,6 +99,27 @@ impl Cluster { schema: [u8; 32], store: Arc, minimum: Durability, + ) -> Result { + Self::with_store_retention( + options, + root, + filesystem, + schema, + store, + minimum, + Default::default(), + ) + .await + } + + async fn with_store_retention( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, + retention: CellGroupLogRetentionLimits, ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { @@ -145,13 +166,17 @@ impl Cluster { let gate = fault::Gate::new(None); let log = domain .run(ReplicaLog::open( - ReplicaLogOptions::new( - base.join("log"), - NodeId::new([n; 16])?, - DiskLineage::new([n + 10; 16])?, - ProcessGeneration::new([20; 16])?, - LogOpenMode::Create, - ), + { + let mut log = ReplicaLogOptions::new( + base.join("log"), + NodeId::new([n; 16])?, + DiskLineage::new([n + 10; 16])?, + ProcessGeneration::new([20; 16])?, + LogOpenMode::Create, + ); + log.cell_group_retention = retention; + log + }, Arc::new(fault::FaultFileSystem { gate: gate.clone() }), )) .await??; diff --git a/crates/cellar/tests/support/cell_group_log_retention.rs b/crates/cellar/tests/support/cell_group_log_retention.rs new file mode 100644 index 000000000..75792dcde --- /dev/null +++ b/crates/cellar/tests/support/cell_group_log_retention.rs @@ -0,0 +1,239 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual three-voter SQL and checkpoint boundaries at a small entry ceiling. +//! Full-backend outage authority and physical file amplification are separate. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { + let mut retention = CellGroupLogRetentionLimits::default(); + retention.group.entries = 5; + retention.progress_group.entries = 1; + let cluster = Cluster::with_store_retention( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + [6; 32], + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + retention, + ) + .await?; + let fixture = RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000).await?; + let (mut fixture, route, a, b) = + enroll_sql_cells(fixture, rooted, Arc::new(DirectFileSystem)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + assert_eq!(group.log_retention_usage()?.retained_entries, 4); + let mut executor = fixture.private_cell(&a).await?; + let (mut executor, prepared) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(prepared) = + executor.stage(cell_attempt(91)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"retention-result".to_vec()) + })? + else { + anyhow::bail!("prepared A") + }; + executor.mark_submitted(prepared.proposal())?; + Ok((executor, prepared)) + })? + .await??; + let reserved = group.log_retention_usage()?; + assert_eq!( + (reserved.retained_entries, reserved.reserved_entries), + (4, 1) + ); + let calls = Arc::new(AtomicUsize::new(0)); + for committed in [false, true] { + if committed { + let accepted = accepted_private(&fixture.runtimes[0], &prepared).await?; + fixture.drive(Some(accepted.position.index)).await?; + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + assert_eq!(group.log_retention_usage()?.retained_entries, 5); + } + let mut other = fixture.private_cell(&b).await?; + let observed = calls.clone(); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(matches!( + other.stage(cell_attempt(92)?, true, move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(vec![]) + }), + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + other.close()?; + Ok(()) + })? + .await??; + assert_eq!(calls.load(Ordering::SeqCst), 0); + } + let authority = fixture.runtimes[0] + .execution_authority(route.history, a.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"retention-result" + ); + executor.close()?; + Ok(()) + })? + .await??; + // A known receipt still resolves when no new ordinary entry can fit. + let mut duplicate = fixture.private_cell(&a).await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + let CellStageOutcome::Duplicate(receipt) = + duplicate.stage(cell_attempt(93)?, true, |_| { + panic!("duplicate callback must not run at the retention ceiling") + })? + else { + anyhow::bail!("known receipt") + }; + assert_eq!(receipt.receipt.result(), b"retention-result"); + duplicate.close()?; + Ok(()) + })? + .await??; + let d = history(&route, "retention-new")?; + let pool = CellGroupSqlPool::start( + &fixture.cluster.voters[0].log, + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + )?; + let authority = fixture.runtimes[0] + .enrollment_authority(route.history, d.clone()) + .await?; + let observed = calls.clone(); + let result = pool + .initialize( + fixture.cell_options(&d).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([94; 16])?, + preview: ImageId::new([94; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(()) + }, + ) + .await; + assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(calls.load(Ordering::SeqCst), 0); + pool.close().await?; + drop(pool); + // The saturated group must not consume the sibling's own entry allowance. + let sibling = fixture.add_group(47).await?; + fixture.runtimes[0].campaign(sibling).await?; + fixture.settle(&[route.history, sibling]).await?; + let other_route = fixture.extra[&sibling][0] + .log + .cell_group(&sibling)? + .group_manifest()? + .context("sibling root")? + .route() + .clone(); + let image = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(41);")?; + let cut = images::Cut::new(&[], &image, 0, true, false); + let bytes = request(&other_route, history(&other_route, "healthy")?, 1, &cut)?; + fixture.runtimes[0].propose(sibling, bytes).await?; + fixture.settle(&[route.history, sibling]).await?; + assert_eq!(group.log_retention_usage()?.retained_entries, 5); + // Only actual selected local checkpoints retire debt. Use the original + // one-source pool and wait for each physical publication before recapture. + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + for node in 0..3 { + let source = fixture.runtimes[node] + .capture_checkpoint(route.history, &checkpoints) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + fixture.runtimes[node] + .publish_checkpoint(route.history, prepared) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let usage = fixture.cluster.voters[node] + .log + .cell_group(&route.history)? + .log_retention_usage()?; + if usage.retained_entries == 0 + && fixture.runtimes[node].status(route.history).await?.ready + { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + assert_eq!(checkpoints.usage().sources, 0); + let mut executor = fixture.private_cell(&b).await?; + let (mut executor, next) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let CellStageOutcome::Prepared(next) = + executor.stage(cell_attempt(95)?, true, |tx| { + tx.execute("UPDATE value SET n=n+7", [])?; + Ok(b"after-checkpoint".to_vec()) + })? + else { + anyhow::bail!("new callback after actual reclaim") + }; + executor.mark_submitted(next.proposal())?; + Ok((executor, next)) + })? + .await??; + let accepted = accepted_private(&fixture.runtimes[0], &next).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"after-checkpoint" + ); + executor.close()?; + Ok(()) + })? + .await??; + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + drop(prepared); + drop(next); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 5f6755713..2050fbf70 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -922,3 +922,7 @@ pub(super) use sql::{seed_crash, seed_process_fixture}; pub(super) use sql::{membership_crash, membership_process_fixture}; pub(super) use sql::{seed_retry, shared_inventory}; + +pub(super) async fn retained_log_admission(rooted: bool) -> Result<()> { + Box::pin(sql::retained_log_admission(rooted)).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 979d11d44..c13854f8f 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -667,3 +667,7 @@ mod node_reclaim; pub(crate) use node_reclaim::shared_inventory; pub(crate) use seed_crash::seed_retry; + +#[path = "cell_group_log_retention.rs"] +mod log_retention; +pub(super) use log_retention::retained_log_admission; diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index e7f543700..7b89b09dd 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -1044,3 +1044,37 @@ owns the current remote hold, and no idle per-group replay workspace is kept. This gate covers local checkpoint publication. Full-backend outage retention admission, Raft file purge, peer-install integration of deployment archive policy, fresh-quorum recovery and the public hard cutover remain separate requirements. + + +### Engine-local retained entry ledger + +The shared-group storage engine owns one node ledger and one small group record +per physical history. `ReplicaLogOptions.cell_group_retention` supplies explicit +node/group byte and entry ceilings and bounded progress sublimits. Ordinary data +cannot consume the progress allowance for Raft's empty election/automatic leave +entries. Per-entry charges include fixed bookkeeping. Advisory debt observations +are not physical file measurements, durability proofs or admission capabilities. + +Duplicate receipts resolve before new callback admission. A new command or +initializer must obtain an owned worst-envelope reservation before executing user +code. Encoded preparation shrinks that reservation; delayed submission, ambiguous +outcomes and entered physical work retain it. Definitively sealed pre-append +rejection and authoritative receipt reconciliation may release unused leases. + +Before RawNode mutation, the owner admits local proposals, inherited peer entries +and possible election no-ops. Automatic joint leave receives promised capacity +before the configuration can require it; apply must not seek new admission after +its durable configuration is selected. Read-index envelopes are not log appends. +Heartbeat/read/completion work preserves its independent lanes at saturation. + +Persistence validates exact engine, group and entry bindings. Its selected cache +and root state precede an atomic reservation-to-debt transfer. Failed preflight +cannot change durable debt. Unknown physical outcomes close the engine; reopen +reconstructs actual retained debt without trusting old in-memory reservations. +Conflicting suffixes and local checkpoints retire only actual selected entries. +Archive coverage/status alone never releases ledger debt. A smaller restart limit +preserves selected entries and closes new admission until a reservation fits. + +This contract is being implemented and qualified. It does not by itself prove +physical disk bounds, full-backend S3 outage authority, fenced fresh-quorum serving +or the complete CG-06 gate. Those original requirements remain unchanged. From 2df640c995f1d9e478792ac64ebebc592909130c Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 01:21:26 -0700 Subject: [PATCH 217/568] fix: use Raft configuration accessors --- crates/cellar/durability/raft/cell_groups/log_admission.rs | 2 +- crates/cellar/durability/raft/cell_groups/owner.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/log_admission.rs b/crates/cellar/durability/raft/cell_groups/log_admission.rs index 6ebbc20da..cf7f60f56 100644 --- a/crates/cellar/durability/raft/cell_groups/log_admission.rs +++ b/crates/cellar/durability/raft/cell_groups/log_admission.rs @@ -191,7 +191,7 @@ fn needs_automatic_leave( group: &GroupLog, node: &RawNode, ) -> Result { - if node.raft.prs().conf().auto_leave { + if node.raft.prs().conf().auto_leave() { return Ok(true); } let cache = group.read()?; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index fc8861fb1..36eb0d869 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -1436,7 +1436,7 @@ impl CellGroupOwner { } let last = self.node.raft.raft_log.last_index(); let automatic = self.node.raft.state == StateRole::Leader - && self.node.raft.prs().conf().auto_leave + && self.node.raft.prs().conf().auto_leave() && self.node.raft.raft_log.applied <= self.node.raft.pending_conf_index && ticket.position.index.get() >= self.node.raft.pending_conf_index; let admission = self.log_admission.applying(automatic)?; From e090757f3c09ad3d90cd50a73609d3f171e81ea6 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 01:27:30 -0700 Subject: [PATCH 218/568] fix: read borrowed Raft configuration flag --- crates/cellar/durability/raft/cell_groups/log_admission.rs | 2 +- crates/cellar/durability/raft/cell_groups/owner.rs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/log_admission.rs b/crates/cellar/durability/raft/cell_groups/log_admission.rs index cf7f60f56..dcef04c14 100644 --- a/crates/cellar/durability/raft/cell_groups/log_admission.rs +++ b/crates/cellar/durability/raft/cell_groups/log_admission.rs @@ -191,7 +191,7 @@ fn needs_automatic_leave( group: &GroupLog, node: &RawNode, ) -> Result { - if node.raft.prs().conf().auto_leave() { + if *node.raft.prs().conf().auto_leave() { return Ok(true); } let cache = group.read()?; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 36eb0d869..bc3f16af4 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -1436,7 +1436,7 @@ impl CellGroupOwner { } let last = self.node.raft.raft_log.last_index(); let automatic = self.node.raft.state == StateRole::Leader - && self.node.raft.prs().conf().auto_leave() + && *self.node.raft.prs().conf().auto_leave() && self.node.raft.raft_log.applied <= self.node.raft.pending_conf_index && ticket.position.index.get() >= self.node.raft.pending_conf_index; let admission = self.log_admission.applying(automatic)?; From 1a8970840e2214bb88186e23dbd37effcbe80229 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 01:35:48 -0700 Subject: [PATCH 219/568] refactor: keep group storage handles compact --- crates/cellar/durability/raft/cache.rs | 12 +++++++----- .../raft/cell_groups/archive_compaction.rs | 7 ++++++- .../raft/cell_groups/enrollment_authority.rs | 3 +-- .../raft/cell_groups/execution_authority.rs | 3 +-- .../durability/raft/cell_groups/log_admission.rs | 3 +-- crates/cellar/durability/raft/storage.rs | 6 +++--- .../durability/raft/storage_cell_genesis_tests.rs | 8 ++------ 7 files changed, 21 insertions(+), 21 deletions(-) diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index f922c60d7..a86d203eb 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -199,7 +199,6 @@ impl GroupWrite { /// The engine rejects mixed application domains and never casts between them. #[derive(Clone)] pub struct GroupLog { - pub(crate) log_retention: Option>, pub(crate) archive_compaction: Arc, pub(crate) consensus_claimed: Arc, pub(crate) scratch_directory: std::path::PathBuf, @@ -225,6 +224,12 @@ impl std::fmt::Debug for GroupLog { } impl GroupLog { + pub(crate) fn log_retention( + &self, + ) -> Option<&Arc> { + self.archive_compaction.log_retention.as_ref() + } + pub fn identity(&self) -> &ReplicaIdentity { &self.identity } @@ -998,9 +1003,6 @@ impl GroupLog { pub fn log_retention_usage( &self, ) -> Result { - self.log_retention - .as_ref() - .ok_or(LogError::WrongIdentity)? - .usage() + self.log_retention().ok_or(LogError::WrongIdentity)?.usage() } } diff --git a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs index f21ca2c10..3186189f3 100644 --- a/crates/cellar/durability/raft/cell_groups/archive_compaction.rs +++ b/crates/cellar/durability/raft/cell_groups/archive_compaction.rs @@ -145,12 +145,17 @@ impl Drop for ArchiveActivation { /// restart in the group record; the frontier never survives without rechecking /// the actual remote hold. No replay workspace or per-cell state is retained. pub(crate) struct ArchiveCompaction { + pub(crate) log_retention: Option>, state: Mutex, } impl ArchiveCompaction { - pub(crate) fn new(required: bool) -> Self { + pub(crate) fn new( + required: bool, + log_retention: Option>, + ) -> Self { Self { + log_retention, state: Mutex::new(State { required, covered: None, diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs index c77883723..b050f9cf0 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -27,8 +27,7 @@ impl CellEnrollmentState { envelope: crate::application::consensus::EnvelopeLimits, ) -> Result, RuntimeError> { self.group - .log_retention - .as_ref() + .log_retention() .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? .reserve_execution(envelope) .map_err(|error| match error { diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index d982afee1..8f805fb02 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -30,8 +30,7 @@ impl CellExecutionAuthority { envelope: crate::application::consensus::EnvelopeLimits, ) -> Result, RuntimeError> { self.group - .log_retention - .as_ref() + .log_retention() .ok_or_else(|| RuntimeError::Protocol("missing cell-group log retention".into()))? .reserve_execution(envelope) .map_err(|error| match error { diff --git a/crates/cellar/durability/raft/cell_groups/log_admission.rs b/crates/cellar/durability/raft/cell_groups/log_admission.rs index dcef04c14..20e73ff46 100644 --- a/crates/cellar/durability/raft/cell_groups/log_admission.rs +++ b/crates/cellar/durability/raft/cell_groups/log_admission.rs @@ -28,8 +28,7 @@ fn retention( group: &GroupLog, ) -> Result<&Arc, RuntimeError> { group - .log_retention - .as_ref() + .log_retention() .ok_or_else(|| RuntimeError::Protocol("missing group log admission".into())) } diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index fc73f37fd..2a4c6f2db 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -1209,10 +1209,10 @@ impl Worker { None }; Ok(GroupLog { - log_retention, archive_compaction: Arc::new( super::cell_groups::archive_compaction::ArchiveCompaction::new( cache.archive_required, + log_retention, ), ), consensus_claimed: Arc::new(AtomicBool::new(false)), @@ -1540,7 +1540,7 @@ impl Worker { mapping.history()?, )?; let mut leases = Vec::new(); - if let Some(retention) = &write.group.log_retention { + if let Some(retention) = write.group.log_retention() { let indices: BTreeSet<_> = write.entries.iter().map(|e| e.index).collect(); if !write .log_reservations @@ -1661,7 +1661,7 @@ impl Worker { .map_err(|e| LogError::Corrupt(e.to_string()))?; } for (write, leases) in writes.iter().zip(&log_admissions) { - if let Some(retention) = &write.group.log_retention { + if let Some(retention) = write.group.log_retention() { let cache = write.group.read()?; retention .publish(cache.entries.values(), leases) diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index f79530642..d855f87e7 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -594,11 +594,7 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: data: vec![1; 256].into(), ..Default::default() }; - let lease = group - .log_retention - .as_ref() - .unwrap() - .reserve_entry(&entry)?; + let lease = group.log_retention().unwrap().reserve_entry(&entry)?; let reserved = group.log_retention_usage()?; assert_eq!( (reserved.retained_entries, reserved.reserved_entries), @@ -652,7 +648,7 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: // Keeping the original lease does not double-charge selected debt. It also // cannot authorize another write after its exact reservation was consumed. assert!(matches!( - lease.validate(group.log_retention.as_ref().unwrap(), &entry), + lease.validate(group.log_retention().unwrap(), &entry), Err(LogError::WrongIdentity) )); drop(lease); From b36ab33d10be7edf150882929c56cf43504660f1 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 01:49:42 -0700 Subject: [PATCH 220/568] fix: revoke authority before log bookkeeping --- crates/cellar/durability/raft/cell_groups/owner.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index bc3f16af4..2a33a7043 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -513,8 +513,8 @@ impl CellGroupOwner { .log_admission .election(&self.group, &self.node, Default::default())?; let result = self.node.campaign(); - self.finish_log_admission(admission)?; self.refresh_execution_fence(); + self.finish_log_admission(admission)?; result?; Ok(()) } @@ -568,8 +568,8 @@ impl CellGroupOwner { .election(&self.group, &self.node, Default::default())? }; self.node.tick(); - self.finish_log_admission(admission)?; self.refresh_execution_fence(); + self.finish_log_admission(admission)?; Ok(()) } @@ -725,8 +725,8 @@ impl CellGroupOwner { return Err(RuntimeError::Overloaded); } let result = self.node.step(message); - self.finish_log_admission(Default::default())?; self.refresh_execution_fence(); + self.finish_log_admission(Default::default())?; result?; if let Some(snapshot) = self.node.snap() { if snapshot != checkpoint.snapshot() { @@ -769,8 +769,8 @@ impl CellGroupOwner { .election(&self.group, &self.node, admission)?; } let result = self.node.step(message); - self.finish_log_admission(admission)?; self.refresh_execution_fence(); + self.finish_log_admission(admission)?; result?; Ok(()) } From 4e05c21430216ab12e5341244ec59c7c355f76fc Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 02:11:20 -0700 Subject: [PATCH 221/568] test: prove log lease and joint progress ownership --- .../raft/storage_cell_genesis_tests.rs | 33 +++++- .../tests/support/cell_group_log_retention.rs | 101 ++++++++++++++++++ 2 files changed, 130 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index d855f87e7..e2a12bd4a 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -635,7 +635,13 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: gate.entered.notified(), )) .await??; + let physical_lease = Arc::downgrade(&lease); drop(saving); + drop(lease); + assert!( + physical_lease.upgrade().is_some(), + "entered storage owns the final strong lease" + ); assert_eq!(group.log_retention_usage()?, reserved); drop(release); log.reserve_node_revision().await?; @@ -645,14 +651,33 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: (1, 0) ); assert_eq!(selected.retained_bytes, reserved.reserved_bytes); - // Keeping the original lease does not double-charge selected debt. It also - // cannot authorize another write after its exact reservation was consumed. + assert!( + physical_lease.upgrade().is_none(), + "FIFO join releases every physical lease clone" + ); + assert_eq!(group.log_retention_usage()?, selected); + // A separately retained client lease is consumed exactly once. Keeping it + // cannot double-charge selected debt or authorize reuse of the capability. + let next = Entry { + index: 3, + ..entry.clone() + }; + let lease = group.log_retention().unwrap().reserve_entry(&next)?; + let mut write = GroupWrite::new(&group); + write.entries.push(next.clone()); + write.log_reservations.insert(3, lease.clone()); + log.persist_cells(OperationId::new(63), vec![write]).await?; + let complete = group.log_retention_usage()?; + assert_eq!( + (complete.retained_entries, complete.reserved_entries), + (2, 0) + ); assert!(matches!( - lease.validate(group.log_retention().unwrap(), &entry), + lease.validate(group.log_retention().unwrap(), &next), Err(LogError::WrongIdentity) )); drop(lease); - assert_eq!(group.log_retention_usage()?, selected); + assert_eq!(group.log_retention_usage()?, complete); log.close().await?; domain.shutdown().await; Ok(()) diff --git a/crates/cellar/tests/support/cell_group_log_retention.rs b/crates/cellar/tests/support/cell_group_log_retention.rs index 75792dcde..63c05d62c 100644 --- a/crates/cellar/tests/support/cell_group_log_retention.rs +++ b/crates/cellar/tests/support/cell_group_log_retention.rs @@ -3,6 +3,10 @@ //! Actual three-voter SQL and checkpoint boundaries at a small entry ceiling. //! Full-backend outage authority and physical file amplification are separate. use super::*; +use raft::{ + prelude::{ConfChangeSingle, ConfChangeTransition, ConfChangeType, ConfChangeV2}, + Storage as _, +}; use std::sync::atomic::{AtomicUsize, Ordering}; pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { @@ -233,6 +237,103 @@ pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { })? .await??; assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + // Fill three of the four ordinary slots with real cell commands. The + // revision-bound joint configuration consumes the fourth; its automatic + // leave must use promised progress capacity while ordinary admission is full. + for sequence in 2..=3 { + let mut executor = fixture.private_cell(&b).await?; + let (mut executor, command) = fixture + .cluster + .domain + .blocking(move || -> Result<_> { + let mut attempt = cell_attempt(94 + sequence as u8)?; + attempt.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + let CellStageOutcome::Prepared(command) = executor.stage(attempt, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"joint-progress".to_vec()) + })? + else { + anyhow::bail!("ordinary command before joint transition") + }; + executor.mark_submitted(command.proposal())?; + Ok((executor, command)) + })? + .await??; + let accepted = accepted_private(&fixture.runtimes[0], &command).await?; + fixture.drive(Some(accepted.position.index)).await?; + let authority = fixture.runtimes[0] + .execution_authority(route.history, b.clone()) + .await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!( + executor.finish(authority)?.receipt.result(), + b"joint-progress" + ); + executor.close()?; + Ok(()) + })? + .await??; + } + assert_eq!(group.log_retention_usage()?.retained_entries, 3); + let configuration = ConfChangeV2 { + transition: ConfChangeTransition::Auto, + changes: vec![ + ConfChangeSingle { + node_id: 3, + change_type: ConfChangeType::AddLearnerNode, + ..Default::default() + }, + ConfChangeSingle { + node_id: 2, + change_type: ConfChangeType::AddNode, + ..Default::default() + }, + ] + .into(), + ..Default::default() + }; + let request = CellGroupMembershipChange::new(&route, &group.membership()?, configuration, &[])?; + let accepted = fixture.runtimes[0] + .configure(route.history, request) + .await?; + tokio::time::timeout(Duration::from_secs(20), async { + loop { + fixture.pump(&[], &mut BTreeMap::new()).await?; + let conf = group.initial_state()?.conf_state; + let status = fixture.runtimes[0].status(route.history).await?; + if conf.voters_outgoing.is_empty() + && conf.learners == vec![3] + && status.applied.index > accepted.position.index + && fixture.applied[&route.history] + .iter() + .all(|index| *index >= status.applied.index) + { + break; + } + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await + .context("automatic joint leave progresses at the ordinary entry ceiling")??; + let usage = group.log_retention_usage()?; + assert_eq!((usage.retained_entries, usage.reserved_entries), (5, 0)); + for voter in &fixture.cluster.voters { + let group = voter.log.cell_group(&route.history)?; + assert_eq!(group.initial_state()?.conf_state.learners, vec![3]); + assert_eq!( + group + .group_manifest()? + .context("complete post-joint root")? + .root() + .cells(), + 3 + ); + assert_eq!(group.log_retention_usage()?.reserved_entries, 0); + } drop(prepared); drop(next); fixture.close().await From 5238b9ac3e7cb07e3fb113642bc8f664d227f37a Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 02:25:42 -0700 Subject: [PATCH 222/568] test: isolate lease and byte admission boundaries --- .../raft/storage_cell_genesis_tests.rs | 126 +++++++++++++++++- 1 file changed, 122 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs index e2a12bd4a..51eb9c774 100644 --- a/crates/cellar/durability/raft/storage_cell_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_cell_genesis_tests.rs @@ -612,11 +612,20 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: term: 2, ..entry.clone() }); + // Keep the proposed cache state valid so the negative reaches the exact + // lease binding check, rather than failing the independent term invariant. + let mut hard = group.initial_state()?.hard_state; + hard.term = 2; + changed.hard_state = Some(hard); changed.log_reservations.insert(2, lease.clone()); - assert!(matches!( - log.persist_cells(OperationId::new(61), vec![changed]).await, - Err(LogError::WrongIdentity) - )); + let changed_result = log.persist_cells(OperationId::new(61), vec![changed]).await; + assert!( + matches!(&changed_result, Err(LogError::WrongIdentity)), + "wrong-term lease refusal: {changed_result:?}" + ); + assert_eq!(group.initial_state()?.hard_state.term, 1); + assert_eq!(group.last_index()?, 1); + assert_eq!(group.log_retention_usage()?, reserved); let release = gate.arm(fault::HOLD_SYNC); let mut write = GroupWrite::new(&group); write.entries.push(entry.clone()); @@ -682,3 +691,112 @@ async fn retained_log_lease_survives_waiter_loss_and_transfers_once() -> anyhow: domain.shutdown().await; Ok(()) } + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn retained_log_byte_limits_and_selected_suffix_replacement() -> anyhow::Result<()> { + use crate::raft::cell_groups::{CellGroupLogCapacity, CellGroupLogRetentionLimits}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut settings = options(root.path()); + settings.cell_group_retention = CellGroupLogRetentionLimits { + node: CellGroupLogCapacity { + bytes: 8192, + entries: 100, + }, + group: CellGroupLogCapacity { + bytes: 4096, + entries: 20, + }, + progress_node: CellGroupLogCapacity { + bytes: 2048, + entries: 20, + }, + progress_group: CellGroupLogCapacity { + bytes: 1024, + entries: 2, + }, + }; + let log = domain + .run(ReplicaLog::open( + settings, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let a = log.create_cell_group(bootstrap(5)).await?; + let b = log.create_cell_group(bootstrap(7)).await?; + let c = log.create_cell_group(bootstrap(9)).await?; + let large = Entry { + index: 2, + term: 1, + data: vec![1; 1800].into(), + ..Default::default() + }; + let mut write = GroupWrite::new(&a); + write.entries.push(large.clone()); + log.persist_cells(OperationId::new(70), vec![write]).await?; + let original = a.log_retention_usage()?; + assert_eq!( + (original.retained_entries, original.reserved_entries), + (1, 0) + ); + let mut write = GroupWrite::new(&a); + write.entries.push(Entry { + index: 3, + ..large.clone() + }); + assert!( + matches!( + log.persist_cells(OperationId::new(71), vec![write]).await, + Err(LogError::Overloaded) + ), + "group bytes refuse while entry counts have capacity" + ); + assert_eq!(a.last_index()?, 2); + assert_eq!(a.log_retention_usage()?, original); + let mut write = GroupWrite::new(&b); + write.entries.push(large.clone()); + log.persist_cells(OperationId::new(72), vec![write]).await?; + let before = log.cell_group_log_retention_usage()?; + assert_eq!((before.retained_entries, before.reserved_entries), (2, 0)); + let mut write = GroupWrite::new(&c); + write.entries.push(large.clone()); + assert!( + matches!( + log.persist_cells(OperationId::new(73), vec![write]).await, + Err(LogError::Overloaded) + ), + "node bytes refuse a fitting group while entry counts have capacity" + ); + assert_eq!(c.last_index()?, 1); + assert_eq!(log.cell_group_log_retention_usage()?, before); + // A smaller replacement of an uncommitted suffix releases debt only after + // its new entry and term are actually selected by storage. Both versions + // fit the admitted overlap; no archive observation provides the refund. + let mut write = GroupWrite::new(&a); + write.entries.push(Entry { + index: 2, + term: 2, + data: vec![2; 64].into(), + ..Default::default() + }); + let mut hard = a.initial_state()?.hard_state; + hard.term = 2; + write.hard_state = Some(hard); + log.persist_cells(OperationId::new(74), vec![write]).await?; + let smaller = a.log_retention_usage()?; + assert_eq!((smaller.retained_entries, smaller.reserved_entries), (1, 0)); + assert!(smaller.retained_bytes < original.retained_bytes); + let mut write = GroupWrite::new(&c); + write.entries.push(large); + log.persist_cells(OperationId::new(75), vec![write]).await?; + let complete = log.cell_group_log_retention_usage()?; + assert_eq!( + (complete.retained_entries, complete.reserved_entries), + (3, 0) + ); + assert_eq!(a.last_index()?, 2); + assert_eq!(a.term(2)?, 2); + log.close().await?; + domain.shutdown().await; + Ok(()) +} From 462db264cac1a4e3f499b16227b905abc03329e0 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 02:46:50 -0700 Subject: [PATCH 223/568] test: await exact checkpoints on followers Leader readiness cannot become true on followers. Require the selected cut, retired log debt and released source quota on every voter. --- .../tests/support/cell_group_log_retention.rs | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_log_retention.rs b/crates/cellar/tests/support/cell_group_log_retention.rs index 63c05d62c..b35fdc45f 100644 --- a/crates/cellar/tests/support/cell_group_log_retention.rs +++ b/crates/cellar/tests/support/cell_group_log_retention.rs @@ -180,18 +180,20 @@ pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { .domain .blocking(move || source.prepare_publication())? .await??; + let cut = prepared.marker().clone(); fixture.runtimes[node] .publish_checkpoint(route.history, prepared) .await?; tokio::time::timeout(Duration::from_secs(20), async { loop { fixture.pump(&[], &mut BTreeMap::new()).await?; - let usage = fixture.cluster.voters[node] + let group = fixture.cluster.voters[node] .log - .cell_group(&route.history)? - .log_retention_usage()?; + .cell_group(&route.history)?; + let usage = group.log_retention_usage()?; if usage.retained_entries == 0 - && fixture.runtimes[node].status(route.history).await?.ready + && group.checkpoint_manifest()?.as_ref() == Some(&cut) + && checkpoints.usage().sources == 0 { break; } @@ -199,7 +201,8 @@ pub(crate) async fn retained_log_admission(rooted: bool) -> Result<()> { } Ok::<_, anyhow::Error>(()) }) - .await??; + .await + .with_context(|| format!("voter {node} selects checkpoint and releases its source"))??; } assert_eq!(checkpoints.usage().sources, 0); let mut executor = fixture.private_cell(&b).await?; From ad73f03f278f5845b311fe929d184cc947382b2d Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 03:08:17 -0700 Subject: [PATCH 224/568] test: construct snapshot futures off poll frame The composite restart case overflows the default test stack. Use the existing non-inlined fixture constructor, preserving all assertions and resource limits. --- crates/cellar/tests/native_cell_groups_snapshot.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 60be8f8b2..15bf7aadf 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -76,8 +76,8 @@ async fn snapshot_empty_cell_and_extent_restart() -> Result<()> { // Raft generates the snapshot over TLS after prefix compaction. The // selected/retained closures include empty enrollment and later writes, // and cold reopen verifies the exact checkpoint plus surviving suffix. - runtime_fixture::tcp_publication(rooted).await?; - runtime_fixture::install_checkpoint(rooted).await?; + fixture(|| runtime_fixture::tcp_publication(rooted)).await?; + fixture(|| runtime_fixture::install_checkpoint(rooted)).await?; } Ok(()) } From a6e80176fafe8d1055b33e30c396b224d5c6b51e Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 03:31:02 -0700 Subject: [PATCH 225/568] docs: record retained-log qualification --- .../multi-raft/cell-groups/baseline.json | 10 +- docs/design/multi-raft/cell-groups/handoff.md | 113 ++++++++---------- .../design/multi-raft/cell-groups/progress.md | 77 ++++++++++++ 3 files changed, 132 insertions(+), 68 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index c748607d7..ad7153860 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ab0abf26dea4df436b5ced926f28a9ede4f360fd", - "runtime_revision": "ab0abf26dea4df436b5ced926f28a9ede4f360fd", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged.", + "source_revision": "ad73f03f278f5845b311fe929d184cc947382b2d", + "runtime_revision": "ad73f03f278f5845b311fe929d184cc947382b2d", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "b9c7fc76a87a6ec0ae5ee28c79f24e59c6e582daecb58727d1f1a64c7b75a2e7", - "line": 451, + "sha256": "c8e7da3fdd12cc788389b69b13d2799b7930e666484aa23fa84660a24ed9bb6c", + "line": 460, "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 07620b33b..0050607bc 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -16,69 +16,56 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). The latest runtime/qualification source is -`ab0abf26dea4df436b5ced926f28a9ede4f360fd`. CG-06 remains IN_PROGRESS. Archive -registration owns a pending activation that closes later checkpoint admission. -Earlier owner/storage work shares an exact cut-bound lease and drains before -durable policy publication. Entered metadata persistence owns activation after -waiter loss. An unentered intent releases its temporary barrier on -rejection/shutdown; only sync and cache/root publication make the archive -requirement irreversible. A once-only pre-I/O frontier reset avoids discarding -newer coverage on duplicate completion. Wrong-group, wrong-cut and -non-checkpoint lease use refuse. Marker-1 records and original quotas are -unchanged; the cut lease fits the existing 4 KiB charge. - -Archive-configured local checkpoint compaction still requires independently -verified complete coverage and a fresh frontier after retained-disk reopen. -Missing-field records refuse without migration or a legacy reader. Ordinary -writes and peer snapshot installation cannot clear policy or manufacture -coverage. Scheduler prechecks allow apply and sibling checkpoints to progress. -Public archive status grants no compaction or response authority. The frontier -retains bounded metadata; the journal owns the remote hold. Deployment policy on -peer install remains separately unqualified. - -Local and remote command/enrollment/receipt S3 outputs retain the exact receiver -controller/runtime archive owner, complete independently recovered cuts, -original request/namespace/stored minima and deadlines, fresh catalog/leader -checks and authenticated achieved durability. Native outputs own physical proof -through TLS flush or ordered stream destruction. Quorum outputs release SQL -independently. No retry replays callbacks; missing receipts remain ambiguous. -Enrollment preserves original creation provenance. These contracts remain -required at public cutover. - -All three exact storage ownership cases pass, including the unchanged overtaking regression, real checkpoint drain and a held-sync cancelled waiter. Seventeen exact RustFS cases pass in both image forms with seven signed probes. Twenty-two regression/check commands pass with 262 Rust cases across seven copied executables, including nineteen storage cases. Locked all-target Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests and system-Python hashing pass. Independent source/hash/selection and credential-exclusion audit passes. Source-bound records, frozen plans, orchestration, original red regression and copied executables are under `.cellar/cell-groups-evidence/CG-06-retention-admission/`. Earlier evidence stays retained. The `8dfe9ff3` slow-reader timeout remains unexplained despite unchanged passing reruns. Sustained activation/load fairness and zero-voter-disk-loss recovery are not established by this matrix. See [progress](progress.md) and the [local compaction contract](protocols.md#archive-aware-local-checkpoint-compaction). - -Next implement explicit retained-log byte/entry admission before new callbacks, -preserving known receipt reconciliation. Account for follower/control work, -suffix replacement, restart and uncertain physical outcomes; transfer -reservations into durable debt without release gaps or double charges. Preserve -reserved completion/apply/checkpoint capacity so a full budget cannot strand -collected Ready. Qualify a full-backend S3 outage, resolving fresh catalog -authority without weakening ownership or epoch fencing. Wrapper faults and local -compaction checks alone do not establish a disk bound. Add physical -shared-engine file purge constrained by every checkpoint, archive, reader, -transfer and sibling group. Fenced fresh-incarnation serving must inventory -every possible old voter, including joins after the selected archive cut. -Fencing-authority API plus Docker qualification remains accepted. Peer-install -policy, process-kill and multi-head/physical-read shutdown cuts still need -qualification. - -CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover, then -topology, placement, hibernation, managed recovery, measured load and legacy -removal. Four original CG-06 cases/full gate remain pending. Voters are -host-native tasks and RustFS one VM; independent domains, container node limits, -measured load, fresh-quorum disaster serving and the hard cutover remain -unqualified. The full end-to-end goal remains active. - -Preserve archive-compaction runtime `8dfe9ff3`, its source-bound failures and -the unresolved slow-reader timing limitation. Preserve remote response source -`25938fd8` with validation-only inventory correction `debdc30c`, receipt -`cde0eea2`, enrollment `c2801b65`, command `3d564f3d`, prior RustFS `d739bf79`, -controller `7914177d`, catalog cut `df0ba1f6`, protected publication `87348477`, -isolated verifier `14955f2d`, committed tail `7a29c2a9` and their evidence. -Exact intents precede remote mutation; uncertain outcomes cannot allocate new -identities or reset counters. Namespace binding, stable cuts, protected -publication and independent complete recovery preserve their contracts. The -archive controller still coalesces complete cuts. +`ad73f03f278f5845b311fe929d184cc947382b2d`. CG-06 remains IN_PROGRESS. Logical +retained-log bytes/entries and reservations now share explicit node/group +ceilings. Restart reconstructs selected debt; smaller ceilings preserve history +while refusing new work. New callback/initializer envelopes reserve before +execution, known receipts still resolve at capacity, and exact leases survive +uncertain submission and entered physical work. Durable selection transfers the +charge atomically into retained debt. Checkpoints and suffix replacement release +only actually retired entries. These limits do not bound physical engine files. + +All shared Raft append paths use the ledger, including inherited entries and +election/configuration controls. Joint changes reserve their future automatic +leave before append; the native scenario fills ordinary admission and verifies +that leave uses the single progress slot. Role revocation precedes bookkeeping. +Selected checkpoint identity and released physical source quota, rather than +leader-only readiness, gate recapture on each voter. + +The prior archive activation and checkpoint-drain contract remains required: +pending activation closes later checkpoint admission, earlier exact cut-bound +owner/storage leases drain, entered metadata work owns activation through waiter +loss, and sync/cache publication makes policy irreversible. Archive-configured +local compaction requires independently verified complete coverage and fresh +verification after reopen. Public archive status grants no compaction, output or +serving authority. Marker-1 serialization and original task oracles remain +unchanged. + +Local/remote command, enrollment and receipt S3 responses retain their exact +archive owners and independently verified cuts through physical output. Original +durability minima/deadlines, fresh catalog/leader checks, authenticated achieved +durability and creation provenance remain required. Unknown outcomes never +replay callbacks. Public SDK/fleet cutover still has to integrate these +contracts. + +Three exact storage cases and both native image scenarios pass. Seventeen real RustFS controller cases pass in both image forms with seven signed probes. Twenty-two regression/check commands pass with 267 distinct Rust cases across seven copied executables, including all 22 storage cases. Final-source all-target Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests and independent source/hash/selection/credential audit pass. Evidence binds 2574 sources under `.cellar/cell-groups-evidence/CG-06-log-retention/`; see [progress](progress.md) for precise scope and retained failures. The earlier `8dfe9ff3` slow-reader timeout remains unexplained. This matrix does not establish sustained load or delete all voter disks. + +Next resolve full S3-outage catalog/epoch authority without weakening fencing, +then physical shared-engine file reclamation across every +checkpoint/archive/reader/transfer/sibling hold. Logical debt admission and +wrapper-fault tests are insufficient for a physical disk bound. Fenced +fresh-incarnation serving must inventory every possible old voter, including +joins after the archived cut. The accepted fencing-authority API plus Docker +qualification remains the target. Peer-install deployment policy, process-kill +and physical-read/output shutdown cuts still need qualification. + +CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover. +Topology, placement, hibernation, managed recovery, measured load and final +legacy removal follow. Four original CG-06 cases/full gate remain pending. +Voters are host-native tasks and RustFS one VM; independent failure domains and +the full hard cutover remain unqualified. The full end-to-end goal remains +active. Preserve predecessor runtime `ab0abf26`, archive-compaction runtime +`8dfe9ff3`, all source-bound failures and earlier evidence. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index b5a536148..c1b3651da 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,83 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 retained-log admission and progress capacity + +Source: `ad73f03f278f5845b311fe929d184cc947382b2d`. CG-06 remains IN_PROGRESS. +Shared-group logs now account for logical retained entries and outstanding +reservations against explicit node and group byte/count ceilings. The ledger +reconstructs selected durable debt on reopen. Preexisting debt is preserved when +a restart uses smaller limits, and new admission waits until it fits. +Encoded-entry bytes include bounded accounting overhead; these are logical +limits, not physical Raft Engine file-byte bounds. + +New SQL callbacks and initializers reserve a worst-case command envelope before +execution. Known receipt reconciliation remains available at the ceiling. A +preparation carries an exact engine/group/entry lease through submission and +uncertain outcomes. Only sealed pre-append rejection or authoritative +reconciliation releases unused preparation capacity. Entered storage retains the +lease after waiter loss, then transfers its charge into selected durable entries +after sync/cache publication without a release gap or double debit. Consumed +leases cannot authorize another append. + +Local, inherited follower, election and configuration entries use the same +ledger. Empty election and automatic configuration entries have separately +reserved progress capacity. A joint transition promises its automatic leave +before it can enter RawNode, avoiding capacity acquisition after collected apply +work changes configuration. Capacity refusal on a deferred tick is nonfatal. +Role changes revoke concurrent execution authority before accounting. Actual +selected checkpoints and suffix replacement retire debt; archive status alone +does not. Existing archive-required compaction still needs independently +verified complete coverage. + +Three exact real-engine cases pass for entry ceilings/reopen, independent +node/group byte ceilings with slack counts, suffix replacement, exact lease +identity and held physical sync after all caller owners disappear. Both native +SQL image forms pass before-callback and initializer refusal, known receipt +recovery, sibling progress, exact local checkpoint selection/source release and +resumed writes. They fill ordinary admission before a real automatic joint +transition and require its automatic leave through the unchanged single progress +entry. The local checkpoint scenario does not claim a full S3 backend outage or +physical file purge. + +All seventeen exact real RustFS controller cases pass in both image forms, with +seven signed S3 protocol probes. Twenty-two regression/check commands pass with +267 distinct Rust cases and seven copied executables; all 22 storage cases and +four native shared-group suites run. Locked all-target Cellar/logic/LTX Clippy, +format/layout, native dependency/docs/preflight/plan checks, 37 Python tests and +system-Python hashing pass. Source/artifact/selection and credential-exclusion +audit passes across 2574 source inputs. Records, copied binaries/outputs, +orchestration and frozen plan inputs are under +`.cellar/cell-groups-evidence/CG-06-log-retention/`. + +Source-bound failures remain retained: initial accessor/type/layout compile +checks, the coherent-hardstate correction to a wrong-term lease negative, the +native fixture's impossible leader-only wait on followers, and the aggregate +snapshot test's default-stack overflow. Copied binaries reproduce these fixture +failures. The successor keeps the original identity assertions, quotas and +deadlines and waits for the exact selected cut, retired debt and released +checkpoint source on each voter. The aggregate snapshot case uses the existing +non-inlined fixture constructor to keep large async construction off its poll +frame; the exact case and complete suite pass on the original default stack. No +temporary DEBUG instrumentation remains. The earlier `8dfe9ff3` slow-reader +timeout remains unexplained despite unchanged passing reruns; this change is not +claimed to fix it. The RustFS controller matrix does not delete voter disks. +Original CG-06 cases, acceptance oracles and evidence array remain unchanged; +four original cases/full task gate remain incomplete. + +Reflection: next qualify a full backend S3 outage with fresh catalog/epoch +authority, without caching ownership past fencing. Complete and qualify physical +shared-engine file reclamation constrained by every +checkpoint/archive/reader/transfer and sibling hold; logical admission alone +does not establish a disk bound. Complete fenced fresh-group/cell-incarnation +recovery and replacement-quorum serving with the complete inventory of possible +old voters, including joins after the selected cut, using the accepted +fencing-authority API plus Docker qualification. Then finish CG-08 +effect/cache/result lifetimes before CG-07 public SDK/fleet cutover. Topology, +placement, hibernation, managed recovery, measured load and final legacy removal +follow. Voters remain host-native tasks and RustFS one local VM; independent +failure domains and the full hard cutover remain unqualified. + ## CG-06 owned archive activation and checkpoint drain Source: `ab0abf26dea4df436b5ced926f28a9ede4f360fd`. CG-06 remains IN_PROGRESS. A From 0ecd34db504104354adc034c2959e268696790b7 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 03:58:58 -0700 Subject: [PATCH 226/568] test: exercise physical group log reclamation --- .../tests/native_cell_groups_archive.rs | 16 + .../native_cell_groups_archive_controller.rs | 13 + .../support/cell_group_controller_archive.rs | 18 +- .../support/cell_group_engine_fixture.rs | 48 +- .../tests/support/cell_group_log_reclaim.rs | 652 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 7 + crates/cellar/tests/support/cell_group_sql.rs | 4 + .../support/cell_group_sql_registered.rs | 2 +- tools/multi-raft/cell_groups.py | 1 + 9 files changed, 755 insertions(+), 6 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_log_reclaim.rs diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 3d678460b..c7ebc9b25 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -27,8 +27,24 @@ mod publication_fixture; #[path = "support/cell_group_archive_tail.rs"] mod tail_fixture; +#[path = "support/cell_group_runtime.rs"] +#[allow(dead_code, unused_imports)] +mod runtime_fixture; + include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_reclaim_restart_keeps_history() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::physical_log_reclamation( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn archive_cut_upload_survives_quorum_progress() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index e11a2e197..efa54d5b6 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -20,6 +20,19 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_group_reclaim_restart_keeps_history() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::physical_log_reclamation( + rooted, + rustfs::store("physical-log-reclaim", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn archive_coverage_gates_local_compaction() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 86d938343..797a7eeea 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -209,12 +209,12 @@ async fn setup_store( Ok((fixture, route)) } -async fn audit( +pub(super) async fn verified_current( fixture: &RuntimeCluster, config: &CellGroupArchiveRegistration, rooted: bool, - last: u8, -) -> Result<()> { + name: &str, +) -> Result { let route = config.birth.route(); let registry = config .directory @@ -240,7 +240,7 @@ async fn audit( let hold: CellGroupArchiveHold = serde_json::from_value(envelope["record"]["head"].clone())?; assert!(envelope["record"]["retire"].is_null()); assert_eq!(envelope["record"]["pending"], "None"); - let destination = verifier(fixture, route, rooted, "controller-archive-audit").await?; + let destination = verifier(fixture, route, rooted, name).await?; let coverage = registry .verify_hold( &fixture.cluster.domain, @@ -253,6 +253,16 @@ async fn audit( )?, ) .await?; + Ok(coverage) +} + +async fn audit( + fixture: &RuntimeCluster, + config: &CellGroupArchiveRegistration, + rooted: bool, + last: u8, +) -> Result<()> { + let coverage = verified_current(fixture, config, rooted, "controller-archive-audit").await?; let root = coverage.recovery().manifest(); assert_eq!(root.pinned_images(), usize::from(last) + 2); let names = ["registered-a".to_owned(), "registered-b".to_owned()] diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 61571c051..0d296a9c9 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -41,6 +41,25 @@ struct Cluster { options: RaftRuntimeOptions, } +#[derive(Clone, Copy)] +struct ClusterLogLimits { + retention: CellGroupLogRetentionLimits, + target_file_bytes: u64, + purge_threshold_bytes: u64, + purge_interval_ms: Option, +} + +impl Default for ClusterLogLimits { + fn default() -> Self { + Self { + retention: Default::default(), + target_file_bytes: 16 * 1024 * 1024, + purge_threshold_bytes: 64 * 1024 * 1024, + purge_interval_ms: Some(1000), + } + } +} + impl Cluster { async fn new() -> Result { Self::with_options(RaftRuntimeOptions::default()).await @@ -120,6 +139,30 @@ impl Cluster { store: Arc, minimum: Durability, retention: CellGroupLogRetentionLimits, + ) -> Result { + Box::pin(Self::with_store_log_limits( + options, + root, + filesystem, + schema, + store, + minimum, + ClusterLogLimits { + retention, + ..Default::default() + }, + )) + .await + } + + async fn with_store_log_limits( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + minimum: Durability, + log_limits: ClusterLogLimits, ) -> Result { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { @@ -174,7 +217,10 @@ impl Cluster { ProcessGeneration::new([20; 16])?, LogOpenMode::Create, ); - log.cell_group_retention = retention; + log.cell_group_retention = log_limits.retention; + log.target_file_bytes = log_limits.target_file_bytes; + log.purge_threshold_bytes = log_limits.purge_threshold_bytes; + log.purge_interval_ms = log_limits.purge_interval_ms; log }, Arc::new(fault::FaultFileSystem { gate: gate.clone() }), diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs new file mode 100644 index 000000000..dc4de2f60 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -0,0 +1,652 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical Raft file reclamation, independent archive verification and retained +//! restart. Filesystem observations are assertions, not cache-debt estimates. +use super::*; +use raft::{GetEntriesContext, Storage as _}; +use std::path::PathBuf; +use std::sync::atomic::{AtomicUsize, Ordering}; + +const PREFIX: u64 = 8; +const TOTAL: u64 = 12; + +fn log_limits() -> ClusterLogLimits { + ClusterLogLimits { + target_file_bytes: 4096, + purge_threshold_bytes: 8192, + purge_interval_ms: None, + ..Default::default() + } +} + +fn payload(sequence: u64, cell: usize) -> Vec { + let mut bytes = sequence.to_be_bytes().to_vec(); + for block in 0..512_u64 { + let mut hash = Sha256::new(); + hash.update(sequence.to_be_bytes()); + hash.update((cell as u64).to_be_bytes()); + hash.update(block.to_be_bytes()); + bytes.extend_from_slice(&hash.finalize()); + } + bytes +} + +fn attempt(sequence: u64, cell: usize, input: &[u8]) -> Result { + let mut request = cell_attempt(100 + (sequence as u8) * 2 + cell as u8)?; + request.request.id = CommandId::new(ClientId::new([81; 16])?, sequence)?; + request.request.operation = "Write".into(); + request.request.contract_digest = [83; 32]; + request.request.request_digest = command_request_digest(input); + Ok(request) +} + +fn read_image(pin: &CellImagePin) -> Result> { + Ok(match pin { + CellImagePin::Empty => Vec::new(), + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for n in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(n).context("extent")?)?, + ); + } + bytes + } + }) +} + +fn assert_image(root: &PreparedGroupManifest, name: &CellId, count: u64) -> Result<()> { + let (leaf, pin) = root.pin_cell(name).context("selected SQL cell")?; + assert_eq!(leaf.version.sequence.get(), count + 1); + assert_eq!(leaf.effects.next_alarm_unix_millis, Some(5000)); + let temp = tempfile::tempdir()?; + let file = temp.path().join("oracle.sqlite"); + std::fs::write(&file, read_image(&pin)?)?; + let db = + rusqlite::Connection::open_with_flags(file, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + assert_eq!( + db.query_row("SELECT n FROM value", [], |r| r.get::<_, u64>(0))?, + count + ); + assert_eq!( + db.query_row("SELECT COUNT(*) FROM payload", [], |r| r.get::<_, u64>(0))?, + count + ); + Ok(()) +} + +async fn command( + fixture: &RuntimeCluster, + class: Arc, + cell: &CellHistory, + ordinal: usize, + sequence: u64, +) -> Result { + let group = fixture.cluster.birth.route().history; + let pool = &fixture.appliers[&(0, group)].0; + let input = payload(sequence, ordinal); + let expected = Sha256::digest(&input).to_vec(); + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let CellSqlOutcome::Prepared(prepared) = pool + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(sequence, ordinal, &input)?, + class, + input, + true, + ) + .await? + else { + anyhow::bail!("new registered command must prepare") + }; + pool.mark_submitted(cell.clone(), prepared.proposal()) + .await?; + let accepted = accepted_sql(&fixture.runtimes[0], &prepared).await?; + controller::applied(fixture, group, accepted.position.index).await?; + let authority = fixture.runtimes[0] + .execution_authority(group, cell.clone()) + .await?; + let receipt = pool.finish(authority).await?; + assert_eq!(receipt.receipt().receipt.result(), expected); + assert_eq!(receipt.receipt().version.history, *cell); + assert_eq!(receipt.receipt().version.sequence.get(), sequence + 1); + Ok(accepted.position.index) +} + +#[derive(Clone)] +struct SelectedLog { + marker: GroupManifest, + checkpoint: Option, + entries: Vec, +} + +impl SelectedLog { + fn capture(log: &ReplicaLog, history: GroupHistory) -> Result { + let group = log.cell_group(&history)?; + Ok(Self { + marker: group.group_manifest()?.context("selected manifest")?, + checkpoint: group.checkpoint_manifest()?, + entries: group.entries( + group.first_index()?, + group.last_index()? + 1, + None, + GetEntriesContext::empty(false), + )?, + }) + } + + fn check(&self, log: &ReplicaLog) -> Result<()> { + let actual = Self::capture(log, self.marker.route().history)?; + assert_eq!(actual.marker, self.marker); + assert_eq!(actual.checkpoint, self.checkpoint); + assert_eq!(actual.entries, self.entries); + Ok(()) + } +} + +async fn files(domain: &ExecutionDomain, path: PathBuf) -> Result> { + domain + .blocking(move || { + let mut files = BTreeMap::new(); + for item in std::fs::read_dir(path)? { + let item = item?; + let name = item.file_name().to_string_lossy().into_owned(); + if name.ends_with(".raftlog") || name.ends_with(".rewrite") { + files.insert(name, item.metadata()?.len()); + } + } + Ok(files) + })? + .await? +} + +async fn checkpoint( + fixture: &RuntimeCluster, + pool: &CellGroupCheckpointPool, + marker: &GroupManifest, +) -> Result<()> { + let history = marker.route().history; + let source = fixture.runtimes[0] + .capture_checkpoint(history, pool) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + assert_eq!(prepared.marker(), marker); + fixture.runtimes[0] + .publish_checkpoint(history, prepared) + .await?; + controller::wait_for(async || { + Ok(fixture.cluster.voters[0] + .log + .cell_group(&history)? + .checkpoint_manifest()? + .as_ref() + == Some(marker) + && pool.usage().sources == 0) + }) + .await +} + +async fn recover_primary( + disk: tempfile::TempDir, + directory: RangeDirectory, + birth: CellGroupBirth, + store: Arc, + rooted: bool, + saved: &[Vec], +) -> Result { + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let mut voters = Vec::new(); + let mut appliers = Vec::new(); + for (node, selected) in saved.iter().enumerate() { + let number = node as u8 + 1; + let base = disk.path().join(format!("node-{number}")); + let mut options = ReplicaLogOptions::new( + base.join("log"), + NodeId::new([number; 16])?, + DiskLineage::new([number + 10; 16])?, + ProcessGeneration::new([91; 16])?, + LogOpenMode::Resume, + ); + options.target_file_bytes = log_limits().target_file_bytes; + options.purge_threshold_bytes = log_limits().purge_threshold_bytes; + options.purge_interval_ms = None; + let log = domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + for expected in selected { + expected.check(&log)?; + } + let images = GenerationStore::open( + base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let extents = if rooted { + Some(NodeExtentStore::open( + disk.path().join(format!("rooted-{node}")), + Arc::new(DirectFileSystem), + Default::default(), + )?) + } else { + None + }; + let config = checkpoint_receive::config(images.clone(), extents.clone(), &base)?; + let group = log.cell_group(&birth.route().history)?; + let (applier, root) = domain + .blocking(move || -> Result<_> { + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + Ok(CellGroupApplier::recover( + group, + pages, + config, + Default::default(), + Default::default(), + 100, + )?) + })? + .await??; + let sibling = selected[1].clone(); + let sibling_group = log.cell_group(&sibling.marker.route().history)?; + let sibling_base = disk.path().join(format!("extra-31-node-{node}")); + domain + .blocking(move || -> Result<()> { + let pages = ManifestPageStore::open( + &sibling_base.join("pages"), + Arc::new(DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let images = GenerationStore::open( + sibling_base.join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?; + let (applier, root) = CellGroupApplier::recover( + sibling_group, + pages, + checkpoint_receive::config(images, None, &sibling_base)?, + Default::default(), + Default::default(), + 100, + )?; + assert_eq!(root.marker(), &sibling.marker); + let (_, pin) = root + .pin_cell(&CellId::new("live-sibling")?) + .context("sibling image")?; + assert_eq!( + read_image(&pin)?, + images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(197);" + )? + ); + drop(root); + drop(applier); + Ok(()) + })? + .await??; + let permit = domain + .run(directory.bootstrap_group_member(&birth, &log)) + .await??; + let owner = domain + .run(CellGroupOwner::start( + permit, + root, + RaftRuntimeOptions::default(), + [number + 70; 32], + )) + .await??; + let pool = CellGroupSqlPool::start(&log, sql_limits())?; + appliers.push((pool, applier)); + voters.push(Voter { + log, + owner: Some(owner), + pages: None, + images, + extents, + tree: ManifestTree::new(birth.route().clone(), Default::default())?, + gate: fault::Gate::new(None), + paused: false, + held: None, + }); + } + let history = birth.route().history; + let cluster = Cluster { + _root: disk, + domain, + directory: directory.clone(), + store, + birth, + voters, + network: VecDeque::new(), + reads: Vec::new(), + options: RaftRuntimeOptions::default(), + }; + let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; + for (node, applier) in appliers.into_iter().enumerate() { + fixture.appliers.insert((node, history), applier); + } + Ok(fixture) +} + +fn sql_limits() -> CellSqlLimits { + CellSqlLimits { + workers: 1, + queue_per_worker: 2, + max_pending_cells: 2, + group_pending_cells: 2, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + } +} + +pub(super) async fn physical_log_reclamation( + rooted: bool, + store: Arc, +) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .alarm("Wake", [82; 32], |_, _| Ok(()))? + .command("Write", [83; 32], move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let sequence = u64::from_be_bytes(input[..8].try_into()?); + context.sql().execute_batch( + "CREATE TABLE IF NOT EXISTS payload(seq INTEGER PRIMARY KEY, body BLOB);", + )?; + context.sql().execute("UPDATE value SET n=n+1", [])?; + context.sql().execute( + "INSERT INTO payload VALUES(?1,?2)", + rusqlite::params![sequence, &input[8..]], + )?; + context.set_alarm("Wake", 5000, b"reclaimed-but-retained")?; + Ok(Sha256::digest(input).to_vec()) + })?, + ); + let cluster = Box::pin(Cluster::with_store_log_limits( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + class.schema_digest(), + store, + Durability::Quorum, + log_limits(), + )) + .await?; + let (mut fixture, route, a, b) = Box::pin(registered::setup_cluster( + rooted, + cluster, + Arc::new(DirectFileSystem), + )) + .await?; + let second = controller::add_second(&mut fixture, route.history).await?; + let (archive, remote) = archive_controller::registration(&fixture, &route, rooted).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + fixture.runtimes[0].campaign(second).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; + let sibling_route = fixture.cluster.voters[0] + .log + .cell_group(&second)? + .group_manifest()? + .context("sibling root")? + .route() + .clone(); + let sibling = history(&sibling_route, "live-sibling")?; + let sibling_image = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(197);", + )?; + let accepted = fixture.runtimes[0] + .propose( + second, + apply::enrollment_bytes(&sibling_route, &sibling, &sibling_image, false)?, + ) + .await?; + controller::applied(&fixture, second, accepted.position.index).await?; + controllers[0].archive(archive.clone())?.response().await?; + let mut reader = None; + for sequence in 1..=PREFIX { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, sequence).await?; + } + if sequence == 1 { + reader = Some(fixture.appliers[&(0, route.history)].1.recover_selected()?); + } + } + let reader = reader.context("old physical reader")?; + let (_, pin) = reader.pin_cell(&a.cell).context("old image")?; + let old_bytes = read_image(&pin)?; + drop(pin); + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let cut = group.group_manifest()?.context("covered cut")?; + archive_controller::published(&controllers[0], route.history, cut.position().index).await?; + let coverage = + archive_controller::verified_current(&fixture, &archive, rooted, "reclaim-audit").await?; + assert_eq!(coverage.marker(), &cut); + assert_image(coverage.recovery().manifest(), &a.cell, PREFIX)?; + assert_image(coverage.recovery().manifest(), &b.cell, PREFIX)?; + let pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let log_path = fixture.cluster._root.path().join("node-1/log/log"); + let before = files(&fixture.cluster.domain, log_path.clone()).await?; + assert!(before.len() > 8, "actual file rotation is required"); + checkpoint(&fixture, &pool, &cut).await?; + assert_eq!(group.first_index()?, cut.position().index.get() + 1); + let current = fixture.appliers[&(0, route.history)].1.recover_selected()?; + let mut retained_source = pool.capture_snapshot(&group, ¤t, &group.snapshot(0, 3)?)?; + drop(current); + remote.outage.store(true, Ordering::Release); + for sequence in PREFIX + 1..=TOTAL { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, sequence).await?; + } + } + let saved: Vec<_> = fixture + .cluster + .voters + .iter() + .map(|voter| { + Ok(vec![ + SelectedLog::capture(&voter.log, route.history)?, + SelectedLog::capture(&voter.log, second)?, + ]) + }) + .collect::>()?; + assert_eq!(saved[0][0].entries.len(), ((TOTAL - PREFIX) * 2) as usize); + assert!(!saved[0][1].entries.is_empty()); + let mut after = before.clone(); + for _ in 0..64 { + fixture.cluster.voters[0].log.purge().await?; + for expected in &saved[0] { + expected.check(&fixture.cluster.voters[0].log)?; + } + let (_, pin) = reader.pin_cell(&a.cell).context("retained old reader")?; + assert_eq!(read_image(&pin)?, old_bytes); + after = files(&fixture.cluster.domain, log_path.clone()).await?; + if before.keys().any(|name| !after.contains_key(name)) + && after.values().sum::() < before.values().sum::() + { + break; + } + } + let removed: Vec<_> = before + .keys() + .filter(|name| !after.contains_key(*name)) + .cloned() + .collect(); + assert!( + !removed.is_empty(), + "purge must remove actual pre-checkpoint files" + ); + assert!( + after.values().sum::() < before.values().sum::(), + "actual log-file length must decrease" + ); + let receiver_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let mut receiver = receiver_pool.receive( + fixture.appliers[&(2, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?; + let received = fixture + .cluster + .domain + .blocking(move || { + retained_source.stream(|chunk| receiver.append(chunk))?; + receiver.finish() + })? + .await??; + assert_eq!(received.marker(), &cut); + assert_eq!(pool.usage().sources, 0); + assert_eq!(receiver_pool.usage().sources, 1); + drop(received); + assert_eq!(receiver_pool.usage().sources, 0); + assert_eq!(pool.usage().sources, 0); + remote.outage.store(false, Ordering::Release); + archive_controller::published( + &controllers[0], + route.history, + saved[0][0].marker.position().index, + ) + .await?; + for controller in controllers { + controller.close().await?; + } + // Quiesce all node-zero SQL owners, then reconcile the complete two-group + // image inventory. Old readers and the selected checkpoint still pin their + // closures while obsolete image/page files can be physically collected. + let inventory: Vec<_> = fixture + .appliers + .iter() + .filter(|((node, _), _)| *node == 0) + .map(|(_, (_, applier))| applier.clone()) + .collect(); + fixture.appliers.retain(|(node, _), _| *node != 0); + let mut image_files = 0; + let mut page_files = 0; + for _ in 0..32 { + let report = fixture.cluster.voters[0] + .log + .reclaim_cell_storage( + inventory.clone(), + vec![], + CellGroupReclaimLimits { + max_selected_cells: 64, + max_files: 1, + max_page_scan: 64, + }, + ) + .await?; + assert_eq!(report.groups, 2); + assert!(report.image_files + report.page_files <= 1); + image_files += report.image_files; + page_files += report.page_files; + let (_, pin) = reader + .pin_cell(&a.cell) + .context("reader during collection")?; + assert_eq!(read_image(&pin)?, old_bytes); + } + assert!(image_files + page_files > 0, "actual orphan collection"); + let birth = fixture.cluster.birth.clone(); + let directory = fixture.cluster.directory.clone(); + let store = fixture.cluster.store.clone(); + let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + drop(reader); + drop(group); + drop(coverage); + drop(archive); + drop(inventory); + fixture.close().await?; + let mut fixture = Box::pin(recover_primary( + disk, directory, birth, store, rooted, &saved, + )) + .await?; + for node in 0..3 { + let root = fixture.appliers[&(node, route.history)] + .1 + .recover_selected()?; + assert_image(&root, &a.cell, TOTAL)?; + assert_image(&root, &b.cell, TOTAL)?; + } + fixture.runtimes[0].campaign(route.history).await?; + fixture.drive(None).await?; + for sequence in 1..=TOTAL { + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + let input = payload(sequence, ordinal); + let authority = fixture.runtimes[0] + .execution_authority(route.history, cell.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = fixture.appliers[&(0, route.history)] + .0 + .stage_registered( + fixture.cell_options(cell).await?, + authority, + attempt(sequence, ordinal, &input)?, + class.clone(), + input.clone(), + false, + ) + .await? + else { + anyhow::bail!("retained receipt") + }; + assert_eq!( + receipt.receipt().receipt.result(), + Sha256::digest(input).to_vec() + ); + } + } + assert_eq!(calls.load(Ordering::SeqCst), TOTAL as usize * 2); + let source = fixture.runtimes[0] + .capture_checkpoint(route.history, &pool) + .await?; + let prepared = fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let mut write = GroupWrite::new(&group); + write.snapshot = Some(SnapshotChange::CellGroupCheckpoint(prepared)); + assert!( + matches!( + fixture.cluster.voters[0] + .log + .persist_cells(cellar_logic::consensus::OperationId::new(912), vec![write]) + .await, + Err(LogError::Overloaded) + ), + "reopen must not trust the lost archive frontier" + ); + drop(group); + println!( + "cell_group_log_reclamation_observations {}", + serde_json::json!({ + "mode": if rooted { "extents" } else { "materialized" }, + "files_before": before.len(), "files_after": after.len(), "removed": removed, + "bytes_before": before.values().sum::(), "bytes_after": after.values().sum::(), + "checkpoint": cut.position().index.get(), "retained_suffix_entries": saved[0][0].entries.len(), + "receipts_retried_without_callback": TOTAL * 2, "voters_reopened": 3, "groups_per_engine": 2, + "obsolete_image_files_removed": image_files, "obsolete_page_files_removed": page_files + }) + ); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 2050fbf70..aa01d8d00 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -926,3 +926,10 @@ pub(super) use sql::{seed_retry, shared_inventory}; pub(super) async fn retained_log_admission(rooted: bool) -> Result<()> { Box::pin(sql::retained_log_admission(rooted)).await } + +pub(super) async fn physical_log_reclamation( + rooted: bool, + store: Arc, +) -> Result<()> { + Box::pin(sql::physical_log_reclamation(rooted, store)).await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c13854f8f..c0bf5baf3 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -671,3 +671,7 @@ pub(crate) use seed_crash::seed_retry; #[path = "cell_group_log_retention.rs"] mod log_retention; pub(super) use log_retention::retained_log_admission; + +#[path = "cell_group_log_reclaim.rs"] +mod log_reclaim; +pub(super) use log_reclaim::physical_log_reclamation; diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index e9bfe6b17..76e7c6d91 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -207,7 +207,7 @@ pub(super) async fn setup_store_minimum( setup_cluster(rooted, cluster, filesystem).await } -async fn setup_cluster( +pub(super) async fn setup_cluster( rooted: bool, cluster: Cluster, filesystem: Arc, diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 532a8976a..3fb7241d1 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -25,6 +25,7 @@ ROOT = Path(__file__).resolve().parents[2] TARGET = "native_cell_groups_archive_controller" CASES = ( + "rustfs_group_reclaim_restart_keeps_history", "rustfs_archive_coverage_gates_local_compaction", "rustfs_s3_remote_outputs_preserve_durability_and_creation", "rustfs_s3_remote_deadlines_preserve_ambiguity_and_callbacks", From 2d9be1b52d39223f9ec71d85b42fb3c16276c004 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:06:51 -0700 Subject: [PATCH 227/568] test: cover serving after retained log recovery Expose the fixture to its enclosing module and require fresh TLS writes after receipt recovery. Preserve the original physical reclamation assertions and limits. --- .../tests/support/cell_group_log_reclaim.rs | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index dc4de2f60..cb1c7683f 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -358,7 +358,7 @@ fn sql_limits() -> CellSqlLimits { } } -pub(super) async fn physical_log_reclamation( +pub(in super::super) async fn physical_log_reclamation( rooted: bool, store: Arc, ) -> Result<()> { @@ -637,6 +637,21 @@ pub(super) async fn physical_log_reclamation( "reopen must not trust the lost archive frontier" ); drop(group); + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + for (ordinal, cell) in [&a, &b].into_iter().enumerate() { + command(&fixture, class.clone(), cell, ordinal, TOTAL + 1).await?; + } + assert_eq!(calls.load(Ordering::SeqCst), (TOTAL + 1) as usize * 2); + for node in 0..3 { + let root = fixture.appliers[&(node, route.history)] + .1 + .recover_selected()?; + assert_image(&root, &a.cell, TOTAL + 1)?; + assert_image(&root, &b.cell, TOTAL + 1)?; + } + for controller in controllers { + controller.close().await?; + } println!( "cell_group_log_reclamation_observations {}", serde_json::json!({ @@ -644,8 +659,9 @@ pub(super) async fn physical_log_reclamation( "files_before": before.len(), "files_after": after.len(), "removed": removed, "bytes_before": before.values().sum::(), "bytes_after": after.values().sum::(), "checkpoint": cut.position().index.get(), "retained_suffix_entries": saved[0][0].entries.len(), - "receipts_retried_without_callback": TOTAL * 2, "voters_reopened": 3, "groups_per_engine": 2, - "obsolete_image_files_removed": image_files, "obsolete_page_files_removed": page_files + "receipts_retried_without_callback": TOTAL * 2, "voters_reopened": 3, "groups_per_engine": 2, + "obsolete_image_files_removed": image_files, "obsolete_page_files_removed": page_files, + "new_commands_after_restart": 2 }) ); fixture.close().await From fae2f9e341e490de841eed070c68b9080c0b52a9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:12:49 -0700 Subject: [PATCH 228/568] test: read retained images in bounded chunks --- .../cellar/tests/support/cell_group_log_reclaim.rs | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index cb1c7683f..f514c1565 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -43,9 +43,15 @@ fn attempt(sequence: u64, cell: usize, input: &[u8]) -> Result { fn read_image(pin: &CellImagePin) -> Result> { Ok(match pin { CellImagePin::Empty => Vec::new(), - CellImagePin::Materialized(pin) => pin - .open_reader()? - .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut bytes = Vec::new(); + while (bytes.len() as u64) < reader.length() { + let count = (reader.length() - bytes.len() as u64).min(64 * 1024) as usize; + bytes.extend(reader.read_exact_at(bytes.len() as u64, count)?); + } + bytes + } CellImagePin::Extents(pin) => { let mut bytes = Vec::new(); for n in 0..pin.manifest().geometry().extent_count() { From 51a846be978c296c491509a997e8a91703b0f838 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:20:57 -0700 Subject: [PATCH 229/568] test: preserve live checkpoint rollback refusal --- .../support/cell_group_controller_archive.rs | 2 +- .../tests/support/cell_group_log_reclaim.rs | 35 +++++++++++++++++-- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 797a7eeea..02737269e 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -11,7 +11,7 @@ use std::sync::atomic::Ordering; #[allow(dead_code)] pub(super) mod archive_store; -async fn verifier( +pub(super) async fn verifier( fixture: &RuntimeCluster, route: &GroupRoute, rooted: bool, diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index f514c1565..b279737ca 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -507,8 +507,10 @@ pub(in super::super) async fn physical_log_reclamation( "actual log-file length must decrease" ); let receiver_pool = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; - let mut receiver = receiver_pool.receive( - fixture.appliers[&(2, route.history)].1.clone(), + let destination = + archive_controller::verifier(&fixture, &route, rooted, "post-purge-snapshot").await?; + let mut receiver = receiver_pool.receive_archive( + destination, checkpoint_receive::scratch()?, fixture.cluster._root.path().into(), )?; @@ -523,9 +525,38 @@ pub(in super::super) async fn physical_log_reclamation( assert_eq!(received.marker(), &cut); assert_eq!(pool.usage().sources, 0); assert_eq!(receiver_pool.usage().sources, 1); + assert_image(received.manifest(), &a.cell, PREFIX)?; + assert_image(received.manifest(), &b.cell, PREFIX)?; drop(received); assert_eq!(receiver_pool.usage().sources, 0); assert_eq!(pool.usage().sources, 0); + // A retained checkpoint is data for lagging/recovering destinations, never + // permission to roll back an already-ahead live voter. Exercise its original + // cut guard with a separate one-shot source, without enlarging either pool. + assert!(saved[2][0].marker.position().index > cut.position().index); + let current = fixture.appliers[&(0, route.history)].1.recover_selected()?; + let mut source = pool.capture_snapshot(&group, ¤t, &group.snapshot(0, 3)?)?; + drop(current); + let mut live = receiver_pool.receive( + fixture.appliers[&(2, route.history)].1.clone(), + checkpoint_receive::scratch()?, + fixture.cluster._root.path().into(), + )?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(matches!( + source.stream(|chunk| live.append(chunk)), + Err(CellGroupCheckpointError::Invalid( + "receiver history/range/cut" + )) + )); + Ok(()) + })? + .await??; + assert_eq!(pool.usage().sources, 0); + assert_eq!(receiver_pool.usage().sources, 0); remote.outage.store(false, Ordering::Release); archive_controller::published( &controllers[0], From 58d6559956eb7d37082a7bf066a21cda0df70945 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:28:38 -0700 Subject: [PATCH 230/568] test: release completed archive verification --- .../cellar/tests/support/cell_group_log_reclaim.rs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index b279737ca..92751d229 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -557,13 +557,24 @@ pub(in super::super) async fn physical_log_reclamation( .await??; assert_eq!(pool.usage().sources, 0); assert_eq!(receiver_pool.usage().sources, 0); + // Independent verification owns the archive pool's sole receiver slot. + // Its checks and post-purge use are complete; the journal still owns the + // remote hold. Release this physical result before waiting for a new cut. + assert_eq!(archive.checkpoints.usage().sources, 1); + drop(coverage); remote.outage.store(false, Ordering::Release); archive_controller::published( &controllers[0], route.history, saved[0][0].marker.position().index, ) - .await?; + .await + .with_context(|| { + format!( + "post-purge archive catch-up (checkpoint slots {})", + archive.checkpoints.usage().sources + ) + })?; for controller in controllers { controller.close().await?; } @@ -608,7 +619,6 @@ pub(in super::super) async fn physical_log_reclamation( let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); drop(reader); drop(group); - drop(coverage); drop(archive); drop(inventory); fixture.close().await?; From 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:35:06 -0700 Subject: [PATCH 231/568] fix: parse interleaved reclamation observations --- tools/multi-raft/cell_groups.py | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 3fb7241d1..6fed942ed 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -46,6 +46,18 @@ ) +def exact_case_passed(output, case): + """Keep exact selection strict when validated observations interrupt libtest.""" + pattern = r"cell_group_log_reclamation_observations (\{[^\n]+\})\n" + observations = [json.loads(value) for value in re.findall(pattern, output)] + if observations: + if [value.get("mode") for value in observations] != ["materialized", "extents"]: + raise ValueError("incomplete or duplicated reclamation observations") + output = re.sub(pattern, "", output) + return bool(re.search(rf"^test {re.escape(case)} \.\.\. ok$", output, re.M) + and "test result: ok. 1 passed; 0 failed; 0 ignored;" in output) + + def sha(path): digest = hashlib.sha256() with path.open("rb") as source: @@ -171,8 +183,7 @@ def save(): for case in CASES: if not re.search(rf"^{re.escape(case)}: test$", output, re.M): raise RuntimeError("missing exact RustFS test in executable inventory") - elif not (re.search(rf"^test {re.escape(selection)} \.\.\. ok$", output, re.M) - and "test result: ok. 1 passed; 0 failed; 0 ignored;" in output): + elif not exact_case_passed(output, selection): raise RuntimeError("exact RustFS test was not executed successfully") match = re.search(r"Running [^\n]*\(([^\n)]+/native_cell_groups_archive_controller-[a-f0-9]+)\)", output) if not match: From ea2717b62457f5038fedd6bf4dc54c22f464af16 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:41:15 -0700 Subject: [PATCH 232/568] test: use project reclamation diagnostics --- crates/cellar/tests/support/cell_group_log_reclaim.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index 92751d229..025803d24 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -699,7 +699,7 @@ pub(in super::super) async fn physical_log_reclamation( for controller in controllers { controller.close().await?; } - println!( + cellar::note!( "cell_group_log_reclamation_observations {}", serde_json::json!({ "mode": if rooted { "extents" } else { "materialized" }, From 37f3def8e06c4c05ff1bd605d77db0062feaa3bb Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 04:49:56 -0700 Subject: [PATCH 233/568] test: share archive store fixture modules --- crates/cellar/tests/native_cell_groups_archive_controller.rs | 3 +++ crates/cellar/tests/native_cell_groups_execution.rs | 3 +++ crates/cellar/tests/native_cell_groups_snapshot.rs | 3 +++ crates/cellar/tests/support/cell_group_controller_archive.rs | 4 +--- 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index efa54d5b6..33de9d385 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -3,6 +3,9 @@ //! Controller-owned archive publication; no fixture event pumping after startup. #![allow(clippy::disallowed_methods)] +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; #[path = "support/cell_group_checkpoint.rs"] #[allow(dead_code)] mod checkpoint_fixture; diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index bfc6f0175..fd04a00d7 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -4,6 +4,9 @@ //! Selected fixed-pool and consensus boundaries; not SDK or snapshot qualification. #![allow(clippy::disallowed_methods)] +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; #[path = "support/raft_storage_fault.rs"] #[allow(dead_code)] mod fault; diff --git a/crates/cellar/tests/native_cell_groups_snapshot.rs b/crates/cellar/tests/native_cell_groups_snapshot.rs index 15bf7aadf..5b581a73a 100644 --- a/crates/cellar/tests/native_cell_groups_snapshot.rs +++ b/crates/cellar/tests/native_cell_groups_snapshot.rs @@ -4,6 +4,9 @@ //! as native_cell_groups_execution. Each named case retains direct fixture oracles. #![allow(clippy::disallowed_methods)] +#[path = "support/cell_group_archive_store.rs"] +#[allow(dead_code)] +mod archive_store; #[path = "support/raft_storage_fault.rs"] #[allow(dead_code)] mod fault; diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index 02737269e..cd9924f89 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -7,9 +7,7 @@ use futures_util::TryStreamExt; use object_store::ObjectStore; use std::sync::atomic::Ordering; -#[path = "cell_group_archive_store.rs"] -#[allow(dead_code)] -pub(super) mod archive_store; +pub(super) use crate::archive_store; pub(super) async fn verifier( fixture: &RuntimeCluster, From ea2f802475aa849a1b00f0990eff6929beb4f207 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 05:01:26 -0700 Subject: [PATCH 234/568] test: scope reclamation to archive harness --- .../tests/support/cell_group_controller_archive.rs | 4 ++++ crates/cellar/tests/support/cell_group_log_reclaim.rs | 11 +++++++++-- crates/cellar/tests/support/cell_group_runtime.rs | 8 ++------ crates/cellar/tests/support/cell_group_sql.rs | 5 ++--- 4 files changed, 17 insertions(+), 11 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index cd9924f89..b1dc65702 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -9,6 +9,10 @@ use std::sync::atomic::Ordering; pub(super) use crate::archive_store; +#[path = "cell_group_log_reclaim.rs"] +mod log_reclaim; +pub(crate) use log_reclaim::physical_log_reclamation; + pub(super) async fn verifier( fixture: &RuntimeCluster, route: &GroupRoute, diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index 025803d24..403d993a4 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -364,10 +364,17 @@ fn sql_limits() -> CellSqlLimits { } } -pub(in super::super) async fn physical_log_reclamation( +// Construct the large scenario outside its caller's async poll frame, as the +// snapshot acceptance harness does for its composite physical scenarios. +#[inline(never)] +pub(crate) fn physical_log_reclamation( rooted: bool, store: Arc, -) -> Result<()> { +) -> std::pin::Pin>>> { + Box::pin(run(rooted, store)) +} + +async fn run(rooted: bool, store: Arc) -> Result<()> { let calls = Arc::new(AtomicUsize::new(0)); let observed = calls.clone(); let class = Arc::new( diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index aa01d8d00..5fb684c7d 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -927,9 +927,5 @@ pub(super) async fn retained_log_admission(rooted: bool) -> Result<()> { Box::pin(sql::retained_log_admission(rooted)).await } -pub(super) async fn physical_log_reclamation( - rooted: bool, - store: Arc, -) -> Result<()> { - Box::pin(sql::physical_log_reclamation(rooted, store)).await -} +#[allow(unused_imports)] +pub(super) use sql::physical_log_reclamation; diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c0bf5baf3..66681332a 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -672,6 +672,5 @@ pub(crate) use seed_crash::seed_retry; mod log_retention; pub(super) use log_retention::retained_log_admission; -#[path = "cell_group_log_reclaim.rs"] -mod log_reclaim; -pub(super) use log_reclaim::physical_log_reclamation; +#[allow(unused_imports)] +pub(super) use archive_controller::physical_log_reclamation; From 7741e4049704f3e1c6519b736e62deaf263c39a2 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 05:03:50 -0700 Subject: [PATCH 235/568] test: expose archive scenario to target --- crates/cellar/tests/support/cell_group_sql.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 66681332a..30bcb30bb 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -673,4 +673,4 @@ mod log_retention; pub(super) use log_retention::retained_log_admission; #[allow(unused_imports)] -pub(super) use archive_controller::physical_log_reclamation; +pub(crate) use archive_controller::physical_log_reclamation; From 8ffc843a8e34c61fca3e15851d75f78ed39dc965 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 05:24:39 -0700 Subject: [PATCH 236/568] docs: qualify physical group log reclamation --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 104 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 94 ++++++++++++++++ 3 files changed, 151 insertions(+), 53 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index ad7153860..e40e0efc6 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "ad73f03f278f5845b311fe929d184cc947382b2d", - "runtime_revision": "ad73f03f278f5845b311fe929d184cc947382b2d", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged.", + "source_revision": "7741e4049704f3e1c6519b736e62deaf263c39a2", + "runtime_revision": "7741e4049704f3e1c6519b736e62deaf263c39a2", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 0050607bc..9a552ba31 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -16,56 +16,60 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). The latest runtime/qualification source is -`ad73f03f278f5845b311fe929d184cc947382b2d`. CG-06 remains IN_PROGRESS. Logical -retained-log bytes/entries and reservations now share explicit node/group -ceilings. Restart reconstructs selected debt; smaller ceilings preserve history -while refusing new work. New callback/initializer envelopes reserve before -execution, known receipts still resolve at capacity, and exact leases survive -uncertain submission and entered physical work. Durable selection transfers the -charge atomically into retained debt. Checkpoints and suffix replacement release -only actually retired entries. These limits do not bound physical engine files. - -All shared Raft append paths use the ledger, including inherited entries and -election/configuration controls. Joint changes reserve their future automatic -leave before append; the native scenario fills ordinary admission and verifies -that leave uses the single progress slot. Role revocation precedes bookkeeping. -Selected checkpoint identity and released physical source quota, rather than -leader-only readiness, gate recapture on each voter. - -The prior archive activation and checkpoint-drain contract remains required: -pending activation closes later checkpoint admission, earlier exact cut-bound -owner/storage leases drain, entered metadata work owns activation through waiter -loss, and sync/cache publication makes policy irreversible. Archive-configured -local compaction requires independently verified complete coverage and fresh -verification after reopen. Public archive status grants no compaction, output or -serving authority. Marker-1 serialization and original task oracles remain -unchanged. - -Local/remote command, enrollment and receipt S3 responses retain their exact -archive owners and independently verified cuts through physical output. Original -durability minima/deadlines, fresh catalog/leader checks, authenticated achieved -durability and creation provenance remain required. Unknown outcomes never -replay callbacks. Public SDK/fleet cutover still has to integrate these -contracts. - -Three exact storage cases and both native image scenarios pass. Seventeen real RustFS controller cases pass in both image forms with seven signed probes. Twenty-two regression/check commands pass with 267 distinct Rust cases across seven copied executables, including all 22 storage cases. Final-source all-target Clippy, format/layout, dependency/docs/preflight/plan checks, 37 Python tests and independent source/hash/selection/credential audit pass. Evidence binds 2574 sources under `.cellar/cell-groups-evidence/CG-06-log-retention/`; see [progress](progress.md) for precise scope and retained failures. The earlier `8dfe9ff3` slow-reader timeout remains unexplained. This matrix does not establish sustained load or delete all voter disks. - -Next resolve full S3-outage catalog/epoch authority without weakening fencing, -then physical shared-engine file reclamation across every -checkpoint/archive/reader/transfer/sibling hold. Logical debt admission and -wrapper-fault tests are insufficient for a physical disk bound. Fenced -fresh-incarnation serving must inventory every possible old voter, including -joins after the archived cut. The accepted fencing-authority API plus Docker -qualification remains the target. Peer-install deployment policy, process-kill -and physical-read/output shutdown cuts still need qualification. - -CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover. -Topology, placement, hibernation, managed recovery, measured load and final -legacy removal follow. Four original CG-06 cases/full gate remain pending. -Voters are host-native tasks and RustFS one VM; independent failure domains and -the full hard cutover remain unqualified. The full end-to-end goal remains -active. Preserve predecessor runtime `ab0abf26`, archive-compaction runtime -`8dfe9ff3`, all source-bound failures and earlier evidence. +`7741e4049704f3e1c6519b736e62deaf263c39a2`. CG-06 remains IN_PROGRESS. Its +original physical-reclamation case now proves actual Raft-file removal and +file-length reduction across two shared histories, with an archive-covered +checkpoint, retained suffix, uncheckpointed sibling, old physical reader and +post-purge checkpoint streaming. Bounded complete-inventory image/page +maintenance runs after service quiescence. All three retained voter disks +reopen, verify cells/effects/checkpoints/tails and establish fresh quorum. Every +old receipt resolves without callback replay; two new commands commit over TLS. +Reopen restores archive policy without trusting the lost coverage frontier. The +new scenario alone uses 4 KiB files/8KiB purge threshold/manual purge. Existing +defaults, quotas, production code and marker-1 records are unchanged. + +The preceding logical retained-log admission, owned archive activation, exact physical lease transfer and S3-output lifetime contracts remain required. Node/group capacity is reserved before new callbacks; known receipts remain available. Unknown outcomes and entered physical work retain capacity until authoritative resolution or durable selection. Election/configuration controls retain separately bounded progress. S3 outputs still own complete independent coverage through physical delivery; missing receipts remain ambiguous. See [progress](progress.md) and [protocols](protocols.md) for retained contracts and precise qualification scope. + +Eighteen exact real RustFS controller cases pass in both image forms with seven +signed probes. Twenty-two regression/check commands pass with 268 Rust cases +across seven copied executables, including all 22 storage cases. Locked +final-source all-target Clippy, format/layout, dependency/docs/preflight/plan +checks, 37 Python tests, system-Python hashing and independent +source/artifact/selection/credential audit pass across 2575 sources. Evidence is +under `.cellar/cell-groups-evidence/CG-06-physical-log-reclaim/`. Original +helper-visibility and bounded-reader failures and copied-binary reproduction +remain retained. The earlier 8dfe9ff3 slow-reader timing cause remains +unresolved. Voters are host-native tasks and RustFS one VM. + +Two original CG-06 cases pass: archive tails and physical reclaim/restart. Three +original cases/full task gate remain pending: archive dependencies, full-backend +S3 outage and fenced fresh group/cell incarnations serving from a replacement +quorum. This retained-disk case does not erase all voters, qualify process-kill +recovery, establish a global physical disk bound or measure throughput. Fenced +recovery must inventory every possible old voter, including joins after the +selected cut, using the accepted fencing-authority API plus Docker +qualification. + +Next complete the original archive-dependency acknowledgement case, then qualify +both local cached and remote fresh-discovery paths against a fully stopped +backend. Preserve catalog and epoch fencing; source sealing and activation must +be implemented before any new cache-based authority shortcut. +Seal/Install/Activate codecs exist, but live apply rejects those controls and +manifests lack persistent sealed/active lifecycle. Remote command routing +requests fresh catalog confirmation; local known-route cache paths differ and +must be exercised independently against an actually stopped backend. Durable +cancellation and seal authorization must compete on one CAS decision, sources +must close admission in committed order, and children must stay non-serving +until committed install/publication-bound activation. See private +`outage-authority-investigation.md` in this evidence directory for inspected +seams; no source-seal implementation is claimed. + +CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover, +followed by transitions, topology, placement, hibernation, managed recovery, +measured load and final legacy removal. Independent failure domains and the full +hard cutover remain unqualified. Preserve predecessor runtime ad73f03f, all +prior failure records/qualifications and the original task oracles. The full +end-to-end goal remains active. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index c1b3651da..c3c2ae19a 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,100 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 physical log reclamation and retained-disk recovery + +Source: `7741e4049704f3e1c6519b736e62deaf263c39a2`. CG-06 remains IN_PROGRESS. +The original `group_reclaim_restart_keeps_history` case now runs in both image +forms, together with an exact real RustFS counterpart. It uses the production +purge, checkpoint, image/page maintenance, SQL and recovery APIs. Production +behavior and marker-1 records are unchanged. The new scenario explicitly selects +4 KiB target log files, an 8 KiB purge threshold and manual purge so filesystem +observations cannot race automatic maintenance; all prior fixture defaults and +quotas remain unchanged. + +Two real groups share each engine. The primary commits 24 registered commands +across two SQLite cells, retaining exact results and alarm state. Independently +verified complete S3 coverage permits its exact checkpoint. Eight subsequent +entries remain in the local suffix during archive lag. The sibling retains its +entire uncheckpointed history. Purge must remove actual original Raft files and +reduce their aggregate file length, while both histories remain byte-for-byte +intact and an old image reader remains readable. A retained checkpoint streams +into an independent private receiver after purge; an ahead live voter still +rejects rollback. Quiesced node services then perform bounded image/page +maintenance over the complete two-group inventory, preserving reader, current +and checkpoint closures. + +All three voter disks close and reopen under new process generations, preserving +their group/cell identities. Recovery verifies every selected image, effect +summary, checkpoint and log suffix, including the sibling. Fresh quorum resolves +all 24 exact receipts without callback replay, refuses local compaction without +a newly verified archive frontier and commits two new commands over TLS. This is +retained-disk restart, not deletion of all voter disks or fresh-incarnation +disaster recovery. No cached status grants archive or serving authority. + +Real RustFS observations from this controlled test geometry: + +| Image mode | Raft file bytes before → after purge | Original files removed | Retained suffix entries | +| --- | ---: | ---: | ---: | +| materialized | 474445 → 256891 | 42 | 8 | +| extents | 474469 → 256817 | 42 | 8 | + +These measure actual file lengths at the purge boundary, not allocated +filesystem blocks, a global disk bound, throughput or production sizing. +Reader/transition holds and sustained workloads need their broader gates. + +Eighteen exact RustFS controller cases pass in both image forms with seven +signed S3 protocol probes. Twenty-two regression/check commands pass with 268 +distinct Rust cases across seven copied executables, including all four native +shared-group suites and 22 storage cases. Final-source locked all-target +Cellar/logic/LTX Clippy, format/layout, dependency/docs/preflight/plan checks, +37 Python tests and system-Python hashing pass. Source/artifact/selection and +credential-exclusion audit binds 2575 source inputs, copied +outputs/binaries/orchestration and frozen plan inputs under +`.cellar/cell-groups-evidence/CG-06-physical-log-reclaim/`. + +Source-bound failures remain retained. The initial helper visibility failure did +not run a case. A copied binary reproduces the subsequent oracle-reader failure: +`GenerationReader` refuses reads above 64 KiB. The helper now reads every byte +in bounded chunks without raising the reader limit, shrinking payloads or +dropping retained owners. A second copied-binary reproduction confirms that an +ahead live voter must reject the older retained cut. Positive data verification +now uses the existing private archive receiver, while an explicit negative +transfer preserves the live rollback guard and one-slot pool limits. A further +copied-binary timeout exposes the test retaining the archive pool's sole +verification slot past its completed checks; it now drops that result before +awaiting a new cut, while the durable journal retains the remote hold. The quota +and deadline remain unchanged. The first successful Rust run emitted +observations inside the libtest progress line; its wrapper incorrectly refused +selection. Raw output and the original record stay unchanged. Validated +observation normalization now preserves exact matching and the zero-failure +summary; five checks reject wrong, failed, empty and incomplete selections. +Final-source Clippy also requires the project diagnostic macro instead of std +println; the structured observations use that existing API without suppressing +the lint. The archive-store fixture is declared once per integration target, and +the archive-only scenario lives under the owning archive harness with its +existing target-specific exports. Source-bound duplicate-module, unused-helper +and reexport-visibility compile/lint failures remain retained; no production +lint setting or test assertion changed. Original assertions, task cases and +acceptance oracles remain intact. Earlier qualifications and the unexplained +`8dfe9ff3` slow-reader timing limitation remain retained. + +Reflection: two original CG-06 cases now pass: archive tails and physical +reclaim/restart. Complete archive-dependency acknowledgement, full S3-backend +outage authority and fenced fresh-incarnation serving remain required, together +with the full task gate. Investigation confirms that Seal/Install/Activate wire +entries exist but live apply still rejects them and manifests lack the durable +transition lifecycle. Before introducing a cache-based outage shortcut, +cancellation/authorization must share one durable decision, sources must seal, +and children must install and activate under an exact publication. Existing +local cached and remote fresh-discovery paths must first be qualified separately +against a stopped backend; the next independent deliverable is the original +complete-group dependency acknowledgement gate. The independent reclamation case +does not waive this dependency. Finish CG-08 effect/cache/result lifetimes +before CG-07 public SDK/fleet cutover, then source transitions, topology, +placement, hibernation, managed recovery, measured load and final legacy +removal. The full goal remains active and incomplete. + ## CG-06 retained-log admission and progress capacity Source: `ad73f03f278f5845b311fe929d184cc947382b2d`. CG-06 remains IN_PROGRESS. From dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 05:42:35 -0700 Subject: [PATCH 237/568] test: enforce complete group S3 dependencies --- .../tests/native_cell_groups_archive.rs | 16 +- .../native_cell_groups_archive_controller.rs | 13 + .../cell_group_archive_dependencies.rs | 507 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 13 +- .../tests/support/cell_group_s3_commands.rs | 4 + crates/cellar/tests/support/cell_group_sql.rs | 7 +- tools/multi-raft/cell_groups.py | 1 + 7 files changed, 550 insertions(+), 11 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_archive_dependencies.rs diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index c7ebc9b25..cb874a1b7 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! CG-06 checkpoint and committed-tail data recovery. Archive-head protection, -//! S3 settlement, reclamation and fresh-incarnation recovery remain separate. +//! CG-06 complete recovery, S3 dependencies and physical log reclamation. +//! Fresh-incarnation disaster serving remains a separate required gate. #![allow(clippy::disallowed_methods)] // The shared engine fixture also defines fault/membership helpers selected by // sibling integration targets and are not all selected by this archive gate. @@ -33,6 +33,18 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_archive_requires_all_dependencies() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::archive_dependencies( + rooted, + Arc::new(object_store::memory::InMemory::new()), + ) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_reclaim_restart_keeps_history() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index 33de9d385..dc6bc9541 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -23,6 +23,19 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_group_archive_requires_all_dependencies() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::archive_dependencies( + rooted, + rustfs::store("archive-dependencies", rooted)?, + ) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] #[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] async fn rustfs_group_reclaim_restart_keeps_history() -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_archive_dependencies.rs b/crates/cellar/tests/support/cell_group_archive_dependencies.rs new file mode 100644 index 000000000..c1931ff10 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_archive_dependencies.rs @@ -0,0 +1,507 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete-group coverage must include B even when only A is awaiting S3. +use super::*; + +struct Base { + h: Harness, + registry: CellGroupArchiveRegistry, + hold: CellGroupArchiveHold, + dependency: ObjectPath, + bytes: Vec, +} + +fn image_value(pin: CellImagePin, expected: u64) -> Result<()> { + let bytes = match pin { + CellImagePin::Empty => anyhow::bail!("expected a physical SQLite image"), + CellImagePin::Materialized(pin) => { + let mut reader = pin.open_reader()?; + let mut bytes = Vec::new(); + while (bytes.len() as u64) < reader.length() { + let count = (reader.length() - bytes.len() as u64).min(64 * 1024) as usize; + bytes.extend(reader.read_exact_at(bytes.len() as u64, count)?); + } + bytes + } + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for ordinal in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(ordinal).context("SQLite extent")?)?, + ); + } + bytes + } + }; + let root = tempfile::tempdir()?; + let path = root.path().join("value.sqlite"); + std::fs::write(&path, bytes)?; + let db = + rusqlite::Connection::open_with_flags(path, rusqlite::OpenFlags::SQLITE_OPEN_READ_ONLY)?; + assert_eq!( + db.query_row("SELECT n FROM value", [], |r| r.get::<_, u64>(0))?, + expected + ); + Ok(()) +} + +async fn archive_values(h: &Harness, coverage: &CellGroupArchiveCoverage) -> Result<()> { + let root = coverage.recovery().manifest().clone(); + h.fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert_eq!(root.pinned_images(), 2); + for (name, expected) in [("registered-a", 0), ("registered-b", 1)] { + let (leaf, pin) = root + .pin_cell(&CellId::new(name)?) + .context("archived cell")?; + assert_eq!(leaf.version.sequence.get(), expected + 1); + image_value(pin, expected)?; + } + Ok(()) + })? + .await? +} + +async fn local_values(h: &Harness, index: LogIndex) -> Result<()> { + controller::applied(&h.fixture, h.route.history, index).await?; + let b = history(&h.route, "registered-b")?; + for runtime in &h.fixture.runtimes { + for (cell, value) in [(&h.cell, 2), (&b, 1)] { + let selected = runtime + .reconciliation_authority(h.route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), value + 1); + let image = selected.image().clone(); + h.fixture + .cluster + .domain + .blocking(move || image_value(image, value))? + .await??; + } + } + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + assert!(h.controllers[0].group_failure(h.route.history)?.is_none()); + Ok(()) +} + +async fn setup(rooted: bool, store: Arc) -> Result { + let h = Harness::new(rooted, store, 2).await?; + // Give B different actual SQLite bytes, so the fault cannot hit a shared + // empty/header chunk or any part of A's image. + let b = history(&h.route, "registered-b")?; + let invocation = h + .dispatcher + .command( + b.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([189; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"b-base".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + let response = h.controllers[0] + .command(invocation, h.fixture.cell_options(&b).await?)? + .response() + .await?; + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!(response.receipt().receipt.result(), b"b-base"); + let index = response.receipt().position.index; + drop(response); + archive_controller::published(&h.controllers[0], h.route.history, index).await?; + let coverage = + archive_controller::verified_current(&h.fixture, &h.archive, rooted, "dependency-base") + .await?; + archive_values(&h, &coverage).await?; + let marker = coverage.marker().clone(); + // The controller retains its durable hold. Release this completed receiver + // before asking the same one-slot production pool to archive a new cut. + drop(coverage); + + // A separate recovery client owns its own bounded two-operation I/O pool, + // one checkpoint receiver and one tail operation. Its persistent hold is + // independent of the controller's head retirement. + let registry = h + .archive + .directory + .open_group_archive( + &h.fixture.cluster.domain, + &CellGroupArchivePool::new(Default::default())?, + &h.archive.birth, + ) + .await?; + let intent = registry.prepare_acquire(&h.fixture.cluster.domain).await?; + let encoded = serde_json::to_vec(&intent)?; + let path = h + .fixture + .cluster + ._root + .path() + .join("dependency-recovery-intent"); + h.fixture + .cluster + .domain + .blocking(move || -> Result<()> { + use std::io::Write; + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&path)?; + file.write_all(&encoded)?; + file.sync_all()?; + std::fs::File::open(path.parent().context("intent parent")?)?.sync_all()?; + Ok(()) + })? + .await??; + let hold = registry.acquire(&h.fixture.cluster.domain, &intent).await?; + assert_eq!(hold.marker(), &marker); + + let objects = h.remote.list(None).try_collect::>().await?; + let mut dependency = None; + for object in objects + .iter() + .filter(|o| o.location.as_ref().contains("/indexes/")) + { + let value: serde_json::Value = + serde_json::from_slice(&h.remote.get(&object.location).await?.bytes().await?)?; + let Some(chunks) = value.get("chunks").and_then(|v| v.as_array()) else { + continue; + }; + if serde_json::from_value::(value["marker"].clone())? != marker { + continue; + } + let prefix = object + .location + .as_ref() + .split_once("/generations/") + .context("generation prefix")? + .0; + let path = |chunk: &serde_json::Value| -> Result { + let reference = &chunk["object"]; + let digest: Vec = serde_json::from_value(reference["digest"].clone())?; + Ok(ObjectPath::from(format!( + "{prefix}/generations/{}/chunks/{}-{}", + reference["generation"], + reference["length"], + hex::encode(digest) + ))) + }; + let a = chunks + .iter() + .filter(|c| c["part"]["cell"] == "registered-a") + .map(&path) + .collect::>>()?; + for chunk in chunks.iter().filter(|c| { + c["part"]["cell"] == "registered-b" + && c["part"]["kind"] == if rooted { "Extent" } else { "Materialized" } + }) { + let candidate = path(chunk)?; + if !a.contains(&candidate) { + dependency = Some(candidate); + break; + } + } + if dependency.is_some() { + break; + } + } + let dependency = dependency.context("B image bytes outside A's entire image closure")?; + let bytes = h + .remote + .inner + .get(&dependency) + .await? + .bytes() + .await? + .to_vec(); + assert!(!bytes.is_empty()); + assert!(dependency + .as_ref() + .ends_with(&hex::encode(Sha256::digest(&bytes)))); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + Ok(Base { + h, + registry, + hold, + dependency, + bytes, + }) +} + +async fn selected_stays_before_a(base: &Base) -> Result<()> { + // Read the actual registry selection, not the advisory controller status. + // The uncommitted acquire intent below is only an observation. + let intent = base + .registry + .prepare_acquire(&base.h.fixture.cluster.domain) + .await?; + let value = serde_json::to_value(intent)?; + let selected: GroupManifest = + serde_json::from_value(value["action"]["Acquire"]["root"]["marker"].clone())?; + assert_eq!(&selected, base.hold.marker()); + Ok(()) +} + +async fn receiver( + base: &Base, + rooted: bool, + name: &str, + checkpoints: &CellGroupCheckpointPool, +) -> Result { + let destination = + archive_controller::verifier(&base.h.fixture, &base.h.route, rooted, name).await?; + Ok(checkpoints.receive_archive( + destination, + ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + base.h.fixture.cluster._root.path().into(), + )?) +} + +async fn exact_local_retry(h: &Harness) -> Result<()> { + let mut request = cell_attempt(207)?; + request.request.id = CommandId::new(ClientId::new([188; 16])?, 1)?; + request.request.operation = "Add".into(); + request.request.contract_digest = [82; 32]; + request.request.request_digest = command_request_digest(b"later-a"); + request.request.durability = Durability::S3; + request.request.retry_until_ms = h.retry_until; + let authority = h.fixture.runtimes[0] + .execution_authority(h.route.history, h.cell.clone()) + .await?; + let CellSqlOutcome::Receipt(receipt) = h.fixture.appliers[&(0, h.route.history)] + .0 + .stage_registered( + h.fixture.cell_options(&h.cell).await?, + authority, + request, + h.class.clone(), + b"later-a".to_vec(), + false, + ) + .await? + else { + anyhow::bail!("known command must resolve without callback") + }; + assert_eq!(receipt.receipt().receipt.result(), b"later-a"); + assert_eq!( + receipt.receipt().receipt.minimum_durability(), + Durability::S3 + ); + assert_eq!(h.calls.load(Ordering::SeqCst), 3); + // This internal receipt carries no archive proof and is never emitted as an + // S3 response. The stored minimum remains S3, including during repair. + Ok(()) +} + +async fn delayed(rooted: bool, store: Arc) -> Result<()> { + let base = Box::pin(setup(rooted, store)).await?; + let gate = base.h.remote.hold(base.dependency.clone()); + let call = base + .h + .command(1, b"later-a", Durability::S3, Duration::from_secs(15)) + .await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + base.h.waiting().await?; + let mut output = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), output.as_mut()) + .await + .is_err() + ); + + let next = base + .h + .command( + 2, + b"quorum-successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + assert_eq!(next.receipt().receipt.result(), b"quorum-successor"); + let index = next.receipt().position.index; + assert!(index > base.hold.marker().position().index); + drop(next); + local_values(&base.h, index).await?; + selected_stays_before_a(&base).await?; + + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let target = receiver(&base, rooted, "dependency-delayed", &checkpoints).await?; + let mut coverage = Box::pin(base.registry.verify_hold( + &base.h.fixture.cluster.domain, + &base.hold, + &tails, + target, + )); + assert!( + tokio::time::timeout(Duration::from_millis(30), coverage.as_mut()) + .await + .is_err() + ); + assert_eq!(checkpoints.usage().sources, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), output.as_mut()) + .await + .is_err() + ); + + base.h.remote.clear(); + gate.release.notify_waiters(); + let recovered = coverage.await?; + archive_values(&base.h, &recovered).await?; + drop(recovered); + assert_eq!(checkpoints.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + let response = output.await?; + assert_eq!(response.durability(), Durability::S3); + assert_eq!(response.receipt().receipt.result(), b"later-a"); + drop(response); + archive_controller::published(&base.h.controllers[0], base.h.route.history, index).await?; + exact_local_retry(&base.h).await?; + base.registry + .release(&base.h.fixture.cluster.domain, &base.hold) + .await?; + base.h.close().await +} + +async fn corrupt(rooted: bool, store: Arc) -> Result<()> { + let base = Box::pin(setup(rooted, store)).await?; + let mut damaged = base.bytes.clone(); + damaged[0] ^= 1; + assert_ne!(Sha256::digest(&base.bytes), Sha256::digest(&damaged)); + base.h + .remote + .inner + .put(&base.dependency, damaged.clone().into()) + .await?; + assert_eq!( + base.h + .remote + .inner + .get(&base.dependency) + .await? + .bytes() + .await? + .as_ref(), + damaged + ); + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let target = receiver(&base, rooted, "dependency-corrupt", &checkpoints).await?; + let error = base + .registry + .verify_hold(&base.h.fixture.cluster.domain, &base.hold, &tails, target) + .await + .err() + .context("corrupt B must refuse complete coverage")?; + assert!(matches!( + error, + CellGroupArchiveError::Invalid("object digest/length") + )); + assert_eq!(checkpoints.usage().sources, 0); + assert_eq!(tails.usage().operations, 0); + + let error = base + .h + .command(1, b"later-a", Durability::S3, Duration::from_secs(15)) + .await? + .response() + .await + .err() + .context("corrupt B must refuse A S3 acknowledgement")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Archive(_) + )); + let status = base.h.controllers[0] + .archive_status(base.h.route.history)? + .context("archive status")?; + assert!(status.stopped); + assert!(status + .error + .context("corruption failure")? + .to_string() + .contains("object digest/length")); + selected_stays_before_a(&base).await?; + let next = base + .h + .command( + 2, + b"quorum-successor", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(next.durability(), Durability::Quorum); + let index = next.receipt().position.index; + assert!(index > base.hold.marker().position().index); + drop(next); + local_values(&base.h, index).await?; + + // Repair only the fault-injected object in this isolated fixture. Recovery + // uses a fresh physical receiver, while the failed archive owner stays stopped. + base.h + .remote + .inner + .put(&base.dependency, base.bytes.clone().into()) + .await?; + let target = receiver(&base, rooted, "dependency-repaired", &checkpoints).await?; + let recovered = base + .registry + .verify_hold(&base.h.fixture.cluster.domain, &base.hold, &tails, target) + .await?; + archive_values(&base.h, &recovered).await?; + drop(recovered); + exact_local_retry(&base.h).await?; + assert!( + base.h.controllers[0] + .archive_status(base.h.route.history)? + .context("archive status")? + .stopped + ); + base.registry + .release(&base.h.fixture.cluster.domain, &base.hold) + .await?; + base.h.close().await +} + +#[inline(never)] +pub(crate) fn archive_dependencies( + rooted: bool, + store: Arc, +) -> std::pin::Pin>>> { + Box::pin(async move { + // Each scenario creates real fixed-identity voter disks. Isolate their + // catalogs and permanent claims instead of trying to reuse that birth. + let delayed_store = Arc::new(object_store::prefix::PrefixStore::new( + store.clone(), + ObjectPath::from("delayed"), + )); + let corrupt_store = Arc::new(object_store::prefix::PrefixStore::new( + store, + ObjectPath::from("corrupt"), + )); + Box::pin(delayed(rooted, delayed_store)).await?; + Box::pin(corrupt(rooted, corrupt_store)).await + }) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 5fb684c7d..bdf28f6e9 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -811,16 +811,17 @@ pub(super) use sql::{ }; #[allow(unused_imports)] pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; +#[allow(unused_imports)] pub(super) use sql::{ - controller_apply_faults, controller_close, controller_owners, controller_replication, + archive_dependencies, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, + s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, + s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, + s3_remote_slow_delivery, }; -pub(super) use sql::{owned_command_failure, owned_commands}; -#[allow(unused_imports)] pub(super) use sql::{ - s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, - s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, - s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, + controller_apply_faults, controller_close, controller_owners, controller_replication, }; +pub(super) use sql::{owned_command_failure, owned_commands}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index ce88b5f28..183bc419c 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -1,6 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! SQL output proofs on the actual controller and archive lane. +#[path = "cell_group_archive_dependencies.rs"] +mod dependencies; +pub(crate) use dependencies::archive_dependencies; + #[path = "cell_group_s3_enrollment.rs"] mod enrollment; #[path = "cell_group_s3_receipts.rs"] diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 30bcb30bb..7a23d7ae1 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -538,9 +538,10 @@ mod s3_commands; mod s3_response; #[allow(unused_imports)] pub(crate) use s3_commands::{ - s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, - s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, - s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, + archive_dependencies, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, + s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, + s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, + s3_remote_slow_delivery, }; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 6fed942ed..48b757479 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -25,6 +25,7 @@ ROOT = Path(__file__).resolve().parents[2] TARGET = "native_cell_groups_archive_controller" CASES = ( + "rustfs_group_archive_requires_all_dependencies", "rustfs_group_reclaim_restart_keeps_history", "rustfs_archive_coverage_gates_local_compaction", "rustfs_s3_remote_outputs_preserve_durability_and_creation", From 60170a7850f0b2594a9e31a6513725ac7bbb726b Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 06:03:35 -0700 Subject: [PATCH 238/568] docs: qualify whole-group archive dependencies --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 105 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 69 ++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 8 +- 4 files changed, 129 insertions(+), 59 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index e40e0efc6..31b5e4e02 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "7741e4049704f3e1c6519b736e62deaf263c39a2", - "runtime_revision": "7741e4049704f3e1c6519b736e62deaf263c39a2", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending.", + "source_revision": "dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8", + "runtime_revision": "dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 9a552ba31..1dbd9c4ff 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -16,60 +16,57 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). The latest runtime/qualification source is -`7741e4049704f3e1c6519b736e62deaf263c39a2`. CG-06 remains IN_PROGRESS. Its -original physical-reclamation case now proves actual Raft-file removal and -file-length reduction across two shared histories, with an archive-covered -checkpoint, retained suffix, uncheckpointed sibling, old physical reader and -post-purge checkpoint streaming. Bounded complete-inventory image/page -maintenance runs after service quiescence. All three retained voter disks -reopen, verify cells/effects/checkpoints/tails and establish fresh quorum. Every -old receipt resolves without callback replay; two new commands commit over TLS. -Reopen restores archive policy without trusting the lost coverage frontier. The -new scenario alone uses 4 KiB files/8KiB purge threshold/manual purge. Existing -defaults, quotas, production code and marker-1 records are unchanged. - -The preceding logical retained-log admission, owned archive activation, exact physical lease transfer and S3-output lifetime contracts remain required. Node/group capacity is reserved before new callbacks; known receipts remain available. Unknown outcomes and entered physical work retain capacity until authoritative resolution or durable selection. Election/configuration controls retain separately bounded progress. S3 outputs still own complete independent coverage through physical delivery; missing receipts remain ambiguous. See [progress](progress.md) and [protocols](protocols.md) for retained contracts and precise qualification scope. - -Eighteen exact real RustFS controller cases pass in both image forms with seven -signed probes. Twenty-two regression/check commands pass with 268 Rust cases -across seven copied executables, including all 22 storage cases. Locked -final-source all-target Clippy, format/layout, dependency/docs/preflight/plan -checks, 37 Python tests, system-Python hashing and independent -source/artifact/selection/credential audit pass across 2575 sources. Evidence is -under `.cellar/cell-groups-evidence/CG-06-physical-log-reclaim/`. Original -helper-visibility and bounded-reader failures and copied-binary reproduction -remain retained. The earlier 8dfe9ff3 slow-reader timing cause remains -unresolved. Voters are host-native tasks and RustFS one VM. - -Two original CG-06 cases pass: archive tails and physical reclaim/restart. Three -original cases/full task gate remain pending: archive dependencies, full-backend -S3 outage and fenced fresh group/cell incarnations serving from a replacement -quorum. This retained-disk case does not erase all voters, qualify process-kill -recovery, establish a global physical disk bound or measure throughput. Fenced -recovery must inventory every possible old voter, including joins after the -selected cut, using the accepted fencing-authority API plus Docker -qualification. - -Next complete the original archive-dependency acknowledgement case, then qualify -both local cached and remote fresh-discovery paths against a fully stopped -backend. Preserve catalog and epoch fencing; source sealing and activation must -be implemented before any new cache-based authority shortcut. -Seal/Install/Activate codecs exist, but live apply rejects those controls and -manifests lack persistent sealed/active lifecycle. Remote command routing -requests fresh catalog confirmation; local known-route cache paths differ and -must be exercised independently against an actually stopped backend. Durable -cancellation and seal authorization must compete on one CAS decision, sources -must close admission in committed order, and children must stay non-serving -until committed install/publication-bound activation. See private -`outage-authority-investigation.md` in this evidence directory for inspected -seams; no source-seal implementation is claimed. - -CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover, -followed by transitions, topology, placement, hibernation, managed recovery, -measured load and final legacy removal. Independent failure domains and the full -hard cutover remain unqualified. Preserve predecessor runtime ad73f03f, all -prior failure records/qualifications and the original task oracles. The full -end-to-end goal remains active. +`dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8`. CG-06 remains IN_PROGRESS. The +original complete-group dependency case now holds or corrupts an actual B image +chunk/extent outside A's entire image closure, after a real B mutation. Later A +S3 settlement cannot complete; independent recovery is pending or fails with the +exact digest error and the registry selection stays at the preceding cut. Quorum +successors and all three voter data remain healthy. Delay release returns the +exact S3 result; repairing injected bytes permits fresh independent data +recovery but does not silently restart the failed archive owner. Original +receipts resolve without callbacks and retain their S3 minimum. + +The fixture persists and fsyncs the independent recovery client's acquire intent +before CAS, retains its durable hold across controller head retirement, and +releases physical receiver/tail ownership before later work. Existing controller +quotas, timeouts, production behavior and marker-1 records are unchanged. The +reviewed scope extension adds the owning test helper and its common exports; +original acceptance cases and invariants remain intact. + +Nineteen exact real RustFS controller cases pass in both image forms with seven +signed probes. Twenty-two regression/check commands pass with 269 Rust cases, +seven copied executables and 2576 source hashes. All four shared-group native +suites, all 22 storage cases, locked all-target Clippy, format/layout, +dependency/docs/preflight/plan validation, 37 Python tests, system-Python +hashing and independent source/artifact/selection/credential audit pass. Current +evidence is under `.cellar/cell-groups-evidence/CG-06-archive-dependencies/`. +Preceding physical-reclamation source 7741e404 and its evidence remain retained, +including actual old-file removal, reader/checkpoint/sibling preservation, +all-three retained-disk reopen, exact receipts and new TLS writes. The physical +case also passes at the new source. Earlier source-bound failures and the +unresolved 8dfe9ff3 slow-reader timing limitation remain retained. + +Three original CG-06 cases pass: whole-group dependencies, archive tails and +physical reclaim/restart. Two original cases and the full task gate remain +pending: bounded Quorum progress against a fully stopped S3 backend, and fenced +fresh-incarnation serving from a replacement quorum. The next step is to test +existing local cached and remote fresh-discovery paths separately, then +implement the missing bounded-outage behavior without weakening catalog/epoch +authority. Seal/Install/Activate codecs exist, but live apply still rejects +them; source sealing and exact publication-bound activation must precede any new +cached-authority shortcut. The preceding private physical-reclaim evidence +retains the source-bound outage-authority investigation. + +Fenced recovery must inventory every possible old voter, including joins after +the selected archive cut, through the accepted fencing-authority API plus Docker +qualification. CG-08 effect/cache/result lifetimes precede CG-07 public +SDK/fleet cutover, followed by transitions, split/merge, placement, hibernation, +managed recovery, measured load and final legacy removal. The local RustFS +matrix uses host-native voters and one VM. It does not qualify full-backend +outage bounds, independent failure domains, deleted-voter fresh-incarnation +serving, sustained throughput, global physical disk bounds or the final hard +cutover. Preserve all earlier evidence and the user's untracked diagram. The +full end-to-end goal remains active and incomplete. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index c3c2ae19a..1120315d4 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,75 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 complete-group dependency acknowledgement + +Source: `dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8`. CG-06 remains IN_PROGRESS. +The original `group_archive_requires_all_dependencies` case and an exact real +RustFS counterpart now run delay and corruption phases in both image forms. This +qualifies the existing protected-object reuse, complete recovery and S3 +settlement paths; production behavior, markers, deadlines and existing resource +limits are unchanged. The recorded scope extension adds the owning +archive-dependency fixture and its shared test exports without changing any +original case or acceptance oracle. + +Each phase commits a real registered B mutation before selecting the complete +group archive. B now has different SQLite data from A. The fixture finds an +actual materialized image chunk or extent unique to B, outside A's entire image +closure, and checks its content digest. A separate recovery client persists and +fsyncs its exact acquire intent before obtaining a durable hold. It owns a +bounded two-operation I/O pool, one checkpoint receiver and one tail-operation +pool; the controller's original pools remain unchanged. + +The delay phase holds GETs for that exact B image object. A later registered A +command commits to the local quorum but its S3 output stays pending. Another A +command receives a Quorum acknowledgement, and all three voters expose the exact +A/B data and versions. The actual remote registry selection stays at the +preceding cut. Independent whole-group recovery is also pending with its one +receiver occupied. Releasing the GET permits complete historical recovery and +A's exact S3 result. Receivers and tail reservations return to zero before +subsequent work, and the known command resolves locally without callback replay +while preserving its stored S3 minimum. + +The corruption phase overwrites that actual stored B image object with +same-length, different-digest bytes and reads them back. Independent recovery +refuses with the exact object-digest failure. A later A entry still commits, but +its S3 output fails ambiguously through the typed archive error; the archive +owner stops and its selected head cannot advance. Another A Quorum command +succeeds, all voter data remains correct and the group is healthy. Repairing +only the injected bytes permits complete recovery through a fresh physical +receiver. It does not silently restart the stopped archive owner or lower the +original command's S3 minimum. Internal exact receipt lookup observes the +original result without callback replay and supplies no external S3 +acknowledgement. + +Nineteen exact real RustFS controller cases pass in both image forms, with seven +signed protocol probes. Twenty-two regression/check commands pass with 269 +distinct Rust cases across seven copied executables and 2576 source inputs. All +four native shared-group suites, all 22 storage cases, locked all-target +Cellar/logic/LTX Clippy, format/layout, dependency/docs/preflight/plan +validation, 37 Python tests and system-Python hashing pass. Independent +source/artifact/selection and credential-exclusion audit passes. Evidence under +`.cellar/cell-groups-evidence/CG-06-archive-dependencies/` binds raw outputs, +copied executables, orchestration, reviewed plan inputs and the preceding +physical-reclamation qualification. The physical purge/restart case also passes +at this source. Earlier retained failures and the unresolved `8dfe9ff3` +slow-reader timing limitation remain recorded at their original source +boundaries. + +Reflection: three of the five original CG-06 cases now pass: complete-group +dependencies, archive tails and physical reclaim/restart. The next critical work +is the original full-backend S3 outage case: exercise local cached and remote +fresh-discovery paths separately, establish the exact bounded Quorum admission +boundary, and refuse S3 outputs until complete coverage recovers. Preserve +catalog/epoch fencing; durable source sealing and activation are required before +introducing any new cached-authority shortcut. Fenced fresh-incarnation serving +from a replacement quorum remains the other original case. The accepted +fencing-authority API plus Docker qualification must inventory every possible +old voter, including later joins. CG-08 lifetimes precede CG-07 public SDK/fleet +cutover, followed by transitions, split/merge, placement, hibernation, managed +recovery, real load and final legacy removal. No full task, production cutover, +independent-domain qualification or zero-quorum disaster-serving claim is made. + ## CG-06 physical log reclamation and retained-disk recovery Source: `7741e4049704f3e1c6519b736e62deaf263c39a2`. CG-06 remains IN_PROGRESS. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 951943fb2..1fd7b7c43 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1724,7 +1724,10 @@ "crates/cellar/durability/raft/", "crates/cellar-ltx/src/", "crates/cellar/tests/support/cell_group_checkpoint.rs", - "crates/cellar/tests/support/cell_group_engine_fixture.rs" + "crates/cellar/tests/support/cell_group_engine_fixture.rs", + "crates/cellar/tests/support/cell_group_runtime.rs", + "crates/cellar/tests/support/cell_group_sql.rs", + "crates/cellar/tests/support/cell_group_s3_commands.rs" ], "creates": [ "crates/cellar/durability/raft/cell_groups/archive/mod.rs", @@ -1771,7 +1774,8 @@ "crates/cellar/durability/raft/query_reply.rs", "crates/cellar/tests/support/cell_group_s3_remote.rs", "crates/cellar/tests/support/cell_group_s3_remote_wire.rs", - "crates/cellar/durability/raft/cell_groups/archive_compaction.rs" + "crates/cellar/durability/raft/cell_groups/archive_compaction.rs", + "crates/cellar/tests/support/cell_group_archive_dependencies.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From cfbeaab04be0df734128668402e7388bc34b949f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 06:33:13 -0700 Subject: [PATCH 239/568] test: bound quorum during full S3 outages --- .../tests/native_cell_groups_archive.rs | 13 + .../native_cell_groups_archive_controller.rs | 14 + .../cell_group_archive_dependencies.rs | 2 +- .../tests/support/cell_group_archive_store.rs | 3 + .../support/cell_group_backend_control.rs | 131 +++++ .../tests/support/cell_group_runtime.rs | 8 +- .../cellar/tests/support/cell_group_rustfs.rs | 39 +- .../tests/support/cell_group_s3_commands.rs | 48 +- .../tests/support/cell_group_s3_outage.rs | 454 ++++++++++++++++++ .../tests/support/cell_group_s3_remote.rs | 34 +- crates/cellar/tests/support/cell_group_sql.rs | 8 +- tools/multi-raft/backend_control.py | 199 ++++++++ tools/multi-raft/cell_groups.py | 38 +- tools/multi-raft/test_backend_control.py | 99 ++++ 14 files changed, 1052 insertions(+), 38 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_backend_control.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_outage.rs create mode 100644 tools/multi-raft/backend_control.py create mode 100644 tools/multi-raft/test_backend_control.py diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index cb874a1b7..2434864c8 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -33,6 +33,19 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn quorum_continues_bounded_s3_outage() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::full_backend_outage( + rooted, + Arc::new(object_store::memory::InMemory::new()), + None, + ) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_archive_requires_all_dependencies() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/native_cell_groups_archive_controller.rs b/crates/cellar/tests/native_cell_groups_archive_controller.rs index dc6bc9541..93e1a16ec 100644 --- a/crates/cellar/tests/native_cell_groups_archive_controller.rs +++ b/crates/cellar/tests/native_cell_groups_archive_controller.rs @@ -23,6 +23,20 @@ mod rustfs; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] +async fn rustfs_quorum_continues_bounded_s3_outage() -> Result<()> { + for rooted in [false, true] { + runtime_fixture::full_backend_outage( + rooted, + rustfs::outage_store("full-backend-outage", rooted)?, + Some(rustfs::control_directory()?), + ) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] #[ignore = "requires isolated local RustFS; tools/multi-raft/cell_groups.py"] async fn rustfs_group_archive_requires_all_dependencies() -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_archive_dependencies.rs b/crates/cellar/tests/support/cell_group_archive_dependencies.rs index c1931ff10..f9476e715 100644 --- a/crates/cellar/tests/support/cell_group_archive_dependencies.rs +++ b/crates/cellar/tests/support/cell_group_archive_dependencies.rs @@ -11,7 +11,7 @@ struct Base { bytes: Vec, } -fn image_value(pin: CellImagePin, expected: u64) -> Result<()> { +pub(super) fn image_value(pin: CellImagePin, expected: u64) -> Result<()> { let bytes = match pin { CellImagePin::Empty => anyhow::bail!("expected a physical SQLite image"), CellImagePin::Materialized(pin) => { diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index 544131645..e4958961d 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -20,6 +20,7 @@ pub struct Gate { pub struct Store { pub inner: Arc, pub puts: AtomicUsize, + pub gets: AtomicUsize, pub lose_put_response: AtomicBool, pub deletes: AtomicUsize, pub lose_delete_response: AtomicBool, @@ -40,6 +41,7 @@ impl Store { Arc::new(Self { inner, puts: AtomicUsize::new(0), + gets: AtomicUsize::new(0), lose_put_response: AtomicBool::new(false), deletes: AtomicUsize::new(0), lose_delete_response: AtomicBool::new(false), @@ -155,6 +157,7 @@ impl ObjectStore for Store { } async fn get_opts(&self, path: &Path, opts: GetOptions) -> Result { + self.gets.fetch_add(1, Ordering::AcqRel); self.available()?; let gate = self .held diff --git a/crates/cellar/tests/support/cell_group_backend_control.rs b/crates/cellar/tests/support/cell_group_backend_control.rs new file mode 100644 index 000000000..72823a45b --- /dev/null +++ b/crates/cellar/tests/support/cell_group_backend_control.rs @@ -0,0 +1,131 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! An explicit test-only handshake with the runner that owns the RustFS container. +use super::*; +use serde::{Deserialize, Serialize}; +use std::{ + io::{Read, Write}, + path::PathBuf, +}; + +#[derive(Serialize)] +struct Request { + format: u8, + sequence: u8, + mode: &'static str, + action: &'static str, +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct Response { + format: u8, + sequence: u8, + mode: String, + action: String, + running: bool, +} + +pub(super) struct Control { + store: Arc, + directory: Option, + rooted: bool, +} + +impl Control { + pub(super) fn new( + store: Arc, + directory: Option, + rooted: bool, + ) -> Self { + Self { + store, + directory, + rooted, + } + } + + pub(super) async fn set_running(&self, domain: &ExecutionDomain, running: bool) -> Result<()> { + let Some(directory) = self.directory.clone() else { + self.store.outage.store(!running, Ordering::Release); + return Ok(()); + }; + let sequence = u8::from(self.rooted) * 2 + u8::from(running); + let mode = if self.rooted { + "extents" + } else { + "materialized" + }; + let action = if running { "start" } else { "stop" }; + let encoded = serde_json::to_vec(&Request { + format: 1, + sequence, + mode, + action, + })?; + let response = directory.join(format!("{sequence:02}-response.json")); + domain + .blocking(move || -> Result<()> { + anyhow::ensure!( + directory.is_absolute() && directory.is_dir(), + "explicit control directory" + ); + let pending = directory.join(format!("{sequence:02}-request.pending")); + let request = directory.join(format!("{sequence:02}-request.json")); + let mut file = std::fs::OpenOptions::new() + .create_new(true) + .write(true) + .open(&pending)?; + file.write_all(&encoded)?; + file.sync_all()?; + // Atomic create, never replacement of another operation's request. + std::fs::hard_link(&pending, request)?; + std::fs::remove_file(pending)?; + std::fs::File::open(directory)?.sync_all()?; + Ok(()) + })? + .await??; + let deadline = domain + .run(async { cellar::asyncrt::mono_ms().saturating_add(30_000) }) + .await?; + loop { + let path = response.clone(); + let reply = domain + .blocking(move || -> Result> { + let file = match std::fs::File::open(path) { + Ok(file) => file, + Err(error) if error.kind() == std::io::ErrorKind::NotFound => { + return Ok(None) + } + Err(error) => return Err(error.into()), + }; + let mut bytes = Vec::new(); + file.take(4097).read_to_end(&mut bytes)?; + anyhow::ensure!(bytes.len() <= 4096, "bounded control reply"); + Ok(Some(serde_json::from_slice(&bytes)?)) + })? + .await??; + if let Some(reply) = reply { + anyhow::ensure!( + reply.format == 1 + && reply.sequence == sequence + && reply.mode == mode + && reply.action == action + && reply.running == running, + "exact backend transition reply" + ); + return Ok(()); + } + domain + .run(async { + anyhow::ensure!( + cellar::asyncrt::mono_ms() < deadline, + "backend control deadline" + ); + cellar::asyncrt::sleep(Duration::from_millis(5)).await; + Ok::<_, anyhow::Error>(()) + }) + .await??; + } + } +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index bdf28f6e9..f35d52c70 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -813,10 +813,10 @@ pub(super) use sql::{ pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; #[allow(unused_imports)] pub(super) use sql::{ - archive_dependencies, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, - s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, - s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, - s3_remote_slow_delivery, + archive_dependencies, full_backend_outage, s3_commands, s3_deadline, s3_enrollment, + s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, + s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, + s3_remote_outputs, s3_remote_slow_delivery, }; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, diff --git a/crates/cellar/tests/support/cell_group_rustfs.rs b/crates/cellar/tests/support/cell_group_rustfs.rs index 518f78498..62353be71 100644 --- a/crates/cellar/tests/support/cell_group_rustfs.rs +++ b/crates/cellar/tests/support/cell_group_rustfs.rs @@ -6,6 +6,36 @@ use object_store::{aws::AmazonS3Builder, path::Path, prefix::PrefixStore, Object use std::{env, sync::Arc}; pub fn store(case: &str, rooted: bool) -> Result> { + configured_store(case, rooted, None) +} + +pub fn outage_store(case: &str, rooted: bool) -> Result> { + // A stopped backend must expose its actual connection failure promptly. + // Other RustFS cases keep their existing adapter retry policy. + configured_store( + case, + rooted, + Some(object_store::RetryConfig { + max_retries: 0, + ..Default::default() + }), + ) +} + +pub fn control_directory() -> Result { + let path = std::path::PathBuf::from(env::var("CELLAR_CELL_GROUPS_RUSTFS_CONTROL")?); + ensure!( + path.is_absolute() && path.is_dir(), + "explicit owned backend control" + ); + Ok(path) +} + +fn configured_store( + case: &str, + rooted: bool, + retry: Option, +) -> Result> { let endpoint = env::var("CELLAR_CELL_GROUPS_RUSTFS_ENDPOINT")?; ensure!( endpoint.starts_with("http://127.0.0.1:"), @@ -17,14 +47,17 @@ pub fn store(case: &str, rooted: bool) -> Result> { "{prefix}/{case}/{}", if rooted { "extents" } else { "materialized" } ))?; - let store = AmazonS3Builder::new() + let mut builder = AmazonS3Builder::new() .with_bucket_name(env::var("CELLAR_CELL_GROUPS_RUSTFS_BUCKET")?) .with_endpoint(endpoint) .with_access_key_id(env::var("AWS_ACCESS_KEY_ID")?) .with_secret_access_key(env::var("AWS_SECRET_ACCESS_KEY")?) .with_region("us-east-1") .with_allow_http(true) - .with_conditional_put(object_store::aws::S3ConditionalPut::ETagMatch) - .build()?; + .with_conditional_put(object_store::aws::S3ConditionalPut::ETagMatch); + if let Some(retry) = retry { + builder = builder.with_retry(retry); + } + let store = builder.build()?; Ok(Arc::new(PrefixStore::new(store, prefix))) } diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 183bc419c..466241d28 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -5,6 +5,10 @@ mod dependencies; pub(crate) use dependencies::archive_dependencies; +#[path = "cell_group_s3_outage.rs"] +mod outage; +pub(crate) use outage::full_backend_outage; + #[path = "cell_group_s3_enrollment.rs"] mod enrollment; #[path = "cell_group_s3_receipts.rs"] @@ -70,6 +74,27 @@ impl Harness { minimum: Durability, result_size: Option, send_buffer: Option, + ) -> Result { + Box::pin(Self::configured_retention( + rooted, + store, + max_commands, + minimum, + result_size, + send_buffer, + None, + )) + .await + } + + async fn configured_retention( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, + retention: Option, ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -97,10 +122,25 @@ impl Harness { }, )?, ); - let (mut fixture, route, cell, _) = Box::pin(registered::setup_store_minimum( - rooted, &class, store, minimum, - )) - .await?; + let (mut fixture, route, cell, _) = if let Some(retention) = retention { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Box::pin(Cluster::with_store_retention( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + minimum, + retention, + )) + .await?; + Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? + } else { + Box::pin(registered::setup_store_minimum( + rooted, &class, store, minimum, + )) + .await? + }; let (archive, remote) = archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; if send_buffer.is_some() { diff --git a/crates/cellar/tests/support/cell_group_s3_outage.rs b/crates/cellar/tests/support/cell_group_s3_outage.rs new file mode 100644 index 000000000..d328324a0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_outage.rs @@ -0,0 +1,454 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete backend outage bounds Quorum admission without inventing S3 proof. +use super::*; +use raft::Storage as _; +use std::path::PathBuf; + +#[path = "cell_group_backend_control.rs"] +mod backend_control; + +const ENTRY_LIMIT: usize = 9; + +async fn gateway( + h: &Harness, + store: Arc, +) -> Result { + let catalog = Catalog::new(store, Default::default())?; + let config = NamespaceConfig::new( + h.route.history.cluster, + h.route.history.namespace, + cellar_logic::NamespaceMode::Raft, + h.class.schema_digest(), + ); + let namespace = h + .fixture + .cluster + .domain + .run(catalog.existing_namespace(config)) + .await??; + Ok(CellGroupDispatcher::new( + &h.fixture.cluster.domain, + RangeDirectory::new(namespace, Default::default())?, + h.class.clone(), + VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&h.controllers[0])?) +} + +async fn state(h: &Harness, index: LogIndex, value: u64) -> Result<()> { + controller::applied(&h.fixture, h.route.history, index).await?; + let b = history(&h.route, "registered-b")?; + for runtime in &h.fixture.runtimes { + for (cell, expected) in [(&h.cell, value), (&b, 0)] { + let selected = runtime + .reconciliation_authority(h.route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), expected + 1); + let pin = selected.image().clone(); + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(pin, expected))? + .await??; + } + assert_eq!(h.controllers[0].usage().quarantined_groups, 0); + } + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, value); + Ok(()) +} + +async fn initialize_at_capacity(h: &Harness) -> Result<()> { + let cell = history(&h.route, "outage-new")?; + let authority = h.fixture.runtimes[0] + .enrollment_authority(h.route.history, cell.clone()) + .await?; + let observed = h.initializations.clone(); + let result = h.fixture.appliers[&(0, h.route.history)] + .0 + .initialize( + h.fixture.cell_options(&cell).await?, + authority, + CellEnrollmentAttempt { + creation: ProposalId::new([203; 16])?, + preview: ImageId::new([203; 16])?, + inputs: cellar::storage::CommandInputs { + now_unix_millis: 10, + random: [17; 32], + }, + }, + move |_| { + observed.fetch_add(1, Ordering::SeqCst); + Ok(()) + }, + ) + .await; + assert!(matches!( + result, + Err(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(h.initializations.load(Ordering::SeqCst), 0); + Ok(()) +} + +async fn run( + rooted: bool, + store: Arc, + control_directory: Option, +) -> Result<()> { + // This adapter is shared by catalog and archive. In real RustFS mode it + // never injects an outage: the owning runner actually stops the container. + let global = archive_controller::archive_store::Store::around(store); + let control = backend_control::Control::new(global.clone(), control_directory, rooted); + let mut retention = CellGroupLogRetentionLimits::default(); + retention.group.entries = ENTRY_LIMIT; + retention.progress_group.entries = 1; + let h = Box::pin(Harness::configured_retention( + rooted, + global.clone(), + 2, + Durability::Quorum, + None, + None, + Some(retention), + )) + .await?; + let group = h.fixture.cluster.voters[0] + .log + .cell_group(&h.route.history)?; + assert_eq!(group.log_retention_usage()?.retained_entries, 4); + let warm = h + .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + let warm_index = warm.receipt().position.index; + assert_eq!(warm.durability(), Durability::Quorum); + drop(warm); + archive_controller::published(&h.controllers[0], h.route.history, warm_index).await?; + let gateway_store = archive_controller::archive_store::Store::around(global.clone()); + let remote = remote::Remote::start(&h, gateway(&h, gateway_store.clone()).await?).await?; + // Warm the receiver too. It must still perform fresh discovery after stop. + let found = remote::command_using( + &h, + &remote, + remote::command_request(&h, 1, b"warm", Durability::Quorum)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + ) + .await?; + assert_eq!(found.result(), b"warm"); + drop(found); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + let fresh_gets = gateway_store.gets.load(Ordering::SeqCst); + let probe = ObjectPath::from("backend-probe"); + let probe_bytes = vec![29_u8; 128]; + global + .put_opts( + &probe, + probe_bytes.clone().into(), + object_store::PutOptions::from(object_store::PutMode::Create), + ) + .await?; + let before_checkpoint = group.checkpoint_manifest()?; + let before_first = group.first_index()?; + control + .set_running(&h.fixture.cluster.domain, false) + .await?; + let failed_probe = tokio::time::timeout(Duration::from_secs(2), global.get(&probe)).await?; + assert!( + failed_probe.is_err(), + "stopped backend must refuse actual S3 I/O" + ); + + let remote_request = CellCommandRequest { + id: CommandId::new(ClientId::new([187; 16])?, 1)?, + contract: h.class.contract("Add").context("Add")?, + input: b"remote-outage".to_vec(), + durability: Durability::Quorum, + retry_until_ms: h.retry_until, + }; + let mut remote_invocation = h + .dispatcher + .command( + h.cell.clone(), + remote_request, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + remote_invocation.begin().await?; + let refused = remote_invocation + .execute_remote(remote.transport(), MemberId::new(1)?) + .await; + assert!( + refused.is_err(), + "remote reception requires fresh catalog discovery" + ); + assert!(gateway_store.gets.load(Ordering::SeqCst) > fresh_gets); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + + let pending = h + .command(2, b"archive-wait", Durability::S3, Duration::from_secs(2)) + .await?; + h.waiting().await?; + let mut pending = Box::pin(pending.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), pending.as_mut()) + .await + .is_err() + ); + let mut latest = warm_index; + for sequence in 3..=5 { + let response = h + .command( + sequence, + b"quorum-offline", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(response.durability(), Durability::Quorum); + assert_eq!(response.receipt().receipt.result(), b"quorum-offline"); + latest = response.receipt().position.index; + drop(response); + } + state(&h, latest, 5).await?; + let usage = group.log_retention_usage()?; + assert_eq!( + (usage.retained_entries, usage.reserved_entries), + (ENTRY_LIMIT, 0) + ); + let refused = h + .command( + 6, + b"after-ceiling", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await + .err() + .context("retention ceiling must refuse new command")?; + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Overloaded)) + )); + assert_eq!(h.calls.load(Ordering::SeqCst), 5); + initialize_at_capacity(&h).await?; + let known = h + .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) + .await? + .response() + .await?; + assert_eq!(known.durability(), Durability::Quorum); + assert_eq!(known.receipt().receipt.result(), b"warm"); + drop(known); + let error = pending + .await + .err() + .context("S3 output cannot settle during full outage")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + assert_eq!(h.calls.load(Ordering::SeqCst), 5); + assert!( + !h.controllers[0] + .archive_status(h.route.history)? + .context("archive status")? + .stopped + ); + controller::wait_for(async || { + Ok(h.controllers[0].usage().retained_s3 == 0 + && h.controllers[0].usage().retained_commands == 0) + }) + .await?; + let downgraded = h + .command( + 2, + b"archive-wait", + Durability::Quorum, + Duration::from_millis(800), + ) + .await? + .response() + .await + .err() + .context("stored S3 minimum survives caller downgrade")?; + assert!(downgraded.ambiguous); + assert!(matches!( + downgraded.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + assert_eq!(h.calls.load(Ordering::SeqCst), 5); + + let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let source = h.fixture.runtimes[0] + .capture_checkpoint(h.route.history, &checkpoints) + .await?; + let prepared = h + .fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + let cut = prepared.marker().clone(); + assert_eq!(cut.position().index, latest); + controller::wait_for(async || { + match h.fixture.runtimes[0] + .publish_checkpoint(h.route.history, prepared.clone()) + .await + { + Err(RuntimeError::Overloaded) => Ok(true), + Err(RuntimeError::NotReady) => Ok(false), + other => anyhow::bail!("uncovered checkpoint must refuse: {other:?}"), + } + }) + .await?; + assert_eq!(group.checkpoint_manifest()?, before_checkpoint); + assert_eq!(group.first_index()?, before_first); + assert_eq!(group.log_retention_usage()?.retained_entries, ENTRY_LIMIT); + + control.set_running(&h.fixture.cluster.domain, true).await?; + assert_eq!( + global.get(&probe).await?.bytes().await?.as_ref(), + probe_bytes + ); + archive_controller::published(&h.controllers[0], h.route.history, latest).await?; + let coverage = + archive_controller::verified_current(&h.fixture, &h.archive, rooted, "outage-recovery") + .await?; + assert!(coverage.marker().position().index >= latest); + let image = coverage + .recovery() + .manifest() + .pin_cell(&h.cell.cell) + .context("archived A")? + .1; + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(image, 5))? + .await??; + let image = coverage + .recovery() + .manifest() + .pin_cell(&CellId::new("registered-b")?) + .context("archived B")? + .1; + h.fixture + .cluster + .domain + .blocking(move || dependencies::image_value(image, 0))? + .await??; + // Release the completed receiver before another S3 output uses that same + // unchanged one-source archive pool. + drop(coverage); + let recovered = h + .command( + 2, + b"archive-wait", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(recovered.durability(), Durability::S3); + assert_eq!(recovered.receipt().receipt.result(), b"archive-wait"); + drop(recovered); + assert_eq!(h.calls.load(Ordering::SeqCst), 5); + + h.fixture.runtimes[0] + .publish_checkpoint(h.route.history, prepared) + .await?; + for node in 0..3 { + if node != 0 { + controller::wait_for(async || Ok(checkpoints.usage().sources == 0)).await?; + let source = h.fixture.runtimes[node] + .capture_checkpoint(h.route.history, &checkpoints) + .await?; + let prepared = h + .fixture + .cluster + .domain + .blocking(move || source.prepare_publication())? + .await??; + assert_eq!(prepared.marker(), &cut); + h.fixture.runtimes[node] + .publish_checkpoint(h.route.history, prepared) + .await?; + } + let log = h.fixture.cluster.voters[node] + .log + .cell_group(&h.route.history)?; + controller::wait_for(async || { + Ok(log.checkpoint_manifest()?.as_ref() == Some(&cut) + && log.log_retention_usage()?.retained_entries == 0 + && checkpoints.usage().sources == 0) + }) + .await?; + } + let resumed = h + .command( + 6, + b"after-ceiling", + Durability::Quorum, + Duration::from_secs(15), + ) + .await? + .response() + .await?; + assert_eq!(resumed.durability(), Durability::Quorum); + drop(resumed); + // Preserve the same invocation, captured inputs, original deadline and + // attempt budget after the failed remote response. + remote_invocation.retry().await?; + let resumed = remote_invocation + .execute_remote(remote.transport(), MemberId::new(1)?) + .await?; + assert_eq!(resumed.durability(), Durability::Quorum); + assert_eq!(resumed.result(), b"remote-outage"); + drop(resumed); + drop(remote_invocation); + latest = h.fixture.runtimes[0] + .status(h.route.history) + .await? + .applied + .index; + state(&h, latest, 7).await?; + archive_controller::published(&h.controllers[0], h.route.history, latest).await?; + remote.close().await?; + drop(group); + h.close().await +} + +#[inline(never)] +pub(crate) fn full_backend_outage( + rooted: bool, + store: Arc, + control_directory: Option, +) -> std::pin::Pin>>> { + Box::pin(run(rooted, store, control_directory)) +} diff --git a/crates/cellar/tests/support/cell_group_s3_remote.rs b/crates/cellar/tests/support/cell_group_s3_remote.rs index e3252dd15..67cef2f04 100644 --- a/crates/cellar/tests/support/cell_group_s3_remote.rs +++ b/crates/cellar/tests/support/cell_group_s3_remote.rs @@ -10,13 +10,17 @@ use cellar::application::consensus::verification::cell_groups::{ mod wire; pub(crate) use wire::s3_remote_slow_delivery; -struct Remote { +pub(super) struct Remote { gateway: CellCommandGateway, router: super::super::query_origins::Router, } impl Remote { - async fn start(h: &Harness, dispatcher: CellGroupDispatcher) -> Result { + pub(super) fn transport(&self) -> &RaftTransport { + &self.router.transport + } + + pub(super) async fn start(h: &Harness, dispatcher: CellGroupDispatcher) -> Result { let gateway = h.fixture.transports[0].install_cell_command_gateway( dispatcher, h.controllers[0].clone(), @@ -38,13 +42,13 @@ impl Remote { Ok(Self { gateway, router }) } - async fn close(self) -> Result<()> { + pub(super) async fn close(self) -> Result<()> { drop(self.gateway); self.router.close().await } } -fn command_request( +pub(super) fn command_request( h: &Harness, sequence: u64, input: &[u8], @@ -65,15 +69,25 @@ async fn command( sequence: u64, input: &[u8], durability: Durability, +) -> Result { + command_using( + h, + remote, + command_request(h, sequence, input, durability)?, + bounds(), + ) + .await +} + +pub(super) async fn command_using( + h: &Harness, + remote: &Remote, + request: CellCommandRequest, + limits: CellRoutingLimits, ) -> Result { let mut invocation = h .dispatcher - .command( - h.cell.clone(), - command_request(h, sequence, input, durability)?, - bounds(), - None, - ) + .command(h.cell.clone(), request, limits, None) .await?; invocation.begin().await?; Ok(invocation diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 7a23d7ae1..c940698b0 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -538,10 +538,10 @@ mod s3_commands; mod s3_response; #[allow(unused_imports)] pub(crate) use s3_commands::{ - archive_dependencies, s3_commands, s3_deadline, s3_enrollment, s3_enrollment_deadline, - s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, s3_receipt_failures, - s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, - s3_remote_slow_delivery, + archive_dependencies, full_backend_outage, s3_commands, s3_deadline, s3_enrollment, + s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, + s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, + s3_remote_outputs, s3_remote_slow_delivery, }; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; diff --git a/tools/multi-raft/backend_control.py b/tools/multi-raft/backend_control.py new file mode 100644 index 000000000..0ba6f7578 --- /dev/null +++ b/tools/multi-raft/backend_control.py @@ -0,0 +1,199 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Run one exact outage case with four owner-checked RustFS transitions.""" + +import json +import os +from pathlib import Path +import re +import signal +import subprocess +import tempfile +import time +import urllib.error +import urllib.request + +from local import command + +TRANSITIONS = ( + ("materialized", "stop"), ("materialized", "start"), + ("extents", "stop"), ("extents", "start"), +) + + +def atomic_json(path, value): + temporary = path.with_suffix(".pending") + with temporary.open("x") as output: + json.dump(value, output) + output.write("\n") + output.flush() + os.fsync(output.fileno()) + temporary.replace(path) + descriptor = os.open(path.parent, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +class BackendControl: + def __init__(self, remote, directory): + if (not re.fullmatch(r"cellar-multi-raft-[0-9a-f]{16}", remote["owner"]) + or remote["container"] != remote["owner"] + "-rustfs" + or not re.fullmatch(r"colima-cellar-multi-raft(?:-[a-z0-9-]+)?", remote["context"]) + or not re.fullmatch(r"http://127\.0\.0\.1:[0-9]+", remote["endpoint"])): + raise ValueError("requires the captured owned loopback backend") + self.remote = remote + self.directory = Path(directory).resolve() + self.directory.mkdir(mode=0o700, parents=True, exist_ok=False) + self.docker = ["docker", "--context", remote["context"]] + self.target = remote["container"] + self.identity = self.inspect() + self.verify(True) + self.target = self.identity["id"] + self.events = [] + self.status = "RUNNING" + self.error = None + self.cleanup = [] + self.save() + + def inspect(self): + # Deliberately exclude Config.Env and all credential-bearing fields. + return json.loads(command(self.docker + [ + "inspect", "--format", + '{"id":{{json .Id}},"owner":{{json (index .Config.Labels "cellar.multi-raft.run")}},' + '"image":{{json .Image}},"running":{{json .State.Running}},' + '"cpus":{{json .HostConfig.NanoCpus}},"memory":{{json .HostConfig.Memory}},' + '"swap":{{json .HostConfig.MemorySwap}},"ports":{{json .HostConfig.PortBindings}},' + '"mounts":{{json .Mounts}}}', self.target, + ])) + + def verify(self, running): + actual = self.inspect() + port = self.remote["endpoint"].rsplit(":", 1)[1] + if (not re.fullmatch(r"[0-9a-f]{64}", actual["id"]) + or actual["id"] != self.identity["id"] + or actual["owner"] != self.remote["owner"] + or actual["image"] != self.remote["image_id"] + or type(actual["running"]) is not bool or actual["running"] != running + or (actual["cpus"], actual["memory"], actual["swap"]) + != (10**9, 2 * 1024**3, 2 * 1024**3) + or actual["ports"].get("9000/tcp") != [{"HostIp": "127.0.0.1", "HostPort": port}] + or actual["mounts"] != self.identity["mounts"]): + raise ValueError("backend identity, owner, resource, mount or state changed") + return actual + + def healthy(self): + deadline = time.monotonic() + 20 + while True: + try: + with urllib.request.urlopen(self.remote["endpoint"] + "/health", timeout=1) as response: + if response.status == 200: + return + except (urllib.error.URLError, TimeoutError, OSError): + pass + if time.monotonic() >= deadline: + raise RuntimeError("owned RustFS restart health deadline expired") + time.sleep(0.02) + + def transition(self, request): + sequence = len(self.events) + if sequence >= len(TRANSITIONS): + raise ValueError("too many backend control operations") + mode, action = TRANSITIONS[sequence] + if (set(request) != {"format", "sequence", "mode", "action"} + or type(request["format"]) is not int or request["format"] != 1 + or type(request["sequence"]) is not int or request["sequence"] != sequence + or (request["mode"], request["action"]) != (mode, action)): + raise ValueError("unexpected backend transition request") + running = action == "start" + before = self.verify(not running) + started = time.monotonic() + if running: + command(self.docker + ["start", self.target]) + self.healthy() + else: + command(self.docker + ["stop", "--time", "5", self.target]) + after = self.verify(running) + self.events.append({"sequence": sequence, "mode": mode, "action": action, + "before": before, "after": after, + "elapsed_seconds": round(time.monotonic() - started, 3)}) + self.save() + atomic_json(self.directory / f"{sequence:02}-response.json", + {**request, "running": running}) + + def save(self): + path = self.directory / "record.json" + temporary = path.with_suffix(".tmp") + temporary.write_text(json.dumps({"format": 1, "status": self.status, + "container_id": self.identity["id"], + "events": self.events, "cleanup": self.cleanup, + "error": self.error}, + indent=2) + "\n") + temporary.replace(path) + + def restore(self): + actual = self.inspect() + # Identity/resource checks also apply to cleanup; never start a replacement. + self.verify(actual["running"]) + if not actual["running"]: + command(self.docker + ["start", self.target]) + self.healthy() + self.cleanup.append({"restarted_owned_backend": self.verify(True)}) + else: + self.cleanup.append({"already_running": True}) + self.save() + + def run(self, argv, cwd, env): + child_env = dict(env, CELLAR_CELL_GROUPS_RUSTFS_CONTROL=str(self.directory)) + with tempfile.TemporaryFile(mode="w+") as output: + process = subprocess.Popen(argv, cwd=cwd, env=child_env, stdout=output, + stderr=subprocess.STDOUT, start_new_session=True) + deadline = None # Build time precedes the infrastructure experiment. + returncode = 125 + try: + while process.poll() is None: + sequence = len(self.events) + request = self.directory / f"{sequence:02}-request.json" + for observed in self.directory.glob("*-request.json"): + if not re.fullmatch(r"[0-3][0-9]-request\.json", observed.name): + raise ValueError("unexpected backend control filename") + if int(observed.name[:2]) > sequence: + raise ValueError("out-of-order backend control request") + if request.exists(): + with request.open("rb") as source: + encoded = source.read(4097) + if len(encoded) > 4096: + raise ValueError("backend control request exceeds limit") + if deadline is None: + deadline = time.monotonic() + 180 + self.transition(json.loads(encoded)) + if deadline is not None and time.monotonic() >= deadline: + raise RuntimeError("backend outage experiment deadline expired") + time.sleep(0.01) + returncode = process.wait() + if returncode or len(self.events) != len(TRANSITIONS): + raise RuntimeError("outage case failed or omitted a backend transition") + self.verify(True) + self.status = "PASSED" + except BaseException as error: + self.status = "FAILED" + self.error = type(error).__name__ + ": " + str(error) + if process.poll() is None: + os.killpg(process.pid, signal.SIGTERM) + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + os.killpg(process.pid, signal.SIGKILL) + process.wait() + returncode = process.wait() or 125 + finally: + try: + self.restore() + except BaseException as error: + self.status = "FAILED" + self.error = "cleanup " + type(error).__name__ + ": " + str(error) + returncode = returncode or 125 + self.save() + output.seek(0) + return subprocess.CompletedProcess(argv, returncode, output.read()) diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index 48b757479..ac9fc1855 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -21,10 +21,12 @@ from local import command from store_probe import probe +from backend_control import BackendControl ROOT = Path(__file__).resolve().parents[2] TARGET = "native_cell_groups_archive_controller" CASES = ( + "rustfs_quorum_continues_bounded_s3_outage", "rustfs_group_archive_requires_all_dependencies", "rustfs_group_reclaim_restart_keeps_history", "rustfs_archive_coverage_gates_local_compaction", @@ -134,8 +136,10 @@ def main(): "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " - "No byte/disk outage bound, " - "Raft log file purge, process-kill restart, fencing/DR, performance or independent hosts.", + "The bounded full-backend outage case controls only this owned container. " + "Logical retained-entry admission and controlled-geometry Raft file purge are tested; " + "global physical disk bounds, process-kill restart, fencing/DR, sustained performance " + "and independent hosts remain separate gates.", } def save(): @@ -167,8 +171,17 @@ def save(): argv = cargo + ([selection] if number == 0 else [selection, "--exact", "--ignored", "--test-threads=1"]) print(f"START {number + 1}/{len(CASES) + 1}: {selection}", flush=True) started = time.monotonic() - result = subprocess.run(argv, cwd=ROOT, env=env, text=True, - stdout=subprocess.PIPE, stderr=subprocess.STDOUT) + if selection == "rustfs_quorum_continues_bounded_s3_outage": + control = BackendControl(remote, out / "backend-control") + result = control.run(argv, ROOT, env) + control_record = control.directory / "record.json" + record["backend_controls"] = [{ + "case": selection, "path": str(control_record.relative_to(out)), + "sha256": sha(control_record), + }] + else: + result = subprocess.run(argv, cwd=ROOT, env=env, text=True, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT) output = result.stdout for secret in credentials.values(): output = output.replace(secret, "[REDACTED]") @@ -177,6 +190,15 @@ def save(): record["commands"].append({"argv": argv, "exit_code": result.returncode, "elapsed_seconds": round(time.monotonic() - started, 3), "output": log.name, "sha256": sha(log)}) + match = re.search(r"Running [^\n]*\(([^\n)]+/native_cell_groups_archive_controller-[a-f0-9]+)\)", output) + if match: + executable = Path(match[1]) + selected_hash = sha(executable) + if "executable" in record and record["executable"]["sha256"] != selected_hash: + raise RuntimeError("executable changed during qualification") + destination = out / executable.name + shutil.copy2(executable, destination) + record["executable"] = {"path": destination.name, "sha256": selected_hash} save() if result.returncode: raise RuntimeError(f"{selection} failed; see {log}") @@ -186,16 +208,8 @@ def save(): raise RuntimeError("missing exact RustFS test in executable inventory") elif not exact_case_passed(output, selection): raise RuntimeError("exact RustFS test was not executed successfully") - match = re.search(r"Running [^\n]*\(([^\n)]+/native_cell_groups_archive_controller-[a-f0-9]+)\)", output) if not match: raise RuntimeError("missing executable provenance") - executable = Path(match[1]) - selected_hash = sha(executable) - if "executable" in record and record["executable"]["sha256"] != selected_hash: - raise RuntimeError("executable changed during qualification") - destination = out / executable.name - shutil.copy2(executable, destination) - record["executable"] = {"path": destination.name, "sha256": selected_hash} print(f"PASS {selection}", flush=True) if sources() != record["source_inputs"]: raise RuntimeError("source changed during qualification") diff --git a/tools/multi-raft/test_backend_control.py b/tools/multi-raft/test_backend_control.py new file mode 100644 index 000000000..03cf1b1e1 --- /dev/null +++ b/tools/multi-raft/test_backend_control.py @@ -0,0 +1,99 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""The test runner must never control a foreign or substituted container.""" + +import copy +import json +import os +from pathlib import Path +import tempfile +import sys +import unittest +from unittest.mock import patch + +from backend_control import BackendControl, TRANSITIONS + + +class BackendOwnershipTests(unittest.TestCase): + def setUp(self): + self.root = tempfile.TemporaryDirectory() + self.addCleanup(self.root.cleanup) + owner = "cellar-multi-raft-" + "a" * 16 + self.remote = {"owner": owner, "container": owner + "-rustfs", + "context": "colima-cellar-multi-raft", "endpoint": "http://127.0.0.1:9123", + "image_id": "sha256:" + "b" * 64} + self.actual = {"id": "c" * 64, "owner": owner, "image": self.remote["image_id"], + "running": True, "cpus": 10**9, "memory": 2 * 1024**3, + "swap": 2 * 1024**3, + "ports": {"9000/tcp": [{"HostIp": "127.0.0.1", "HostPort": "9123"}]}, + "mounts": [{"Name": owner + "-objects", "Destination": "/data"}]} + self.actions = [] + self.mock = patch("backend_control.command", self.command) + self.mock.start() + self.addCleanup(self.mock.stop) + + def command(self, argv): + if "inspect" in argv: + return json.dumps(self.actual) + action = next(value for value in ("start", "stop") if value in argv) + self.actions.append((action, argv[-1])) + self.actual["running"] = action == "start" + return argv[-1] + + def control(self): + return BackendControl(self.remote, Path(self.root.name) / "control") + + def test_foreign_owner_cannot_be_stopped(self): + self.actual["owner"] = "foreign" + with self.assertRaises(ValueError): + self.control() + self.assertEqual(self.actions, []) + + def test_substituted_container_cannot_be_stopped(self): + control = self.control() + self.actual["id"] = "d" * 64 + with self.assertRaises(ValueError): + control.transition({"format": 1, "sequence": 0, "mode": "materialized", "action": "stop"}) + self.assertEqual(self.actions, []) + + def test_unexpected_operations_never_reach_docker(self): + control = self.control() + valid = {"format": 1, "sequence": 0, "mode": "materialized", "action": "stop"} + for changes in [{"action": "rm"}, {"sequence": 2}, {"sequence": False}, + {"container": "foreign"}, {"mode": "extents"}]: + with self.subTest(changes=changes), self.assertRaises(ValueError): + control.transition({**valid, **changes}) + self.assertEqual(self.actions, []) + + def test_fixed_transitions_preserve_identity_resources_and_mounts(self): + control = self.control() + identity = copy.deepcopy(self.actual) + with patch.object(control, "healthy"): + for sequence, (mode, action) in enumerate(TRANSITIONS): + control.transition({"format": 1, "sequence": sequence, "mode": mode, "action": action}) + self.assertEqual(self.actions, [(action, identity["id"]) for _, action in TRANSITIONS]) + self.assertEqual(self.actual, identity) + self.assertEqual([event["after"]["running"] for event in control.events], + [False, True, False, True]) + changed = copy.deepcopy(self.actual) + self.actual["mounts"] = [] + with self.assertRaises(ValueError): + control.verify(True) + self.actual = changed + + def test_failed_child_keeps_output_and_running_backend(self): + control = self.control() + result = control.run( + [sys.executable, "-c", "print('failure-provenance'); raise SystemExit(9)"], + Path(self.root.name), os.environ, + ) + self.assertEqual(result.returncode, 9) + self.assertIn("failure-provenance", result.stdout) + record = json.loads((control.directory / "record.json").read_text()) + self.assertEqual(record["status"], "FAILED") + self.assertEqual(record["cleanup"], [{"already_running": True}]) + self.assertEqual(self.actions, []) + + +if __name__ == "__main__": + unittest.main() From b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 06:41:27 -0700 Subject: [PATCH 240/568] test: count the registered log inventory --- .../tests/support/cell_group_s3_outage.rs | 45 ++++++++++++++----- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_outage.rs b/crates/cellar/tests/support/cell_group_s3_outage.rs index d328324a0..44c98ab80 100644 --- a/crates/cellar/tests/support/cell_group_s3_outage.rs +++ b/crates/cellar/tests/support/cell_group_s3_outage.rs @@ -9,6 +9,8 @@ use std::path::PathBuf; mod backend_control; const ENTRY_LIMIT: usize = 9; +const OFFLINE_COMMANDS: u64 = 6; +const REFUSED_COMMAND: u64 = OFFLINE_COMMANDS + 1; async fn gateway( h: &Harness, @@ -127,7 +129,28 @@ async fn run( let group = h.fixture.cluster.voters[0] .log .cell_group(&h.route.history)?; - assert_eq!(group.log_retention_usage()?.retained_entries, 4); + // This registered fixture enrolls two cells. Genesis is snapshot index one; + // the retained inventory is one election no-op and two enrollments. + for voter in &h.fixture.cluster.voters { + let selected = voter.log.cell_group(&h.route.history)?; + assert_eq!(selected.first_index()?, 2); + let entries = selected.entries( + selected.first_index()?, + selected.last_index()? + 1, + None, + raft::GetEntriesContext::empty(false), + )?; + assert_eq!( + entries.iter().map(|entry| entry.index).collect::>(), + vec![2, 3, 4] + ); + assert!(entries[0].data.is_empty()); + assert!(entries[1..].iter().all(|entry| !entry.data.is_empty())); + assert_eq!( + selected.log_retention_usage()?.retained_entries, + entries.len() + ); + } let warm = h .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) .await? @@ -215,7 +238,7 @@ async fn run( .is_err() ); let mut latest = warm_index; - for sequence in 3..=5 { + for sequence in 3..=OFFLINE_COMMANDS { let response = h .command( sequence, @@ -231,7 +254,7 @@ async fn run( latest = response.receipt().position.index; drop(response); } - state(&h, latest, 5).await?; + state(&h, latest, OFFLINE_COMMANDS).await?; let usage = group.log_retention_usage()?; assert_eq!( (usage.retained_entries, usage.reserved_entries), @@ -239,7 +262,7 @@ async fn run( ); let refused = h .command( - 6, + REFUSED_COMMAND, b"after-ceiling", Durability::Quorum, Duration::from_secs(15), @@ -253,7 +276,7 @@ async fn run( refused.cause.as_ref(), CellDispatchError::Execution(ExecutionError::Refused(CommandFailure::Overloaded)) )); - assert_eq!(h.calls.load(Ordering::SeqCst), 5); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); initialize_at_capacity(&h).await?; let known = h .command(1, b"warm", Durability::Quorum, Duration::from_secs(15)) @@ -272,7 +295,7 @@ async fn run( error.cause.as_ref(), CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) )); - assert_eq!(h.calls.load(Ordering::SeqCst), 5); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); assert!( !h.controllers[0] .archive_status(h.route.history)? @@ -301,7 +324,7 @@ async fn run( downgraded.cause.as_ref(), CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) )); - assert_eq!(h.calls.load(Ordering::SeqCst), 5); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); let checkpoints = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; let source = h.fixture.runtimes[0] @@ -349,7 +372,7 @@ async fn run( h.fixture .cluster .domain - .blocking(move || dependencies::image_value(image, 5))? + .blocking(move || dependencies::image_value(image, OFFLINE_COMMANDS))? .await??; let image = coverage .recovery() @@ -378,7 +401,7 @@ async fn run( assert_eq!(recovered.durability(), Durability::S3); assert_eq!(recovered.receipt().receipt.result(), b"archive-wait"); drop(recovered); - assert_eq!(h.calls.load(Ordering::SeqCst), 5); + assert_eq!(h.calls.load(Ordering::SeqCst) as u64, OFFLINE_COMMANDS); h.fixture.runtimes[0] .publish_checkpoint(h.route.history, prepared) @@ -412,7 +435,7 @@ async fn run( } let resumed = h .command( - 6, + REFUSED_COMMAND, b"after-ceiling", Durability::Quorum, Duration::from_secs(15), @@ -437,7 +460,7 @@ async fn run( .await? .applied .index; - state(&h, latest, 7).await?; + state(&h, latest, OFFLINE_COMMANDS + 2).await?; archive_controller::published(&h.controllers[0], h.route.history, latest).await?; remote.close().await?; drop(group); From 1159e9358c106fe1260c3479759e909e413257a4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 07:07:53 -0700 Subject: [PATCH 241/568] test: await S3 data readiness after restart --- .../tests/support/cell_group_s3_outage.rs | 57 +++++++++++++++++-- 1 file changed, 53 insertions(+), 4 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_s3_outage.rs b/crates/cellar/tests/support/cell_group_s3_outage.rs index 44c98ab80..2f17bbf99 100644 --- a/crates/cellar/tests/support/cell_group_s3_outage.rs +++ b/crates/cellar/tests/support/cell_group_s3_outage.rs @@ -104,6 +104,48 @@ async fn initialize_at_capacity(h: &Harness) -> Result<()> { Ok(()) } +async fn restored_probe( + h: &Harness, + store: &Arc, + path: &ObjectPath, + expected: &[u8], + deadline: u64, +) -> Result<()> { + let mut last_transient = None; + let result = h + .fixture + .cluster + .domain + .run(async { + cellar::asyncrt::timeout_at(deadline, async { + loop { + let read = async { store.get(path).await?.bytes().await }.await; + match read { + Ok(bytes) => return Ok(bytes), + // Container health can precede S3 data readiness. Retry only + // the idempotent read; absent/authentication failures remain + // failures and no operation receives a new deadline. + Err(error @ object_store::Error::Generic { .. }) => { + last_transient = Some(error) + } + Err(error) => return Err(error), + } + cellar::asyncrt::sleep(Duration::from_millis(5)).await; + } + }) + .await + }) + .await?; + let bytes = match result { + Ok(read) => read?, + Err(_) => anyhow::bail!( + "S3 data readiness exhausted the original invocation deadline: {last_transient:?}" + ), + }; + assert_eq!(bytes.as_ref(), expected); + Ok(()) +} + async fn run( rooted: bool, store: Arc, @@ -354,10 +396,17 @@ async fn run( assert_eq!(group.log_retention_usage()?.retained_entries, ENTRY_LIMIT); control.set_running(&h.fixture.cluster.domain, true).await?; - assert_eq!( - global.get(&probe).await?.bytes().await?.as_ref(), - probe_bytes - ); + restored_probe( + &h, + &global, + &probe, + &probe_bytes, + remote_invocation + .attempt() + .context("retained remote attempt")? + .deadline_ms(), + ) + .await?; archive_controller::published(&h.controllers[0], h.route.history, latest).await?; let coverage = archive_controller::verified_current(&h.fixture, &h.archive, rooted, "outage-recovery") From 8e5ca2fc9826cbe4ec35c327496cc653a5d408ee Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 07:37:28 -0700 Subject: [PATCH 242/568] docs: qualify bounded quorum during S3 outage --- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 103 +++++++++--------- .../design/multi-raft/cell-groups/progress.md | 91 ++++++++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 6 +- 4 files changed, 151 insertions(+), 55 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 31b5e4e02..13f86720e 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8", - "runtime_revision": "dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending.", + "source_revision": "1159e9358c106fe1260c3479759e909e413257a4", + "runtime_revision": "1159e9358c106fe1260c3479759e909e413257a4", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 1dbd9c4ff..51c1869e4 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -16,57 +16,58 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). The latest runtime/qualification source is -`dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8`. CG-06 remains IN_PROGRESS. The -original complete-group dependency case now holds or corrupts an actual B image -chunk/extent outside A's entire image closure, after a real B mutation. Later A -S3 settlement cannot complete; independent recovery is pending or fails with the -exact digest error and the registry selection stays at the preceding cut. Quorum -successors and all three voter data remain healthy. Delay release returns the -exact S3 result; repairing injected bytes permits fresh independent data -recovery but does not silently restart the failed archive owner. Original -receipts resolve without callbacks and retain their S3 minimum. - -The fixture persists and fsyncs the independent recovery client's acquire intent -before CAS, retains its durable hold across controller head retirement, and -releases physical receiver/tail ownership before later work. Existing controller -quotas, timeouts, production behavior and marker-1 records are unchanged. The -reviewed scope extension adds the owning test helper and its common exports; -original acceptance cases and invariants remain intact. - -Nineteen exact real RustFS controller cases pass in both image forms with seven -signed probes. Twenty-two regression/check commands pass with 269 Rust cases, -seven copied executables and 2576 source hashes. All four shared-group native -suites, all 22 storage cases, locked all-target Clippy, format/layout, -dependency/docs/preflight/plan validation, 37 Python tests, system-Python -hashing and independent source/artifact/selection/credential audit pass. Current -evidence is under `.cellar/cell-groups-evidence/CG-06-archive-dependencies/`. -Preceding physical-reclamation source 7741e404 and its evidence remain retained, -including actual old-file removal, reader/checkpoint/sibling preservation, -all-three retained-disk reopen, exact receipts and new TLS writes. The physical -case also passes at the new source. Earlier source-bound failures and the -unresolved 8dfe9ff3 slow-reader timing limitation remain retained. - -Three original CG-06 cases pass: whole-group dependencies, archive tails and -physical reclaim/restart. Two original cases and the full task gate remain -pending: bounded Quorum progress against a fully stopped S3 backend, and fenced -fresh-incarnation serving from a replacement quorum. The next step is to test -existing local cached and remote fresh-discovery paths separately, then -implement the missing bounded-outage behavior without weakening catalog/epoch -authority. Seal/Install/Activate codecs exist, but live apply still rejects -them; source sealing and exact publication-bound activation must precede any new -cached-authority shortcut. The preceding private physical-reclaim evidence -retains the source-bound outage-authority investigation. - -Fenced recovery must inventory every possible old voter, including joins after -the selected archive cut, through the accepted fencing-authority API plus Docker -qualification. CG-08 effect/cache/result lifetimes precede CG-07 public -SDK/fleet cutover, followed by transitions, split/merge, placement, hibernation, -managed recovery, measured load and final legacy removal. The local RustFS -matrix uses host-native voters and one VM. It does not qualify full-backend -outage bounds, independent failure domains, deleted-voter fresh-incarnation -serving, sustained throughput, global physical disk bounds or the final hard -cutover. Preserve all earlier evidence and the user's untracked diagram. The -full end-to-end goal remains active and incomplete. +1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. Its +original bounded full-S3-backend outage case now passes in both image forms. The +normal case faults the adapter shared by catalog and archive. The real RustFS +case stops and restarts the owning runner's actual backend container, preserving +its pinned identity, owner, image, volume, resource limits and endpoint. Actual +S3 probe I/O fails during stop and recovers after start; no real-mode wrapper +fault substitutes for that outage. + +Warm local Quorum commands progress to the explicit nine-entry retention ceiling +while fresh remote discovery refuses without executing a callback. S3 commands +may commit locally but cannot return S3, and stored minima survive downgrade +attempts. At capacity, callbacks/initialization refuse before execution and +known receipts remain available. Independent complete archive recovery precedes +publication of the held exact checkpoint. Every voter then selects its complete +cut and retires actual debt, allowing new work. A failed remote invocation +retries with unchanged captured inputs, identity, deadline and budget. All three +voter images and logical versions remain exact; eight callbacks occur once. + +Twenty exact real RustFS cases pass in both image forms, with seven signed +probes and four owner-checked backend stop/start transitions. Twenty-three +checks pass with 270 Rust cases, seven copied executables and 2580 source +hashes. All four native shared-group suites, all 22 storage cases, locked +all-target Clippy, format/layout, dependency/docs/preflight/plan validation, 37 +plan tests, five controller tests, system-Python hashing and independent +source/artifact/selection/credential/backend-transition audit pass. Evidence is +under .cellar/cell-groups-evidence/CG-06-full-backend-outage/. The first-source +inventory mismatch and copied executable reproduction remain retained; the +correction verifies actual indexes 2, 3 and 4 on all voters and fills the +unchanged entry ceiling with one additional real command. Earlier dependency and +physical-reclamation qualifications and the unresolved 8dfe9ff3 slow-reader +timing limitation remain retained. + +Four original CG-06 cases pass: whole-group dependencies, bounded S3 outage, +archive tails and physical reclaim/restart. The remaining original case and full +task gate require fenced unmanaged recovery into fresh group/cell incarnations +and a serving replacement quorum. Inventory every possible old voter, including +joins after the selected cut. Use the accepted fencing-authority API plus Docker +qualification, permanently prevent old-host restart, preserve selected-cut +results/receipts, reject old tokens, initialize new histories and require fresh +quorum before serving. + +CG-08 effect/cache/result lifetimes precede CG-07 public SDK/fleet cutover. +Irreversible source sealing and activation, split/merge, placement, hibernation, +managed recovery, real load/crash qualification and final legacy removal follow. +Do not add a new cached-authority shortcut: live Seal/Install/Activate controls +still need their durable transition lifecycle and exact publication proofs. +Current outage evidence covers existing static groups, warm local routes and +fresh remote refusals. Host-native voters share one machine and RustFS one VM; +global physical disk bounds, sustained throughput, independent domains and +deleted-voter fresh-incarnation disaster serving remain unqualified. Preserve +all predecessor evidence, excluded MR-13/MR-14 progress basenames and the user's +untracked diagram. The full sixteen-task goal remains active and incomplete. CG-01 through CG-05 are DONE at their declared native task boundaries. CG-05 closes at `e0550acd` with all five exact snapshot cases and eighteen passing diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1120315d4..1af84ec50 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,97 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 bounded Quorum progress during a full S3 backend outage + +Source: 1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. The +original quorum_continues_bounded_s3_outage case now runs in both image forms, +alongside an exact real RustFS counterpart. It exercises existing static-group +local-quorum durability, routing, owned SQL, archive settlement and reclamation +without changing production behavior or introducing a cached-authority shortcut. + +Each scenario explicitly selects a nine-entry group ceiling with one progress +slot. All three voters first verify the actual retained indexes 2, 3 and 4: one +election no-op and two enrollments after genesis snapshot index 1. A warm local +command and remote receipt retry establish current routes and archived state. +The fixture then disables the adapter shared by catalog and archive; the RustFS +counterpart instead requests actual Docker stop/start from its owning runner, +never toggling that adapter's outage flag. Actual S3 probe I/O fails while the +backend is stopped and returns the same probe bytes after restart. + +Warm local commands continue through the actual quorum. A later S3 command +commits locally but cannot settle; four subsequent Quorum mutations reach the +exact retained-entry ceiling, preserving every A/B value and logical version on +all three voters. The next command and initialization refuse before their +callbacks. A known Quorum receipt remains available at capacity. S3 output +expires with its original deadline, and a retry asking for Quorum still obeys +the stored S3 minimum. The healthy group is not quarantined. + +A separate, already-warmed remote receiver still performs fresh catalog GETs +during the outage and refuses before its callback. The same originating +invocation survives that error and later retries with its exact captured inputs, +identity, original deadline and attempt budget. No failed response becomes a +new-command or non-commit proof. + +An exact prepared checkpoint cannot publish beyond verified coverage while S3 is +stopped; the selected checkpoint, retained prefix and debt remain unchanged. +Restart uses the same RustFS container, image, volume, endpoint and resource +limits. The durable journal catches up, independent complete recovery verifies +both A and B, and the original failed S3 command returns its exact result +without callback replay. The completed verifier releases its one-source pool +before subsequent settlement. The same prepared checkpoint then selects +successfully; every voter selects its exact complete cut, retires actual debt to +zero and releases physical source quota before reuse. New local work and the +retained remote invocation commit, with eight total callbacks and the expected +voter state. + +The narrow Python controller accepts only four ordered stop/start transitions +for the two image modes. It checks the captured container ID, owner label, +pinned image, CPU/memory, port bindings and mounts before and after every +operation, uses explicit Docker context, and retains actual state observations. +Requests/replies are bounded, atomic local files. The native fixture performs no +Docker process calls. Failed children are joined and preserve output; cleanup +restores only the same verified owned backend. Five protocol/ownership tests +reject foreign or substituted containers and unexpected actions, verify the +fixed transitions, and retain failed-child output. + +Twenty exact real RustFS cases pass in both image forms with seven signed +protocol probes and four recorded actual backend transitions. Twenty-three +regression/check commands pass with 270 Rust cases, seven copied executables and +2580 source hashes. All four shared-group native suites, all 22 storage cases, +locked all-target Cellar/logic/LTX Clippy, format/layout, +dependency/docs/preflight/plan checks, 37 plan tests, five controller tests and +system-Python hashing pass. Source/artifact/selection/credential and +backend-transition audit passes under +.cellar/cell-groups-evidence/CG-06-full-backend-outage/. + +The original first-source failure and copied-binary reproduction remain +retained: an expected count of four came from the older three-cell fixture, +whereas registered setup enrolls two cells. The successor asserts the physical +log inventory on every voter and adds one real mutation to reach the unchanged +nine-entry ceiling. No original oracle, quota, deadline, payload or production +behavior was weakened. The first real stop/start run and a copied-binary run on +a fresh backend both return503 for the first probe GET after health200. The +successor retries only idempotent Generic read failures within the retained +remote invocation original deadline; missing/authentication failures still +refuse and exact data must match before proceeding. The outage adapter remains +noRetry and no mutation retry or production behavior changes. Both failed real +traces, container observations, copied executable and stopped backend data +remain retained. Earlier qualifications and the unresolved 8dfe9ff3 slow-reader +timing limitation remain retained. + +Reflection: four original CG-06 cases now pass: complete-group dependencies, +bounded S3 outage, archive tails and physical reclaim/restart. The remaining +original case is fenced_group_seed_new_incarnations: permanent old-host fencing, +complete old-voter inventory including later joins, fresh group/cell histories, +exact selected-cut results, old-token rejection and a serving replacement +quorum. Use the accepted fencing-authority API plus Docker qualification. The +full CG-06 gate is still pending. CG-08 lifetimes precede CG-07 production +SDK/fleet cutover, followed by irreversible transitions, split/merge, placement, +hibernation, managed recovery, measured load and final legacy removal. This is +bounded static-group outage qualification, not a throughput benchmark, +independent-domain proof, global physical disk bound, voter-disk erasure or +completed hard cutover. + ## CG-06 complete-group dependency acknowledgement Source: `dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8`. CG-06 remains IN_PROGRESS. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 1fd7b7c43..b62ba67a9 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1775,7 +1775,11 @@ "crates/cellar/tests/support/cell_group_s3_remote.rs", "crates/cellar/tests/support/cell_group_s3_remote_wire.rs", "crates/cellar/durability/raft/cell_groups/archive_compaction.rs", - "crates/cellar/tests/support/cell_group_archive_dependencies.rs" + "crates/cellar/tests/support/cell_group_archive_dependencies.rs", + "crates/cellar/tests/support/cell_group_s3_outage.rs", + "crates/cellar/tests/support/cell_group_backend_control.rs", + "tools/multi-raft/backend_control.py", + "tools/multi-raft/test_backend_control.py" ], "test_target": "native_cell_groups_archive", "steps": [ From 626621214f6b92d49a081b69ff0a3d05166fcd89 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 08:21:13 -0700 Subject: [PATCH 243/568] feat: fence exact cell-group recovery intents Bind recovery to a verified complete archive and serialize its permanent fence with topology and fleet import. Resume only the same intent and independently verify the winning receipt. This barrier grants data and fencing proofs only. Fresh-genesis seeding, receipt rebinding, activation and replacement-quorum serving remain required before disaster recovery is complete. --- crates/cellar/durability/raft/catalog.rs | 5 +- .../durability/raft/catalog_disaster.rs | 9 +- .../raft/cell_groups/archive/replay.rs | 39 ++ .../raft/cell_groups/directory/archive.rs | 54 +- .../raft/cell_groups/directory/birth.rs | 21 +- .../raft/cell_groups/directory/mod.rs | 12 + .../cell_groups/directory/recovery/mod.rs | 336 +++++++++++++ .../cell_groups/directory/recovery/request.rs | 173 +++++++ .../raft/cell_groups/directory/wire.rs | 11 + .../cellar/durability/raft/cell_groups/mod.rs | 5 +- crates/cellar/durability/raft/member_state.rs | 6 + .../tests/native_cell_groups_archive.rs | 46 ++ .../tests/support/cell_group_archive_store.rs | 23 + .../support/cell_group_disaster_fence.rs | 464 ++++++++++++++++++ .../multi-raft/cell-groups/baseline.json | 2 +- .../multi-raft/cell-groups/protocols.md | 25 + docs/design/multi-raft/cell-groups/tasks.json | 5 +- 17 files changed, 1197 insertions(+), 39 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs create mode 100644 crates/cellar/tests/support/cell_group_disaster_fence.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 0bf9e85ec..554f21caf 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -27,10 +27,11 @@ mod range_directory; pub(crate) use range_directory::RouteCache; pub use range_directory::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, + CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryError, + CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, - PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, + FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/catalog_disaster.rs b/crates/cellar/durability/raft/catalog_disaster.rs index 4584f77fe..db7782907 100644 --- a/crates/cellar/durability/raft/catalog_disaster.rs +++ b/crates/cellar/durability/raft/catalog_disaster.rs @@ -48,15 +48,12 @@ impl DisasterFenceRequest { /// controller and bound to `request.digest()`. Fail closed on partial/unknown /// outcomes. Fences are permanent; retries must be idempotent and cancellation /// must never undo a fence. Cellar supplies no permissive default implementation. -pub trait DisasterFenceAuthority: Send + Sync { - fn fence<'a>( - &'a self, - request: &'a DisasterFenceRequest, - ) -> BoxFuture<'a, anyhow::Result>>; +pub trait DisasterFenceAuthority: Send + Sync { + fn fence<'a>(&'a self, request: &'a Request) -> BoxFuture<'a, anyhow::Result>>; fn verify<'a>( &'a self, - request: &'a DisasterFenceRequest, + request: &'a Request, evidence: &'a [u8], ) -> BoxFuture<'a, anyhow::Result<()>>; } diff --git a/crates/cellar/durability/raft/cell_groups/archive/replay.rs b/crates/cellar/durability/raft/cell_groups/archive/replay.rs index 5e92940d5..2f7cc1068 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/replay.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/replay.rs @@ -40,6 +40,45 @@ impl VerifiedCellGroupArchiveRecovery { pub fn membership(&self) -> &GroupMembership { &self.membership } + + /// Static unmanaged recovery only. A fence controller must independently + /// exclude later joins and retired hosts from its deployment inventory. + pub(crate) fn check_disaster_target( + &self, + target: &crate::raft::catalog::CellGroupBirthRequest, + ) -> Result<()> { + if !self.configuration.voters_outgoing.is_empty() + || !self.configuration.learners_next.is_empty() + || self.configuration.auto_leave + || self + .membership + .repair_status() + .map_err(CellGroupCheckpointError::from)? + .is_some() + || self.membership.fleet_status().is_some() + { + return Err(CellGroupArchiveError::Invalid( + "recovery requires static unmanaged membership", + )); + } + for member in target.voters().iter().chain(target.learners()) { + if member.member.get() <= self.membership.high_water() + || self.membership.has_node_binding(member.node) + { + return Err(CellGroupArchiveError::Invalid( + "reused recovery member or node", + )); + } + } + self.root.images.visit(&mut |_, leaf, _, _, _| { + if leaf.version.history.incarnation == target.route().history.incarnation { + return Err(CellGroupArchiveError::Invalid( + "reused recovery cell incarnation", + )); + } + Ok(()) + }) + } } pub(super) fn replay( diff --git a/crates/cellar/durability/raft/cell_groups/directory/archive.rs b/crates/cellar/durability/raft/cell_groups/directory/archive.rs index 801c68cda..bbc72de1f 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/archive.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/archive.rs @@ -179,33 +179,41 @@ impl RangeDirectory { domain .run(async { self.require_current_group_birth(birth).await?; - let bytes = self - .bounded(async { - self.namespace - .inner - .read_versioned_bounded( - &self.archive_binding_key(birth.route()), - wire::LIMIT, - ) - .await? - .map(|(bytes, _)| bytes) - .ok_or(CatalogError::Corrupt("missing group archive binding")) - }) - .await?; - let binding: Binding = record::decode(wire::GROUP_ARCHIVE, &bytes, wire::LIMIT)?; - if binding.namespace != self.namespace.root_digest() - || binding.creation.route() != birth.route() - { - return Err(CatalogError::Corrupt("group archive binding identity").into()); - } - let registry = self - .group_archive(pool, birth.route())? - .open_registry(domain, &binding.creation) - .await?; + let registry = self.open_recovery_archive(domain, pool, birth).await?; self.require_current_group_birth(birth).await?; Ok(registry) }) .await .map_err(|_| CellGroupArchiveError::Stopped)? } + + // The recovery owner checks the exact intent, selected source, immutable + // birth and topology lock. Ordinary archive admission remains fenced. + pub(super) async fn open_recovery_archive( + &self, + domain: &impl CellGroupArchiveExecution, + pool: &CellGroupArchivePool, + birth: &CellGroupBirth, + ) -> Result { + let bytes = self + .bounded(async { + self.namespace + .inner + .read_versioned_bounded(&self.archive_binding_key(birth.route()), wire::LIMIT) + .await? + .map(|(bytes, _)| bytes) + .ok_or(CatalogError::Corrupt("missing group archive binding")) + }) + .await?; + let binding: Binding = record::decode(wire::GROUP_ARCHIVE, &bytes, wire::LIMIT)?; + if binding.namespace != self.namespace.root_digest() + || binding.creation.route() != birth.route() + { + return Err(CatalogError::Corrupt("group archive binding identity").into()); + } + Ok(self + .group_archive(pool, birth.route())? + .open_registry(domain, &binding.creation) + .await?) + } } diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs index 5cd02aa4b..167570d12 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/birth.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -52,8 +52,8 @@ impl CellGroupBirthRequest { /// This grants neither membership nor permission to initialize a blank disk. #[derive(Clone)] pub struct CellGroupBirth { - request: CellGroupBirthRequest, - owner: Arc<()>, + pub(super) request: CellGroupBirthRequest, + pub(super) owner: Arc<()>, } impl CellGroupBirth { @@ -191,7 +191,7 @@ impl RangeDirectory { )) } - fn group_birth_key(&self, route: &GroupRoute) -> Path { + pub(super) fn group_birth_key(&self, route: &GroupRoute) -> Path { self.key(&format!( "births/{}/{}", hex::encode(route.history.group.bytes()), @@ -244,7 +244,20 @@ impl RangeDirectory { async fn selected_birth_route(&self, route: &GroupRoute) -> Result<(), CatalogError> { let (view, _) = self.current().await?; - let mut next = Some(view.head.tree); + if let Some(lock) = &view.head.recovery { + if lock.source_route(self)? == *route { + return Err(CatalogError::Fenced); + } + } + self.selected_route_in(&view, route).await + } + + pub(super) async fn selected_route_in( + &self, + view: &DirectoryView, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + let mut next = Some(view.head.tree.clone()); for _ in 0..32 { let reference = next.take().ok_or(CatalogError::Fenced)?; let page = self.page(&reference).await?; diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index c3c82c075..c4a852f39 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -21,6 +21,11 @@ mod archive; pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding}; mod birth; mod bootstrap; +mod recovery; +pub use recovery::{ + CellGroupDisasterFenceRequest, CellGroupRecoveryError, CellGroupRecoveryRequest, + FencedCellGroupRecovery, +}; mod route_cache; mod routing; pub use route_cache::CellRouteCacheUsage; @@ -295,6 +300,9 @@ impl RangeDirectory { { return Err(CatalogError::Corrupt("directory head")); } + if let Some(recovery) = &head.recovery { + recovery.check(self)?; + } head.tree.require_complete(self) } @@ -479,6 +487,9 @@ impl RangeDirectory { } // An interrupted genesis resumes only its identical first head. } + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } if request .expected .is_some_and(|expected| expected != view.digest) @@ -546,6 +557,7 @@ impl RangeDirectory { operations, groups, journal: id, + recovery: None, }; self.check_head(&head)?; let bytes = record::encode(wire::HEAD, &head, wire::LIMIT)?; diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs new file mode 100644 index 000000000..d85a3ec92 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs @@ -0,0 +1,336 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact, irreversible disaster fencing before fresh-genesis installation. +//! The returned data proof grants no bootstrap, voting or serving permission. +mod request; +use super::*; +use crate::raft::{ + catalog::DisasterFenceAuthority, + cell_groups::{ + CellGroupArchiveBindingError, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCoverage, + CellGroupArchiveError, CellGroupArchiveExecution, CellGroupArchivePool, + CellGroupArchiveTailPool, + }, +}; +pub use request::{CellGroupDisasterFenceRequest, CellGroupRecoveryRequest}; +use serde::{Deserialize, Serialize}; + +#[derive(Debug)] +pub enum CellGroupRecoveryError { + Catalog(CatalogError), + Archive(CellGroupArchiveError), +} + +impl std::fmt::Display for CellGroupRecoveryError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "cell group recovery: {self:?}") + } +} + +impl std::error::Error for CellGroupRecoveryError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Catalog(error) => error, + Self::Archive(error) => error, + }) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CatalogError) -> Self { + Self::Catalog(error) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CellGroupArchiveError) -> Self { + Self::Archive(error) + } +} + +impl From for CellGroupRecoveryError { + fn from(error: CellGroupArchiveBindingError) -> Self { + match error { + CellGroupArchiveBindingError::Catalog(error) => Self::Catalog(error), + CellGroupArchiveBindingError::Archive(error) => Self::Archive(error), + } + } +} + +/// A topology-head CAS closes old startup and all competing topology mutations +/// atomically. It never stands in for stopping a running deployment. +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::raft::catalog::range_directory) struct RecoveryLock { + request: [u8; 32], + base: [u8; 32], + source: wire::Route, +} + +impl RecoveryLock { + pub(super) fn check(&self, directory: &RangeDirectory) -> Result<(), CatalogError> { + if self.request == [0; 32] || self.base == [0; 32] { + return Err(CatalogError::Corrupt("group recovery lock")); + } + self.source_route(directory).map(|_| ()) + } + + pub(in crate::raft::catalog::range_directory) fn source_route( + &self, + directory: &RangeDirectory, + ) -> Result { + self.source.decode(directory) + } +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Complete { + request: [u8; 32], + evidence: Vec, +} + +/// A complete selected archive and independently verified permanent fence. +/// Retain its dedicated remote hold until recovery and entered work finish. +/// This deliberately cannot be supplied to ordinary bootstrap or live apply. +pub struct FencedCellGroupRecovery { + request: CellGroupRecoveryRequest, + coverage: CellGroupArchiveCoverage, +} + +impl FencedCellGroupRecovery { + pub fn request(&self) -> &CellGroupRecoveryRequest { + &self.request + } + + pub fn coverage(&self) -> &CellGroupArchiveCoverage { + &self.coverage + } +} + +impl RangeDirectory { + fn recovery_key(&self, request: &CellGroupRecoveryRequest, suffix: &str) -> Path { + self.key(&format!( + "recovery/{}/{suffix}", + hex::encode(request.digest()) + )) + } + + async fn recovery_source( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result { + let (view, _) = self.current().await?; + self.check_recovery_view(&view, request)?; + if Index::new(self) + .lookup( + view.head.operations, + &hex::encode(request.operation().bytes()), + ) + .await? + .is_some() + { + return Err(CatalogError::Conflict( + "group recovery operation already published", + )); + } + self.selected_route_in(&view, request.source().route()) + .await?; + self.verify_birth_reservation(request.source().route()) + .await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(request.source().route()), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery source birth"))? + .0; + if bytes != request.source().encode() { + return Err(CatalogError::Corrupt("recovery source birth binding")); + } + Ok(CellGroupBirth { + request: request.source().clone(), + owner: self.namespace.inner.owner.clone(), + }) + } + + fn check_recovery_view( + &self, + view: &DirectoryView, + request: &CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + match &view.head.recovery { + Some(lock) + if lock.request == request.digest() + && Some(lock.base) == request.mutation().request.expected + && lock.source_route(self)? == *request.source().route() => + { + Ok(()) + } + Some(_) => Err(CatalogError::Conflict( + "different group recovery owns topology", + )), + None if Some(view.digest) == request.mutation().request.expected => Ok(()), + None => Err(CatalogError::Conflict("stale group recovery topology")), + } + } + + async fn lock_recovery(&self, request: &CellGroupRecoveryRequest) -> Result<(), CatalogError> { + self.bounded(async { + let (mut view, version) = self.current().await?; + self.check_recovery_view(&view, request)?; + if view.head.recovery.is_some() { + return Ok(()); + } + self.selected_route_in(&view, request.source().route()) + .await?; + view.head.recovery = Some(RecoveryLock { + request: request.digest(), + base: view.digest, + source: wire::Route::encode(request.source().route(), self)?, + }); + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + // A lost response can follow a committed CAS. Observe the exact + // lock; never issue another request or rebase onto a newer head. + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + self.check_recovery_view(&after, request)?; + if after.head.recovery.is_none() { + return Err(CatalogError::Unconfirmed); + } + Ok(()) + }) + .await + } + + /// Verify the exact held closure, serialize recovery with fleet import and + /// topology, then permanently fence every possible old writer. Persist + /// `request.encode()` before calling. A failed/unknown outcome is resumable + /// ONLY with those bytes; there is no abort or unfence operation. + /// + /// The authority must use external deployment/launch inventory, including + /// later joins, removed hosts, routers and effects; independently verify + /// evidence bound to the request digest and exclude every replacement node + /// from the old lineage. A marker or caller assertion cannot satisfy it. + /// This static unmanaged boundary does not publish a replacement or seed a + /// Raft engine. Fresh genesis, restored histories and quorum remain required. + pub async fn fence_group_recovery( + &self, + domain: &impl CellGroupArchiveExecution, + archives: &CellGroupArchivePool, + tails: &CellGroupArchiveTailPool, + request: &CellGroupRecoveryRequest, + receiver: CellGroupArchiveCheckpointReceiver, + authority: &dyn DisasterFenceAuthority, + ) -> Result { + domain + .run(async { + let request = self.decode_group_recovery(request.encode())?; + let birth = self.bounded(self.recovery_source(&request)).await?; + let registry = self.open_recovery_archive(domain, archives, &birth).await?; + // No inventory claim, topology lock or infrastructure action until + // every cell/base/control dependency has been physically verified. + let coverage = registry + .verify_hold(domain, request.hold(), tails, receiver) + .await?; + coverage + .recovery() + .check_disaster_target(request.replacement())?; + self.namespace.claim_disaster_namespace().await?; + self.reserve_group_birth(request.replacement()).await?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_key(&request, "intent"), + request.encode(), + wire::LIMIT, + ) + .await?; + if observed != request.encode() { + return Err(CatalogError::Conflict("group recovery intent")); + } + Ok(()) + }) + .await?; + self.lock_recovery(&request).await?; + let fence = CellGroupDisasterFenceRequest::new(request); + let complete_key = self.recovery_key(fence.recovery(), "fence"); + let existing = self + .bounded(async { + Ok(self + .namespace + .inner + .read_versioned_bounded(&complete_key, wire::LIMIT) + .await? + .map(|v| v.0)) + }) + .await?; + let bytes = if let Some(bytes) = existing { + bytes + } else { + let evidence = authority + .fence(&fence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + check_evidence(&evidence)?; + authority + .verify(&fence, &evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + let bytes = record::encode( + wire::GROUP_FENCE, + &Complete { + request: fence.digest(), + evidence, + }, + wire::LIMIT, + )?; + self.bounded(self.namespace.inner.create_and_observe_bounded( + &complete_key, + &bytes, + wire::LIMIT, + )) + .await? + }; + // Concurrent controllers may return different valid evidence. Verify + // the winning durable receipt independently, including on reopen. + let complete: Complete = record::decode(wire::GROUP_FENCE, &bytes, wire::LIMIT)?; + if complete.request != fence.digest() { + return Err(CatalogError::Corrupt("group fence request binding").into()); + } + check_evidence(&complete.evidence)?; + authority + .verify(&fence, &complete.evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string()))?; + self.bounded(self.recovery_source(fence.recovery())).await?; + Ok(FencedCellGroupRecovery { + request: fence.into_recovery(), + coverage, + }) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +fn check_evidence(evidence: &[u8]) -> Result<(), CatalogError> { + if evidence.is_empty() || evidence.len() > 4096 { + return Err(CatalogError::Invalid("group fence evidence length")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs new file mode 100644 index 000000000..c7ced2b2c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs @@ -0,0 +1,173 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Persistable operator intent. No mutable-head selection or authority is +//! reconstructed from a fence receipt or an untrusted archive locator. +use super::*; +use crate::raft::cell_groups::CellGroupArchiveHold; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Request { + namespace: [u8; 32], + source: Vec, + replacement: Vec, + mutation: Vec, + hold: CellGroupArchiveHold, +} + +/// Persist before submission, including the dedicated remote hold. Never +/// release that hold while the intent or any recovery user remains live. +/// Replacement cell histories use the replacement group's fresh incarnation; +/// old witnesses remain provenance, not authority in the replacement history. +pub struct CellGroupRecoveryRequest { + bytes: Vec, + source: CellGroupBirthRequest, + replacement: CellGroupBirthRequest, + mutation: DirectoryMutation, + hold: CellGroupArchiveHold, +} + +impl CellGroupRecoveryRequest { + pub fn encode(&self) -> &[u8] { + &self.bytes + } + + pub fn digest(&self) -> [u8; 32] { + record::digest(&self.bytes) + } + + pub fn source(&self) -> &CellGroupBirthRequest { + &self.source + } + + pub fn replacement(&self) -> &CellGroupBirthRequest { + &self.replacement + } + + pub fn hold(&self) -> &CellGroupArchiveHold { + &self.hold + } + + pub fn operation(&self) -> ProposalId { + self.mutation.operation() + } + + pub(super) fn mutation(&self) -> &DirectoryMutation { + &self.mutation + } +} + +/// Whole-group binding for `DisasterFenceAuthority`. It is created only after +/// complete archive verification and a durable topology/namespace claim. +pub struct CellGroupDisasterFenceRequest { + recovery: CellGroupRecoveryRequest, +} + +impl CellGroupDisasterFenceRequest { + pub fn recovery(&self) -> &CellGroupRecoveryRequest { + &self.recovery + } + + pub fn digest(&self) -> [u8; 32] { + self.recovery.digest() + } + + pub(super) fn new(recovery: CellGroupRecoveryRequest) -> Self { + Self { recovery } + } + + pub(super) fn into_recovery(self) -> CellGroupRecoveryRequest { + self.recovery + } +} + +impl RangeDirectory { + /// Prepare one exact static-group rollback. This allocates no metadata and + /// grants no fence, startup or serving permission. The hold must cover the + /// selected birth; callers explicitly choose its recovery point. + pub fn prepare_group_recovery( + &self, + view: &DirectoryView, + operation: ProposalId, + source: &CellGroupBirth, + replacement: &CellGroupBirthRequest, + hold: CellGroupArchiveHold, + ) -> Result { + self.unmanaged_birth()?; + self.check_view(view)?; + if !Arc::ptr_eq(&source.owner, &self.namespace.inner.owner) { + return Err(CatalogError::Conflict("foreign recovery source birth")); + } + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + let mutation = self.prepare_replacement( + view, + operation, + std::slice::from_ref(source.route()), + std::slice::from_ref(replacement.route()), + )?; + let bytes = record::encode( + wire::GROUP_RECOVERY, + &Request { + namespace: self.namespace.root_digest, + source: source.request.encode().to_vec(), + replacement: replacement.encode().to_vec(), + mutation: mutation.encode().to_vec(), + hold, + }, + wire::LIMIT, + )?; + self.decode_group_recovery(&bytes) + } + + /// Decode exactly the persisted intent under this namespace. A new catalog + /// owner may resume it; it cannot reinterpret or rebase its selected cut. + pub fn decode_group_recovery( + &self, + bytes: &[u8], + ) -> Result { + self.unmanaged_birth()?; + let value: Request = record::decode(wire::GROUP_RECOVERY, bytes, wire::LIMIT)?; + if value.namespace != self.namespace.root_digest { + return Err(CatalogError::Conflict("group recovery namespace")); + } + let source = self.decode_group_birth(&value.source)?; + let replacement = self.decode_group_birth(&value.replacement)?; + let mutation = self.decode_mutation(&value.mutation)?; + let old = source.route(); + let new = replacement.route(); + if mutation.request.expected.is_none() + || mutation.request.parents.len() != 1 + || mutation.request.children.len() != 1 + || mutation.request.parents[0].decode(self)? != *old + || mutation.request.children[0].decode(self)? != *new + || value.hold.marker().route() != old + || old.range != new.range + || old.history.group == new.history.group + || old.history.incarnation == new.history.incarnation + || source.creation() == replacement.creation() + { + return Err(CatalogError::Conflict("group recovery identities or cut")); + } + for member in replacement.voters().iter().chain(replacement.learners()) { + if source + .voters() + .iter() + .chain(source.learners()) + .any(|old| old.node == member.node || old.member == member.member) + { + return Err(CatalogError::Conflict( + "reused initial recovery member or node", + )); + } + } + Ok(CellGroupRecoveryRequest { + bytes: bytes.to_vec(), + source, + replacement, + mutation, + hold: value.hold, + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 86f2e8dec..176c33d7f 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -13,6 +13,8 @@ pub(super) const REQUEST: u8 = 85; pub(super) const GROUP_BIRTH: u8 = 186; pub(super) const GROUP_CLAIM: u8 = 187; pub(super) const GROUP_ARCHIVE: u8 = 188; +pub(super) const GROUP_RECOVERY: u8 = 189; +pub(super) const GROUP_FENCE: u8 = 190; pub(super) const LIMIT: usize = 64 * 1024; #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -205,6 +207,15 @@ pub(super) struct Head { pub operations: Option<[u8; 32]>, pub groups: Option<[u8; 32]>, pub journal: [u8; 32], + #[serde(deserialize_with = "required_recovery")] + pub recovery: Option, +} + +// Marker 1 has one record shape. Missing fields are not an older-reader mode. +fn required_recovery<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) } #[derive(Serialize, Deserialize, PartialEq, Eq)] diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 38ce2de26..952cfbedc 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -102,10 +102,11 @@ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsag pub use crate::raft::catalog::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, + CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryError, + CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, - PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, + FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index 3d58d5b65..4b74b63bb 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -125,6 +125,12 @@ impl GroupMembership { self.high_water } + pub(crate) fn has_node_binding(&self, node: NodeId) -> bool { + self.bindings + .iter() + .any(|binding| binding.node == node.bytes()) + } + pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { if self.validate_scope(&replica.scope).is_err() { return false; diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 2434864c8..4fe352fe8 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -14,6 +14,8 @@ mod binding_fixture; #[path = "support/cell_group_checkpoint.rs"] #[allow(dead_code)] mod checkpoint_fixture; +#[path = "support/cell_group_disaster_fence.rs"] +mod disaster_fixture; #[path = "support/raft_storage_fault.rs"] #[allow(dead_code)] mod fault; @@ -24,6 +26,8 @@ mod images; mod isolation_fixture; #[path = "support/cell_group_archive_publication.rs"] mod publication_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; #[path = "support/cell_group_archive_tail.rs"] mod tail_fixture; @@ -33,6 +37,48 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_fence_resumes_exact_intent() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::resume( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_topology_race_refuses_stale_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::topology_wins( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_fence_intent_and_topology() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::resume( + rooted, + rustfs::store("recovery-fence-resume", rooted)?, + )) + .await?; + Box::pin(disaster_fixture::topology_wins( + rooted, + rustfs::store("recovery-fence-race", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn quorum_continues_bounded_s3_outage() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index e4958961d..18d36ade9 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -25,6 +25,7 @@ pub struct Store { pub deletes: AtomicUsize, pub lose_delete_response: AtomicBool, held: Mutex)>>, + held_put: Mutex)>>, held_delete: Mutex)>>, publication: Mutex>>, pub publication_cuts: Mutex>, @@ -46,6 +47,7 @@ impl Store { deletes: AtomicUsize::new(0), lose_delete_response: AtomicBool::new(false), held: Mutex::new(None), + held_put: Mutex::new(None), held_delete: Mutex::new(None), publication: Mutex::new(None), publication_cuts: Mutex::new(Vec::new()), @@ -86,6 +88,16 @@ impl Store { gate } + pub fn hold_put(&self, path: Path) -> Arc { + let gate = Arc::new(Gate::default()); + *self.held_put.lock().unwrap() = Some((path, gate.clone())); + gate + } + + pub fn clear_put(&self) { + self.held_put.lock().unwrap().take(); + } + pub fn clear(&self) { self.held.lock().unwrap().take(); } @@ -107,6 +119,17 @@ impl ObjectStore for Store { ) -> Result { self.available()?; self.puts.fetch_add(1, Ordering::AcqRel); + let gate = self + .held_put + .lock() + .unwrap() + .as_ref() + .filter(|(held, _)| held == path) + .map(|(_, gate)| gate.clone()); + if let Some(gate) = gate { + gate.entered.notify_one(); + gate.release.notified().await; + } let publication = if path.as_ref().ends_with("/ownership/registry") { let bytes: Vec = payload .iter() diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs new file mode 100644 index 000000000..52373ba61 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -0,0 +1,464 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real catalog CAS/archive I/O and voter disks, with a contract-test authority. +//! This is not Docker host fencing or fresh-incarnation serving qualification. +use super::*; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; + +#[derive(Default)] +struct Authority { + fences: AtomicUsize, + verifications: AtomicUsize, + permanently_stopped: AtomicBool, + lose_response: AtomicBool, + reject: AtomicBool, +} + +impl DisasterFenceAuthority for Authority { + fn fence<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + ) -> futures_util::future::BoxFuture<'a, Result>> { + Box::pin(async move { + self.fences.fetch_add(1, Ordering::AcqRel); + self.permanently_stopped.store(true, Ordering::Release); + if self.lose_response.swap(false, Ordering::AcqRel) { + anyhow::bail!("lost authority response after fence"); + } + Ok(request.digest().to_vec()) + }) + } + + fn verify<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + evidence: &'a [u8], + ) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + self.verifications.fetch_add(1, Ordering::AcqRel); + anyhow::ensure!( + self.permanently_stopped.load(Ordering::Acquire), + "no external fence" + ); + anyhow::ensure!( + !self.reject.load(Ordering::Acquire), + "independent verification rejected" + ); + anyhow::ensure!(evidence == request.digest(), "foreign fence evidence"); + Ok(()) + }) + } +} + +struct Scenario { + cluster: Cluster, + expected: std::collections::BTreeMap>, + remote: Arc, + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + receivers: CellGroupCheckpointPool, + destination: ArchiveDestination, + request: CellGroupRecoveryRequest, + alternate: CellGroupRecoveryRequest, + competing: DirectoryMutation, + checkpoint: CellGroupArchiveReference, + authority: Authority, + original_head: [u8; 32], +} + +impl Scenario { + async fn new(rooted: bool, store: Arc) -> Result { + let remote = archive_store::Store::around(store); + let mut cluster = Cluster::with_store( + Default::default(), + tempfile::tempdir()?, + Arc::new(DirectFileSystem), + [6; 32], + remote.clone(), + ) + .await?; + cluster.elect().await?; + let expected = + checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 0, 8192).await?; + let archives = CellGroupArchivePool::new(Default::default())?; + let binding = cluster + .directory + .prepare_group_archive_binding(&cluster.domain, &archives, &cluster.birth) + .await?; + let registry = binding.commit(&cluster.domain).await?; + let owners = CellGroupArchiveUploadPool::new(Default::default())?; + let tails = CellGroupArchiveTailPool::new(Default::default())?; + let intent = registry.prepare_upload(&cluster.domain).await?; + let ticket = registry.reserve(&cluster.domain, &intent).await?; + let upload = registry + .open_upload(&cluster.domain, &owners, &ticket, &tails) + .await?; + let sources = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let checkpoint = upload + .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) + .await?; + let group = cluster.voters[0] + .log + .cell_group(&cluster.birth.route().history)?; + let uploaded = upload + .upload_tail( + &cluster.domain, + &tails, + &checkpoint, + &group, + cluster.voters[0].owner.as_ref().unwrap().manifest(), + ) + .await?; + drop(group); + let destination = destination(&cluster).await?; + let receivers = CellGroupCheckpointPool::new(checkpoint_fixture::limits())?; + let coverage = upload + .prepare_publication( + &cluster.domain, + &tails, + &uploaded, + receiver(&receivers, &destination, &cluster)?, + ) + .await? + .commit(&cluster.domain) + .await?; + let hold = coverage.hold().clone(); + drop(coverage); + let route = cluster.birth.route(); + let target = cluster.directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([70; 16])?, + incarnation: IncarnationId::new([71; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([72; 16])?, + voters: (11..=13) + .map(|n| InitialMember { + member: MemberId::new(n).unwrap(), + node: NodeId::new([n as u8 + 100; 16]).unwrap(), + }) + .collect(), + learners: vec![], + })?; + let view = cluster.domain.run(cluster.directory.observe()).await??; + let request = cluster.directory.prepare_group_recovery( + &view, + ProposalId::new([73; 16])?, + &cluster.birth, + &target, + hold.clone(), + )?; + let alternate = cluster.directory.prepare_group_recovery( + &view, + ProposalId::new([74; 16])?, + &cluster.birth, + &target, + hold, + )?; + let competing = cluster.directory.prepare_replacement( + &view, + ProposalId::new([75; 16])?, + std::slice::from_ref(route), + std::slice::from_ref(target.route()), + )?; + cluster + .domain + .run(cluster.directory.reserve_group_birth(&target)) + .await??; + let saved: Vec = persist_archive_intent(&cluster, &request.encode().to_vec()).await?; + assert_eq!(saved, request.encode()); + let decoded = cluster.directory.decode_group_recovery(&saved)?; + assert_eq!(decoded.digest(), request.digest()); + assert_eq!(decoded.hold(), request.hold()); + Ok(Self { + cluster, + expected, + remote, + archives, + tails, + receivers, + destination, + request, + alternate, + competing, + checkpoint, + authority: Default::default(), + original_head: view.digest(), + }) + } + + fn fence<'a>( + &'a self, + directory: &'a RangeDirectory, + request: &'a CellGroupRecoveryRequest, + ) -> futures_util::future::BoxFuture<'a, Result> + { + // Repeated by-value recovery futures made the debug caller's polling + // frame exceed the default test-thread stack before its first poll. + // Keep the complete admitted operation owned behind this one boundary. + Box::pin(async move { + directory + .fence_group_recovery( + &self.cluster.domain, + &self.archives, + &self.tails, + request, + receiver(&self.receivers, &self.destination, &self.cluster).unwrap(), + &self.authority, + ) + .await + }) + } + + async fn reopen(&self) -> Result { + let route = self.cluster.birth.route(); + let catalog = Catalog::new(self.remote.clone(), Default::default())?; + let namespace = self + .cluster + .domain + .run(catalog.existing_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + Ok(RangeDirectory::new(namespace, Default::default())?) + } + + async fn assert_closed(&self) -> Result<()> { + let cluster = &self.cluster; + assert!(matches!( + cluster + .domain + .run( + cluster + .directory + .require_current_group_birth(&cluster.birth) + ) + .await?, + Err(CatalogError::Fenced) + )); + assert!(cluster + .domain + .run( + cluster + .directory + .materialize_selected_group(cluster.birth.route()) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run( + cluster + .directory + .bootstrap_group_member(&cluster.birth, &cluster.voters[0].log) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run( + cluster + .directory + .materialize_group_birth(self.request.replacement()) + ) + .await? + .is_err()); + assert!(cluster + .domain + .run(cluster.directory.publish(&self.competing)) + .await? + .is_err()); + let before = self.authority.fences.load(Ordering::Acquire); + assert!(self + .fence(&cluster.directory, &self.alternate) + .await + .is_err()); + assert_eq!(self.authority.fences.load(Ordering::Acquire), before); + let route = cluster.birth.route(); + let catalog = Catalog::new(self.remote.clone(), Default::default())?; + let namespace = cluster + .domain + .run(catalog.existing_namespace(NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ))) + .await??; + assert!(matches!( + cluster + .domain + .run(namespace.freeze_births(ProposalId::new([80; 16])?, Duration::from_secs(5))) + .await?, + Err(CatalogError::Conflict( + "disaster namespace cannot enter managed ownership" + )) + )); + Ok(()) + } + + async fn finish(self) -> Result<()> { + assert_eq!(self.receivers.usage().sources, 0); + assert_eq!(self.tails.usage().operations, 0); + self.destination.store.close(); + self.cluster.close().await + } +} + +pub(super) async fn resume(rooted: bool, store: Arc) -> Result<()> { + let s = Box::pin(Scenario::new(rooted, store)).await?; + let (path, index) = index(s.remote.as_ref(), &s.checkpoint).await?; + let dependency = cell_objects(&path, &index, "beta")[0].clone(); + let bytes = s.remote.get(&dependency).await?.bytes().await?; + let mut corrupt = bytes.to_vec(); + corrupt[0] ^= 1; + s.remote.put(&dependency, corrupt.into()).await?; + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + assert!(!s.authority.permanently_stopped.load(Ordering::Acquire)); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.observe()) + .await?? + .digest(), + s.original_head + ); + s.cluster + .domain + .run( + s.cluster + .directory + .require_current_group_birth(&s.cluster.birth), + ) + .await??; + s.remote.put(&dependency, bytes.into()).await?; + + // Both the catalog CAS and the external stop can commit before their + // responses are lost. Read back the exact lock, then resume that same fence. + s.authority.lose_response.store(true, Ordering::Release); + let head = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().ends_with("/cell-groups/head")) + .context("directory head")? + .location; + let gate = s.remote.hold_put(head); + let mut work = s.fence(&s.cluster.directory, &s.request); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("recovery skipped topology lock"), + } + s.remote.lose_put_response.store(true, Ordering::Release); + s.remote.clear_put(); + gate.release.notify_one(); + assert!(matches!( + work.await, + Err(CellGroupRecoveryError::Catalog(CatalogError::Fence(_))) + )); + assert!(!s.remote.lose_put_response.load(Ordering::Acquire)); + assert!(s.authority.permanently_stopped.load(Ordering::Acquire)); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 1); + s.assert_closed().await?; + + s.authority.reject.store(true, Ordering::Release); + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 2); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 1); + s.assert_closed().await?; + s.authority.reject.store(false, Ordering::Release); + let proof = s.fence(&s.cluster.directory, &s.request).await?; + assert_eq!(proof.request().encode(), s.request.encode()); + assert_eq!(proof.coverage().marker(), s.request.hold().marker()); + tail_fixture::check_images( + &s.cluster, + proof.coverage().recovery().manifest(), + &s.expected, + ) + .await?; + for voter in &s.cluster.voters { + assert_eq!( + voter.owner.as_ref().unwrap().manifest().marker(), + proof.coverage().marker() + ); + assert!(voter + .log + .cell_group(&s.request.replacement().route().history) + .is_err()); + } + drop(proof); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 3); + s.assert_closed().await?; + + // Fresh catalog owner reconstructs intent from durable bytes and rechecks + // complete physical data and the winning receipt, without calling fence. + let directory = s.reopen().await?; + let request = directory.decode_group_recovery(s.request.encode())?; + s.authority.reject.store(true, Ordering::Release); + assert!(s.fence(&directory, &request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + s.authority.reject.store(false, Ordering::Release); + let resumed = s.fence(&directory, &request).await?; + tail_fixture::check_images( + &s.cluster, + resumed.coverage().recovery().manifest(), + &s.expected, + ) + .await?; + assert_eq!(resumed.request().encode(), s.request.encode()); + drop(resumed); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 3); + assert_eq!(s.authority.verifications.load(Ordering::Acquire), 5); + s.finish().await +} + +pub(super) async fn topology_wins( + rooted: bool, + store: Arc, +) -> Result<()> { + let s = Box::pin(Scenario::new(rooted, store)).await?; + let head = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().ends_with("/cell-groups/head")) + .context("directory head")? + .location; + let gate = s.remote.hold_put(head); + let mut work = Box::pin(s.fence(&s.cluster.directory, &s.request)); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("recovery skipped topology CAS"), + } + s.remote.clear_put(); + s.cluster + .domain + .run(s.cluster.directory.publish(&s.competing)) + .await??; + gate.release.notify_one(); + assert!(work.await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + assert!(!s.authority.permanently_stopped.load(Ordering::Acquire)); + assert!(s.fence(&s.cluster.directory, &s.request).await.is_err()); + assert_eq!(s.authority.fences.load(Ordering::Acquire), 0); + s.finish().await +} diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 13f86720e..78272de27 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -2,7 +2,7 @@ "format": 1, "source_revision": "1159e9358c106fe1260c3479759e909e413257a4", "runtime_revision": "1159e9358c106fe1260c3479759e909e413257a4", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 7b89b09dd..6fc47cc90 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -1002,6 +1002,31 @@ recovery, not a globally consistent snapshot of unrelated groups or external eff ## Archive-aware local checkpoint compaction +The unmanaged recovery fence barrier is implemented separately from replacement +installation. `RangeDirectory::prepare_group_recovery` binds the exact source +birth, dedicated archive hold, replacement birth and expected topology mutation +in canonical marker-1 bytes. Persist those bytes and retain the hold before +`fence_group_recovery`; resume the identical bytes after an unknown outcome. +The barrier independently verifies every held archive dependency before claiming +the namespace or asking infrastructure to fence it. It refuses joint, repair +and fleet membership records at this static unmanaged boundary. + +A recovery claim shares the directory-head CAS with ordinary topology publication +and the namespace lifecycle CAS with fleet import. Once selected, its source +cannot bootstrap again and competing topology operations refuse. The existing +`DisasterFenceAuthority` contract accepts a typed `CellGroupDisasterFenceRequest`: +the deployment controller must permanently stop/revoke all possible old writers, +including later joins and removed hosts, and independently verify evidence bound +to the complete request digest. Cellar stores the winning bounded receipt and +independently verifies it again on resume. No permissive controller is supplied. + +`FencedCellGroupRecovery` retains verified data under the original histories. +It grants no engine, voting or serving permission and does not release the +topology lock or publish a replacement. Fresh-history image/receipt rebinding, +fresh-genesis installation, activation and fresh-quorum admission must complete +the recovery protocol above. The contract fixture and real RustFS CAS tests do +not qualify Docker host fencing or the deleted-voter serving drill. + An archive registration first makes the local group's retention requirement irreversible through an engine-synced metadata write. The required marker-1 `archive_required` field belongs to the local group application record; a peer diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index b62ba67a9..6b2b7cb7e 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1779,7 +1779,10 @@ "crates/cellar/tests/support/cell_group_s3_outage.rs", "crates/cellar/tests/support/cell_group_backend_control.rs", "tools/multi-raft/backend_control.py", - "tools/multi-raft/test_backend_control.py" + "tools/multi-raft/test_backend_control.py", + "crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs", + "crates/cellar/tests/support/cell_group_disaster_fence.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 65c5176975907fb52a53deca58e1ae1f5ba7adec Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 08:46:45 -0700 Subject: [PATCH 244/568] perf: box rare directory recovery locks A by-value recovery lock enlarged common directory futures and made the retained learner-replacement test overflow its default stack. The previous copied executable passes and the candidate reproduces the failure; boxing the optional lock restores the exact test without changing its stack limit or serialized record. --- .../durability/raft/cell_groups/directory/recovery/mod.rs | 4 ++-- crates/cellar/durability/raft/cell_groups/directory/wire.rs | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs index d85a3ec92..1fe418b5a 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs @@ -184,11 +184,11 @@ impl RangeDirectory { } self.selected_route_in(&view, request.source().route()) .await?; - view.head.recovery = Some(RecoveryLock { + view.head.recovery = Some(Box::new(RecoveryLock { request: request.digest(), base: view.digest, source: wire::Route::encode(request.source().route(), self)?, - }); + })); let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; // A lost response can follow a committed CAS. Observe the exact // lock; never issue another request or rebase onto a newer head. diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 176c33d7f..147cc84eb 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -208,13 +208,13 @@ pub(super) struct Head { pub groups: Option<[u8; 32]>, pub journal: [u8; 32], #[serde(deserialize_with = "required_recovery")] - pub recovery: Option, + pub recovery: Option>, } // Marker 1 has one record shape. Missing fields are not an older-reader mode. fn required_recovery<'de, D: serde::Deserializer<'de>>( decoder: D, -) -> Result, D::Error> { +) -> Result>, D::Error> { Option::deserialize(decoder) } From 9f3b5b86e05b5dd7e62fdb088145a3d6f8a8c111 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 10:08:20 -0700 Subject: [PATCH 245/568] docs: qualify group recovery fencing barrier Record the source-bound native and RustFS checks, retained failures and independent audit. Keep CG-06 in progress until fresh-genesis restoration, activation and replacement-quorum serving pass their original gate. --- .../multi-raft/cell-groups/baseline.json | 14 ++--- docs/design/multi-raft/cell-groups/handoff.md | 22 ++++++- .../design/multi-raft/cell-groups/progress.md | 57 +++++++++++++++++++ .../multi-raft/cell-groups/protocols.md | 4 +- 4 files changed, 87 insertions(+), 10 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 78272de27..16bb760ce 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "1159e9358c106fe1260c3479759e909e413257a4", - "runtime_revision": "1159e9358c106fe1260c3479759e909e413257a4", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged.", + "source_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", + "runtime_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3dd15a523c1ba6bb326b84f3aefa415ba3ea0714913ee346f08702c444387718", - "line": 115, + "sha256": "068d12f48753389e1a46dba4a2c0a744765136622ed4c99a9b24652c8966bc2d", + "line": 116, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "3dd15a523c1ba6bb326b84f3aefa415ba3ea0714913ee346f08702c444387718", - "line": 184, + "sha256": "068d12f48753389e1a46dba4a2c0a744765136622ed4c99a9b24652c8966bc2d", + "line": 185, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 51c1869e4..314107a24 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,7 +15,27 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest runtime/qualification source is +The latest runtime source is +65c5176975907fb52a53deca58e1ae1f5ba7adec. The exact whole-group +recovery-intent/fencing barrier passes memory and real RustFS scenarios in both +image forms, 211 native shared-group cases, eleven legacy disaster cases and +all scoped static checks. Source/artifact/selection/credential audit passes under +`.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Qualification SHA256: +`0c94cfd5071d49e59bdefd0d1fae78aff15237602eebcfc4ade48283d50b254c`. +The retained stack diagnoses and unresolved socket/deadline timing +limits are recorded in [progress](progress.md). + +Consume the fenced data proof to restore fresh histories, images, receipts and +client floors without callbacks; install fresh genesis, publish once under the +same topology lock, commit activation and require fresh quorum before serving. +The proof currently grants no bootstrap, voting or serving permission. Complete +the accepted Docker fencing-authority qualification including later/removed +voters and permanent launch revocation, erase every owned voter disk and verify +real RustFS restore, old-token refusal, exact receipt retries, new writes and +replacement restart. Preserve the original full CG-06 gate and all later tasks. +The full hard-cutover goal remains active and incomplete. + +The prior bounded-outage qualification source is 1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. Its original bounded full-S3-backend outage case now passes in both image forms. The normal case faults the adapter shared by catalog and archive. The real RustFS diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1af84ec50..9882aad1f 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,63 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 exact whole-group recovery intents and fencing barrier + +Source: 65c5176975907fb52a53deca58e1ae1f5ba7adec. CG-06 remains IN_PROGRESS. The new +`RangeDirectory::fence_group_recovery` boundary independently verifies an exact +held complete-group archive before claiming the namespace or asking infrastructure +to fence it. Canonical operator bytes bind the source birth, archive hold, +replacement identities and original topology mutation. Unknown outcomes retain +that exact request; no latest-head substitution, abort or unfence is supplied. + +The recovery lock shares the directory-head CAS with topology publication and +the namespace lifecycle CAS with fleet import. Old source bootstrap and competing +topology mutations refuse once selected. The existing fencing-authority contract +accepts a typed whole-group request, requires permanent external inventory/launch +fencing and independently verifies the winning bounded receipt, including after +reopening the catalog. Archive corruption prevents any fencing call. The current +static unmanaged boundary refuses joint, repair and fleet membership records. + +Both image forms pass exact resume and topology-race scenarios against memory +and real RustFS. The actual S3 case covers four scenarios and seven signed probes, +including lost CAS/fence responses, failed independent verification, immutable +intent resume, all-cell image checks and startup/import/publication refusals. +All five native shared-group suites pass (211 cases), as do eleven legacy +disaster cases, locked all-target Cellar/logic Clippy, formatting/layout over 1966 +Rust files, native dependency/docs/preflight/plan checks, 37 plan tests and five +backend-controller tests. Independent audit verifies 2583 +source hashes, copied executables, exact selection and credential exclusion. +Owned RustFS backends/profiles are stopped with data retained. + +Two stack failures are retained with copied executables and diagnosis. The new +fixture's recovery adapter is boxed; the existing extent helper uses its actual +schema contract. A deterministic learner-replacement regression from by-value +recovery-lock metadata is corrected by boxing that rare metadata in the common +directory head. The exact default-stack test passes; canonical record bytes, +stack limits and test assertions are unchanged by the layout correction. + +Initial controller-suite socket-address/command-deadline failures and a checkpoint +fixture-driver deadline remain retained, not claimed fixed. Eight controller and +four checkpoint comparisons pass on both copied source revisions. A contemporaneous +extreme host-load snapshot is context, not proof of causation. The full unchanged +source repeat passes; the earlier 8dfe9ff3 slow-reader timing limitation remains. +This is not sustained-load, independent-domain, Docker voter-fence or deleted-voter +replacement-serving qualification. + +Evidence: `.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Qualification record +SHA256: `0c94cfd5071d49e59bdefd0d1fae78aff15237602eebcfc4ade48283d50b254c`. Independent audit SHA256: `f2b13f7590220b06a7c23368563b176ce2abcf5d9506869b21538dde20db2c12`. +Prior outage/dependency/reclamation evidence remains unchanged. + +The critical next feature is fresh-history image/receipt/client-floor restoration, +fresh engine genesis, one fenced replacement publication, committed activation and +fresh-quorum admission. The returned `FencedCellGroupRecovery` still contains +original-history data and grants no engine, voting or serving permission. Qualify +Docker permanent restart/launch revocation including later/removed voters, delete +all owned voter disks, restore from real RustFS, reject old tokens, prove no +callback replay, perform new writes and restart the retained replacement quorum. +Only then run the full original CG-06 gate. CG-08 effects/cache/result lifetimes +precede CG-07 SDK/fleet cutover; CG-09–16 remain mandatory. + ## CG-06 bounded Quorum progress during a full S3 backend outage Source: 1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. The diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 6fc47cc90..32529c3ad 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -1000,8 +1000,6 @@ receipt retries, new writes and retained-disk restart. Repeat during split/merge crash cuts and after physical log reclamation. This remains group/component recovery, not a globally consistent snapshot of unrelated groups or external effects. -## Archive-aware local checkpoint compaction - The unmanaged recovery fence barrier is implemented separately from replacement installation. `RangeDirectory::prepare_group_recovery` binds the exact source birth, dedicated archive hold, replacement birth and expected topology mutation @@ -1027,6 +1025,8 @@ fresh-genesis installation, activation and fresh-quorum admission must complete the recovery protocol above. The contract fixture and real RustFS CAS tests do not qualify Docker host fencing or the deleted-voter serving drill. +## Archive-aware local checkpoint compaction + An archive registration first makes the local group's retention requirement irreversible through an engine-synced metadata write. The required marker-1 `archive_required` field belongs to the local group application record; a peer From 7052d218d60181d7b0bb5a4c12686e80e294a61c Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 11:19:55 -0700 Subject: [PATCH 246/568] feat: prepare fresh-history group recovery --- crates/cellar/durability/raft/cache.rs | 14 + .../raft/cell_groups/apply/recovery.rs | 17 +- .../raft/cell_groups/apply/transition.rs | 36 ++ .../raft/cell_groups/archive/mod.rs | 2 + .../raft/cell_groups/checkpoint/receive.rs | 10 +- .../cell_groups/directory/recovery/mod.rs | 6 +- .../raft/cell_groups/enrollment_apply.rs | 3 + .../raft/cell_groups/enrollment_authority.rs | 3 +- .../raft/cell_groups/execution_authority.rs | 6 +- .../durability/raft/cell_groups/membership.rs | 3 + .../cellar/durability/raft/cell_groups/mod.rs | 2 + .../durability/raft/cell_groups/owner.rs | 6 +- .../raft/cell_groups/read_authority.rs | 2 +- .../raft/cell_groups/recovery/admission.rs | 89 ++++ .../raft/cell_groups/recovery/import.rs | 293 +++++++++++++ .../raft/cell_groups/recovery/mod.rs | 13 + .../raft/cell_groups/recovery/state.rs | 155 +++++++ .../raft/cell_groups/recovery/tests.rs | 402 ++++++++++++++++++ .../durability/raft/cell_groups/root.rs | 111 ++++- .../raft/cell_groups/seed_authority.rs | 5 +- .../tests/native_cell_groups_archive.rs | 43 ++ .../support/cell_group_disaster_fence.rs | 216 ++++++++++ docs/design/multi-raft/cell-groups/tasks.json | 7 +- 23 files changed, 1411 insertions(+), 33 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/admission.rs create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/import.rs create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/state.rs create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/tests.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index a86d203eb..cf88f9f67 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -682,6 +682,17 @@ impl CachedGroup { { return Err(LogError::Invalid("cell-group snapshot binding/base".into())); } + let previous = self.group_manifest.as_ref().unwrap(); + if !marker.preserves_recovery(previous) + && !(previous.recovery_request().is_none() + && previous.root().cells() == 0 + && previous.position().index.get() == 1 + && write.seed_grant.is_some()) + { + return Err(LogError::Invalid( + "snapshot recovery lineage changed".into(), + )); + } prepared.membership().succeeds(&self.membership)?; next.entries.clear(); next.applied = DurableApply { @@ -835,6 +846,9 @@ impl CachedGroup { )); } if let Some(previous) = &self.group_manifest { + if !marker.preserves_recovery(previous) { + return Err(LogError::Invalid("applied recovery lineage changed".into())); + } if marker.route() != previous.route() { return Err(LogError::Invalid( "group route changed outside topology protocol".into(), diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs index e03c1e650..d8d6ca031 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -64,18 +64,11 @@ impl CellGroupApplier { .. } = owner; let mut prepare = |marker: &GroupManifest| { - tree.prepare_complete( - &self.group, - pages, - marker.root(), - marker.position(), - marker.membership_revision(), - |leaf| { - images - .recover_image(leaf, marker.route().schema_digest) - .map_err(|error| ManifestError::Io(std::io::Error::other(error))) - }, - ) + tree.prepare_bound_complete(&self.group, pages, marker.clone(), &mut |leaf| { + images + .recover_image(leaf, marker.route().schema_digest) + .map_err(|error| ManifestError::Io(std::io::Error::other(error))) + }) .map_err(storage) }; let current = prepare(&marker)?; diff --git a/crates/cellar/durability/raft/cell_groups/apply/transition.rs b/crates/cellar/durability/raft/cell_groups/apply/transition.rs index 35f945327..5341e7ecd 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/transition.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/transition.rs @@ -40,6 +40,42 @@ impl Storage { } let decoded = CellGroupEntry::decode(&entry.data, base.marker().route(), *limits).map_err(storage)?; + if let GroupCommand::Activate { + transition, + publication, + manifest, + } = decoded.command + { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(ExecutionError::Invalid("activation proposal binding")); + } + let state = base + .marker() + .recovery + .as_deref() + .ok_or(ExecutionError::Invalid("activation without recovery"))?; + let expected = match state.activation_manifest() { + Some(manifest) => manifest, + None => state.seed_commitment(base.marker()).map_err(storage)?, + }; + let mut next = unchanged; + next.marker + .recovery + .as_deref_mut() + .ok_or(ExecutionError::Invalid("activation without recovery"))? + .activate( + transition, + publication, + manifest, + expected, + entry_position(entry), + ) + .map_err(storage)?; + return Ok(next); + } + if !base.marker().is_active() { + return Err(ExecutionError::Invalid("recovery is not activated")); + } let witness = CommitWitness { group: base.marker().route().history, position: entry_position(entry), diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index d5b420586..1477d606d 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -64,6 +64,7 @@ pub enum CellGroupArchiveError { MissingRegistry, Checkpoint(CellGroupCheckpointError), Replay(crate::raft::execution::ExecutionError), + Enrollment(super::EnrollmentError), Storage(object_store::Error), Worker(crate::asyncrt::TaskPanic), } @@ -79,6 +80,7 @@ impl std::error::Error for CellGroupArchiveError { match self { Self::Checkpoint(e) => Some(e), Self::Replay(e) => Some(e), + Self::Enrollment(e) => Some(e), Self::Storage(e) => Some(e), Self::Worker(e) => Some(e), _ => None, diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs index 9621c7e6a..0ca021e1a 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -91,14 +91,8 @@ impl Target for EngineTarget { ) -> std::result::Result, ) -> Result { - let mut root = tree.prepare_complete( - &self.applier.group, - pages, - header.root.root(), - header.root.position(), - header.root.membership_revision(), - image, - )?; + let mut root = + tree.prepare_bound_complete(&self.applier.group, pages, header.root.clone(), image)?; root.base = Some(self.base.clone()); Ok(root) } diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs index 1fe418b5a..2ee76535c 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs @@ -93,8 +93,10 @@ struct Complete { /// A complete selected archive and independently verified permanent fence. /// Retain its dedicated remote hold until recovery and entered work finish. /// This deliberately cannot be supplied to ordinary bootstrap or live apply. +/// Clones share the same admitted cut when preparing multiple destinations. +#[derive(Clone)] pub struct FencedCellGroupRecovery { - request: CellGroupRecoveryRequest, + request: Arc, coverage: CellGroupArchiveCoverage, } @@ -319,7 +321,7 @@ impl RangeDirectory { .map_err(|e| CatalogError::Fence(e.to_string()))?; self.bounded(self.recovery_source(fence.recovery())).await?; Ok(FencedCellGroupRecovery { - request: fence.into_recovery(), + request: Arc::new(fence.into_recovery()), coverage, }) }) diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs index bd7b724e4..a3de27884 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_apply.rs @@ -80,6 +80,9 @@ impl ManifestTree { apply: EnrollmentApply<'_>, image: impl FnOnce(&EnrollmentRecord) -> Result, ) -> Result<(PreparedGroupManifest, EnrollmentDecision), EnrollmentApplyError> { + if !previous.marker().is_active() { + return Err(ManifestError::Invalid("recovery is not activated").into()); + } // Validate route, store ownership and contiguous position before image // I/O. The unchanged candidate also supplies the idempotent apply path. let unchanged = self.prepare_unchanged(store, previous, apply.position)?; diff --git a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs index b050f9cf0..f6765d6ed 100644 --- a/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/enrollment_authority.rs @@ -99,7 +99,8 @@ impl CellEnrollmentState { .as_ref() .ok_or(RuntimeError::NotReady)?; let key = cell_routing_key(self.cell.namespace, &self.cell.cell); - if self.term.get() == 0 + if !current.is_active() + || self.term.get() == 0 || cache.hard_state.term != self.term.get() || current.position().term != self.term || current.position().index.get() > cache.hard_state.commit diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index 8f805fb02..a89d1f6ed 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -122,7 +122,8 @@ impl CellExecutionAuthority { self.route.history.namespace, &self.leaf.version.history.cell, ); - if cache.hard_state.term != self.term.get() + if !current.is_active() + || cache.hard_state.term != self.term.get() || !cache .conf_state .voters @@ -206,7 +207,8 @@ impl CellReconciliationAuthority { self.route.history.namespace, &self.leaf.version.history.cell, ); - if self.term.get() == 0 + if !current.is_active() + || self.term.get() == 0 || cache.hard_state.term != self.term.get() || current.position().term != self.term || current.position().index.get() > cache.hard_state.commit diff --git a/crates/cellar/durability/raft/cell_groups/membership.rs b/crates/cellar/durability/raft/cell_groups/membership.rs index dff3bb096..43016f22c 100644 --- a/crates/cellar/durability/raft/cell_groups/membership.rs +++ b/crates/cellar/durability/raft/cell_groups/membership.rs @@ -193,6 +193,9 @@ impl super::ManifestTree { before: &ConfState, membership: &GroupMembership, ) -> Result { + if !previous.marker().is_active() { + return Err(super::ManifestError::Invalid("recovery is not activated")); + } let mut prepared = self.prepare_unchanged( pages, previous, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 952cfbedc..e78ace537 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -80,6 +80,8 @@ pub use log_retention::{ CellGroupLogCapacity, CellGroupLogRetentionLimits, CellGroupLogRetentionUsage, }; mod read_authority; +mod recovery; +pub use recovery::{CellGroupRecoveryLimits, CellGroupRecoveryPool, PreparedCellGroupRecovery}; pub(crate) mod reclaim; pub use read_authority::{CellReadAuthority, CellReadConsistency}; pub use reclaim::{CellGroupReclaimError, CellGroupReclaimLimits, CellGroupReclaimReport}; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 2a33a7043..75c872c79 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -368,6 +368,9 @@ impl CellGroupOwner { ) -> Result { self.live()?; let status = self.status()?; + if !self.current.marker().is_active() { + return Err(RuntimeError::NotReady); + } let leader = if matches!(consistency, super::CellReadConsistency::Linearizable) { if self.node.raft.state != StateRole::Leader || Some(status.term) != term { return Err(RuntimeError::NotLeader); @@ -486,7 +489,8 @@ impl CellGroupOwner { leader: (self.node.raft.leader_id != 0).then_some(self.node.raft.leader_id), applied: self.current.marker().position(), committed: LogIndex::new(cache.hard_state.commit), - ready: self.node.raft.state == StateRole::Leader + ready: self.current.marker().is_active() + && self.node.raft.state == StateRole::Leader && self.node.raft.lead_transferee.is_none() && self .node diff --git a/crates/cellar/durability/raft/cell_groups/read_authority.rs b/crates/cellar/durability/raft/cell_groups/read_authority.rs index 5f700e656..22103896d 100644 --- a/crates/cellar/durability/raft/cell_groups/read_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/read_authority.rs @@ -101,7 +101,7 @@ impl CellReadAuthority { if cache .group_manifest .as_ref() - .is_none_or(|m| m.route() != &s.route) + .is_none_or(|m| m.route() != &s.route || !m.is_active()) || !cache.membership.permits_group(s.group.identity()) || s.leader.as_ref().is_some_and(|(term, leader)| { cache.hard_state.term != term.get() diff --git a/crates/cellar/durability/raft/cell_groups/recovery/admission.rs b/crates/cellar/durability/raft/cell_groups/recovery/admission.rs new file mode 100644 index 000000000..830aac927 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/admission.rs @@ -0,0 +1,89 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Separate import admission: the held source already owns archive replay and +//! checkpoint permits. Reacquiring either single-slot pool would deadlock. +use super::super::{CellGroupArchiveError, CellGroupArchiveUsage, ManifestPins}; +use crate::raft::budget::{Budget, Permit}; +use std::sync::Arc; + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupRecoveryLimits { + pub max_operations: usize, + pub max_retained_bytes: usize, + pub max_cells: u32, + pub max_image_bytes: u64, + pub max_restore_bytes: u64, +} + +impl Default for CellGroupRecoveryLimits { + fn default() -> Self { + Self { + max_operations: 1, + max_retained_bytes: 256 * 1024 * 1024, + max_cells: 64, + max_image_bytes: 256 * 1024 * 1024, + max_restore_bytes: 1024 * 1024 * 1024, + } + } +} + +/// Share across the node. The returned preparation retains admission until its +/// physical pins are dropped; cancellation does not release entered worker debt. +#[derive(Clone)] +pub struct CellGroupRecoveryPool { + pub(super) limits: CellGroupRecoveryLimits, + pub(super) budget: Arc, + workspace: usize, +} + +impl CellGroupRecoveryPool { + pub fn new(limits: CellGroupRecoveryLimits) -> Result { + if !(1..=4096).contains(&limits.max_operations) + || !(1..=1_000_000).contains(&limits.max_cells) + || !(512..=cellar_ltx::stream_snapshot::MAX_IMAGE_BYTES) + .contains(&limits.max_image_bytes) + || limits.max_restore_bytes < limits.max_image_bytes + { + return Err(CellGroupArchiveError::Limit); + } + // Source metadata has its own retained archive admission. Account the + // target tree, temporary AVL paths, both verification walks, all extent + // references and two full extent buffers without duplicating an inventory. + let extents_per_image = limits + .max_image_bytes + .div_ceil(cellar_ltx::internal::EXTENT_BYTES); + let workspace = usize::try_from(extents_per_image) + .ok() + .and_then(|count| count.checked_mul(1024)) + .and_then(|bytes| bytes.checked_add(1024 * 1024)) + .and_then(|bytes| bytes.checked_mul(limits.max_cells as usize)) + .and_then(|bytes| { + bytes.checked_add(ManifestPins::maximum_bytes(limits.max_cells).checked_mul(4)?) + }) + .and_then(|bytes| bytes.checked_add(3 * cellar_ltx::internal::EXTENT_BYTES as usize)) + .ok_or(CellGroupArchiveError::Limit)?; + if workspace > limits.max_retained_bytes { + return Err(CellGroupArchiveError::Limit); + } + Ok(Self { + limits, + budget: Budget::new(limits.max_retained_bytes, limits.max_operations), + workspace, + }) + } + + pub fn usage(&self) -> CellGroupArchiveUsage { + let (retained_bytes, operations) = self.budget.usage(); + CellGroupArchiveUsage { + retained_bytes, + operations, + } + } + + pub(super) fn admit(&self) -> Result, CellGroupArchiveError> { + self.budget + .acquire(self.workspace) + .map(Arc::new) + .ok_or(CellGroupArchiveError::Busy) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/import.rs b/crates/cellar/durability/raft/cell_groups/recovery/import.rs new file mode 100644 index 000000000..8a3bf1779 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/import.rs @@ -0,0 +1,293 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh-history data preparation. No engine mapping, catalog publication, +//! membership, callback, log acknowledgement or serving permission is created. +use super::{super::*, CellGroupRecoveryPool}; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::{ + internal::{ImageBinding, EXTENT_BYTES}, + stream_snapshot::ScratchPool, +}; +use std::{ + path::PathBuf, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }, +}; + +type Result = std::result::Result; + +pub struct PreparedCellGroupRecovery { + fence: FencedCellGroupRecovery, + store: CellGroupArchiveStore, + manifest: PreparedGroupManifest, + commitment: [u8; 32], +} + +impl PreparedCellGroupRecovery { + pub fn request(&self) -> &CellGroupRecoveryRequest { + self.fence.request() + } + + pub fn source(&self) -> &FencedCellGroupRecovery { + &self.fence + } + + pub fn store(&self) -> &CellGroupArchiveStore { + &self.store + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.manifest + } + + pub fn commitment(&self) -> [u8; 32] { + self.commitment + } +} + +struct Stop(Arc); + +impl Drop for Stop { + fn drop(&mut self) { + self.0.store(true, Ordering::Release); + } +} + +impl CellGroupRecoveryPool { + /// Copy the complete fenced cut into dedicated replacement stores, with + /// new cell histories and genesis witnesses. SQLite bytes (including receipt + /// results, client floors and effect identity) remain exact. The source hold + /// is provenance and must remain retained after this preparation is dropped. + pub async fn prepare( + &self, + domain: &impl CellGroupArchiveExecution, + fence: FencedCellGroupRecovery, + target: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + domain + .run(async { + let admission = self.admit()?; + let stop = Stop(Arc::new(AtomicBool::new(false))); + let cancelled = stop.0.clone(); + let limits = self.limits; + domain + .blocking(move || { + let check = || { + if cancelled.load(Ordering::Acquire) { + return Err(CellGroupArchiveError::Stopped); + } + target.require_live()?; + fence + .coverage() + .recovery() + .checkpoint() + .checkpoint() + .store() + .require_live()?; + Ok(()) + }; + check()?; + if target.route() != fence.request().replacement().route() + || !directory.is_absolute() + { + return Err(CellGroupArchiveError::Invalid( + "recovery destination binding", + )); + } + let source = fence.coverage().recovery().manifest(); + if source.pinned_images() > limits.max_cells as usize { + return Err(CellGroupArchiveError::Limit); + } + let mut total = 0u64; + source.images.visit(&mut |_, leaf, _, _, _| -> Result<()> { + if leaf.geometry.length > limits.max_image_bytes { + return Err(CellGroupArchiveError::Limit); + } + total = total + .checked_add(leaf.geometry.length) + .filter(|n| *n <= limits.max_restore_bytes) + .ok_or(CellGroupArchiveError::Limit)?; + Ok(()) + })?; + let mut owner = target.storage.try_lock().map_err(|error| match error { + std::sync::TryLockError::WouldBlock => CellGroupArchiveError::Busy, + std::sync::TryLockError::Poisoned(_) => { + CellGroupArchiveError::Invalid("poisoned recovery destination") + } + })?; + let apply::Storage { + tree, + pages, + images, + .. + } = &mut *owner; + let mut pins = ManifestPins::default(); + let mut root = ManifestRoot::default(); + source + .images + .visit(&mut |_, old, pin, _, _| -> Result<()> { + check()?; + let mut leaf = old.clone(); + leaf.version.history.incarnation = + target.route().history.incarnation; + leaf.version.sequence = MutationSequence::new(1).map_err(|_| { + CellGroupArchiveError::Invalid("recovery sequence") + })?; + let pin = copy_image( + images, pin, &mut leaf, &scratch, &directory, &check, + )?; + leaf.enrollment = EnrollmentRecord::new( + leaf.version.history.clone(), + fence.request().operation(), + target.route().schema_digest, + leaf.version.image, + ) + .map_err(CellGroupArchiveError::Enrollment)?; + leaf.last_mutation = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + leaf.witness = CommitWitness { + group: target.route().history, + position: leaf.last_mutation, + proposal: fence.request().operation(), + }; + root = tree + .update(pages, root, leaf.clone(), None) + .map_err(manifest_error)?; + pins = pins + .updated(tree, pages, root, &leaf, &pin) + .map_err(manifest_error)?; + Ok(()) + })?; + let marker = GroupManifest::recovery_genesis( + target.route().clone(), + root, + fence.request().digest(), + fence.request().operation(), + ) + .map_err(manifest_error)?; + let commitment = marker + .recovery + .as_deref() + .ok_or(CellGroupArchiveError::Invalid("recovery lifecycle missing"))? + .seed_commitment(&marker) + .map_err(manifest_error)?; + let mut manifest = tree + .prepare_archive_complete(&target.owner, pages, marker, &mut |leaf| { + let key = crate::application::consensus::cell_routing_key( + target.route().history.namespace, + &leaf.version.history.cell, + ); + pins.get(&key) + .filter(|(selected, _)| *selected == leaf) + .map(|(_, pin)| pin) + .ok_or(ManifestError::Invalid("recovery closure changed")) + }) + .map_err(manifest_error)?; + manifest.recovery_admission = Some(admission); + check()?; + drop(owner); + Ok(PreparedCellGroupRecovery { + fence, + store: target, + manifest, + commitment, + }) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} + +fn manifest_error(error: ManifestError) -> CellGroupArchiveError { + CellGroupCheckpointError::from(error).into() +} + +fn physical(error: impl Into) -> CellGroupArchiveError { + error.into().into() +} + +fn copy_image( + images: &CellGroupApplyImages, + source: &CellImagePin, + leaf: &mut CellManifestLeaf, + scratch: &ScratchPool, + directory: &std::path::Path, + check: &impl Fn() -> Result<()>, +) -> Result { + match (images, source) { + (_, CellImagePin::Empty) => Ok(CellImagePin::Empty), + (CellGroupApplyImages::Materialized { images, .. }, CellImagePin::Materialized(source)) => { + source.verify_and_sync().map_err(physical)?; + let state = source.state(); + let id = apply::images::image_id(&leaf.version.history, 1, state) + .map_err(CellGroupArchiveError::Replay)?; + let mut receive = images + .receive_image(id, state, scratch, directory) + .map_err(physical)?; + let mut reader = source.open_reader().map_err(physical)?; + let mut offset = 0; + while offset < state.length { + check()?; + let bytes = reader + .read_exact_at(offset, (state.length - offset).min(64 * 1024) as usize) + .map_err(physical)?; + receive.append(offset, &bytes).map_err(physical)?; + offset += bytes.len() as u64; + } + Ok(CellImagePin::Materialized( + receive.finish().map_err(physical)?, + )) + } + (CellGroupApplyImages::Extents { images, .. }, CellImagePin::Extents(source)) => { + let old = source.manifest(); + let manifest = old + .with_updates( + ImageBinding { + history: cell_image_history_digest(&leaf.version.history), + ..old.binding() + }, + old.geometry(), + &[], + ) + .map_err(cellar_ltx::internal::StoreError::from) + .map_err(physical)?; + // Protect introduced object identities from concurrent collection + // before the first copy, including deduplicated existing extents. + let staged = images + .stage_transient_root(manifest.clone()) + .map_err(physical)?; + for index in 0..old.geometry().extent_count() { + check()?; + let id = old + .extent(index) + .ok_or(CellGroupArchiveError::Invalid("source extent missing"))?; + let bytes = source.view().read_extent(id).map_err(physical)?; + images.put_extent_bytes(id, &bytes).map_err(physical)?; + } + let pin = Arc::new( + images + .seal_image_root(manifest, &mut vec![0; EXTENT_BYTES as usize]) + .map_err(physical)?, + ); + drop(staged); + leaf.version.image.commitment = CellImageCommitment::Extents { + root: pin.reference().root().bytes(), + manifest_bytes: pin.reference().encoded_length(), + }; + Ok(CellImagePin::Extents(pin)) + } + _ => Err(CellGroupArchiveError::Invalid( + "recovery image representation", + )), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs new file mode 100644 index 000000000..1f5497a07 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs @@ -0,0 +1,13 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fresh-history restoration and activation. Archive integrity, infrastructure +//! fencing, physical genesis and a committed serving decision are distinct proofs. +mod admission; +mod import; +mod state; +pub use admission::{CellGroupRecoveryLimits, CellGroupRecoveryPool}; +pub use import::PreparedCellGroupRecovery; +pub(super) use state::RecoveryState; + +#[cfg(test)] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/recovery/state.rs b/crates/cellar/durability/raft/cell_groups/recovery/state.rs new file mode 100644 index 000000000..7664fbc4f --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/state.rs @@ -0,0 +1,155 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Applied recovery lifecycle. A source intent is provenance, never authority +//! to serve its replacement. Only replicated activation opens admission. +use crate::raft::cell_groups::{GroupManifest, ManifestError}; +use cellar_logic::{cell_group::TransitionId, consensus::*}; +use serde::{Deserialize, Serialize}; + +/// The immutable request commits to the held source archive and all its original +/// cell witnesses. Retain that source hold while this provenance is referenced; +/// no old witness is installed as authority in the replacement history. +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(in crate::raft::cell_groups) struct RecoveryState { + request: [u8; 32], + operation: [u8; 16], + #[serde(deserialize_with = "required_activation")] + activation: Option, +} + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Activation { + publication: [u8; 32], + manifest: [u8; 32], + index: u64, + term: u64, +} + +fn required_activation<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +impl RecoveryState { + pub(in crate::raft::cell_groups) fn pending( + request: [u8; 32], + operation: ProposalId, + ) -> Result { + if request == [0; 32] { + return Err(ManifestError::Invalid("recovery request digest")); + } + Ok(Self { + request, + operation: operation.bytes(), + activation: None, + }) + } + + pub(in crate::raft::cell_groups) fn request(&self) -> [u8; 32] { + self.request + } + + pub(in crate::raft::cell_groups) fn active(&self) -> bool { + self.activation.is_some() + } + + pub(in crate::raft::cell_groups) fn activation_manifest(&self) -> Option<[u8; 32]> { + self.activation + .as_ref() + .map(|activation| activation.manifest) + } + + pub(in crate::raft::cell_groups) fn succeeds(&self, previous: &Self) -> bool { + self.request == previous.request + && self.operation == previous.operation + && previous + .activation + .as_ref() + .is_none_or(|activation| self.activation.as_ref() == Some(activation)) + } + + pub(in crate::raft::cell_groups) fn validate( + &self, + position: LogPosition, + ) -> Result<(), ManifestError> { + if self.request == [0; 32] || ProposalId::new(self.operation).is_err() { + return Err(ManifestError::Invalid("recovery identity")); + } + if let Some(a) = &self.activation { + if a.publication == [0; 32] + || a.manifest == [0; 32] + || a.index <= 1 + || a.term <= 1 + || a.index > position.index.get() + || a.term > position.term.get() + || (a.index == position.index.get() && a.term != position.term.get()) + { + return Err(ManifestError::Invalid("recovery activation position")); + } + } + Ok(()) + } + + pub(in crate::raft::cell_groups) fn activate( + &mut self, + transition: TransitionId, + publication: [u8; 32], + manifest: [u8; 32], + expected_manifest: [u8; 32], + position: LogPosition, + ) -> Result<(), ManifestError> { + if transition.bytes() != self.operation + || publication == [0; 32] + || manifest == [0; 32] + || manifest != expected_manifest + { + return Err(ManifestError::Invalid("recovery activation binding")); + } + if let Some(existing) = &self.activation { + if existing.publication != publication || existing.manifest != manifest { + return Err(ManifestError::Conflict); + } + return self.validate(position); + } + let next = Self { + activation: Some(Activation { + publication, + manifest, + index: position.index.get(), + term: position.term.get(), + }), + ..self.clone() + }; + next.validate(position)?; + *self = next; + Ok(()) + } + + pub(in crate::raft::cell_groups) fn seed_commitment( + &self, + marker: &GroupManifest, + ) -> Result<[u8; 32], ManifestError> { + use sha2::{Digest, Sha256}; + if self.active() + || marker.membership_revision() != 1 + || marker.recovery.as_deref() != Some(self) + { + return Err(ManifestError::Invalid("recovery seed commitment phase")); + } + let mut genesis = marker.clone(); + genesis.position = LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }; + genesis.check()?; + let bytes = serde_json::to_vec(&genesis) + .map_err(|_| ManifestError::Invalid("recovery seed encoding"))?; + let mut digest = Sha256::new(); + digest.update(b"cellar-cell-group-recovery-genesis-1\0"); + digest.update(bytes); + Ok(digest.finalize().into()) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/tests.rs b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs new file mode 100644 index 000000000..03d6bd04d --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs @@ -0,0 +1,402 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Storage preparation tests; no marker here grants catalog or quorum authority. +use super::super::*; +use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, +}; +use cellar_logic::{cell_group::*, consensus::*}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + DirectFileSystem, +}; +use raft::prelude::Entry; +use std::sync::Arc; + +fn route() -> GroupRoute { + GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(2).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + } +} + +fn operation() -> ProposalId { + ProposalId::new([7; 16]).unwrap() +} + +fn position(index: u64) -> LogPosition { + LogPosition { + index: LogIndex::new(index), + term: Term::new(2), + } +} + +fn pending() -> GroupManifest { + GroupManifest::recovery_genesis(route(), ManifestRoot::default(), [8; 32], operation()).unwrap() +} + +#[test] +fn recovery_marker_requires_explicit_lifecycle_and_valid_activation() { + let marker = pending(); + assert!(!marker.is_active()); + let value = serde_json::to_value(&marker).unwrap(); + assert_eq!( + serde_json::from_value::(value.clone()).unwrap(), + marker + ); + let mut malformed = value.clone(); + malformed.as_object_mut().unwrap().remove("recovery"); + assert!(serde_json::from_value::(malformed).is_err()); + let mut malformed = value.clone(); + malformed["recovery"] + .as_object_mut() + .unwrap() + .remove("activation"); + assert!(serde_json::from_value::(malformed).is_err()); + for invalid in [ + serde_json::json!({"publication":vec![0; 32],"manifest":vec![9; 32],"index":2,"term":2}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":1,"term":1}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":3,"term":2}), + serde_json::json!({"publication":vec![9; 32],"manifest":vec![9; 32],"index":2,"term":1}), + ] { + let mut malformed = value.clone(); + malformed["recovery"]["activation"] = invalid; + assert!(serde_json::from_value::(malformed).is_err()); + } + let ordinary = GroupManifest::genesis(route()).unwrap(); + assert!(ordinary.is_active()); + assert_eq!( + serde_json::to_value(ordinary).unwrap()["recovery"], + serde_json::Value::Null + ); + let mut invalid = pending(); + invalid.position = position(2); + invalid.membership_revision = 2; + assert!(invalid.check().is_err()); +} + +#[test] +fn recovery_commitment_is_bound_and_activation_cannot_be_replaced() { + let mut marker = pending(); + let expected = marker + .recovery + .as_ref() + .unwrap() + .seed_commitment(&marker) + .unwrap(); + let original = marker.clone(); + marker.position = position(2); + assert_eq!( + marker + .recovery + .as_ref() + .unwrap() + .seed_commitment(&marker) + .unwrap(), + expected + ); + let foreign = super::RecoveryState::pending([11; 32], operation()).unwrap(); + assert!(foreign.seed_commitment(&marker).is_err()); + let transition = TransitionId::new(operation().bytes()).unwrap(); + let state = marker.recovery.as_mut().unwrap(); + for (id, publication, manifest) in [ + (TransitionId::new([12; 16]).unwrap(), [13; 32], expected), + (transition, [0; 32], expected), + (transition, [13; 32], [14; 32]), + ] { + assert!(state + .activate(id, publication, manifest, expected, position(2)) + .is_err()); + assert!(!state.active()); + } + state + .activate(transition, [13; 32], expected, expected, position(2)) + .unwrap(); + let activated = state.clone(); + state + .activate(transition, [13; 32], expected, expected, position(3)) + .unwrap(); + assert_eq!(*state, activated); + assert!(state + .activate(transition, [15; 32], expected, expected, position(3)) + .is_err()); + assert!(marker.is_active()); + assert!(marker.preserves_recovery(&original)); + assert!(!original.preserves_recovery(&marker)); + assert!(!GroupManifest::genesis(route()) + .unwrap() + .preserves_recovery(&marker)); + assert!(!marker.preserves_recovery(&GroupManifest::genesis(route()).unwrap())); + marker.check().unwrap(); +} + +fn activation(marker: &GroupManifest, index: u64, manifest: [u8; 32]) -> Entry { + let bytes = CellGroupEntry { + binding: GroupEntryBinding { + group: marker.route().history, + route_epoch: marker.route().epoch, + birth_digest: marker.route().birth_digest, + schema_digest: marker.route().schema_digest, + proposal: operation(), + }, + command: GroupCommand::Activate { + transition: TransitionId::new(operation().bytes()).unwrap(), + publication: [9; 32], + manifest, + }, + } + .encode(marker.route(), GroupEntryLimits::default()) + .unwrap(); + Entry { + index, + term: 2, + data: bytes.into(), + ..Default::default() + } +} + +#[test] +fn recovery_apply_requires_exact_activation_and_keeps_noop_prefix_closed() -> anyhow::Result<()> { + let root = tempfile::tempdir()?; + let pages = ManifestPageStore::open( + &root.path().join("pages"), + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let images = CellGroupApplyImages::Materialized { + images: GenerationStore::open( + root.path().join("images"), + Arc::new(DirectFileSystem), + Default::default(), + )?, + services: ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }, + }; + let mut storage = apply::Storage::new( + route(), + pages, + images, + Default::default(), + Default::default(), + 100, + )?; + let marker = pending(); + let commitment = marker.recovery.as_ref().unwrap().seed_commitment(&marker)?; + let owner = root::ArchiveManifestOwner::new(); + let base = + storage + .tree + .prepare_archive_complete(&owner, &mut storage.pages, marker, &mut |_| { + unreachable!("empty root") + })?; + let noop = storage.prepare_entry( + &base, + &Entry { + index: 2, + term: 2, + ..Default::default() + }, + None, + false, + )?; + assert!(!noop.marker().is_active()); + assert_eq!( + noop.marker().recovery_request(), + base.marker().recovery_request() + ); + assert!(storage + .prepare_entry(&noop, &activation(noop.marker(), 3, [99; 32]), None, false) + .is_err()); + assert!(!noop.marker().is_active()); + let active = storage.prepare_entry( + &noop, + &activation(noop.marker(), 3, commitment), + None, + false, + )?; + assert!(active.marker().is_active()); + assert_eq!(active.marker().root(), base.marker().root()); + assert!(active.marker().preserves_recovery(base.marker())); + let duplicate = storage.prepare_entry( + &active, + &activation(active.marker(), 4, commitment), + None, + false, + )?; + assert_eq!(duplicate.marker().recovery, active.marker().recovery); + let ordinary = storage.tree.prepare_archive_complete( + &owner, + &mut storage.pages, + GroupManifest::genesis(route())?, + &mut |_| unreachable!("empty root"), + )?; + assert!(storage + .prepare_entry( + &ordinary, + &activation(ordinary.marker(), 2, commitment), + None, + false + ) + .is_err()); + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn recovery_pending_root_denies_stale_and_execution_authorities() -> anyhow::Result<()> { + use crate::{ + asyncrt::ExecutionDomain, + raft::{ + budget::Budget, + options::{LogOpenMode, ReplicaLogOptions}, + storage::{CellGroupBootstrap, ReplicaLog}, + }, + }; + use sha2::{Digest, Sha256}; + use std::sync::atomic::AtomicU64; + let directory = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + directory.path().join("log"), + NodeId::new([21; 16])?, + DiskLineage::new([22; 16])?, + ProcessGeneration::new([23; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log + .create_cell_group(CellGroupBootstrap { + route: route(), + member: MemberId::new(1)?, + voters: vec![MemberId::new(1)?], + learners: vec![], + }) + .await?; + domain + .blocking(move || -> anyhow::Result<()> { + let cell = CellHistory { + cluster: route().history.cluster, + namespace: route().history.namespace, + cell: CellId::new("restored")?, + incarnation: route().history.incarnation, + }; + let image = CellImage { + epoch: 1, + txid: 0, + checksum: 0, + commitment: CellImageCommitment::Materialized(Sha256::digest([]).into()), + }; + let leaf = CellManifestLeaf { + enrollment: EnrollmentRecord::new( + cell.clone(), + operation(), + route().schema_digest, + image, + )?, + version: CellVersion { + history: cell.clone(), + sequence: MutationSequence::new(1)?, + image, + }, + geometry: CellImageGeometry { + length: 0, + page_size: 512, + }, + last_mutation: pending().position(), + witness: CommitWitness { + group: route().history, + position: pending().position(), + proposal: operation(), + }, + effects: Default::default(), + }; + let mut pages = ManifestPageStore::open( + &directory.path().join("pages"), + Arc::new(DirectFileSystem), + 128, + 1024 * 1024, + 16, + )?; + let tree = ManifestTree::new(route(), Default::default())?; + let root = tree.update(&mut pages, ManifestRoot::default(), leaf.clone(), None)?; + let marker = GroupManifest::recovery_genesis(route(), root, [8; 32], operation())?; + let prepared = + tree.prepare_bound_complete(&group, &mut pages, marker.clone(), &mut |_| { + Ok(CellImagePin::Empty) + })?; + // This is an admission-negative fixture, not a supported genesis installer: + // emulate the selected cache to exercise every actual authority boundary. + group.cache.write().unwrap().group_manifest = Some(marker.clone()); + let leader = Arc::new(AtomicU64::new(1)); + assert!(CellExecutionAuthority::new( + group.clone(), + &prepared, + &cell, + Term::new(1), + leader.clone() + ) + .is_err()); + assert!(CellReconciliationAuthority::new(group.clone(), &prepared, &cell).is_err()); + assert!(CellEnrollmentState::new(group.clone(), &prepared, cell.clone()).is_err()); + let node = Budget::new(1024 * 1024, 1); + let local = Budget::new(1024 * 1024, 1); + let read = CellReadAuthority::new( + read_authority::ReadSelection { + group: group.clone(), + route: route(), + cell: leaf.clone(), + image: CellImagePin::Empty, + applied: marker.position(), + barrier: None, + leader: None, + }, + 100, + node.acquire(4096).unwrap(), + local.acquire(4096).unwrap(), + )?; + assert!(matches!( + read.check(10), + Err(crate::raft::completion::RuntimeError::NotReady) + )); + assert!(tree + .prepare_complete(&group, &mut pages, root, position(2), 1, |_| Ok( + CellImagePin::Empty + )) + .is_err()); + let reopened = + tree.prepare_complete(&group, &mut pages, root, marker.position(), 1, |_| { + Ok(CellImagePin::Empty) + })?; + assert_eq!(reopened.marker(), &marker); + assert!(!reopened.marker().is_active()); + let mut changed = leaf; + changed.last_mutation = position(2); + changed.witness.position = position(2); + assert!(tree + .prepare_update(&mut pages, &prepared, changed, None, CellImagePin::Empty) + .is_err()); + drop(read); + drop(reopened); + drop(prepared); + Ok(()) + })? + .await??; + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 0735aeab8..086759c8a 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -19,6 +19,7 @@ pub struct GroupManifest { pub(super) position: LogPosition, pub(super) membership_revision: u64, pub(super) root: ManifestRoot, + pub(super) recovery: Option>, } impl GroupManifest { @@ -38,6 +39,28 @@ impl GroupManifest { self.root } + /// Data imported into a fresh history remains unavailable to application + /// reads/writes until its replicated activation decision is selected. + pub fn is_active(&self) -> bool { + self.recovery.as_ref().is_none_or(|state| state.active()) + } + + /// Commitment to the immutable recovery intent and retained source archive. + /// This is provenance, never a fence, membership or serving permission. + pub fn recovery_request(&self) -> Option<[u8; 32]> { + self.recovery.as_ref().map(|state| state.request()) + } + + /// A retained history cannot forget or replace its recovery lineage. Only + /// an authenticated empty learner may acquire that lineage via a snapshot. + pub(crate) fn preserves_recovery(&self, previous: &Self) -> bool { + match (&previous.recovery, &self.recovery) { + (None, None) => true, + (Some(previous), Some(next)) => next.succeeds(previous), + _ => false, + } + } + pub(crate) fn check(&self) -> Result<(), ManifestError> { self.position .validate() @@ -49,6 +72,14 @@ impl GroupManifest { return Err(ManifestError::Invalid("root membership")); } ManifestTree::new(self.route.clone(), ManifestLimits::default())?; + if let Some(recovery) = &self.recovery { + recovery.validate(self.position)?; + if !recovery.active() && self.membership_revision != 1 { + return Err(ManifestError::Invalid( + "pending recovery membership changed", + )); + } + } if let Some(root) = self.root.0 { if root.cells == 0 || root.cells > 1_000_000 || root.height == 0 || root.height > 32 { return Err(ManifestError::Invalid("root bounds")); @@ -73,6 +104,24 @@ impl GroupManifest { }, membership_revision: 1, root: ManifestRoot::default(), + recovery: None, + }; + marker.check()?; + Ok(marker) + } + + pub(crate) fn recovery_genesis( + route: GroupRoute, + root: ManifestRoot, + request: [u8; 32], + operation: ProposalId, + ) -> Result { + let marker = Self { + root, + recovery: Some(Box::new(super::recovery::RecoveryState::pending( + request, operation, + )?)), + ..Self::genesis(route)? }; marker.check()?; Ok(marker) @@ -89,6 +138,8 @@ pub struct PreparedGroupManifest { pub(crate) images: ManifestPins, pub(crate) engine_owner: Arc<()>, pub(crate) base: Option, + /// Import admission follows every clone of this private physical closure. + pub(super) recovery_admission: Option>, pub(super) page_owner: Arc, retention: Arc, } @@ -183,6 +234,7 @@ impl PreparedGroupManifest { images, engine_owner: group.owner.clone(), base: None, + recovery_admission: None, page_owner: store.owner.clone(), retention: store.retention.clone(), }) @@ -243,12 +295,50 @@ impl ManifestTree { where for<'a> GroupHistory: From<&'a S>, { - let marker = GroupManifest { + let mut marker = GroupManifest { route: self.route.clone(), position, membership_revision, root, + recovery: None, }; + // Reopening must retain the selected recovery lifecycle, rather than + // silently turning a pending imported root into an ordinary active one. + let mut recovery_selected = false; + let mut matched = false; + for selected in [group.group_manifest(), group.checkpoint_manifest()] { + if let Some(selected) = selected.map_err(|_| ManifestError::Invalid("closed engine"))? { + recovery_selected |= selected.recovery.is_some(); + if selected.route == marker.route + && selected.position == marker.position + && selected.membership_revision == marker.membership_revision + && selected.root == marker.root + { + marker = selected; + matched = true; + break; + } + } + } + if recovery_selected && !matched { + return Err(ManifestError::Invalid("complete recovery marker required")); + } + self.prepare_bound_complete(group, store, marker, &mut image) + } + + pub(super) fn prepare_bound_complete( + &self, + group: &crate::raft::cache::GroupLog, + store: &mut ManifestPageStore, + marker: GroupManifest, + image: &mut dyn FnMut(&CellManifestLeaf) -> Result, + ) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { + if marker.route != self.route { + return Err(ManifestError::Conflict); + } marker.check()?; group .cache_revision() @@ -256,8 +346,7 @@ impl ManifestTree { if !marker.matches_group(&group.identity().scope, group.birth_digest) { return Err(ManifestError::Invalid("root/engine identity")); } - let prepared = - self.prepare_physical(store, marker.clone(), group.owner.clone(), &mut image)?; + let prepared = self.prepare_physical(store, marker.clone(), group.owner.clone(), image)?; // Recover physical ownership only from the engine's selected marker. // A prospective checkpoint or mutation remains a preparation capability. if group @@ -312,6 +401,7 @@ impl ManifestTree { images: pins, engine_owner: owner, base: None, + recovery_admission: None, page_owner: store.owner.clone(), retention: store.retention.clone(), }) @@ -332,6 +422,9 @@ impl ManifestTree { if previous.marker.route != self.route || !Arc::ptr_eq(&previous.page_owner, &store.owner) { return Err(ManifestError::Invalid("root/page owner")); } + if !previous.marker.is_active() { + return Err(ManifestError::Invalid("recovery is not activated")); + } if previous.marker.position.index.get().checked_add(1) != Some(leaf.last_mutation.index.get()) || leaf.last_mutation.term < previous.marker.position.term @@ -350,6 +443,7 @@ impl ManifestTree { position: leaf.last_mutation, membership_revision: previous.marker.membership_revision, root, + recovery: previous.marker.recovery.clone(), }; marker.check()?; Ok(PreparedGroupManifest { @@ -357,6 +451,7 @@ impl ManifestTree { images: pins, engine_owner: previous.engine_owner.clone(), base: Some(previous.marker.clone()), + recovery_admission: previous.recovery_admission.clone(), page_owner: store.owner.clone(), retention: store.retention.clone(), }) @@ -381,6 +476,14 @@ struct WireRoot { membership_revision: u64, #[serde(deserialize_with = "required_root")] root: Option, + #[serde(deserialize_with = "required_recovery")] + recovery: Option>, +} + +fn required_recovery<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result>, D::Error> { + Option::deserialize(decoder) } fn required_root<'de, D: serde::Deserializer<'de>>( @@ -435,6 +538,7 @@ impl From for WireRoot { term: m.position.term.get(), membership_revision: m.membership_revision, root: m.root.0, + recovery: m.recovery, } } } @@ -468,6 +572,7 @@ impl TryFrom for GroupManifest { }, membership_revision: w.membership_revision, root: ManifestRoot(w.root), + recovery: w.recovery, }; result.check()?; Ok(result) diff --git a/crates/cellar/durability/raft/cell_groups/seed_authority.rs b/crates/cellar/durability/raft/cell_groups/seed_authority.rs index cad987ebb..d6504a4d0 100644 --- a/crates/cellar/durability/raft/cell_groups/seed_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/seed_authority.rs @@ -119,12 +119,13 @@ impl CellGroupSeedAuthority { let marker = s.root.marker(); let local = s.group.identity(); let term = s.barrier.term.get(); - if s.leader.load(Ordering::Acquire) != term + if !marker.is_active() + || s.leader.load(Ordering::Acquire) != term || cache.hard_state.term != term || cache .group_manifest .as_ref() - .is_none_or(|m| m.route() != marker.route()) + .is_none_or(|m| m.route() != marker.route() || !m.is_active()) || cache.membership.revision().get() != marker.membership_revision() || !cache.membership.permits_group(local) || !cache diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 4fe352fe8..8ee439b79 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -37,6 +37,39 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_import_rebinds_complete_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::import( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_cancelled_import_retains_physical_admission() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::cancel_import(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_import_rebinds_complete_cut() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::import( + rooted, + rustfs::store("recovery-import", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_recovery_fence_resumes_exact_intent() -> Result<()> { for rooted in [false, true] { @@ -266,6 +299,16 @@ async fn destination_mode( ) -> Result { let base = cluster._root.path().join("archive-data"); let route = cluster.birth.route().clone(); + destination_at(cluster, rooted, filesystem, base, route).await +} + +async fn destination_at( + cluster: &Cluster, + rooted: bool, + filesystem: Arc, + base: std::path::PathBuf, + route: GroupRoute, +) -> Result { cluster .domain .blocking(move || { diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs index 52373ba61..1a48adce8 100644 --- a/crates/cellar/tests/support/cell_group_disaster_fence.rs +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -310,6 +310,222 @@ impl Scenario { } } +pub(super) async fn cancel_import(rooted: bool) -> Result<()> { + let s = Box::new(Scenario::new(rooted, Arc::new(object_store::memory::InMemory::new())).await?); + let gate = fault::Gate::new(None); + let target = destination_at( + &s.cluster, + rooted, + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + s.cluster._root.path().join("recovery-cancel"), + s.request.replacement().route().clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let release = gate.arm(fault::HOLD_DEPENDENCY_SYNC); + let mut work = Box::pin(pool.prepare( + &s.cluster.domain, + proof, + target.store, + scratch.clone(), + s.cluster._root.path().into(), + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => { + anyhow::bail!("import skipped physical dependency sync") + } + } + drop(work); + assert_eq!(pool.usage().operations, 1); + assert_eq!(s.receivers.usage().sources, 1); + drop(release); + let s = *s; + let receivers = s.receivers.clone(); + s.cluster.close().await?; + assert_eq!(pool.usage().operations, 0); + assert_eq!(receivers.usage().sources, 0); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + Ok(()) +} + +pub(super) async fn import(rooted: bool, store: Arc) -> Result<()> { + let s = Box::new(Scenario::new(rooted, store).await?); + let route = s.request.replacement().route().clone(); + let target = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster._root.path().join("recovery-import"), + route.clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let retained_proof = proof.clone(); + let bounded = CellGroupRecoveryPool::new(CellGroupRecoveryLimits { + max_image_bytes: 512, + max_restore_bytes: 512, + ..Default::default() + })?; + assert!(matches!( + bounded + .prepare( + &s.cluster.domain, + proof.clone(), + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await, + Err(CellGroupArchiveError::Limit) + )); + assert_eq!(bounded.usage().operations, 0); + let source = proof.coverage().recovery().manifest().clone(); + let preparation = pool + .prepare( + &s.cluster.domain, + proof, + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let marker = preparation.manifest().marker(); + let expected_marker = marker.clone(); + let expected_commitment = preparation.commitment(); + assert_eq!(marker.route(), &route); + assert_eq!( + marker.position(), + LogPosition { + index: LogIndex::new(1), + term: Term::new(1) + } + ); + assert_eq!(marker.membership_revision(), 1); + assert_eq!(marker.recovery_request(), Some(s.request.digest())); + assert!(!marker.is_active()); + assert_ne!(preparation.commitment(), [0; 32]); + assert_eq!(preparation.request().encode(), s.request.encode()); + assert_eq!(pool.usage().operations, 1); + assert_eq!(scratch.usage()?.reserved_bytes, 0); + tail_fixture::check_images(&s.cluster, preparation.manifest(), &s.expected).await?; + let keys = VersionKeys::new( + VersionSigningKey::new([91; 32], 100_000)?, + vec![], + Duration::from_secs(10), + )?; + for name in s.expected.keys() { + let cell = CellId::new(name)?; + let (old, old_pin) = source.pin_cell(&cell).context("old leaf")?; + let (new, new_pin) = preparation.manifest().pin_cell(&cell).context("new leaf")?; + assert_eq!(new.version.history.incarnation, route.history.incarnation); + assert_ne!(new.version.history, old.version.history); + assert_eq!(new.version.sequence.get(), 1); + assert_eq!(new.enrollment.cell(), &new.version.history); + assert_eq!(new.enrollment.creation(), s.request.operation()); + assert_eq!(new.enrollment.initial(), new.version.image); + assert_eq!(new.effects, old.effects); + assert_eq!(new.geometry, old.geometry); + assert_eq!(new.witness.group, route.history); + assert_eq!(new.witness.position, marker.position()); + assert_eq!(new.witness.proposal, s.request.operation()); + assert_eq!(new.last_mutation, marker.position()); + match (old_pin, new_pin) { + (CellImagePin::Materialized(old), CellImagePin::Materialized(new)) => { + assert_ne!(old.id(), new.id()); + assert_eq!(old.state(), new.state()); + } + (CellImagePin::Extents(old), CellImagePin::Extents(new)) => { + assert_ne!(old.reference(), new.reference()); + assert_ne!( + old.manifest().binding().history, + new.manifest().binding().history + ); + for n in 0..old.manifest().geometry().extent_count() { + assert_eq!(old.manifest().extent(n), new.manifest().extent(n)); + } + } + (CellImagePin::Empty, CellImagePin::Empty) => {} + _ => anyhow::bail!("recovery changed image representation"), + } + let old_token = keys.issue_cell(&old.observation(), route.schema_digest, 10)?; + assert_eq!( + keys.verify_cell(&old_token, &new.version.history, route.schema_digest, 11), + Err(VersionError::WrongHistory) + ); + } + // Holding the result owns its admission. Refusal precedes physical work; + // the replacement still has no local Raft mapping or startup permission. + assert!(matches!( + pool.prepare( + &s.cluster.domain, + retained_proof, + target.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await, + Err(CellGroupArchiveError::Busy) + )); + for voter in &s.cluster.voters { + assert!(voter.log.cell_group(&route.history).is_err()); + } + let retained = preparation.manifest().clone(); + drop(preparation); + assert_eq!(pool.usage().operations, 1); + drop(retained); + assert_eq!(pool.usage().operations, 0); + // Exact retry produces the same imported genesis/commitment. A wrong route + // cannot turn the fenced old data owner into a replacement data owner. + let proof = s.fence(&s.cluster.directory, &s.request).await?; + assert!(pool + .prepare( + &s.cluster.domain, + proof, + s.destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into() + ) + .await + .is_err()); + assert_eq!(pool.usage().operations, 0); + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let retried = pool + .prepare( + &s.cluster.domain, + proof, + target.store, + scratch, + s.cluster._root.path().into(), + ) + .await?; + tail_fixture::check_images(&s.cluster, retried.manifest(), &s.expected).await?; + assert!(!retried.manifest().marker().is_active()); + assert_eq!(retried.manifest().marker(), &expected_marker); + assert_eq!(retried.commitment(), expected_commitment); + drop(retried); + drop(source); + s.assert_closed().await?; + assert_eq!(pool.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + let s = *s; + s.cluster.close().await +} + pub(super) async fn resume(rooted: bool, store: Arc) -> Result<()> { let s = Box::pin(Scenario::new(rooted, store)).await?; let (path, index) = index(s.remote.as_ref(), &s.checkpoint).await?; diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 6b2b7cb7e..7c6babc5a 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1782,7 +1782,12 @@ "tools/multi-raft/test_backend_control.py", "crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs", "crates/cellar/durability/raft/cell_groups/directory/recovery/request.rs", - "crates/cellar/tests/support/cell_group_disaster_fence.rs" + "crates/cellar/tests/support/cell_group_disaster_fence.rs", + "crates/cellar/durability/raft/cell_groups/recovery/mod.rs", + "crates/cellar/durability/raft/cell_groups/recovery/state.rs", + "crates/cellar/durability/raft/cell_groups/recovery/admission.rs", + "crates/cellar/durability/raft/cell_groups/recovery/import.rs", + "crates/cellar/durability/raft/cell_groups/recovery/tests.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 973cad3474b688ea3703c4f3ed5309cd7e1662a0 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 11:56:27 -0700 Subject: [PATCH 247/568] docs: qualify fresh group recovery preparation --- docs/design/multi-raft/cell-groups/README.md | 10 ++- .../multi-raft/cell-groups/baseline.json | 6 +- docs/design/multi-raft/cell-groups/handoff.md | 74 ++++++++++++++----- .../design/multi-raft/cell-groups/progress.md | 65 ++++++++++++++++ 4 files changed, 129 insertions(+), 26 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index b32fe011b..c7b7303d4 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -7,10 +7,12 @@ CG-01 through CG-05 have passed their scoped native evidence gates. CG-05's complete checkpoint, authenticated learner/rebuild, membership, restart and resource-ownership requirements pass at `e0550acd`. Its current gate includes all five exact cases and 205 Rust regressions across eighteen commands. -CG-06 is IN_PROGRESS: independent data recovery after voter-directory erasure and -protected complete-cut publication/remote collection pass native gates. S3 output -settlement, bounded outage admission, local Raft file purge and fenced seeding -remain the next priorities. +CG-06 is IN_PROGRESS: complete S3 checkpoint/tail recovery, bounded S3 settlement, +local log reclamation, exact recovery fencing and fresh-history data preparation +pass their scoped gates. Fresh engine genesis, atomic fenced publication, +committed activation and a fresh serving quorum remain the immediate priorities. +CG-08 due effects and safe cold-cache eviction follow, before CG-07 SDK/fleet +cutover. See the latest [progress](progress.md) and [handoff](handoff.md). The production SDK/fleet has not yet cut over; topology, hibernation, complete S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 16bb760ce..6249b1cbb 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", + "source_revision": "7052d218d60181d7b0bb5a4c12686e80e294a61c", "runtime_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,7 +19,7 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "c8e7da3fdd12cc788389b69b13d2799b7930e666484aa23fa84660a24ed9bb6c", + "sha256": "1e7517328428b06269fa62e8492219e4d820c2a5a1e808c5e33c5e9653bf6354", "line": 460, "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 314107a24..2d495e424 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,25 +15,61 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch `codex/shared-cell-groups` and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest runtime source is -65c5176975907fb52a53deca58e1ae1f5ba7adec. The exact whole-group -recovery-intent/fencing barrier passes memory and real RustFS scenarios in both -image forms, 211 native shared-group cases, eleven legacy disaster cases and -all scoped static checks. Source/artifact/selection/credential audit passes under -`.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Qualification SHA256: -`0c94cfd5071d49e59bdefd0d1fae78aff15237602eebcfc4ade48283d50b254c`. -The retained stack diagnoses and unresolved socket/deadline timing -limits are recorded in [progress](progress.md). - -Consume the fenced data proof to restore fresh histories, images, receipts and -client floors without callbacks; install fresh genesis, publish once under the -same topology lock, commit activation and require fresh quorum before serving. -The proof currently grants no bootstrap, voting or serving permission. Complete -the accepted Docker fencing-authority qualification including later/removed -voters and permanent launch revocation, erase every owned voter disk and verify -real RustFS restore, old-token refusal, exact receipt retries, new writes and -replacement restart. Preserve the original full CG-06 gate and all later tasks. -The full hard-cutover goal remains active and incomplete. +The latest runtime source is 7052d218d60181d7b0bb5a4c12686e80e294a61c. +Fenced source data now imports into complete fresh-history data roots, with +pending application admission and bounded physical ownership. Seven native suites +pass 246 cases; 36 recovery-related unit tests, eleven legacy disaster cases and +the exact real RustFS import case in both image forms pass. Scoped static checks +and independent source/artifact/selection/credential audit pass. Evidence is at +`.cellar/cell-groups-evidence/CG-06-recovery-import/`; qualification SHA256: +`1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3`. See [progress](progress.md) for scope and retained limitations. + +The mandatory nullable recovery field is a marker-1 hard cutover, with no reader +for older intermediate cell-group manifest/archive shapes. Keep recovery state +through snapshots and retained reopen. `PreparedCellGroupRecovery` is data owned +by an archive store; never cast that owner into a Raft engine or learner permit. +Its source hold remains provenance and must stay retained while referenced. + +The most critical next work is physical fresh genesis and safe activation: + +1. Consume the exact preparation through a node/member/disk/process/engine-bound + recovery permit and atomically sync the complete imported root with the new + Raft mapping/genesis. Every seeded voter must bind the same request and seed + commitment. Preserve physical pins through unknown sync results. Do not copy + old Raft terms/configuration or reopen an already owned page directory. +2. Persist the target recovery binding before publication. Ordinary bootstrap + must refuse blank replacement disks or an empty/incorrect genesis. Reverify + the permanent external fence on process resume. Continue using the accepted + fencing-authority API; no new approval is required for Docker qualification. +3. Publish the exact replacement mutation with one CAS under the existing + recovery lock. Never clear the lock before ordinary publication. Mint a + catalog-bound leader activation permit only after exact readback and seed + validation. Raw administration remains unattached. Commit, persist and apply + activation through the fresh quorum before opening any application authority. +4. Implement the original `fenced_group_seed_new_incarnations` oracle with real + registered commands, receipt results/minima/client floors, old-token refusal + and zero duplicate callbacks. The current import checks exact SQLite bytes; + it does not qualify the new serving receipt path. Exercise pending/active + recovered roots through actual engine restart and snapshot installation. +5. Qualify Docker permanent launch/restart revocation for every possible old + writer, including later/removed voters. Erase every owned voter disk and local + recovery cache, restore all cells from real RustFS under fresh identities, + perform new writes and restart the retained replacement quorum. Keep effect + ambiguity closed. Complete the original full CG-06 gate before marking DONE. + +CG-08 is next after CG-06: replicated effect summaries exist, but due-work +scheduling independent of client requests and bounded idle SQLite caches remain +missing. Preserve effect identity, unknown outcomes and joined worker ownership. +Then perform CG-07 public SDK/fleet cutover. CG-09–12 transfers, split/merge, +placement and hibernation; CG-14/13/15 workload and managed recovery/qualification; +and CG-16 final legacy removal and release matrix all remain required. The full +hard-cutover goal remains active and incomplete. + +The prior whole-group fencing-barrier source is +65c5176975907fb52a53deca58e1ae1f5ba7adec, with 211 native cases and four real RustFS +resume/race scenarios. Its evidence remains at +`.cellar/cell-groups-evidence/CG-06-recovery-fence/`. Historical qualifications +remain scoped to their recorded sources. The prior bounded-outage qualification source is 1159e9358c106fe1260c3479759e909e413257a4. CG-06 remains IN_PROGRESS. Its diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 9882aad1f..f37676428 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,71 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 fresh-history data import and pending admission + +Source: 7052d218d60181d7b0bb5a4c12686e80e294a61c. CG-06 remains IN_PROGRESS. A fenced, +complete source cut can now be prepared into dedicated replacement stores using +`CellGroupRecoveryPool::prepare`. Every cell receives the replacement group's +fresh incarnation, sequence 1, a new enrollment record and a genesis witness. +SQLite bytes stay exact; materialized generation identity is rederived and extent +history metadata is rebound while verified payload identities are reused. The +source intent and its retained remote hold preserve original-witness provenance. +No application callback or Raft engine is created by this data preparation. + +The replacement manifest carries mandatory nullable recovery lifecycle metadata. +Pending roots refuse writes, enrollment, stale/strong reads, receipt reconciliation, +membership changes and learner seed authority. No-ops preserve pending state. +Snapshot receive and selected-root reopen retain the full marker; storage refuses +forgetting or replacing recovery lineage. Deterministic apply checks the exact +activation operation, proposal and seed commitment, preserves the first activation +on an identical retry and rejects conflicting activation. A catalog-bound leader +activation admission API and actual replacement-quorum activation remain required. + +One node-wide import pool separately bounds cells, restored bytes, metadata and +image workspace. Result/root clones retain import admission. Cancellation during +actual disk sync retains source pins and permits until entered physical work +finishes; shutdown joins it. Clones of the fenced data proof share one verified +cut and its existing admission when preparing multiple destinations. + +At this source, seven native shared-group suites pass 246 cases; 36 recovery-related +unit cases and eleven legacy disaster cases also pass. The exact real RustFS +import case passes in both image forms with seven signed S3 probes. It checks +all-cell byte equality, fresh identities/witnesses, old-token rejection, admission +limits and stable retries. All-target locked Cellar/logic Clippy, format/layout +(1971 Rust files), native dependency/docs/preflight/plan checks, 37 plan tests and +five backend-controller tests pass. The owned RustFS backend/profile are stopped +with data retained and the default Docker context preserved. Independent audit +checks 2588 source inputs, Git blobs/symlinks, copied executables, exact case +selection and credential exclusion. + +Evidence: `.cellar/cell-groups-evidence/CG-06-recovery-import/qualification/record.json` +(SHA256 `1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3`); independent audit +`1eae08c2bd03eb6fc42d81ae665533716aff5c90cc9217e92221733119d5b75d`. The initial test-fixture private-import compilation error and +its interrupted compiler are retained in `focus-01/`; corrected source passes. +The initial uncommitted-baseline preflight refusal is retained and the pinned +source preflight passes. Earlier socket/deadline/host-pressure and slow-reader +limitations remain historical; no causal fix is claimed for them. + +This follows the repository's internal marker-1 hard cutover: older intermediate +cell-group manifest/archive shapes have no compatibility reader. Current native +and RustFS fixtures use the complete new shape. Qualification here covers data +preparation and admission barriers. The replacement still has no engine genesis, +node/member/disk/engine seed binding, fenced publication or serving permission. +Docker voter fencing, fresh-quorum serving after erasing every voter disk, actual +registered-command duplicate receipt recovery and the original +`fenced_group_seed_new_incarnations` oracle remain pending. + +Next, consume the complete typed preparation through atomic fresh-genesis storage +and durable physical identity claims. Ordinary bootstrap must refuse an empty or +missing replacement seed after publication. Publish exactly once under the +existing recovery lock, then admit activation only with a catalog-bound permit +and prove its actual commit/apply through a fresh quorum. Reverify fencing on +resume and keep the source hold while its provenance is referenced. Complete the +accepted Docker authority drill and the original full CG-06 gate before marking +this task DONE. Then CG-08 due effects and safe cold-cache eviction precede CG-07 +public SDK/fleet cutover; transfer, split/merge, placement, hibernation, load/crash +qualification and final legacy removal remain required. + ## CG-06 exact whole-group recovery intents and fencing barrier Source: 65c5176975907fb52a53deca58e1ae1f5ba7adec. CG-06 remains IN_PROGRESS. The new From 12944b762631c300ec6da2a8f6528394a19e4eda Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 15:13:47 -0700 Subject: [PATCH 248/568] feat: activate recovered cell-group quorums Bind each restored genesis to a permanent physical seed receipt before publishing its replacement. Require independently fenced startup and a committed activation before application admission, retaining topology ownership through completion. Share immutable genesis metadata through engine caches and box cold recovery operations to bound handles and polling frames. --- crates/cellar/durability/raft/cache.rs | 9 + crates/cellar/durability/raft/catalog.rs | 12 +- .../durability/raft/cell_groups/README.md | 36 + .../raft/cell_groups/directory/birth.rs | 61 +- .../raft/cell_groups/directory/bootstrap.rs | 54 +- .../cell_groups/directory/joined_bootstrap.rs | 2 +- .../raft/cell_groups/directory/mod.rs | 36 +- .../directory/recovery/activation.rs | 329 ++++++++ .../directory/recovery/bootstrap.rs | 96 +++ .../cell_groups/directory/recovery/mod.rs | 40 +- .../directory/recovery/publication.rs | 220 ++++++ .../cell_groups/directory/recovery/seed.rs | 381 +++++++++ .../raft/cell_groups/directory/wire.rs | 19 + .../cellar/durability/raft/cell_groups/mod.rs | 20 +- .../durability/raft/cell_groups/owner.rs | 187 ++++- .../raft/cell_groups/recovery/genesis.rs | 196 +++++ .../raft/cell_groups/recovery/import.rs | 4 + .../raft/cell_groups/recovery/mod.rs | 3 + .../raft/cell_groups/recovery/state.rs | 15 + .../durability/raft/cell_groups/retention.rs | 34 +- .../durability/raft/cell_groups/root.rs | 59 ++ .../raft/cell_groups/runtime/mod.rs | 23 + .../raft/cell_groups/runtime/reads.rs | 3 + .../raft/cell_groups/runtime/worker.rs | 4 + crates/cellar/durability/raft/member_state.rs | 6 + .../cellar/durability/raft/node_directory.rs | 57 ++ crates/cellar/durability/raft/records.rs | 53 +- .../cellar/durability/raft/snapshot_seed.rs | 1 + crates/cellar/durability/raft/storage.rs | 108 ++- .../tests/native_cell_groups_archive.rs | 50 ++ .../tests/support/cell_group_archive_store.rs | 13 + .../tests/support/cell_group_checkpoint.rs | 2 +- .../support/cell_group_disaster_activation.rs | 748 ++++++++++++++++++ .../support/cell_group_disaster_fence.rs | 23 +- .../tests/support/cell_group_disaster_seed.rs | 294 +++++++ .../multi-raft/cell-groups/baseline.json | 16 +- docs/design/multi-raft/cell-groups/tasks.json | 9 +- 37 files changed, 3135 insertions(+), 88 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs create mode 100644 crates/cellar/durability/raft/cell_groups/recovery/genesis.rs create mode 100644 crates/cellar/tests/support/cell_group_disaster_activation.rs create mode 100644 crates/cellar/tests/support/cell_group_disaster_seed.rs diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index cf88f9f67..83e048e79 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -248,6 +248,12 @@ impl GroupLog { Ok(cache.applied.clone()) } + /// Immutable mapping metadata, shared by the engine cache. This observation + /// grants neither complete physical recovery nor startup authority. + pub(crate) fn recovery_genesis(&self) -> Result>, LogError> { + Ok(self.read()?.recovery_genesis.clone()) + } + /// Engine-selected metadata only. Reverify every page/image before admission. pub fn group_manifest(&self) -> Result, LogError> { Ok(self.read()?.group_manifest.clone()) @@ -466,6 +472,8 @@ pub(crate) struct CachedGroup { pub entries: BTreeMap, pub applied: DurableApply, pub group_manifest: Option, + /// Derived from the immutable mapping; never supplied by a write/snapshot. + pub recovery_genesis: Option>, pub manifest_pins: Option, pub checkpoint_manifest: Option, /// Local receipt; never imported from another node's checkpoint. @@ -998,6 +1006,7 @@ pub(crate) fn genesis(conf_state: ConfState) -> CachedGroup { snapshot, entries: BTreeMap::new(), group_manifest: None, + recovery_genesis: None, manifest_pins: None, checkpoint_manifest: None, group_seed: None, diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 554f21caf..56c8016be 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -24,14 +24,16 @@ pub use births::BirthDisposition; #[path = "cell_groups/directory/mod.rs"] mod range_directory; +pub(crate) use range_directory::RecoverySeedPermit; pub(crate) use range_directory::RouteCache; pub use range_directory::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryError, - CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, - CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, - FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, + CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, + CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, + RangeDirectory, ResolvedCellRoute, }; #[path = "catalog_inventory/mod.rs"] diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index aa97aeb88..c8a5de1da 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -321,6 +321,42 @@ namespaces refuse these unmanaged APIs until their rooted fleet authorization is connected. This staged boundary does not permit blank-disk recovery or replace current quorum membership. +## Fresh-history disaster recovery + +Unmanaged recovery preserves one canonical `CellGroupRecoveryRequest` through +all unknown outcomes. `fence_group_recovery` verifies its exact held archive cut +and an independently verified permanent infrastructure fence. The authority must +stop every possible old writer and revoke restart; unreachable hosts and S3 +markers cannot supply that proof. Retain the source hold while its provenance is +referenced. + +`CellGroupRecoveryPool::prepare` restores complete data under fresh cell histories. +It grants no vote. `seed_group_recovery` binds the imported genesis to the actual +node, member, disk and engine, atomically syncs its mapping/root and then confirms +a permanent seed receipt. A lost waiter cannot release entered physical work. +`publish_group_recovery` requires every initial member to have the same complete +seed commitment and retains the topology lock after selecting the replacement. + +Ordinary bootstrap refuses recovery births. `bootstrap_recovered_group_member` +requires the existing seed, immutable engine genesis, storage pin and selected +publication, and rechecks the external fence. Physical reopen must verify the +complete selected and checkpoint closures before attachment. Startup alone grants +no serving authority: pending roots reject application work, including reads. + +`prepare_group_recovery_activation` issues an engine/process-bound permit for +`CellGroupRuntime::activate_recovery`. The owner requires a fresh-term quorum +barrier and admits one bounded activation proposal. Only committed, persisted +and applied activation opens application admission. An applied acknowledgement +allows `complete_group_recovery` to confirm the activation receipt and release +the topology lock. Repeated attempts preserve the first activation position. +Rare catalog operations own boxed futures to bound composed control polling +frames; they do not start per-cell tasks. + +Recovery origin, seed and activation records use the internal marker-1 hard +cutover, with no reader for older intermediate shapes. These low-level contracts +do not complete the public SDK/fleet cutover; implementation and qualification +status remains in the [design progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). + ## Physical log identity cutover The engine registry uses `GroupHistory` (cluster, namespace, group and incarnation), diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs index 167570d12..65acc29a1 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/birth.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -204,19 +204,49 @@ impl RangeDirectory { pub async fn reserve_group_birth( &self, request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + self.reserve_birth_record( + request, + &wire::BirthReservation { + birth: request.encode().to_vec(), + recovery: None, + }, + ) + .await + } + + pub(super) async fn reserve_recovery_birth( + &self, + request: &super::CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + self.reserve_birth_record( + request.replacement(), + &wire::BirthReservation { + birth: request.replacement().encode().to_vec(), + recovery: Some(request.digest()), + }, + ) + .await + } + + async fn reserve_birth_record( + &self, + request: &CellGroupBirthRequest, + reservation: &wire::BirthReservation, ) -> Result<(), CatalogError> { self.decode_group_birth(request.encode())?; + let bytes = record::encode(wire::GROUP_BIRTH_RESERVATION, reservation, wire::LIMIT)?; self.bounded(async { let observed = self .namespace .inner .create_and_observe_bounded( &self.reservation_key(request.route()), - request.encode(), + &bytes, wire::LIMIT, ) .await?; - if observed != request.encode() { + if observed != bytes { return Err(CatalogError::Conflict("group birth identity reserved")); } Ok(()) @@ -228,6 +258,24 @@ impl RangeDirectory { &self, route: &GroupRoute, ) -> Result { + self.birth_reservation(route).await.map(|record| record.0) + } + + pub(super) async fn require_ordinary_birth_reservation( + &self, + route: &GroupRoute, + ) -> Result { + let (request, recovery) = self.birth_reservation(route).await?; + if recovery.is_some() { + return Err(CatalogError::Fenced); + } + Ok(request) + } + + pub(super) async fn birth_reservation( + &self, + route: &GroupRoute, + ) -> Result<(CellGroupBirthRequest, Option<[u8; 32]>), CatalogError> { let bytes = self .namespace .inner @@ -235,11 +283,16 @@ impl RangeDirectory { .await? .ok_or(CatalogError::Corrupt("missing group birth reservation"))? .0; - let request = self.decode_group_birth(&bytes)?; + let reservation: wire::BirthReservation = + record::decode(wire::GROUP_BIRTH_RESERVATION, &bytes, wire::LIMIT)?; + if reservation.recovery == Some([0; 32]) { + return Err(CatalogError::Corrupt("recovery reservation identity")); + } + let request = self.decode_group_birth(&reservation.birth)?; if request.route() != route { return Err(CatalogError::Conflict("group birth reservation binding")); } - Ok(request) + Ok((request, reservation.recovery)) } async fn selected_birth_route(&self, route: &GroupRoute) -> Result<(), CatalogError> { diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs index 1adb179fd..584b81502 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -22,10 +22,32 @@ pub struct CellGroupBootstrapPermit { birth: CellGroupBirth, directory: RangeDirectory, log: ReplicaLog, - joined: bool, + origin: BootstrapOrigin, +} + +enum BootstrapOrigin { + Initial, + Joined, + Recovery(Box), } impl CellGroupBootstrapPermit { + pub(in crate::raft::catalog::range_directory) fn recovered( + group: GroupLog, + birth: CellGroupBirth, + directory: RangeDirectory, + log: ReplicaLog, + startup: recovery::RecoveryStartup, + ) -> Self { + Self { + group, + birth, + directory, + log, + origin: BootstrapOrigin::Recovery(Box::new(startup)), + } + } + pub(crate) fn log(&self) -> &ReplicaLog { &self.log } @@ -39,16 +61,26 @@ impl CellGroupBootstrapPermit { } pub async fn revalidate(&self) -> Result<(), CatalogError> { - if self.joined { - self.directory + match &self.origin { + BootstrapOrigin::Initial => self + .directory + .bootstrap_group_member(&self.birth, &self.log) + .await + .map(|_| ()), + BootstrapOrigin::Joined => self + .directory .bootstrap_joined_group_member(&self.birth, self.group.identity().member, &self.log) .await - .map(|_| ()) - } else { - self.directory - .bootstrap_group_member(&self.birth, &self.log) + .map(|_| ()), + BootstrapOrigin::Recovery(startup) => self + .directory + .bootstrap_recovered_group_member( + &startup.request, + &self.log, + startup.authority.clone(), + ) .await - .map(|_| ()) + .map(|_| ()), } } } @@ -66,7 +98,7 @@ struct Claim { engine: [u8; 16], } -fn check_member(group: &GroupLog) -> Result<(), CatalogError> { +pub(super) fn check_member(group: &GroupLog) -> Result<(), CatalogError> { let cache = group.read().map_err(CatalogError::Log)?; if !crate::raft::member_state::members(&cache.conf_state) .contains(&group.identity().member.get()) @@ -98,6 +130,8 @@ impl RangeDirectory { self.unmanaged_birth()?; self.bounded(async { self.require_current_group_birth(birth).await?; + self.require_ordinary_birth_reservation(birth.route()) + .await?; let selected = birth .voters() .iter() @@ -167,7 +201,7 @@ impl RangeDirectory { birth: birth.clone(), directory: self.clone(), log: log.clone(), - joined: false, + origin: BootstrapOrigin::Initial, }) }) .await diff --git a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs index b9786e3c8..283bb4568 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs @@ -144,7 +144,7 @@ impl RangeDirectory { birth: birth.clone(), directory: self.clone(), log: log.clone(), - joined: true, + origin: BootstrapOrigin::Joined, }) }) .await diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index c4a852f39..c1e2a083d 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -22,8 +22,10 @@ pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding} mod birth; mod bootstrap; mod recovery; +pub(crate) use recovery::RecoverySeedPermit; pub use recovery::{ - CellGroupDisasterFenceRequest, CellGroupRecoveryError, CellGroupRecoveryRequest, + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, FencedCellGroupRecovery, }; mod route_cache; @@ -455,12 +457,13 @@ impl RangeDirectory { mutation: &DirectoryMutation, ) -> Result { self.check_request(&mutation.request)?; - self.bounded(self.publish_inner(mutation)).await + self.bounded(self.publish_inner(mutation, None)).await } async fn publish_inner( &self, mutation: &DirectoryMutation, + recovery: Option<&recovery::RecoveryPublication>, ) -> Result { let request = &mutation.request; let current = if request.expected.is_some() { @@ -487,14 +490,18 @@ impl RangeDirectory { } // An interrupted genesis resumes only its identical first head. } - if view.head.recovery.is_some() { - return Err(CatalogError::Fenced); - } - if request - .expected - .is_some_and(|expected| expected != view.digest) - { - return Err(CatalogError::Conflict("stale directory root")); + if let Some(recovery) = recovery { + recovery.check(self, view, mutation)?; + } else { + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + if request + .expected + .is_some_and(|expected| expected != view.digest) + { + return Err(CatalogError::Conflict("stale directory root")); + } } } let base = if request.expected.is_some() { @@ -512,7 +519,12 @@ impl RangeDirectory { // A route can be lazy, but its exact cell-independent birth and global // group identity must be reserved before it can enter the directory. for child in &request.children { - self.verify_birth_reservation(&child.decode(self)?).await?; + let route = child.decode(self)?; + if let Some(recovery) = recovery { + recovery.check_child(&route)?; + } else { + self.require_ordinary_birth_reservation(&route).await?; + } } for parent in &request.parents { tree = self.remove(tree, parent.clone(), 0).await?; @@ -557,7 +569,7 @@ impl RangeDirectory { operations, groups, journal: id, - recovery: None, + recovery: recovery.map(|grant| grant.published_lock(id)).transpose()?, }; self.check_head(&head)?; let bytes = record::encode(wire::HEAD, &head, wire::LIMIT)?; diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs new file mode 100644 index 000000000..d177880aa --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs @@ -0,0 +1,329 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A catalog grant admits one activation proposal. Only the real Raft owner can +//! produce an applied acknowledgement and release the topology recovery lock. +use super::*; +use crate::raft::{ + cache::GroupLog, cell_groups::GroupManifest, completion::RuntimeError, storage::ReplicaLog, +}; +use cellar_logic::consensus::{EngineInstanceId, LogIndex, LogPosition, Term}; +use std::sync::atomic::Ordering; + +pub struct CellGroupRecoveryActivationPermit { + request: Arc, + group: GroupLog, + engine: EngineInstanceId, + publication: [u8; 32], + manifest: [u8; 32], + deadline: u64, +} + +impl CellGroupRecoveryActivationPermit { + pub fn history(&self) -> GroupHistory { + self.group.identity().scope + } + + pub(crate) fn check( + &self, + log: &ReplicaLog, + group: &GroupLog, + ) -> Result<(), RuntimeError> { + if !Arc::ptr_eq(&group.owner, &self.group.owner) + || group.identity() != self.group.identity() + || log.storage_instance()? != self.engine + || log.domain().mono_ms().map_err(|_| RuntimeError::Closed)? >= self.deadline + { + return Err(RuntimeError::StaleCompletion); + } + drop(group.read()?); + Ok(()) + } + + pub(crate) fn matches(&self, marker: &GroupManifest) -> Result { + if marker.route() != self.request.replacement().route() + || marker.recovery_request() != Some(self.request.digest()) + || marker + .recovery_commitment() + .map_err(|e| RuntimeError::Invalid(e.to_string()))? + != Some(self.manifest) + { + return Err(RuntimeError::StaleCompletion); + } + match marker.recovery_activation() { + Some((publication, manifest, _)) + if publication == self.publication && manifest == self.manifest => + { + Ok(true) + } + Some(_) => Err(RuntimeError::StaleCompletion), + None => Ok(false), + } + } + + pub(crate) fn entry(&self) -> Result, RuntimeError> { + use crate::application::consensus::group_protocol::{ + CellGroupEntry, GroupCommand, GroupEntryBinding, GroupEntryLimits, + }; + let route = self.request.replacement().route(); + CellGroupEntry { + binding: GroupEntryBinding { + group: route.history, + route_epoch: route.epoch, + birth_digest: route.birth_digest, + schema_digest: route.schema_digest, + proposal: self.request.operation(), + }, + command: GroupCommand::Activate { + transition: TransitionId::new(self.request.operation().bytes()) + .map_err(|e| RuntimeError::Invalid(e.to_string()))?, + publication: self.publication, + manifest: self.manifest, + }, + } + .encode(route, GroupEntryLimits::default()) + .map_err(|e| RuntimeError::Invalid(e.to_string())) + } + + pub(crate) fn acknowledgement( + &self, + group: &GroupLog, + selected: &GroupManifest, + ) -> Result { + let cache = group.read()?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + if marker != selected || !self.matches(marker)? { + return Err(RuntimeError::NotReady); + } + let (_, _, position) = marker.recovery_activation().ok_or(RuntimeError::NotReady)?; + if cache.hard_state.commit < position.index.get() { + return Err(RuntimeError::NotReady); + } + Ok(CellGroupRecoveryActivationAck { + group: group.clone(), + request: self.request.digest(), + publication: self.publication, + manifest: self.manifest, + position, + }) + } +} + +/// An observation of activation committed and selected by an actual local Raft +/// owner. This is neither a transferable serving grant nor an archive witness. +pub struct CellGroupRecoveryActivationAck { + group: GroupLog, + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + position: LogPosition, +} + +impl CellGroupRecoveryActivationAck { + pub fn position(&self) -> LogPosition { + self.position + } + + fn check( + &self, + request: &CellGroupRecoveryRequest, + publication: [u8; 32], + manifest: [u8; 32], + ) -> Result<(), CatalogError> { + if self.request != request.digest() + || self.publication != publication + || self.manifest != manifest + || self.group.closed.load(Ordering::Acquire) + || self.group.identity().scope != request.replacement().route().history + { + return Err(CatalogError::Fenced); + } + let cache = self.group.read().map_err(CatalogError::Log)?; + let root = cache + .group_manifest + .as_ref() + .ok_or(CatalogError::LostStableState)?; + if root.recovery_request() != Some(self.request) + || root.recovery_activation() != Some((self.publication, self.manifest, self.position)) + || cache.hard_state.commit < self.position.index.get() + { + return Err(CatalogError::Fenced); + } + Ok(()) + } +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +struct Activated { + request: [u8; 32], + publication: [u8; 32], + manifest: [u8; 32], + index: u64, + term: u64, +} + +impl RangeDirectory { + pub(super) async fn require_recovery_activation( + &self, + request: &CellGroupRecoveryRequest, + publication: &DirectoryPublication, + manifest: [u8; 32], + ) -> Result<(), CatalogError> { + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.recovery_key(request, "activation"), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery activation"))? + .0; + let activated: Activated = + record::decode(wire::GROUP_RECOVERY_ACTIVATED, &bytes, wire::LIMIT)?; + if activated.request != request.digest() + || activated.publication != publication.journal_digest() + || activated.manifest != manifest + || activated.index <= 1 + || activated.term <= 1 + || (LogPosition { + index: LogIndex::new(activated.index), + term: Term::new(activated.term), + }) + .validate() + .is_err() + { + return Err(CatalogError::Corrupt("recovery activation binding")); + } + Ok(()) + } + + /// Recheck publication, physical member identity and the permanent external + /// fence before admitting the exact activation to the actual Raft owner. + pub fn prepare_group_recovery_activation<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + log: &'a ReplicaLog, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> + { + Box::pin(async move { + let request = Arc::new(self.decode_group_recovery(request.encode())?); + self.bounded(async { + self.verify_recovery_fence(&request, authority).await?; + let group = self.recovered_member(&request, log).await?; + let (publication, binding, _) = self.published_recovery(&request).await?; + let deadline = log + .domain() + .mono_ms() + .map_err(|_| CatalogError::Unconfirmed)? + .checked_add(self.limits.operation_timeout_ms) + .ok_or(CatalogError::Limit)?; + Ok(CellGroupRecoveryActivationPermit { + request, + group, + engine: log.storage_instance().map_err(CatalogError::Log)?, + publication: publication.journal_digest(), + manifest: binding.commitment, + deadline, + }) + }) + .await + }) + } + + /// Release topology only after committed/applied activation. Unknown S3 + /// responses retain the original request and are reconciled by read-back. + pub fn complete_group_recovery<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + acknowledgement: &'a CellGroupRecoveryActivationAck, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = self.decode_group_recovery(request.encode())?; + self.bounded(async { + self.verify_recovery_fence(&request, authority).await?; + let (publication, binding, _) = self.published_recovery(&request).await?; + acknowledgement.check( + &request, + publication.journal_digest(), + binding.commitment, + )?; + let position = acknowledgement.position(); + let complete = record::encode( + wire::GROUP_RECOVERY_ACTIVATED, + &Activated { + request: request.digest(), + publication: publication.journal_digest(), + manifest: binding.commitment, + index: position.index.get(), + term: position.term.get(), + }, + wire::LIMIT, + )?; + if self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_key(&request, "activation"), + &complete, + wire::LIMIT, + ) + .await? + != complete + { + return Err(CatalogError::Conflict("recovery activation receipt")); + } + let (mut view, version) = self.current().await?; + self.selected_route_in(&view, request.replacement().route()) + .await?; + if view.head.recovery.is_none() { + self.require_recovery_activation(&request, &publication, binding.commitment) + .await?; + return Ok(publication); + } + if view.head.recovery.as_deref().is_none_or(|lock| { + lock.request != request.digest() + || lock.phase + != (RecoveryPhase::Published { + publication: publication.journal_digest(), + manifest: binding.commitment, + }) + }) { + return Err(CatalogError::Fenced); + } + acknowledgement.check( + &request, + publication.journal_digest(), + binding.commitment, + )?; + view.head.recovery = None; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + let (after, _) = self + .current() + .await + .map_err(|_| CatalogError::Unconfirmed)?; + self.require_recovery_activation(&request, &publication, binding.commitment) + .await?; + if after.head.recovery.is_some() { + return Err(CatalogError::Unconfirmed); + } + self.confirmed(&after.head, request.mutation()) + .await? + .ok_or(CatalogError::Unconfirmed) + }) + .await + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs new file mode 100644 index 000000000..0c03b4621 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs @@ -0,0 +1,96 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Recovered startup never creates an empty mapping or a missing storage pin. +use super::*; +use crate::raft::{ + cache::GroupLog, + storage::{LogError, ReplicaLog}, +}; + +pub(in crate::raft::catalog::range_directory) struct RecoveryStartup { + pub(in crate::raft::catalog::range_directory) request: Arc, + pub(in crate::raft::catalog::range_directory) authority: + Arc>, +} + +impl RangeDirectory { + pub(super) async fn recovered_member( + &self, + request: &CellGroupRecoveryRequest, + log: &ReplicaLog, + ) -> Result, CatalogError> { + let (_, binding, digest) = self.published_recovery(request).await?; + let selected = request + .replacement() + .voters() + .iter() + .chain(request.replacement().learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + let receipt = self + .recovery_seed_receipt(request, selected, digest) + .await?; + receipt.check_log(log)?; + let group = log + .cell_group(&request.replacement().route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + if group.identity().member != selected.member + || group + .recovery_genesis() + .map_err(CatalogError::Log)? + .as_deref() + != Some(&binding.genesis) + { + return Err(CatalogError::LostStableState); + } + let marker = group + .group_manifest() + .map_err(CatalogError::Log)? + .ok_or(CatalogError::LostStableState)?; + if marker.route() != request.replacement().route() + || marker.recovery_request() != Some(request.digest()) + || marker + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("recovery lineage"))? + != Some(binding.commitment) + { + return Err(CatalogError::LostStableState); + } + super::super::bootstrap::check_member(&group)?; + Ok(group) + } + + /// Resume only the exact physically seeded member of the selected recovery. + /// The owner must still verify all local bytes and establish a fresh-term + /// quorum barrier; the pending lifecycle refuses every application request. + pub fn bootstrap_recovered_group_member<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + log: &'a ReplicaLog, + authority: Arc>, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = Arc::new(self.decode_group_recovery(request.encode())?); + self.bounded(async { + self.verify_recovery_fence(&request, authority.as_ref()) + .await?; + let group = self.recovered_member(&request, log).await?; + let birth = self.materialize_group_birth(request.replacement()).await?; + self.verify_recovery_fence(&request, authority.as_ref()) + .await?; + self.published_recovery(&request).await?; + Ok(CellGroupBootstrapPermit::recovered( + group, + birth, + self.clone(), + log.clone(), + RecoveryStartup { request, authority }, + )) + }) + .await + }) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs index 2ee76535c..8fc2ae4d9 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/mod.rs @@ -1,8 +1,14 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Exact, irreversible disaster fencing before fresh-genesis installation. -//! The returned data proof grants no bootstrap, voting or serving permission. +//! Irreversible fencing, complete fresh genesis, publication and activation. +//! Archive data and catalog observations never grant quorum or serving authority. +//! Rare catalog operations own boxed futures so composing them cannot inflate +//! a controller polling frame with multiple complete recovery state machines. +mod activation; +mod bootstrap; +mod publication; mod request; +mod seed; use super::*; use crate::raft::{ catalog::DisasterFenceAuthority, @@ -12,7 +18,11 @@ use crate::raft::{ CellGroupArchiveTailPool, }, }; +pub use activation::{CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit}; +pub(in crate::raft::catalog::range_directory) use bootstrap::RecoveryStartup; +pub(in crate::raft::catalog::range_directory) use publication::RecoveryPublication; pub use request::{CellGroupDisasterFenceRequest, CellGroupRecoveryRequest}; +pub(crate) use seed::RecoverySeedPermit; use serde::{Deserialize, Serialize}; #[derive(Debug)] @@ -65,6 +75,17 @@ pub(in crate::raft::catalog::range_directory) struct RecoveryLock { request: [u8; 32], base: [u8; 32], source: wire::Route, + phase: RecoveryPhase, +} + +#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +enum RecoveryPhase { + Source, + Published { + publication: [u8; 32], + manifest: [u8; 32], + }, } impl RecoveryLock { @@ -72,6 +93,15 @@ impl RecoveryLock { if self.request == [0; 32] || self.base == [0; 32] { return Err(CatalogError::Corrupt("group recovery lock")); } + if let RecoveryPhase::Published { + publication, + manifest, + } = self.phase + { + if publication == [0; 32] || manifest == [0; 32] { + return Err(CatalogError::Corrupt("published recovery lock")); + } + } self.source_route(directory).map(|_| ()) } @@ -163,7 +193,8 @@ impl RangeDirectory { ) -> Result<(), CatalogError> { match &view.head.recovery { Some(lock) - if lock.request == request.digest() + if lock.phase == RecoveryPhase::Source + && lock.request == request.digest() && Some(lock.base) == request.mutation().request.expected && lock.source_route(self)? == *request.source().route() => { @@ -190,6 +221,7 @@ impl RangeDirectory { request: request.digest(), base: view.digest, source: wire::Route::encode(request.source().route(), self)?, + phase: RecoveryPhase::Source, })); let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; // A lost response can follow a committed CAS. Observe the exact @@ -251,7 +283,7 @@ impl RangeDirectory { .recovery() .check_disaster_target(request.replacement())?; self.namespace.claim_disaster_namespace().await?; - self.reserve_group_birth(request.replacement()).await?; + self.reserve_recovery_birth(&request).await?; self.bounded(async { let observed = self .namespace diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs new file mode 100644 index 000000000..2368039dc --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs @@ -0,0 +1,220 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The same complete genesis must be durably seeded everywhere before the +//! topology CAS. Keep its lock until replicated activation has been confirmed. +use super::*; +use crate::raft::cell_groups::GroupManifest; + +pub(in crate::raft::catalog::range_directory) struct RecoveryPublication { + request: Arc, + manifest: [u8; 32], +} + +impl RecoveryPublication { + pub(in crate::raft::catalog::range_directory) fn check( + &self, + directory: &RangeDirectory, + view: &DirectoryView, + mutation: &DirectoryMutation, + ) -> Result<(), CatalogError> { + if mutation.encode() != self.request.mutation().encode() { + return Err(CatalogError::Fenced); + } + directory.check_recovery_view(view, &self.request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) fn check_child( + &self, + route: &GroupRoute, + ) -> Result<(), CatalogError> { + if route != self.request.replacement().route() { + return Err(CatalogError::Fenced); + } + Ok(()) + } + + pub(in crate::raft::catalog::range_directory) fn published_lock( + &self, + publication: [u8; 32], + ) -> Result, CatalogError> { + Ok(Box::new(RecoveryLock { + request: self.request.digest(), + base: self + .request + .mutation() + .request + .expected + .ok_or(CatalogError::Invalid("recovery base"))?, + source: self + .request + .mutation() + .request + .parents + .first() + .ok_or(CatalogError::Invalid("recovery source"))? + .clone(), + phase: RecoveryPhase::Published { + publication, + manifest: self.manifest, + }, + })) + } +} + +impl RangeDirectory { + pub(super) async fn recovery_binding( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(seed::Binding, [u8; 32]), CatalogError> { + let (_, origin) = self + .birth_reservation(request.replacement().route()) + .await?; + if origin != Some(request.digest()) { + return Err(CatalogError::Fenced); + } + let bytes = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_binding_key(request.replacement().route()), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Conflict("recovery genesis not bound"))? + .0; + let binding: seed::Binding = + record::decode(wire::GROUP_RECOVERY_BINDING, &bytes, wire::LIMIT)?; + let genesis: &GroupManifest = &binding.genesis; + if binding.request != request.encode() + || !genesis.is_recovery_genesis() + || genesis.route() != request.replacement().route() + || genesis.recovery_request() != Some(request.digest()) + || genesis + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("recovery genesis"))? + != Some(binding.commitment) + { + return Err(CatalogError::Corrupt("recovery genesis binding")); + } + Ok((binding, record::digest(&bytes))) + } + + pub(super) async fn recovery_seed_receipt( + &self, + request: &CellGroupRecoveryRequest, + selected: &crate::raft::catalog::InitialMember, + binding: [u8; 32], + ) -> Result { + let bytes = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(request, selected.member.get(), "complete"), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Conflict("recovery member not seeded"))? + .0; + let receipt: seed::MemberSeed = + record::decode(wire::GROUP_RECOVERY_SEED_COMPLETE, &bytes, wire::LIMIT)?; + receipt.check(selected, binding)?; + let intent = record::encode(wire::GROUP_RECOVERY_SEED_INTENT, &receipt, wire::LIMIT)?; + let observed = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(request, selected.member.get(), "intent"), + wire::LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing recovery seed intent"))? + .0; + if observed != intent { + return Err(CatalogError::Corrupt("recovery seed intent binding")); + } + self.namespace + .verify_node_storage_identity(selected.node, receipt.disk()?, receipt.engine()?) + .await?; + Ok(receipt) + } + + /// Publish only after all initial members have synced the same complete + /// pending genesis. This metadata result never grants voting or serving. + /// Retry the original request bytes after an unknown CAS outcome. + pub fn publish_group_recovery<'a>( + &'a self, + request: &'a CellGroupRecoveryRequest, + authority: &'a dyn DisasterFenceAuthority, + ) -> futures_util::future::BoxFuture<'a, Result> { + Box::pin(async move { + let request = Arc::new(self.decode_group_recovery(request.encode())?); + self.bounded(async { + if let Some(publication) = self.publication(request.mutation()).await? { + return Ok(publication); + } + let (view, _) = self.current().await?; + self.check_recovery_view(&view, &request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + self.recovery_source(&request).await?; + self.verify_recovery_fence(&request, authority).await?; + let (binding, digest) = self.recovery_binding(&request).await?; + let mut disks = BTreeSet::new(); + let mut engines = BTreeSet::new(); + for member in request + .replacement() + .voters() + .iter() + .chain(request.replacement().learners()) + { + let receipt = self.recovery_seed_receipt(&request, member, digest).await?; + if !disks.insert(receipt.disk()?) || !engines.insert(receipt.engine()?) { + return Err(CatalogError::Conflict("duplicate recovery storage")); + } + } + let permit = RecoveryPublication { + request: request.clone(), + manifest: binding.commitment, + }; + self.publish_inner(request.mutation(), Some(&permit)).await + }) + .await + }) + } + + pub(super) async fn published_recovery( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(DirectoryPublication, seed::Binding, [u8; 32]), CatalogError> { + let (view, _) = self.current().await?; + let publication = self + .confirmed(&view.head, request.mutation()) + .await? + .ok_or(CatalogError::Fenced)?; + self.selected_route_in(&view, request.replacement().route()) + .await?; + let (binding, digest) = self.recovery_binding(request).await?; + match view.head.recovery.as_deref() { + Some(lock) + if lock.request == request.digest() + && Some(lock.base) == request.mutation().request.expected + && lock.source_route(self)? == *request.source().route() + && lock.phase + == (RecoveryPhase::Published { + publication: publication.journal_digest(), + manifest: binding.commitment, + }) => {} + Some(_) => return Err(CatalogError::Fenced), + None => { + self.require_recovery_activation(request, &publication, binding.commitment) + .await? + } + } + Ok((publication, binding, digest)) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs b/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs new file mode 100644 index 000000000..7002c7cc4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs @@ -0,0 +1,381 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A permanent recovery origin and stable seed receipts precede topology +//! publication. Losing any response never authorizes a new disk or empty root. +use super::*; +use crate::raft::{ + cell_groups::{ + CellGroupApplier, GroupManifest, PreparedCellGroupRecovery, RecoverySeed, + SeededCellGroupRecovery, + }, + storage::{CellGroupBootstrap, LogError, ReplicaLog}, +}; +use cellar_logic::consensus::{EngineInstanceId, ReplicaIdentity}; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Binding { + pub(super) request: Vec, + pub(super) genesis: GroupManifest, + pub(super) commitment: [u8; 32], +} + +#[derive(Serialize, Deserialize, PartialEq, Eq)] +#[serde(deny_unknown_fields)] +pub(super) struct MemberSeed { + binding: [u8; 32], + member: u64, + node: [u8; 16], + disk: [u8; 16], + engine: [u8; 16], +} + +impl MemberSeed { + pub(super) fn disk(&self) -> Result { + cellar_logic::consensus::DiskLineage::new(self.disk) + .map_err(|_| CatalogError::Corrupt("recovery seed disk")) + } + + pub(super) fn engine(&self) -> Result { + EngineInstanceId::new(self.engine) + .map_err(|_| CatalogError::Corrupt("recovery seed engine")) + } + + pub(super) fn check( + &self, + selected: &crate::raft::catalog::InitialMember, + binding: [u8; 32], + ) -> Result<(), CatalogError> { + if self.binding != binding + || self.member != selected.member.get() + || self.node != selected.node.bytes() + { + return Err(CatalogError::Corrupt("recovery seed identity")); + } + self.disk()?; + self.engine()?; + Ok(()) + } + + pub(super) fn check_log(&self, log: &ReplicaLog) -> Result<(), CatalogError> { + if self.node != log.limits().node.bytes() + || self.disk()? != log.limits().disk + || self.engine()? != log.storage_instance().map_err(CatalogError::Log)? + { + return Err(CatalogError::LostStableState); + } + Ok(()) + } +} + +/// Process-local engine grant, minted only after the durable fence/origin and +/// exact seed intent are confirmed. It is not a startup or serving permission. +pub(crate) struct RecoverySeedPermit { + owner: Arc<()>, + local: ReplicaIdentity, + engine: EngineInstanceId, + request: Arc, + bootstrap: CellGroupBootstrap, + genesis: GroupManifest, + commitment: [u8; 32], +} + +impl RecoverySeedPermit { + pub(crate) fn owner(&self) -> &Arc<()> { + &self.owner + } + + pub(crate) fn request(&self) -> &CellGroupRecoveryRequest { + &self.request + } + + pub(crate) fn bootstrap(&self) -> &CellGroupBootstrap { + &self.bootstrap + } + + pub(crate) fn genesis(&self) -> &GroupManifest { + &self.genesis + } + + pub(crate) fn commitment(&self) -> [u8; 32] { + self.commitment + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + local: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + if !Arc::ptr_eq(owner, &self.owner) || local != &self.local || engine != self.engine { + return Err(LogError::WrongIdentity); + } + Ok(()) + } +} + +impl RangeDirectory { + pub(in crate::raft::catalog::range_directory::recovery) fn recovery_binding_key( + &self, + route: &GroupRoute, + ) -> Path { + self.key(&format!( + "recovery-bindings/{}/{}", + hex::encode(route.history.group.bytes()), + hex::encode(route.history.incarnation.bytes()) + )) + } + + pub(super) fn recovery_seed_key( + &self, + request: &CellGroupRecoveryRequest, + member: u64, + leaf: &str, + ) -> Path { + self.recovery_key(request, &format!("seeds/{member:016x}/{leaf}")) + } + + async fn require_locked_recovery( + &self, + request: &CellGroupRecoveryRequest, + ) -> Result<(), CatalogError> { + let (view, _) = self.current().await?; + self.check_recovery_view(&view, request)?; + if view.head.recovery.is_none() { + return Err(CatalogError::Fenced); + } + let (_, origin) = self + .birth_reservation(request.replacement().route()) + .await?; + if origin != Some(request.digest()) { + return Err(CatalogError::Corrupt("recovery birth origin")); + } + self.recovery_source(request).await?; + Ok(()) + } + + pub(super) async fn verify_recovery_fence( + &self, + request: &CellGroupRecoveryRequest, + authority: &dyn DisasterFenceAuthority, + ) -> Result<(), CatalogError> { + let request = self.decode_group_recovery(request.encode())?; + let fence = CellGroupDisasterFenceRequest::new(request); + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.recovery_key(fence.recovery(), "fence"), wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing recovery fence"))? + .0; + let complete: Complete = record::decode(wire::GROUP_FENCE, &bytes, wire::LIMIT)?; + if complete.request != fence.digest() { + return Err(CatalogError::Corrupt("recovery fence identity")); + } + check_evidence(&complete.evidence)?; + authority + .verify(&fence, &complete.evidence) + .await + .map_err(|e| CatalogError::Fence(e.to_string())) + } + + /// Sync the imported complete root as a fresh Raft genesis and confirm its + /// permanent physical seed receipt. The exact preparation can be cloned for + /// retry after a lost reply. The replacement remains closed to startup. + pub async fn seed_group_recovery( + &self, + domain: &impl CellGroupArchiveExecution, + prepared: PreparedCellGroupRecovery, + log: &ReplicaLog, + authority: &dyn DisasterFenceAuthority, + ) -> Result { + domain + .run(async { + let request = Arc::new(self.decode_group_recovery(prepared.request().encode())?); + self.bounded(self.require_locked_recovery(&request)).await?; + self.bounded(self.verify_recovery_fence(&request, authority)) + .await?; + if !prepared.manifest().marker().is_recovery_genesis() + || prepared.manifest().marker().route() != request.replacement().route() + || prepared.manifest().marker().recovery_request() != Some(request.digest()) + || prepared + .manifest() + .marker() + .recovery_commitment() + .map_err(|_| CatalogError::Corrupt("seed commitment"))? + != Some(prepared.commitment()) + { + return Err(CatalogError::Conflict("recovery preparation identity").into()); + } + let target = request.replacement(); + let member = target + .voters() + .iter() + .chain(target.learners()) + .find(|member| member.node == log.limits().node) + .ok_or(CatalogError::NotInitialMember)?; + if prepared + .source() + .coverage() + .recovery() + .membership() + .has_disk_binding(log.limits().disk) + { + return Err(CatalogError::Conflict("reused archived recovery disk").into()); + } + let nodes = request + .source() + .voters() + .iter() + .chain(request.source().learners()) + .map(|member| member.node) + .collect::>(); + self.bounded(self.namespace.verify_recovery_storage_fresh(log, &nodes)) + .await?; + let _node = self.namespace.require_node_storage(log).await?; + let binding_bytes = record::encode( + wire::GROUP_RECOVERY_BINDING, + &Binding { + request: request.encode().to_vec(), + genesis: prepared.manifest().marker().clone(), + commitment: prepared.commitment(), + }, + wire::LIMIT, + )?; + let binding = record::digest(&binding_bytes); + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_binding_key(target.route()), + &binding_bytes, + wire::LIMIT, + ) + .await?; + if observed != binding_bytes { + return Err(CatalogError::Conflict("recovery genesis binding")); + } + Ok(()) + }) + .await?; + let member_seed = MemberSeed { + binding, + member: member.member.get(), + node: member.node.bytes(), + disk: log.limits().disk.bytes(), + engine: log.storage_instance().map_err(CatalogError::Log)?.bytes(), + }; + let intent = + record::encode(wire::GROUP_RECOVERY_SEED_INTENT, &member_seed, wire::LIMIT)?; + let complete = record::encode( + wire::GROUP_RECOVERY_SEED_COMPLETE, + &member_seed, + wire::LIMIT, + )?; + self.bounded(async { + let existing = self + .namespace + .inner + .read_versioned_bounded( + &self.recovery_seed_key(&request, member_seed.member, "complete"), + wire::LIMIT, + ) + .await?; + if let Some((bytes, _)) = existing { + if bytes != complete { + return Err(CatalogError::Conflict("recovery seed receipt")); + } + log.cell_group(&target.route().history) + .map_err(|error| match error { + LogError::NotFound => CatalogError::LostStableState, + error => CatalogError::Log(error), + })?; + } + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_seed_key(&request, member_seed.member, "intent"), + &intent, + wire::LIMIT, + ) + .await?; + if observed != intent { + return Err(CatalogError::Conflict("recovery seed identity")); + } + Ok(()) + }) + .await?; + let local = ReplicaIdentity { + scope: target.route().history, + member: member.member, + node: member.node, + disk: log.limits().disk, + process: log.limits().process, + }; + let permit = RecoverySeedPermit { + owner: log.owner(), + local, + engine: log.storage_instance().map_err(CatalogError::Log)?, + request: request.clone(), + bootstrap: CellGroupBootstrap { + route: target.route().clone(), + member: member.member, + voters: target.voters().iter().map(|member| member.member).collect(), + learners: target + .learners() + .iter() + .map(|member| member.member) + .collect(), + }, + genesis: prepared.manifest().marker().clone(), + commitment: prepared.commitment(), + }; + let seed = + RecoverySeed::new(permit, prepared.clone()).map_err(CatalogError::Log)?; + let group = log + .install_recovery_seed(seed) + .await + .map_err(CatalogError::Log)?; + let store = prepared.store().clone(); + let genesis = prepared.manifest().marker().clone(); + let selected = group.clone(); + let retained = prepared.clone(); + let (applier, root) = domain + .blocking(move || { + let _retained = retained; + CellGroupApplier::recover_seeded(selected, &store, &genesis) + }) + .map_err(|_| CellGroupArchiveError::Stopped)? + .await + .map_err(CellGroupArchiveError::Worker)? + .map_err(CellGroupArchiveError::Replay)?; + self.bounded(self.require_locked_recovery(&request)).await?; + self.bounded(self.verify_recovery_fence(&request, authority)) + .await?; + self.bounded(async { + let observed = self + .namespace + .inner + .create_and_observe_bounded( + &self.recovery_seed_key(&request, member_seed.member, "complete"), + &complete, + wire::LIMIT, + ) + .await?; + if observed != complete { + return Err(CatalogError::Conflict("recovery seed receipt")); + } + self.require_locked_recovery(&request).await?; + Ok(()) + }) + .await?; + Ok(SeededCellGroupRecovery::new( + request, group, applier, root, binding, + )) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index 147cc84eb..b47364786 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -15,8 +15,27 @@ pub(super) const GROUP_CLAIM: u8 = 187; pub(super) const GROUP_ARCHIVE: u8 = 188; pub(super) const GROUP_RECOVERY: u8 = 189; pub(super) const GROUP_FENCE: u8 = 190; +pub(super) const GROUP_RECOVERY_BINDING: u8 = 191; +pub(super) const GROUP_RECOVERY_SEED_INTENT: u8 = 192; +pub(super) const GROUP_RECOVERY_SEED_COMPLETE: u8 = 193; +pub(super) const GROUP_BIRTH_RESERVATION: u8 = 194; +pub(super) const GROUP_RECOVERY_ACTIVATED: u8 = 195; pub(super) const LIMIT: usize = 64 * 1024; +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct BirthReservation { + pub birth: Vec, + #[serde(deserialize_with = "required_origin")] + pub recovery: Option<[u8; 32]>, +} + +fn required_origin<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(tag = "bound", content = "cell", deny_unknown_fields)] pub(super) enum Bound { diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index e78ace537..312a1ca59 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -81,7 +81,11 @@ pub use log_retention::{ }; mod read_authority; mod recovery; -pub use recovery::{CellGroupRecoveryLimits, CellGroupRecoveryPool, PreparedCellGroupRecovery}; +pub(crate) use recovery::RecoverySeed; +pub use recovery::{ + CellGroupRecoveryLimits, CellGroupRecoveryPool, PreparedCellGroupRecovery, + SeededCellGroupRecovery, +}; pub(crate) mod reclaim; pub use read_authority::{CellReadAuthority, CellReadConsistency}; pub use reclaim::{CellGroupReclaimError, CellGroupReclaimLimits, CellGroupReclaimReport}; @@ -104,11 +108,12 @@ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsag pub use crate::raft::catalog::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryError, - CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, - CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, - DirectoryLimits, DirectoryMutation, DirectoryPublication, DirectoryView, - FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, + CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, + CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, + CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, + DirectoryPublication, DirectoryView, FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, + RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, @@ -120,7 +125,8 @@ pub use root::{GroupManifest, PreparedGroupManifest}; pub use owner::{ CellGroupApply, CellGroupEffect, CellGroupOwner, CellGroupPersist, CellGroupPersisted, - CellGroupProposal, CellGroupSnapshotAdmission, CellGroupStatus, CellGroupTicket, + CellGroupProposal, CellGroupRecoveryActivation, CellGroupSnapshotAdmission, CellGroupStatus, + CellGroupTicket, }; pub(crate) use dispatch::RemoteMutationRoute; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index 75c872c79..c45cd4610 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -160,6 +160,12 @@ pub struct CellGroupStatus { pub ready: bool, } +/// Admission is distinct from activation committed and selected by this owner. +pub enum CellGroupRecoveryActivation { + Accepted(CellGroupProposal), + Applied(Box), +} + enum Phase { Ready(Box, Term), Light(LightReady), @@ -193,10 +199,11 @@ pub struct CellGroupOwner { proposals: super::proposal_admission::ProposalAdmission, pub(super) proposal_budget: Arc, prepared_only: bool, + recovery_activation: Option>, } impl CellGroupOwner { - /// Attach only a claimed initial voter with a completely verified selected + /// Attach a durably claimed member with a completely verified selected /// root. The catalog permit is revalidated; serving additionally waits for /// the current term's committed/applied Raft barrier. Managed members and /// ordinary snapshot messages remain closed; step_checkpoint requires the @@ -284,6 +291,7 @@ impl CellGroupOwner { failed: None, proposals: Default::default(), prepared_only: false, + recovery_activation: None, }) } @@ -302,6 +310,7 @@ impl CellGroupOwner { pub(super) fn revoke_execution(&mut self) { self.leader_term.store(0, Ordering::Release); self.proposals.clear(); + self.recovery_activation = None; } fn refresh_execution_fence(&mut self) { @@ -321,6 +330,7 @@ impl CellGroupOwner { }; if self.leader_term.swap(term, Ordering::AcqRel) != term { self.proposals.clear(); + self.recovery_activation = None; } } @@ -350,6 +360,14 @@ impl CellGroupOwner { &self.current } + pub(super) fn require_application_active(&self) -> Result<(), RuntimeError> { + self.live()?; + if !self.current.marker().is_active() { + return Err(RuntimeError::NotReady); + } + Ok(()) + } + pub(super) fn capture_checkpoint( &self, pool: &super::CellGroupCheckpointPool, @@ -366,11 +384,8 @@ impl CellGroupOwner { term: Option, barrier: Option, ) -> Result { - self.live()?; + self.require_application_active()?; let status = self.status()?; - if !self.current.marker().is_active() { - return Err(RuntimeError::NotReady); - } let leader = if matches!(consistency, super::CellReadConsistency::Linearizable) { if self.node.raft.state != StateRole::Leader || Some(status.term) != term { return Err(RuntimeError::NotLeader); @@ -489,25 +504,114 @@ impl CellGroupOwner { leader: (self.node.raft.leader_id != 0).then_some(self.node.raft.leader_id), applied: self.current.marker().position(), committed: LogIndex::new(cache.hard_state.commit), - ready: self.current.marker().is_active() - && self.node.raft.state == StateRole::Leader - && self.node.raft.lead_transferee.is_none() - && self - .node - .raft - .prs() - .conf() - .voters() - .contains(self.node.raft.id) - && self.phase.is_none() - && self.checkpoint.is_none() - && self.apply_bytes < self.options.max_group_apply_bytes - && self.current.marker().position().term.get() == self.node.raft.term - && cache.hard_state.term == self.node.raft.term - && cache.group_manifest.as_ref() == Some(self.current.marker()), + ready: self.current.marker().is_active() && self.protocol_ready(&cache), }) } + fn protocol_ready(&self, cache: &crate::raft::cache::CachedGroup) -> bool { + self.node.raft.state == StateRole::Leader + && self.node.raft.lead_transferee.is_none() + && self + .node + .raft + .prs() + .conf() + .voters() + .contains(self.node.raft.id) + && self.phase.is_none() + && self.checkpoint.is_none() + && self.apply_bytes < self.options.max_group_apply_bytes + && self.current.marker().position().term.get() == self.node.raft.term + && cache.hard_state.term == self.node.raft.term + && cache.group_manifest.as_ref() == Some(self.current.marker()) + } + + /// Admit only the selected recovery's exact activation. Retries in the same + /// term share one bounded proposal; a new leader must apply its fresh-term + /// barrier before it can decide whether an inherited activation survived. + pub fn activate_recovery( + &mut self, + permit: super::CellGroupRecoveryActivationPermit, + ) -> Result { + self.live()?; + permit.check(&self.log, &self.group)?; + if permit.matches(self.current.marker())? { + self.recovery_activation = None; + return Ok(CellGroupRecoveryActivation::Applied(Box::new( + permit.acknowledgement(&self.group, self.current.marker())?, + ))); + } + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if let Some(pending) = &self.recovery_activation { + if pending.0.get() == self.node.raft.term { + return Ok(CellGroupRecoveryActivation::Accepted(pending.1.clone())); + } + } + let cache = self.group.read()?; + if !self.protocol_ready(&cache) + || self.current.marker().position().index.get() != cache.hard_state.commit + || !self.apply.is_empty() + || self.applying.is_some() + { + return Err(RuntimeError::NotReady); + } + drop(cache); + let data = permit.entry()?; + if data.len() > self.options.max_proposal_bytes { + return Err(RuntimeError::Overloaded); + } + let before = self.node.raft.raft_log.last_index(); + let expected = Entry { + index: before + 1, + term: self.node.raft.term, + data: data.clone().into(), + ..Default::default() + }; + let admission = + self.log_admission + .prepare(&self.group, std::slice::from_ref(&expected), None)?; + let result = self.node.propose(Vec::new(), data); + self.finish_log_admission(admission)?; + if let Err(error) = result { + let error = RuntimeError::from(error); + if self.node.raft.raft_log.last_index() != before { + self.failed = Some(error.clone()); + self.revoke_execution(); + } + return Err(error); + } + let observed = (|| { + let entry = self + .node + .raft + .raft_log + .unstable_entries() + .last() + .ok_or_else(|| RuntimeError::Protocol("missing recovery activation".into()))?; + if entry != &expected { + return Err(RuntimeError::Protocol("changed recovery activation".into())); + } + Ok(CellGroupProposal { + group: self.group.identity().scope, + position: entry_position(entry), + digest: entry_digest(entry)?, + }) + })(); + let proposal = match observed { + Ok(proposal) => proposal, + Err(error) => { + self.failed = Some(error.clone()); + self.revoke_execution(); + return Err(error); + } + }; + self.recovery_activation = + Some(Box::new((Term::new(self.node.raft.term), proposal.clone()))); + Ok(CellGroupRecoveryActivation::Accepted(proposal)) + } + pub fn campaign(&mut self) -> Result<(), RuntimeError> { self.live()?; if self.phase.is_some() || self.checkpoint.is_some() { @@ -1566,7 +1670,11 @@ impl CellGroupOwner { next: &PreparedGroupManifest, ) -> Result<(), RuntimeError> { let invalid = || RuntimeError::Invalid("group apply does not match committed entry".into()); + let same_recovery = next.marker().recovery == self.current.marker().recovery; if entry.get_entry_type() == EntryType::EntryConfChangeV2 { + if !self.current.marker().is_active() || !same_recovery { + return Err(invalid()); + } let cache = self.group.read()?; super::membership::transition( entry, @@ -1588,7 +1696,7 @@ impl CellGroupOwner { return Err(invalid()); } if entry.data.is_empty() { - return if next.marker().root() == self.current.marker().root() { + return if next.marker().root() == self.current.marker().root() && same_recovery { Ok(()) } else { Err(invalid()) @@ -1600,12 +1708,47 @@ impl CellGroupOwner { GroupEntryLimits::default(), ) .map_err(|_| invalid())?; + if !matches!(&decoded.command, GroupCommand::Activate { .. }) + && (!self.current.marker().is_active() || !same_recovery) + { + return Err(invalid()); + } let witness = cellar_logic::cell_group::CommitWitness { group: self.current.marker().route().history, position: entry_position(entry), proposal: decoded.binding.proposal, }; match decoded.command { + GroupCommand::Activate { + transition, + publication, + manifest, + } => { + if decoded.binding.proposal.bytes() != transition.bytes() { + return Err(invalid()); + } + let mut expected = self.current.marker().clone(); + let commitment = expected + .recovery_commitment() + .map_err(|_| invalid())? + .ok_or_else(invalid)?; + expected.position = entry_position(entry); + expected + .recovery + .as_deref_mut() + .ok_or_else(invalid)? + .activate( + transition, + publication, + manifest, + commitment, + entry_position(entry), + ) + .map_err(|_| invalid())?; + if next.marker() != &expected { + return Err(invalid()); + } + } GroupCommand::Enroll { ref cell, .. } => { let old = self.current.pin_cell(&cell.cell); let decision = super::decide_enrollment( diff --git a/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs b/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs new file mode 100644 index 000000000..d5f37fe3e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/recovery/genesis.rs @@ -0,0 +1,196 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog-authorized, fresh Raft genesis. The node writer owns the sync and +//! physical root publication; neither an archive owner nor an image is a grant. +use super::super::*; +use crate::raft::{cache::GroupLog, catalog::RecoverySeedPermit, storage::LogError}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, ReplicaIdentity}, +}; +use std::sync::{atomic::AtomicBool, Arc}; + +pub(crate) struct RecoverySeed { + permit: RecoverySeedPermit, + prepared: PreparedCellGroupRecovery, +} + +impl RecoverySeed { + pub(crate) fn new( + permit: RecoverySeedPermit, + prepared: PreparedCellGroupRecovery, + ) -> Result { + if prepared.request().encode() != permit.request().encode() + || prepared.manifest().marker() != permit.genesis() + || prepared.commitment() != permit.commitment() + || !prepared.store().owner.owns(prepared.manifest()) + { + return Err(LogError::WrongIdentity); + } + Ok(Self { permit, prepared }) + } + + pub(crate) fn bootstrap(&self) -> &crate::raft::storage::CellGroupBootstrap { + self.permit.bootstrap() + } + + pub(crate) fn genesis(&self) -> &GroupManifest { + self.permit.genesis() + } + + pub(crate) fn check( + &self, + owner: &Arc<()>, + identity: &ReplicaIdentity, + engine: EngineInstanceId, + ) -> Result<(), LogError> { + self.permit.check(owner, identity, engine) + } + + /// Reverify the complete closure on this actual engine's blocking writer. + /// Hold the source/import proof until the writer finishes even on a lost reply. + pub(crate) fn prepare( + &self, + closed: &Arc, + ) -> Result< + ( + PreparedGroupManifest, + crate::raft::cell_groups::retention::PublicationPin, + ), + LogError, + > { + let mut owner = self + .prepared + .store() + .storage + .try_lock() + .map_err(|error| match error { + std::sync::TryLockError::WouldBlock => LogError::Overloaded, + std::sync::TryLockError::Poisoned(_) => { + LogError::Io("poisoned recovery store".into()) + } + })?; + let apply::Storage { tree, pages, .. } = &mut *owner; + let root = tree + .prepare_recovery_genesis(&self.permit, pages, self.prepared.manifest()) + .map_err(preparation_error)?; + let publication = root + .begin_genesis(&self.permit, closed) + .map_err(preparation_error)?; + // Revoke further archive admission. Entered immutable I/O retains its + // own permits/pins, while the engine now owns this page-store lifetime. + self.prepared.store().close(); + Ok((root, publication)) + } +} + +/// One synced physical seed and independently confirmed catalog receipt. +/// It has no startup, voting or application permission. The directory must +/// select the replacement, then startup and a fresh committed activation follow. +#[derive(Clone)] +pub struct SeededCellGroupRecovery { + request: Arc, + group: GroupLog, + applier: CellGroupApplier, + root: PreparedGroupManifest, + binding: [u8; 32], +} + +impl SeededCellGroupRecovery { + pub(crate) fn new( + request: Arc, + group: GroupLog, + applier: CellGroupApplier, + root: PreparedGroupManifest, + binding: [u8; 32], + ) -> Self { + Self { + request, + group, + applier, + root, + binding, + } + } + + pub fn request(&self) -> &CellGroupRecoveryRequest { + &self.request + } + + pub fn group(&self) -> &GroupLog { + &self.group + } + + pub fn applier(&self) -> &CellGroupApplier { + &self.applier + } + + pub fn manifest(&self) -> &PreparedGroupManifest { + &self.root + } + + pub fn binding(&self) -> [u8; 32] { + self.binding + } +} + +impl CellGroupApplier { + /// Preserve the already opened physical owner during archive-to-engine + /// transfer. Only a checked recovery grant invokes this boundary. + pub(crate) fn recover_seeded( + group: GroupLog, + store: &CellGroupArchiveStore, + genesis: &GroupManifest, + ) -> Result<(Self, PreparedGroupManifest), crate::raft::execution::ExecutionError> { + if group + .group_manifest() + .map_err(crate::raft::execution::storage)? + .as_ref() + != Some(genesis) + || store.route() != genesis.route() + || !genesis.is_recovery_genesis() + { + return Err(crate::raft::execution::ExecutionError::Invalid( + "recovery seed selection", + )); + } + store.close(); + let applier = Self { + group, + storage: store.storage.clone(), + }; + let root = applier.recover_selected()?; + // A same-process retry can encounter cached seals. Confirm bytes and + // persistence independently before a permanent seed receipt is written. + root.images.visit(&mut |_, _, pin, _, _| { + match pin { + CellImagePin::Materialized(pin) => pin + .verify_and_sync() + .map_err(crate::raft::execution::storage)?, + CellImagePin::Extents(pin) => { + for index in 0..pin.manifest().geometry().extent_count() { + let extent = pin.manifest().extent(index).ok_or( + crate::raft::execution::ExecutionError::Invalid("recovery extent"), + )?; + pin.view() + .read_extent(extent) + .map_err(crate::raft::execution::storage)?; + } + } + CellImagePin::Empty => (), + } + Ok::<_, crate::raft::execution::ExecutionError>(()) + })?; + Ok((applier, root)) + } +} + +fn preparation_error(error: ManifestError) -> LogError { + match error { + ManifestError::Limit => LogError::Overloaded, + ManifestError::Conflict => LogError::WrongIdentity, + ManifestError::Invalid(message) => LogError::Invalid(message.into()), + ManifestError::Quarantined => LogError::LostStableState, + ManifestError::Io(error) => LogError::Io(error.to_string()), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/recovery/import.rs b/crates/cellar/durability/raft/cell_groups/recovery/import.rs index 8a3bf1779..cca40687c 100644 --- a/crates/cellar/durability/raft/cell_groups/recovery/import.rs +++ b/crates/cellar/durability/raft/cell_groups/recovery/import.rs @@ -18,6 +18,10 @@ use std::{ type Result = std::result::Result; +/// Complete imported data under fresh histories. Clones retain the same +/// physical owners and admission. This grants no engine or serving authority; +/// only a catalog-authorized genesis installation may select it in a Raft log. +#[derive(Clone)] pub struct PreparedCellGroupRecovery { fence: FencedCellGroupRecovery, store: CellGroupArchiveStore, diff --git a/crates/cellar/durability/raft/cell_groups/recovery/mod.rs b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs index 1f5497a07..eda0041ba 100644 --- a/crates/cellar/durability/raft/cell_groups/recovery/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/recovery/mod.rs @@ -3,9 +3,12 @@ //! Fresh-history restoration and activation. Archive integrity, infrastructure //! fencing, physical genesis and a committed serving decision are distinct proofs. mod admission; +mod genesis; mod import; mod state; pub use admission::{CellGroupRecoveryLimits, CellGroupRecoveryPool}; +pub(crate) use genesis::RecoverySeed; +pub use genesis::SeededCellGroupRecovery; pub use import::PreparedCellGroupRecovery; pub(super) use state::RecoveryState; diff --git a/crates/cellar/durability/raft/cell_groups/recovery/state.rs b/crates/cellar/durability/raft/cell_groups/recovery/state.rs index 7664fbc4f..f585b93fb 100644 --- a/crates/cellar/durability/raft/cell_groups/recovery/state.rs +++ b/crates/cellar/durability/raft/cell_groups/recovery/state.rs @@ -62,6 +62,21 @@ impl RecoveryState { .map(|activation| activation.manifest) } + pub(in crate::raft::cell_groups) fn activation( + &self, + ) -> Option<([u8; 32], [u8; 32], LogPosition)> { + self.activation.as_ref().map(|a| { + ( + a.publication, + a.manifest, + LogPosition { + index: LogIndex::new(a.index), + term: Term::new(a.term), + }, + ) + }) + } + pub(in crate::raft::cell_groups) fn succeeds(&self, previous: &Self) -> bool { self.request == previous.request && self.operation == previous.operation diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs index e5a192bf9..2a986939b 100644 --- a/crates/cellar/durability/raft/cell_groups/retention.rs +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -175,7 +175,33 @@ impl RootRetentions { previous: Option<&GroupManifest>, checkpoint: bool, ) -> Result { - if group.closed.load(Ordering::Acquire) { + self.begin_owned(&group.owner, &group.closed, prepared, previous, checkpoint) + } + + pub(super) fn begin_genesis( + self: &Arc, + permit: &crate::raft::catalog::RecoverySeedPermit, + closed: &Arc, + prepared: &PreparedGroupManifest, + ) -> Result { + if prepared.marker() != permit.genesis() + || !Arc::ptr_eq(&prepared.engine_owner, permit.owner()) + || !prepared.marker().is_recovery_genesis() + { + return Err(ManifestError::Invalid("genesis retention grant")); + } + self.begin_owned(permit.owner(), closed, prepared, None, false) + } + + fn begin_owned( + self: &Arc, + engine: &Arc<()>, + closed: &Arc, + prepared: &PreparedGroupManifest, + previous: Option<&GroupManifest>, + checkpoint: bool, + ) -> Result { + if closed.load(Ordering::Acquire) { return Err(ManifestError::Quarantined); } let mut state = self.state()?; @@ -197,15 +223,15 @@ impl RootRetentions { return Err(ManifestError::Limit); } let selected = state.groups.entry(key).or_insert_with(|| Group { - engine: group.owner.clone(), - closed: group.closed.clone(), + engine: engine.clone(), + closed: closed.clone(), current: None, pending: None, checkpoint: None, expected_checkpoint: None, pending_checkpoint: false, }); - if !Arc::ptr_eq(&selected.engine, &group.owner) + if !Arc::ptr_eq(&selected.engine, engine) || selected.pending.is_some() || selected.current.as_ref().map(|r| &r.marker) != previous { diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 086759c8a..5770782a5 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -51,6 +51,28 @@ impl GroupManifest { self.recovery.as_ref().map(|state| state.request()) } + pub(crate) fn is_recovery_genesis(&self) -> bool { + self.recovery.is_some() + && !self.is_active() + && self.position.index.get() == 1 + && self.position.term.get() == 1 + && self.membership_revision == 1 + } + + pub(crate) fn recovery_commitment(&self) -> Result, ManifestError> { + self.recovery + .as_deref() + .map(|state| match state.activation_manifest() { + Some(commitment) => Ok(commitment), + None => state.seed_commitment(self), + }) + .transpose() + } + + pub(crate) fn recovery_activation(&self) -> Option<([u8; 32], [u8; 32], LogPosition)> { + self.recovery.as_ref().and_then(|state| state.activation()) + } + /// A retained history cannot forget or replace its recovery lineage. Only /// an authenticated empty learner may acquire that lineage via a snapshot. pub(crate) fn preserves_recovery(&self, previous: &Self) -> bool { @@ -252,6 +274,14 @@ impl PreparedGroupManifest { ) -> Result { self.retention.begin(group, self, previous, checkpoint) } + + pub(super) fn begin_genesis( + &self, + permit: &crate::raft::catalog::RecoverySeedPermit, + closed: &Arc, + ) -> Result { + self.retention.begin_genesis(permit, closed, self) + } } impl ManifestTree { @@ -384,6 +414,35 @@ impl ManifestTree { self.prepare_physical(store, marker, owner.identity.clone(), image) } + /// Called only with the catalog's process-local engine grant. No engine + /// selection/retention is recovered before sync. + pub(super) fn prepare_recovery_genesis( + &self, + permit: &crate::raft::catalog::RecoverySeedPermit, + store: &mut ManifestPageStore, + source: &PreparedGroupManifest, + ) -> Result { + if !source.marker.is_recovery_genesis() + || source.marker.route != self.route + || !Arc::ptr_eq(&source.page_owner, &store.owner) + || source.marker != *permit.genesis() + { + return Err(ManifestError::Invalid("recovery genesis physical binding")); + } + self.prepare_physical( + store, + source.marker.clone(), + permit.owner().clone(), + &mut |leaf| { + source + .pin_cell(&leaf.version.history.cell) + .filter(|(selected, _)| *selected == leaf) + .map(|(_, pin)| pin) + .ok_or(ManifestError::Invalid("recovery genesis closure")) + }, + ) + } + fn prepare_physical( &self, store: &mut ManifestPageStore, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index a0c529706..4df4bca93 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -133,6 +133,7 @@ enum Command { Campaign, TransferLeader(u64), Configure(CellGroupMembershipChange), + ActivateRecovery(Box), Execution(cellar_logic::cell_group::CellHistory), Enrollment(cellar_logic::cell_group::CellHistory), EnrollmentState(cellar_logic::cell_group::CellHistory), @@ -162,6 +163,7 @@ enum Response { Unit, Checkpoint(CellGroupSnapshotAdmission), Proposal(CellGroupProposal), + RecoveryActivation(Box), Submission(crate::raft::execution::CellSubmissionOutcome), Execution(Box), Enrollment(Box), @@ -630,6 +632,27 @@ impl CellGroupRuntime { } /// Admit whole-group leadership handoff; observe status for completion. + /// Admit the catalog-bound activation on the multiplexed physical owner. + pub async fn activate_recovery( + &self, + permit: CellGroupRecoveryActivationPermit, + ) -> Result { + let history = permit.history(); + match self + .call( + Some(history), + Command::ActivateRecovery(Box::new(permit)), + 4096, + ) + .await? + { + Response::RecoveryActivation(result) => Ok(*result), + _ => Err(RuntimeError::Protocol( + "recovery activation response".into(), + )), + } + } + pub async fn transfer_leader( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs index 4638a8c6f..a715e6950 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -103,6 +103,9 @@ impl Reads { } ReadKind::Seed(target) => owner.check_seed_target(target)?, } + // A pending recovery is an admission refusal, not a read barrier + // that may occupy the ordinary wait queue until its deadline. + owner.require_application_active()?; self.groups.retain(|_, budget| budget.strong_count() != 0); let budget = self .groups diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index dfdec7361..01a9addcb 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -416,6 +416,10 @@ impl Worker { .owner .reconciliation_authority(&cell) .map(|authority| Response::Reconciliation(Box::new(authority))), + Command::ActivateRecovery(permit) => group + .owner + .activate_recovery(*permit) + .map(|result| Response::RecoveryActivation(Box::new(result))), Command::Configure(change) => { group.owner.configure(change).map(Response::Proposal) } diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index 4b74b63bb..07cf8a25d 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -131,6 +131,12 @@ impl GroupMembership { .any(|binding| binding.node == node.bytes()) } + pub(crate) fn has_disk_binding(&self, disk: cellar_logic::consensus::DiskLineage) -> bool { + self.bindings + .iter() + .any(|binding| binding.disk == disk.bytes()) + } + pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { if self.validate_scope(&replica.scope).is_err() { return false; diff --git a/crates/cellar/durability/raft/node_directory.rs b/crates/cellar/durability/raft/node_directory.rs index 7c73cf81f..052793cdb 100644 --- a/crates/cellar/durability/raft/node_directory.rs +++ b/crates/cellar/durability/raft/node_directory.rs @@ -172,6 +172,63 @@ pub struct NodePublication { } impl NamespaceCatalog { + /// Initial members' immutable pins remain available after host loss. A new + /// node ID cannot disguise reuse of an old disk lineage or engine instance. + pub(crate) async fn verify_recovery_storage_fresh( + &self, + log: &ReplicaLog, + source_nodes: &[NodeId], + ) -> Result<(), CatalogError> { + if source_nodes.len() > 64 { + return Err(CatalogError::Limit); + } + let local = log.limits(); + let instance = log.storage_instance().map_err(CatalogError::Log)?; + for node in source_nodes { + let bytes = self + .inner + .read_versioned_bounded( + &node_key(&self.inner, self.config.cluster, *node, "pin"), + self.inner.limit, + ) + .await? + .ok_or(CatalogError::Corrupt("missing recovery source storage pin"))? + .0; + let pin: Pin = wire::decode(NODE_PIN, &bytes, self.inner.limit)?; + let disk = DiskLineage::new(pin.disk) + .map_err(|_| CatalogError::Corrupt("source storage lineage"))?; + let old_instance = pin.validate(self.config.cluster, *node, disk)?; + if *node == local.node || disk == local.disk || old_instance == instance { + return Err(CatalogError::Conflict("reused recovery storage")); + } + } + Ok(()) + } + + /// Re-admission requires the existing immutable pin; it must not recreate a + /// missing physical claim from a caller's proposed engine identity. + pub(crate) async fn verify_node_storage_identity( + &self, + node: NodeId, + disk: DiskLineage, + engine: EngineInstanceId, + ) -> Result<(), CatalogError> { + let bytes = self + .inner + .read_versioned_bounded( + &node_key(&self.inner, self.config.cluster, node, "pin"), + self.inner.limit, + ) + .await? + .ok_or(CatalogError::LostStableState)? + .0; + let pin: Pin = wire::decode(NODE_PIN, &bytes, self.inner.limit)?; + if pin.validate(self.config.cluster, node, disk)? != engine { + return Err(CatalogError::LostStableState); + } + Ok(()) + } + /// Import reads the permanent physical pin without requiring a live /// advisory process view. Presence of an arbitrary fleet pin is insufficient. pub(crate) async fn verify_fleet_storage_pin( diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index c111f317d..391f8115d 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -122,6 +122,32 @@ impl MappingRecord { self.history.decode() } + /// Recovery persists the actual fresh genesis, including its complete root + /// commitment. Ordinary creation cannot later substitute an empty genesis. + pub(crate) fn new_recovered_cells( + key: u64, + bootstrap: &CellGroupBootstrap, + conf: &ConfState, + genesis: &GroupManifest, + ) -> Result { + genesis.check().map_err(corrupt)?; + if !genesis.is_recovery_genesis() || genesis.route() != &bootstrap.route { + return Err(LogError::WrongIdentity); + } + let mut mapping = Self::new_cells(key, bootstrap, conf)?; + mapping.application = MappingApplication::CellGroup(Box::new(genesis.clone())); + Ok(mapping) + } + + pub(crate) fn recovery_genesis(&self) -> Option<&GroupManifest> { + match &self.application { + MappingApplication::CellGroup(marker) if marker.recovery_request().is_some() => { + Some(marker) + } + _ => None, + } + } + pub fn is_cells(&self) -> bool { matches!(&self.application, MappingApplication::CellGroup(_)) } @@ -131,6 +157,9 @@ impl MappingRecord { return Err(corrupt("unsupported group marker")); } let history = self.history()?; + if cache.recovery_genesis.as_deref() != self.recovery_genesis() { + return Err(LogError::WrongIdentity); + } if let Some(marker) = &cache.group_manifest { if !marker.matches_group(&history, self.birth_digest) { return Err(LogError::WrongIdentity); @@ -142,8 +171,7 @@ impl MappingRecord { } MappingApplication::CellGroup(genesis) => { let expected = GroupManifest::genesis(genesis.route().clone()).map_err(corrupt)?; - if genesis.as_ref() != &expected - || !genesis.matches_group(&history, self.birth_digest) + if !genesis.matches_group(&history, self.birth_digest) || cache .group_manifest .as_ref() @@ -153,6 +181,22 @@ impl MappingRecord { { return Err(LogError::WrongIdentity); } + if genesis.recovery_request().is_some() { + let current = cache + .group_manifest + .as_ref() + .ok_or(LogError::WrongIdentity)?; + if !genesis.is_recovery_genesis() + || !current.preserves_recovery(genesis) + || current.recovery_commitment().map_err(corrupt)? + != genesis.recovery_commitment().map_err(corrupt)? + || (!current.is_active() && current.root() != genesis.root()) + { + return Err(LogError::WrongIdentity); + } + } else if genesis.as_ref() != &expected { + return Err(LogError::WrongIdentity); + } Ok(()) } } @@ -327,6 +371,11 @@ impl GroupRecord { snapshot, entries: Default::default(), group_manifest, + recovery_genesis: self + .mapping + .recovery_genesis() + .cloned() + .map(std::sync::Arc::new), group_seed, manifest_pins: None, checkpoint_manifest, diff --git a/crates/cellar/durability/raft/snapshot_seed.rs b/crates/cellar/durability/raft/snapshot_seed.rs index b410c00aa..b39e270fa 100644 --- a/crates/cellar/durability/raft/snapshot_seed.rs +++ b/crates/cellar/durability/raft/snapshot_seed.rs @@ -295,6 +295,7 @@ impl PreparedSeed { snapshot, entries: std::collections::BTreeMap::new(), group_manifest: None, + recovery_genesis: None, manifest_pins: None, checkpoint_manifest: None, group_seed: None, diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 2a4c6f2db..42debf9b9 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -20,7 +20,9 @@ use super::records::{ }; use ::raft::prelude::{ConfState, Entry}; use cellar_logic::cell_group::{GroupHistory, GroupRoute}; -use cellar_logic::consensus::{GroupScope, MemberId, OperationId, ReplicaIdentity}; +use cellar_logic::consensus::{ + EngineInstanceId, GroupScope, MemberId, OperationId, ReplicaIdentity, +}; use cellar_ltx::{FileSystem, HostFileLock}; use protobuf::Message; use raft_engine::{Command, Engine, LogBatch, MessageExt, ReadableSize, RecoveryMode}; @@ -256,6 +258,10 @@ enum Work { Option>, Reply>, ), + RecoverCells( + Box, + Reply>, + ), Seeded(Box, Reply), Persist(OperationId, Vec, Reply), PersistCells( @@ -665,6 +671,22 @@ impl ReplicaLog { .await } + pub(crate) async fn install_recovery_seed( + &self, + seed: super::cell_groups::RecoverySeed, + ) -> Result, LogError> { + let identity = ReplicaIdentity { + scope: seed.bootstrap().route.history, + member: seed.bootstrap().member, + node: self.limits().node, + disk: self.limits().disk, + process: self.limits().process, + }; + seed.check(&self.owner(), &identity, self.storage_instance()?)?; + self.request(96 * 1024, |reply| Work::RecoverCells(Box::new(seed), reply)) + .await + } + /// Read occupied identities and locally durable completed membership under /// the engine catalog lock and short per-group cache reads. This catalog is /// append-only for an engine instance; retired bindings remain. @@ -1197,9 +1219,10 @@ impl Worker { fn scoped_handle( &self, mapping: &MappingRecord, - cache: CachedGroup, + mut cache: CachedGroup, scope: S, ) -> Result, LogError> { + cache.recovery_genesis = mapping.recovery_genesis().cloned().map(Arc::new); let log_retention = if mapping.is_cells() { Some( self.log_retention @@ -1369,7 +1392,7 @@ impl Worker { seed: Option<&super::snapshot::PreparedSeed>, ) -> Result { let mapping = MappingRecord::new(self.root.next_key, &bootstrap, &conf)?; - self.insert_mapping(mapping, cache, seed)? + self.insert_mapping(mapping, cache, seed, None)? .as_cell() .cloned() } @@ -1414,7 +1437,54 @@ impl Worker { super::cell_groups::GroupManifest::genesis(bootstrap.route) .map_err(|error| LogError::Invalid(error.to_string()))?, ); - self.insert_mapping(mapping, cache, None)? + self.insert_mapping(mapping, cache, None, None)? + .as_cells() + .cloned() + } + + fn recover_cells( + &mut self, + seed: &super::cell_groups::RecoverySeed, + ) -> Result, LogError> { + let bootstrap = seed.bootstrap(); + seed.check( + &self.owner, + &ReplicaIdentity { + scope: bootstrap.route.history, + member: bootstrap.member, + node: self.options.node, + disk: self.options.disk, + process: self.options.process, + }, + EngineInstanceId::new(self.root.identity.storage_instance).map_err(records::corrupt)?, + )?; + let conf = bootstrap.configuration()?; + if let Some(group) = self + .groups + .read() + .map_err(|_| LogError::Closed)? + .get(&bootstrap.route.history) + { + let group = group.as_cells()?; + let expected = + MappingRecord::new_recovered_cells(group.key, bootstrap, &conf, seed.genesis())?; + if self.mappings.get(&group.key) != Some(&expected) + || group.group_manifest()?.as_ref() != Some(seed.genesis()) + { + return Err(LogError::WrongIdentity); + } + return Ok(group.clone()); + } + let mapping = MappingRecord::new_recovered_cells( + self.root.next_key, + bootstrap, + &conf, + seed.genesis(), + )?; + let mut cache = genesis(conf); + cache.group_manifest = Some(seed.genesis().clone()); + cache.recovery_genesis = Some(Arc::new(seed.genesis().clone())); + self.insert_mapping(mapping, cache, None, Some(seed))? .as_cells() .cloned() } @@ -1422,8 +1492,9 @@ impl Worker { fn insert_mapping( &mut self, mapping: MappingRecord, - cache: CachedGroup, + mut cache: CachedGroup, seed: Option<&super::snapshot::PreparedSeed>, + recovery: Option<&super::cell_groups::RecoverySeed>, ) -> Result { if self.mappings.len() >= self.options.max_groups { return Err(LogError::Overloaded); @@ -1453,6 +1524,16 @@ impl Worker { { return Err(LogError::Overloaded); } + if seed.is_some() && recovery.is_some() { + return Err(LogError::WrongIdentity); + } + let mut publication = if let Some(recovery) = recovery { + let (root, publication) = recovery.prepare(&self.closed)?; + cache.manifest_pins = Some(root.images.clone()); + Some(publication) + } else { + None + }; let mut batch = LogBatch::default(); batch .put(0, mapping.catalog_key()?, records::encode(&mapping)?) @@ -1470,6 +1551,9 @@ impl Worker { } let previous_key = self.root.next_key; self.root.next_key = next_key; + if let Some(publication) = &mut publication { + publication.arm(); + } if let Err(error) = self.fence(&mut batch) { self.root.next_key = previous_key; return Err(error); @@ -1477,6 +1561,11 @@ impl Worker { if let Some(seed) = seed { seed.retain().map_err(|e| LogError::Io(e.to_string()))?; } + if let Some(publication) = publication { + publication + .commit() + .map_err(|error| LogError::Io(error.to_string()))?; + } let handle = self.handle(&mapping, cache)?; self.groups .write() @@ -1799,6 +1888,15 @@ impl Worker { let _ = reply.send(result); fatal } + Work::RecoverCells(seed, reply) => { + let result = self.recover_cells(&seed); + let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); + if fatal.is_some() { + self.closed.store(true, Ordering::Release); + } + let _ = reply.send(result); + fatal + } Work::PersistCells(operation, writes, reply) => { let timer = self.performance.timer(Measured::EnginePersist); let result = self.persist(operation, writes); diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index 8ee439b79..c4b0df2c2 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -37,6 +37,56 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_publication_requires_seeded_quorum_activation() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::activate_quorum( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_publication_requires_seeded_quorum_activation() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::activate_quorum( + rooted, + rustfs::store("recovery-activation", rooted)?, + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn group_recovery_genesis_survives_lost_reply_and_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::seed_engine( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_group_recovery_genesis_survives_lost_reply_and_restart() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::seed_engine( + rooted, + rustfs::store("recovery-genesis", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_recovery_import_rebinds_complete_cut() -> Result<()> { for rooted in [false, true] { diff --git a/crates/cellar/tests/support/cell_group_archive_store.rs b/crates/cellar/tests/support/cell_group_archive_store.rs index 18d36ade9..df3c4f4b0 100644 --- a/crates/cellar/tests/support/cell_group_archive_store.rs +++ b/crates/cellar/tests/support/cell_group_archive_store.rs @@ -22,6 +22,7 @@ pub struct Store { pub puts: AtomicUsize, pub gets: AtomicUsize, pub lose_put_response: AtomicBool, + pub lose_directory_head_and_reads: AtomicBool, pub deletes: AtomicUsize, pub lose_delete_response: AtomicBool, held: Mutex)>>, @@ -44,6 +45,7 @@ impl Store { puts: AtomicUsize::new(0), gets: AtomicUsize::new(0), lose_put_response: AtomicBool::new(false), + lose_directory_head_and_reads: AtomicBool::new(false), deletes: AtomicUsize::new(0), lose_delete_response: AtomicBool::new(false), held: Mutex::new(None), @@ -162,6 +164,17 @@ impl ObjectStore for Store { source: "injected".into(), }); } + if path.as_ref().ends_with("/cell-groups/head") + && self + .lose_directory_head_and_reads + .swap(false, Ordering::AcqRel) + { + self.outage.store(true, Ordering::Release); + return Err(Error::Generic { + store: "lost directory head response", + source: "injected after real CAS".into(), + }); + } if self.lose_put_response.swap(false, Ordering::AcqRel) { return Err(Error::Generic { store: "lost PUT response", diff --git a/crates/cellar/tests/support/cell_group_checkpoint.rs b/crates/cellar/tests/support/cell_group_checkpoint.rs index fd78a9ad2..ca9633e50 100644 --- a/crates/cellar/tests/support/cell_group_checkpoint.rs +++ b/crates/cellar/tests/support/cell_group_checkpoint.rs @@ -137,7 +137,7 @@ pub(super) async fn enroll_images( Ok(expected) } -fn extent_commitment( +pub(super) fn extent_commitment( route: &GroupRoute, cell: &CellHistory, state: ImageState, diff --git a/crates/cellar/tests/support/cell_group_disaster_activation.rs b/crates/cellar/tests/support/cell_group_disaster_activation.rs new file mode 100644 index 000000000..4a3e48c2d --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_activation.rs @@ -0,0 +1,748 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real fresh voters, scheduler, physical apply and topology CAS after all old +//! voter disks are gone. The external fence is still a contract fixture. +use super::*; + +struct FreshVoter { + log: ReplicaLog, + applier: CellGroupApplier, + member: u64, + base: std::path::PathBuf, + options: ReplicaLogOptions, + runtime: CellGroupRuntime, + events: CellGroupRuntimeEvents, + sql: CellGroupSqlPool, +} + +async fn pump( + voters: &mut [FreshVoter], + history: GroupHistory, + messages: &mut VecDeque, +) -> Result<()> { + for (ordinal, voter) in voters.iter_mut().enumerate() { + for _ in 0..64 { + let Ok(event) = voter.events.try_recv() else { + break; + }; + match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { + CellGroupEffect::Message(message) => messages.push_back(message.clone()), + CellGroupEffect::Apply(_) => { + let work = event.retain_apply().context("recovered apply")?; + Box::pin(voter.sql.prepare_apply(&voter.applier, work)) + .await? + .complete(&voter.runtime) + .await?; + } + CellGroupEffect::Applied { .. } | CellGroupEffect::ReadIndex { .. } => (), + _ => anyhow::bail!("unexpected recovered event on {ordinal}"), + } + } + } + for _ in 0..messages.len() { + let message = messages.pop_front().context("queued fresh message")?; + let target = voters + .iter() + .position(|v| v.member == message.to) + .context("fresh target member")?; + match voters[target].runtime.step(history, message.clone()).await { + Ok(()) => (), + Err(RuntimeError::NotReady) => messages.push_back(message), + Err(error) => return Err(error.into()), + } + } + tokio::task::yield_now().await; + Ok(()) +} + +async fn applied(voters: &mut [FreshVoter], history: GroupHistory, index: LogIndex) -> Result<()> { + let mut messages = VecDeque::new(); + tokio::time::timeout(Duration::from_secs(15), async { + loop { + pump(voters, history, &mut messages).await?; + let mut done = true; + for voter in voters.iter() { + done &= voter.runtime.status(history).await?.applied.index >= index; + } + if done { + return Ok::<_, anyhow::Error>(()); + } + } + }) + .await? +} + +// Each cold control phase owns its future. A single inline state machine for +// the entire disk-loss drill otherwise consumes the default test-thread stack +// before nested archive verification can run. +type SeededVoters = ( + Vec, + Vec, + Vec<(std::path::PathBuf, ReplicaLogOptions)>, +); + +struct FreshQuorum { + voters: Vec, + route: GroupRoute, + cell: CellHistory, + authority: Arc, + publication: DirectoryPublication, +} + +pub(crate) async fn activate_quorum(rooted: bool, store: Arc) -> Result<()> { + let mut s = Box::new(Scenario::new(rooted, store).await?); + let seeds = seed_voters(&mut s, rooted).await?; + let mut quorum = publish_voters(&mut s, seeds).await?; + activate_voters(&s, &mut quorum).await?; + mutate_voters(&mut s, &mut quorum, rooted).await?; + restart_voters(&s, &mut quorum, rooted).await?; + for voter in quorum.voters { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + } + s.cluster.close().await +} + +fn seed_voters( + s: &mut Scenario, + rooted: bool, +) -> futures_util::future::BoxFuture<'_, Result> { + Box::pin(async move { + seed::erase_voters(s, rooted).await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let route = s.request.replacement().route().clone(); + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let mut seeds = Vec::new(); + let mut logs = Vec::new(); + let mut reopen = Vec::new(); + for (ordinal, member) in s.request.replacement().voters().iter().enumerate() { + assert!(s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await? + .is_err()); + let base = s.cluster._root.path().join(format!("fresh-{ordinal}")); + let destination = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + base.join("data"), + route.clone(), + ) + .await?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof.clone(), + destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let options = ReplicaLogOptions::new( + base.join("log"), + member.node, + DiskLineage::new([210 + ordinal as u8; 16])?, + ProcessGeneration::new([220; 16])?, + LogOpenMode::Create, + ); + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + reopen.push((base, options)); + let seed = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared, &log, &s.authority) + .await?; + assert!(s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + Arc::new(Authority::default()) + )) + .await? + .is_err()); + if let Some(first) = seeds.first() { + let first: &SeededCellGroupRecovery = first; + assert_eq!(seed.manifest().marker(), first.manifest().marker()); + assert_eq!(seed.binding(), first.binding()); + } + seeds.push(seed); + logs.push(log); + } + assert_eq!(pool.usage().operations, 0); + drop(proof); + assert_eq!(s.receivers.usage().sources, 0); + Ok((seeds, logs, reopen)) + }) +} + +fn publish_voters( + s: &mut Scenario, + seeded: SeededVoters, +) -> futures_util::future::BoxFuture<'_, Result> { + Box::pin(async move { + let (seeds, logs, reopen) = seeded; + let route = s.request.replacement().route().clone(); + s.authority.reject.store(true, Ordering::Release); + assert!(s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await? + .is_err()); + s.authority.reject.store(false, Ordering::Release); + // Lose the actual head CAS response and all immediate reads. The replacement + // is selected; retry must recover that exact publication without rebasing. + s.remote + .lose_directory_head_and_reads + .store(true, Ordering::Release); + assert!(matches!( + s.cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await?, + Err(CatalogError::Unconfirmed) + )); + s.remote.outage.store(false, Ordering::Release); + let publication = s + .cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority), + ) + .await??; + assert_eq!( + publication.journal_digest(), + s.cluster + .domain + .run( + s.cluster + .directory + .publish_group_recovery(&s.request, &s.authority) + ) + .await?? + .journal_digest() + ); + let birth = s + .cluster + .domain + .run( + s.cluster + .directory + .materialize_group_birth(s.request.replacement()), + ) + .await??; + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.bootstrap_group_member(&birth, &logs[0])) + .await?, + Err(CatalogError::Fenced) + )); + // The same head still fences ordinary topology while activation is pending. + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.publish(&s.competing)) + .await?, + Err(CatalogError::Fenced) + )); + let cell = seeds[0] + .manifest() + .pin_cell(&CellId::new("alpha")?) + .context("restored alpha")? + .0 + .version + .history + .clone(); + let authority = Arc::new(std::mem::take(&mut s.authority)); + let binding_key = s + .remote + .list(None) + .try_collect::>() + .await? + .into_iter() + .find(|meta| meta.location.as_ref().contains("/recovery-bindings/")) + .context("recovery binding key")? + .location; + let binding_bytes = s.remote.get(&binding_key).await?.bytes().await?; + s.remote.delete(&binding_key).await?; + assert!(s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &logs[0], + authority.clone() + )) + .await? + .is_err()); + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.bootstrap_group_member(&birth, &logs[0])) + .await?, + Err(CatalogError::Fenced) + )); + s.remote.put(&binding_key, binding_bytes.into()).await?; + let mut voters = Vec::new(); + for ((seed, log), (base, options)) in seeds.into_iter().zip(logs).zip(reopen) { + let permit = s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + authority.clone(), + )) + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([231; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, seed.manifest().clone()).await?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 128 * 1024 * 1024, + group_bytes: 64 * 1024 * 1024, + }, + )?; + voters.push(FreshVoter { + member: seed.group().identity().member.get(), + applier: seed.applier().clone(), + base, + options, + log, + runtime, + events, + sql, + }); + } + Ok(FreshQuorum { + voters, + route, + cell, + authority, + publication, + }) + }) +} + +fn activate_voters<'a>( + s: &'a Scenario, + quorum: &'a mut FreshQuorum, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + publication, + } = quorum; + let history = route.history; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + assert!(matches!( + voters[0].runtime.activate_recovery(grant).await, + Err(RuntimeError::NotLeader) + )); + voters[0].runtime.campaign(history).await?; + applied(voters, history, LogIndex::new(2)).await?; + let status = voters[0].runtime.status(history).await?; + assert_eq!(status.leader, Some(11)); + assert!(status.term.get() > 1 && !status.ready); + assert!(matches!( + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await, + Err(RuntimeError::NotReady) + )); + let deadline = s + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let pending_read = voters[0] + .runtime + .read_authority(history, cell.clone(), CellReadConsistency::Stale, deadline) + .await; + assert!( + matches!(&pending_read, Err(RuntimeError::NotReady)), + "pending recovery read refusal: {:?}", + pending_read.as_ref().err() + ); + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + assert!(matches!( + voters[1].runtime.activate_recovery(grant).await, + Err(RuntimeError::StaleCompletion) + )); + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Accepted(proposal) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation skipped replication") + }; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Accepted(retry) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation committed without network") + }; + assert_eq!(proposal.position, retry.position); + assert_eq!(proposal.digest, retry.digest); + assert!(!voters[0].runtime.status(history).await?.ready); + applied(voters, history, proposal.position.index).await?; + assert!(voters[0].runtime.status(history).await?.ready); + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await?; + let grant = s + .cluster + .domain + .run(s.cluster.directory.prepare_group_recovery_activation( + &s.request, + &voters[0].log, + authority.as_ref(), + )) + .await??; + let CellGroupRecoveryActivation::Applied(acknowledgement) = + voters[0].runtime.activate_recovery(grant).await? + else { + anyhow::bail!("activation not selected") + }; + s.remote + .lose_directory_head_and_reads + .store(true, Ordering::Release); + assert!(matches!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?, + Err(CatalogError::Unconfirmed) + )); + s.remote.outage.store(false, Ordering::Release); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?? + .journal_digest(), + publication.journal_digest() + ); + assert_eq!( + s.cluster + .domain + .run(s.cluster.directory.complete_group_recovery( + &s.request, + &acknowledgement, + authority.as_ref() + )) + .await?? + .journal_digest(), + publication.journal_digest() + ); + drop(acknowledgement); + Ok(()) + }) +} + +fn mutate_voters<'a>( + s: &'a mut Scenario, + quorum: &'a mut FreshQuorum, + rooted: bool, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + .. + } = quorum; + let history = route.history; + // Exercise a real cell-addressed LTX mutation on the restored group, through + // the bounded production physical applier, in both image representations. + let before = &s.expected["alpha"]; + let after = images::image(Some(before), "UPDATE value SET n=n+1")?; + let cut = images::Cut::new(before, &after, 1, true, false); + let bytes = request(route, cell.clone(), 2, &cut)?; + let mut entry = CellGroupEntry::decode(&bytes, route, GroupEntryLimits::default())?; + if rooted { + let GroupCommand::Mutation(mutation) = &mut entry.command else { + anyhow::bail!("mutation") + }; + mutation.base.commitment = + checkpoint_fixture::extent_commitment(route, cell, cut.base, before)?; + mutation.result.commitment = + checkpoint_fixture::extent_commitment(route, cell, cut.result, &after)?; + } + let proposal = voters[0] + .runtime + .propose(history, entry.encode(route, GroupEntryLimits::default())?) + .await?; + applied(voters, history, proposal.position.index).await?; + s.expected.insert("alpha".into(), after); + for voter in voters.iter() { + let applier = voter.applier.clone(); + let selected = s + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + assert!(selected.marker().is_active()); + assert_eq!( + selected + .pin_cell(&cell.cell) + .context("mutated cell")? + .0 + .version + .sequence + .get(), + 2 + ); + tail_fixture::check_images(&s.cluster, &selected, &s.expected).await?; + s.cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &voter.log, + authority.clone(), + )) + .await??; + } + Ok(()) + }) +} + +fn restart_voters<'a>( + s: &'a Scenario, + quorum: &'a mut FreshQuorum, + rooted: bool, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let FreshQuorum { + voters, + route, + cell, + authority, + .. + } = quorum; + let history = route.history; + let mut reopen = Vec::new(); + for voter in voters.drain(..) { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + let mut options = voter.options.clone(); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([240; 16])?; + reopen.push((voter.base.clone(), options)); + drop(voter); + } + for (base, options) in reopen { + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + let group = log.cell_group(&history)?; + let target = base.join("data"); + let (applier, root) = s + .cluster + .domain + .blocking(move || reopen_applier(group, target, rooted))? + .await??; + assert!(root.marker().is_active()); + tail_fixture::check_images(&s.cluster, &root, &s.expected).await?; + let permit = s + .cluster + .domain + .run(s.cluster.directory.bootstrap_recovered_group_member( + &s.request, + &log, + authority.clone(), + )) + .await??; + let member = permit.group().identity().member.get(); + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: None, + election_seed: Some([241; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, root).await?; + assert!(!runtime.status(history).await?.ready); + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 128 * 1024 * 1024, + group_bytes: 64 * 1024 * 1024, + }, + )?; + voters.push(FreshVoter { + log, + applier, + member, + runtime, + events, + sql, + base, + options, + }); + } + let old_term = voters[0].runtime.status(history).await?.term; + let before = voters[0].runtime.status(history).await?.applied.index; + voters[0].runtime.campaign(history).await?; + applied(voters, history, LogIndex::new(before.get() + 1)).await?; + assert!(voters[0].runtime.status(history).await?.ready); + assert!(voters[0].runtime.status(history).await?.term > old_term); + voters[0] + .runtime + .execution_authority(history, cell.clone()) + .await?; + Ok(()) + }) +} + +fn reopen_applier( + group: cellar::application::consensus::verification::GroupLog, + base: std::path::PathBuf, + rooted: bool, +) -> Result<(CellGroupApplier, PreparedGroupManifest)> { + let filesystem: Arc = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: cellar_ltx::internal::NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open(base.join("images"), filesystem, Default::default())?, + services, + } + }; + Ok(CellGroupApplier::recover( + group, + pages, + images, + ManifestLimits::default(), + GroupEntryLimits::default(), + 100, + )?) +} diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs index 1a48adce8..ff9ac155f 100644 --- a/crates/cellar/tests/support/cell_group_disaster_fence.rs +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -5,6 +5,13 @@ use super::*; use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +#[path = "cell_group_disaster_activation.rs"] +mod activation; +#[path = "cell_group_disaster_seed.rs"] +mod seed; +pub(super) use activation::activate_quorum; +pub(super) use seed::seed_engine; + #[derive(Default)] struct Authority { fences: AtomicUsize, @@ -157,15 +164,27 @@ impl Scenario { &target, hold, )?; + let competing_birth = cluster.directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + group: GroupId::new([80; 16])?, + incarnation: IncarnationId::new([81; 16])?, + ..route.history + }, + range: route.range.clone(), + epoch: RouteEpoch::new(2)?, + creation: ProposalId::new([82; 16])?, + voters: target.voters().to_vec(), + learners: vec![], + })?; let competing = cluster.directory.prepare_replacement( &view, ProposalId::new([75; 16])?, std::slice::from_ref(route), - std::slice::from_ref(target.route()), + std::slice::from_ref(competing_birth.route()), )?; cluster .domain - .run(cluster.directory.reserve_group_birth(&target)) + .run(cluster.directory.reserve_group_birth(&competing_birth)) .await??; let saved: Vec = persist_archive_intent(&cluster, &request.encode().to_vec()).await?; assert_eq!(saved, request.encode()); diff --git a/crates/cellar/tests/support/cell_group_disaster_seed.rs b/crates/cellar/tests/support/cell_group_disaster_seed.rs new file mode 100644 index 000000000..4c59284d0 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_seed.rs @@ -0,0 +1,294 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual fresh engine seeding after removing every old voter directory. The +//! authority remains a contract fixture; no Docker launch fence is qualified. +use super::*; + +pub(super) async fn erase_voters(s: &mut Scenario, rooted: bool) -> Result<()> { + for mut voter in std::mem::take(&mut s.cluster.voters) { + drop(voter.owner.take()); + drop(voter.held.take()); + voter.log.close().await?; + } + s.cluster.network.clear(); + let directory = s.cluster._root.path().to_owned(); + s.cluster + .domain + .blocking(move || -> Result<()> { + for n in 1..=3 { + let node = directory.join(format!("node-{n}")); + assert!(node.join("log").exists()); + std::fs::remove_dir_all(&node)?; + assert!(!node.exists()); + if rooted { + let extents = directory.join(format!("extents-{}", n - 1)); + assert!(extents.exists()); + std::fs::remove_dir_all(extents)?; + } + } + Ok(()) + })? + .await??; + // Replace and erase the earlier publication verifier cache as well. The + // fence/recovery now has neither a voter disk nor that warm recovery image. + let cold = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster._root.path().join("verify-after-voter-loss"), + s.cluster.birth.route().clone(), + ) + .await?; + drop(std::mem::replace(&mut s.destination, cold)); + let old_cache = s.cluster._root.path().join("archive-data"); + s.cluster + .domain + .blocking(move || std::fs::remove_dir_all(old_cache))? + .await??; + Ok(()) +} + +pub(crate) async fn seed_engine( + rooted: bool, + store: Arc, +) -> Result<()> { + let mut s = Box::new(Scenario::new(rooted, store).await?); + let old_disk = s.cluster.voters[0] + .log + .cell_group(&s.cluster.birth.route().history)? + .identity() + .disk; + erase_voters(&mut s, rooted).await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let route = s.request.replacement().route().clone(); + let node = s.request.replacement().voters()[0].node; + let member = s.request.replacement().voters()[0].member; + let physical = s.cluster._root.path().join("recovered-images"); + let destination = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + physical.clone(), + route.clone(), + ) + .await?; + let pool = CellGroupRecoveryPool::new(Default::default())?; + let scratch = ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof, + destination.store.clone(), + scratch.clone(), + s.cluster._root.path().into(), + ) + .await?; + let marker = prepared.manifest().marker().clone(); + let wrong = s + .cluster + .domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + s.cluster._root.path().join("wrong-recovery-disk/log"), + node, + old_disk, + ProcessGeneration::new([199; 16])?, + LogOpenMode::Create, + ), + Arc::new(DirectFileSystem), + )) + .await??; + assert!(matches!( + s.cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &wrong, &s.authority) + .await, + Err(CellGroupRecoveryError::Catalog(CatalogError::Conflict( + "reused recovery storage" + ))) + )); + wrong.close().await?; + drop(wrong); + let gate = fault::Gate::new(None); + let mut options = ReplicaLogOptions::new( + s.cluster._root.path().join("recovered-node/log"), + node, + DiskLineage::new([200; 16])?, + ProcessGeneration::new([201; 16])?, + LogOpenMode::Create, + ); + options.purge_interval_ms = None; + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(fault::FaultFileSystem { gate: gate.clone() }), + )) + .await??; + let release = gate.arm(fault::HOLD_SYNC); + let mut work = Box::pin(s.cluster.directory.seed_group_recovery( + &s.cluster.domain, + prepared.clone(), + &log, + &s.authority, + )); + let entered = Box::pin(tokio::time::timeout( + Duration::from_secs(15), + gate.entered.notified(), + )); + match futures_util::future::select(entered, work.as_mut()).await { + futures_util::future::Either::Left((result, _)) => result?, + futures_util::future::Either::Right(_) => anyhow::bail!("genesis skipped engine sync"), + } + assert!(matches!( + log.cell_group(&route.history), + Err(LogError::NotFound) + )); + drop(work); + drop(release); + let group = tokio::time::timeout(Duration::from_secs(15), async { + loop { + match log.cell_group(&route.history) { + Ok(group) => break Ok::<_, LogError>(group), + Err(LogError::NotFound) => tokio::task::yield_now().await, + Err(error) => break Err(error), + } + } + }) + .await??; + assert_eq!(group.group_manifest()?.as_ref(), Some(&marker)); + assert!(!group.group_manifest()?.unwrap().is_active()); + assert_eq!(group.identity().member, member); + let key = group.numeric_key(); + assert!(!s + .remote + .list(None) + .try_collect::>() + .await? + .iter() + .any(|meta| meta.location.as_ref().contains(&format!( + "recovery/{}/seeds/", + hex::encode(s.request.digest()) + )) && meta.location.as_ref().ends_with("/complete"))); + let seeded = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &log, &s.authority) + .await?; + assert_eq!(seeded.group().numeric_key(), key); + assert_eq!(seeded.manifest().marker(), &marker); + assert_ne!(seeded.binding(), [0; 32]); + tail_fixture::check_images(&s.cluster, seeded.manifest(), &s.expected).await?; + let ordinary = CellGroupBootstrap { + route: route.clone(), + member, + voters: s + .request + .replacement() + .voters() + .iter() + .map(|m| m.member) + .collect(), + learners: vec![], + }; + assert!(matches!( + log.create_cell_group(ordinary.clone()).await, + Err(LogError::WrongIdentity) + )); + drop(prepared); + assert_eq!(pool.usage().operations, 0); + // The engine owns the complete physical closure after import admission ends. + let applier = seeded.applier().clone(); + let selected = s + .cluster + .domain + .blocking(move || applier.recover_selected())? + .await??; + tail_fixture::check_images(&s.cluster, &selected, &s.expected).await?; + drop(selected); + drop(group); + drop(seeded); + drop(destination); + log.close().await?; + drop(log); + options.mode = LogOpenMode::Resume; + options.process = ProcessGeneration::new([202; 16])?; + let log = s + .cluster + .domain + .run(ReplicaLog::open( + options.clone(), + Arc::new(DirectFileSystem), + )) + .await??; + let target = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + physical, + route.clone(), + ) + .await?; + let proof = s.fence(&s.cluster.directory, &s.request).await?; + let prepared = pool + .prepare( + &s.cluster.domain, + proof, + target.store.clone(), + scratch, + s.cluster._root.path().into(), + ) + .await?; + let seeded = s + .cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &log, &s.authority) + .await?; + assert_eq!(seeded.group().numeric_key(), key); + assert_eq!(seeded.group().identity().process, options.process); + assert_eq!(seeded.manifest().marker(), &marker); + tail_fixture::check_images(&s.cluster, seeded.manifest(), &s.expected).await?; + assert!(matches!( + log.create_cell_group(ordinary).await, + Err(LogError::WrongIdentity) + )); + drop(seeded); + log.close().await?; + drop(log); + let path = options.directory.clone(); + s.cluster + .domain + .blocking(move || std::fs::remove_dir_all(path))? + .await??; + options.mode = LogOpenMode::Create; + options.process = ProcessGeneration::new([203; 16])?; + let blank = s + .cluster + .domain + .run(ReplicaLog::open(options, Arc::new(DirectFileSystem))) + .await??; + assert!(matches!( + s.cluster + .directory + .seed_group_recovery(&s.cluster.domain, prepared.clone(), &blank, &s.authority) + .await, + Err(CellGroupRecoveryError::Catalog( + CatalogError::LostStableState + )) + )); + assert!(matches!( + blank.cell_group(&route.history), + Err(LogError::NotFound) + )); + blank.close().await?; + drop(prepared); + drop(target); + assert_eq!(pool.usage().operations, 0); + assert_eq!(s.receivers.usage().sources, 0); + let s = *s; + s.cluster.close().await +} diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 6249b1cbb..95bd9abef 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -2,7 +2,7 @@ "format": 1, "source_revision": "7052d218d60181d7b0bb5a4c12686e80e294a61c", "runtime_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending.", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,9 +19,9 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "1e7517328428b06269fa62e8492219e4d820c2a5a1e808c5e33c5e9653bf6354", - "line": 460, - "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" + "sha256": "944ba3c1f181d1af42976d1e19e2c7c0f0d2e2981b121b145acfad9e4fd4e461", + "line": 466, + "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n /// Derived from the immutable mapping; never supplied by a write/snapshot.\n pub recovery_genesis: Option>,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { "path": "crates/cellar/durability/raft/apply.rs", @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "068d12f48753389e1a46dba4a2c0a744765136622ed4c99a9b24652c8966bc2d", - "line": 116, + "sha256": "19523ded3a4ef95720f9f5ac130fec5b41533e7f6a2dfe627344c5819cfd7d66", + "line": 118, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "068d12f48753389e1a46dba4a2c0a744765136622ed4c99a9b24652c8966bc2d", - "line": 185, + "sha256": "19523ded3a4ef95720f9f5ac130fec5b41533e7f6a2dfe627344c5819cfd7d66", + "line": 187, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 7c6babc5a..7c9f8ac89 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1787,7 +1787,14 @@ "crates/cellar/durability/raft/cell_groups/recovery/state.rs", "crates/cellar/durability/raft/cell_groups/recovery/admission.rs", "crates/cellar/durability/raft/cell_groups/recovery/import.rs", - "crates/cellar/durability/raft/cell_groups/recovery/tests.rs" + "crates/cellar/durability/raft/cell_groups/recovery/tests.rs", + "crates/cellar/durability/raft/cell_groups/recovery/genesis.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/seed.rs", + "crates/cellar/tests/support/cell_group_disaster_seed.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/publication.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/bootstrap.rs", + "crates/cellar/durability/raft/cell_groups/directory/recovery/activation.rs", + "crates/cellar/tests/support/cell_group_disaster_activation.rs" ], "test_target": "native_cell_groups_archive", "steps": [ From 2c4326d5cf29f3935d4859e97a52af0ee08b170a Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 16:38:41 -0700 Subject: [PATCH 249/568] fix: retry stale receipt observations Concurrent apply can invalidate a proof while its producer remains owned. Return Uncertain so settlement obtains a fresh proof without replaying the callback. Keep identity, closed-engine and newer-term checks intact. Join actual query cancellation before expecting another one-slot admission. Preserve existing limits, deadlines and result assertions. --- .../durability/raft/cell_groups/README.md | 5 +++++ .../cellar/durability/raft/execution/cell.rs | 12 ++++++++++-- .../tests/support/cell_group_sql_reconcile.rs | 11 +++++++---- .../cell_group_sql_registered_queries.rs | 19 +++++++++++++++++++ 4 files changed, 41 insertions(+), 6 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index c8a5de1da..997caf477 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -464,6 +464,11 @@ apply and ambiguous command owners remain charged through physical shutdown. Quarantine cannot be cleared by registration. Recovery requires joined restart; a late persistence completion cannot restore the revoked serving fence. +Concurrent apply can invalidate a reconciliation observation before SQL inspects +it. This returns `ExecutionError::Uncertain` with the original producer retained. +The owned controller obtains a fresh observation and retries receipt inspection +without executing the callback again. Identity validation precedes this check. + Shared log-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Unknown events and controller ownership violations fail closed. Request-driven group provisioning, configuration lifecycle and public SDK diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 0985e214d..7b6359114 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -334,6 +334,8 @@ impl CellPrivateExecutor { /// Resolve the original request without requiring the original log entry. /// Missing receipts cannot retire an ambiguous same-term proposal. The /// caller must close this disposable producer after successful resolution. + /// A selected image that advances during inspection leaves the original + /// producer uncertain; retry with a fresh reconciliation observation. pub fn reconcile( &mut self, proposal: ProposalId, @@ -345,7 +347,13 @@ impl CellPrivateExecutor { { return Err(ExecutionError::Invalid("cell reconciliation identity")); } - authority.check().map_err(authority_failure)?; + let check = || { + authority.check().map_err(|error| match error { + RuntimeError::NotReady => ExecutionError::Uncertain, + error => authority_failure(error), + }) + }; + check()?; // Staging already established absence in this exact immutable base. // In particular an enrollment image need not yet have receipt tables. let receipt = if authority.cell().version == self.authority.cell().version { @@ -378,7 +386,7 @@ impl CellPrivateExecutor { Err(error) => return Err(error), Ok(None) => return Err(ExecutionError::Uncertain), }; - authority.check().map_err(authority_failure)?; + check()?; pending .prepared .release_log_reservation() diff --git a/crates/cellar/tests/support/cell_group_sql_reconcile.rs b/crates/cellar/tests/support/cell_group_sql_reconcile.rs index 1606eed9e..f80bf0276 100644 --- a/crates/cellar/tests/support/cell_group_sql_reconcile.rs +++ b/crates/cellar/tests/support/cell_group_sql_reconcile.rs @@ -175,10 +175,13 @@ pub(super) async fn reconcile(rooted: bool) -> Result<()> { stale_b.check().is_err(), "an old cell pin cannot prove absence after commit" ); - assert!(pool - .reconcile(prepared[1].proposal(), stale_b) - .await - .is_err()); + let stale = pool.reconcile(prepared[1].proposal(), stale_b).await; + assert!( + matches!(&stale, Err(ExecutionError::Uncertain)), + "stale observation must remain retryable: {:?}", + stale.as_ref().err() + ); + assert_eq!(pool.usage()?.active_cells, 2); // B committed; deliberately lose its exact-entry completion. Transfer real // leadership so node 0 must reconcile while it is a follower. diff --git a/crates/cellar/tests/support/cell_group_sql_registered_queries.rs b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs index 28bef7150..30cefe1f6 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered_queries.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered_queries.rs @@ -28,6 +28,20 @@ fn verify( reads::verify_observation(output, schema) } +async fn query_released(pool: &CellGroupSqlPool) -> Result<()> { + tokio::time::timeout(Duration::from_secs(3), async { + loop { + let usage = pool.usage()?; + if usage.queued_or_running == 0 && usage.retained_operations == 0 { + return Ok::<_, anyhow::Error>(()); + } + tokio::task::yield_now().await; + } + }) + .await + .context("query physical ownership did not settle")? +} + pub(crate) async fn queries(rooted: bool) -> Result<()> { let calls = Arc::new(AtomicUsize::new(0)); let observed = calls.clone(); @@ -206,6 +220,7 @@ pub(crate) async fn queries(rooted: bool) -> Result<()> { .await, Err(ExecutionError::Storage(_)) )); + query_released(&pool).await?; let deadline = reads::deadline(&fixture, 100).await?; let authority = fixture.runtimes[0] .read_authority( @@ -227,6 +242,10 @@ pub(crate) async fn queries(rooted: bool) -> Result<()> { ) .await? .is_err()); + // Deadline delivery cancels the waiter. Entered SQL and its sole queue + // slot remain owned until the worker observes cancellation and closes the + // image; verify that physical completion before expecting new admission. + query_released(&pool).await?; let authority = reads::selected(&mut fixture, 0, &a, CellReadConsistency::Stale).await?; let output = pool .query_registered( From e6e9072dc03100a020b4febd5da054962875321d Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 17:38:49 -0700 Subject: [PATCH 250/568] test: recover registered cell-group receipts --- .../tests/native_cell_groups_archive.rs | 29 +- .../tests/support/cell_group_directory.rs | 27 +- .../support/cell_group_disaster_activation.rs | 42 +- .../support/cell_group_disaster_fence.rs | 19 +- .../support/cell_group_disaster_receipts.rs | 694 ++++++++++++++++++ .../support/cell_group_engine_fixture.rs | 5 +- .../tests/support/cell_group_runtime.rs | 7 + 7 files changed, 803 insertions(+), 20 deletions(-) create mode 100644 crates/cellar/tests/support/cell_group_disaster_receipts.rs diff --git a/crates/cellar/tests/native_cell_groups_archive.rs b/crates/cellar/tests/native_cell_groups_archive.rs index c4b0df2c2..f622caa0b 100644 --- a/crates/cellar/tests/native_cell_groups_archive.rs +++ b/crates/cellar/tests/native_cell_groups_archive.rs @@ -37,6 +37,31 @@ mod runtime_fixture; include!("support/cell_group_engine_fixture.rs"); +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn fenced_group_seed_new_incarnations() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::registered_receipts( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_fenced_group_seed_new_incarnations() -> Result<()> { + for rooted in [false, true] { + Box::pin(disaster_fixture::registered_receipts( + rooted, + rustfs::store("recovery-receipts", rooted)?, + )) + .await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn group_recovery_publication_requires_seeded_quorum_activation() -> Result<()> { for rooted in [false, true] { @@ -325,6 +350,7 @@ use std::sync::atomic::Ordering; struct ArchiveDestination { store: CellGroupArchiveStore, images: GenerationStore, + apply_images: CellGroupApplyImages, } async fn destination(cluster: &Cluster) -> Result { @@ -403,7 +429,7 @@ async fn destination_at( let store = CellGroupArchiveStore::new( route, pages, - images, + images.clone(), ManifestLimits::default(), GroupEntryLimits::default(), 100, @@ -411,6 +437,7 @@ async fn destination_at( Ok(ArchiveDestination { store, images: generations, + apply_images: images, }) })? .await? diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 6786c9e53..98b757113 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -653,10 +653,31 @@ pub async fn birth_reservations() -> Result<()> { fault::missing_read(key.to_string(), f.run(d.reserve_group_birth(&original))).await, Err(CatalogError::Unconfirmed) )); + let reservation = f.store.inner.get(&key).await?.bytes().await?; + let payload = serde_json::to_vec(&serde_json::json!({ + "birth": original.encode(), + "recovery": null, + }))?; + let end = 13 + payload.len(); + assert_eq!(reservation.len(), end + 32); + assert_eq!(&reservation[..9], b"CLRCAT01\xc2"); + assert_eq!(&reservation[9..13], &(payload.len() as u32).to_be_bytes()); + assert_eq!(&reservation[13..end], payload.as_slice()); assert_eq!( - f.store.inner.get(&key).await?.bytes().await?.as_ref(), - original.encode() + &reservation[end..], + &Sha256::digest(&reservation[..end])[..] ); + // A bare birth is no longer a global reservation. The required origin + // wrapper must not be synthesized while reading an older record. + f.store + .inner + .put(&key, original.encode().to_vec().into()) + .await?; + assert!(matches!( + f.run(d.reserve_group_birth(&original)).await, + Err(CatalogError::Conflict(_)) + )); + f.store.inner.put(&key, reservation.clone().into()).await?; f.run(d.reserve_group_birth(&original)).await?; for which in 0..4 { let mut spec = birth_spec(&route); @@ -711,7 +732,7 @@ pub async fn birth_reservations() -> Result<()> { // The original reservation survives every conflicting attempt unchanged. assert_eq!( f.store.inner.get(&key).await?.bytes().await?.as_ref(), - original.encode() + reservation.as_ref() ); // Configuring a managed policy does not authorize the unmanaged APIs. No // fleet permit is fabricated, and refusal must precede any store access. diff --git a/crates/cellar/tests/support/cell_group_disaster_activation.rs b/crates/cellar/tests/support/cell_group_disaster_activation.rs index 4a3e48c2d..702b4fe6e 100644 --- a/crates/cellar/tests/support/cell_group_disaster_activation.rs +++ b/crates/cellar/tests/support/cell_group_disaster_activation.rs @@ -4,6 +4,11 @@ //! voter disks are gone. The external fence is still a contract fixture. use super::*; +#[path = "cell_group_disaster_receipts.rs"] +mod receipts; +pub(crate) use receipts::registered_receipts; +pub(super) use receipts::ReceiptWorkload; + struct FreshVoter { log: ReplicaLog, applier: CellGroupApplier, @@ -11,7 +16,8 @@ struct FreshVoter { base: std::path::PathBuf, options: ReplicaLogOptions, runtime: CellGroupRuntime, - events: CellGroupRuntimeEvents, + events: Option, + images: CellGroupApplyImages, sql: CellGroupSqlPool, } @@ -22,7 +28,12 @@ async fn pump( ) -> Result<()> { for (ordinal, voter) in voters.iter_mut().enumerate() { for _ in 0..64 { - let Ok(event) = voter.events.try_recv() else { + let Ok(event) = voter + .events + .as_mut() + .context("owned recovery events")? + .try_recv() + else { break; }; match event.effect().map_err(|e| anyhow::anyhow!(e.to_string()))? { @@ -78,7 +89,7 @@ async fn applied(voters: &mut [FreshVoter], history: GroupHistory, index: LogInd type SeededVoters = ( Vec, Vec, - Vec<(std::path::PathBuf, ReplicaLogOptions)>, + Vec<(std::path::PathBuf, ReplicaLogOptions, CellGroupApplyImages)>, ); struct FreshQuorum { @@ -164,7 +175,7 @@ fn seed_voters( Arc::new(DirectFileSystem), )) .await??; - reopen.push((base, options)); + reopen.push((base, options, destination.apply_images.clone())); let seed = s .cluster .directory @@ -315,7 +326,7 @@ fn publish_voters( )); s.remote.put(&binding_key, binding_bytes.into()).await?; let mut voters = Vec::new(); - for ((seed, log), (base, options)) in seeds.into_iter().zip(logs).zip(reopen) { + for ((seed, log), (base, options, images)) in seeds.into_iter().zip(logs).zip(reopen) { let permit = s .cluster .domain @@ -353,7 +364,8 @@ fn publish_voters( options, log, runtime, - events, + events: Some(events), + images, sql, }); } @@ -631,7 +643,7 @@ fn restart_voters<'a>( .await??; let group = log.cell_group(&history)?; let target = base.join("data"); - let (applier, root) = s + let (applier, root, images) = s .cluster .domain .blocking(move || reopen_applier(group, target, rooted))? @@ -675,7 +687,8 @@ fn restart_voters<'a>( applier, member, runtime, - events, + events: Some(events), + images, sql, base, options, @@ -699,7 +712,11 @@ fn reopen_applier( group: cellar::application::consensus::verification::GroupLog, base: std::path::PathBuf, rooted: bool, -) -> Result<(CellGroupApplier, PreparedGroupManifest)> { +) -> Result<( + CellGroupApplier, + PreparedGroupManifest, + CellGroupApplyImages, +)> { let filesystem: Arc = Arc::new(DirectFileSystem); let pages = ManifestPageStore::open( &base.join("pages"), @@ -737,12 +754,13 @@ fn reopen_applier( services, } }; - Ok(CellGroupApplier::recover( + let (applier, root) = CellGroupApplier::recover( group, pages, - images, + images.clone(), ManifestLimits::default(), GroupEntryLimits::default(), 100, - )?) + )?; + Ok((applier, root, images)) } diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs index ff9ac155f..a2a7589dc 100644 --- a/crates/cellar/tests/support/cell_group_disaster_fence.rs +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -9,7 +9,7 @@ use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; mod activation; #[path = "cell_group_disaster_seed.rs"] mod seed; -pub(super) use activation::activate_quorum; +pub(super) use activation::{activate_quorum, registered_receipts}; pub(super) use seed::seed_engine; #[derive(Default)] @@ -75,18 +75,31 @@ struct Scenario { impl Scenario { async fn new(rooted: bool, store: Arc) -> Result { + Box::pin(Self::with_workload(rooted, store, None)).await + } + + async fn with_workload( + rooted: bool, + store: Arc, + mut workload: Option<&mut activation::ReceiptWorkload>, + ) -> Result { let remote = archive_store::Store::around(store); let mut cluster = Cluster::with_store( Default::default(), tempfile::tempdir()?, Arc::new(DirectFileSystem), - [6; 32], + workload + .as_ref() + .map_or([6; 32], |w| w.class.schema_digest()), remote.clone(), ) .await?; cluster.elect().await?; - let expected = + let mut expected = checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 0, 8192).await?; + if let Some(workload) = workload.as_mut() { + Box::pin(workload.populate(&mut cluster, &mut expected)).await?; + } let archives = CellGroupArchivePool::new(Default::default())?; let binding = cluster .directory diff --git a/crates/cellar/tests/support/cell_group_disaster_receipts.rs b/crates/cellar/tests/support/cell_group_disaster_receipts.rs new file mode 100644 index 000000000..3e08eccbf --- /dev/null +++ b/crates/cellar/tests/support/cell_group_disaster_receipts.rs @@ -0,0 +1,694 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registered receipts and non-reusable client floors across a cold group seed. +use super::*; +use cellar::storage::CommandInputs; +use std::collections::BTreeMap; + +pub(in super::super) struct ReceiptWorkload { + pub(in super::super) class: Arc, + calls: Arc, + now: u64, + old: BTreeMap, + results: BTreeMap<(String, u64), Vec>, +} + +impl ReceiptWorkload { + fn new() -> Result { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("RecoveryCounter", 1, [84; 32])?.command( + "Add", + [85; 32], + move |context, input| { + count.fetch_add(1, Ordering::AcqRel); + context.sql().execute("UPDATE value SET n=n+1", [])?; + // Include the incarnation in the result: re-execution under the + // fresh identity could not produce the archived result. + let mut result = context.identity().cell.as_str().as_bytes().to_vec(); + result.extend_from_slice(&context.identity().incarnation.bytes()); + result.extend_from_slice(input); + Ok(result) + }, + )?); + Ok(Self { + class, + calls, + now: 0, + old: BTreeMap::new(), + results: BTreeMap::new(), + }) + } + + fn request(&self, sequence: u64, input: &[u8]) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.class.contract("Add").context("Add")?, + input: input.to_vec(), + durability: Durability::Quorum, + retry_until_ms: self.now + 300_000, + }) + } + + pub(in super::super) async fn populate( + &mut self, + cluster: &mut Cluster, + expected: &mut BTreeMap>, + ) -> Result<()> { + self.now = cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + for (name, sequence) in [("alpha", 1), ("alpha", 2), ("alpha", 3), ("beta", 1)] { + let cell = history(cluster.birth.route(), name)?; + let authority = cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .execution_authority(&cell)?; + let voter = &cluster.voters[0]; + let services = ReadServices { + now_unix_millis: Arc::new(|| 10), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = match &voter.extents { + Some(images) => CellGroupApplyImages::Extents { + images: images.clone(), + services, + limits: Default::default(), + cache_directory: cluster._root.path().join("node-1/source-query-cache"), + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache: CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + }, + }, + None => CellGroupApplyImages::Materialized { + images: voter.images.clone(), + services, + }, + }; + let mut options = sql_options( + images, + cluster + ._root + .path() + .join(format!("node-1/source-sql-{name}")), + self.class.schema_digest(), + )?; + match &mut options { + CellSqlOptions::Materialized(o) => o.limits.max_retained_receipts = 2, + CellSqlOptions::Extents(o) => o.limits.max_retained_receipts = 2, + } + let input = vec![sequence as u8]; + let mut request = self.request(sequence, &input)?; + request.durability = if name == "alpha" && sequence >= 2 { + Durability::S3 + } else { + Durability::Quorum + }; + if name == "alpha" && sequence == 1 { + request.retry_until_ms = self.now - 1000; + } + let attempt = CommandAttempt { + request: CommandRequest { + id: request.id, + operation: "Add".into(), + contract_digest: [85; 32], + request_digest: command_request_digest(&input), + durability: request.durability, + retry_until_ms: request.retry_until_ms, + }, + proposal: ProposalId::new([100 + self.calls.load(Ordering::Acquire) as u8; 16])?, + preview: ImageId::new([100 + self.calls.load(Ordering::Acquire) as u8; 16])?, + inputs: CommandInputs { + now_unix_millis: if sequence <= 2 && name == "alpha" { + self.now - 2000 + } else { + self.now + }, + random: [17; 32], + }, + }; + let class = self.class.clone(); + let (executor, prepared) = cluster + .domain + .blocking(move || -> Result<_> { + let mut executor = open_private(options, authority)?; + let CellStageOutcome::Prepared(prepared) = + executor.stage_registered(attempt, &class, &input, true)? + else { + anyhow::bail!("source command must execute once") + }; + executor.mark_submitted(prepared.proposal())?; + Ok((executor, prepared)) + })? + .await??; + let CellSubmissionOutcome::Accepted(accepted) = cluster.voters[0] + .owner + .as_mut() + .context("source owner")? + .submit(prepared)? + else { + anyhow::bail!("source command refused") + }; + cluster.drive(false).await?; + let authority = cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .execution_authority(&cell)?; + assert!(authority.cell().last_mutation.index >= accepted.position.index); + let receipt = cluster + .domain + .blocking(move || -> Result<_> { + let mut executor = executor; + let receipt = executor.finish(authority)?; + executor.close()?; + Ok(receipt) + })? + .await??; + self.results + .insert((name.into(), sequence), receipt.receipt.result().to_vec()); + self.old.insert(name.into(), receipt.observation()); + expected.insert( + name.into(), + image_bytes( + cluster, + cluster.voters[0] + .owner + .as_ref() + .context("source owner")? + .manifest(), + name, + ) + .await?, + ); + } + assert_eq!(self.calls.load(Ordering::Acquire), 4); + Ok(()) + } +} + +fn sql_options( + images: CellGroupApplyImages, + directory: std::path::PathBuf, + schema: [u8; 32], +) -> Result { + let host = cellar_ltx::LtxHost::new( + || 10, + |_| Ok(Duration::ZERO), + |path| async move { std::fs::read(path) }, + |job| async move { + job(); + Ok(()) + }, + ) + .with_filesystem(Arc::new(DirectFileSystem)); + let mut limits = ExecutionLimits::default(); + // This small workload has a 1 MiB capture contract. The default 16 MiB + // payload contract alone reserves four buffers, exceeding the fixture's + // unchanged 64 MiB group SQL budget before receipt lookup can run. + limits.envelope.max_payload_bytes = 1024 * 1024; + Ok(match images { + CellGroupApplyImages::Materialized { images, services } => { + CellSqlOptions::Materialized(PrivateOptions { + directory, + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + }) + } + CellGroupApplyImages::Extents { + images, + services, + cache, + .. + } => CellSqlOptions::Extents(CellSqlRootOptions { + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + cache_directory: directory.with_extension("read-cache"), + // The writer and immutable receipt reader may coexist. Neither may + // borrow the applier's independently admitted single-file cache. + cache_pool: ScratchPool::new(ScratchLimits { + max_bytes: 2 * EXTENT_BYTES, + max_files: 2, + })?, + cache, + }), + }) +} + +fn open_private( + options: CellSqlOptions, + authority: CellExecutionAuthority, +) -> Result { + Ok(match options { + CellSqlOptions::Materialized(o) => CellPrivateExecutor::open(o, authority)?, + CellSqlOptions::Extents(o) => { + let workspace = o.images.create_workspace( + cell_image_history_digest(&authority.cell().version.history), + RootPrivateOptions::workspace_bounds(o.limits)?, + )?; + CellPrivateExecutor::open_rooted( + RootPrivateOptions { + workspace, + host: o.host, + sqlite_vfs: o.sqlite_vfs, + images: o.images, + query_services: o.query_services, + application_digest: o.application_digest, + limits: o.limits, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + authority, + )? + } + }) +} + +async fn image_bytes( + cluster: &Cluster, + root: &PreparedGroupManifest, + name: &str, +) -> Result> { + let (_, pin) = root + .pin_cell(&CellId::new(name)?) + .context("selected image")?; + pin_bytes(cluster, pin).await +} + +async fn pin_bytes(cluster: &Cluster, pin: CellImagePin) -> Result> { + cluster + .domain + .blocking(move || -> Result<_> { + Ok(match pin { + CellImagePin::Empty => vec![], + CellImagePin::Materialized(pin) => pin + .open_reader()? + .read_exact_at(0, pin.state().length as usize)?, + CellImagePin::Extents(pin) => { + let mut bytes = Vec::new(); + for ordinal in 0..pin.manifest().geometry().extent_count() { + bytes.extend( + pin.view() + .read_extent(pin.manifest().extent(ordinal).context("extent")?)?, + ); + } + bytes + } + }) + })? + .await? +} + +fn keys() -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(600), + )?) +} + +async fn controllers_applied( + voters: &[FreshVoter], + history: GroupHistory, + index: LogIndex, +) -> Result<()> { + tokio::time::timeout(Duration::from_secs(15), async { + loop { + let mut done = true; + for voter in voters { + done &= voter.runtime.status(history).await?.applied.index >= index; + } + if done { + return Ok::<_, anyhow::Error>(()); + } + tokio::time::sleep(Duration::from_millis(2)).await; + } + }) + .await? +} + +struct Serving { + controllers: Vec, + transports: Vec, + dispatcher: CellGroupDispatcher, +} + +impl Serving { + async fn start( + s: &Scenario, + quorum: &mut FreshQuorum, + workload: &ReceiptWorkload, + rooted: bool, + ordinal: u8, + ) -> Result { + let mut transports = Vec::new(); + for (node, voter) in quorum.voters.iter().enumerate() { + transports.push(runtime_fixture::start_recovery_transport(&voter.runtime, node).await?); + } + let route = cellar::application::consensus::verification::GroupRoute { + scope: quorum.route.history, + schema_digest: quorum.route.schema_digest, + revision: 1, + peers: quorum + .voters + .iter() + .zip(&transports) + .enumerate() + .map(|(node, (voter, transport))| -> Result<_> { + Ok(RaftPeer { + replica: voter + .log + .cell_group(&quorum.route.history)? + .identity() + .clone(), + label: format!("node-{}", node + 1), + server_name: "localhost".into(), + control_address: transport.control_address(), + data_address: transport.data_address(), + query_address: transport.query_address(), + }) + }) + .collect::>()?, + }; + for transport in &transports { + transport.install_physical_group(route.clone())?; + } + let mut controllers = Vec::new(); + for (voter, transport) in quorum.voters.iter_mut().zip(&transports) { + controllers.push( + CellGroupController::start( + voter.runtime.clone(), + voter.events.take().context("exclusive recovered events")?, + transport.clone(), + voter.sql.clone(), + vec![voter.applier.clone()], + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + ) + .await?, + ); + } + let io = CellGroupArchivePool::new(Default::default())?; + let birth = s + .cluster + .domain + .run( + s.cluster + .directory + .materialize_selected_group(&quorum.route), + ) + .await??; + if ordinal == 0 { + s.cluster + .directory + .prepare_group_archive_binding(&s.cluster.domain, &io, &birth) + .await? + .commit(&s.cluster.domain) + .await?; + } else { + s.cluster + .directory + .open_group_archive(&s.cluster.domain, &io, &birth) + .await?; + } + let verifier = destination_at( + &s.cluster, + rooted, + Arc::new(DirectFileSystem), + s.cluster + ._root + .path() + .join(format!("fresh-verifier-{ordinal}")), + quorum.route.clone(), + ) + .await?; + controllers[0] + .archive(CellGroupArchiveRegistration { + directory: s.cluster.directory.clone(), + birth, + io, + owners: CellGroupArchiveUploadPool::new(Default::default())?, + tails: CellGroupArchiveTailPool::new(Default::default())?, + checkpoints: CellGroupCheckpointPool::new(checkpoint_fixture::limits())?, + store: verifier.store, + scratch: ScratchPool::new(ScratchLimits { + max_bytes: 32 * 1024 * 1024, + max_files: 1, + })?, + scratch_directory: s.cluster._root.path().into(), + })? + .response() + .await?; + let dispatcher = CellGroupDispatcher::new( + &s.cluster.domain, + s.cluster.directory.clone(), + workload.class.clone(), + keys()?, + CellDispatchLimits { + max_cached_routes: 4, + route_cache_bytes: 256 * 1024, + max_invocations: 4, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )? + .with_s3(&controllers[0])?; + Ok(Self { + controllers, + transports, + dispatcher, + }) + } + + async fn command( + &self, + s: &Scenario, + quorum: &FreshQuorum, + w: &ReceiptWorkload, + name: &str, + sequence: u64, + input: &[u8], + ) -> Result { + let cell = fresh_cell(s, quorum, name)?; + let call = self + .dispatcher + .command( + cell.clone(), + w.request(sequence, input)?, + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + }, + None, + ) + .await?; + Ok(self.controllers[0] + .command( + call, + sql_options( + quorum.voters[0].images.clone(), + quorum.voters[0].base.join(format!("sql-{name}")), + quorum.route.schema_digest, + )?, + )? + .response() + .await?) + } + + async fn close(self) -> Result<()> { + for controller in self.controllers { + controller.close().await?; + } + for transport in self.transports { + transport.close().await; + } + Ok(()) + } +} + +fn fresh_cell(_s: &Scenario, quorum: &FreshQuorum, name: &str) -> Result { + Ok(CellHistory { + cell: CellId::new(name)?, + ..quorum.cell.clone() + }) +} + +pub(crate) async fn registered_receipts(rooted: bool, store: Arc) -> Result<()> { + let mut workload = ReceiptWorkload::new()?; + let mut s = + Box::new(Box::pin(Scenario::with_workload(rooted, store, Some(&mut workload))).await?); + let seeds = seed_voters(&mut s, rooted).await?; + let mut quorum = publish_voters(&mut s, seeds).await?; + activate_voters(&s, &mut quorum).await?; + serving_phase(&mut s, &mut quorum, &workload, rooted, 0).await?; + restart_voters(&s, &mut quorum, rooted).await?; + serving_phase(&mut s, &mut quorum, &workload, rooted, 1).await?; + for voter in quorum.voters { + voter.runtime.close().await?; + voter.sql.close().await?; + voter.log.close().await?; + } + s.cluster.close().await +} + +fn serving_phase<'a>( + s: &'a mut Scenario, + quorum: &'a mut FreshQuorum, + w: &'a ReceiptWorkload, + rooted: bool, + ordinal: u8, +) -> futures_util::future::BoxFuture<'a, Result<()>> { + Box::pin(async move { + let serving = Serving::start(s, quorum, w, rooted, ordinal).await?; + let now = s + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + for name in ["alpha", "beta"] { + let cell = fresh_cell(s, quorum, name)?; + let old = &w.old[name]; + let old_token = keys()?.issue_cell(old, quorum.route.schema_digest, w.now)?; + // Prove rejection is due to the history, while the old token remains + // valid for its original identity at the same real wall time. + assert_eq!( + keys()?.verify_cell( + &old_token, + &old.version.history, + quorum.route.schema_digest, + now + )?, + *old + ); + assert_eq!( + keys()?.verify_cell(&old_token, &cell, quorum.route.schema_digest, now), + Err(VersionError::WrongHistory) + ); + assert_ne!(old.version.history, cell); + assert_ne!(old.witness.group, quorum.route.history); + let sequence = if name == "alpha" { 2 } else { 1 }; + let response = serving + .command(s, quorum, w, name, sequence, &[sequence as u8]) + .await?; + assert_eq!( + response.receipt().receipt.result(), + &w.results[&(name.into(), sequence)] + ); + assert_eq!(response.receipt().version.history, cell); + assert_eq!(response.receipt().witness.group, quorum.route.history); + assert_eq!( + response.receipt().version.sequence.get(), + if ordinal == 1 && name == "alpha" { + 2 + } else { + 1 + } + ); + assert_eq!( + response.receipt().receipt.minimum_durability(), + if name == "alpha" { + Durability::S3 + } else { + Durability::Quorum + } + ); + assert_eq!( + response.durability(), + if name == "alpha" { + Durability::S3 + } else { + Durability::Quorum + } + ); + let response_time = s + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 }) + .await?; + assert_eq!( + keys()?.verify_cell( + response.token(), + &cell, + quorum.route.schema_digest, + response_time + )?, + response.receipt().observation() + ); + } + assert_eq!( + w.calls.load(Ordering::Acquire), + if ordinal == 0 { 4 } else { 5 } + ); + let floor = serving + .command(s, quorum, w, "alpha", 1, &[1]) + .await + .err() + .context("restored expired floor")?; + assert!(matches!( + floor + .downcast_ref::>() + .map(|e| e.cause.as_ref()), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandExpired + ))) + )); + let conflict = serving + .command(s, quorum, w, "alpha", 2, b"foreign input") + .await + .err() + .context("restored receipt identity")?; + assert!(matches!( + conflict + .downcast_ref::>() + .map(|e| e.cause.as_ref()), + Some(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::CommandConflict + ))) + )); + if ordinal == 0 { + let response = serving.command(s, quorum, w, "alpha", 4, &[4]).await?; + assert_eq!(response.receipt().version.sequence.get(), 2); + assert_eq!(response.receipt().version.history, quorum.cell); + let index = response.receipt().position.index; + drop(response); + controllers_applied(&quorum.voters, quorum.route.history, index).await?; + let root = quorum.voters[0].applier.history(); + // Read from the actual fresh engine-selected pin; retained restart + // below must preserve this new image as well as both old receipts. + let authority = quorum.voters[0] + .runtime + .reconciliation_authority(root, quorum.cell.clone()) + .await?; + s.expected.insert( + "alpha".into(), + pin_bytes(&s.cluster, authority.image().clone()).await?, + ); + } else { + let response = serving.command(s, quorum, w, "alpha", 4, &[4]).await?; + assert_eq!(response.receipt().version.sequence.get(), 2); + } + assert_eq!(w.calls.load(Ordering::Acquire), 5); + serving.close().await + }) +} diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 0d296a9c9..740b33ba4 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -440,6 +440,7 @@ fn apply_image( fn prepare_cell_image( images: &GenerationStore, + schema: [u8; 32], extents: Option<&cellar_ltx::internal::NodeExtentStore>, cell: &CellHistory, sequence: u64, @@ -484,7 +485,7 @@ fn prepare_cell_image( }; let binding = ImageBinding { history: cell_image_history_digest(cell), - schema: [6; 32], + schema, epoch: image.epoch, txid: image.txid, checksum: image.checksum, @@ -584,6 +585,7 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result Result Result { + tcp::start(runtime, node).await +} + struct RuntimeCluster { snapshot_sources: BTreeMap<(usize, GroupHistory), PreparedGroupManifest>, snapshot_transfers: usize, From 820fa955c31d414923b99e0e8be147d7f21cf7b4 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 18:38:22 -0700 Subject: [PATCH 251/568] test: restore receipts from archived LTX tails --- .../cellar/tests/support/cell_group_disaster_fence.rs | 8 +++++--- .../cellar/tests/support/cell_group_engine_fixture.rs | 10 ++++------ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_disaster_fence.rs b/crates/cellar/tests/support/cell_group_disaster_fence.rs index a2a7589dc..ce0c32900 100644 --- a/crates/cellar/tests/support/cell_group_disaster_fence.rs +++ b/crates/cellar/tests/support/cell_group_disaster_fence.rs @@ -97,9 +97,6 @@ impl Scenario { cluster.elect().await?; let mut expected = checkpoint_fixture::enroll_images(&mut cluster, rooted, false, 0, 8192).await?; - if let Some(workload) = workload.as_mut() { - Box::pin(workload.populate(&mut cluster, &mut expected)).await?; - } let archives = CellGroupArchivePool::new(Default::default())?; let binding = cluster .directory @@ -117,6 +114,11 @@ impl Scenario { let checkpoint = upload .upload_checkpoint(&cluster.domain, source(&sources, &cluster)?) .await?; + // Put registered receipts and client-floor maintenance in the archived + // LTX tail, after the independent base checkpoint was uploaded. + if let Some(workload) = workload.as_mut() { + Box::pin(workload.populate(&mut cluster, &mut expected)).await?; + } let group = cluster.voters[0] .log .cell_group(&cluster.birth.route().history)?; diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 740b33ba4..486c994c9 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -439,9 +439,8 @@ fn apply_image( } fn prepare_cell_image( - images: &GenerationStore, + stores: (&GenerationStore, Option<&cellar_ltx::internal::NodeExtentStore>), schema: [u8; 32], - extents: Option<&cellar_ltx::internal::NodeExtentStore>, cell: &CellHistory, sequence: u64, base: Option<&CellImagePin>, @@ -451,6 +450,7 @@ fn prepare_cell_image( use cellar_ltx::internal::{ ExtentManifest, ExtentRootReference, Geometry, ImageBinding, ImageRootUpdate, EXTENT_BYTES, }; + let (images, extents) = stores; if image.txid == 0 { anyhow::ensure!(base.is_none() && files.is_empty()); return Ok(EnrollmentImage { @@ -584,9 +584,8 @@ fn prepare(voter: &mut Voter, apply: &CellGroupApply) -> Result Result { let (old, base) = apply.base.pin_cell(&mutation.cell.cell).context("cell")?; let prepared = prepare_cell_image( - &voter.images, + (&voter.images, voter.extents.as_ref()), apply.base.marker().route().schema_digest, - voter.extents.as_ref(), &mutation.cell, mutation.sequence.get(), Some(&base), From 9a0bad9124f0089055a0ba63b2952e5ef5e77853 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 18:48:14 -0700 Subject: [PATCH 252/568] test: reuse shared transport fixture directly --- .../cellar/tests/support/cell_group_disaster_receipts.rs | 2 +- crates/cellar/tests/support/cell_group_engine_fixture.rs | 5 ++++- crates/cellar/tests/support/cell_group_runtime.rs | 9 +-------- crates/cellar/tests/support/cell_group_tcp.rs | 2 +- 4 files changed, 7 insertions(+), 11 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_disaster_receipts.rs b/crates/cellar/tests/support/cell_group_disaster_receipts.rs index 3e08eccbf..765fc85f6 100644 --- a/crates/cellar/tests/support/cell_group_disaster_receipts.rs +++ b/crates/cellar/tests/support/cell_group_disaster_receipts.rs @@ -358,7 +358,7 @@ impl Serving { ) -> Result { let mut transports = Vec::new(); for (node, voter) in quorum.voters.iter().enumerate() { - transports.push(runtime_fixture::start_recovery_transport(&voter.runtime, node).await?); + transports.push(runtime_fixture::tcp::start(&voter.runtime, node).await?); } let route = cellar::application::consensus::verification::GroupRoute { scope: quorum.route.history, diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 486c994c9..14dc2c739 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -439,7 +439,10 @@ fn apply_image( } fn prepare_cell_image( - stores: (&GenerationStore, Option<&cellar_ltx::internal::NodeExtentStore>), + stores: ( + &GenerationStore, + Option<&cellar_ltx::internal::NodeExtentStore>, + ), schema: [u8; 32], cell: &CellHistory, sequence: u64, diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 29a2dd5e4..9020c189d 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -31,16 +31,9 @@ mod checkpoint_receive; pub(super) use checkpoint_receive::{install_checkpoint, refuse_checkpoint}; #[path = "cell_group_tcp.rs"] -mod tcp; +pub(super) mod tcp; pub(super) use tcp::physical_transport; -pub(super) async fn start_recovery_transport( - runtime: &CellGroupRuntime, - node: usize, -) -> Result { - tcp::start(runtime, node).await -} - struct RuntimeCluster { snapshot_sources: BTreeMap<(usize, GroupHistory), PreparedGroupManifest>, snapshot_transfers: usize, diff --git a/crates/cellar/tests/support/cell_group_tcp.rs b/crates/cellar/tests/support/cell_group_tcp.rs index 5f2648cc2..da3af27af 100644 --- a/crates/cellar/tests/support/cell_group_tcp.rs +++ b/crates/cellar/tests/support/cell_group_tcp.rs @@ -21,7 +21,7 @@ pub(super) fn tls() -> Result { )?) } -pub(super) async fn start( +pub(crate) async fn start( runtime: &CellGroupRuntime, n: usize, ) -> Result { From 1e4570929b731c2d9d0a2c8c8ce3f8d883c92489 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 19:02:57 -0700 Subject: [PATCH 253/568] docs: record recovered receipt qualification --- docs/design/multi-raft/cell-groups/README.md | 13 ++- .../multi-raft/cell-groups/baseline.json | 18 ++-- docs/design/multi-raft/cell-groups/handoff.md | 97 +++++++++---------- .../design/multi-raft/cell-groups/progress.md | 57 +++++++++++ 4 files changed, 121 insertions(+), 64 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index c7b7303d4..df5966dd8 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -8,12 +8,12 @@ complete checkpoint, authenticated learner/rebuild, membership, restart and resource-ownership requirements pass at `e0550acd`. Its current gate includes all five exact cases and 205 Rust regressions across eighteen commands. CG-06 is IN_PROGRESS: complete S3 checkpoint/tail recovery, bounded S3 settlement, -local log reclamation, exact recovery fencing and fresh-history data preparation -pass their scoped gates. Fresh engine genesis, atomic fenced publication, -committed activation and a fresh serving quorum remain the immediate priorities. +local reclamation, fresh physical genesis, fenced publication, committed activation +and registered receipt recovery through a fresh quorum pass scoped native/RustFS +checks. Actual shared-group Docker launch/restart fencing is the immediate gate. CG-08 due effects and safe cold-cache eviction follow, before CG-07 SDK/fleet cutover. See the latest [progress](progress.md) and [handoff](handoff.md). -The production SDK/fleet has not yet cut over; topology, hibernation, complete S3 +The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. Package validation alone does not qualify implementation or a release. @@ -48,7 +48,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package only with verified evidence and completed dependencies. Final qualification repeats the full capability matrix at one release revision. -This checkout retains evidence under the ignored +Historical native gate records are referenced under the ignored `.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/`, `.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` and `.cellar/cell-groups-evidence/CG-03-routing-final/` directories. CG-04 adds @@ -68,6 +68,9 @@ checkpoints and local publication in `.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` and membership in `.cellar/cell-groups-evidence/CG-05-group-membership/` remain historical partial records; the complete current-source gate above closes CG-05. +The referenced CG-01–05 raw directories are currently unavailable locally. +Restore their records or rerun the unchanged exact gates before complete package +validation. The scoped CG-06 receipt checkpoint does not replace them. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 95bd9abef..5581c61a0 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "7052d218d60181d7b0bb5a4c12686e80e294a61c", - "runtime_revision": "65c5176975907fb52a53deca58e1ae1f5ba7adec", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged.", + "source_revision": "9a0bad9124f0089055a0ba63b2952e5ef5e77853", + "runtime_revision": "9a0bad9124f0089055a0ba63b2952e5ef5e77853", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -37,7 +37,7 @@ }, "host": { "path": "crates/cellar/durability/raft/host.rs", - "sha256": "94d5b62a99129b057377243a737c9e8ac642438eb677ad50ed78ccc15379bcc2", + "sha256": "6cdbaacdd4fd1dab6be10c9ce3f4619847e65ace5395b5eefe66eb9adb3cb08c", "line": 1313, "excerpt": "struct Cell {\n extents: Option,\n birth: GroupBirth,\n route_revision: LogIndex,\n configuration: Option,\n retired: bool,\n archive: CellArchive,\n archived: LogIndex,\n archive_retained: admission::Retained," }, @@ -49,14 +49,14 @@ }, "runtime": { "path": "crates/cellar/durability/raft/runtime.rs", - "sha256": "8f35279c42414980eca58899cdc87e17c98b45ae26bac9c68e1209a58452b890", - "line": 731, + "sha256": "9b205876c4f28ec325eaab732de9f687efa7364b8b808dd67944c22fd370e07a", + "line": 759, "excerpt": "struct Group {\n replica: Replica,\n scheduled: bool,\n write_scheduled: bool,\n deferred: VecDeque,\n inbound: Arc,\n hints: BTreeMap,\n tick: bool,\n failed: Option," }, "ticks": { "path": "crates/cellar/durability/raft/runtime.rs", - "sha256": "8f35279c42414980eca58899cdc87e17c98b45ae26bac9c68e1209a58452b890", - "line": 1059, + "sha256": "9b205876c4f28ec325eaab732de9f687efa7364b8b808dd67944c22fd370e07a", + "line": 1084, "excerpt": " fn tick(&mut self) {\n for (scope, group) in &mut self.groups {\n group.tick = true;\n group.wake(scope, &mut self.order);\n }\n }\n\n fn progress(&mut self) -> Result {\n let mut made = false;" }, "catalog": { @@ -145,7 +145,7 @@ }, "ci": { "path": ".github/workflows/native-sdk.yml", - "sha256": "05c5dc73b4ed7c2a29af7ada4765db3dbd6ec7645c8d2c997223417b18145de0", + "sha256": "cccc977d7cf0bb2f03cf2a604fee53fa7a6f1e077294ab96368b8aaadc5d09f2", "line": 67, "excerpt": " RUST_VERSION: 1.98.1\n # Process tests publish executable artifacts; DWARF makes those fixtures huge.\n # Keep debug assertions and test behavior while omitting symbol information.\n CARGO_PROFILE_DEV_DEBUG: \"0\"\n CARGO_PROFILE_TEST_DEBUG: \"0\"\n\njobs:\n filesystem_release:\n if: github.event_name != 'workflow_dispatch' || inputs.mode == 'qualification'" } diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 2d495e424..228eda274 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -13,57 +13,54 @@ node resources. A group spans voter nodes; it is not all cells on one node. ## Starting point -Use branch `codex/shared-cell-groups` and the existing +Use branch codex/shared-cell-groups and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest runtime source is 7052d218d60181d7b0bb5a4c12686e80e294a61c. -Fenced source data now imports into complete fresh-history data roots, with -pending application admission and bounded physical ownership. Seven native suites -pass 246 cases; 36 recovery-related unit tests, eleven legacy disaster cases and -the exact real RustFS import case in both image forms pass. Scoped static checks -and independent source/artifact/selection/credential audit pass. Evidence is at -`.cellar/cell-groups-evidence/CG-06-recovery-import/`; qualification SHA256: -`1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3`. See [progress](progress.md) for scope and retained limitations. - -The mandatory nullable recovery field is a marker-1 hard cutover, with no reader -for older intermediate cell-group manifest/archive shapes. Keep recovery state -through snapshots and retained reopen. `PreparedCellGroupRecovery` is data owned -by an archive store; never cast that owner into a Raft engine or learner permit. -Its source hold remains provenance and must stay retained while referenced. - -The most critical next work is physical fresh genesis and safe activation: - -1. Consume the exact preparation through a node/member/disk/process/engine-bound - recovery permit and atomically sync the complete imported root with the new - Raft mapping/genesis. Every seeded voter must bind the same request and seed - commitment. Preserve physical pins through unknown sync results. Do not copy - old Raft terms/configuration or reopen an already owned page directory. -2. Persist the target recovery binding before publication. Ordinary bootstrap - must refuse blank replacement disks or an empty/incorrect genesis. Reverify - the permanent external fence on process resume. Continue using the accepted - fencing-authority API; no new approval is required for Docker qualification. -3. Publish the exact replacement mutation with one CAS under the existing - recovery lock. Never clear the lock before ordinary publication. Mint a - catalog-bound leader activation permit only after exact readback and seed - validation. Raw administration remains unattached. Commit, persist and apply - activation through the fresh quorum before opening any application authority. -4. Implement the original `fenced_group_seed_new_incarnations` oracle with real - registered commands, receipt results/minima/client floors, old-token refusal - and zero duplicate callbacks. The current import checks exact SQLite bytes; - it does not qualify the new serving receipt path. Exercise pending/active - recovered roots through actual engine restart and snapshot installation. -5. Qualify Docker permanent launch/restart revocation for every possible old - writer, including later/removed voters. Erase every owned voter disk and local - recovery cache, restore all cells from real RustFS under fresh identities, - perform new writes and restart the retained replacement quorum. Keep effect - ambiguity closed. Complete the original full CG-06 gate before marking DONE. - -CG-08 is next after CG-06: replicated effect summaries exist, but due-work -scheduling independent of client requests and bounded idle SQLite caches remain -missing. Preserve effect identity, unknown outcomes and joined worker ownership. -Then perform CG-07 public SDK/fleet cutover. CG-09–12 transfers, split/merge, -placement and hibernation; CG-14/13/15 workload and managed recovery/qualification; -and CG-16 final legacy removal and release matrix all remain required. The full -hard-cutover goal remains active and incomplete. +The latest tested source is 9a0bad9124f0089055a0ba63b2952e5ef5e77853, containing origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. +Fresh physical genesis, exact fenced publication, committed activation and +fresh-quorum admission are implemented. Registered receipt recovery now passes +through production TLS/controllers/SQL and the replacement archive in both image +forms. Archive/catalog suites pass 47 cases; five exact real RustFS cases pass in +both forms with seven signed S3 probes. All-target Cellar/logic Clippy and the +scoped format/layout/dependency/documentation checks pass. + +See [progress](progress.md) for the scope and limitations. The scoped record is +.cellar/cell-groups-evidence/CG-06-recovery-genesis-settlement/receipt-recovery-qualification.json (SHA256 eb0efe6af7d4e6bed3a9801d821e0ea8e48058f7ce949cdc01912a74642fe25d). This is not a complete CG-06 gate. +CG-06 remains IN_PROGRESS; the full hard-cutover goal remains active. + +The most critical next work is the accepted Docker fencing qualification: + +1. Add an actual shared-group voter path to the Docker qualification harness. + Its current raft_node.rs/raft_disaster.py path still uses the older per-cell + runtime. Use three real voter containers and a durable source command ledger; + archive complete group checkpoint/tail state against RustFS. +2. Reuse the controller's exact owner/container/volume checks and private signing + key. Durably revoke the old deployment in the sole launcher's policy before + signing. Disable restarts, stop/join every possible old writer, remove the + immutable containers and all voter volumes, and confirm absence. Reload the + controller and prove old-deployment launch remains refused. No additional + provider choice or approval is needed: fencing-authority API plus Docker + qualification was already accepted. +3. Run the recovery operator with only the exact held S3 cut, request and signed + certificate. Seed fresh node/member/disk/process/engine identities, publish + once under the recovery lock and commit/apply activation through the fresh + quorum. Recheck original results, stored S3 minimums, client floors, old-token + refusal and no duplicate callback; perform new writes and retained restart. +4. Finish the original CG-06 gate without changing its commands or oracle. + Restore or refresh the unavailable CG-01–05 raw evidence before claiming + complete package validation. Keep the source hold as recovery provenance. + +The required nullable recovery metadata remains an internal marker-1 hard +cutover. There is no legacy reader, state migration or permanent mode flag. +PreparedCellGroupRecovery is private data preparation; only the sealed physical +seed, recovered-bootstrap and activation contracts admit their respective next +steps. Never reopen owned page/image directories or cast archive ownership into +engine authority. Keep unknown outcomes and physical joins owned. + +CG-08 follows CG-06: implement bounded fair due-work scheduling independent of +client requests and qualify the current close-on-settle/exact-root reopen policy. +Preserve effect identity and unresolved delivery outcomes. Then perform CG-07 +public SDK/fleet cutover. CG-09–16 topology, placement, hibernation, managed +recovery, crash/load qualification and final legacy removal remain required. The prior whole-group fencing-barrier source is 65c5176975907fb52a53deca58e1ae1f5ba7adec, with 211 native cases and four real RustFS diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index f37676428..a2f863a06 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,63 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 fresh quorum and registered receipt recovery + +Source: 9a0bad9124f0089055a0ba63b2952e5ef5e77853, including origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. CG-06 remains IN_PROGRESS. +Fresh recovery now installs the complete imported root and physical Raft genesis +in one synced engine publication. Permanent seed records bind the exact recovery +request, genesis commitment, node, member, disk and engine. Ordinary bootstrap +refuses recovered births; recovered startup revalidates the seed and external +fencing authority. Replacement publication retains the recovery lock until a +catalog-bound activation is committed and applied through the fresh quorum. + +The original `fenced_group_seed_new_incarnations` oracle now exercises registered +commands. It writes actual receipts and an expired client floor after the base +checkpoint, archives their LTX tails, and deletes every old native voter directory, +private SQL files, extent store and the previous +verification cache, then seeds and activates three fresh voters. Production TLS +transports, controllers, SQL workers and the replacement group's archive owner +serve retries. Exact opaque results survive without callback replay; Quorum +retries honor stored S3 minimums through actual replacement-history coverage. +Expired sequences and conflicting inputs refuse. Old tokens remain valid for +their original history at the same wall time and fail with WrongHistory for the +fresh history. A new registered write and retained-quorum restart preserve the +imported receipts and client floors. + +At this source, archive/catalog suites pass 47 cases; five explicitly ignored +RustFS cases pass when selected against the owned real backend, each in both +materialized and extent modes. Seven signed S3 operations verify conditional +creation, CAS and readback behavior. All-target locked Cellar/logic Clippy, +format/layout over 1985 Rust files, native dependency and documentation checks +pass. The RustFS backend and dedicated VM are stopped with data retained; the +default Docker context remains colima. + +Scoped record: .cellar/cell-groups-evidence/CG-06-recovery-genesis-settlement/receipt-recovery-qualification.json (SHA256 eb0efe6af7d4e6bed3a9801d821e0ea8e48058f7ce949cdc01912a74642fe25d). Records bind the tested source +inputs, copied test artifact, exact selections, logs and backend cleanup. This is +a physical/receipt recovery checkpoint, not a full CG-06 DONE or release record. +Earlier failed fixture builds, capture-admission refusal, extent-schema mismatch +and cache-admission failure remain recorded beside the corrected passing run. +The catalog golden now verifies the required recovery-origin wrapper and refuses +the older bare birth record; no compatibility reader was added. + +The remaining feature gate is actual Docker fencing. The current native and +RustFS drills use an explicit contract-test authority; they do not qualify +permanent infrastructure launch/restart revocation. The existing Docker harness +uses the older per-cell runtime. Replace its voter path with actual shared-group +nodes, retain its ownership checks/signing flow, revoke the old deployment in a +durable sole-launcher policy, erase all owned voter containers and volumes, then +cold-recover through a fresh quorum. Reopening the launcher must still refuse +the old deployment. Complete that drill and the original CG-06 gate before DONE. + +Historical CG-01–05 raw gate directories are currently unavailable in this +checkout. Their prior source-scoped results remain historical; complete package +validation requires restoring those records or rerunning the unchanged exact +gates. This checkpoint does not replace them or weaken the validator. The 37 validator +unit tests pass and all 24 reviewed source anchors match; complete validation +currently refuses the missing CG-01 record. +After CG-06, prioritize CG-08 bounded due-work scheduling and qualification of +the current close-on-settle SQL policy, then CG-07 public SDK/fleet cutover. + ## CG-06 fresh-history data import and pending admission Source: 7052d218d60181d7b0bb5a4c12686e80e294a61c. CG-06 remains IN_PROGRESS. A fenced, From 131b4e6e630efa38746e518514255d88d37f8041 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 19:39:38 -0700 Subject: [PATCH 254/568] test: add shared-group Docker recovery drill --- crates/cellar/Cargo.toml | 4 + tools/multi-raft/Dockerfile | 31 ++ tools/multi-raft/README.md | 45 +++ tools/multi-raft/group_disaster.py | 487 ++++++++++++++++++++++ tools/multi-raft/node/config.rs | 196 +++++++++ tools/multi-raft/node/main.rs | 63 +++ tools/multi-raft/node/physical.rs | 174 ++++++++ tools/multi-raft/node/recovery.rs | 224 +++++++++++ tools/multi-raft/node/service.rs | 514 ++++++++++++++++++++++++ tools/multi-raft/test_group_disaster.py | 57 +++ 10 files changed, 1795 insertions(+) create mode 100644 tools/multi-raft/Dockerfile create mode 100644 tools/multi-raft/group_disaster.py create mode 100644 tools/multi-raft/node/config.rs create mode 100644 tools/multi-raft/node/main.rs create mode 100644 tools/multi-raft/node/physical.rs create mode 100644 tools/multi-raft/node/recovery.rs create mode 100644 tools/multi-raft/node/service.rs create mode 100644 tools/multi-raft/test_group_disaster.py diff --git a/crates/cellar/Cargo.toml b/crates/cellar/Cargo.toml index 7607a0726..aecee43af 100644 --- a/crates/cellar/Cargo.toml +++ b/crates/cellar/Cargo.toml @@ -79,6 +79,10 @@ path = "../../tools/perf/raft_node.rs" name = "cellar-perf-baseline" path = "../../tools/perf/baseline_node.rs" +[[bin]] +name = "cellar-perf-cell-group" +path = "../../tools/multi-raft/node/main.rs" + [[bin]] name = "cellar-perf-storage" path = "../../tools/perf/storage_operations.rs" diff --git a/tools/multi-raft/Dockerfile b/tools/multi-raft/Dockerfile new file mode 100644 index 000000000..9bdca6c94 --- /dev/null +++ b/tools/multi-raft/Dockerfile @@ -0,0 +1,31 @@ +# syntax=docker/dockerfile:1 +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +FROM rust:1.98.1-bookworm AS build +ARG BUILD_JOBS=1 +ARG TARGET_CACHE_ID=cellar-3617-cell-group-dr-target +WORKDIR /src +COPY Cargo.toml Cargo.lock ./ +COPY crates ./crates +COPY vendor ./vendor +COPY examples ./examples +COPY tools ./tools +# Archived sources can predate cached artifacts. Recheck the exact snapshot. +RUN --mount=type=cache,id=cellar-3617-cell-group-registry,target=/usr/local/cargo/registry \ + --mount=type=cache,id=${TARGET_CACHE_ID},target=/src/target \ + find Cargo.toml Cargo.lock crates examples tools vendor -type f -exec touch {} + && \ + CARGO_PROFILE_DEV_DEBUG=0 cargo build -j ${BUILD_JOBS} --locked -p cellar --bin cellar-perf-cell-group && \ + mkdir /out && cp target/debug/cellar-perf-cell-group /out/ +# Match source_manifest(): include build recipes and embedded non-Rust inputs. +RUN find Cargo.toml Cargo.lock crates vendor examples tools \ + \( -name .git -o -name .cellar -o -name node_modules -o -name __pycache__ \) -prune -o \ + \( -type f -o -type l \) ! -name '*.pyc' ! -name .DS_Store -print0 | \ + LC_ALL=C sort -z | xargs -0 sha256sum > /out/raft-source.sha256 +FROM debian:bookworm-slim +ARG SOURCE_REVISION=unspecified +LABEL org.opencontainers.image.revision=${SOURCE_REVISION} +RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/* +COPY --from=build /out/ /usr/local/bin/ +COPY LICENSE NOTICE /usr/share/doc/cellar/ +WORKDIR /data +ENTRYPOINT ["/usr/local/bin/cellar-perf-cell-group"] diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index d766c3ff0..04a6126a6 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -159,3 +159,48 @@ Fenced new-history disaster restore needs its own verified deployment fence. Stopping containers alone does not prevent an old quorum from returning. Release/durability qualification, performance measurement and federation remain separate gates in the [roadmap](../../docs/design/multi-raft/implementation-plan.md). + +## Shared-group all-voter-disk-loss drill + +`group_disaster.py` is the CG-06 Docker qualification runner. It launches the +`cellar-perf-cell-group` executable with three actual shared-group voters and +three independently versioned SQLite cells in one group. It uses real RustFS, +TLS replication, S3-gated commands, physical image and receipt recovery, new +incarnations, committed activation, fresh writes and retained-disk restart. +The runner checks every recovered voter's final values and cell versions. + +Commit the sources before building. Use the dedicated profile described above, +a checkout-specific build cache and the immutable image ID produced by Docker: + +```sh +docker --context colima-cellar-multi-raft build -f tools/multi-raft/Dockerfile \ + --build-arg TARGET_CACHE_ID=cellar-my-checkout-group-dr-target \ + --build-arg SOURCE_REVISION="$(git rev-parse HEAD)" \ + --iidfile /tmp/cellar-my-checkout-group-image-id . +python3 tools/multi-raft/group_disaster.py \ + --context colima-cellar-multi-raft \ + --image "$(cat /tmp/cellar-my-checkout-group-image-id)" \ + --mode materialized --output /tmp/cellar-group-disaster-materialized-001 +``` + +Repeat with `--mode extents` and a fresh output directory. The image embeds the +complete build-input manifest; the runner refuses a source mismatch. This uses +the unoptimized, debug-info-free Cargo development profile for correctness +qualification. It establishes no throughput or recovery-time claim. + +The controller owns the complete disposable deployment and is its sole launcher. +It persists permanent generation revocation under a launch lock, verifies and +removes the exact owned old containers and voter volumes, then signs the exact +recovery request. Its private signing key and launch ledger never enter a voter +or operator container. Reloading the controller cannot authorize the revoked +generation; an unknown launch outcome prevents certification. Production needs +an equivalent trusted infrastructure fencing authority that revokes every old +writer's launch capability. Stopping processes alone does not supply that proof. + +Only this run's three old voter disks are deleted. The RustFS object volume, +new voter volumes and evidence remain. The runner stops all remaining owned +containers on exit, including failure, and verifies normal joined shutdown. +The fresh private output directory contains credentials, opaque tokens, command +ledgers, signing material and sanitized process logs; keep it out of Git and +publish only deliberately sanitized reports. A successful drill is an exact +local recovery result, separate from the complete CG-06 task and release gates. diff --git a/tools/multi-raft/group_disaster.py b/tools/multi-raft/group_disaster.py new file mode 100644 index 000000000..4ce489bfa --- /dev/null +++ b/tools/multi-raft/group_disaster.py @@ -0,0 +1,487 @@ +#!/usr/bin/env python3 +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Owned three-container shared-group/RustFS cold recovery, with permanent fencing. + +The controller is the sole launcher. Its durable revocation ledger and private +Ed25519 key are never mounted in voters/operators. Source volumes alone are +removed after complete inventory checks; new volumes and sanitized evidence stay. +""" +import argparse +from contextlib import contextmanager +import fcntl +import ipaddress +import json +import os +from pathlib import Path +import re +import secrets +import subprocess +import sys +import time + +ROOT = Path(__file__).resolve().parents[2] +sys.path.insert(0, str(ROOT / 'tools/perf')) +from raft_disaster import LABEL, command, fence_and_delete, owned_container +from raft_run import input_digest, parse_manifest, request, require, source_manifest +from run import create_bucket + +RUSTFS = 'ghcr.io/rustfs/rustfs@sha256:bffcab0c9d647aab0055d1c69d340b202d0909966b385932d4ead1aeb7602858' +PURPOSE = b'cellar-docker-cell-group-fence-1\0' + + +def save(path, value): + temporary = path.with_name(path.name + '.' + secrets.token_hex(8) + '.pending') + with open(temporary, 'x', opener=lambda p, f: os.open(p, f, 0o600)) as stream: + json.dump(value, stream, indent=2) + stream.write('\n') + stream.flush() + os.fsync(stream.fileno()) + temporary.replace(path) + descriptor = os.open(path.parent, os.O_RDONLY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +class Launcher: + """Reloaded for each admission; revocation has no reverse operation. + + Holding the lock through launch resolution prevents a launch from passing + admission before revocation and materializing after the fence inventory. + """ + def __init__(self, root, run_id): + self.root, self.run_id = Path(root), run_id + self.path = self.root / 'launcher.json' + with self.lock(): + if not self.path.exists(): + save(self.path, {'format': 1, 'run_id': run_id, 'revoked': [], 'nodes': []}) + self.read() + + @contextmanager + def lock(self): + with open(self.root / 'launcher.lock', 'a+') as lock: + fcntl.flock(lock, fcntl.LOCK_EX) + try: + yield + finally: + fcntl.flock(lock, fcntl.LOCK_UN) + + def read(self): + value = json.loads(self.path.read_text()) + require(value['format'] == 1 and value['run_id'] == self.run_id, 'foreign launcher ledger') + return value + + def launch(self, generation, name, volume, invoke): + with self.lock(): + value = self.read() + require(generation not in value['revoked'], 'deployment permanently revoked') + require(all(row['name'] != name for row in value['nodes']), 'duplicate launch identity') + row = {'generation': generation, 'name': name, 'volume': volume, 'container_id': None} + value['nodes'].append(row) + save(self.path, value) + identifier = invoke() + require(re.fullmatch('[0-9a-f]{64}', identifier) is not None, 'unknown launch outcome') + row['container_id'] = identifier + save(self.path, value) + return identifier + + def revoke(self, generation): + with self.lock(): + value = self.read() + nodes = [row for row in value['nodes'] if row['generation'] == generation] + require(len(nodes) == 3 and len({row['container_id'] for row in nodes}) == 3 + and all(row['container_id'] for row in nodes), 'incomplete writer inventory') + if generation not in value['revoked']: + value['revoked'].append(generation) + save(self.path, value) + return [(row['container_id'], row['volume']) for row in nodes] + + +def verify_image(image, docker): + require(re.fullmatch('sha256:[0-9a-f]{64}', image), 'immutable Docker image ID required') + actual = parse_manifest(docker('run', '--rm', '--network', 'none', '--entrypoint', 'cat', image, '/usr/local/bin/raft-source.sha256')) + require(actual == source_manifest(), 'stale compiled image source manifest') + binary = docker('run', '--rm', '--network', 'none', '--entrypoint', 'sha256sum', image, '/usr/local/bin/cellar-perf-cell-group').split()[0] + return {'image': image, 'binary_sha256': binary, 'source_manifest_sha256': input_digest(actual), 'profile': 'dev-debug0', 'source_revision': subprocess.check_output(['git', 'rev-parse', 'HEAD'], cwd=ROOT, text=True).strip()} + + +class Qualification: + def __init__(self, args): + self.args = args + self.root = args.output.resolve() + self.root.mkdir(mode=0o700, parents=True, exist_ok=False) + self.config_dir = self.root / 'config' + self.config_dir.mkdir(mode=0o700) + self.run_id = secrets.token_hex(12) + self.prefix = 'cellar-cg-dr-' + self.run_id + self.owner = LABEL + '=' + self.run_id + self.launcher = Launcher(self.root, self.run_id) + self.containers, self.targets = [], [] + self.config = {'rooted': args.mode == 'extents', 'endpoint': 'http://rustfs:9000', + 'bucket': 'cellar-cg-dr', 'prefix': self.prefix, 'access': secrets.token_hex(10), + 'secret': secrets.token_hex(32), 'token': secrets.token_hex(32), + 'peers': [[{'host': '127.0.0.1', 'disk': list(secrets.token_bytes(16)), 'process': list(secrets.token_bytes(16))} for _ in range(3)] for _ in range(2)], + 'fence_public': [], 'certificate': [], 'recovery': []} + self.report = {'format': 1, 'run_id': self.run_id, 'mode': args.mode, 'completed': False, + 'group_count': 1, 'cells': ['alpha', 'beta', 'gamma'], 'voters': 3, 'checks': {}, + 'limitations': ['Single local VM, not independent failure domains or Amazon S3.', + 'Trusted sole-launcher Docker controller; production requires an equivalent fencing authority.', + 'No external effects, managed recovery, sustained-load or recovery-time SLA qualification.']} + self.ledger = {'commands': [], 'observations': {}} + + def docker(self, *args): + return command(['docker', '--context', self.args.context, *args]).stdout.strip() + + def save(self): + save(self.root / 'report.json', self.report) + save(self.config_dir / 'config.json', self.config) + save(self.config_dir / 'ledger.json', self.ledger) + + def setup(self): + require(self.args.context.startswith('colima-cellar-multi-raft'), 'dedicated qualification context required') + self.report['build'] = verify_image(self.args.image, self.docker) + info = json.loads(self.docker('info', '--format', '{{json .}}')) + require(info['OSType'] == 'linux' and info['NCPU'] >= 5 and info['MemTotal'] >= 12 * 1024**3, + 'insufficient isolated Docker resources') + self.docker('network', 'create', '--label', self.owner, self.prefix) + network = json.loads(self.docker('network', 'inspect', self.prefix))[0] + subnet = ipaddress.ip_network(network['IPAM']['Config'][0]['Subnet']) + require(subnet.version == 4 and subnet.prefixlen <= 24, 'private IPv4 network required') + prefix = '.'.join(str(subnet.network_address).split('.')[:3]) + for peers in self.config['peers']: + for n, peer in enumerate(peers): + peer['host'] = prefix + '.' + str(11+n) + self.key = self.root / 'controller-private.pem' + command(['openssl', 'genpkey', '-algorithm', 'ED25519', '-out', str(self.key)]) + self.key.chmod(0o600) + public = self.root / 'controller-public.der' + command(['openssl', 'pkey', '-in', str(self.key), '-pubout', '-outform', 'DER', '-out', str(public)]) + data = public.read_bytes() + require(len(data) == 44 and data[:12].hex() == '302a300506032b6570032100', 'unexpected public key encoding') + self.config['fence_public'] = list(data[12:]) + self.save() + self.docker('pull', RUSTFS) + volume = self.prefix + '-objects' + self.docker('volume', 'create', '--label', self.owner, volume) + # RustFS's pinned image runs as UID 10001. Use the measurement image's + # root user for this one owned volume, without any backend credentials. + self.docker('run', '--rm', '--network', 'none', '--label', self.owner, '--entrypoint', 'chown', + '-v', volume + ':/data', self.args.image, '10001:10001', '/data') + identifier = self.docker('run', '-d', '--label', self.owner, '--name', self.prefix + '-rustfs', '--network', self.prefix, '--network-alias', 'rustfs', + '--cpus', '1', '--memory', '2g', '--memory-swap', '2g', '-p', f'127.0.0.1:{self.args.port_base}:9000', '-v', volume + ':/data', + '-e', 'RUSTFS_ACCESS_KEY=' + self.config['access'], '-e', 'RUSTFS_SECRET_KEY=' + self.config['secret'], '-e', 'RUSTFS_CONSOLE_ENABLE=false', RUSTFS, '/data') + self.containers.append(identifier) + store = owned_container(self.docker, identifier, self.run_id) + self.capture_resources(store, 2) + self.report['rustfs'] = {'container_id':identifier, 'image':store['Image'], 'pinned_image':RUSTFS} + self.until(lambda: self.bucket(), 90) + self.operator('init') + + def bucket(self): + create_bucket(f'http://127.0.0.1:{self.args.port_base}', self.config['bucket'], self.config['access'], self.config['secret']) + return True + + @staticmethod + def until(operation, seconds=90): + deadline = time.monotonic() + seconds + last = None + while time.monotonic() < deadline: + try: + value = operation() + if value: + return value + except (OSError, ValueError, RuntimeError) as error: + last = type(error).__name__ + time.sleep(.2) + raise RuntimeError('bounded qualification wait expired: ' + str(last)) + + def operator(self, action, node=0, *, volume=None, generation=0, check=True): + args = ['run', '--rm', '--label', self.owner, '--network', self.prefix, + '-v', str(self.config_dir) + ':/config:ro', '-e', 'CELLAR_GROUP_ACTION=' + action, + '-e', 'CELLAR_GROUP_NODE=' + str(node), '-e', 'CELLAR_GROUP_GENERATION=' + str(generation)] + if volume: + args += ['-v', volume + ':/data'] + result = command(['docker', '--context', self.args.context, *args, self.args.image], check=False) + text = self.redact(result.stdout + result.stderr) + with (self.root / 'operator.log').open('a') as stream: + stream.write(action + ': exit ' + str(result.returncode) + '\n' + text + '\n') + if check: + require(result.returncode == 0, 'operator failed; inspect private sanitized operator log') + return result, text + + def launch(self, node, generation, volume=None): + name = f'{self.prefix}-g{generation}-node{node}' + volume = volume or name + if volume == name: + self.docker('volume', 'create', '--label', self.owner, volume) + def invoke(): + identifier = self.docker('run', '-d', '--label', self.owner, '--restart=no', '--name', name, + '--network', self.prefix, '--ip', self.config['peers'][generation][node]['host'], + '--cpus', '1', '--memory', '3g', '--memory-swap', '3g', '-p', f'127.0.0.1:{self.args.port_base+node+1}:{8080+node}', + '-v', volume + ':/data', '-v', str(self.config_dir) + ':/config:ro', + '-e', 'CELLAR_GROUP_NODE=' + str(node), '-e', 'CELLAR_GROUP_GENERATION=' + str(generation), + '-e', 'CELLAR_GROUP_RETAINED=' + str(generation == 1).lower(), self.args.image) + self.containers.append(identifier) + captured = owned_container(self.docker, identifier, self.run_id) + self.capture_resources(captured, 3) + self.save() + return identifier + return self.launcher.launch(generation, name, volume, invoke), volume + + def capture_resources(self, container, gib): + limits = container['HostConfig'] + require(limits['NanoCpus'] == 1_000_000_000 and limits['Memory'] == gib*1024**3 + and limits['MemorySwap'] == gib*1024**3, 'actual container resource limits differ') + self.report.setdefault('resources', []).append({'container_id':container['Id'], + 'nano_cpus':limits['NanoCpus'], 'memory':limits['Memory'], 'memory_swap':limits['MemorySwap']}) + + def call(self, node, action): + result = request(f'http://127.0.0.1:{self.args.port_base+node+1}', '/control', self.config['token'], method='POST', body=action, timeout=25) + with (self.root / 'http.jsonl').open('a') as stream: + stream.write(json.dumps({'node': node, 'request': action, 'status': result[0], 'response': result[1]})+'\n') + return result + + def leader(self, generation, ready=True): + for node in range(3): + try: + status, value = self.call(node, {'action': 'status'}) + if status == 200 and value['leader'] == node + (1 if generation == 0 else 11) and (not ready or value['ready']): + return node + 1 + except (OSError, ValueError): + pass + return None + + def ready(self, generation): + return self.until(lambda: self.leader(generation)) - 1 + + def ok(self, action, generation): + def attempt(): + leader = self.leader(generation) + if leader is None: + return None + code, value = self.call(leader-1, action) + return value if code == 200 else None + return self.until(attempt) + + def command(self, cell, sequence, generation, *, retry=None, s3=True, input_value=None): + body = {'action': 'command', 'cell': cell, 'sequence': sequence, 'input': input_value or [1], + 'retry_until_ms': retry or int(time.time()*1000)+300000, 's3': s3} + row = {'generation': generation, 'request': body, 'acknowledged': None} + self.ledger['commands'].append(row) + self.save() + response = self.ok(body, generation) + require(response['s3'] and response['minimum_s3'], 'stored S3 minimum was downgraded') + row['acknowledged'] = response + self.save() + return response + + def observations(self, generation, old_tokens=False): + values = {} + for cell in ('alpha', 'beta', 'gamma'): + action = {'action': 'query', 'cell': cell, 'token': None} + if old_tokens: + action['token'] = self.ledger['observations'][cell]['token'] + node = self.ready(generation) + code, reply = self.call(node, action) + require(code == 503 and 'WrongHistory' in reply['error'], 'old token not refused by fresh serving path') + else: + values[cell] = self.ok(action, generation) + return values + + def counts(self): + result = [self.call(node, {'action': 'status'}) for node in range(3)] + require(all(code == 200 for code, _ in result), 'all three voters must be live') + return {key: sum(value[key] for _, value in result) for key in ('commands', 'initializations')} + + def sign(self, digest): + message = self.root / 'fence-message' + message.write_bytes(PURPOSE + bytes.fromhex(digest)) + signature = self.root / 'fence-signature' + command(['openssl', 'pkeyutl', '-sign', '-inkey', str(self.key), '-rawin', '-in', str(message), '-out', str(signature)]) + data = signature.read_bytes() + require(len(data) == 64, 'unexpected Ed25519 signature length') + self.config['certificate'] = list(bytes.fromhex(digest) + data) + self.save() + + def drill(self): + old = [self.launch(n, 0) for n in range(3)] + self.ready(0) + self.ok({'action': 'archive'}, 0) + for cell in ('alpha', 'beta', 'gamma'): + require(self.ok({'action': 'enroll', 'cell': cell}, 0)['s3'], 'enrollment lacks S3 proof') + expiry = int(time.time()*1000)+1000 + self.command('alpha', 1, 0, retry=expiry) + source = self.command('alpha', 2, 0) + time.sleep(max(0, (expiry-int(time.time()*1000))/1000)+.05) + self.command('alpha', 3, 0) + beta = self.command('beta', 1, 0) + require(source['result']['value'] == 2 and beta['result']['value'] == 1, 'independent source counter oracle') + self.ledger['observations'] = self.observations(0) + require({k:v['value'] for k,v in self.ledger['observations'].items()} == {'alpha':3,'beta':1,'gamma':0}, 'source group state mismatch') + require(self.counts() == {'commands':4,'initializations':3}, 'source callbacks not exact') + self.save() + plan = self.ok({'action': 'plan'}, 0) + self.config['recovery'] = plan['recovery'] + self.save() + missing, text = self.operator('fence', check=False) + require(missing.returncode != 0 and 'fence certificate' in text, 'missing infrastructure fence accepted') + self.config['certificate'] = list(bytes.fromhex(plan['fence_digest']) + bytes(64)) + self.save() + forged, text = self.operator('fence', check=False) + require(forged.returncode != 0 and 'untrusted Docker controller fence' in text, 'forged infrastructure certificate accepted') + inventory = self.launcher.revoke(0) + require(inventory == old, 'durable launcher inventory differs') + for identifier, _ in old: + captured = command(['docker','--context',self.args.context,'logs',identifier],check=False) + (self.root / (identifier[:12]+'-before-erasure.log')).write_text(self.redact(captured.stdout+captured.stderr)) + self.report['fence'] = fence_and_delete(self.docker, self.run_id, inventory) + for identifier, _ in old: + result = command(['docker', '--context', self.args.context, 'start', identifier], check=False) + require(result.returncode != 0, 'old immutable container restarted') + # A newly opened controller must refuse before any Docker mutation. + reloaded = Launcher(self.root, self.run_id) + for node in range(3): + try: + reloaded.launch(0, 'forbidden-'+str(node), 'forbidden', lambda: (_ for _ in ()).throw(RuntimeError('launcher reached Docker'))) + except ValueError as error: + require(str(error) == 'deployment permanently revoked', 'wrong launch refusal') + else: + raise ValueError('revoked deployment launch accepted') + self.report['checks']['old_launch_after_controller_reload_refused'] = True + self.sign(plan['fence_digest']) + self.operator('fence') + bindings = [] + volumes = [] + for node in range(3): + volume = f'{self.prefix}-target-{node}' + self.docker('volume', 'create', '--label', self.owner, volume) + _, text = self.operator('seed', node, volume=volume, generation=1) + result = json.loads(text.strip().splitlines()[-1]) + require(result['seeded'] and result['cells'] == 3, 'incomplete physical seed') + bindings.append(result['binding']) + volumes.append(volume) + require(len(set(bindings)) == 1, 'voters seeded different roots') + self.operator('publish') + for node in range(3): + self.operator('probe-old', node) + self.report['checks']['recreated_old_disks_refused'] = True + for peer in self.config['peers'][1]: + peer['process'] = list(secrets.token_bytes(16)) + self.save() + self.targets = [self.launch(node, 1, volumes[node]) for node in range(3)] + self.until(lambda: self.leader(1, ready=False)) + for node in range(3): + code, _ = self.call(node, {'action':'command','cell':'alpha','sequence':4,'input':[1], 'retry_until_ms':int(time.time()*1000)+300000,'s3':False}) + require(code == 503, 'pending group admitted application work') + code, _ = self.call(node, {'action':'query','cell':'alpha','token':None}) + require(code == 503, 'pending group admitted a strong read') + code, _ = self.call(node, {'action':'query','cell':'alpha','token':None,'stale':True}) + require(code == 503, 'pending group admitted a stale read') + require(self.counts() == {'commands':0,'initializations':0}, 'pending group invoked callback') + def activate(): + node = self.leader(1, ready=False) + if not node: return False + code, value = self.call(node-1, {'action':'activate'}) + return code == 200 and value.get('activated') + self.until(activate) + self.ready(1) + self.operator('target-archive') + self.ok({'action':'archive'}, 1) + self.operator('verify-old-tokens') + self.observations(1, old_tokens=True) + restored = self.observations(1) + require({k:v['value'] for k,v in restored.items()} == {'alpha':3,'beta':1,'gamma':0}, 'cold group state mismatch') + require(all(v['version']==1 and v['incarnation']=='47'*16 for v in restored.values()), 'fresh cell histories not reset') + for cell, sequence, original in [('alpha',2,source),('beta',1,beta)]: + reply = self.command(cell, sequence, 1, s3=False) + require(reply['result'] == original['result'] and reply['version']==1, 'original receipt result was re-executed or retagged') + node = self.ready(1) + for sequence, input_value, expected in [(1,[1],'CommandExpired'),(2,[2],'CommandConflict')]: + code, value = self.call(node, {'action':'command','cell':'alpha','sequence':sequence,'input':input_value,'retry_until_ms':int(time.time()*1000)+300000,'s3':False}) + require(code == 503 and expected in value['error'], 'restored receipt floor/identity lost') + require(self.counts() == {'commands':0,'initializations':0}, 'recovery replayed callbacks') + fresh = self.command('alpha', 4, 1) + require(fresh['result']['value']==4 and fresh['result']['incarnation']=='47'*16 and fresh['version']==2, 'fresh write not in new history') + require(self.counts() == {'commands':1,'initializations':0}, 'fresh callback count mismatch') + # Retain disks but stop every process before giving the next processes + # new generation IDs. Removing containers is not needed for this restart. + self.docker('stop', '--time','60', *(identifier for identifier,_ in self.targets)) + for peer in self.config['peers'][1]: peer['process'] = list(secrets.token_bytes(16)) + self.save() + self.docker('start', *(identifier for identifier,_ in self.targets)) + self.ready(1) + self.ok({'action':'archive'},1) + for cell, sequence, original in [('alpha',2,source),('beta',1,beta)]: + restored_retry = self.command(cell,sequence,1,s3=False) + require(restored_retry['result']==original['result'] and restored_retry['version']==1, + 'imported receipt lost on retained restart') + retry = self.command('alpha',4,1,s3=False) + require(retry['result']==fresh['result'] and retry['version']==2, 'retained receipt changed') + require(self.counts()=={'commands':0,'initializations':0}, 'retained restart replayed callback') + after = self.command('alpha',5,1) + require(after['result']['value']==5 and after['version']==3, 'post-restart write failed') + self.verify_replicas({'alpha':5,'beta':1,'gamma':0}, {'alpha':3,'beta':1,'gamma':1}) + self.report['checks'].update(native_shared_group=True, all_source_disks_deleted=True, source_cut_restored=True, + fresh_quorum_activation=True, exact_receipts_no_replay=True, s3_minimums=True, client_floors=True, + old_tokens_refused=True, fresh_writes=True, retained_restart=True) + self.report['completed'] = True + self.save() + + def verify_replicas(self, expected_values, expected_versions): + leader = self.ready(1) + code, selected = self.call(leader, {'action':'status'}) + require(code == 200, 'selected leader unavailable') + for node in range(3): + self.until(lambda: self.call(node, {'action':'status'})[1]['applied'] >= selected['applied']) + for cell, expected in expected_values.items(): + code, reply = self.call(node, {'action':'query','cell':cell,'token':None,'stale':True}) + require(code == 200 and reply['value']==expected and reply['version']==expected_versions[cell] + and reply['incarnation']=='47'*16, 'replica state/history/version mismatch') + self.report['checks']['all_three_replica_images_verified'] = True + + def redact(self, value): + for key in ('access','secret','token'): + value = value.replace(self.config[key], '[REDACTED]') + return value + + def close(self): + states = [] + present = set(self.docker('ps','-aq','--no-trunc','--filter','label='+self.owner).split()) + for identifier in sorted(present): + value = owned_container(self.docker, identifier, self.run_id) + if value['State']['Running']: self.docker('stop','--time','60',identifier) + value = owned_container(self.docker, identifier, self.run_id) + result = command(['docker','--context',self.args.context,'logs',identifier],check=False) + (self.root / (identifier[:12]+'.log')).write_text(self.redact(result.stdout+result.stderr)) + states.append({'id':identifier,'running':value['State']['Running'],'exit_code':value['State']['ExitCode'],'oom_killed':value['State']['OOMKilled']}) + self.report['stopped_containers']=states + self.report['shutdown_verified']=bool(states) and all(not s['running'] and not s['oom_killed'] and s['exit_code']==0 for s in states) + if not self.report['shutdown_verified']: self.report['completed']=False + self.save() + + +def main(): + parser=argparse.ArgumentParser(description=__doc__) + parser.add_argument('--context',required=True) + parser.add_argument('--image',required=True) + parser.add_argument('--output',type=Path,required=True) + parser.add_argument('--mode',choices=['materialized','extents'],required=True) + parser.add_argument('--port-base',type=int,default=19800) + args=parser.parse_args() + require(1024<=args.port_base<65530,'invalid loopback port range') + os.umask(0o077) + q=Qualification(args) + try: + q.setup();q.drill() + except BaseException as error: + q.report['failure']=type(error).__name__+': '+q.redact(str(error));q.report['completed']=False + raise + finally: + q.close() + require(q.report['completed'] and q.report['shutdown_verified'],'incomplete drill') + print('PASS shared-group cold recovery after permanent Docker fencing ('+args.mode+')',flush=True) + + +if __name__=='__main__': main() diff --git a/tools/multi-raft/node/config.rs b/tools/multi-raft/node/config.rs new file mode 100644 index 000000000..090a44b5e --- /dev/null +++ b/tools/multi-raft/node/config.rs @@ -0,0 +1,196 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::ReplicaIdentity; +use object_store::{ + aws::{AmazonS3Builder, S3ConditionalPut}, + prefix::PrefixStore, + ObjectStore, +}; +use serde::{Deserialize, Serialize}; + +#[derive(Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Peer { + pub host: String, + pub disk: [u8; 16], + pub process: [u8; 16], +} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +pub(super) struct Config { + pub rooted: bool, + pub endpoint: String, + pub bucket: String, + pub prefix: String, + pub access: String, + pub secret: String, + pub token: String, + pub peers: [Vec; 2], + pub fence_public: [u8; 32], + pub certificate: Vec, + pub recovery: Vec, +} + +pub(super) struct Options { + pub config: PathBuf, + pub data: PathBuf, + pub action: String, + pub node: usize, + pub generation: usize, + pub retained: bool, +} + +impl Options { + pub fn load() -> Result { + let value = + |name: &str, fallback: &str| std::env::var(name).unwrap_or_else(|_| fallback.into()); + let options = Self { + config: value("CELLAR_GROUP_CONFIG", "/config/config.json").into(), + data: value("CELLAR_GROUP_DATA", "/data").into(), + action: value("CELLAR_GROUP_ACTION", "node"), + node: value("CELLAR_GROUP_NODE", "0").parse()?, + generation: value("CELLAR_GROUP_GENERATION", "0").parse()?, + retained: value("CELLAR_GROUP_RETAINED", "false").parse()?, + }; + ensure!( + options.node < 3 && options.generation < 2, + "invalid disposable voter identity" + ); + ensure!( + options.config.is_absolute() && options.data.is_absolute(), + "absolute private paths required" + ); + Ok(options) + } +} + +impl Config { + pub fn load(path: &std::path::Path) -> Result { + let bytes = std::fs::read(path)?; + ensure!(bytes.len() <= 256 * 1024, "config too large"); + let config: Self = serde_json::from_slice(&bytes)?; + ensure!( + config.endpoint == "http://rustfs:9000" + || config.endpoint.starts_with("http://127.0.0.1:"), + "isolated RustFS endpoint required" + ); + ensure!( + config.prefix.starts_with("cellar-cg-dr-") && config.token.len() == 64, + "isolated run required" + ); + for peers in &config.peers { + ensure!(peers.len() == 3, "three peers required"); + for peer in peers { + let _: std::net::Ipv4Addr = peer.host.parse()?; + } + } + Ok(config) + } + + pub fn store(&self) -> Result> { + let store = AmazonS3Builder::new() + .with_endpoint(&self.endpoint) + .with_bucket_name(&self.bucket) + .with_region("us-east-1") + .with_access_key_id(&self.access) + .with_secret_access_key(&self.secret) + .with_allow_http(true) + .with_conditional_put(S3ConditionalPut::ETagMatch) + .build()?; + Ok(Arc::new(PrefixStore::new(store, self.prefix.as_str()))) + } + + pub fn replica(&self, generation: usize, node: usize) -> Result> { + let peer = &self.peers[generation][node]; + Ok(ReplicaIdentity { + scope: history(generation)?, + node: node_id(generation, node)?, + member: member(generation, node)?, + disk: DiskLineage::new(peer.disk)?, + process: ProcessGeneration::new(peer.process)?, + }) + } + + pub async fn directory( + &self, + domain: &ExecutionDomain, + schema: [u8; 32], + ) -> Result { + let catalog = Catalog::new(self.store()?, Default::default())?; + let namespace = domain + .run(catalog.existing_namespace(namespace(schema)?)) + .await??; + Ok(RangeDirectory::new(namespace, Default::default())?) + } + + pub fn spec(&self, generation: usize) -> Result { + Ok(CellGroupBirthSpec { + history: history(generation)?, + range: CellRange::all(), + epoch: RouteEpoch::new(generation as u64 + 1)?, + creation: ProposalId::new([5 + generation as u8; 16])?, + voters: (0..3) + .map(|n| { + Ok(InitialMember { + member: member(generation, n)?, + node: node_id(generation, n)?, + }) + }) + .collect::>()?, + learners: vec![], + }) + } + + pub fn log_options(&self, options: &Options, mode: LogOpenMode) -> Result { + let identity = self.replica(options.generation, options.node)?; + Ok(ReplicaLogOptions::new( + options.data.join("log"), + identity.node, + identity.disk, + identity.process, + mode, + )) + } +} + +pub(super) fn namespace(schema: [u8; 32]) -> Result { + Ok(NamespaceConfig::new( + ClusterId::new([1; 16])?, + NamespaceId::new([2; 16])?, + NamespaceMode::Raft, + schema, + )) +} + +pub(super) fn history(generation: usize) -> Result { + Ok(GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([if generation == 0 { 3 } else { 70 }; 16])?, + incarnation: IncarnationId::new([if generation == 0 { 4 } else { 71 }; 16])?, + }) +} + +pub(super) fn node_id(generation: usize, node: usize) -> Result { + Ok(NodeId::new( + [node as u8 + if generation == 0 { 1 } else { 111 }; 16], + )?) +} + +pub(super) fn member(generation: usize, node: usize) -> Result { + Ok(MemberId::new( + node as u64 + if generation == 0 { 1 } else { 11 }, + )?) +} + +pub(super) fn cell(generation: usize, name: &str) -> Result { + let group = history(generation)?; + Ok(CellHistory { + cluster: group.cluster, + namespace: group.namespace, + cell: CellId::new(name)?, + incarnation: IncarnationId::new([if generation == 0 { 8 } else { 71 }; 16])?, + }) +} diff --git a/tools/multi-raft/node/main.rs b/tools/multi-raft/node/main.rs new file mode 100644 index 000000000..f8ebbb421 --- /dev/null +++ b/tools/multi-raft/node/main.rs @@ -0,0 +1,63 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Disposable shared-group voters/operator for the Docker disaster drill. +//! This is qualification infrastructure, not production deployment policy. +#![allow(clippy::disallowed_methods)] +mod config; +mod physical; +mod recovery; +mod service; + +use anyhow::{ensure, Context, Result}; +use cellar::application::consensus::{ + verification::{cell_groups::*, *}, + *, +}; +use cellar::sdk::ExecutionDomain; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + internal::{CacheLimits, NodeExtentStore, EXTENT_BYTES}, + stream_snapshot::{ScratchLimits, ScratchPool}, + DirectFileSystem, +}; +use config::{Config, Options}; +use serde_json::{json, Value}; +use std::{path::PathBuf, sync::Arc, time::Duration}; + +#[tokio::main(flavor = "multi_thread", worker_threads = 2)] +async fn main() -> Result<()> { + let options = Options::load()?; + let config = Arc::new(Config::load(&options.config)?); + let domain = Arc::new(ExecutionDomain::new(tokio::runtime::Handle::current())); + let result = domain + .run(async { + match options.action.as_str() { + "node" => service::run(domain.clone(), config.clone(), &options).await, + "seed" => recovery::seed(&domain, &config, &options).await, + action => recovery::operator(&domain, &config, &options, action).await, + } + }) + .await?; + domain.shutdown().await; + result +} + +fn output(value: Value) -> Result<()> { + cellar::cli_output::Output::new(cellar::cli_output::Format::Json) + .line(format_args!("{}", serde_json::to_string(&value)?)) +} + +fn bounds() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(20), + max_attempts: 4, + } +} + +fn keys() -> Result { + Ok(VersionKeys::new( + VersionSigningKey::new([89; 32], u64::MAX)?, + vec![], + Duration::from_secs(600), + )?) +} diff --git a/tools/multi-raft/node/physical.rs b/tools/multi-raft/node/physical.rs new file mode 100644 index 000000000..c2ce1ed3f --- /dev/null +++ b/tools/multi-raft/node/physical.rs @@ -0,0 +1,174 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +pub(super) struct Physical { + pub pages: ManifestPageStore, + pub images: CellGroupApplyImages, +} + +impl Physical { + pub fn open(base: &std::path::Path, rooted: bool) -> Result { + let filesystem: Arc = Arc::new(DirectFileSystem); + let pages = ManifestPageStore::open( + &base.join("pages"), + filesystem.clone(), + 1024, + 16 * 1024 * 1024, + 16, + )?; + let services = ReadServices { + now_unix_millis: Arc::new(cellar::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| bytes.fill(17)), + }; + let images = if rooted { + CellGroupApplyImages::Extents { + images: NodeExtentStore::open( + base.join("extents"), + filesystem, + Default::default(), + )?, + services, + limits: Default::default(), + cache_directory: base.join("apply-cache"), + cache_pool: scratch(1)?, + cache: cache(), + } + } else { + CellGroupApplyImages::Materialized { + images: GenerationStore::open(base.join("images"), filesystem, Default::default())?, + services, + } + }; + Ok(Self { pages, images }) + } + + pub fn archive( + self, + route: cellar::application::consensus::verification::cell_groups::GroupRoute, + ) -> Result { + Ok(CellGroupArchiveStore::new( + route, + self.pages, + self.images, + Default::default(), + Default::default(), + 100, + )?) + } + + pub fn recover( + self, + group: cellar::application::consensus::verification::GroupLog, + ) -> Result<(CellGroupApplier, PreparedGroupManifest)> { + Ok(CellGroupApplier::recover( + group, + self.pages, + self.images, + Default::default(), + Default::default(), + 100, + )?) + } + + pub fn initial( + mut self, + group: cellar::application::consensus::verification::GroupLog, + ) -> Result<(CellGroupApplier, PreparedGroupManifest)> { + let marker = group.group_manifest()?.context("initial manifest")?; + let tree = ManifestTree::new(marker.route().clone(), Default::default())?; + let root = tree.prepare_complete( + &group, + &mut self.pages, + marker.root(), + marker.position(), + marker.membership_revision(), + |_| Err(ManifestError::Invalid("source genesis is not empty")), + )?; + let applier = CellGroupApplier::new( + group, + self.pages, + self.images, + Default::default(), + Default::default(), + 100, + )?; + Ok((applier, root)) + } +} + +pub(super) fn scratch(files: usize) -> Result { + Ok(ScratchPool::new(ScratchLimits { + max_bytes: files as u64 * EXTENT_BYTES, + max_files: files, + })?) +} + +fn cache() -> CacheLimits { + CacheLimits { + max_cached_bytes: EXTENT_BYTES, + max_cached_extents: 1, + } +} + +pub(super) fn sql_options( + images: CellGroupApplyImages, + base: PathBuf, + schema: [u8; 32], + source: bool, +) -> Result { + let host = cellar_ltx::LtxHost::new( + cellar::asyncrt::wall_ms, + |_| Ok(Duration::ZERO), + |path| async move { std::fs::read(path) }, + |job| async move { + job(); + Ok(()) + }, + ) + .with_filesystem(Arc::new(DirectFileSystem)); + let limits = ExecutionLimits { + envelope: EnvelopeLimits { + max_payload_bytes: 1024 * 1024, + ..Default::default() + }, + max_retained_receipts: if source { 2 } else { 4096 }, + ..Default::default() + }; + Ok(match images { + CellGroupApplyImages::Materialized { images, services } => { + CellSqlOptions::Materialized(PrivateOptions { + directory: base, + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + }) + } + CellGroupApplyImages::Extents { + images, services, .. + } => CellSqlOptions::Extents(CellSqlRootOptions { + host, + sqlite_vfs: None, + images, + query_services: services, + application_digest: schema, + limits, + cache_directory: base.with_extension("read-cache"), + cache_pool: scratch(2)?, + cache: cache(), + }), + }) +} + +pub(super) fn checkpoint_pool() -> Result { + Ok(CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + })?) +} diff --git a/tools/multi-raft/node/recovery.rs b/tools/multi-raft/node/recovery.rs new file mode 100644 index 000000000..e467770f9 --- /dev/null +++ b/tools/multi-raft/node/recovery.rs @@ -0,0 +1,224 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use futures_util::future::BoxFuture; + +pub(super) const PURPOSE: &[u8] = b"cellar-docker-cell-group-fence-1\0"; + +pub(super) struct SignedFence { + public: ed25519_dalek::VerifyingKey, + certificate: Vec, +} + +impl SignedFence { + pub fn new(config: &Config) -> Result { + Ok(Self { + public: ed25519_dalek::VerifyingKey::from_bytes(&config.fence_public)?, + certificate: config.certificate.clone(), + }) + } + + fn check(&self, request: &CellGroupDisasterFenceRequest, evidence: &[u8]) -> Result<()> { + ensure!( + evidence.len() == 96 && evidence[..32] == request.digest(), + "missing or foreign permanent fence certificate" + ); + let mut message = PURPOSE.to_vec(); + message.extend_from_slice(&request.digest()); + self.public + .verify_strict( + &message, + &ed25519_dalek::Signature::from_slice(&evidence[32..])?, + ) + .context("untrusted Docker controller fence") + } +} + +impl DisasterFenceAuthority for SignedFence { + fn fence<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + ) -> BoxFuture<'a, Result>> { + Box::pin(async move { + self.check(request, &self.certificate)?; + Ok(self.certificate.clone()) + }) + } + + fn verify<'a>( + &'a self, + request: &'a CellGroupDisasterFenceRequest, + evidence: &'a [u8], + ) -> BoxFuture<'a, Result<()>> { + Box::pin(async move { self.check(request, evidence) }) + } +} + +pub(super) async fn fenced( + domain: &ExecutionDomain, + config: &Config, + data: &std::path::Path, + directory: &RangeDirectory, +) -> Result { + let request = directory.decode_group_recovery(&config.recovery)?; + let route = request.source().route().clone(); + let rooted = config.rooted; + let base = data.join("cold-source-verifier"); + let store = domain + .blocking(move || physical::Physical::open(&base, rooted)?.archive(route))? + .await??; + Ok(directory + .fence_group_recovery( + domain, + &CellGroupArchivePool::new(Default::default())?, + &CellGroupArchiveTailPool::new(Default::default())?, + &request, + physical::checkpoint_pool()?.receive_archive( + store, + physical::scratch(8)?, + data.into(), + )?, + &SignedFence::new(config)?, + ) + .await?) +} + +pub(super) async fn seed( + domain: &ExecutionDomain, + config: &Config, + options: &Options, +) -> Result<()> { + ensure!( + options.generation == 1 && !options.retained, + "fresh target seed required" + ); + let app = service::application()?; + let directory = config.directory(domain, app.class.schema_digest()).await?; + let proof = Box::pin(fenced(domain, config, &options.data, &directory)).await?; + let route = proof.request().replacement().route().clone(); + let base = options.data.join("state"); + let rooted = config.rooted; + let destination = domain + .blocking(move || physical::Physical::open(&base, rooted)?.archive(route))? + .await??; + let prepared = CellGroupRecoveryPool::new(Default::default())? + .prepare( + domain, + proof, + destination, + physical::scratch(8)?, + options.data.clone(), + ) + .await?; + let log = ReplicaLog::open( + config.log_options(options, LogOpenMode::Create)?, + Arc::new(DirectFileSystem), + ) + .await?; + let seeded = directory + .seed_group_recovery(domain, prepared, &log, &SignedFence::new(config)?) + .await?; + let value = json!({"seeded":true,"member":seeded.group().identity().member.get(),"binding":hex::encode(seeded.binding()),"cells":seeded.manifest().pinned_images()}); + drop(seeded); + log.close().await?; + output(value) +} + +pub(super) async fn operator( + domain: &ExecutionDomain, + config: &Config, + options: &Options, + action: &str, +) -> Result<()> { + let app = service::application()?; + let schema = app.class.schema_digest(); + if action == "init" { + let catalog = Catalog::new(config.store()?, Default::default())?; + let namespace = catalog.open_namespace(config::namespace(schema)?).await?; + let directory = RangeDirectory::new(namespace, Default::default())?; + let birth = directory.prepare_group_birth(config.spec(0)?)?; + directory.reserve_group_birth(&birth).await?; + let mutation = directory + .prepare_initialization(ProposalId::new([7; 16])?, &[birth.route().clone()])?; + directory.publish(&mutation).await?; + let birth = directory.materialize_group_birth(&birth).await?; + directory + .prepare_group_archive_binding( + domain, + &CellGroupArchivePool::new(Default::default())?, + &birth, + ) + .await? + .commit(domain) + .await?; + return output(json!({"initialized":true})); + } + let directory = config.directory(domain, schema).await?; + if action == "probe-old" { + let log = ReplicaLog::open( + config.log_options(options, LogOpenMode::Create)?, + Arc::new(DirectFileSystem), + ) + .await?; + let birth = directory.prepare_group_birth(config.spec(0)?)?; + let refused = matches!( + directory.materialize_group_birth(&birth).await, + Err(CatalogError::Fenced) + ); + log.close().await?; + ensure!( + refused, + "old birth was not fenced on a recreated empty disk" + ); + return output(json!({"old_identity_refused":true})); + } + if action == "fence" { + let proof = Box::pin(fenced(domain, config, &options.data, &directory)).await?; + return output(json!({"fenced":true,"request":hex::encode(proof.request().digest())})); + } + let request = directory.decode_group_recovery(&config.recovery)?; + let authority = SignedFence::new(config)?; + match action { + "publish" => { + directory + .publish_group_recovery(&request, &authority) + .await?; + output(json!({"published":true})) + } + "target-archive" => { + let birth = directory + .materialize_selected_group(request.replacement().route()) + .await?; + directory + .prepare_group_archive_binding( + domain, + &CellGroupArchivePool::new(Default::default())?, + &birth, + ) + .await? + .commit(domain) + .await?; + output(json!({"initialized":true})) + } + "verify-old-tokens" => { + let ledger: Value = serde_json::from_slice(&std::fs::read( + options + .config + .parent() + .context("config parent")? + .join("ledger.json"), + )?)?; + let now = cellar::asyncrt::wall_ms().max(0) as u64; + for name in ["alpha", "beta", "gamma"] { + let token = CellVersionToken::from_encoded( + ledger["observations"][name]["token"] + .as_str() + .context("source token")?, + )?; + keys()?.verify_cell(&token, &config::cell(0, name)?, schema, now)?; + } + output(json!({"old_tokens_valid":true})) + } + _ => anyhow::bail!("unknown operator action"), + } +} diff --git a/tools/multi-raft/node/service.rs b/tools/multi-raft/node/service.rs new file mode 100644 index 000000000..f74f5cf70 --- /dev/null +++ b/tools/multi-raft/node/service.rs @@ -0,0 +1,514 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use axum::{ + extract::{DefaultBodyLimit, State}, + http::{HeaderMap, StatusCode}, + response::IntoResponse, + routing::{get, post}, + Json, Router, +}; +use cellar::peer_auth::PeerAuth; +use serde::Deserialize; +use std::sync::atomic::{AtomicUsize, Ordering}; +use tokio::net::TcpListener; + +pub(super) struct Application { + pub class: Arc, + commands: Arc, + initialized: Arc, +} + +pub(super) fn application() -> Result { + let commands = Arc::new(AtomicUsize::new(0)); + let initialized = Arc::new(AtomicUsize::new(0)); + let c = commands.clone(); + let i = initialized.clone(); + let class = Arc::new(RaftClass::new("DockerCounter", 1, [84;32])? + .initializer([87;32], move |_,tx| { i.fetch_add(1,Ordering::AcqRel); tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(0); CREATE TABLE padding(bytes); INSERT INTO padding VALUES(zeroblob(8192));")?; Ok(()) })? + .command("Add",[85;32],move |context,input| { + c.fetch_add(1,Ordering::AcqRel); + ensure!(input == [1], "bounded increment input"); + context.sql().execute("UPDATE value SET n=n+1",[])?; + let value:i64=context.sql().query_row("SELECT n FROM value",[],|r|r.get(0))?; + Ok(serde_json::to_vec(&json!({"cell":context.identity().cell.as_str(),"incarnation":hex::encode(context.identity().incarnation.bytes()),"command":context.command().sequence(),"value":value}))?) + })? + .query("Value",[86;32],|context,_| { let value:i64=context.query_row("SELECT n FROM value",[],|r|r.get(0))?; Ok(serde_json::to_vec(&value)?) })?); + Ok(Application { + class, + commands, + initialized, + }) +} + +struct Node { + domain: Arc, + config: Arc, + generation: usize, + ordinal: usize, + data: PathBuf, + app: Application, + directory: RangeDirectory, + birth: CellGroupBirth, + log: ReplicaLog, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + controller: CellGroupController, + images: CellGroupApplyImages, +} + +#[derive(Deserialize)] +#[serde(tag = "action", rename_all = "snake_case", deny_unknown_fields)] +enum Action { + Status, + Archive, + Activate, + Plan, + Enroll { + cell: String, + }, + Command { + cell: String, + sequence: u64, + input: Vec, + retry_until_ms: u64, + s3: bool, + }, + Query { + cell: String, + token: Option, + #[serde(default)] + stale: bool, + }, +} + +pub(super) async fn run( + domain: Arc, + config: Arc, + options: &Options, +) -> Result<()> { + let app = application()?; + let directory = config.directory(&domain, app.class.schema_digest()).await?; + let request = directory.prepare_group_birth(config.spec(options.generation)?)?; + let birth = directory.materialize_group_birth(&request).await?; + let mode = if options.retained { + LogOpenMode::Resume + } else { + LogOpenMode::Create + }; + let log = ReplicaLog::open( + config.log_options(options, mode)?, + Arc::new(DirectFileSystem), + ) + .await?; + let permit = if options.generation == 0 { + directory.bootstrap_group_member(&birth, &log).await? + } else { + directory + .bootstrap_recovered_group_member( + &directory.decode_group_recovery(&config.recovery)?, + &log, + Arc::new(recovery::SignedFence::new(&config)?), + ) + .await? + }; + let group = permit.group().clone(); + let base = options.data.join("state"); + let rooted = config.rooted; + let retained = options.retained; + let (applier, root, images) = domain + .blocking(move || -> Result<_> { + let physical = physical::Physical::open(&base, rooted)?; + let images = physical.images.clone(); + let (applier, root) = if retained { + physical.recover(group)? + } else { + physical.initial(group)? + }; + Ok((applier, root, images)) + })? + .await??; + let (runtime, events) = CellGroupRuntime::start( + log.clone(), + RaftRuntimeOptions { + tick_interval_ms: Some(100), + election_seed: Some([17 + options.node as u8; 32]), + ..Default::default() + }, + ) + .await?; + runtime.attach(permit, root).await?; + let n = options.node as u16; + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([17; 32], format!("node-{}", n + 1))?), + tls()?, + TcpListener::bind(("0.0.0.0", 7100 + n * 3)).await?, + TcpListener::bind(("0.0.0.0", 7101 + n * 3)).await?, + TcpListener::bind(("0.0.0.0", 7102 + n * 3)).await?, + Default::default(), + ) + .await?; + transport.install_physical_group(cellar::application::consensus::verification::GroupRoute { + scope: config::history(options.generation)?, + schema_digest: app.class.schema_digest(), + revision: 1, + peers: (0..3) + .map(|node| -> Result<_> { + let p = &config.peers[options.generation][node]; + let n = node as u16; + Ok(RaftPeer { + replica: config.replica(options.generation, node)?, + label: format!("node-{}", n + 1), + server_name: "localhost".into(), + control_address: format!("{}:{}", p.host, 7100 + n * 3).parse()?, + data_address: format!("{}:{}", p.host, 7101 + n * 3).parse()?, + query_address: format!("{}:{}", p.host, 7102 + n * 3).parse()?, + }) + }) + .collect::>()?, + })?; + let sql = CellGroupSqlPool::start( + &log, + CellSqlLimits { + workers: 1, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 128 * 1024 * 1024, + group_bytes: 64 * 1024 * 1024, + }, + )?; + let controller = CellGroupController::start( + runtime.clone(), + events, + transport, + sql.clone(), + vec![applier], + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + ) + .await?; + let node = Arc::new(Node { + domain: domain.clone(), + config, + generation: options.generation, + ordinal: options.node, + data: options.data.clone(), + app, + directory, + birth, + log, + runtime, + sql, + controller, + images, + }); + let router = Router::new() + .route("/health", get(|| async { Json(json!({"live":true})) })) + .route("/control", post(control)) + .layer(DefaultBodyLimit::max(64 * 1024)) + .with_state(node.clone()); + let listener = TcpListener::bind(("0.0.0.0", 8080 + n)).await?; + axum::serve(listener, router) + .with_graceful_shutdown(async { + #[cfg(unix)] + { + if let Ok(mut signal) = + tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()) + { + signal.recv().await; + } + } + #[cfg(not(unix))] + { + let _ = tokio::signal::ctrl_c().await; + } + }) + .await?; + node.controller.close().await?; + node.log.close().await?; + Ok(()) +} + +async fn control( + State(node): State>, + headers: HeaderMap, + Json(action): Json, +) -> impl IntoResponse { + if headers.get("authorization").and_then(|v| v.to_str().ok()) + != Some(format!("Bearer {}", node.config.token).as_str()) + { + return ( + StatusCode::UNAUTHORIZED, + Json(json!({"error":"unauthorized"})), + ); + } + let result = node.domain.run(node.call(action)).await; + match result { + Ok(Ok(value)) => (StatusCode::OK, Json(value)), + Ok(Err(error)) => ( + StatusCode::SERVICE_UNAVAILABLE, + Json(json!({"error":format!("{error:#}")})), + ), + Err(_) => ( + StatusCode::SERVICE_UNAVAILABLE, + Json(json!({"error":"domain closed"})), + ), + } +} + +impl Node { + fn dispatcher(&self, s3: bool) -> Result { + let dispatcher = CellGroupDispatcher::new( + &self.domain, + self.directory.clone(), + self.app.class.clone(), + keys()?, + CellDispatchLimits { + max_cached_routes: 4, + route_cache_bytes: 256 * 1024, + max_invocations: 4, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + Ok(if s3 { + dispatcher.with_s3(&self.controller)? + } else { + dispatcher + }) + } + + fn options(&self, name: &str) -> Result { + ensure!( + ["alpha", "beta", "gamma"].contains(&name), + "unknown synthetic cell" + ); + physical::sql_options( + self.images.clone(), + self.data.join(format!("sql-{name}")), + self.app.class.schema_digest(), + self.generation == 0, + ) + } + + async fn call(&self, action: Action) -> Result { + let history = config::history(self.generation)?; + match action { + Action::Status => { + let s = self.runtime.status(history).await?; + Ok( + json!({"member":config::member(self.generation,self.ordinal)?.get(),"leader":s.leader,"ready":s.ready,"term":s.term.get(),"applied":s.applied.index.get(),"commands":self.app.commands.load(Ordering::Acquire),"initializations":self.app.initialized.load(Ordering::Acquire)}), + ) + } + Action::Archive => self.archive().await, + Action::Activate => { + ensure!(self.generation == 1, "target activation only"); + let request = self + .directory + .decode_group_recovery(&self.config.recovery)?; + let authority = recovery::SignedFence::new(&self.config)?; + let permit = self + .directory + .prepare_group_recovery_activation(&request, &self.log, &authority) + .await?; + match self.runtime.activate_recovery(permit).await? { + CellGroupRecoveryActivation::Accepted(p) => { + Ok(json!({"activated":false,"accepted":p.position.index.get()})) + } + CellGroupRecoveryActivation::Applied(ack) => { + self.directory + .complete_group_recovery(&request, &ack, &authority) + .await?; + Ok(json!({"activated":true})) + } + } + } + Action::Enroll { cell } => { + let options = self.options(&cell)?; + let invocation = self + .dispatcher(true)? + .enroll( + config::cell(self.generation, &cell)?, + CellEnrollmentRequest { + creation: ProposalId::new([100 + cell.as_bytes()[0]; 16])?, + durability: Durability::S3, + }, + bounds(), + None, + ) + .await?; + let result = self + .controller + .enroll(invocation, options)? + .response() + .await?; + Ok(json!({"enrolled":true,"s3":result.durability()==Durability::S3})) + } + Action::Command { + cell, + sequence, + input, + retry_until_ms, + s3, + } => { + let invocation = self + .dispatcher(true)? + .command( + config::cell(self.generation, &cell)?, + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, sequence)?, + contract: self.app.class.contract("Add").context("Add")?, + input, + durability: if s3 { + Durability::S3 + } else { + Durability::Quorum + }, + retry_until_ms, + }, + bounds(), + None, + ) + .await?; + let result = self + .controller + .command(invocation, self.options(&cell)?)? + .response() + .await?; + Ok( + json!({"result":serde_json::from_slice::(result.receipt().receipt.result())?,"s3":result.durability()==Durability::S3,"minimum_s3":result.receipt().receipt.minimum_durability()==Durability::S3,"version":result.receipt().version.sequence.get(),"token":result.token().as_str()}), + ) + } + Action::Query { cell, token, stale } => { + ensure!( + !stale || token.is_none(), + "stale inspection cannot carry an AtLeast token" + ); + let consistency = match token { + Some(token) => { + CellQueryConsistency::AtLeast(CellVersionToken::from_encoded(&token)?) + } + None if stale => CellQueryConsistency::Stale, + None => CellQueryConsistency::Linearizable, + }; + let mut invocation = self.dispatcher(false)?.query( + config::cell(self.generation, &cell)?, + CellQueryRequest { + invocation: ProposalId::new([180; 16])?, + contract: self.app.class.contract("Value").context("Value")?, + input: vec![], + consistency, + }, + bounds(), + None, + )?; + invocation.begin().await?; + let response = invocation + .execute(&self.runtime, &self.sql, self.options(&cell)?) + .await?; + Ok( + json!({"value":serde_json::from_slice::(response.result())?,"token":response.token().as_str(),"version":response.observation().version.sequence.get(),"incarnation":hex::encode(response.observation().version.history.incarnation.bytes())}), + ) + } + Action::Plan => self.plan().await, + } + } + + async fn archive(&self) -> Result { + let history = config::history(self.generation)?; + if self.controller.archive_status(history)?.is_some() { + return Ok(json!({"registered":true})); + } + let base = self.data.join("archive-verifier"); + let rooted = self.config.rooted; + let route = self.birth.route().clone(); + let store = self + .domain + .blocking(move || physical::Physical::open(&base, rooted)?.archive(route))? + .await??; + self.controller + .archive(CellGroupArchiveRegistration { + directory: self.directory.clone(), + birth: self.birth.clone(), + io: CellGroupArchivePool::new(Default::default())?, + owners: CellGroupArchiveUploadPool::new(Default::default())?, + tails: CellGroupArchiveTailPool::new(Default::default())?, + checkpoints: physical::checkpoint_pool()?, + store, + scratch: physical::scratch(8)?, + scratch_directory: self.data.clone(), + })? + .response() + .await?; + Ok(json!({"registered":true})) + } + + async fn plan(&self) -> Result { + ensure!(self.generation == 0, "source plan only"); + let path = self.data.join("log/cell-group-archive-journals"); + let hold = self + .domain + .blocking(move || -> Result { + let entries = std::fs::read_dir(path)?.collect::>>()?; + ensure!( + entries.len() == 1, + "exactly one group archive journal required" + ); + let value: Value = + serde_json::from_slice(&std::fs::read(entries[0].path().join("state"))?)?; + ensure!( + value["record"]["pending"] == "None" && value["record"]["retire"].is_null(), + "archive journal still advancing" + ); + Ok(serde_json::from_value(value["record"]["head"].clone())?) + })? + .await??; + ensure!( + hold.marker().position().index + >= self + .runtime + .status(config::history(0)?) + .await? + .applied + .index, + "archive behind selected cut" + ); + let target = self.directory.prepare_group_birth(self.config.spec(1)?)?; + let view = self.directory.observe().await?; + let request = self.directory.prepare_group_recovery( + &view, + ProposalId::new([73; 16])?, + &self.birth, + &target, + hold, + )?; + Ok(json!({"recovery":request.encode(),"fence_digest":hex::encode(request.digest())})) + } +} + +fn tls() -> Result { + Ok(RaftTls::new( + vec![ + include_bytes!("../../../crates/cellar/transport/tcp/fixtures/ca.der") + .to_vec() + .into(), + ], + vec![ + include_bytes!("../../../crates/cellar/transport/tcp/fixtures/server.der") + .to_vec() + .into(), + ], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../../crates/cellar/transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?) +} diff --git a/tools/multi-raft/test_group_disaster.py b/tools/multi-raft/test_group_disaster.py new file mode 100644 index 000000000..46dc970e1 --- /dev/null +++ b/tools/multi-raft/test_group_disaster.py @@ -0,0 +1,57 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""The trusted launcher's complete inventory and durable, irreversible revocation.""" +import json +from pathlib import Path +import tempfile +import unittest + +from group_disaster import Launcher + + +class LauncherTests(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.launcher = Launcher(self.root, 'owned') + + def populate(self): + return [(self.launcher.launch(0, f'node{n}', f'disk{n}', lambda n=n: str(n)*64), f'disk{n}') + for n in range(1, 4)] + + def test_reload_refuses_revoked_generation_before_invoking_docker(self): + old = self.populate() + self.assertEqual(self.launcher.revoke(0), old) + reloaded = Launcher(self.root, 'owned') + with self.assertRaisesRegex(ValueError, 'deployment permanently revoked'): + reloaded.launch(0, 'recreated', 'lost', lambda: self.fail('Docker called after revocation')) + self.assertEqual(reloaded.revoke(0), old) + self.assertEqual(reloaded.launch(1, 'fresh', 'new', lambda: 'a'*64), 'a'*64) + + def test_incomplete_inventory_cannot_certify_a_fence(self): + self.launcher.launch(0, 'only-one', 'disk', lambda: '1'*64) + with self.assertRaisesRegex(ValueError, 'incomplete writer inventory'): + self.launcher.revoke(0) + self.assertEqual(json.loads(self.launcher.path.read_text())['revoked'], []) + + def test_unknown_launch_outcome_prevents_fence(self): + self.populate() + with self.assertRaisesRegex(RuntimeError, 'unknown outcome'): + self.launcher.launch(0, 'uncertain', 'disk4', lambda: (_ for _ in ()).throw(RuntimeError('unknown outcome'))) + with self.assertRaisesRegex(ValueError, 'incomplete writer inventory'): + Launcher(self.root, 'owned').revoke(0) + + def test_duplicate_container_ids_prevent_fence(self): + for n in range(3): + self.launcher.launch(0, f'node{n}', f'disk{n}', lambda: 'a'*64) + with self.assertRaisesRegex(ValueError, 'incomplete writer inventory'): + self.launcher.revoke(0) + + def test_foreign_run_cannot_adopt_the_launch_ledger(self): + with self.assertRaisesRegex(ValueError, 'foreign launcher ledger'): + Launcher(self.root, 'foreign') + + +if __name__ == '__main__': + unittest.main() From c5231b6474474b744b72a78e3324d433f023acb9 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 19:57:08 -0700 Subject: [PATCH 255/568] fix: bind drill voters to advertised addresses Shared transport requires exact listener and route endpoints. Bind the private voter address and verify host configuration visibility before backend startup. --- tools/multi-raft/README.md | 6 ++++-- tools/multi-raft/group_disaster.py | 8 ++++++++ tools/multi-raft/node/config.rs | 2 +- tools/multi-raft/node/service.rs | 9 ++++++--- 4 files changed, 19 insertions(+), 6 deletions(-) diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 04a6126a6..b2d99ae9a 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -180,10 +180,12 @@ docker --context colima-cellar-multi-raft build -f tools/multi-raft/Dockerfile \ python3 tools/multi-raft/group_disaster.py \ --context colima-cellar-multi-raft \ --image "$(cat /tmp/cellar-my-checkout-group-image-id)" \ - --mode materialized --output /tmp/cellar-group-disaster-materialized-001 + --mode materialized --output "$HOME/cellar-qualification/group-disaster-materialized-001" ``` -Repeat with `--mode extents` and a fresh output directory. The image embeds the +Use an output directory shared with the VM; the runner verifies the mounted +config file before starting the backend. Repeat with `--mode extents` and a fresh +output directory. The image embeds the complete build-input manifest; the runner refuses a source mismatch. This uses the unoptimized, debug-info-free Cargo development profile for correctness qualification. It establishes no throughput or recovery-time claim. diff --git a/tools/multi-raft/group_disaster.py b/tools/multi-raft/group_disaster.py index 4ce489bfa..12345238b 100644 --- a/tools/multi-raft/group_disaster.py +++ b/tools/multi-raft/group_disaster.py @@ -10,6 +10,7 @@ import argparse from contextlib import contextmanager import fcntl +import hashlib import ipaddress import json import os @@ -162,6 +163,13 @@ def setup(self): require(len(data) == 44 and data[:12].hex() == '302a300506032b6570032100', 'unexpected public key encoding') self.config['fence_public'] = list(data[12:]) self.save() + # Docker must see the host file, rather than create an empty guest path. + mounted = self.docker('run', '--rm', '--network', 'none', '--entrypoint', 'sha256sum', + '--mount', 'type=bind,source='+str(self.config_dir)+',target=/config,readonly', + self.args.image, '/config/config.json').split()[0] + require(mounted == hashlib.sha256((self.config_dir/'config.json').read_bytes()).hexdigest(), + 'configuration output directory is not shared with the dedicated VM') + self.report['config_mount_verified'] = True self.docker('pull', RUSTFS) volume = self.prefix + '-objects' self.docker('volume', 'create', '--label', self.owner, volume) diff --git a/tools/multi-raft/node/config.rs b/tools/multi-raft/node/config.rs index 090a44b5e..1862baa7b 100644 --- a/tools/multi-raft/node/config.rs +++ b/tools/multi-raft/node/config.rs @@ -68,7 +68,7 @@ impl Options { impl Config { pub fn load(path: &std::path::Path) -> Result { - let bytes = std::fs::read(path)?; + let bytes = std::fs::read(path).context("read mounted qualification config")?; ensure!(bytes.len() <= 256 * 1024, "config too large"); let config: Self = serde_json::from_slice(&bytes)?; ensure!( diff --git a/tools/multi-raft/node/service.rs b/tools/multi-raft/node/service.rs index f74f5cf70..51580f946 100644 --- a/tools/multi-raft/node/service.rs +++ b/tools/multi-raft/node/service.rs @@ -139,13 +139,16 @@ pub(super) async fn run( .await?; runtime.attach(permit, root).await?; let n = options.node as u16; + let bind: std::net::Ipv4Addr = config.peers[options.generation][options.node] + .host + .parse()?; let transport = RaftTransport::start_shared( runtime.clone(), Arc::new(PeerAuth::new([17; 32], format!("node-{}", n + 1))?), tls()?, - TcpListener::bind(("0.0.0.0", 7100 + n * 3)).await?, - TcpListener::bind(("0.0.0.0", 7101 + n * 3)).await?, - TcpListener::bind(("0.0.0.0", 7102 + n * 3)).await?, + TcpListener::bind((bind, 7100 + n * 3)).await?, + TcpListener::bind((bind, 7101 + n * 3)).await?, + TcpListener::bind((bind, 7102 + n * 3)).await?, Default::default(), ) .await?; From 503abd3c5ff1c982aeee492bf662826fe535793f Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 20:08:15 -0700 Subject: [PATCH 256/568] fix: snapshot Docker drill configuration via API --- tools/multi-raft/README.md | 7 ++- tools/multi-raft/group_configuration.py | 57 +++++++++++++++++++ tools/multi-raft/group_disaster.py | 39 ++++++++----- tools/multi-raft/test_group_configuration.py | 58 ++++++++++++++++++++ 4 files changed, 144 insertions(+), 17 deletions(-) create mode 100644 tools/multi-raft/group_configuration.py create mode 100644 tools/multi-raft/test_group_configuration.py diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index b2d99ae9a..37d5977db 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -183,9 +183,10 @@ python3 tools/multi-raft/group_disaster.py \ --mode materialized --output "$HOME/cellar-qualification/group-disaster-materialized-001" ``` -Use an output directory shared with the VM; the runner verifies the mounted -config file before starting the backend. Repeat with `--mode extents` and a fresh -output directory. The image embeds the +Repeat with `--mode extents` and a fresh output directory. Configuration reaches +each stopped container through the Docker API with exact byte readback before +startup. Mutable host bind mounts are avoided because virtiofs can expose mixed +file size/content across atomic replacements. The image embeds the complete build-input manifest; the runner refuses a source mismatch. This uses the unoptimized, debug-info-free Cargo development profile for correctness qualification. It establishes no throughput or recovery-time claim. diff --git a/tools/multi-raft/group_configuration.py b/tools/multi-raft/group_configuration.py new file mode 100644 index 000000000..5f8575062 --- /dev/null +++ b/tools/multi-raft/group_configuration.py @@ -0,0 +1,57 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Copy a private, immutable configuration snapshot through the Docker API. + +Do not bind mutable host files: virtiofs may expose mixed file size/content across +atomic host replacements. No process starts until Docker returns exact bytes. +""" +import hashlib +import io +from pathlib import Path +import re +import subprocess +import tarfile +import tempfile + + +FILES = ('config.json', 'ledger.json') + + +def read_copy(context, identifier, name): + if re.fullmatch('[0-9a-f]{64}', identifier) is None or name not in FILES: + raise ValueError('invalid configuration copy identity') + result = subprocess.run(['docker', '--context', context, 'cp', identifier+':/config/'+name, '-'], + capture_output=True, timeout=240) + if result.returncode: + raise RuntimeError('Docker configuration readback failed') + with tarfile.open(fileobj=io.BytesIO(result.stdout)) as archive: + members = archive.getmembers() + if len(members) != 1 or not members[0].isfile() or members[0].name != name or members[0].size > 4*1024*1024: + raise ValueError('invalid Docker configuration readback') + return archive.extractfile(members[0]).read() + + +def copy_configuration(context, identifier, directory): + """Only configuration and verification tokens enter the container. + + The authority key and durable launcher ledger stay on the controller host. + Caller owns the immutable container ID and keeps its process stopped. + """ + if re.fullmatch('[0-9a-f]{64}', identifier) is None: + raise ValueError('invalid configuration copy identity') + contents = {name: (Path(directory)/name).read_bytes() for name in FILES} + with tempfile.TemporaryDirectory(prefix='cellar-config-snapshot-') as temporary: + snapshot = Path(temporary)/'config' + snapshot.mkdir(mode=0o700) + for name, data in contents.items(): + path = snapshot/name + path.write_bytes(data) + path.chmod(0o400) + result = subprocess.run(['docker', '--context', context, 'cp', str(snapshot), identifier+':/'], + capture_output=True, timeout=240) + if result.returncode: + raise RuntimeError('Docker configuration copy failed') + for name, expected in contents.items(): + if read_copy(context, identifier, name) != expected: + raise ValueError('Docker configuration snapshot differs from controller bytes') + return {name: hashlib.sha256(data).hexdigest() for name, data in contents.items()} diff --git a/tools/multi-raft/group_disaster.py b/tools/multi-raft/group_disaster.py index 12345238b..ddcc763be 100644 --- a/tools/multi-raft/group_disaster.py +++ b/tools/multi-raft/group_disaster.py @@ -10,7 +10,6 @@ import argparse from contextlib import contextmanager import fcntl -import hashlib import ipaddress import json import os @@ -26,6 +25,7 @@ from raft_disaster import LABEL, command, fence_and_delete, owned_container from raft_run import input_digest, parse_manifest, request, require, source_manifest from run import create_bucket +from group_configuration import copy_configuration RUSTFS = 'ghcr.io/rustfs/rustfs@sha256:bffcab0c9d647aab0055d1c69d340b202d0909966b385932d4ead1aeb7602858' PURPOSE = b'cellar-docker-cell-group-fence-1\0' @@ -163,13 +163,6 @@ def setup(self): require(len(data) == 44 and data[:12].hex() == '302a300506032b6570032100', 'unexpected public key encoding') self.config['fence_public'] = list(data[12:]) self.save() - # Docker must see the host file, rather than create an empty guest path. - mounted = self.docker('run', '--rm', '--network', 'none', '--entrypoint', 'sha256sum', - '--mount', 'type=bind,source='+str(self.config_dir)+',target=/config,readonly', - self.args.image, '/config/config.json').split()[0] - require(mounted == hashlib.sha256((self.config_dir/'config.json').read_bytes()).hexdigest(), - 'configuration output directory is not shared with the dedicated VM') - self.report['config_mount_verified'] = True self.docker('pull', RUSTFS) volume = self.prefix + '-objects' self.docker('volume', 'create', '--label', self.owner, volume) @@ -205,16 +198,30 @@ def until(operation, seconds=90): time.sleep(.2) raise RuntimeError('bounded qualification wait expired: ' + str(last)) + def stage_configuration(self, identifier): + value = owned_container(self.docker, identifier, self.run_id) + require(not value['State']['Running'], 'configuration cannot change under a live process') + hashes = copy_configuration(self.args.context, identifier, self.config_dir) + self.report.setdefault('configuration_copies', []).append({'container_id':identifier, 'sha256':hashes}) + self.save() + def operator(self, action, node=0, *, volume=None, generation=0, check=True): - args = ['run', '--rm', '--label', self.owner, '--network', self.prefix, - '-v', str(self.config_dir) + ':/config:ro', '-e', 'CELLAR_GROUP_ACTION=' + action, + args = ['create', '--label', self.owner, '--network', self.prefix, + '-e', 'CELLAR_GROUP_ACTION=' + action, '-e', 'CELLAR_GROUP_NODE=' + str(node), '-e', 'CELLAR_GROUP_GENERATION=' + str(generation)] if volume: args += ['-v', volume + ':/data'] - result = command(['docker', '--context', self.args.context, *args, self.args.image], check=False) + identifier = self.docker(*args, self.args.image) + self.stage_configuration(identifier) + result = command(['docker', '--context', self.args.context, 'start', '--attach', identifier], check=False) + value = owned_container(self.docker, identifier, self.run_id) + require(not value['State']['Running'], 'operator did not join') + require(result.returncode == 0 or value['State']['ExitCode'] != 0, 'Docker attachment failed') + result.returncode = value['State']['ExitCode'] text = self.redact(result.stdout + result.stderr) with (self.root / 'operator.log').open('a') as stream: stream.write(action + ': exit ' + str(result.returncode) + '\n' + text + '\n') + self.docker('rm', identifier) if check: require(result.returncode == 0, 'operator failed; inspect private sanitized operator log') return result, text @@ -225,13 +232,15 @@ def launch(self, node, generation, volume=None): if volume == name: self.docker('volume', 'create', '--label', self.owner, volume) def invoke(): - identifier = self.docker('run', '-d', '--label', self.owner, '--restart=no', '--name', name, + identifier = self.docker('create', '--label', self.owner, '--restart=no', '--name', name, '--network', self.prefix, '--ip', self.config['peers'][generation][node]['host'], '--cpus', '1', '--memory', '3g', '--memory-swap', '3g', '-p', f'127.0.0.1:{self.args.port_base+node+1}:{8080+node}', - '-v', volume + ':/data', '-v', str(self.config_dir) + ':/config:ro', + '-v', volume + ':/data', '-e', 'CELLAR_GROUP_NODE=' + str(node), '-e', 'CELLAR_GROUP_GENERATION=' + str(generation), '-e', 'CELLAR_GROUP_RETAINED=' + str(generation == 1).lower(), self.args.image) self.containers.append(identifier) + self.stage_configuration(identifier) + self.docker('start', identifier) captured = owned_container(self.docker, identifier, self.run_id) self.capture_resources(captured, 3) self.save() @@ -338,7 +347,7 @@ def drill(self): self.config['certificate'] = list(bytes.fromhex(plan['fence_digest']) + bytes(64)) self.save() forged, text = self.operator('fence', check=False) - require(forged.returncode != 0 and 'untrusted Docker controller fence' in text, 'forged infrastructure certificate accepted') + require(forged.returncode != 0 and 'untrusted Docker controller fence' in text, 'forged certificate did not reach signature-verification refusal') inventory = self.launcher.revoke(0) require(inventory == old, 'durable launcher inventory differs') for identifier, _ in old: @@ -418,6 +427,8 @@ def activate(): self.docker('stop', '--time','60', *(identifier for identifier,_ in self.targets)) for peer in self.config['peers'][1]: peer['process'] = list(secrets.token_bytes(16)) self.save() + for identifier, _ in self.targets: + self.stage_configuration(identifier) self.docker('start', *(identifier for identifier,_ in self.targets)) self.ready(1) self.ok({'action':'archive'},1) diff --git a/tools/multi-raft/test_group_configuration.py b/tools/multi-raft/test_group_configuration.py new file mode 100644 index 000000000..e8315ecaa --- /dev/null +++ b/tools/multi-raft/test_group_configuration.py @@ -0,0 +1,58 @@ +# Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +"""Real Docker regression for snapshot configuration through changing host files. + +Set CELLAR_GROUP_TEST_IMAGE to an immutable measurement image and explicitly set +CELLAR_GROUP_TEST_CONTEXT to the dedicated qualification context. No backend or +voter is started. Image contents are irrelevant to this Docker API boundary. +""" +import hashlib +import json +import os +from pathlib import Path +import re +import subprocess +import tempfile +import unittest + +from group_configuration import copy_configuration, read_copy +from group_disaster import save + + +class DockerConfigurationTests(unittest.TestCase): + def test_host_replacement_cannot_change_a_copied_process_snapshot(self): + image = os.environ.get('CELLAR_GROUP_TEST_IMAGE') + context = os.environ.get('CELLAR_GROUP_TEST_CONTEXT') + if not image or not context: + self.skipTest('requires explicit isolated Docker image/context') + self.assertRegex(image, r'^sha256:[0-9a-f]{64}$') + self.assertEqual(context, 'colima-cellar-multi-raft') + with tempfile.TemporaryDirectory(prefix='cellar-config-regression-') as temporary: + source = Path(temporary) + config = {'values':[17]*9000, 'certificate':[]} + save(source/'config.json', config) + save(source/'ledger.json', {'tokens':[]}) + (source/'private-key').write_text('must stay on the controller') + result = subprocess.run(['docker','--context',context,'create','--network','none', + '--label','io.cellar.config.regression=owned',image],capture_output=True,text=True,check=True) + identifier = result.stdout.strip() + self.assertTrue(re.fullmatch('[0-9a-f]{64}',identifier)) + try: + first = (source/'config.json').read_bytes() + copied = copy_configuration(context,identifier,source) + self.assertEqual(copied['config.json'],hashlib.sha256(first).hexdigest()) + config['certificate'] = [0]*96 + save(source/'config.json',config) + self.assertEqual(read_copy(context,identifier,'config.json'),first) + copy_configuration(context,identifier,source) + actual = read_copy(context,identifier,'config.json') + self.assertEqual(json.loads(actual),config) + self.assertEqual(actual,(source/'config.json').read_bytes()) + absent = subprocess.run(['docker','--context',context,'cp',identifier+':/config/private-key','-'],capture_output=True) + self.assertNotEqual(absent.returncode,0) + finally: + subprocess.run(['docker','--context',context,'rm',identifier],capture_output=True,check=True) + + +if __name__ == '__main__': + unittest.main() From f8aacd052591b7d3a31ea7ca426bef10d9d845cf Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 20:11:46 -0700 Subject: [PATCH 257/568] test: verify current recovered receipt versions --- tools/multi-raft/group_disaster.py | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/tools/multi-raft/group_disaster.py b/tools/multi-raft/group_disaster.py index ddcc763be..5e16b13f9 100644 --- a/tools/multi-raft/group_disaster.py +++ b/tools/multi-raft/group_disaster.py @@ -207,11 +207,13 @@ def stage_configuration(self, identifier): def operator(self, action, node=0, *, volume=None, generation=0, check=True): args = ['create', '--label', self.owner, '--network', self.prefix, + '--cpus', '1', '--memory', '3g', '--memory-swap', '3g', '-e', 'CELLAR_GROUP_ACTION=' + action, '-e', 'CELLAR_GROUP_NODE=' + str(node), '-e', 'CELLAR_GROUP_GENERATION=' + str(generation)] if volume: args += ['-v', volume + ':/data'] identifier = self.docker(*args, self.args.image) + self.capture_resources(owned_container(self.docker, identifier, self.run_id), 3) self.stage_configuration(identifier) result = command(['docker', '--context', self.args.context, 'start', '--attach', identifier], check=False) value = owned_container(self.docker, identifier, self.run_id) @@ -338,6 +340,7 @@ def drill(self): self.ledger['observations'] = self.observations(0) require({k:v['value'] for k,v in self.ledger['observations'].items()} == {'alpha':3,'beta':1,'gamma':0}, 'source group state mismatch') require(self.counts() == {'commands':4,'initializations':3}, 'source callbacks not exact') + self.report['checks']['source_s3_ledger_verified'] = True self.save() plan = self.ok({'action': 'plan'}, 0) self.config['recovery'] = plan['recovery'] @@ -367,6 +370,7 @@ def drill(self): else: raise ValueError('revoked deployment launch accepted') self.report['checks']['old_launch_after_controller_reload_refused'] = True + self.report['checks']['all_source_disks_deleted'] = True self.sign(plan['fence_digest']) self.operator('fence') bindings = [] @@ -404,6 +408,7 @@ def activate(): return code == 200 and value.get('activated') self.until(activate) self.ready(1) + self.report['checks']['fresh_quorum_activation'] = True self.operator('target-archive') self.ok({'action':'archive'}, 1) self.operator('verify-old-tokens') @@ -419,6 +424,7 @@ def activate(): code, value = self.call(node, {'action':'command','cell':'alpha','sequence':sequence,'input':input_value,'retry_until_ms':int(time.time()*1000)+300000,'s3':False}) require(code == 503 and expected in value['error'], 'restored receipt floor/identity lost') require(self.counts() == {'commands':0,'initializations':0}, 'recovery replayed callbacks') + self.report['checks'].update(source_cut_restored=True, exact_receipts_no_replay=True, s3_minimums=True, client_floors=True, old_tokens_refused=True) fresh = self.command('alpha', 4, 1) require(fresh['result']['value']==4 and fresh['result']['incarnation']=='47'*16 and fresh['version']==2, 'fresh write not in new history') require(self.counts() == {'commands':1,'initializations':0}, 'fresh callback count mismatch') @@ -434,8 +440,11 @@ def activate(): self.ok({'action':'archive'},1) for cell, sequence, original in [('alpha',2,source),('beta',1,beta)]: restored_retry = self.command(cell,sequence,1,s3=False) - require(restored_retry['result']==original['result'] and restored_retry['version']==1, - 'imported receipt lost on retained restart') + # The original opaque result stays fixed. Its proof observes the + # current cell image: alpha advanced once after genesis; beta did not. + expected_version = 2 if cell == 'alpha' else 1 + require(restored_retry['result']==original['result'] and restored_retry['version']==expected_version, + 'imported receipt or current observation lost on retained restart') retry = self.command('alpha',4,1,s3=False) require(retry['result']==fresh['result'] and retry['version']==2, 'retained receipt changed') require(self.counts()=={'commands':0,'initializations':0}, 'retained restart replayed callback') From e32115aa3218dc6eaa7ed53e134f413f26305938 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 20:22:48 -0700 Subject: [PATCH 258/568] docs: record Docker-fenced group recovery --- docs/design/multi-raft/cell-groups/README.md | 4 +- docs/design/multi-raft/cell-groups/handoff.md | 67 ++++++++++--------- .../design/multi-raft/cell-groups/progress.md | 59 +++++++++++++++- 3 files changed, 97 insertions(+), 33 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index df5966dd8..555229f77 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -10,7 +10,9 @@ all five exact cases and 205 Rust regressions across eighteen commands. CG-06 is IN_PROGRESS: complete S3 checkpoint/tail recovery, bounded S3 settlement, local reclamation, fresh physical genesis, fenced publication, committed activation and registered receipt recovery through a fresh quorum pass scoped native/RustFS -checks. Actual shared-group Docker launch/restart fencing is the immediate gate. +checks. Actual shared-group Docker launch/restart fencing and all-voter-disks- +deleted RustFS recovery now pass in both image modes; the original complete +CG-06 gate and unavailable historical raw evidence still need refresh. CG-08 due effects and safe cold-cache eviction follow, before CG-07 SDK/fleet cutover. See the latest [progress](progress.md) and [handoff](handoff.md). The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 228eda274..3eb833b65 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,39 +15,44 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch codex/shared-cell-groups and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest tested source is 9a0bad9124f0089055a0ba63b2952e5ef5e77853, containing origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. -Fresh physical genesis, exact fenced publication, committed activation and -fresh-quorum admission are implemented. Registered receipt recovery now passes -through production TLS/controllers/SQL and the replacement archive in both image -forms. Archive/catalog suites pass 47 cases; five exact real RustFS cases pass in -both forms with seven signed S3 probes. All-target Cellar/logic Clippy and the -scoped format/layout/dependency/documentation checks pass. - -See [progress](progress.md) for the scope and limitations. The scoped record is -.cellar/cell-groups-evidence/CG-06-recovery-genesis-settlement/receipt-recovery-qualification.json (SHA256 eb0efe6af7d4e6bed3a9801d821e0ea8e48058f7ce949cdc01912a74642fe25d). This is not a complete CG-06 gate. +The latest Docker-qualified source is f8aacd052591b7d3a31ea7ca426bef10d9d845cf, +containing freshly fetched origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. +Actual shared-group Docker voters pass the complete permanent-fencing and +all-voter-disks-deleted RustFS drill in both image modes. Exact genesis, pending +admission refusal, fresh-quorum activation, restored receipts/floors/S3 minimums, +old-token and old-launch refusal, new writes, retained restart and every replica's +final state are checked. Every remaining owned process joins normally. + +See [progress](progress.md) and tools/multi-raft/README.md for the scope and runner. +The scoped record is +.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json +(SHA256 08cf2112d4f91519c00d481a6cea3f8c0e378a242ecef258ac67a182c265e77d). +It is backed up outside this checkout. Raw command/token/credential files remain +private outside Git. Four failed predecessors are retained and explained. +Do not repeat the former mutable host config bind: a 150-read probe reproduced +38 truncated JSON observations through virtiofs. Docker API copies now verify +exact immutable config bytes before process startup, including retained restart. +The real Docker configuration regression passes. + +The earlier native/RustFS receipt source is +9a0bad9124f0089055a0ba63b2952e5ef5e77853: 47 archive/catalog cases and five exact +RustFS recovery cases in both forms, seven signed probes and all-target checks. +Its scoped record remains in progress.md. Historical results stay source-scoped. CG-06 remains IN_PROGRESS; the full hard-cutover goal remains active. -The most critical next work is the accepted Docker fencing qualification: - -1. Add an actual shared-group voter path to the Docker qualification harness. - Its current raft_node.rs/raft_disaster.py path still uses the older per-cell - runtime. Use three real voter containers and a durable source command ledger; - archive complete group checkpoint/tail state against RustFS. -2. Reuse the controller's exact owner/container/volume checks and private signing - key. Durably revoke the old deployment in the sole launcher's policy before - signing. Disable restarts, stop/join every possible old writer, remove the - immutable containers and all voter volumes, and confirm absence. Reload the - controller and prove old-deployment launch remains refused. No additional - provider choice or approval is needed: fencing-authority API plus Docker - qualification was already accepted. -3. Run the recovery operator with only the exact held S3 cut, request and signed - certificate. Seed fresh node/member/disk/process/engine identities, publish - once under the recovery lock and commit/apply activation through the fresh - quorum. Recheck original results, stored S3 minimums, client floors, old-token - refusal and no duplicate callback; perform new writes and retained restart. -4. Finish the original CG-06 gate without changing its commands or oracle. - Restore or refresh the unavailable CG-01–05 raw evidence before claiming - complete package validation. Keep the source hold as recovery provenance. +The critical next work is formal CG-06 closure: + +1. Run the unchanged original five exact native cases, inventory and all-target + check/Clippy, format/layout commands at the current source. Retain the actual + executable, outputs and source hashes in one task record. The Docker result + closes the requested deployment drill, not this full record by itself. +2. Restore or refresh unavailable CG-01–05 raw evidence before claiming complete + package validation. The validator currently refuses the missing CG-01 record; + do not waive that check or rewrite historical results. +3. Close CG-06 only after its complete evidence passes. Then implement CG-08's + bounded fair due-work scheduler and qualify close-on-settle/exact-root reopen, + followed by CG-07 public SDK/fleet cutover. Keep the accepted external fencing + contract; no provider approval is needed. The required nullable recovery metadata remains an internal marker-1 hard cutover. There is no legacy reader, state migration or permanent mode flag. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index a2f863a06..a3be48056 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,7 +10,64 @@ CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. -## CG-06 fresh quorum and registered receipt recovery +## CG-06 Docker-fenced, all-voter-disks-deleted recovery + +Source: f8aacd052591b7d3a31ea7ca426bef10d9d845cf, containing freshly fetched +origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. CG-06 remains IN_PROGRESS +pending its original complete gate and evidence refresh. + +The actual shared-group Docker/RustFS drill now passes in both materialized and +extent modes. Three voter processes replicate three independent SQLite cells in +one group. Four registered source commands receive S3 acknowledgements. The +trusted sole-launcher controller fsyncs permanent generation revocation, confirms +all three immutable old containers and voter volumes are absent, and only then +signs the exact recovery request. Reloading the controller refuses the revoked +generation; missing/forged certificates and recreated old disks are refused. +Its private key and launcher ledger never enter a voter or operator container. + +Three cold S3 restorations select the same complete genesis binding under fresh +group/cell/node/member/disk/engine identities. Pending application writes and all +read modes refuse. Actual TLS Raft replication commits and applies activation +before serving. Exact old receipt results preserve their original incarnation +without callbacks, Quorum retries retain stored S3 minimums, expired client floors +and conflicts refuse, and old tokens remain valid only for their old history. +New S3 writes succeed. Every fresh voter stops and resumes its retained disk with +new process identities; imported and new receipts still avoid replay, another new +write succeeds, and all three replicas' values, versions and incarnations match. +Every remaining owned process joins normally; both runs retain their object and +new voter volumes. All voter and operator containers capture one CPU/3 GiB with +no extra swap; each RustFS captures one CPU/2 GiB. + +The runner and native node live in tools/multi-raft/group_disaster.py and +tools/multi-raft/node/. Configurations are copied through the Docker API with +exact byte readback before startup. A real Docker regression verifies snapshot +isolation from later host replacements and exclusion of the controller key. +A separate 150-read probe reproduced 38 truncated JSON reads through mutable +virtiofs bind mounts; that transport is no longer used. Retained failed runs also +cover the initial unshared output path, wildcard transport endpoint mismatch and +an oracle that incorrectly expected genesis version after a later write. +The corrected oracle keeps exact original results and checks the current cell +observation (alpha version 2 after the first new write, beta version 1), with zero +replayed callbacks. No production assertion or protocol was weakened. + +Scoped record: .cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json +(SHA256 08cf2112d4f91519c00d481a6cea3f8c0e378a242ecef258ac67a182c265e77d). +Deployed executable SHA256: +ea5e554d781cdd8004891d28ec8ecfb0abb0664ac5e372a135f0585842045be8. +The record binds both reports, source manifest, raw ledger/log hashes and actual +container/VM resources. Private credentials/signing material stay outside Git; +sanitized scoped evidence is copied locally and backed up on the workspace volume. +This is a local correctness result, without managed recovery, external effects, +sustained load, independent hosts, Amazon S3 or a recovery-time SLA claim. + +The original five-case CG-06 native gate and all-target checks still need a fresh +complete record. Historical CG-01–05 raw records remain unavailable; package +validation refuses the missing CG-01 record. Restore or rerun the original gates, +without changing validation or historical task boundaries. Then prioritize CG-08 +bounded effect discovery/execution and exact-root cold-cache reopening, followed +by CG-07 public SDK/fleet cutover. + +## Earlier CG-06 native quorum and receipt checkpoint Source: 9a0bad9124f0089055a0ba63b2952e5ef5e77853, including origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. CG-06 remains IN_PROGRESS. Fresh recovery now installs the complete imported root and physical Raft genesis From b29fb32a1b56bacdb4fe89671f867e3d6bdee316 Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 21:27:52 -0700 Subject: [PATCH 259/568] docs: close cell-group archive and recovery gate --- docs/design/multi-raft/cell-groups/README.md | 53 +++++------- docs/design/multi-raft/cell-groups/handoff.md | 84 +++++++++++-------- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 47 ++++++++++- docs/design/multi-raft/cell-groups/tasks.json | 16 ++-- 5 files changed, 120 insertions(+), 82 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 555229f77..bb17ac324 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -3,17 +3,14 @@ # Cell group delivery package Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 through CG-05 have passed their scoped native evidence gates. CG-05's -complete checkpoint, authenticated learner/rebuild, membership, restart and -resource-ownership requirements pass at `e0550acd`. Its current gate includes -all five exact cases and 205 Rust regressions across eighteen commands. -CG-06 is IN_PROGRESS: complete S3 checkpoint/tail recovery, bounded S3 settlement, -local reclamation, fresh physical genesis, fenced publication, committed activation -and registered receipt recovery through a fresh quorum pass scoped native/RustFS -checks. Actual shared-group Docker launch/restart fencing and all-voter-disks- -deleted RustFS recovery now pass in both image modes; the original complete -CG-06 gate and unavailable historical raw evidence still need refresh. -CG-08 due effects and safe cold-cache eviction follow, before CG-07 SDK/fleet +CG-01 through CG-06 have passed their current scoped native evidence gates at +e32115aa. The unchanged gate refresh passes 123 commands: 110 exact native cases, +nineteen pure-logic tests and the shared repository checks. Fresh retained records +replace the unavailable historical raw artifacts. CG-06's complete archive, +local reclamation and fresh-history recovery boundary also passes both actual +Docker/RustFS all-voter-disks-deleted drills with permanent launch revocation, +fresh-quorum activation, exact receipts, new writes and retained restart. +CG-08 due effects and safe cache lifetimes follow, before CG-07 SDK/fleet cutover. See the latest [progress](progress.md) and [handoff](handoff.md). The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. @@ -50,29 +47,17 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package only with verified evidence and completed dependencies. Final qualification repeats the full capability matrix at one release revision. -Historical native gate records are referenced under the ignored -`.cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/`, -`.cellar/cell-groups-evidence/CG-02-ec9d277e5041/` and -`.cellar/cell-groups-evidence/CG-03-routing-final/` directories. CG-04 adds -`.cellar/cell-groups-evidence/CG-04-qualified/` and its supporting -`.cellar/cell-groups-evidence/CG-04-local-provisioning-final/` directory. A fresh -checkout must receive these retained directories at the same relative paths -before validating the first four DONE checkpoints. CG-05 additionally requires -`.cellar/cell-groups-evidence/CG-05-completion-admission/qualification/` and its -retained predecessor/reproduction records. Missing evidence fails validation. -CG-05's earlier partial source checkpoint is retained in -`.cellar/cell-groups-evidence/CG-05-checkpoint-source/` and the receiver slice in -`.cellar/cell-groups-evidence/CG-05-checkpoint-receiver/`, followed by owner/runtime -acceptance in `.cellar/cell-groups-evidence/CG-05-snapshot-owner/`. These partial -checkpoints and local publication in -`.cellar/cell-groups-evidence/CG-05-local-checkpoint/`, authenticated transport in -`.cellar/cell-groups-evidence/CG-05-snapshot-transport/`, automatic policy in -`.cellar/cell-groups-evidence/CG-05-controller-checkpoints/` and membership in -`.cellar/cell-groups-evidence/CG-05-group-membership/` remain historical partial records; -the complete current-source gate above closes CG-05. -The referenced CG-01–05 raw directories are currently unavailable locally. -Restore their records or rerun the unchanged exact gates before complete package -validation. The scoped CG-06 receipt checkpoint does not replace them. +Current source-scoped records live under the ignored +`.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/` directory, with one +`CG-NN-record.json` per completed task. Logs, six copied executables and all source +hashes are retained locally and backed up on the workspace volume. A fresh +checkout must receive the same repository-relative evidence paths, or rerun the +unchanged exact gates before full package validation. The separately scoped local +Docker/RustFS record is retained under +`.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/`. Historical CG-01–05 raw +records remain unavailable; their earlier source-scoped results are documented in +progress.md and are superseded by the fresh native records. Missing or changed +evidence still fails validation. The design, task briefs and validator unit tests are committed in this package. ```sh diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 3eb833b65..f5b182ef6 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -15,44 +15,56 @@ node resources. A group spans voter nodes; it is not all cells on one node. Use branch codex/shared-cell-groups and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). -The latest Docker-qualified source is f8aacd052591b7d3a31ea7ca426bef10d9d845cf, -containing freshly fetched origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. -Actual shared-group Docker voters pass the complete permanent-fencing and -all-voter-disks-deleted RustFS drill in both image modes. Exact genesis, pending -admission refusal, fresh-quorum activation, restored receipts/floors/S3 minimums, -old-token and old-launch refusal, new writes, retained restart and every replica's -final state are checked. Every remaining owned process joins normally. - -See [progress](progress.md) and tools/multi-raft/README.md for the scope and runner. -The scoped record is +CG-01 through CG-06 now pass their current source-scoped gates. CG-06 is DONE +at its unmanaged archive/reclamation/disaster-recovery boundary. Native gate +source: e32115aa3218dc6eaa7ed53e134f413f26305938, including origin/main +81cbbc2f72f302751dd471cf06cf51394146ac63. All 123 unchanged gate commands pass: +110 exact native cases, nineteen pure-logic tests, all-target check/Clippy, +format/layout and dependency/documentation checks. Six retained executable hashes +and 3257 source hashes bind the evidence. + +Fresh records replace the unavailable CG-01–05 raw artifacts in tasks.json: +.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-01-record.json through +CG-06-record.json. They are backed up outside the checkout. Each record passes the +unchanged evidence validator. Full package validation passes with six DONE +tasks and 44 local links. CG-06 record SHA256: +29cfff144bb38f5cda6968e5a8d2cf163ff440df0562ec8948059176ab7827b8. +The reclaim case was rerun with separate real stdout/stderr after its original +passing merged log's diagnostics interrupted the test result line. Both logs +remain; no test/validator expectation was weakened. + +Actual shared-group Docker voters pass permanent fencing and all-voter-disks- +deleted RustFS recovery in both image forms at f8aacd05. Its executable inputs +match the native gate source (only plan documents differ). Pending admission, +fresh activation/quorum, receipts/floors/S3 minima, old-token/launch refusal, new +writes, retained restart and every replica's final state pass. All remaining +processes join normally; the VM is stopped, data retained and default context +preserved. See [progress](progress.md) and tools/multi-raft/README.md. +Scoped Docker record: .cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json (SHA256 08cf2112d4f91519c00d481a6cea3f8c0e378a242ecef258ac67a182c265e77d). -It is backed up outside this checkout. Raw command/token/credential files remain -private outside Git. Four failed predecessors are retained and explained. -Do not repeat the former mutable host config bind: a 150-read probe reproduced -38 truncated JSON observations through virtiofs. Docker API copies now verify -exact immutable config bytes before process startup, including retained restart. -The real Docker configuration regression passes. - -The earlier native/RustFS receipt source is -9a0bad9124f0089055a0ba63b2952e5ef5e77853: 47 archive/catalog cases and five exact -RustFS recovery cases in both forms, seven signed probes and all-target checks. -Its scoped record remains in progress.md. Historical results stay source-scoped. -CG-06 remains IN_PROGRESS; the full hard-cutover goal remains active. - -The critical next work is formal CG-06 closure: - -1. Run the unchanged original five exact native cases, inventory and all-target - check/Clippy, format/layout commands at the current source. Retain the actual - executable, outputs and source hashes in one task record. The Docker result - closes the requested deployment drill, not this full record by itself. -2. Restore or refresh unavailable CG-01–05 raw evidence before claiming complete - package validation. The validator currently refuses the missing CG-01 record; - do not waive that check or rewrite historical results. -3. Close CG-06 only after its complete evidence passes. Then implement CG-08's - bounded fair due-work scheduler and qualify close-on-settle/exact-root reopen, - followed by CG-07 public SDK/fleet cutover. Keep the accepted external fencing - contract; no provider approval is needed. +Keep credentials, signing material and raw tokens private. Docker API config +copies avoid the reproduced virtiofs truncation race; controller keys/launch +ledgers never enter containers. Failed predecessors remain recorded. + +The next critical implementation is CG-08, then CG-07 SDK/fleet cutover. An +isolated codex/cell-group-effects-3617 branch at +/Users/haipingfu/.codex/worktrees/cell-group-effects/cellar contains an unqualified +CG-08 due-index draft while the qualified checkout stays unchanged. It adds +subtree next-due hints, required nullable marker-1 fields and bounded fair +manifest lookup without SQLite hydration. Focused unit validation is in progress; +no scheduler or full CG-08 qualification is claimed. Merge qualified changes +back into the existing PR branch; do not create a permanent second runtime. + +1. Finish the due-index validation and wire bounded fair node effect discovery, + exact committed-image scans and sealed private system commands into the shared + controller. Preserve current CellHistory-based identities and receiver dedupe. +2. Drive alarm/outbox work without client requests, retaining unknown delivery and + SQL producer ownership through leadership change, cancellation and shutdown. + Qualify the current close-on-settle/exact-root reopen policy under hot/cold work. +3. Complete the original five CG-08 cases, then cut public SDK/fleet consumers over + in CG-07. Topology, hibernation, managed recovery, load/crash qualification and + removal of old consumers remain later tasks. The full goal stays active. The required nullable recovery metadata remains an internal marker-1 hard cutover. There is no legacy reader, state migration or permanent mode flag. diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 43b72ac5d..b52bbb2a8 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -17,7 +17,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-03 | Prepare paged range routing and enrollment admission | CG-01, CG-02 | DONE | 8 | | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 | -| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | IN_PROGRESS | 5 | +| CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | DONE | 5 | | CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | | CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | | CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | TODO | 7 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index a3be48056..3ec91ac70 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -5,16 +5,55 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). -CG-01 through CG-05 have passed their scoped native gates; -CG-06 is IN_PROGRESS; CG-07 through CG-16 remain TODO. Intermediate contracts +CG-01 through CG-06 have passed their scoped native gates; +CG-07 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-06 complete native and local disaster-recovery gate + +CG-06 is DONE at its declared unmanaged archive/reclamation/recovery boundary. +Qualified native source: e32115aa3218dc6eaa7ed53e134f413f26305938, containing +origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. Its executable inputs match +the Docker-qualified f8aacd05 source; their only differences are plan documents. +CG-07–16 and the complete hard cutover remain unfinished. + +All 123 unique commands in the unchanged CG-01–06 gate refresh pass: 110 exact +native cases, nineteen pure-logic unit tests, all-target locked check/Clippy, +format/layout over 1990 Rust files, native dependency and documentation checks. +Six copied test executables and 3257 source hashes bind the new records. The +unavailable historical CG-01–05 raw records have been superseded by fresh exact +native gate evidence at this source; their earlier results remain historical. +No acceptance command, oracle or validator was weakened. + +Current records live under +.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/ and are backed up on the +workspace volume. CG-06-record.json SHA256: +29cfff144bb38f5cda6968e5a8d2cf163ff440df0562ec8948059176ab7827b8. +Its native executable SHA256: +8b18b5677e07862c892dcea3a02fceb36a11234a2636b20d6efae87f4bddd5ac. +Each task record independently passes the unchanged source/artifact/command +validator. Full package validation passes with six DONE tasks, all source-bound +records and 44 local links; no unavailable-record exemption is used. + +The reclaim case originally passed but its stderr observations interrupted the +libtest result line in the merged log. That original output is retained. The +unchanged case passes again with stdout and stderr retained and hashed separately; +the strict one-case result requirement stays intact. The original gate runner's +initial unsupported ls-tree exclude-pathspec failure is also retained. + +Both complete Docker/RustFS disk-loss drills below pass, including exact physical +recovery and retained restart. These establish local correctness only; managed +recovery, external effects, independent failure domains, sustained load and final +SDK/fleet cutover belong to later gates. The dedicated VM is stopped and data +retained. CG-08's bounded effect scheduler and safe cache lifetimes now have +priority, followed by CG-07 public SDK/fleet cutover. + ## CG-06 Docker-fenced, all-voter-disks-deleted recovery Source: f8aacd052591b7d3a31ea7ca426bef10d9d845cf, containing freshly fetched -origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. CG-06 remains IN_PROGRESS -pending its original complete gate and evidence refresh. +origin/main 81cbbc2f72f302751dd471cf06cf51394146ac63. At this source CG-06 +remained IN_PROGRESS; the complete gate above closes it. The actual shared-group Docker/RustFS drill now passes in both materialized and extent modes. Three voter processes replicate three independent SQLite cells in diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 7c9f8ac89..57120acc2 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -117,7 +117,7 @@ "I12" ], "evidence": [ - ".cellar/cell-groups-evidence/CG-01-ee7f4a8d2dec/record.json" + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-01-record.json" ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", @@ -247,7 +247,7 @@ "I12" ], "evidence": [ - ".cellar/cell-groups-evidence/CG-02-ec9d277e5041/record.json" + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-02-record.json" ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", @@ -403,7 +403,7 @@ "I12" ], "evidence": [ - ".cellar/cell-groups-evidence/CG-03-routing-final/record.json" + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-03-record.json" ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", @@ -1541,7 +1541,7 @@ "I12" ], "evidence": [ - ".cellar/cell-groups-evidence/CG-04-qualified/checkpoint.json" + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-04-record.json" ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", @@ -1689,7 +1689,7 @@ "I12" ], "evidence": [ - ".cellar/cell-groups-evidence/CG-05-completion-admission/qualification/checkpoint.json" + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-05-record.json" ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", @@ -1706,7 +1706,7 @@ { "id": "CG-06", "title": "Archive and reclaim the complete group recovery history", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -1875,7 +1875,9 @@ "I11", "I12" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-06-record.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From 783c270b372925b1f75cd352279d6d4702952a6b Mon Sep 17 00:00:00 2001 From: forhappy Date: Sun, 1 Nov 2026 21:58:32 -0700 Subject: [PATCH 260/568] feat: index due cell effects in group roots --- .../durability/raft/cell_groups/README.md | 16 +- .../raft/cell_groups/effect_index.rs | 68 ++++ .../raft/cell_groups/effect_index_tests.rs | 291 ++++++++++++++++++ .../durability/raft/cell_groups/manifest.rs | 42 +++ .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../durability/raft/cell_groups/page_wire.rs | 20 ++ .../durability/raft/cell_groups/pins.rs | 38 +++ .../durability/raft/cell_groups/root.rs | 21 +- 8 files changed, 495 insertions(+), 2 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/effect_index.rs create mode 100644 crates/cellar/durability/raft/cell_groups/effect_index_tests.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 997caf477..f3710aabf 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -8,12 +8,26 @@ is not yet wired to these contracts; a stored root is not quorum authority. A manifest is a persistent AVL tree ordered by the namespace routing digest and full cell ID. Each bounded page holds one cell leaf and at most two child -references. References bind SHA-256, height and subtree count. Insertions and +references. References bind SHA-256, height, subtree count and earliest due-effect time. Insertions and updates copy bounded paths; rotations preserve balance. Unchanged cell images are reused. The format uses `CLRCGM01`, the marker-1 page hash domain, canonical JSON and a 16 KiB decode bound. Tree depth and cell count are explicit limits. These limits are admission ceilings, not qualified fleet capacities. +The due-effect index stores each subtree's earliest alarm/outbox time in its +required nullable page reference. `ManifestRoot::next_due_unix_millis` is a cheap +hint after complete-root verification. `ManifestTree::next_due` visits bounded +paths to select one due cell after a routing-key cursor, wrapping once for fair +selection. Future-only subtrees need no SQLite hydration. A prepared physical root also +exposes `next_due` over its verified persistent pin tree, with no filesystem or +SQLite I/O. Its returned image pin is a data view and grants no execution rights. +The caller runs page +I/O on its blocking execution domain and retains the selected verified root; +returned metadata still requires fresh leader, route, image and private-command +admission before execution. Missing scheduling fields refuse at marker 1, with +no compatibility reader. Node scheduler/system-command integration and the full +CG-08 effect/cache qualification remain unfinished. + Leaves bind logical cell history, mutation sequence, image commitment and geometry, last group installation/mutation position, schema, effect summary, immutable original enrollment identity and the selected version's complete diff --git a/crates/cellar/durability/raft/cell_groups/effect_index.rs b/crates/cellar/durability/raft/cell_groups/effect_index.rs new file mode 100644 index 000000000..98b478fe8 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/effect_index.rs @@ -0,0 +1,68 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated next-due hints prune cold subtrees without hydrating SQLite. +//! Hints are scheduling data, never execution or delivery authority. +use super::page_wire::PageRef; +use super::{ + CellManifestLeaf, ManifestError, ManifestLimits, ManifestPageStore, ManifestRoot, ManifestTree, +}; +use crate::application::consensus::cell_routing_key; +use cellar_logic::cell_group::{CellId, CellKey}; + +pub(super) fn next( + tree: &ManifestTree, + store: &ManifestPageStore, + root: ManifestRoot, + now: u64, + after: Option<&CellId>, + limits: ManifestLimits, +) -> Result, ManifestError> { + let after = after.map(|cell| cell_routing_key(tree.route().history.namespace, cell)); + // One cursor path can precede a path to the successor; wrapping adds at + // most one more root-to-leaf path. Exhaustion is an error, never "no work". + let mut remaining = usize::from(limits.max_depth) * 3; + let selected = find(tree, store, root.0, now, after.as_ref(), &mut remaining)?; + if selected.is_some() || after.is_none() { + return Ok(selected); + } + find(tree, store, root.0, now, None, &mut remaining) +} + +fn find( + tree: &ManifestTree, + store: &ManifestPageStore, + reference: Option, + now: u64, + after: Option<&CellKey>, + remaining: &mut usize, +) -> Result, ManifestError> { + let Some(reference) = + reference.filter(|p| p.next_due_unix_millis.is_some_and(|due| due <= now)) + else { + return Ok(None); + }; + *remaining = remaining.checked_sub(1).ok_or(ManifestError::Limit)?; + let page = tree.read(store, reference)?; + let key = cell_routing_key( + tree.route().history.namespace, + &page.leaf.version.history.cell, + ); + if after.is_none_or(|after| &key > after) { + if let Some(selected) = find(tree, store, page.left, now, after, remaining)? { + return Ok(Some(selected)); + } + if page + .leaf + .effects + .next_due_unix_millis() + .is_some_and(|due| due <= now) + { + return Ok(Some(page.leaf)); + } + } + find(tree, store, page.right, now, after, remaining) +} + +#[cfg(test)] +#[path = "effect_index_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs new file mode 100644 index 000000000..e81d08a01 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs @@ -0,0 +1,291 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Metadata-index tests only. They do not qualify effect execution or durability. +use super::*; +use crate::raft::cell_groups::{CellImageGeometry, EffectSummary, EnrollmentRecord}; +use cellar_logic::cell_group::*; +use cellar_logic::consensus::*; +use std::sync::Arc; + +struct Fixture { + _directory: tempfile::TempDir, + store: ManifestPageStore, + tree: ManifestTree, + root: ManifestRoot, + leaves: Vec, +} + +fn fixture(empty: bool) -> Fixture { + let directory = tempfile::tempdir().unwrap(); + let mut store = ManifestPageStore::open( + directory.path(), + Arc::new(cellar_ltx::DirectFileSystem), + 4096, + 32 * 1024 * 1024, + 16, + ) + .unwrap(); + let group = GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([4; 16]).unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + }; + let route = GroupRoute { + history: group, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [7; 32], + }; + let tree = ManifestTree::new(route, ManifestLimits::default()).unwrap(); + let image = CellImage { + epoch: 1, + txid: if empty { 0 } else { 1 }, + checksum: 0, + commitment: CellImageCommitment::Materialized(if empty { + use sha2::Digest as _; + sha2::Sha256::digest([]).into() + } else { + [8; 32] + }), + }; + let mut root = ManifestRoot::default(); + let mut leaves = vec![]; + for n in 0..64 { + let history = CellHistory { + cluster: group.cluster, + namespace: group.namespace, + cell: CellId::new(format!("cell-{n}")).unwrap(), + incarnation: IncarnationId::new([3; 16]).unwrap(), + }; + let enrollment = EnrollmentRecord::new( + history.clone(), + ProposalId::new([9; 16]).unwrap(), + tree.route().schema_digest, + image, + ) + .unwrap(); + let position = LogPosition { + index: LogIndex::new(2), + term: Term::new(1), + }; + let effects = match n { + 3 => EffectSummary { + next_alarm_unix_millis: Some(10), + ..Default::default() + }, + 19 => EffectSummary { + next_outbox_unix_millis: Some(10), + pending_outbox: 1, + ..Default::default() + }, + 49 => EffectSummary { + next_alarm_unix_millis: Some(20), + ..Default::default() + }, + _ => EffectSummary { + next_alarm_unix_millis: Some(1000 + n), + ..Default::default() + }, + }; + let leaf = CellManifestLeaf { + enrollment: enrollment.clone(), + version: CellVersion { + history, + sequence: MutationSequence::new(1).unwrap(), + image, + }, + geometry: CellImageGeometry { + length: if empty { 0 } else { 512 }, + page_size: 512, + }, + last_mutation: position, + witness: CommitWitness { + group, + position, + proposal: enrollment.creation(), + }, + effects, + }; + root = tree.update(&mut store, root, leaf.clone(), None).unwrap(); + leaves.push(leaf); + } + Fixture { + _directory: directory, + store, + tree, + root, + leaves, + } +} + +#[test] +fn due_selection_matches_independent_order_and_wraps_fairly() { + let f = fixture(false); + assert_eq!(f.root.next_due_unix_millis(), Some(10)); + for now in [0, 9, 10, 19, 20, 1000, 9999] { + let mut eligible: Vec<_> = f + .leaves + .iter() + .filter(|leaf| { + leaf.effects + .next_due_unix_millis() + .is_some_and(|at| at <= now) + }) + .collect(); + eligible.sort_by_key(|leaf| { + cell_routing_key(f.tree.route().history.namespace, &leaf.version.history.cell) + }); + for cursor in f + .leaves + .iter() + .map(|leaf| Some(&leaf.version.history.cell)) + .chain([None]) + { + let key = cursor.map(|cell| cell_routing_key(f.tree.route().history.namespace, cell)); + let expected = eligible + .iter() + .copied() + .find(|leaf| { + key.as_ref().is_none_or(|key| { + &cell_routing_key( + f.tree.route().history.namespace, + &leaf.version.history.cell, + ) > key + }) + }) + .or_else(|| eligible.first().copied()); + let actual = f.tree.next_due(&f.store, f.root, now, cursor).unwrap(); + assert_eq!(actual.as_ref(), expected); + } + } + let mut cursor = None; + let mut seen = std::collections::BTreeMap::new(); + for _ in 0..20 { + let leaf = f + .tree + .next_due(&f.store, f.root, 10, cursor.as_ref()) + .unwrap() + .unwrap(); + *seen.entry(leaf.version.history.cell.clone()).or_insert(0) += 1; + cursor = Some(leaf.version.history.cell); + } + assert_eq!(seen.len(), 2); + assert!(seen.values().all(|count| *count == 10)); +} + +#[test] +fn changing_effects_updates_new_root_and_preserves_earlier_root() { + let mut f = fixture(false); + let original = f.root; + for n in [3, 19, 49] { + let previous = f.leaves[n].clone(); + let mut next = previous.clone(); + next.effects = EffectSummary::default(); + next.version.sequence = next.version.sequence.checked_next().unwrap(); + next.version.image.txid += 1; + next.version.image.commitment = CellImageCommitment::Materialized([10; 32]); + next.last_mutation.index = LogIndex::new(100 + n as u64); + next.witness.position = next.last_mutation; + next.witness.proposal = ProposalId::new([10; 16]).unwrap(); + f.root = f + .tree + .update(&mut f.store, f.root, next.clone(), Some(&previous.version)) + .unwrap(); + f.leaves[n] = next; + } + assert_eq!(f.root.next_due_unix_millis(), Some(1000)); + assert!(f + .tree + .next_due(&f.store, f.root, 999, None) + .unwrap() + .is_none()); + assert!(f + .tree + .next_due(&f.store, original, 10, None) + .unwrap() + .is_some()); + f.tree + .verify( + &f.store, + f.root, + LogPosition { + index: LogIndex::new(1000), + term: Term::new(1), + }, + |_| Ok(()), + ) + .unwrap(); + let mut forged = f.root; + forged.0.as_mut().unwrap().next_due_unix_millis = None; + assert!(f + .tree + .verify( + &f.store, + forged, + LogPosition { + index: LogIndex::new(1000), + term: Term::new(1) + }, + |_| Ok(()) + ) + .is_err()); +} + +#[test] +fn marker_one_references_require_nullable_due_summary() { + let f = fixture(false); + let reference = PageRef { + digest: [1; 32], + height: 1, + cells: 1, + next_due_unix_millis: None, + }; + let mut value = serde_json::to_value(reference).unwrap(); + assert!(value["next_due_unix_millis"].is_null()); + assert_eq!( + serde_json::from_value::(value.clone()).unwrap(), + reference + ); + value + .as_object_mut() + .unwrap() + .remove("next_due_unix_millis"); + assert!(serde_json::from_value::(value).is_err()); + let mut invalid = f.leaves[3].clone(); + invalid.effects.next_outbox_unix_millis = Some(i64::MAX as u64 + 1); + invalid.effects.pending_outbox = 1; + assert!(invalid.validate(f.tree.route()).is_err()); +} + +#[test] +fn verified_pinned_index_matches_page_lookup_without_opening_images() { + let mut f = fixture(true); + let position = LogPosition { + index: LogIndex::new(1000), + term: Term::new(1), + }; + f.tree + .verify(&f.store, f.root, position, |_| Ok(())) + .unwrap(); + let pins = crate::raft::cell_groups::ManifestPins::complete( + &f.tree, + &mut f.store, + f.root, + &mut |_| Ok(crate::raft::cell_groups::CellImagePin::Empty), + ) + .unwrap(); + for now in [0, 10, 20, 1000, 9999] { + for leaf in &f.leaves { + let cell = &leaf.version.history.cell; + let key = cell_routing_key(f.tree.route().history.namespace, cell); + let expected = f.tree.next_due(&f.store, f.root, now, Some(cell)).unwrap(); + let actual = pins.next_due(now, Some(&key)); + assert_eq!(actual.as_ref().map(|(leaf, _)| *leaf), expected.as_ref()); + if let Some((leaf, image)) = actual { + image.check(leaf, f.tree.route().schema_digest).unwrap(); + } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 9ae542f6d..426875204 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -87,6 +87,16 @@ pub struct EffectSummary { pub pending_inbox: u64, } +impl EffectSummary { + /// Earliest durable work hint. It grants no callback or delivery authority. + pub fn next_due_unix_millis(self) -> Option { + [self.next_alarm_unix_millis, self.next_outbox_unix_millis] + .into_iter() + .flatten() + .min() + } +} + #[derive(Clone, Debug, PartialEq, Eq)] pub struct CellManifestLeaf { /// Original creation identity, retained atomically with every selected image. @@ -113,6 +123,17 @@ impl CellManifestLeaf { } pub(super) fn validate(&self, route: &GroupRoute) -> Result { + if [ + self.effects.next_alarm_unix_millis, + self.effects.next_outbox_unix_millis, + ] + .into_iter() + .flatten() + .any(|due| due > i64::MAX as u64) + || (self.effects.pending_outbox == 0) != self.effects.next_outbox_unix_millis.is_none() + { + return Err(ManifestError::Invalid("effect summary")); + } let key = cell_routing_key(route.history.namespace, &self.version.history.cell); route .check_cell(&self.version.history, &key) @@ -188,6 +209,12 @@ impl ManifestRoot { self.0.map_or(0, |p| p.height) } + /// Authenticated subtree hint after complete-root verification. No SQLite + /// database needs to be opened to decide whether this root has due work. + pub fn next_due_unix_millis(self) -> Option { + self.0.and_then(|p| p.next_due_unix_millis) + } + pub fn digest(self) -> Option<[u8; 32]> { self.0.map(|p| p.digest) } @@ -218,6 +245,21 @@ impl ManifestTree { Ok(Self { route, limits }) } + /// Select one due cell after the cursor in routing-key order, wrapping once. + /// Run on the owning blocking domain while retaining the verified selected + /// root. Reads at most three tree depths of pages, and opens no SQLite DB. + /// The returned leaf is advisory: execution still needs fresh leader, route, + /// exact-image and private-command admission checks. + pub fn next_due( + &self, + store: &ManifestPageStore, + root: ManifestRoot, + now_unix_millis: u64, + after: Option<&CellId>, + ) -> Result, ManifestError> { + super::effect_index::next(self, store, root, now_unix_millis, after, self.limits) + } + pub fn get( &self, store: &ManifestPageStore, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 312a1ca59..9682d9198 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -57,6 +57,7 @@ pub use dispatch::{ CellReceiptSelection, }; +mod effect_index; mod execution_authority; pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs index 48e64a1e5..156983f7a 100644 --- a/crates/cellar/durability/raft/cell_groups/page_wire.rs +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -21,6 +21,15 @@ pub(super) struct PageRef { pub digest: [u8; 32], pub height: u8, pub cells: u32, + // Required even when null: internal marker-1 hard cutover, no old reader. + #[serde(deserialize_with = "required_due")] + pub next_due_unix_millis: Option, +} + +fn required_due<'de, D: serde::Deserializer<'de>>( + deserializer: D, +) -> Result, D::Error> { + Option::::deserialize(deserializer) } pub(super) struct Page { @@ -42,6 +51,9 @@ impl Page { || child.height > limits.max_depth || child.cells == 0 || child.cells > limits.max_cells + || child + .next_due_unix_millis + .is_some_and(|due| due > i64::MAX as u64) { return Err(ManifestError::Invalid("child bounds")); } @@ -64,6 +76,14 @@ impl Page { digest: digest(bytes), height, cells, + next_due_unix_millis: [ + self.leaf.effects.next_due_unix_millis(), + self.left.and_then(|p| p.next_due_unix_millis), + self.right.and_then(|p| p.next_due_unix_millis), + ] + .into_iter() + .flatten() + .min(), }) } } diff --git a/crates/cellar/durability/raft/cell_groups/pins.rs b/crates/cellar/durability/raft/cell_groups/pins.rs index 5697a3a47..173ead51e 100644 --- a/crates/cellar/durability/raft/cell_groups/pins.rs +++ b/crates/cellar/durability/raft/cell_groups/pins.rs @@ -77,6 +77,44 @@ impl ManifestPins { self.find(key).map(|p| (&p.leaf, p.image.clone())) } + /// The complete verified mirror already owns every referenced page/image. + /// Discovery uses immutable metadata only: no filesystem or SQLite work. + pub(super) fn next_due( + &self, + now: u64, + after: Option<&CellKey>, + ) -> Option<(&CellManifestLeaf, CellImagePin)> { + let page = self + .find_due(now, after) + .or_else(|| after.and_then(|_| self.find_due(now, None)))?; + Some((&page.leaf, page.image.clone())) + } + + fn find_due(&self, now: u64, after: Option<&CellKey>) -> Option<&PinnedPage> { + let page = self.0.as_deref()?; + if !page + .reference + .next_due_unix_millis + .is_some_and(|due| due <= now) + { + return None; + } + if after.is_none_or(|after| &page.key > after) { + if let Some(found) = page.left.find_due(now, after) { + return Some(found); + } + if page + .leaf + .effects + .next_due_unix_millis() + .is_some_and(|due| due <= now) + { + return Some(page); + } + } + page.right.find_due(now, after) + } + fn find(&self, key: &CellKey) -> Option<&Arc> { let mut cursor = &self.0; while let Some(page) = cursor { diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 5770782a5..790f975c9 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -103,7 +103,14 @@ impl GroupManifest { } } if let Some(root) = self.root.0 { - if root.cells == 0 || root.cells > 1_000_000 || root.height == 0 || root.height > 32 { + if root.cells == 0 + || root.cells > 1_000_000 + || root.height == 0 + || root.height > 32 + || root + .next_due_unix_millis + .is_some_and(|due| due > i64::MAX as u64) + { return Err(ManifestError::Invalid("root bounds")); } } @@ -214,6 +221,18 @@ impl PreparedGroupManifest { .get(&cell_routing_key(self.marker.route.history.namespace, cell)) } + /// Select one due cell from this immutable physical closure without I/O, + /// wrapping after the routing-key cursor. Like `pin_cell`, this is a data + /// view only; it grants no execution, leadership or delivery permission. + pub fn next_due( + &self, + now_unix_millis: u64, + after: Option<&CellId>, + ) -> Option<(&CellManifestLeaf, CellImagePin)> { + let after = after.map(|cell| cell_routing_key(self.marker.route.history.namespace, cell)); + self.images.next_due(now_unix_millis, after.as_ref()) + } + pub(crate) fn require_checkpoint( &self, group: &crate::raft::cache::GroupLog, From dfdeb983f06767fadc4215d56b37b7ad724fc3ba Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 01:53:02 -0700 Subject: [PATCH 261/568] feat: run effects on shared cell groups --- .../durability/raft/cell_groups/README.md | 77 ++- .../durability/raft/cell_groups/apply/mod.rs | 16 + .../raft/cell_groups/apply/summary.rs | 48 +- .../raft/cell_groups/apply/summary_tests.rs | 27 +- .../raft/cell_groups/controller/commands.rs | 119 ++-- .../cell_groups/controller/effects/mod.rs | 325 +++++++++ .../cell_groups/controller/effects/turn.rs | 349 ++++++++++ .../raft/cell_groups/controller/mod.rs | 24 + .../controller/provision/manager.rs | 8 + .../raft/cell_groups/controller/worker.rs | 14 + .../raft/cell_groups/dispatch/command/mod.rs | 101 ++- .../cell_groups/dispatch/command/remote.rs | 5 + .../raft/cell_groups/dispatch/mod.rs | 29 + .../raft/cell_groups/execution_authority.rs | 75 ++ .../cellar/durability/raft/cell_groups/mod.rs | 5 +- .../durability/raft/cell_groups/owner.rs | 18 + .../raft/cell_groups/runtime/mod.rs | 21 + .../raft/cell_groups/runtime/worker.rs | 4 + .../cellar/durability/raft/effect_outbox.rs | 3 +- crates/cellar/durability/raft/effect_scan.rs | 32 + .../durability/raft/effect_scan_tests.rs | 71 ++ .../cellar/durability/raft/execution/cell.rs | 54 +- .../durability/raft/execution/enrollment.rs | 18 +- .../durability/raft/execution/images.rs | 22 + .../raft/host_sql/cell_groups/mod.rs | 54 +- .../raft/host_sql/cell_groups/query.rs | 79 ++- .../raft/host_sql/cell_groups/worker.rs | 13 +- .../tests/native_cell_groups_effects.rs | 85 +++ .../tests/native_cell_groups_execution.rs | 25 + .../tests/support/cell_group_effects.rs | 328 +++++++++ .../support/cell_group_engine_fixture.rs | 33 +- .../tests/support/cell_group_idle_cache.rs | 298 ++++++++ .../support/cell_group_outbox_effects.rs | 646 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 5 + crates/cellar/tests/support/cell_group_sql.rs | 13 + .../cell_group_sql_command_dispatch.rs | 6 +- .../support/cell_group_sql_registered.rs | 19 +- docs/design/multi-raft/cell-groups/README.md | 2 +- docs/design/multi-raft/cell-groups/handoff.md | 28 +- .../cell-groups/implementation-plan.md | 2 +- .../design/multi-raft/cell-groups/progress.md | 52 +- docs/design/multi-raft/cell-groups/tasks.json | 2 +- 42 files changed, 3032 insertions(+), 123 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs create mode 100644 crates/cellar/durability/raft/effect_scan_tests.rs create mode 100644 crates/cellar/tests/native_cell_groups_effects.rs create mode 100644 crates/cellar/tests/support/cell_group_effects.rs create mode 100644 crates/cellar/tests/support/cell_group_idle_cache.rs create mode 100644 crates/cellar/tests/support/cell_group_outbox_effects.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index f3710aabf..787ac94fa 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -25,8 +25,9 @@ The caller runs page I/O on its blocking execution domain and retains the selected verified root; returned metadata still requires fresh leader, route, image and private-command admission before execution. Missing scheduling fields refuse at marker 1, with -no compatibility reader. Node scheduler/system-command integration and the full -CG-08 effect/cache qualification remain unfinished. +no compatibility reader. The shared controller now integrates indexed discovery +and private system commands. Full CG-08 effect/cache qualification and the public +SDK/fleet cutover remain unfinished; qualification is scoped to its recorded source. Leaves bind logical cell history, mutation sequence, image commitment and geometry, last group installation/mutation position, schema, effect summary, @@ -515,6 +516,72 @@ archive observations cannot refund it. Reopen reconstructs debt from the selecte log, retaining entries even under a newly lowered admission limit. Mutations refuse until a reservation fits. No serialized field or per-cell worker is added. -The admission implementation and new boundary cases are under qualification. -Full-backend S3 outage authority, physical file amplification/reclamation, -sustained-load fairness and the original CG-06 task gates remain pending. +The source-scoped archive/reclamation and local disaster-recovery qualification +is recorded in the [implementation progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). +Sustained-load fairness and final SDK/fleet cutover have separate task gates. + + +## Per-cell effects on shared owners + +`CellGroupController::configure_effects` installs bounded compiled namespace/class +registrations once. Each `CellGroupEffectRegistration` supplies explicit +`RaftEffectOptions` and a synchronous SQL-options factory, executed on the owning +domain's blocking service. The factory derives private paths from the complete +cell history and shares bounded node resources; its captured configuration must +not grow with enrolled cells. Dispatcher, SQL, engine and namespace schema owners +must match. Configuration enforces private-input and metadata capacity, and each +selected SQL execution verifies class-specific image, receipt and retry bounds. + +One owned turn per node rotates ready groups and due cells. Its group queue is +initialized from the selected in-memory markers and updated only after synced +publication. Empty and future-only groups require no runtime probe or SQLite open. +Outbox summaries retain the minimum time among lane fronts; a blocked later +sequence with an earlier timestamp cannot keep a sleeping lane busy. Hints remain +advisory: the actual ready leader issues cell execution authority and a fresh +linearizable image supplies each private client sequence and due generation. + +Discovery runs on the existing SQL workers. A completed scan transfers its +payload into the effect metadata permit and releases SQL read admission before +staging a command, including when the node/group has one SQL slot. Alarm discovery +retains only its generation ID; its bounded payload stays in SQLite until the +private callback loads it. The private operation and reserved logical cell client +are checked before both new execution and duplicate lookup. Public and remote +commands cannot select the private execution path. The last committed private +receipt alternates alarm/outbox priority without a preference table for every cell. + +An outbox claim crosses ordinary quorum and requested S3 output settlement before +send. After consuming the claim result, the turn releases its response admission +before a loopback receiver or source completion needs the same bounded lane. +Published durability survives that consumer release. The durable Dispatched row, +bounded request metadata and fenced image custody retain the unknown outcome. +Delivery custody requires the same active route, authorized member and leader +term, and a current same-history cell version covering the committed claim. +Later business writes and unrelated apply lag do not revoke that custody. A term, +route, activation or lineage change does. The source rechecks custody before send +and reply acceptance, verifies the configured receiver signature, then replicates +conditional generation-matched completion through the normal durability gate. + +Cancellation does not retire admitted command producers. Closing stops new effect +admission, joins factories/scans/delivery waiters, then closes the existing physical +SQL/runtime owners. Pending callback work must finish before close succeeds; +cancelled external delivery preserves committed debt for retry. Controller usage +reports active turns, completed turns and retries; SQL usage separately reports +retained producers and operations. Root publication can precede those resources' +disposal, so inspection through a one-slot pool waits for their ownership to drain. +Settled materialized producers discard the private SQLite/WAL and LTX scratch +under their existing lock, retaining only its lock inode. Failed opens and revoked +post-open authority also discard scratch before releasing that lock. Cleanup is +bounded and its failure is reported. Sealed committed images, receipts and pending +proposal bytes remain independently owned. Extent workspaces retain their existing +bounded store ownership and joined release. + +The [native effect cases](../../../tests/native_cell_groups_effects.rs) exercise +due discovery among cold cells, exact committed-base reopening after disposable +state is removed, retained SQL/delivery ownership, and joined shutdown. Two +independent physical group bindings compare the exact signed request and receiver +proof while each loses a reply and suppresses duplicate receiver execution. Live +transfer sealing/publication belongs to CG-09. The S3 case requires claim coverage +before send, retains the receiver's S3 policy, and independently restores the final +complete image/receipt closure. It has an explicit local RustFS variant without a +memory fallback. Current results and remaining qualification are recorded in the +[implementation progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). diff --git a/crates/cellar/durability/raft/cell_groups/apply/mod.rs b/crates/cellar/durability/raft/cell_groups/apply/mod.rs index d1faf7abe..f28627b36 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/mod.rs @@ -115,6 +115,22 @@ impl CellGroupApplier { Ok(source) } + /// Advisory metadata from the already published in-memory marker. No page + /// or cell database is opened, and no serving authority is granted. + pub(crate) fn effect_hint( + &self, + ) -> Result<(cellar_logic::consensus::LogIndex, Option), ExecutionError> { + let marker = self + .group + .group_manifest() + .map_err(storage)? + .ok_or(ExecutionError::Invalid("effect group manifest missing"))?; + Ok(( + marker.position().index, + marker.root().next_due_unix_millis(), + )) + } + pub fn history(&self) -> GroupHistory { self.group.identity().scope } diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary.rs b/crates/cellar/durability/raft/cell_groups/apply/summary.rs index 3afe46b97..00bf3fdb9 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/summary.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/summary.rs @@ -35,7 +35,8 @@ pub(super) fn read( Ok((count, earliest)) }; let (_, next_alarm_unix_millis) = scan("__sys_raft_alarms", "due")?; - let (pending_outbox, next_outbox_unix_millis) = scan("__sys_raft_outbox", "next_at")?; + let (pending_outbox, _) = scan("__sys_raft_outbox", "next_at")?; + let next_outbox_unix_millis = outbox_front(connection, pending_outbox)?; Ok(EffectSummary { next_alarm_unix_millis, next_outbox_unix_millis, @@ -46,6 +47,51 @@ pub(super) fn read( }) } +fn outbox_front( + connection: &rusqlite::Connection, + count: u64, +) -> Result, ExecutionError> { + if count == 0 { + return Ok(None); + } + // The preceding scan has already bounded every row and validated its time. + // Read fixed-size metadata only; no payload allocation or unbounded sort. + // Later deliveries cannot run before their lane's oldest sequence, even + // when those later rows have next_at=0. Scheduling that zero spins on an + // otherwise sleeping cell until its dispatched head becomes retryable. + let mut statement = connection.prepare( + "SELECT CASE WHEN typeof(lane)='text' AND length(CAST(lane AS BLOB)) BETWEEN 1 AND 64 THEN lane ELSE NULL END,CASE WHEN typeof(sequence)='blob' AND length(sequence)=8 THEN sequence ELSE NULL END,next_at FROM __sys_raft_outbox LIMIT ?1", + ).map_err(storage)?; + let mut rows = statement.query([count]).map_err(storage)?; + let mut fronts = std::collections::BTreeMap::::new(); + while let Some(row) = rows.next().map_err(storage)? { + let lane: String = row.get(0).map_err(storage)?; + let sequence: Vec = row.get(1).map_err(storage)?; + let sequence = u64::from_be_bytes( + sequence + .try_into() + .map_err(|_| ExecutionError::Invalid("effect outbox sequence"))?, + ); + let next: u64 = row.get(2).map_err(storage)?; + if sequence == 0 || !crate::raft::effects::delivery::valid_name(&lane) { + return Err(ExecutionError::Invalid("effect outbox identity")); + } + if !fronts.contains_key(&lane) && fronts.len() == crate::raft::effects::outbox::MAX_LANES { + return Err(ExecutionError::Invalid("effect outbox lane bounds")); + } + let front = fronts.entry(lane).or_insert((sequence, next)); + if sequence < front.0 { + *front = (sequence, next); + } + } + let next = fronts + .values() + .map(|(_, next)| *next) + .min() + .ok_or(ExecutionError::Invalid("effect outbox row count changed"))?; + Ok(Some(next)) +} + #[cfg(test)] #[path = "summary_tests.rs"] mod tests; diff --git a/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs index b945a6002..552bcdfc8 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/summary_tests.rs @@ -6,7 +6,7 @@ use super::*; fn effects_are_bounded_and_inbox_receipts_do_not_schedule_work() { let db = rusqlite::Connection::open_in_memory().unwrap(); assert_eq!(read(&db, 2).unwrap(), EffectSummary::default()); - db.execute_batch("CREATE TABLE __sys_raft_inbox(sequence); INSERT INTO __sys_raft_inbox VALUES(1); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(20),(0); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(30),(10);").unwrap(); + db.execute_batch("CREATE TABLE __sys_raft_inbox(sequence); INSERT INTO __sys_raft_inbox VALUES(1); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(20),(0); CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); INSERT INTO __sys_raft_outbox VALUES('A',x'0000000000000001',30),('B',x'0000000000000001',10);").unwrap(); assert_eq!( read(&db, 2).unwrap(), EffectSummary { @@ -19,6 +19,31 @@ fn effects_are_bounded_and_inbox_receipts_do_not_schedule_work() { assert!(read(&db, 1).is_err()); } +#[test] +fn blocked_later_delivery_does_not_wake_a_sleeping_lane() { + let db = rusqlite::Connection::open_in_memory().unwrap(); + db.execute_batch( + "CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); + INSERT INTO __sys_raft_outbox VALUES + ('Events',x'0000000000000002',0), + ('Events',x'0000000000000001',10000), + ('Alerts',x'0000000000000001',300);", + ) + .unwrap(); + let summary = read(&db, 3).unwrap(); + assert_eq!(summary.pending_outbox, 3); + assert_eq!(summary.next_outbox_unix_millis, Some(300)); + db.execute("DELETE FROM __sys_raft_outbox WHERE lane='Alerts'", []) + .unwrap(); + assert_eq!(read(&db, 3).unwrap().next_outbox_unix_millis, Some(10000)); + db.execute( + "DELETE FROM __sys_raft_outbox WHERE sequence=x'0000000000000001'", + [], + ) + .unwrap(); + assert_eq!(read(&db, 3).unwrap().next_outbox_unix_millis, Some(0)); +} + #[test] fn malformed_effect_state_refuses_instead_of_reporting_idle() { for ddl in [ diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs index 7f42dd903..cac797f40 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/commands.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -89,66 +89,73 @@ impl CellGroupController { pub fn command( &self, invocation: CellCommandInvocation, - mut options: CellSqlOptions, + options: CellSqlOptions, ) -> Result { - let shared = &self.handle.0; - let admitted = (|| { - let (command, deadline, bytes) = invocation.controller_admission(&shared.domain)?; - if shared.closing.load(Ordering::Acquire) { - return Err(error(command, false, RuntimeError::Closed)); - } - let bytes = bytes - .checked_add(option_bytes(&mut options).map_err(|e| error(command, false, e))?) - .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; - let permit = shared - .commands - .acquire(bytes) - .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; - Ok((command, deadline, permit)) - })(); - let (command, deadline, permit) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - return Err(CellCommandAdmissionError { - error, - invocation: Box::new(invocation), - }) - } - }; - let (reply, receiver) = oneshot::channel(); - shared - .requests - .try_send(requests::Request::Command(Box::new(Request { - invocation, - options, - reply, - permit: Arc::new(permit), - }))) - .map_err(|e| { - let (request, reason) = match e { - mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), - mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), - }; - CellCommandAdmissionError { - error: error(command, false, reason), - invocation: match request { - requests::Request::Command(request) => Box::new(request.invocation), - _ => unreachable!("command admission envelope"), - }, - } - })?; - Ok(CellCommandCall { - inner: calls::Call { - receiver, - identity: command, - deadline, - domain: shared.domain.clone(), - failure: |id, reason| error(id, true, reason), - }, - }) + admit(&self.handle.0, invocation, options) } } +pub(super) fn admit( + shared: &Arc, + invocation: CellCommandInvocation, + mut options: CellSqlOptions, +) -> Result { + let admitted = (|| { + let (command, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let bytes = bytes + .checked_add(option_bytes(&mut options).map_err(|e| error(command, false, e))?) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + Ok((command, deadline, permit)) + })(); + let (command, deadline, permit) = match admitted { + Ok(admitted) => admitted, + Err(error) => { + return Err(CellCommandAdmissionError { + error, + invocation: Box::new(invocation), + }) + } + }; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Command(Box::new(Request { + invocation, + options, + reply, + permit: Arc::new(permit), + }))) + .map_err(|e| { + let (request, reason) = match e { + mpsc::error::TrySendError::Closed(request) => (request, RuntimeError::Closed), + mpsc::error::TrySendError::Full(request) => (request, RuntimeError::Overloaded), + }; + CellCommandAdmissionError { + error: error(command, false, reason), + invocation: match request { + requests::Request::Command(request) => Box::new(request.invocation), + _ => unreachable!("command admission envelope"), + }, + } + })?; + Ok(CellCommandCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) +} + pub(super) async fn drive( request: Request, runtime: CellGroupRuntime, diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs new file mode 100644 index 000000000..c5077988e --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs @@ -0,0 +1,325 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded node turn discovers committed work through the manifest index. +//! Cursors are hints per physical group; no cell owns a task, timer or cache. + +mod turn; + +use super::*; +use crate::application::consensus::RaftClass; +use crate::raft::budget::Budget; +use crate::raft::cell_groups::{CellGroupDispatcher, CellSqlOptions}; +use crate::raft::effects::RaftEffectOptions; +use cellar_logic::{ + cell_group::{CellHistory, CellId, GroupHistory}, + consensus::{ClusterId, LogIndex, NamespaceId}, +}; +use futures_util::{future::BoxFuture, stream::FuturesUnordered, FutureExt, StreamExt}; +use std::collections::BTreeSet; +use std::ops::Bound::{Excluded, Included, Unbounded}; + +type SqlFactory = Arc Result + Send + Sync>; + +/// A compiled namespace's effect adapter and explicit execution resources. +/// The factory runs on the owning execution domain's blocking service. It must +/// derive private paths from the complete cell history and must not spawn work. +/// Its captured configuration must be bounded independently of cell count. +pub struct CellGroupEffectRegistration { + dispatcher: CellGroupDispatcher, + options: RaftEffectOptions, + sql: SqlFactory, +} + +impl CellGroupEffectRegistration { + pub fn new( + dispatcher: CellGroupDispatcher, + options: RaftEffectOptions, + sql: impl Fn(&CellHistory) -> Result + Send + Sync + 'static, + ) -> Result { + if !(10..=60_000).contains(&options.interval_ms) + || !(1..=60_000).contains(&options.operation_timeout_ms) + || options + .operation_timeout_ms + .checked_add(100) + .is_none_or(|n| n > options.retry_window_ms) + || options.retry_window_ms > 600_000 + || !(64 * 1024..=8 * 1024 * 1024).contains(&options.metadata_bytes) + { + return Err(CellGroupControllerError::Invalid("effect scheduler bounds")); + } + Ok(Self { + dispatcher, + options, + sql: Arc::new(sql), + }) + } +} + +struct Registration { + dispatcher: CellGroupDispatcher, + class: Arc, + schema: [u8; 32], + options: RaftEffectOptions, + memory: Arc, + sql: SqlFactory, +} + +pub(super) struct Config { + namespaces: BTreeMap<(ClusterId, NamespaceId), Arc>, +} + +impl CellGroupController { + /// Install the node's bounded compiled effect registrations once. All local + /// groups in these namespaces, including later registrations, participate. + /// Commands retain ordinary admission and S3 settlement. A scan or delivery + /// waiter cannot retire a pending command's physical SQL producer. + pub fn configure_effects( + &self, + registrations: Vec, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + if registrations.is_empty() || registrations.len() > shared.limits.max_groups { + return Err(CellGroupControllerError::Invalid("effect registrations")); + } + let mut namespaces = BTreeMap::new(); + for registration in registrations { + let (namespace, class) = registration + .dispatcher + .effect_binding(&shared.domain) + .map_err(|_| CellGroupControllerError::Invalid("effect dispatcher owner/class"))?; + // Alarm discovery retains only Fire{id}; the payload stays on the + // selected SQLite image until the bounded SQL callback loads it. + if class.outbox_definition().is_some() + && registration.options.metadata_bytes < 512 * 1024 + { + return Err(CellGroupControllerError::Invalid( + "effect metadata capacity", + )); + } + let dispatcher = registration + .dispatcher + .with_s3(self) + .map_err(|_| CellGroupControllerError::Invalid("effect archive owner"))?; + let key = (namespace.cluster, namespace.namespace); + let registration = Arc::new(Registration { + dispatcher, + class, + schema: namespace.schema_digest, + options: registration.options, + memory: Budget::new(registration.options.metadata_bytes, 1), + sql: registration.sql, + }); + if namespaces.insert(key, registration).is_some() { + return Err(CellGroupControllerError::Invalid( + "duplicate effect namespace", + )); + } + } + let mut slot = shared + .effects + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if slot.is_some() || shared.closing.load(Ordering::Acquire) { + return Err(CellGroupControllerError::Invalid( + "effect owner already configured/closed", + )); + } + *slot = Some(Arc::new(Config { namespaces })); + Ok(()) + } +} + +#[derive(Default)] +pub(super) struct Manager { + group: Option, + cells: BTreeMap, + hints: BTreeMap)>, + waiting: BTreeSet<(u64, GroupHistory)>, + ready: BTreeSet, + next_at: u64, + jobs: FuturesUnordered>, +} + +pub(super) struct Completed { + pub(super) group: GroupHistory, + cell: Option, + interval: u64, + result: Result<(), turn::Error>, +} + +impl Manager { + pub fn new( + roster: &BTreeMap, + ) -> Result { + let mut manager = Self::default(); + for applier in roster.values() { + manager.register(applier)?; + } + Ok(manager) + } + + pub fn register(&mut self, applier: &CellGroupApplier) -> Result<(), CellGroupControllerError> { + let (position, next) = applier + .effect_hint() + .map_err(|error| CellGroupControllerError::Apply(Arc::new(error)))?; + self.observe(applier.history(), position, next) + } + + pub fn observe( + &mut self, + group: GroupHistory, + position: LogIndex, + next: Option, + ) -> Result<(), CellGroupControllerError> { + if let Some((old_position, old_next)) = self.hints.get(&group) { + if position < *old_position { + return Ok(()); + } + if position == *old_position && next != *old_next { + return Err(CellGroupControllerError::Invalid( + "conflicting effect hint at one apply boundary", + )); + } + if let Some(old_next) = old_next { + self.waiting.remove(&(*old_next, group)); + } + } + self.ready.remove(&group); + self.hints.insert(group, (position, next)); + if let Some(next) = next { + self.waiting.insert((next, group)); + } + Ok(()) + } + + pub fn remove(&mut self, group: &GroupHistory) { + self.cells.remove(group); + self.ready.remove(group); + if let Some((_, Some(at))) = self.hints.remove(group) { + self.waiting.remove(&(at, *group)); + } + } + + pub fn running(&self) -> bool { + !self.jobs.is_empty() + } + + pub fn launch( + &mut self, + quarantined: &BTreeSet, + shared: &Arc, + now: u64, + ) -> Result<(), CellGroupControllerError> { + if self.running() || now < self.next_at || shared.closing.load(Ordering::Acquire) { + return Ok(()); + } + let Some(config) = shared + .effects + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .clone() + else { + return Ok(()); + }; + let Ok(wall) = u64::try_from(crate::asyncrt::wall_ms()) else { + return Ok(()); + }; + // Work scales with due groups. Empty and future-only groups incur no + // runtime RPC or SQLite open. Promotion is bounded per node turn. + for _ in 0..64 { + let Some((at, group)) = self.waiting.first().copied() else { + break; + }; + if at > wall { + break; + } + self.waiting.remove(&(at, group)); + self.ready.insert(group); + } + let after = self.group.map_or(Unbounded, Excluded); + let through = self.group.map_or(Unbounded, Included); + // Rotate ready groups rather than repeatedly choosing the oldest due + // timestamp. A hot cell/group cannot starve another due group. + for group in self + .ready + .range((after, Unbounded)) + .chain(self.ready.range((Unbounded, through))) + .take(4) + { + self.group = Some(*group); + if quarantined.contains(group) || !shared.groups.contains(group)? { + continue; + } + let Some(registration) = config.namespaces.get(&(group.cluster, group.namespace)) + else { + continue; + }; + let Some(reservation) = registration + .memory + .acquire(registration.options.metadata_bytes) + else { + continue; + }; + let group = *group; + let after = self.cells.get(&group).cloned(); + let registration = registration.clone(); + let shared = shared.clone(); + shared.active_effects.store(1, Ordering::Release); + self.jobs.push(Box::pin(async move { + let mut cell = None; + let result = std::panic::AssertUnwindSafe(turn::run( + group, + after, + ®istration, + &shared, + reservation, + &mut cell, + )) + .catch_unwind() + .await + .unwrap_or(Err(turn::Error::Invalid("effect adapter panicked"))); + Completed { + group, + cell, + interval: registration.options.interval_ms, + result, + } + })); + return Ok(()); + } + self.next_at = now.saturating_add(shared.limits.retry_interval_ms); + Ok(()) + } + + pub async fn next(&mut self) -> Option { + self.jobs.next().await + } + + pub fn complete(&mut self, completed: Completed, shared: &Shared) { + if let Some(cell) = completed.cell { + if self.hints.contains_key(&completed.group) { + self.cells.insert(completed.group, cell); + } + } + self.next_at = shared + .domain + .mono_ms() + .unwrap_or(u64::MAX) + .saturating_add(completed.interval); + shared.active_effects.store(0, Ordering::Release); + shared.effect_turns.fetch_add(1, Ordering::Relaxed); + if let Err(error) = completed.result { + shared.effect_retries.fetch_add(1, Ordering::Relaxed); + tracing::debug!(group = ?completed.group, %error, "effect turn requires committed-state reconciliation"); + } + } + + pub async fn join(&mut self, shared: &Shared) { + // Closing stops admission; already admitted commands stay owned by the + // controller. Poll scans/factories/sink waiters to joined completion + // before the controller closes SQL and releases its physical claims. + while let Some(completed) = self.next().await { + self.complete(completed, shared); + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs new file mode 100644 index 000000000..aeaa31dbf --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/turn.rs @@ -0,0 +1,349 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Every turn rereads committed state. Discovery hints never authorize sends. + +use super::*; +use crate::raft::cell_groups::{ + CellCommandRequest, CellCommandResponse, CellEffectAuthority, CellReadConsistency, + CellRoutingLimits, +}; +use crate::raft::effects::{outbox, scan::DueEffect, RaftDeliveryRequest, RaftEffectSink}; +use cellar_logic::consensus::{CommandFailure, CommandId, Durability}; +use std::time::Duration; + +#[derive(Debug)] +pub(super) enum Error { + Runtime(RuntimeError), + Execution(ExecutionError), + Command(Arc), + Effect(CommandFailure), + Invalid(&'static str), +} + +impl std::fmt::Display for Error { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Runtime(e) => std::fmt::Display::fmt(e, f), + Self::Execution(e) => std::fmt::Display::fmt(e, f), + Self::Command(e) => std::fmt::Display::fmt(e, f), + Self::Effect(e) => std::fmt::Display::fmt(e, f), + Self::Invalid(e) => f.write_str(e), + } + } +} + +fn open(shared: &Shared) -> Result<(), Error> { + if shared.closing.load(Ordering::Acquire) { + Err(Error::Runtime(RuntimeError::Closed)) + } else { + Ok(()) + } +} + +fn clock(shared: &Shared) -> Result { + shared + .domain + .mono_ms() + .map_err(|_| Error::Runtime(RuntimeError::Closed)) +} + +fn wall() -> Result { + u64::try_from(crate::asyncrt::wall_ms()).map_err(|_| Error::Runtime(RuntimeError::Closed)) +} + +fn effect(error: anyhow::Error) -> Error { + Error::Effect( + error + .downcast_ref::() + .cloned() + .unwrap_or(CommandFailure::CorruptState), + ) +} + +pub(super) async fn run( + group: GroupHistory, + after: Option, + registration: &Registration, + shared: &Arc, + reservation: crate::raft::budget::Permit, + cursor: &mut Option, +) -> Result<(), Error> { + open(shared)?; + let Some(execution) = shared + .runtime + .next_effect(group, wall()?, after) + .await + .map_err(Error::Runtime)? + else { + return Ok(()); + }; + if execution.route().schema_digest != registration.schema { + return Err(Error::Invalid("effect group schema")); + } + let cell = execution.cell().version.history.clone(); + *cursor = Some(cell.cell.clone()); + // Retain this coherent image and original leader fence through delivery. + let sql = shared + .sql + .upgrade() + .ok_or(Error::Runtime(RuntimeError::Closed))?; + let factory = registration.sql.clone(); + let factory_cell = cell.clone(); + let options = shared + .domain + .blocking(move || factory(&factory_cell)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))? + .await + .map_err(|_| Error::Runtime(RuntimeError::Closed))? + .map_err(Error::Execution)?; + options.limits().validate().map_err(Error::Execution)?; + let limits = options.limits(); + if registration.options.retry_window_ms > limits.max_retry_window_ms { + return Err(Error::Invalid( + "effect retry window exceeds execution limit", + )); + } + if let Some(alarms) = registration.class.alarm_limits() { + if limits.envelope.max_payload_bytes < 1024 + || limits.max_client_identities < 2 + || limits.max_retained_receipts < 2 + || alarms + .max_payload_bytes + .checked_mul(alarms.max_alarms) + .and_then(|n| n.checked_add(128 * 1024)) + .is_none_or(|n| n as u64 > limits.max_image_bytes) + { + return Err(Error::Invalid("alarm execution capacity")); + } + } + if let Some(outbox) = registration.class.outbox_definition() { + if limits.envelope.max_payload_bytes < 32 * 1024 + || limits.max_result_bytes < 33 + crate::raft::effects::delivery::MAX_REQUEST + || limits.max_client_identities < 2 + || limits.max_retained_receipts < 3 + || (outbox.limits.max_total_bytes as u64) + .checked_add(128 * 1024) + .is_none_or(|n| n > limits.max_image_bytes) + { + return Err(Error::Invalid("effect execution capacity")); + } + } + open(shared)?; + let deadline = clock(shared)? + .checked_add(registration.options.operation_timeout_ms) + .ok_or(Error::Invalid("effect deadline"))?; + let authority = shared + .runtime + .read_authority( + group, + cell.clone(), + CellReadConsistency::Linearizable, + deadline, + ) + .await + .map_err(Error::Runtime)?; + let scan = sql + .scan_effects( + options.clone(), + authority, + registration.class.clone(), + wall()?, + reservation, + ) + .await + .map_err(Error::Execution)?; + let crate::raft::effects::scan::EffectScan { + next, + due, + _reservation, + .. + } = *scan.into_scan(); + let _metadata = _reservation; + let Some(due) = due else { + return Ok(()); + }; + let minimum = due.minimum(); + let (operation, input) = due.command().map_err(effect)?; + let retry_until = wall()? + .checked_add(registration.options.retry_window_ms) + .ok_or(Error::Invalid("effect retry window"))?; + let receipt = command( + shared, + registration, + &cell, + options.clone(), + request(registration, next, &operation, input, minimum, retry_until)?, + ) + .await?; + match due { + DueEffect::Alarm(_) => check_result(receipt.receipt().receipt.result()), + DueEffect::Delivery { + selected, + definition, + .. + } => { + let Some(claimed) = outbox::claimed( + &definition, + &cell, + registration.schema, + &selected, + receipt.receipt().receipt.result(), + ) + .map_err(effect)? + else { + return Ok(()); + }; + let authority = execution + .effect_authority(&receipt.receipt().version, receipt.receipt().position.term) + .map_err(Error::Runtime)?; + // The committed claim has been consumed into bounded delivery + // metadata and fenced image custody. Release its response admission + // before a loopback receiver or source completion needs the same + // one-slot SQL/archive lane. Published S3 durability survives a + // consumer dropping its response; the durable Dispatched row stays. + drop(receipt); + let proof = deliver( + shared, + authority, + claimed.request.clone(), + claimed.sink, + registration.options.operation_timeout_ms, + ) + .await?; + claimed.request.verify_receipt(&proof).map_err(effect)?; + let input = + outbox::completion_input(selected, claimed.generation, &proof).map_err(effect)?; + let next = CommandId::new( + next.client, + next.sequence() + .checked_add(1) + .ok_or(Error::Effect(CommandFailure::Overloaded))?, + ) + .map_err(|_| Error::Effect(CommandFailure::Overloaded))?; + let receipt = command( + shared, + registration, + &cell, + options, + request( + registration, + next, + outbox::COMPLETE, + input, + minimum, + retry_until, + )?, + ) + .await?; + check_result(receipt.receipt().receipt.result()) + } + } +} + +fn check_result(result: &[u8]) -> Result<(), Error> { + if result == [0] || result == [1] { + Ok(()) + } else { + Err(Error::Effect(CommandFailure::CorruptState)) + } +} + +fn request( + registration: &Registration, + id: CommandId, + operation: &str, + input: Vec, + durability: Durability, + retry_until_ms: u64, +) -> Result { + let contract = registration + .class + .system_contract(operation) + .ok_or(Error::Invalid("effect system operation"))?; + Ok(CellCommandRequest { + id, + contract, + input, + durability, + retry_until_ms, + }) +} + +async fn command( + shared: &Arc, + registration: &Registration, + cell: &CellHistory, + options: CellSqlOptions, + request: CellCommandRequest, +) -> Result { + open(shared)?; + if wall()? >= request.retry_until_ms { + return Err(Error::Effect(CommandFailure::Unavailable)); + } + let invocation = registration + .dispatcher + .system_command( + cell.clone(), + request, + CellRoutingLimits { + timeout: Duration::from_millis(registration.options.operation_timeout_ms), + max_attempts: 4, + }, + ) + .await + .map_err(|error| Error::Command(Arc::new(error)))?; + let call = commands::admit(shared, invocation, options) + .map_err(|error| Error::Command(Arc::new(error.error)))?; + let response = call.response(); + tokio::pin!(response); + loop { + open(shared)?; + let timer = shared + .domain + .sleep_until(clock(shared)?.saturating_add(100)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + crate::asyncrt::select! { + result = &mut response => return result.map_err(Error::Command), + _ = timer => (), + } + } +} + +async fn deliver( + shared: &Shared, + authority: CellEffectAuthority, + request: RaftDeliveryRequest, + sink: Arc, + timeout: u64, +) -> Result, Error> { + let deadline = clock(shared)? + .checked_add(timeout) + .ok_or(Error::Invalid("delivery deadline"))?; + let mut send = None; + let mut proof = None; + loop { + open(shared)?; + if clock(shared)? >= deadline { + return Err(Error::Effect(CommandFailure::Unavailable)); + } + // Apply lag and unrelated business mutations do not revoke this + // committed dispatch. Term, route, activation, membership and cell + // lineage still must cover the claim before send and reply acceptance. + authority.check().map_err(Error::Runtime)?; + if let Some(proof) = proof { + return Ok(proof); + } + if send.is_none() { + send = Some(sink.deliver(request.clone())); + } + let timer = shared + .domain + .sleep_until(clock(shared)?.saturating_add(100).min(deadline)) + .map_err(|_| Error::Runtime(RuntimeError::Closed))?; + let pending = send.as_mut().ok_or(Error::Invalid("delivery waiter"))?; + crate::asyncrt::select_biased! { "source authority is rechecked before accepting a late delivery"; + _ = timer => (), + result = pending => { proof = Some(result.map_err(Error::Effect)?); } + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 91d4ad6c4..a5a56f516 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -15,7 +15,9 @@ pub use archive::{ mod checkpoints; pub use checkpoints::CellGroupCheckpointPolicy; mod commands; +mod effects; mod enrollments; +pub use effects::CellGroupEffectRegistration; mod failure; mod provision; mod registration; @@ -113,6 +115,11 @@ pub struct CellGroupControllerUsage { /// max_command_bytes. This lane cannot recursively consume command slots. pub retained_s3: usize, pub s3_bytes: usize, + /// At most one effect turn runs on the node. SQL and command admission are + /// shared with ordinary operations; configured metadata has its own permit. + pub active_effects: usize, + pub effect_turns: u64, + pub effect_retries: u64, } struct Shared { @@ -123,6 +130,10 @@ struct Shared { domain: crate::asyncrt::DomainToken, commands: Arc, provisioning: Mutex>>, + effects: Mutex>>, + active_effects: AtomicUsize, + effect_turns: AtomicU64, + effect_retries: AtomicU64, requests: tokio::sync::mpsc::Sender, archives: Mutex>, s3: Arc, @@ -249,6 +260,10 @@ impl CellGroupController { domain: domain.clone(), commands: commands::budget(limits), provisioning: Mutex::new(None), + effects: Mutex::new(None), + active_effects: AtomicUsize::new(0), + effect_turns: AtomicU64::new(0), + effect_retries: AtomicU64::new(0), requests, archives: Mutex::new(BTreeMap::new()), s3: commands::budget(limits), @@ -297,6 +312,12 @@ impl CellGroupController { .lock() .unwrap_or_else(|e| e.into_inner()) .take(); + shared + .effects + .lock() + .unwrap_or_else(|e| e.into_inner()) + .take(); + shared.active_effects.store(0, Ordering::Release); shared.active_checkpoints.store(0, Ordering::Release); shared.active_apply.store(0, Ordering::Release); shared.pending_apply.store(0, Ordering::Release); @@ -331,6 +352,9 @@ impl CellGroupController { }) .unwrap_or_default(); CellGroupControllerUsage { + active_effects: shared.active_effects.load(Ordering::Acquire), + effect_turns: shared.effect_turns.load(Ordering::Relaxed), + effect_retries: shared.effect_retries.load(Ordering::Relaxed), preparation_bytes, retained_preparations, provision_waiters, diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs index 1584925bd..b79eb75fd 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs @@ -29,6 +29,14 @@ pub(in super::super) enum Completed { ), } +impl Completed { + pub(in super::super) fn group(&self) -> GroupHistory { + match self { + Self::Prepared(group, _) | Self::Attached(group, _) => *group, + } + } +} + #[derive(Default)] pub(in super::super) struct Manager { entries: BTreeMap, diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index f1870cf3a..7a09a9ef2 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -47,6 +47,7 @@ pub(super) struct Worker { progress: tokio::sync::watch::Sender<()>, checkpoints: checkpoints::Manager, archives: archive::Manager, + effects: effects::Manager, _checkpoint_routes: Vec, } @@ -71,6 +72,7 @@ impl Worker { } } let archives = archive::Manager::new(shared.domain.clone()); + let effects = effects::Manager::new(&roster)?; Ok(Self { runtime, events, @@ -94,6 +96,7 @@ impl Worker { progress: tokio::sync::watch::channel(()).0, checkpoints: checkpoints::Manager::default(), archives, + effects, _checkpoint_routes: checkpoint_routes, }) } @@ -185,6 +188,7 @@ impl Worker { .map_err(|_| CellGroupControllerError::Closed)?; self.launch(now)?; self.archives.launch(&self.runtime, &self.shared, now); + self.effects.launch(&self.quarantined, &self.shared, now)?; let timer = domain .sleep_until(wake_at) .map_err(|_| CellGroupControllerError::Closed)?; @@ -206,13 +210,16 @@ impl Worker { if self.roster.insert(group, request.registration.applier.clone()).is_some() { return Err(CellGroupControllerError::Invalid("duplicate registered applier")); } + self.effects.register(&request.registration.applier)?; self.registrations.push(Box::pin(registration::drive(*request, self.runtime.clone(), self.transport.clone()))); } other => self.commands.push(Box::pin(requests::drive(other, self.runtime.clone(), self.sql.clone(), self.shared.clone(), self.progress.subscribe()))), } } Some(done) = self.provisioning.next(), if self.provisioning.is_running() => { + let group = done.group(); self.provisioning.complete(done, &self.shared, &self.transport, &mut self.roster, &mut self.routes)?; + if let Some(applier) = self.roster.get(&group) { self.effects.register(applier)?; } } Some(completed) = self.registrations.next(), if !self.registrations.is_empty() => { let group = completed.group; @@ -223,6 +230,7 @@ impl Worker { Err(error) => { if !self.quarantined.contains(&group) { self.roster.remove(&group); + self.effects.remove(&group); } if uncertain { return Err(error); } } @@ -270,6 +278,10 @@ impl Worker { } } Some(()) = self.hints.next(), if !self.hints.is_empty() => (), + Some(completed) = self.effects.next(), if self.effects.running() => { + if let Some(applier) = self.roster.get(&completed.group) { self.effects.register(applier)?; } + self.effects.complete(completed, &self.shared); + } completed = self.archives.next(), if self.archives.running() => { self.archives.complete(completed, &self.shared, now, false); self.progress.send_replace(()); @@ -331,6 +343,7 @@ impl Worker { } CellGroupEffect::Applied { manifest, .. } | CellGroupEffect::CheckpointInstalled { manifest } => { self.archives.applied(group, manifest.position().index); + self.effects.observe(group, manifest.position().index, manifest.root().next_due_unix_millis())?; // Only this post-sync event advances the published count. self.shared.applied.fetch_add(1, Ordering::Relaxed); self.progress.send_replace(()); @@ -390,6 +403,7 @@ impl Worker { while let Some(completed) = self.registrations.next().await { let _ = completed.finish(true); } + self.effects.join(&self.shared).await; self.provisioning.join().await; self.checkpoints.join().await; self.archives.finish(&self.shared).await; diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 39cb9d1e7..0acab5f2a 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -98,6 +98,7 @@ struct Origin { /// Dropping it never releases the SQL worker's unresolved physical producer. pub struct CellCommandInvocation { dispatcher: CellGroupDispatcher, + system: bool, cell: CellHistory, request: CellCommandRequest, inputs: CommandInputs, @@ -123,12 +124,35 @@ impl CellGroupDispatcher { } pub(crate) async fn command_with_inputs( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option, + ) -> Result { + self.command_compiled(cell, request, limits, cached, captured, false) + .await + } + + pub(crate) async fn system_command( + &self, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + ) -> Result { + self.command_compiled(cell, request, limits, None, None, true) + .await + } + + async fn command_compiled( &self, cell: CellHistory, mut request: CellCommandRequest, limits: CellRoutingLimits, cached: Option, captured: Option, + system: bool, ) -> Result { let error = |cause| CellCommandError { command: request.id, @@ -154,9 +178,24 @@ impl CellGroupDispatcher { RegistrationError::KindMismatch, ))); } - self.class - .validate_operation(&request.contract) - .map_err(|e| error(CellDispatchError::Registration(e)))?; + if system { + self.class.validate_system_operation(&request.contract) + } else { + self.class.validate_operation(&request.contract) + } + .map_err(|e| error(CellDispatchError::Registration(e)))?; + if system + && request.id.client + != crate::raft::effects::system_client(&cell).map_err(|failure| { + error(CellDispatchError::Execution(ExecutionError::Refused( + failure, + ))) + })? + { + return Err(error(CellDispatchError::Execution( + ExecutionError::Refused(CommandFailure::UnsupportedCapability), + ))); + } let size = request .input .len() @@ -202,6 +241,7 @@ impl CellGroupDispatcher { .using_cache(self.routes.clone()); Ok(CellCommandInvocation { dispatcher: self.clone(), + system, cell, request, inputs, @@ -412,8 +452,8 @@ impl CellCommandInvocation { .as_ref() .ok_or(CellDispatchError::Invalid("missing origin"))?, ); - let outcome = sql - .stage_registered_owned( + let outcome = if self.system { + sql.stage_system_owned( options, authority, command, @@ -421,7 +461,18 @@ impl CellCommandInvocation { self.request.input.clone(), true, ) - .await; + .await + } else { + sql.stage_registered_owned( + options, + authority, + command, + self.dispatcher.class.clone(), + self.request.input.clone(), + true, + ) + .await + }; match outcome { Ok(outcome) => self.preparation(outcome).await, Err(failure) => { @@ -498,18 +549,32 @@ impl CellCommandInvocation { if authority.route() != &origin.route { return Err(CellDispatchError::Invalid("origin route binding")); } - let outcome = origin - .sql - .stage_registered( - origin.options.clone(), - authority, - self.command_attempt(origin), - self.dispatcher.class.clone(), - self.request.input.clone(), - false, - ) - .await - .map_err(CellDispatchError::Execution)?; + let outcome = if self.system { + origin + .sql + .stage_system_owned( + origin.options.clone(), + authority, + self.command_attempt(origin), + self.dispatcher.class.clone(), + self.request.input.clone(), + false, + ) + .await + } else { + origin + .sql + .stage_registered_owned( + origin.options.clone(), + authority, + self.command_attempt(origin), + self.dispatcher.class.clone(), + self.request.input.clone(), + false, + ) + .await + } + .map_err(|failure| CellDispatchError::Execution(failure.error))?; self.preparation(outcome).await } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs index 640090b88..afdef62a2 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -19,6 +19,11 @@ impl CellCommandInvocation { transport: &RaftTransport, member: MemberId, ) -> Result { + if self.system { + return Err(self.error(CellDispatchError::Invalid( + "private commands require local ownership", + ))); + } let attempt = self.current()?.clone(); self.session .validate_attempt(&attempt) diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 37d589531..2bdb87f98 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -161,6 +161,35 @@ impl CellGroupDispatcher { Ok(self) } + pub(crate) fn effect_binding( + &self, + domain: &DomainToken, + ) -> Result< + ( + crate::application::consensus::NamespaceConfig, + Arc, + ), + CellDispatchError, + > { + let required_input = if self.class.outbox_definition().is_some() { + 32 * 1024 + } else { + 1024 + }; + if !self.domain.same_owner(domain) + || !self.class.has_system_commands() + || self.limits.max_input_bytes < required_input + { + return Err(CellDispatchError::Invalid( + "effect dispatcher domain/class/input capacity", + )); + } + Ok(( + self.directory.namespace_config().clone(), + self.class.clone(), + )) + } + /// Retained namespace range hints, never per-cell execution or authority. pub fn route_cache_usage(&self) -> Result { self.routes.usage().map_err(catalog_failure) diff --git a/crates/cellar/durability/raft/cell_groups/execution_authority.rs b/crates/cellar/durability/raft/cell_groups/execution_authority.rs index a89d1f6ed..3b03ee635 100644 --- a/crates/cellar/durability/raft/cell_groups/execution_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/execution_authority.rs @@ -145,6 +145,81 @@ impl CellExecutionAuthority { } } +/// Custody of an already committed outbox claim. It retains the selected +/// image and leader fence while unrelated cells or later business writes may +/// advance. It grants neither SQL execution nor an acknowledgement. +pub(crate) struct CellEffectAuthority { + execution: CellExecutionAuthority, + minimum: cellar_logic::cell_group::CellVersion, +} + +impl CellExecutionAuthority { + pub(crate) fn effect_authority( + self, + minimum: &cellar_logic::cell_group::CellVersion, + term: Term, + ) -> Result { + // The execution observation was minted at a ready applied barrier. + // Its own cell has since advanced through the exact committed claim; + // checking equality with that older leaf would invalidate every claim. + if term != self.term + || !minimum + .covers(&self.leaf.version) + .map_err(|_| RuntimeError::Invalid("effect claim history".into()))? + { + return Err(RuntimeError::NotReady); + } + let authority = CellEffectAuthority { + execution: self, + minimum: minimum.clone(), + }; + authority.check()?; + Ok(authority) + } +} + +impl CellEffectAuthority { + pub(crate) fn check(&self) -> Result<(), RuntimeError> { + let execution = &self.execution; + if execution.leader_term.load(Ordering::Acquire) != execution.term.get() { + return Err(RuntimeError::NotReady); + } + let cache = execution.group.read()?; + let current = cache + .group_manifest + .as_ref() + .ok_or(RuntimeError::NotReady)?; + let key = cell_routing_key( + execution.route.history.namespace, + &self.minimum.history.cell, + ); + let (leaf, _pin) = cache + .manifest_pins + .as_ref() + .and_then(|pins| pins.get(&key)) + .ok_or(RuntimeError::NotReady)?; + if !current.is_active() + || current.route() != &execution.route + || cache.hard_state.term != execution.term.get() + || !cache + .conf_state + .voters + .iter() + .chain(&cache.conf_state.voters_outgoing) + .any(|id| *id == execution.group.identity().member.get()) + || !cache.membership.permits_group(execution.group.identity()) + || !leaf + .version + .covers(&self.minimum) + .map_err(|_| RuntimeError::NotReady)? + || execution.leader_term.load(Ordering::Acquire) != execution.term.get() + { + return Err(RuntimeError::NotReady); + } + Ok(()) + } +} + /// A durable current-term applied cut, including on a follower. This permits /// receipt inspection and retirement of older-term producers, never new SQL /// execution, a linearizable query, or an S3 acknowledgement. diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 9682d9198..8a2275515 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -43,8 +43,8 @@ pub use controller::{ CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, CellGroupArchiveStatus, CellGroupCheckpointPolicy, CellGroupController, CellGroupControllerError, CellGroupControllerLimits, CellGroupControllerUsage, - CellGroupProvisionCall, CellGroupProvisioning, CellGroupRegistration, - CellGroupRegistrationCall, CellGroupRegistrationError, + CellGroupEffectRegistration, CellGroupProvisionCall, CellGroupProvisioning, + CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; @@ -59,6 +59,7 @@ pub use dispatch::{ mod effect_index; mod execution_authority; +pub(crate) use execution_authority::CellEffectAuthority; pub use execution_authority::{CellExecutionAuthority, CellReconciliationAuthority}; mod enrollment_authority; diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index c45cd4610..e1b6a9f8e 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -356,6 +356,24 @@ impl CellGroupOwner { ) } + pub(crate) fn next_effect( + &self, + now: u64, + after: Option<&CellId>, + ) -> Result, RuntimeError> { + let status = self.status()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + if !status.ready || status.applied.index != status.committed { + return Err(RuntimeError::NotReady); + } + let Some((leaf, _pin)) = self.current.next_due(now, after) else { + return Ok(None); + }; + self.execution_authority(&leaf.version.history).map(Some) + } + pub fn manifest(&self) -> &PreparedGroupManifest { &self.current } diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 4df4bca93..60d1c9cb3 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -135,6 +135,10 @@ enum Command { Configure(CellGroupMembershipChange), ActivateRecovery(Box), Execution(cellar_logic::cell_group::CellHistory), + NextEffect { + now: u64, + after: Option, + }, Enrollment(cellar_logic::cell_group::CellHistory), EnrollmentState(cellar_logic::cell_group::CellHistory), Reconciliation(cellar_logic::cell_group::CellHistory), @@ -166,6 +170,7 @@ enum Response { RecoveryActivation(Box), Submission(crate::raft::execution::CellSubmissionOutcome), Execution(Box), + NextEffect(Option>), Enrollment(Box), EnrollmentState(Box), Reconciliation(Box), @@ -569,6 +574,22 @@ impl CellGroupRuntime { } } + pub(crate) async fn next_effect( + &self, + history: GroupHistory, + now: u64, + after: Option, + ) -> Result, RuntimeError> { + let bytes = after.as_ref().map_or(0, |cell| cell.as_str().len()) + 4096; + match self + .call(Some(history), Command::NextEffect { now, after }, bytes) + .await? + { + Response::NextEffect(authority) => Ok(authority.map(|authority| *authority)), + _ => Err(RuntimeError::Protocol("effect discovery reply".into())), + } + } + pub async fn reconciliation_authority( &self, history: GroupHistory, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index 01a9addcb..b9fa29293 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -404,6 +404,10 @@ impl Worker { .owner .execution_authority(&cell) .map(|authority| Response::Execution(Box::new(authority))), + Command::NextEffect { now, after } => group + .owner + .next_effect(now, after.as_ref()) + .map(|authority| Response::NextEffect(authority.map(Box::new))), Command::Enrollment(cell) => group .owner .enrollment_authority(cell) diff --git a/crates/cellar/durability/raft/effect_outbox.rs b/crates/cellar/durability/raft/effect_outbox.rs index 36ccbb44a..195781e34 100644 --- a/crates/cellar/durability/raft/effect_outbox.rs +++ b/crates/cellar/durability/raft/effect_outbox.rs @@ -16,6 +16,7 @@ use std::{collections::BTreeMap, sync::Arc}; pub(crate) const CLAIM: &str = "__sys_raft_outbox_claim"; pub(crate) const COMPLETE: &str = "__sys_raft_outbox_complete"; +pub(crate) const MAX_LANES: usize = 32; #[derive(Clone, Copy, Debug)] pub struct RaftOutboxLimits { @@ -80,7 +81,7 @@ impl Default for RaftOutboxLimits { impl RaftOutboxLimits { fn validate(self) -> Result<()> { ensure!( - (1..=32).contains(&self.max_lanes) + (1..=MAX_LANES).contains(&self.max_lanes) && (1..=128).contains(&self.max_pending) && (1..=delivery::MAX_PAYLOAD).contains(&self.max_payload_bytes) && (delivery::MAX_REQUEST..=16 * 1024 * 1024).contains(&self.max_total_bytes) diff --git a/crates/cellar/durability/raft/effect_scan.rs b/crates/cellar/durability/raft/effect_scan.rs index e8ecd4a6a..128872ce6 100644 --- a/crates/cellar/durability/raft/effect_scan.rs +++ b/crates/cellar/durability/raft/effect_scan.rs @@ -12,6 +12,10 @@ use cellar_logic::consensus::{CommandFailure, CommandId, Durability, LogPosition use rusqlite::OptionalExtension; use std::sync::Arc; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "effect_scan_tests.rs"] +mod tests; + pub(crate) enum DueEffect { Alarm(alarm::DueAlarm), Delivery { @@ -129,3 +133,31 @@ pub(crate) fn scan_connection( _reservation: reservation, }) } + +/// Alternate adapter preference using the last committed private operation. +/// Placement, process restarts and the number of other cells do not affect it. +pub(crate) fn scan_connection_fair( + connection: &rusqlite::Connection, + scope: &cellar_logic::cell_group::CellHistory, + position: LogPosition, + class: &RaftClass, + now: u64, + reservation: Permit, +) -> Result { + let client = super::system_client(scope)?; + let previous: Option = connection.query_row( + "SELECT CASE WHEN typeof(r.operation)='text' AND length(CAST(r.operation AS BLOB)) BETWEEN 1 AND 128 THEN r.operation ELSE NULL END FROM __sys_raft_clients c JOIN __sys_raft_receipts r ON r.client_id=c.client_id AND r.sequence=c.last_sequence WHERE c.client_id=?1", + [client.bytes().as_slice()], |row| row.get(0), + ).optional()?; + let prefer_delivery = + previous.is_some_and(|operation| operation.starts_with("__sys_raft_alarm_")); + scan_connection( + connection, + scope, + position, + class, + now, + prefer_delivery, + reservation, + ) +} diff --git a/crates/cellar/durability/raft/effect_scan_tests.rs b/crates/cellar/durability/raft/effect_scan_tests.rs new file mode 100644 index 000000000..b68ab5428 --- /dev/null +++ b/crates/cellar/durability/raft/effect_scan_tests.rs @@ -0,0 +1,71 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::budget::Budget; +use cellar_logic::{cell_group::*, consensus::*}; +use rusqlite::types::Value; + +fn scan_operation(operation: Value) -> Result<()> { + let database = rusqlite::Connection::open_in_memory()?; + database.execute_batch( + "CREATE TABLE __sys_raft_clients(client_id BLOB, last_sequence BLOB); + CREATE TABLE __sys_raft_receipts(client_id BLOB, sequence BLOB, operation);", + )?; + let history = CellHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: CellId::new("Counter:one")?, + incarnation: IncarnationId::new([3; 16])?, + }; + let client = super::super::system_client(&history)?; + let sequence = 1_u64.to_be_bytes(); + database.execute( + "INSERT INTO __sys_raft_clients VALUES(?1, ?2)", + rusqlite::params![client.bytes().as_slice(), sequence.as_slice()], + )?; + database.execute( + "INSERT INTO __sys_raft_receipts VALUES(?1, ?2, ?3)", + rusqlite::params![client.bytes().as_slice(), sequence.as_slice(), operation], + )?; + let memory = Budget::new(64 * 1024, 1); + let result = scan_connection_fair( + &database, + &history, + LogPosition { + index: LogIndex::new(1), + term: Term::new(1), + }, + &RaftClass::new("Counter", 1, [4; 32])?, + 1, + memory.acquire(64 * 1024).unwrap(), + ); + match result { + Ok(scan) => { + assert_eq!(scan.next, CommandId::new(client, 2)?); + assert!(scan.due.is_none()); + drop(scan); + assert_eq!(memory.usage(), (0, 0)); + Ok(()) + } + Err(error) => { + assert_eq!(memory.usage(), (0, 0)); + Err(error) + } + } +} + +#[test] +fn previous_private_operation_is_bounded_before_allocation() -> Result<()> { + scan_operation(Value::Text("__sys_raft_alarm_Wake".into()))?; + scan_operation(Value::Text("x".repeat(128)))?; + for invalid in [ + Value::Null, + Value::Integer(1), + Value::Blob(vec![1; 128]), + Value::Text(String::new()), + Value::Text("x".repeat(129)), + ] { + assert!(scan_operation(invalid).is_err()); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/execution/cell.rs b/crates/cellar/durability/raft/execution/cell.rs index 7b6359114..35e5a3f86 100644 --- a/crates/cellar/durability/raft/execution/cell.rs +++ b/crates/cellar/durability/raft/execution/cell.rs @@ -100,13 +100,21 @@ impl CellPrivateExecutor { .map_err(storage)?, _workspace: options.images.workspace().cloned(), }; - let working = WorkingImage::open_cell( + let working = match WorkingImage::open_cell( &options, authority.image(), cell_image_history_digest(&authority.cell().version.history), - ) - .map_err(storage)?; - authority.check().map_err(authority_failure)?; + ) { + Ok(working) => working, + Err(error) => { + options.discard_cell_scratch().map_err(storage)?; + return Err(storage(error)); + } + }; + if let Err(error) = authority.check() { + working.close_cell(&options).map_err(storage)?; + return Err(authority_failure(error)); + } Ok(Self { options, authority, @@ -126,6 +134,27 @@ impl CellPrivateExecutor { class: &RaftClass, input: &[u8], allow_new: bool, + ) -> Result { + self.stage_registered_kind(attempt, class, input, allow_new, false) + } + + pub(crate) fn stage_system_registered( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + ) -> Result { + self.stage_registered_kind(attempt, class, input, allow_new, true) + } + + fn stage_registered_kind( + &mut self, + attempt: CommandAttempt, + class: &RaftClass, + input: &[u8], + allow_new: bool, + system: bool, ) -> Result { let history = self.authority.cell().version.history.clone(); let contract = super::registered::validate( @@ -135,12 +164,23 @@ impl CellPrivateExecutor { &attempt.request, class, input, - false, + system, )?; let command = attempt.request.id; let durability = attempt.request.durability; self.stage(attempt, allow_new, |transaction| { - class.invoke_command(&contract, &history, command, durability, transaction, input) + if system { + class.invoke_system_command( + &contract, + &history, + command, + durability, + transaction, + input, + ) + } else { + class.invoke_command(&contract, &history, command, durability, transaction, input) + } }) } @@ -423,7 +463,7 @@ impl CellPrivateExecutor { pub fn close(mut self) -> Result<(), ExecutionError> { let store = self.options.images.root_store().cloned(); if let Some(working) = self.working.take() { - working.close().map_err(storage)?; + working.close_cell(&self.options).map_err(storage)?; } drop(self.options); drop(self._lock); diff --git a/crates/cellar/durability/raft/execution/enrollment.rs b/crates/cellar/durability/raft/execution/enrollment.rs index 76b5eed16..beeba3d1b 100644 --- a/crates/cellar/durability/raft/execution/enrollment.rs +++ b/crates/cellar/durability/raft/execution/enrollment.rs @@ -75,13 +75,21 @@ impl CellPrivateEnrollment { .map_err(storage)?, _workspace: options.images.workspace().cloned(), }; - let working = WorkingImage::open( + let working = match WorkingImage::open( &options, None, cell_image_history_digest(authority.state().cell()), - ) - .map_err(storage)?; - authority.check().map_err(authority_failure)?; + ) { + Ok(working) => working, + Err(error) => { + options.discard_cell_scratch().map_err(storage)?; + return Err(storage(error)); + } + }; + if let Err(error) = authority.check() { + working.close_cell(&options).map_err(storage)?; + return Err(authority_failure(error)); + } Ok(Self { options, authority, @@ -237,7 +245,7 @@ impl CellPrivateEnrollment { pub fn close(mut self) -> Result<(), ExecutionError> { let store = self.options.images.root_store().cloned(); if let Some(working) = self.working.take() { - working.close().map_err(storage)?; + working.close_cell(&self.options).map_err(storage)?; } drop(self.options); drop(self.lock); diff --git a/crates/cellar/durability/raft/execution/images.rs b/crates/cellar/durability/raft/execution/images.rs index 6f9143f2f..c91da2029 100644 --- a/crates/cellar/durability/raft/execution/images.rs +++ b/crates/cellar/durability/raft/execution/images.rs @@ -109,6 +109,20 @@ impl From for Options { } impl Options { + /// Only shared-cell private scratch is disposable here. Keep the lock inode + /// and sealed image stores; an unresolved producer still owns this directory + /// until its physical callback/settlement has joined. + pub(super) fn discard_cell_scratch(&self) -> anyhow::Result<()> { + if let Images::Materialized(_) = &self.images { + crate::raft::working::clean( + self.host.filesystem().as_ref(), + &self.directory, + &self.directory.join("working.sqlite"), + )?; + } + Ok(()) + } + pub fn materialized(&self) -> anyhow::Result { let Images::Materialized(images) = &self.images else { anyhow::bail!("materialized private storage required"); @@ -329,6 +343,14 @@ impl WorkingImage { } } + pub(super) fn close_cell(self, options: &Options) -> anyhow::Result<()> { + // Close SQLite/WAL capture before removing any path, including when its + // close reports an error. The caller still owns the private lock. + let closed = self.close(); + options.discard_cell_scratch()?; + closed + } + pub fn trim(&self, options: &Options, base: Option<&Published>) -> anyhow::Result<()> { match self { Self::Materialized(working) => working.trim( diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index a967a9ab6..78bc870c3 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -8,7 +8,7 @@ pub(in crate::raft::host_sql) use apply::ApplyRequest; pub use apply::CellGroupPreparedApply; mod query; mod worker; -pub use query::{CellSqlQueryResult, CellSqlReceiptLookup}; +pub use query::{CellSqlEffectScan, CellSqlQueryResult, CellSqlReceiptLookup}; use super::{Budget, Budgets, Envelope, Permit, Pool, ResourceObserver, Worker}; use crate::raft::cell_groups::{ @@ -85,7 +85,7 @@ pub enum CellSqlOptions { } impl CellSqlOptions { - fn limits(&self) -> ExecutionLimits { + pub(crate) fn limits(&self) -> ExecutionLimits { match self { Self::Materialized(o) => o.limits, Self::Extents(o) => o.limits, @@ -321,6 +321,7 @@ pub enum CellSqlOutcome { Rejected, Queried(CellSqlQueryResult), ReceiptLookedUp(CellSqlReceiptLookup), + EffectsScanned(CellSqlEffectScan), } pub enum CellSqlReconciliation { @@ -577,10 +578,30 @@ impl CellGroupSqlPool { input: Vec, allow_new: bool, ) -> Result { - if input.len() > options.limits().envelope.max_payload_bytes { - return Err(CellSqlStageFailure::known(overloaded())); - } - self.stage_owned( + self.stage_compiled_owned( + options, + authority, + attempt, + allow_new, + worker::Callback::Registered { + class, + input: input.into_boxed_slice(), + system: false, + }, + ) + .await + } + + pub(crate) async fn stage_system_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + class: Arc, + input: Vec, + allow_new: bool, + ) -> Result { + self.stage_compiled_owned( options, authority, attempt, @@ -588,11 +609,32 @@ impl CellGroupSqlPool { worker::Callback::Registered { class, input: input.into_boxed_slice(), + system: true, }, ) .await } + async fn stage_compiled_owned( + &self, + options: CellSqlOptions, + authority: CellExecutionAuthority, + attempt: CommandAttempt, + allow_new: bool, + callback: worker::Callback, + ) -> Result { + let worker::Callback::Registered { input, .. } = &callback else { + return Err(CellSqlStageFailure::known(ExecutionError::Invalid( + "compiled command callback", + ))); + }; + if input.len() > options.limits().envelope.max_payload_bytes { + return Err(CellSqlStageFailure::known(overloaded())); + } + self.stage_owned(options, authority, attempt, allow_new, callback) + .await + } + async fn stage_callback( &self, options: CellSqlOptions, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index ca70b54a8..c84e6af12 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -12,6 +12,11 @@ type VerificationCallback = Box< >; enum Callback { + Effects { + class: Arc, + now: u64, + reservation: crate::raft::budget::Permit, + }, Receipt { class: Arc, request: crate::raft::execution::CommandRequest, @@ -34,6 +39,22 @@ pub(super) struct Query { bytes: Arc, } +/// Opaque committed effect observation, charged to the node scheduler until +/// the complete claim/delivery/settlement turn releases it. +pub struct CellSqlEffectScan { + scan: Box, + _bytes: Arc, +} + +impl CellSqlEffectScan { + /// SQL discovery has joined. Its payload is already charged to the effect + /// metadata permit, so release the read slot before admitting the private + /// command. A group with one SQL slot must not recursively acquire it. + pub(crate) fn into_scan(self) -> Box { + self.scan + } +} + pub struct CellSqlQueryResult { result: Vec, observation: crate::raft::version::CellVersionObservation, @@ -89,6 +110,7 @@ impl CellSqlReceiptLookup { } enum Output { + Effects(Box), Query(Vec), Receipt(Option), } @@ -96,6 +118,9 @@ enum Output { impl Output { fn retained_bytes(&self) -> usize { match self { + // The scheduler's metadata permit accounts for scan payloads and + // remains owned by EffectScan, independently of SQL read admission. + Self::Effects(_) => 0, Self::Query(bytes) => bytes.len(), Self::Receipt(receipt) => receipt.as_ref().map_or(0, |r| r.result().len()), } @@ -111,6 +136,37 @@ impl Drop for Cancel { } impl CellGroupSqlPool { + pub(crate) async fn scan_effects( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + now: u64, + reservation: crate::raft::budget::Permit, + ) -> Result { + if authority.barrier().is_none() + || !class.has_system_commands() + || class.schema_digest() != authority.route().schema_digest + { + return Err(ExecutionError::Invalid("effect scan authority/class")); + } + match self + .query_callback( + options, + authority, + Callback::Effects { + class, + now, + reservation, + }, + ) + .await? + { + CellSqlOutcome::EffectsScanned(output) => Ok(output), + _ => Err(ExecutionError::Invalid("effect scan reply")), + } + } + /// Consume an invocation's coherent pin. Queries can coexist with an /// unresolved private writer but read only the selected committed image. /// The registered dispatcher must enforce operation contracts before here. @@ -229,7 +285,7 @@ impl CellGroupSqlPool { .map_err(crate::raft::execution::authority_failure)?; options.limits().validate()?; let input_bytes = match &callback { - Callback::Verification(_) => 0, + Callback::Verification(_) | Callback::Effects { .. } => 0, Callback::Registered { input, .. } | Callback::Receipt { input, .. } => input.len(), }; let bytes = self.inner.admission.reserve_read( @@ -257,7 +313,7 @@ impl CellGroupSqlPool { crate::asyncrt::select! { result = self.call(cell, worker::Action::Query(Box::new(query))) => { match result? { - output @ (CellSqlOutcome::Queried(_) | CellSqlOutcome::ReceiptLookedUp(_)) => { + output @ (CellSqlOutcome::Queried(_) | CellSqlOutcome::ReceiptLookedUp(_) | CellSqlOutcome::EffectsScanned(_)) => { if domain.mono_ms().map_err(|_| unavailable())? >= deadline { return Err(crate::raft::execution::storage(RuntimeError::DeadlineExceeded)); } @@ -358,6 +414,21 @@ pub(super) fn run( check(&query)?; let observation = query.authority.cell().observation(); let result = std::panic::catch_unwind(AssertUnwindSafe(|| match query.callback { + Callback::Effects { + class, + now, + reservation, + } => crate::raft::effects::scan::scan_connection_fair( + database.connection(), + &observation.version.history, + query.authority.cell().last_mutation, + &class, + now, + reservation, + ) + .map(Box::new) + .map(Output::Effects) + .map_err(crate::raft::execution::storage), Callback::Verification(callback) => callback(&database, &observation.version) .map(Output::Query) .map_err(crate::raft::execution::storage), @@ -412,6 +483,10 @@ pub(super) fn run( drop(query.authority); Bytes::shrink(&mut query.bytes, result.retained_bytes() + 4096)?; Ok(match result { + Output::Effects(scan) => CellSqlOutcome::EffectsScanned(CellSqlEffectScan { + scan, + _bytes: query.bytes, + }), Output::Query(result) => CellSqlOutcome::Queried(CellSqlQueryResult { result, observation, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs index 0dd6334aa..a12f44301 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/worker.rs @@ -11,6 +11,7 @@ pub(super) enum Callback { Registered { class: Arc, input: Box<[u8]>, + system: bool, }, } @@ -188,8 +189,16 @@ impl Cells { Callback::Verification(callback) => { executor.stage(attempt, allow_new, callback) } - Callback::Registered { class, input } => { - executor.stage_registered(attempt, &class, &input, allow_new) + Callback::Registered { + class, + input, + system, + } => { + if system { + executor.stage_system_registered(attempt, &class, &input, allow_new) + } else { + executor.stage_registered(attempt, &class, &input, allow_new) + } } }; match result { diff --git a/crates/cellar/tests/native_cell_groups_effects.rs b/crates/cellar/tests/native_cell_groups_effects.rs new file mode 100644 index 000000000..32db89f98 --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_effects.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared-group effect scheduling and disposable execution resource lifetimes. +#![allow(clippy::disallowed_methods)] +// The shared engine fixture has helpers selected by sibling native targets. +#![allow(dead_code, unused_imports)] + +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +mod images; +#[path = "support/cell_group_runtime.rs"] +mod runtime_fixture; +#[path = "support/cell_group_rustfs.rs"] +mod rustfs; + +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn effect_identity_stable_across_group_binding() -> Result<()> { + Box::pin(runtime_fixture::effect_identity_across_bindings()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn idle_cache_reopens_exact_cell_base() -> Result<()> { + Box::pin(runtime_fixture::idle_cache_reopens_exact_base()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn due_effect_without_client_request() -> Result<()> { + Box::pin(runtime_fixture::alarms_without_clients()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn hot_due_cell_rotates_without_hydrating_cold_cells() -> Result<()> { + Box::pin(runtime_fixture::hot_and_cold_alarms()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn pending_command_or_dispatch_pins_cache() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::routed_commands(rooted)).await?; + } + Box::pin(runtime_fixture::effect_leader_change()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn cache_and_effect_shutdown_joins() -> Result<()> { + Box::pin(runtime_fixture::shutdown_joins_alarm()).await?; + Box::pin(runtime_fixture::effect_delivery_shutdown()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn receiver_lost_reply_keeps_exact_delivery_identity() -> Result<()> { + Box::pin(runtime_fixture::effect_lost_reply()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn s3_claim_receiver_and_completion_use_verified_coverage() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::effect_s3_lost_reply( + rooted, + Arc::new(object_store::memory::InMemory::new()), + )) + .await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +#[ignore = "requires isolated local RustFS"] +async fn rustfs_claim_receiver_and_completion_use_verified_coverage() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::effect_s3_lost_reply( + rooted, + rustfs::store("cell-group-effects", rooted)?, + )) + .await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index fd04a00d7..556c63dff 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -17,6 +17,31 @@ mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] mod checkpoint_fixture; +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_retries_lost_receiver_receipt() -> Result<()> { + Box::pin(runtime_fixture::effect_lost_reply()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_leader_change_preserves_dispatch() -> Result<()> { + Box::pin(runtime_fixture::effect_leader_change()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_outbox_shutdown_joins_delivery() -> Result<()> { + Box::pin(runtime_fixture::effect_delivery_shutdown()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_shutdown_joins_held_alarm_callback() -> Result<()> { + Box::pin(runtime_fixture::shutdown_joins_alarm()).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn controller_effects_alarms_run_without_client_traffic() -> Result<()> { + Box::pin(runtime_fixture::alarms_without_clients()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn runtime_archive_capture_retains_cut_without_blocking_quorum() -> Result<()> { Box::pin(runtime_fixture::capture_archive_cut()).await diff --git a/crates/cellar/tests/support/cell_group_effects.rs b/crates/cellar/tests/support/cell_group_effects.rs new file mode 100644 index 000000000..e21e09832 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_effects.rs @@ -0,0 +1,328 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! The production node controller owns all timers, transport and apply work. +//! No fixture event pumping or client traffic drives an armed alarm to commit. + +use super::*; +use cellar::application::consensus::{RaftAlarmLimits, RaftEffectOptions}; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn alarms_without_clients() -> Result<()> { + alarms(false, false).await +} + +pub(crate) async fn shutdown_joins_alarm() -> Result<()> { + alarms(true, false).await +} + +pub(crate) async fn hot_and_cold_alarms() -> Result<()> { + alarms(false, true).await +} + +async fn alarms(hold_callback: bool, rearm_hot: bool) -> Result<()> { + let _ = tracing_subscriber::fmt() + .with_env_filter("cellar::raft::cell_groups::controller::effects=debug") + .with_test_writer() + .try_init(); + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let hot_calls = Arc::new(AtomicUsize::new(0)); + let hot = hot_calls.clone(); + let hot_at_other = Arc::new(AtomicUsize::new(usize::MAX)); + let fairness = hot_at_other.clone(); + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Arc::new(std::sync::Mutex::new(Some(entered))); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Arc::new(std::sync::Mutex::new(held)); + let class = Arc::new( + RaftClass::new("Counter", 1, [if rearm_hot { 225 } else { 201 }; 32])? + .alarms(RaftAlarmLimits { + max_alarms: 2, + max_payload_bytes: 128, + })? + .alarm("Wake", [202; 32], move |context, input| { + assert_eq!(input, b"wake"); + observed.fetch_add(1, Ordering::SeqCst); + if hold_callback { + let _ = entered + .lock() + .unwrap() + .take() + .context("one callback")? + .send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + } + if rearm_hot { + if context.identity().cell.as_str() == "registered-a" { + let count = hot.fetch_add(1, Ordering::SeqCst) + 1; + if count < 4 { + context.set_alarm( + "Wake", + context.inputs().now_unix_millis, + b"wake", + )?; + } + } else { + fairness.store(hot.load(Ordering::SeqCst), Ordering::SeqCst); + } + } + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(()) + })? + .command("Arm", [203; 32], |context, _| { + context.set_alarm("Wake", context.inputs().now_unix_millis + 50, b"wake")?; + Ok(b"armed".to_vec()) + })?, + ); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let mut cold = Vec::new(); + let empty = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + for ordinal in 0..32 { + let cell = history(&route, &format!("cold-{ordinal}"))?; + let accepted = fixture.runtimes[0] + .propose( + route.history, + apply::enrollment_bytes(&route, &cell, &empty, rooted)?, + ) + .await?; + fixture.drive(Some(accepted.position.index)).await?; + cold.push(cell); + } + let keys = VersionKeys::new( + VersionSigningKey::new([204; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 32 * 1024, + }, + )?; + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + for cell in [&a, &b].into_iter().take(if hold_callback { 1 } else { 2 }) { + let mut invocation = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([205; 16])?, 1)?, + contract: class.contract("Arm").context("Arm")?, + input: Vec::new(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + invocation.begin().await?; + assert!(matches!( + invocation + .prepare( + &fixture.runtimes[0], + &fixture.appliers[&(0, route.history)].0, + fixture.cell_options(cell).await? + ) + .await?, + CellCommandPreparation::Prepared + )); + let accepted = command_dispatch::accepted(&mut invocation).await?; + fixture.drive(Some(accepted.position.index)).await?; + let receipt = command_dispatch::committed(&mut invocation).await?; + assert_eq!(receipt.receipt().receipt.result(), b"armed"); + } + // Seed already committed due work before starting the node controller. + // Its initial marker hints, not another client request or fixture pump, + // must discover both cells; one-slot admission stays unchanged. + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + assert!(fixture.events.is_empty()); + let cold_opens = Arc::new(AtomicUsize::new(0)); + for (node, controller) in controllers.iter().enumerate() { + let mut resources = BTreeMap::new(); + for cell in [&a, &b] { + resources.insert(cell.clone(), fixture.cell_options_at(cell, node).await?); + } + let unexpected = cold_opens.clone(); + controller.configure_effects(vec![CellGroupEffectRegistration::new( + dispatcher.clone(), + RaftEffectOptions { + metadata_bytes: 64 * 1024, + ..Default::default() + }, + move |cell| { + resources.get(cell).cloned().ok_or_else(|| { + unexpected.fetch_add(1, Ordering::SeqCst); + ExecutionError::Invalid("cold cell must not open for discovery") + }) + }, + )?])?; + } + if hold_callback { + tokio::time::timeout(Duration::from_secs(5), started) + .await + .with_context(|| { + format!( + "alarm callback never entered; usage={:?}; calls={}", + controllers + .iter() + .map(|controller| controller.usage()) + .collect::>(), + calls.load(Ordering::SeqCst) + ) + })??; + assert_eq!(calls.load(Ordering::SeqCst), 1); + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert!( + tokio::time::timeout(Duration::from_millis(30), controllers[0].close()) + .await + .is_err() + ); + assert_eq!(pool.usage()?.active_cells, 1); + release.send(())?; + controllers[0].close().await?; + // Closing waited for the actual SQL callback. It did not publish + // the unsubmitted private write or erase the durable alarm debt. + for runtime in fixture.runtimes.iter().skip(1) { + let selected = runtime + .reconciliation_authority(route.history, a.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), 2); + assert!(selected.cell().effects.next_alarm_unix_millis.is_some()); + } + } else { + // Only observations follow. The controller's durable index and shared + // SQL/consensus workers must discover and execute both cells unaided. + let completed = controller::wait_for(async || { + for runtime in &fixture.runtimes { + for cell in [&a, &b] { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + let sequence = if rearm_hot && cell == &a { 6 } else { 3 }; + if selected.cell().version.sequence.get() != sequence + || selected.cell().effects.next_due_unix_millis().is_some() + { + return Ok(false); + } + } + } + // Root publication precedes physical producer/response disposal. + // Inspect through the same one-slot pool only after ownership + // has drained; keep the original overall progress deadline. + for controller in &controllers { + let usage = controller.usage(); + if usage.active_effects != 0 || usage.retained_commands != 0 { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 || usage.retained_operations != 0 { + return Ok(false); + } + } + Ok(true) + }) + .await; + if let Err(error) = completed { + let status = fixture.runtimes[0].status(route.history).await; + let alpha = fixture.runtimes[0] + .reconciliation_authority(route.history, a.clone()) + .await + .map(|authority| (authority.cell().version.clone(), authority.cell().effects)); + let beta = fixture.runtimes[0] + .reconciliation_authority(route.history, b.clone()) + .await + .map(|authority| (authority.cell().version.clone(), authority.cell().effects)); + return Err(error.context(format!("alarm progress: status={status:?}; alpha={alpha:?}; beta={beta:?}; usage={:?}; callbacks={}", controllers.iter().map(|controller| controller.usage()).collect::>(), calls.load(Ordering::SeqCst)))); + } + assert_eq!(calls.load(Ordering::SeqCst), if rearm_hot { 5 } else { 2 }); + if rearm_hot { + assert_eq!(hot_calls.load(Ordering::SeqCst), 4); + assert!( + hot_at_other.load(Ordering::SeqCst) <= 1, + "a re-armed cell must rotate behind the other due cell" + ); + } + for (node, runtime) in fixture.runtimes.iter().enumerate() { + let pool = &fixture.appliers[&(node, route.history)].0; + for cell in [&a, &b] { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = pool + .query( + fixture.cell_options_at(cell, node).await?, + authority, + |database, _| { + let n: i64 = database.connection().query_row( + "SELECT n FROM value", + [], + |row| row.get(0), + )?; + Ok(n.to_be_bytes().to_vec()) + }, + ) + .await?; + let expected = if rearm_hot && cell == &a { + 4_i64 + } else { + 1_i64 + }; + assert_eq!(result.result(), expected.to_be_bytes()); + } + } + } + assert!(controllers[0].usage().effect_turns > 0); + assert_eq!(cold_opens.load(Ordering::SeqCst), 0); + for runtime in fixture.runtimes.iter().skip(usize::from(hold_callback)) { + for cell in &cold { + let selected = runtime + .reconciliation_authority(route.history, cell.clone()) + .await?; + assert_eq!(selected.cell().version.sequence.get(), 1); + assert_eq!(selected.cell().effects.next_due_unix_millis(), None); + } + } + for controller in &controllers { + controller.close().await?; + assert_eq!(controller.usage().active_effects, 0); + assert_eq!(controller.usage().retained_commands, 0); + } + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + assert_eq!(pool.usage()?.queued_or_running, 0); + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 14dc2c739..55f3b6752 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -49,6 +49,12 @@ struct ClusterLogLimits { purge_interval_ms: Option, } +struct ClusterBirthOptions { + group: GroupId, + minimum: Durability, + log: ClusterLogLimits, +} + impl Default for ClusterLogLimits { fn default() -> Self { Self { @@ -164,11 +170,36 @@ impl Cluster { minimum: Durability, log_limits: ClusterLogLimits, ) -> Result { + Self::with_store_birth( + options, + root, + filesystem, + schema, + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + minimum, + log: log_limits, + }, + ) + .await + } + + async fn with_store_birth( + options: RaftRuntimeOptions, + root: tempfile::TempDir, + filesystem: Arc, + schema: [u8; 32], + store: Arc, + birth_options: ClusterBirthOptions, + ) -> Result { + let minimum = birth_options.minimum; + let log_limits = birth_options.log; let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let history = GroupHistory { cluster: ClusterId::new([1; 16])?, namespace: NamespaceId::new([2; 16])?, - group: GroupId::new([3; 16])?, + group: birth_options.group, incarnation: IncarnationId::new([4; 16])?, }; let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; diff --git a/crates/cellar/tests/support/cell_group_idle_cache.rs b/crates/cellar/tests/support/cell_group_idle_cache.rs new file mode 100644 index 000000000..c2f8fbe23 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_idle_cache.rs @@ -0,0 +1,298 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Disposable SQL state is never a committed base or receipt authority. +use super::*; +use std::sync::atomic::{AtomicUsize, Ordering}; + +pub(crate) async fn idle_cache_reopens_exact_base() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [221; 32])?.command( + "Add", + [222; 32], + move |context, input| { + observed.fetch_add(1, Ordering::SeqCst); + let delta = i64::from_be_bytes(input.try_into()?); + let before: i64 = context + .sql() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + assert_eq!(before, if delta == 7 { 0 } else { 7 }); + context.sql().execute("UPDATE value SET n=n+?1", [delta])?; + Ok((before + delta).to_be_bytes().to_vec()) + }, + )?); + let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([223; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let term = fixture.runtimes[0].status(route.history).await?.term; + let first = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (1, 7), + retry_until, + ) + .await?; + assert_eq!(first.receipt().receipt.result(), 7_i64.to_be_bytes()); + assert_eq!(first.receipt().version.sequence.get(), 2); + drop(first); + idle(&fixture, &controllers, route.history, &a, 2).await?; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let old = fixture.runtimes[0] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let version = old.cell().version.clone(); + for node in 0..3 { + let options = fixture.cell_options_at(&a, node).await?; + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + match options { + CellSqlOptions::Materialized(options) => { + if node == 0 { + assert!(options.directory.is_dir()); + } + if options.directory.is_dir() { + let mut entries = std::fs::read_dir(&options.directory)?; + let only = entries.next().context("private lock inode")??; + assert_eq!(only.file_name(), "working.lock"); + assert!( + entries.next().is_none(), + "settled private scratch was retained" + ); + // Reacquiring this exact private lock proves the + // settled producer closed before cache eviction. + drop( + options + .host + .filesystem() + .lock_file(&options.directory.join("working.lock"))?, + ); + std::fs::remove_dir_all(&options.directory)?; + std::fs::create_dir_all(&options.directory)?; + std::fs::write( + options.directory.join("working.sqlite"), + b"discarded cache is not SQLite", + )?; + } + } + CellSqlOptions::Extents(options) => { + let usage = options.images.usage()?; + assert_eq!(usage.workspaces, 0); + assert_eq!(usage.workspace_bytes, 0); + if options.cache_directory.is_dir() { + std::fs::remove_dir_all(options.cache_directory)?; + } + } + } + Ok(()) + })? + .await??; + } + // A keeps its group and leadership. The next command must open from + // the verified committed image, including its receipt/client tables. + let next = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (2, 11), + retry_until, + ) + .await?; + assert_eq!(next.receipt().receipt.result(), 18_i64.to_be_bytes()); + assert_eq!(next.receipt().version.sequence.get(), 3); + drop(next); + idle(&fixture, &controllers, route.history, &a, 3).await?; + let duplicate = add( + &fixture, + &controllers[0], + &dispatcher, + &class, + &a, + (1, 7), + retry_until, + ) + .await?; + assert_eq!(duplicate.receipt().receipt.result(), 7_i64.to_be_bytes()); + assert_eq!(duplicate.receipt().version.sequence.get(), 3); + drop(duplicate); + idle(&fixture, &controllers, route.history, &a, 3).await?; + assert_eq!(calls.load(Ordering::SeqCst), 2); + assert_eq!(fixture.runtimes[0].status(route.history).await?.term, term); + let result = fixture.appliers[&(0, route.history)] + .0 + .query( + fixture.cell_options(&a).await?, + old, + move |database, selected| { + assert_eq!(selected, &version); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 7_i64.to_be_bytes()); + drop(result); + for (node, runtime) in fixture.runtimes.iter().enumerate() { + for (cell, expected, sequence) in [(&a, 18_i64, 3_u64), (&b, 0_i64, 1_u64)] { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(cell, node).await?, + authority, + move |database, selected| { + assert_eq!(selected.sequence.get(), sequence); + let value: i64 = database.connection().query_row( + "SELECT n FROM value", + [], + |row| row.get(0), + )?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), expected.to_be_bytes()); + drop(result); + } + } + for controller in &controllers { + controller.close().await?; + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + assert_eq!(usage.running_workers, 0); + assert_eq!(usage.active_cells, 0); + assert_eq!(usage.retained_bytes, 0); + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + +async fn idle( + fixture: &RuntimeCluster, + controllers: &[CellGroupController], + group: GroupHistory, + cell: &CellHistory, + sequence: u64, +) -> Result<()> { + controller::wait_for(async || { + for runtime in &fixture.runtimes { + if runtime + .reconciliation_authority(group, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != sequence + { + return Ok(false); + } + } + for controller in controllers { + if controller.usage().retained_commands != 0 + || controller.usage().active_apply != 0 + || controller.usage().pending_apply != 0 + { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 + || usage.retained_operations != 0 + || usage.retained_bytes != 0 + { + return Ok(false); + } + } + Ok(true) + }) + .await +} + +async fn add( + fixture: &RuntimeCluster, + controller: &CellGroupController, + dispatcher: &CellGroupDispatcher, + class: &RaftClass, + cell: &CellHistory, + (sequence, delta): (u64, i64), + retry_until_ms: u64, +) -> Result { + let invocation = dispatcher + .command( + cell.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([224; 16])?, sequence)?, + contract: class.contract("Add").context("Add")?, + input: delta.to_be_bytes().to_vec(), + durability: Durability::Quorum, + retry_until_ms, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + Ok(controller + .command(invocation, fixture.cell_options(cell).await?)? + .response() + .await?) +} diff --git a/crates/cellar/tests/support/cell_group_outbox_effects.rs b/crates/cellar/tests/support/cell_group_outbox_effects.rs new file mode 100644 index 000000000..bb15c7721 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_outbox_effects.rs @@ -0,0 +1,646 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real receiver commands and signatures; only the source-side reply is lost. +use super::*; +use cellar::application::consensus::{ + RaftDeliveryRequest, RaftEffectOptions, RaftEffectSigningKey, RaftEffectSink, + RaftEffectSourceGrant, RaftEffectTarget, RaftInbox, RaftOutbox, RaftOutboxLimits, +}; +use futures_util::future::BoxFuture; +use std::future::Future; +use std::sync::{ + atomic::{AtomicBool, AtomicUsize, Ordering}, + Mutex, Weak, +}; + +#[derive(Clone, Copy, PartialEq, Eq)] +enum Mode { + LostReply, + LeaderChange, + Shutdown, +} + +struct State { + mode: Mode, + arrivals: AtomicUsize, + first: Mutex)>>, + first_proof: Mutex>>, + changed: AtomicBool, + entered: tokio::sync::Notify, + release: tokio::sync::Notify, + dropped: AtomicBool, + polls: AtomicUsize, +} + +struct Endpoint { + controllers: Vec, + dispatcher: CellGroupDispatcher, + options: Vec, + leader: AtomicUsize, + runtimes: Vec, + group: GroupHistory, + source: CellHistory, +} + +struct Routing { + endpoint: Mutex>, + state: Arc, +} + +struct PendingDelivery(Arc); + +impl Drop for PendingDelivery { + fn drop(&mut self) { + self.0.dropped.store(true, Ordering::SeqCst); + } +} + +impl RaftEffectSink for Routing { + fn deliver( + &self, + request: RaftDeliveryRequest, + ) -> BoxFuture<'static, Result, CommandFailure>> { + let endpoint = self.endpoint.lock().unwrap().upgrade(); + let state = self.state.clone(); + Box::pin(async move { + let endpoint = endpoint.ok_or(CommandFailure::Unavailable)?; + let attempt = state.arrivals.fetch_add(1, Ordering::SeqCst); + { + let mut first = state.first.lock().unwrap(); + if let Some((id, bytes)) = first.as_ref() { + if *id != request.command_id() || bytes != request.encoded() { + state.changed.store(true, Ordering::SeqCst); + return Err(CommandFailure::CommandConflict); + } + } else { + *first = Some((request.command_id(), request.encoded().to_vec())); + } + } + let node = endpoint.leader.load(Ordering::SeqCst); + if request.target().durability() == Durability::S3 { + let selected = endpoint.runtimes[node] + .reconciliation_authority(endpoint.group, endpoint.source.clone()) + .await + .map_err(|_| CommandFailure::Unavailable)?; + let published = endpoint.controllers[node] + .archive_status(endpoint.group) + .map_err(|_| CommandFailure::Unavailable)? + .and_then(|status| status.published) + .ok_or(CommandFailure::Unavailable)?; + assert_eq!(selected.cell().effects.pending_outbox, 1); + assert!( + published.position().index >= selected.cell().last_mutation.index, + "source claim must have published S3 coverage before send" + ); + } + let dispatcher = endpoint + .dispatcher + .clone() + .with_s3(&endpoint.controllers[node]) + .map_err(|_| CommandFailure::Unavailable)?; + let invocation = dispatcher + .command( + request.target().scope().clone(), + CellCommandRequest { + id: request.command_id(), + contract: request.target().contract().clone(), + input: request.encoded().to_vec(), + durability: request.target().durability(), + retry_until_ms: cellar::asyncrt::wall_ms().max(0) as u64 + 60_000, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await + .map_err(|_| CommandFailure::Unavailable)?; + let receipt = endpoint.controllers[node] + .command(invocation, endpoint.options[node].clone()) + .map_err(|_| CommandFailure::Unavailable)? + .response() + .await + .map_err(|_| CommandFailure::Unavailable)?; + assert_eq!(receipt.durability(), request.target().durability()); + let proof = receipt.receipt().receipt.result().to_vec(); + // The adapter has consumed the typed receiver result into bounded + // reply bytes. A simulated lost network reply must not retain the + // receiver's SQL operation slot while another cell runs. + drop(receipt); + request + .verify_receipt(&proof) + .map_err(|_| CommandFailure::CommandConflict)?; + if attempt == 0 { + *state.first_proof.lock().unwrap() = Some(proof.clone()); + state.entered.notify_one(); + if state.mode == Mode::LostReply { + return Err(CommandFailure::Unavailable); + } + let _pending = PendingDelivery(state.clone()); + let released = state.release.notified(); + tokio::pin!(released); + std::future::poll_fn(|cx| { + state.polls.fetch_add(1, Ordering::SeqCst); + released.as_mut().poll(cx) + }) + .await; + } + Ok(proof) + }) + } +} + +pub(crate) async fn lost_reply() -> Result<()> { + run(Mode::LostReply).await +} + +pub(crate) async fn leader_change() -> Result<()> { + run(Mode::LeaderChange).await +} + +pub(crate) async fn shutdown() -> Result<()> { + run(Mode::Shutdown).await +} + +#[derive(Debug, PartialEq, Eq)] +struct DeliveryObservation { + command: CommandId, + bytes: Vec, + proof: Vec, +} + +pub(crate) async fn identity_across_bindings() -> Result<()> { + // Independently boot both physical bindings through real Raft and SQL. + // Each loses one receiver reply, then proves a single receiver mutation. + // A live transfer's seal/publication protocol belongs to CG-09. + for rooted in [false, true] { + let before = Box::pin(run_image( + Mode::LostReply, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([3; 16])?, + )) + .await?; + let after = Box::pin(run_image( + Mode::LostReply, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([31; 16])?, + )) + .await?; + assert_eq!( + before, after, + "group binding changed logical delivery identity or proof" + ); + } + Ok(()) +} + +async fn run(mode: Mode) -> Result<()> { + for rooted in [false, true] { + Box::pin(run_image( + mode, + rooted, + Arc::new(object_store::memory::InMemory::new()), + Durability::Quorum, + GroupId::new([3; 16])?, + )) + .await?; + } + Ok(()) +} + +pub(crate) async fn s3_lost_reply( + rooted: bool, + store: Arc, +) -> Result<()> { + Box::pin(run_image( + Mode::LostReply, + rooted, + store, + Durability::S3, + GroupId::new([3; 16])?, + )) + .await?; + Ok(()) +} + +async fn run_image( + mode: Mode, + rooted: bool, + store: Arc, + minimum: Durability, + group: GroupId, +) -> Result { + let state = Arc::new(State { + mode, + arrivals: AtomicUsize::new(0), + first: Mutex::new(None), + first_proof: Mutex::new(None), + changed: AtomicBool::new(false), + entered: tokio::sync::Notify::new(), + release: tokio::sync::Notify::new(), + dropped: AtomicBool::new(false), + polls: AtomicUsize::new(0), + }); + let routing = Arc::new(Routing { + endpoint: Mutex::new(Weak::new()), + state: state.clone(), + }); + let sender = Arc::new(RaftEffectSigningKey::new([211; 32])?); + let receiver = Arc::new(RaftEffectSigningKey::new([212; 32])?); + let received = Arc::new(AtomicUsize::new(0)); + let count = received.clone(); + let base = RaftClass::new("Counter", 1, [213; 32])?; + let definition = base.definition_digest(); + let target_cell = CellHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + cell: CellId::new("Counter:registered-b")?, + incarnation: IncarnationId::new([8; 16])?, + }; + let receive = OperationContract { + operation: "Receive".into(), + kind: OperationKind::Command, + digest: [214; 32], + }; + let target = RaftEffectTarget::new( + target_cell.clone(), + definition, + receive.clone(), + receiver.verification_key(), + minimum, + )?; + let inbox = RaftInbox::new( + vec![RaftEffectSourceGrant::new( + target_cell.cluster, + target_cell.namespace, + "Counter", + "Events", + [215; 32], + sender.verification_key(), + )?], + receiver, + minimum, + 4, + )?; + let outbox = RaftOutbox::new( + sender, + RaftOutboxLimits { + retry_delay_ms: 100, + ..Default::default() + }, + )? + .binding("Events", [215; 32], target, routing.clone())?; + let class = Arc::new( + base.inbox("Receive", receive.digest, inbox, move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let amount = i64::from_be_bytes(input.try_into()?); + context.sql().execute("UPDATE value SET n=n+?1", [amount])?; + Ok(amount.to_be_bytes().to_vec()) + })? + .outbox(outbox)? + .command("Emit", [216; 32], |context, input| { + Ok(context.emit("Events", input)?.to_be_bytes().to_vec()) + })? + .command("Touch", [217; 32], |context, _| { + context.sql().execute("UPDATE value SET n=n+100", [])?; + Ok(Vec::new()) + })?, + ); + assert_eq!(class.definition_digest(), definition); + let filesystem = Arc::new(DirectFileSystem); + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + group, + minimum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + let (mut fixture, route, source, destination) = Box::pin(registered::setup_cluster_named( + rooted, + cluster, + filesystem, + ["Counter:registered-a", "Counter:registered-b"], + )) + .await?; + assert_eq!(destination, target_cell); + let keys = VersionKeys::new( + VersionSigningKey::new([218; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + keys, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 64 * 1024, + }, + )?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let archive = if minimum == Durability::S3 { + let (archive, _) = + archive_controller::registration_minimum(&fixture, &route, rooted, minimum).await?; + controllers[0].archive(archive.clone())?.response().await?; + archive_controller::published( + &controllers[0], + route.history, + fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index, + ) + .await?; + Some(archive) + } else { + None + }; + let dispatcher = if archive.is_some() { + dispatcher.with_s3(&controllers[0])? + } else { + dispatcher + }; + let mut destination_options = Vec::new(); + for (node, controller) in controllers.iter().enumerate() { + let mut resources = BTreeMap::new(); + for cell in [&source, &destination] { + resources.insert(cell.clone(), fixture.cell_options_at(cell, node).await?); + } + destination_options.push(resources[&destination].clone()); + controller.configure_effects(vec![CellGroupEffectRegistration::new( + dispatcher.clone(), + RaftEffectOptions::default(), + move |cell| { + resources + .get(cell) + .cloned() + .ok_or(ExecutionError::Invalid("effect resources")) + }, + )?])?; + } + let endpoint = Arc::new(Endpoint { + controllers: controllers.clone(), + dispatcher: dispatcher.clone(), + options: destination_options, + leader: AtomicUsize::new(0), + runtimes: fixture.runtimes.clone(), + group: route.history, + source: source.clone(), + }); + *routing.endpoint.lock().unwrap() = Arc::downgrade(&endpoint); + let retry_until = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let invocation = dispatcher + .command( + source.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([219; 16])?, 1)?, + contract: class.contract("Emit").context("Emit")?, + input: 7_i64.to_be_bytes().to_vec(), + durability: minimum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + let receipt = controllers[0] + .command(invocation, fixture.cell_options(&source).await?)? + .response() + .await?; + assert_eq!(receipt.receipt().receipt.result(), 1_u64.to_be_bytes()); + assert_eq!(receipt.durability(), minimum); + drop(receipt); + tokio::time::timeout(Duration::from_secs(5), state.entered.notified()).await?; + if mode == Mode::LeaderChange { + let previous_polls = state.polls.load(Ordering::SeqCst); + let touch = dispatcher + .command( + source.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([220; 16])?, 1)?, + contract: class.contract("Touch").context("Touch")?, + input: Vec::new(), + durability: minimum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }, + None, + ) + .await?; + drop( + controllers[0] + .command(touch, fixture.cell_options(&source).await?)? + .response() + .await?, + ); + controller::wait_for(async || { + Ok(state.polls.load(Ordering::SeqCst) >= previous_polls + 2 + || state.dropped.load(Ordering::SeqCst)) + }) + .await?; + assert!( + !state.dropped.load(Ordering::SeqCst), + "a later cell mutation must not revoke a committed dispatch" + ); + let old_term = fixture.runtimes[0].status(route.history).await?.term; + endpoint.leader.store(1, Ordering::SeqCst); + let mut transferred = false; + controller::wait_for(async || { + // The healthy leader's lease suppresses ordinary pre-votes. + // Use the actual TimeoutNow handoff, including its caught-up + // target check, within this same progress deadline. + if !transferred { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => transferred = true, + Err(RuntimeError::NotReady) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + let status = fixture.runtimes[1].status(route.history).await?; + Ok(status.ready && status.term > old_term && state.dropped.load(Ordering::SeqCst)) + }) + .await?; + } else if mode == Mode::Shutdown { + controllers[0].close().await?; + assert!(state.dropped.load(Ordering::SeqCst)); + assert_eq!(received.load(Ordering::SeqCst), 1); + controller::wait_for(async || { + for runtime in fixture.runtimes.iter().skip(1) { + let source_view = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + let destination_view = runtime + .reconciliation_authority(route.history, destination.clone()) + .await?; + if source_view.cell().version.sequence.get() != 3 + || destination_view.cell().version.sequence.get() != 2 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + } + if mode != Mode::Shutdown { + controller::wait_for(async || { + for runtime in &fixture.runtimes { + let selected = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + if selected.cell().effects.pending_outbox != 0 { + return Ok(false); + } + } + for controller in &controllers { + let usage = controller.usage(); + if usage.active_effects != 0 || usage.retained_commands != 0 { + return Ok(false); + } + } + for (pool, _) in fixture.appliers.values() { + let usage = pool.usage()?; + if usage.active_cells != 0 || usage.retained_operations != 0 { + return Ok(false); + } + } + Ok(true) + }) + .await?; + assert_eq!(state.arrivals.load(Ordering::SeqCst), 2); + } + assert!(!state.changed.load(Ordering::SeqCst)); + assert_eq!(received.load(Ordering::SeqCst), 1); + for (node, runtime) in fixture + .runtimes + .iter() + .enumerate() + .skip(usize::from(mode == Mode::Shutdown)) + { + let source_view = runtime + .reconciliation_authority(route.history, source.clone()) + .await?; + assert_eq!( + source_view.cell().version.sequence.get(), + match mode { + Mode::LostReply => 5, + Mode::LeaderChange => 6, + Mode::Shutdown => 3, + } + ); + assert_eq!( + source_view.cell().effects.pending_outbox, + u64::from(mode == Mode::Shutdown) + ); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + destination.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(&destination, node).await?, + authority, + |database, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 7_i64.to_be_bytes()); + } + if let Some(archive) = &archive { + let current = fixture.runtimes[0] + .status(route.history) + .await? + .applied + .index; + archive_controller::published(&controllers[0], route.history, current).await?; + let coverage = archive_controller::verified_current( + &fixture, + archive, + rooted, + "effect-recovery-audit", + ) + .await?; + assert_eq!(coverage.recovery().manifest().pinned_images(), 2); + for cell in [&source, &destination] { + let selected = fixture.runtimes[0] + .reconciliation_authority(route.history, cell.clone()) + .await?; + let (restored, _) = coverage + .recovery() + .manifest() + .pin_cell(&cell.cell) + .context("S3 effect cell")?; + assert_eq!(restored, selected.cell()); + } + } + for controller in &controllers { + controller.close().await?; + assert_eq!(controller.usage().active_effects, 0); + } + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.retained_bytes, 0); + } + drop(endpoint); + drop(controllers); + fixture.close().await?; + let (command, bytes) = state + .first + .lock() + .unwrap() + .take() + .context("first delivery")?; + let proof = state + .first_proof + .lock() + .unwrap() + .take() + .context("receiver proof")?; + Ok(DeliveryObservation { + command, + bytes, + proof, + }) +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 9020c189d..af2d96bf6 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -805,6 +805,11 @@ pub(super) async fn attachment_reserves_before_owner() -> Result<()> { mod sql; #[allow(unused_imports)] pub(super) use sql::unproved_s3_responses; +pub(super) use sql::{ + alarms_without_clients, effect_delivery_shutdown, effect_identity_across_bindings, + effect_leader_change, effect_lost_reply, effect_s3_lost_reply, hot_and_cold_alarms, + idle_cache_reopens_exact_base, shutdown_joins_alarm, +}; #[allow(unused_imports)] pub(super) use sql::{ archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index c940698b0..4e6431bb7 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -547,6 +547,13 @@ pub(crate) use s3_commands::{ pub(crate) use s3_response::unproved_s3_responses; #[path = "cell_group_controller.rs"] mod controller; +#[path = "cell_group_idle_cache.rs"] +mod idle_cache; +pub(crate) use idle_cache::idle_cache_reopens_exact_base; +#[path = "cell_group_effects.rs"] +mod effects; +#[path = "cell_group_outbox_effects.rs"] +mod outbox_effects; #[allow(unused_imports)] pub(crate) use archive_controller::{ archive_checkpoint_frontier, archive_controller_publish, archive_controller_reconcile, @@ -555,6 +562,12 @@ pub(crate) use archive_controller::{ pub(crate) use archive_controller::{ archive_controller_publish_store, archive_controller_reconcile_store, }; +pub(crate) use effects::{alarms_without_clients, hot_and_cold_alarms, shutdown_joins_alarm}; +pub(crate) use outbox_effects::{ + identity_across_bindings as effect_identity_across_bindings, + leader_change as effect_leader_change, lost_reply as effect_lost_reply, + s3_lost_reply as effect_s3_lost_reply, shutdown as effect_delivery_shutdown, +}; #[path = "cell_group_controller_commands.rs"] mod controller_commands; pub(crate) use controller::{ diff --git a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs index 482c7c8f8..30bd04aa7 100644 --- a/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs +++ b/crates/cellar/tests/support/cell_group_sql_command_dispatch.rs @@ -28,14 +28,16 @@ fn request( }) } -async fn accepted(invocation: &mut CellCommandInvocation) -> Result { +pub(super) async fn accepted(invocation: &mut CellCommandInvocation) -> Result { match invocation.submit().await? { CellCommandSubmission::Accepted(accepted) => Ok(accepted), CellCommandSubmission::Rejected => anyhow::bail!("unexpected sealed rejection"), } } -async fn committed(invocation: &mut CellCommandInvocation) -> Result { +pub(super) async fn committed( + invocation: &mut CellCommandInvocation, +) -> Result { match invocation.settle().await? { CellCommandSettlement::Committed(receipt) => Ok(receipt), _ => anyhow::bail!("expected committed receipt"), diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index 76e7c6d91..b1493ce95 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -211,6 +211,21 @@ pub(super) async fn setup_cluster( rooted: bool, cluster: Cluster, filesystem: Arc, +) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + setup_cluster_named( + rooted, + cluster, + filesystem, + ["registered-a", "registered-b"], + ) + .await +} + +pub(super) async fn setup_cluster_named( + rooted: bool, + cluster: Cluster, + filesystem: Arc, + names: [&str; 2], ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; let route = fixture.cluster.birth.route().clone(); @@ -223,8 +238,8 @@ pub(super) async fn setup_cluster( )?); } } - let a = history(&route, "registered-a")?; - let b = history(&route, "registered-b")?; + let a = history(&route, names[0])?; + let b = history(&route, names[1])?; for node in 0..3 { apply::install(&mut fixture, &a, node).await?; } diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index bb17ac324..e9b7879e9 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -10,7 +10,7 @@ replace the unavailable historical raw artifacts. CG-06's complete archive, local reclamation and fresh-history recovery boundary also passes both actual Docker/RustFS all-voter-disks-deleted drills with permanent launch revocation, fresh-quorum activation, exact receipts, new writes and retained restart. -CG-08 due effects and safe cache lifetimes follow, before CG-07 SDK/fleet +CG-08 due effects and safe cache lifetimes are IN_PROGRESS, before CG-07 SDK/fleet cutover. See the latest [progress](progress.md) and [handoff](handoff.md). The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index f5b182ef6..5c6dd42d1 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -47,14 +47,26 @@ Keep credentials, signing material and raw tokens private. Docker API config copies avoid the reproduced virtiofs truncation race; controller keys/launch ledgers never enter containers. Failed predecessors remain recorded. -The next critical implementation is CG-08, then CG-07 SDK/fleet cutover. An -isolated codex/cell-group-effects-3617 branch at -/Users/haipingfu/.codex/worktrees/cell-group-effects/cellar contains an unqualified -CG-08 due-index draft while the qualified checkout stays unchanged. It adds -subtree next-due hints, required nullable marker-1 fields and bounded fair -manifest lookup without SQLite hydration. Focused unit validation is in progress; -no scheduler or full CG-08 qualification is claimed. Merge qualified changes -back into the existing PR branch; do not create a permanent second runtime. +CG-08 is IN_PROGRESS, followed by CG-07 SDK/fleet cutover. The committed due-index +foundation is 783c270b. The isolated effects checkout retains the exact predecessor +five-case execution run. Its expanded compile check and all five native-10 +lifecycle cases pass in both image modes. The preceding native-09 run passes four +cases while an ordinary follower campaign times out under the healthy leader +lease. The test-only correction uses real TimeoutNow transfer inside the same +deadline. Preserve both source/executable/log records. Native-10 is terminal; +source can now be reconciled with the successor checkout. + +The existing PR checkout contains the successor implementation and new +native_cell_groups_effects target: exact cold-cache reopening, cold-cell discovery, +logical delivery/proof identity under two independent physical group bindings, +SQL/dispatch custody, joined shutdown, and normal/explicit RustFS S3 closure cases. +All eight successor native cases now pass in both image modes, including the five +declared cases and S3 coverage through the normal adapter. The real RustFS case, +new unit/enrollment regressions, strict lint and complete task evidence remain. +See the current [progress](progress.md). Local raw records/continuation remain +ignored and backed up outside the checkout. Preserve the user's untracked diagram +and both excluded MR progress basenames. Reconcile both checkouts only after the +source-bound predecessor run terminates; reuse PR 160 and one final runtime. 1. Finish the due-index validation and wire bounded fair node effect discovery, exact committed-image scans and sealed private system commands into the shared diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index b52bbb2a8..cbc8bc390 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -18,7 +18,7 @@ never executed by the plan runner. Read the [design](design.md), | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | DONE | 5 | -| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | TODO | 5 | +| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | IN_PROGRESS | 5 | | CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | | CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | TODO | 7 | | CG-10 | Split hot groups and merge compatible adjacent ranges | CG-09 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 3ec91ac70..1e8edac09 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -6,10 +6,60 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). CG-01 through CG-06 have passed their scoped native gates; -CG-07 through CG-16 remain TODO. Intermediate contracts +CG-08 is IN_PROGRESS; CG-07 and CG-09 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-08 node effects and disposable SQL lifetimes + +The successor implementation is under qualification on codex/shared-cell-groups. +The committed due-index foundation is 783c270b. The node controller now owns fair +group/cell discovery, exact committed-image scans, private alarm/outbox commands, +claim durability, fenced delivery and joined shutdown. No cell owns a timer or +resident execution worker. The expanded predecessor compile check passes. + +All five predecessor execution cases now pass in both image modes: alarms, lost +receiver replies, actual leader transfer, delivery shutdown and held alarm +shutdown. The successful native-10 run passes five cases in 7.11 seconds after an +18.44-second build. Its retained executable SHA256 is +e9fa57084a95fe02e0af8d8ddaceda5ccd0b8b9efd00c1b1ee92a61dcc06a8b7. +The preceding run passes four cases but times out on an ordinary follower +pre-vote under the healthy leader lease. The test-only correction uses the real +TimeoutNow transfer and caught-up target check inside the original deadline, +preserving every custody/state oracle. Both source-bound records, executables +and real logs remain under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/. + +The native_cell_groups_effects target now passes eight cases with zero failures +and one explicitly ignored RustFS case, in 33.05 seconds after a 3m45s build. Each +case exercises materialized and extent modes, including all five original CG-08 +case names. The source-bound native-11 record and executable are retained locally; +executable SHA256: +54d1af2d19bba8aacf206f23d04ca3fb1f3b23f1dafc554bbd49dc079760ffb9. + +Cold reopening checks the committed base, exact receipt retry, an older coherent +read and an unchanged Raft term. Discovery leaves 32 cold cells unopened; a cell +re-arming through four alarm generations rotates behind the other due cell. +Identity qualification boots two independent physical group bindings, comparing +exact signed deliveries and receiver proofs while suppressing duplicate execution +under each. This does not qualify live CG-09 transfer. The normal S3 adapter case +requires archived claim coverage before send and independently verifies the final +complete image/receipt closure. Its explicit local RustFS variant remains unrun. + +The cache review found materialized settlement closed SQLite while retaining its +private database and LTX files. Shared-cell close now discards those bounded +scratch paths before releasing the private lock, including failed opens and +post-open authority revocation. Sealed images and uncertain proposal custody are +separate. The passing cold-reopen case asserts only the lock inode remains before +injecting a bogus disposable database and checking exact reconstruction. Affected +enrollment regressions and cleanup failure qualification remain pending. + +Private-operation discovery now bounds its SQLite projection before allocating +the scheduling preference string. Its corruption/permit regression, strict +Clippy and the complete CG-08 evidence gate remain required. Real RustFS effects +and the later CG-07 public SDK/fleet cutover are still unqualified. CG-06's prior +source-scoped closure remains intact; the full hard cutover is unfinished. + ## CG-06 complete native and local disaster-recovery gate CG-06 is DONE at its declared unmanaged archive/reclamation/recovery boundary. diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 57120acc2..484b38979 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -2020,7 +2020,7 @@ { "id": "CG-08", "title": "Preserve per-cell effects and bounded idle execution caches", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", From e4901835aea567527bdb062a0568244ea383f532 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 02:02:54 -0700 Subject: [PATCH 262/568] test: qualify shared effects on RustFS --- tools/multi-raft/README.md | 22 +++++++++++++++++- tools/multi-raft/cell_groups.py | 40 ++++++++++++++++++++++++--------- 2 files changed, 51 insertions(+), 11 deletions(-) diff --git a/tools/multi-raft/README.md b/tools/multi-raft/README.md index 37d5977db..ef2027e18 100644 --- a/tools/multi-raft/README.md +++ b/tools/multi-raft/README.md @@ -71,7 +71,7 @@ python3 tools/multi-raft/cell_groups.py \ The runner verifies the backend's ownership label, image, running state, endpoint and actual resource limits, then makes seven signed S3 probes in a new bucket. -It runs seventeen exact ignored RustFS cases, each in materialized and extent modes: +It runs twenty exact ignored RustFS cases, each in materialized and extent modes: archive-gated local compaction with independent sibling checkpoint progress, stable publication with continuing quorum, uncertain-publication reconciliation with joined close, refusal of unproved S3 responses, positive local command S3 @@ -107,6 +107,26 @@ Failed runs retain their record and logs. Every run owns a new remote prefix; remote objects and the backend remain available for inspection until explicitly stopped using the command above. No cleanup deletes another run's data. +For CG-08, select the fixed effects case set: + +```sh +python3 tools/multi-raft/cell_groups.py --suite effects \ + --backend /tmp/cellar-raft-local-001 \ + --output /tmp/cellar-cell-groups-effects-rustfs-001 \ + --target-directory "$HOME/Workspace/crabbuild-target/cellar-my-checkout" +``` + +This runs the exact `rustfs_claim_receiver_and_completion_use_verified_coverage` +case in `native_cell_groups_effects`, in both image forms. Source claims require +published S3 coverage before send, the real receiver preserves its S3 receipt +policy, and a lost reply retries identical delivery bytes with one receiver +callback. Conditional source completion also crosses S3 settlement. A fresh +verifier independently restores the final complete image/receipt closure and +compares every cell leaf with the selected local state. The same ownership, +resource, signed-probe, executable/source and exact-selection checks apply. +The default archive suite keeps its complete existing case set. This effects +suite does not qualify a backend outage or external-effects disaster recovery. + ## Native node runs The repository's `tools/perf/raft.py` harness now provides three native SDK node diff --git a/tools/multi-raft/cell_groups.py b/tools/multi-raft/cell_groups.py index ac9fc1855..ab82f09ce 100644 --- a/tools/multi-raft/cell_groups.py +++ b/tools/multi-raft/cell_groups.py @@ -48,6 +48,18 @@ "rustfs_s3_receipt_lookup_deadlines_refund_read_admission", ) +SUITES = { + "archive": ( + TARGET, CASES, + "shared-group RustFS archive/setup and local/remote command/enrollment/receipt S3 settlement", + ), + "effects": ( + "native_cell_groups_effects", + ("rustfs_claim_receiver_and_completion_use_verified_coverage",), + "shared-group RustFS outbox claim, receiver receipt/deduplication and complete S3 recovery closure", + ), +} + def exact_case_passed(output, case): """Keep exact selection strict when validated observations interrupt libtest.""" @@ -114,11 +126,14 @@ def backend(root): def main(): parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--suite", choices=tuple(SUITES), default="archive", + help="fixed required RustFS case set; archive preserves the existing gate") parser.add_argument("--backend", type=Path, required=True) parser.add_argument("--output", type=Path, required=True) parser.add_argument("--target-directory", type=Path, required=True, help="absolute checkout-private Cargo target on the workspace volume") args = parser.parse_args() + target, cases, scope = SUITES[args.suite] if not args.target_directory.is_absolute(): parser.error("--target-directory must be absolute and unique to this checkout") subprocess.run(["git", "diff", "--exit-code", "HEAD", "--", ".", @@ -132,14 +147,19 @@ def main(): out.mkdir(mode=0o700, parents=True, exist_ok=False) prefix = "cell-groups/" + secrets.token_hex(16) record = { - "format": 1, "status": "RUNNING", "scope": "shared-group RustFS archive/setup and local/remote command/enrollment/receipt S3 settlement", + "format": 1, "status": "RUNNING", "scope": scope, + "suite": args.suite, "test_target": target, "source_revision": subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip(), "source_inputs": sources(), "backend": remote, "prefix": prefix, "commands": [], - "limitations": "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " + "limitations": ("Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " "The bounded full-backend outage case controls only this owned container. " "Logical retained-entry admission and controlled-geometry Raft file purge are tested; " "global physical disk bounds, process-kill restart, fencing/DR, sustained performance " - "and independent hosts remain separate gates.", + "and independent hosts remain separate gates." if args.suite == "archive" else + "Host-native three-voter TLS fixture, both image forms; RustFS runs in one local VM. " + "This suite covers live effect S3 settlement and independent complete archive replay. " + "It does not qualify a backend outage, public SDK/fleet cutover, live group transfer, " + "external-effects disaster recovery, sustained load or independent hosts."), } def save(): @@ -166,10 +186,10 @@ def save(): CARGO_TARGET_DIR=str(args.target_directory), CARGO_PROFILE_DEV_DEBUG="0", CARGO_PROFILE_TEST_DEBUG="0", ) - cargo = ["cargo", "+1.98.1", "test", "-p", "cellar", "--locked", "--test", TARGET, "--"] - for number, selection in enumerate(("--list", *CASES)): + cargo = ["cargo", "+1.98.1", "test", "-p", "cellar", "--locked", "--test", target, "--"] + for number, selection in enumerate(("--list", *cases)): argv = cargo + ([selection] if number == 0 else [selection, "--exact", "--ignored", "--test-threads=1"]) - print(f"START {number + 1}/{len(CASES) + 1}: {selection}", flush=True) + print(f"START {number + 1}/{len(cases) + 1}: {selection}", flush=True) started = time.monotonic() if selection == "rustfs_quorum_continues_bounded_s3_outage": control = BackendControl(remote, out / "backend-control") @@ -190,7 +210,7 @@ def save(): record["commands"].append({"argv": argv, "exit_code": result.returncode, "elapsed_seconds": round(time.monotonic() - started, 3), "output": log.name, "sha256": sha(log)}) - match = re.search(r"Running [^\n]*\(([^\n)]+/native_cell_groups_archive_controller-[a-f0-9]+)\)", output) + match = re.search(rf"Running [^\n]*\(([^\n)]+/{re.escape(target)}-[a-f0-9]+)\)", output) if match: executable = Path(match[1]) selected_hash = sha(executable) @@ -203,7 +223,7 @@ def save(): if result.returncode: raise RuntimeError(f"{selection} failed; see {log}") if number == 0: - for case in CASES: + for case in cases: if not re.search(rf"^{re.escape(case)}: test$", output, re.M): raise RuntimeError("missing exact RustFS test in executable inventory") elif not exact_case_passed(output, selection): @@ -215,14 +235,14 @@ def save(): raise RuntimeError("source changed during qualification") backend(args.backend.resolve()) record["status"] = "PASSED" - record["passed_cases"] = list(CASES) + record["passed_cases"] = list(cases) except Exception: record["status"] = "FAILED" raise finally: save() print(json.dumps({"record": str(out / "record.json"), "status": record["status"], - "rustfs_cases": len(CASES), "image_modes_per_case": 2}), flush=True) + "rustfs_cases": len(cases), "image_modes_per_case": 2}), flush=True) if __name__ == "__main__": From e8edfd12c0d11e6302bd849d865926139bf4a9ae Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 02:49:34 -0700 Subject: [PATCH 263/568] test: preserve outcomes on cache cleanup failure --- .../tests/native_cell_groups_effects.rs | 5 + .../tests/native_cell_groups_execution.rs | 2 + .../tests/support/cell_group_idle_cache.rs | 212 ++++++++++++++++++ .../tests/support/cell_group_runtime.rs | 6 +- crates/cellar/tests/support/cell_group_sql.rs | 2 +- .../tests/support/raft_storage_fault.rs | 10 + docs/design/multi-raft/cell-groups/handoff.md | 9 +- .../design/multi-raft/cell-groups/progress.md | 29 ++- 8 files changed, 266 insertions(+), 9 deletions(-) diff --git a/crates/cellar/tests/native_cell_groups_effects.rs b/crates/cellar/tests/native_cell_groups_effects.rs index 32db89f98..74296d16c 100644 --- a/crates/cellar/tests/native_cell_groups_effects.rs +++ b/crates/cellar/tests/native_cell_groups_effects.rs @@ -30,6 +30,11 @@ async fn idle_cache_reopens_exact_cell_base() -> Result<()> { Box::pin(runtime_fixture::idle_cache_reopens_exact_base()).await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn materialized_cleanup_failure_preserves_outcome_and_group_capacity() -> Result<()> { + Box::pin(runtime_fixture::cleanup_failure_retains_outcome()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn due_effect_without_client_request() -> Result<()> { Box::pin(runtime_fixture::alarms_without_clients()).await diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 556c63dff..d60d88b69 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -12,6 +12,8 @@ mod archive_store; mod fault; #[path = "support/cell_group_runtime.rs"] +// The shared fixture also owns cases selected by native_cell_groups_effects. +#[allow(dead_code, unused_imports)] mod runtime_fixture; #[path = "support/cell_group_checkpoint.rs"] diff --git a/crates/cellar/tests/support/cell_group_idle_cache.rs b/crates/cellar/tests/support/cell_group_idle_cache.rs index c2f8fbe23..99ae63302 100644 --- a/crates/cellar/tests/support/cell_group_idle_cache.rs +++ b/crates/cellar/tests/support/cell_group_idle_cache.rs @@ -296,3 +296,215 @@ async fn add( .response() .await?) } + +/// The cleanup fails after a committed receipt exists, while the private lock +/// is still owned. A real leader handoff must preserve its ambiguous outcome. +pub(crate) async fn cleanup_failure_retains_outcome() -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let observed = calls.clone(); + let gate = fault::Gate::new(None); + let class = Arc::new(RaftClass::new("Counter", 1, [227; 32])?.command( + "Add", + [228; 32], + move |context, _| { + observed.fetch_add(1, Ordering::SeqCst); + context.sql().execute("UPDATE value SET n=n+1", [])?; + Ok(b"committed".to_vec()) + }, + )?); + let (mut fixture, route, cell, _) = Box::pin(registered::setup(false, &class)).await?; + let controllers = controller::start_with_limits( + &mut fixture, + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + ..controller::limits() + }, + ) + .await?; + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + VersionKeys::new( + VersionSigningKey::new([229; 32], u64::MAX)?, + vec![], + Duration::from_secs(30), + )?, + CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + )?; + let retry_until_ms = fixture + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([230; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: Vec::new(), + durability: Durability::Quorum, + retry_until_ms, + }; + let routing = CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 4, + }; + let invocation = dispatcher + .command(cell.clone(), request.clone(), routing, None) + .await?; + let CellSqlOptions::Materialized(mut options) = fixture.cell_options(&cell).await? else { + anyhow::bail!("materialized private state required"); + }; + let directory = options.directory.clone(); + options.host = options + .host + .with_filesystem(Arc::new(fault::FaultFileSystem { gate: gate.clone() })); + let held = gate.arm(fault::HOLD_FAIL_PRIVATE_CLEANUP); + let call = controllers[0].command(invocation, CellSqlOptions::Materialized(options))?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + let pool = &fixture.appliers[&(0, route.history)].0; + assert_eq!(pool.usage()?.active_cells, 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + let old_term = fixture.runtimes[0].status(route.history).await?.term; + let mut transferred = false; + controller::wait_for(async || { + if !transferred { + match fixture.runtimes[0].transfer_leader(route.history, 2).await { + Ok(()) => transferred = true, + Err(RuntimeError::NotReady) => return Ok(false), + Err(error) => return Err(error.into()), + } + } + let status = fixture.runtimes[1].status(route.history).await?; + Ok(status.ready && status.term > old_term) + }) + .await?; + drop(held); + let error = call + .response() + .await + .err() + .context("cleanup failure cannot acknowledge success")?; + assert!(error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Execution(ExecutionError::Storage(_)) + )); + assert!(error + .to_string() + .contains("injected private scratch cleanup failure")); + controller::wait_for(async || Ok(controllers[0].usage().quarantined_groups == 1)).await?; + assert_eq!(controllers[0].usage().groups, 0); + assert_eq!(controllers[0].usage().retained_commands, 1); + assert_eq!(pool.usage()?.active_cells, 0); + assert!(!controllers[0].usage().closing); + let mut replacement = registration::prepare(&fixture, 31, &cell).await?; + let refused = controllers[0] + .register(replacement.inputs.remove(0)) + .err() + .context("quarantined group retains the only registration slot")?; + assert!(matches!( + refused.error, + CellGroupControllerError::Runtime(RuntimeError::Overloaded) + )); + drop(refused); + drop(replacement); + assert_eq!(controllers[0].usage().pending_registrations, 0); + assert_eq!(controllers[0].usage().quarantined_groups, 1); + fixture + .cluster + .domain + .blocking(move || -> Result<()> { + assert!(directory.join("working.sqlite").is_file()); + Ok(()) + })? + .await??; + // The next leader reads the original replicated receipt. Neither the lost + // cleanup response nor a discarded private image can rerun the mutation. + let invocation = dispatcher + .command(cell.clone(), request, routing, None) + .await?; + let receipt = controllers[1] + .command(invocation, fixture.cell_options_at(&cell, 1).await?)? + .response() + .await?; + assert_eq!(receipt.receipt().receipt.result(), b"committed"); + assert_eq!(receipt.receipt().version.sequence.get(), 2); + drop(receipt); + assert_eq!(calls.load(Ordering::SeqCst), 1); + assert_eq!(controllers[0].usage().retained_commands, 1); + controller::wait_for(async || { + for runtime in fixture.runtimes.iter().skip(1) { + if runtime + .reconciliation_authority(route.history, cell.clone()) + .await? + .cell() + .version + .sequence + .get() + != 2 + { + return Ok(false); + } + } + for (node, controller) in controllers.iter().enumerate().skip(1) { + let usage = fixture.appliers[&(node, route.history)].0.usage()?; + if controller.usage().retained_commands != 0 + || usage.active_cells != 0 + || usage.retained_operations != 0 + { + return Ok(false); + } + } + Ok(true) + }) + .await?; + for (node, runtime) in fixture.runtimes.iter().enumerate().skip(1) { + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 5000 }) + .await?; + let authority = runtime + .read_authority( + route.history, + cell.clone(), + CellReadConsistency::Stale, + deadline, + ) + .await?; + let result = fixture.appliers[&(node, route.history)] + .0 + .query( + fixture.cell_options_at(&cell, node).await?, + authority, + |database, version| { + assert_eq!(version.sequence.get(), 2); + let value: i64 = + database + .connection() + .query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_be_bytes().to_vec()) + }, + ) + .await?; + assert_eq!(result.result(), 1_i64.to_be_bytes()); + drop(result); + } + for controller in &controllers { + controller.close().await?; + } + assert_eq!(controllers[0].usage().retained_commands, 0); + for (pool, _) in fixture.appliers.values() { + assert_eq!(pool.usage()?.active_cells, 0); + assert_eq!(pool.usage()?.running_workers, 0); + } + drop(controllers); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index af2d96bf6..36a2090ed 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -806,9 +806,9 @@ mod sql; #[allow(unused_imports)] pub(super) use sql::unproved_s3_responses; pub(super) use sql::{ - alarms_without_clients, effect_delivery_shutdown, effect_identity_across_bindings, - effect_leader_change, effect_lost_reply, effect_s3_lost_reply, hot_and_cold_alarms, - idle_cache_reopens_exact_base, shutdown_joins_alarm, + alarms_without_clients, cleanup_failure_retains_outcome, effect_delivery_shutdown, + effect_identity_across_bindings, effect_leader_change, effect_lost_reply, effect_s3_lost_reply, + hot_and_cold_alarms, idle_cache_reopens_exact_base, shutdown_joins_alarm, }; #[allow(unused_imports)] pub(super) use sql::{ diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 4e6431bb7..18d71033f 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -549,7 +549,7 @@ pub(crate) use s3_response::unproved_s3_responses; mod controller; #[path = "cell_group_idle_cache.rs"] mod idle_cache; -pub(crate) use idle_cache::idle_cache_reopens_exact_base; +pub(crate) use idle_cache::{cleanup_failure_retains_outcome, idle_cache_reopens_exact_base}; #[path = "cell_group_effects.rs"] mod effects; #[path = "cell_group_outbox_effects.rs"] diff --git a/crates/cellar/tests/support/raft_storage_fault.rs b/crates/cellar/tests/support/raft_storage_fault.rs index 226a59779..84d396515 100644 --- a/crates/cellar/tests/support/raft_storage_fault.rs +++ b/crates/cellar/tests/support/raft_storage_fault.rs @@ -30,6 +30,7 @@ pub const HOLD_EXTENT_READ: u8 = 13; pub const FAIL_AFTER_SYNC: u8 = 14; pub const HOLD_DEPENDENCY_PREFIX: u8 = 15; pub const HOLD_DEPENDENCY_SYNC: u8 = 16; +pub const HOLD_FAIL_PRIVATE_CLEANUP: u8 = 17; pub struct Gate { pub mode: AtomicU8, @@ -407,6 +408,15 @@ impl FileSystem for FaultFileSystem { } fn remove_file(&self, path: &Path) -> io::Result<()> { + if path + .file_name() + .is_some_and(|name| name == "working.sqlite") + && self.gate.mode.load(Ordering::Acquire) == HOLD_FAIL_PRIVATE_CLEANUP + && DirectFileSystem.metadata(path).is_ok() + { + self.gate.pause()?; + return Err(io::Error::other("injected private scratch cleanup failure")); + } DirectFileSystem.remove_file(path)?; if path .parent() diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 5c6dd42d1..c3b7a7a6e 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -61,8 +61,13 @@ native_cell_groups_effects target: exact cold-cache reopening, cold-cell discove logical delivery/proof identity under two independent physical group bindings, SQL/dispatch custody, joined shutdown, and normal/explicit RustFS S3 closure cases. All eight successor native cases now pass in both image modes, including the five -declared cases and S3 coverage through the normal adapter. The real RustFS case, -new unit/enrollment regressions, strict lint and complete task evidence remain. +declared cases and S3 coverage through the normal adapter. The exact real RustFS +case also passes in both image modes at e4901835 with seven signed probes and +source/executable/backend binding. Its backend/VM are stopped, data retained. +The extra cleanup-fault case passes actual leader handoff, ambiguous storage +failure, retained one-group quota and receipt retry without callback replay. +Twenty-four library regressions pass. Current complete task evidence, affected +enrollment/execution regressions and strict lint remain before closing CG-08. See the current [progress](progress.md). Local raw records/continuation remain ignored and backed up outside the checkout. Preserve the user's untracked diagram and both excluded MR progress basenames. Reconcile both checkouts only after the diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1e8edac09..fbbc0b0c8 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -52,12 +52,35 @@ scratch paths before releasing the private lock, including failed opens and post-open authority revocation. Sealed images and uncertain proposal custody are separate. The passing cold-reopen case asserts only the lock inode remains before injecting a bogus disposable database and checking exact reconstruction. Affected -enrollment regressions and cleanup failure qualification remain pending. +enrollment regressions remain pending. + +The cleanup I/O failure case now passes. It holds the real private cleanup after +commit, transfers leadership through TimeoutNow, returns a typed ambiguous storage +error, and retains the failed invocation and quarantined group's capacity. With +the group/apply limits tightened to one, an actual fresh registration refuses +with Overloaded. The new leader returns the original receipt without callback +replay; both serving replicas retain the value and joined shutdown releases work. +The first test attempted to count retained capacity using the active-group metric; +its failure remains as native-12. Native-13 corrects the active/quarantined +observation and proves quota refusal directly without raising any limit. + +The real RustFS effects case passes in both image forms at e4901835, with seven +signed S3 probes, exact source/backend/executable checks and independently restored +final images/receipts. Its executable SHA256 remains +54d1af2d19bba8aacf206f23d04ca3fb1f3b23f1dafc554bbd49dc079760ffb9. +The backend and dedicated VM are stopped, data retained, and default Docker context +remains colima. The fixed --suite effects selection preserves the original twenty +archive cases. Twenty-four source-matched library regressions also pass. Strict +Clippy initially identified sibling-only fixture functions as unused in the +execution target; its shared fixture now carries the same scoped annotation as +other native targets. Production lint, test oracles and qualification commands +remain unchanged. These source-scoped records are retained under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/. Private-operation discovery now bounds its SQLite projection before allocating the scheduling preference string. Its corruption/permit regression, strict -Clippy and the complete CG-08 evidence gate remain required. Real RustFS effects -and the later CG-07 public SDK/fleet cutover are still unqualified. CG-06's prior +Clippy and the complete CG-08 evidence gate remain required. The later CG-07 +public SDK/fleet cutover is still unqualified. CG-06's prior source-scoped closure remains intact; the full hard cutover is unfinished. ## CG-06 complete native and local disaster-recovery gate From 8a4c5ae0e92d12fa1ea91b16733342e8a4f2cb83 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 03:26:54 -0700 Subject: [PATCH 264/568] test: preserve lane heads in apply fixtures --- .../tests/support/cell_group_sql_apply.rs | 15 +++++++++-- docs/design/multi-raft/cell-groups/handoff.md | 19 ++++++++++++-- .../design/multi-raft/cell-groups/progress.md | 25 +++++++++++++++++-- 3 files changed, 53 insertions(+), 6 deletions(-) diff --git a/crates/cellar/tests/support/cell_group_sql_apply.rs b/crates/cellar/tests/support/cell_group_sql_apply.rs index 1c99b3440..8d03981ee 100644 --- a/crates/cellar/tests/support/cell_group_sql_apply.rs +++ b/crates/cellar/tests/support/cell_group_sql_apply.rs @@ -99,7 +99,18 @@ pub(crate) async fn apply_images(rooted: bool) -> Result<()> { // Enroll through the production adapter on all voters, including scheduling // summaries derived from real SQLite state rather than fixture defaults. let cell = history(&route, "effects")?; - let bytes = images::image(None, "CREATE TABLE value(n); INSERT INTO value VALUES(7); CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); CREATE TABLE __sys_raft_outbox(next_at); INSERT INTO __sys_raft_outbox VALUES(80),(40);")?; + // Keep the lane/sequence projection used by real outbox discovery. The + // later A row is immediately due but cannot pass its sleeping lane head. + let bytes = images::image( + None, + "CREATE TABLE value(n); INSERT INTO value VALUES(7); + CREATE TABLE __sys_raft_alarms(due); INSERT INTO __sys_raft_alarms VALUES(90),(30); + CREATE TABLE __sys_raft_outbox(lane,sequence,next_at); + INSERT INTO __sys_raft_outbox VALUES + ('A',x'0000000000000001',80), + ('A',x'0000000000000002',0), + ('B',x'0000000000000001',40);", + )?; let encoded = enrollment_bytes(&route, &cell, &bytes, rooted)?; let accepted = fixture.runtimes[0].propose(route.history, encoded).await?; fixture.drive(Some(accepted.position.index)).await?; @@ -109,7 +120,7 @@ pub(crate) async fn apply_images(rooted: bool) -> Result<()> { .await?; assert_eq!(state.cell().effects.next_alarm_unix_millis, Some(30)); assert_eq!(state.cell().effects.next_outbox_unix_millis, Some(40)); - assert_eq!(state.cell().effects.pending_outbox, 2); + assert_eq!(state.cell().effects.pending_outbox, 3); } let options = fixture.cell_options(&a).await?; diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index c3b7a7a6e..0b027fdbd 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -66,13 +66,28 @@ case also passes in both image modes at e4901835 with seven signed probes and source/executable/backend binding. Its backend/VM are stopped, data retained. The extra cleanup-fault case passes actual leader handoff, ambiguous storage failure, retained one-group quota and receipt retry without callback replay. -Twenty-four library regressions pass. Current complete task evidence, affected -enrollment/execution regressions and strict lint remain before closing CG-08. +Twenty-four library regressions and strict lint pass. Complete successor task +evidence and affected execution regressions remain before closing CG-08. See the current [progress](progress.md). Local raw records/continuation remain ignored and backed up outside the checkout. Preserve the user's untracked diagram and both excluded MR progress basenames. Reconcile both checkouts only after the source-bound predecessor run terminates; reuse PR 160 and one final runtime. +The e8edfd12 full gate is now terminal: all ten required commands, four extra +effects cases, twenty-four library regressions and the exact same-binary real +RustFS effects case pass. Its expanded execution target passes 162 cases but +fails two obsolete outbox-projection fixtures; two RustFS cases are ignored. +The test-only correction adds lane/sequence and a blocked later row. Both failing +cases now pass with the earliest runnable head and all pending rows asserted. +Preserve the failed full gate and partial correction records. Rerun the complete +gate at the committed successor before closing CG-08. Its source remains frozen +while a source-bound job runs; poll its exact handle instead of restarting. +The owned RustFS backend and dedicated VM are stopped and default context stays +colima. The separate same-source RustFS record is under rustfs-e8edfd12/ in the +effects checkpoint. The earlier current-source statement above is superseded by +this checkpoint. CG-07's reviewed anchors still match; the spare PR checkout has +a source-bound startup inspection record, without any cutover completion claim. + 1. Finish the due-index validation and wire bounded fair node effect discovery, exact committed-image scans and sealed private system commands into the shared controller. Preserve current CellHistory-based identities and receiver dedupe. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index fbbc0b0c8..019f441cd 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -12,6 +12,27 @@ is being added. ## CG-08 node effects and disposable SQL lifetimes +At e8edfd12, all ten unchanged required commands pass, along with four extra +effects cases and twenty-four library regressions. The same-source/executable +real RustFS effects case passes both image modes with all seven signed probes. +Its record, logs and executable are retained under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/rustfs-e8edfd12/. +The owned backend and dedicated VM are stopped; data and default context remain. + +The expanded execution suite passes 162 cases, fails two and explicitly ignores +two RustFS cases. Both failures use an old summary fixture whose outbox table +contains only next_at; lane-head discovery now also requires lane and sequence. +The fixture correction adds those identities and a third, immediately due row +blocked behind its lane's older sequence. Exact summary assertions still require +the other lane's time 40 and now all three pending rows. Both original failed +cases pass after this test-only correction, in both image forms. Production +validation, resource limits, deadlines and acceptance oracles stay intact. +The failed full gate, source inputs, both regression executables and real streams +remain under .cellar/cell-groups-evidence/CG-08-gates-e8edfd12-01/ and are backed up. +The corrected partial run lives under +.cellar/cell-groups-evidence/CG-08-apply-summary-fixture-e8edfd12-01/. +CG-08 stays IN_PROGRESS until the complete gate passes at the successor source. + The successor implementation is under qualification on codex/shared-cell-groups. The committed due-index foundation is 783c270b. The node controller now owns fair group/cell discovery, exact committed-image scans, private alarm/outbox commands, @@ -78,8 +99,8 @@ remain unchanged. These source-scoped records are retained under .cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/. Private-operation discovery now bounds its SQLite projection before allocating -the scheduling preference string. Its corruption/permit regression, strict -Clippy and the complete CG-08 evidence gate remain required. The later CG-07 +the scheduling preference string. Its corruption/permit regression and strict +Clippy pass; the complete successor CG-08 evidence gate remains required. The later CG-07 public SDK/fleet cutover is still unqualified. CG-06's prior source-scoped closure remains intact; the full hard cutover is unfinished. From 4ea70e483e84ca88d77ca062278edb5ff2389646 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 04:14:14 -0700 Subject: [PATCH 265/568] fix: retain committed outputs during S3 fencing --- .../durability/raft/cell_groups/README.md | 9 + .../raft/cell_groups/dispatch/command/mod.rs | 105 ++--------- .../cell_groups/dispatch/command/output.rs | 129 ++++++++++++++ .../cell_groups/dispatch/command/owned.rs | 82 +++++++-- .../cell_groups/dispatch/command/remote.rs | 6 +- .../tests/native_cell_groups_effects.rs | 8 + .../tests/support/cell_group_runtime.rs | 8 +- .../tests/support/cell_group_s3_commands.rs | 51 +++++- .../tests/support/cell_group_s3_output.rs | 167 ++++++++++++++++++ crates/cellar/tests/support/cell_group_sql.rs | 8 +- docs/design/multi-raft/cell-groups/handoff.md | 13 ++ .../design/multi-raft/cell-groups/progress.md | 26 +++ 12 files changed, 500 insertions(+), 112 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs create mode 100644 crates/cellar/tests/support/cell_group_s3_output.rs diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index 787ac94fa..d11e05dbc 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -484,6 +484,15 @@ it. This returns `ExecutionError::Uncertain` with the original producer retained The owned controller obtains a fresh observation and retries receipt inspection without executing the callback again. Identity validation precedes this check. +After receipt reconciliation retires SQL, S3 output fencing is a separate owned +phase. It buffers the exact committed receipt and verified archive proof under +their existing admission. Later group apply can temporarily refuse current +serving authority; the controller waits for progress within the original deadline +and rechecks route, leader and catalog binding. It neither restarts receipt +reconciliation nor executes another callback. A duplicate receipt uses the same +output phase. Cancellation keeps the admitted owner; terminal errors release +output resources without quarantining a group whose SQL producer already closed. + Shared log-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Unknown events and controller ownership violations fail closed. Request-driven group provisioning, configuration lifecycle and public SDK diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 0acab5f2a..6d5f3359c 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -2,8 +2,10 @@ //! Routed commands retain their originating producer until exact resolution. //! The node controller drives publication; acceptance is only provisional. +mod output; mod owned; mod remote; +use output::PendingOutput; use super::*; use crate::application::consensus::command_request_digest; @@ -107,6 +109,7 @@ pub struct CellCommandInvocation { dispatched: bool, ambiguous: bool, origin: Option, + output: Option, remote_route: Option, _admission: Permit, } @@ -250,6 +253,7 @@ impl CellGroupDispatcher { dispatched: false, ambiguous: false, origin: None, + output: None, remote_route: None, _admission: admission, }) @@ -374,7 +378,7 @@ impl CellCommandInvocation { return Err(self.error(CellDispatchError::Invalid("consumed route attempt"))); } self.dispatched = true; - if self.origin.is_some() { + if self.origin.is_some() || self.output.is_some() { return self.resume().await; } let domain = self.dispatcher.domain.clone(); @@ -523,6 +527,12 @@ impl CellCommandInvocation { } async fn resume_at(&mut self) -> Result { + if self.output.is_some() { + return self + .finish_output() + .await + .map(CellCommandPreparation::Committed); + } let origin = self .origin .as_ref() @@ -659,6 +669,12 @@ impl CellCommandInvocation { } async fn settle_at(&mut self) -> Result { + if self.output.is_some() { + return self + .finish_output() + .await + .map(CellCommandSettlement::Committed); + } let origin = self .origin .as_ref() @@ -705,93 +721,6 @@ impl CellCommandInvocation { Err(error) => Err(CellDispatchError::Execution(error)), } } - - async fn response( - &mut self, - receipt: Box, - ) -> Result { - if receipt.receipt().receipt.command() != self.request.id - || receipt.receipt().receipt.operation != self.request.contract.operation - || receipt.receipt().receipt.contract_digest != self.request.contract.digest - || receipt.receipt().receipt.request_digest - != command_request_digest(&self.request.input) - || receipt.receipt().version.history != self.cell - { - return Err(CellDispatchError::Invalid("command receipt binding")); - } - // SQL has resolved the producer. Archive delay, failure or cancellation - // must not retain an unresolved origin or quarantine this healthy group. - let mut origin = self - .origin - .take() - .ok_or(CellDispatchError::Invalid("resolved command origin"))?; - let required = self - .request - .durability - .max(receipt.receipt().receipt.minimum_durability()); - let (receipt, s3, authority) = if required == Durability::S3 { - let owner = self - .dispatcher - .s3 - .as_ref() - .ok_or(CellDispatchError::Execution(ExecutionError::Refused( - CommandFailure::UnsupportedCapability, - )))?; - let limits = match &origin.options { - CellSqlOptions::Materialized(o) => o.limits, - CellSqlOptions::Extents(o) => o.limits, - }; - // The worker has closed this producer. Its preparation must not - // keep a duplicate SQL admission lease during the archive wait. - origin.prepared = None; - let settled = owner - .command( - &origin.runtime, - origin.route.clone(), - receipt, - limits, - self.session.deadline_ms(), - ) - .await?; - // Archive coverage is historical data durability. Current catalog and - // leader authority remain independent requirements for this output. - let authority = origin - .runtime - .execution_authority(origin.route.history, self.cell.clone()) - .await - .map_err(CellDispatchError::Runtime)?; - if authority.route() != &origin.route { - return Err(CellDispatchError::Invalid("S3 response current route")); - } - settled - .proof - .require_current(&self.dispatcher.domain, self.session.deadline_ms()) - .await?; - authority.check().map_err(CellDispatchError::Runtime)?; - (settled.receipt, Some(settled.proof), Some(authority)) - } else { - (receipt, None, None) - }; - let now = self - .dispatcher - .domain - .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) - .await - .map_err(|_| closed())?; - let token = self - .dispatcher - .keys - .issue_cell( - &receipt.receipt().observation(), - self.dispatcher.class.schema_digest(), - now, - ) - .map_err(CellDispatchError::Version)?; - if let Some(authority) = authority { - authority.check().map_err(CellDispatchError::Runtime)?; - } - Ok(CellCommandResponse { receipt, token, s3 }) - } } fn fingerprint( diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs new file mode 100644 index 000000000..55995adc5 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/output.rs @@ -0,0 +1,129 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Committed outputs retain proof admission, independently of retired SQL work. +use super::*; + +pub(super) struct PendingOutput { + runtime: CellGroupRuntime, + route: GroupRoute, + receipt: Box, + proof: Box, +} + +impl CellCommandInvocation { + pub(super) async fn response( + &mut self, + receipt: Box, + ) -> Result { + if receipt.receipt().receipt.command() != self.request.id + || receipt.receipt().receipt.operation != self.request.contract.operation + || receipt.receipt().receipt.contract_digest != self.request.contract.digest + || receipt.receipt().receipt.request_digest + != command_request_digest(&self.request.input) + || receipt.receipt().version.history != self.cell + { + return Err(CellDispatchError::Invalid("command receipt binding")); + } + // SQL has resolved its producer. Archive failure and output fencing + // cannot turn that committed result back into unresolved SQL ownership. + let mut origin = self + .origin + .take() + .ok_or(CellDispatchError::Invalid("resolved command origin"))?; + let required = self + .request + .durability + .max(receipt.receipt().receipt.minimum_durability()); + if required == Durability::S3 { + let owner = self + .dispatcher + .s3 + .as_ref() + .ok_or(CellDispatchError::Execution(ExecutionError::Refused( + CommandFailure::UnsupportedCapability, + )))?; + let limits = origin.options.limits(); + // Do not retain the preparation's SQL slot across archive waiting. + origin.prepared = None; + let settled = owner + .command( + &origin.runtime, + origin.route.clone(), + receipt, + limits, + self.session.deadline_ms(), + ) + .await?; + self.output = Some(PendingOutput { + runtime: origin.runtime, + route: origin.route, + receipt: settled.receipt, + proof: settled.proof, + }); + return self.finish_output().await; + } + let token = self.output_token(&receipt).await?; + Ok(CellCommandResponse { + receipt, + token, + s3: None, + }) + } + + pub(super) async fn finish_output(&mut self) -> Result { + let output = self + .output + .as_ref() + .ok_or(CellDispatchError::Invalid("missing committed output"))?; + // A later committed entry may temporarily put the group ahead of apply. + // Retain this exact result/proof until a fresh serving fence is available; + // never reconcile again or rerun SQL to reconstruct the response. + let authority = output + .runtime + .execution_authority(output.route.history, self.cell.clone()) + .await + .map_err(CellDispatchError::Runtime)?; + if authority.route() != &output.route { + return Err(CellDispatchError::Invalid("S3 response current route")); + } + output + .proof + .require_current(&self.dispatcher.domain, self.session.deadline_ms()) + .await?; + authority.check().map_err(CellDispatchError::Runtime)?; + let token = self.output_token(&output.receipt).await?; + authority.check().map_err(CellDispatchError::Runtime)?; + if self.dispatcher.domain.mono_ms().map_err(|_| closed())? >= self.session.deadline_ms() { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let output = self + .output + .take() + .ok_or(CellDispatchError::Invalid("consumed committed output"))?; + Ok(CellCommandResponse { + receipt: output.receipt, + token, + s3: Some(output.proof), + }) + } + + async fn output_token( + &self, + receipt: &CellSqlReceipt, + ) -> Result { + let now = self + .dispatcher + .domain + .run(async { crate::asyncrt::wall_ms().max(0) as u64 }) + .await + .map_err(|_| closed())?; + self.dispatcher + .keys + .issue_cell( + &receipt.receipt().observation(), + self.dispatcher.class.schema_digest(), + now, + ) + .map_err(CellDispatchError::Version) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs index 047b03b38..179bde4be 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/owned.rs @@ -13,6 +13,7 @@ impl CellCommandInvocation { || self.attempt.is_some() || self.dispatched || self.origin.is_some() + || self.output.is_some() { return Err(self.error(CellDispatchError::Invalid( "controller requires a fresh local invocation", @@ -112,9 +113,13 @@ impl CellCommandInvocation { self.dispatched = true; // Unlike the response-facing phase API, do not drop this SQL waiter at // the response deadline. It carries the exact preparation or refusal. - match self.prepare_at(&attempt, runtime, sql, options).await? { - CellCommandPreparation::Committed(response) => return Ok(response), - CellCommandPreparation::Prepared => (), + match self.prepare_at(&attempt, runtime, sql, options).await { + Ok(CellCommandPreparation::Committed(response)) => return Ok(response), + Ok(CellCommandPreparation::Prepared) => (), + Err(error) if self.output.is_some() && output_retryable(&error) => { + return self.finish_owned_output(progress).await; + } + Err(error) => return Err(error), } loop { if self.expired()? { @@ -169,20 +174,69 @@ impl CellCommandInvocation { reason, ))) } - Err(CellDispatchError::Runtime( - RuntimeError::NotReady - | RuntimeError::NotLeader - | RuntimeError::Overloaded - | RuntimeError::StaleCompletion, - )) - | Err(CellDispatchError::Execution( - ExecutionError::Uncertain - | ExecutionError::Busy - | ExecutionError::Refused(CommandFailure::Overloaded), - )) => (), + Err( + error @ CellDispatchError::Runtime( + RuntimeError::NotReady + | RuntimeError::NotLeader + | RuntimeError::Overloaded + | RuntimeError::StaleCompletion, + ), + ) + | Err( + error @ CellDispatchError::Execution( + ExecutionError::Uncertain + | ExecutionError::Busy + | ExecutionError::Refused(CommandFailure::Overloaded), + ), + ) => { + if self.output.is_some() && output_retryable(&error) { + return self.finish_owned_output(progress).await; + } + // SQL has already resolved before an archive/output error. + // Never restart reconciliation after retiring its origin. + if self.origin.is_none() { + return Err(error); + } + } Err(error) => return Err(error), } progress.changed().await.map_err(|_| closed())?; } } + + async fn finish_owned_output( + &mut self, + progress: &mut watch::Receiver<()>, + ) -> Result { + let domain = self.dispatcher.domain.clone(); + let deadline = self.session.deadline_ms(); + loop { + let timer = domain.sleep_until(deadline).map_err(|_| closed())?; + let result = crate::asyncrt::select! { + result = self.finish_output() => result, + _ = timer => Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + }; + match result { + Ok(response) => return Ok(response), + Err(error) if output_retryable(&error) => (), + Err(error) => return Err(error), + } + // This is buffered committed output, not an unresolved SQL producer. + // Progress wakes a fresh fence check without callbacks or proposals. + let timer = domain.sleep_until(deadline).map_err(|_| closed())?; + crate::asyncrt::select! { + result = progress.changed() => result.map_err(|_| closed())?, + _ = timer => return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)), + } + } + } +} + +fn output_retryable(error: &CellDispatchError) -> bool { + matches!( + error, + CellDispatchError::Runtime( + RuntimeError::NotReady | RuntimeError::Overloaded | RuntimeError::StaleCompletion + ) + ) } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs index afdef62a2..f210b39c0 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/remote.rs @@ -28,7 +28,11 @@ impl CellCommandInvocation { self.session .validate_attempt(&attempt) .map_err(|e| self.error(CellDispatchError::Routing(e)))?; - if self.dispatched || self.origin.is_some() || self.remote_route.is_some() { + if self.dispatched + || self.origin.is_some() + || self.output.is_some() + || self.remote_route.is_some() + { return Err(self.error(CellDispatchError::Invalid( "consumed or local command attempt", ))); diff --git a/crates/cellar/tests/native_cell_groups_effects.rs b/crates/cellar/tests/native_cell_groups_effects.rs index 74296d16c..c722d44cd 100644 --- a/crates/cellar/tests/native_cell_groups_effects.rs +++ b/crates/cellar/tests/native_cell_groups_effects.rs @@ -76,6 +76,14 @@ async fn s3_claim_receiver_and_completion_use_verified_coverage() -> Result<()> Ok(()) } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn committed_s3_output_survives_later_group_apply() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::output_during_apply(rooted)).await?; + } + Ok(()) +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] #[ignore = "requires isolated local RustFS"] async fn rustfs_claim_receiver_and_completion_use_verified_coverage() -> Result<()> { diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 36a2090ed..b680fb7aa 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -818,10 +818,10 @@ pub(super) use sql::{ pub(super) use sql::{archive_controller_publish_store, archive_controller_reconcile_store}; #[allow(unused_imports)] pub(super) use sql::{ - archive_dependencies, full_backend_outage, s3_commands, s3_deadline, s3_enrollment, - s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, - s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, - s3_remote_outputs, s3_remote_slow_delivery, + archive_dependencies, full_backend_outage, output_during_apply, s3_commands, s3_deadline, + s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, + s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, + s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, }; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 466241d28..52d39d958 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -5,6 +5,10 @@ mod dependencies; pub(crate) use dependencies::archive_dependencies; +#[path = "cell_group_s3_output.rs"] +mod output; +pub(crate) use output::output_during_apply; + #[path = "cell_group_s3_outage.rs"] mod outage; pub(crate) use outage::full_backend_outage; @@ -39,6 +43,14 @@ struct Harness { retry_until: u64, } +enum Resources { + Retention(CellGroupLogRetentionLimits), + Storage { + root: tempfile::TempDir, + filesystem: Arc, + }, +} + impl Harness { async fn new( rooted: bool, @@ -95,6 +107,27 @@ impl Harness { result_size: Option, send_buffer: Option, retention: Option, + ) -> Result { + Self::configured_resources( + rooted, + store, + max_commands, + minimum, + result_size, + send_buffer, + retention.map(Resources::Retention), + ) + .await + } + + async fn configured_resources( + rooted: bool, + store: Arc, + max_commands: usize, + minimum: Durability, + result_size: Option, + send_buffer: Option, + resources: Option, ) -> Result { let calls = Arc::new(AtomicUsize::new(0)); let count = calls.clone(); @@ -122,7 +155,8 @@ impl Harness { }, )?, ); - let (mut fixture, route, cell, _) = if let Some(retention) = retention { + let (mut fixture, route, cell, _) = if let Some(Resources::Retention(retention)) = resources + { let filesystem = Arc::new(DirectFileSystem); let cluster = Box::pin(Cluster::with_store_retention( RaftRuntimeOptions::default(), @@ -135,6 +169,21 @@ impl Harness { )) .await?; Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? + } else if let Some(Resources::Storage { root, filesystem }) = resources { + let cluster = Cluster::with_store_birth( + RaftRuntimeOptions::default(), + root, + filesystem.clone(), + class.schema_digest(), + store, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + minimum, + log: ClusterLogLimits::default(), + }, + ) + .await?; + Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? } else { Box::pin(registered::setup_store_minimum( rooted, &class, store, minimum, diff --git a/crates/cellar/tests/support/cell_group_s3_output.rs b/crates/cellar/tests/support/cell_group_s3_output.rs new file mode 100644 index 000000000..45bdd05ec --- /dev/null +++ b/crates/cellar/tests/support/cell_group_s3_output.rs @@ -0,0 +1,167 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A resolved SQL producer cannot be reconciled again during S3 output fencing. +use super::*; + +pub(crate) async fn output_during_apply(rooted: bool) -> Result<()> { + for duplicate in [false, true] { + Box::pin(run(rooted, duplicate)).await?; + } + Ok(()) +} + +async fn run(rooted: bool, duplicate: bool) -> Result<()> { + let root = tempfile::tempdir()?; + let faults = apply_faults::physical::Faults::new(root.path().join("node-1/pages"), None); + let filesystem = Arc::new(apply_faults::physical::FaultFileSystem { + faults: faults.clone(), + }); + let h = Harness::configured_resources( + rooted, + Arc::new(object_store::memory::InMemory::new()), + 2, + Durability::Quorum, + None, + None, + Some(Resources::Storage { root, filesystem }), + ) + .await?; + let publication = h.remote.hold_publication(); + + struct Publication(Arc); + + impl Drop for Publication { + fn drop(&mut self) { + self.0.release.notify_one(); + } + } + let publication_release = Publication(publication.clone()); + if duplicate { + // The duplicate enters S3 settlement from preparation's receipt path, + // rather than reconciling a newly accepted SQL producer. + let first = h + .command( + 1, + b"retained-output", + Durability::Quorum, + Duration::from_secs(5), + ) + .await? + .response() + .await?; + assert_eq!(first.receipt().receipt.result(), b"retained-output"); + drop(first); + } + let call = h + .command( + 1, + b"retained-output", + Durability::S3, + Duration::from_secs(5), + ) + .await?; + tokio::time::timeout(Duration::from_secs(5), publication.entered.notified()).await?; + h.waiting().await?; + + // The first producer has retired. Hold a later cell's real manifest sync, + // without enlarging the one-item SQL admission or changing the old image. + let other = history(&h.route, "registered-b")?; + let authority = h.fixture.runtimes[0] + .execution_authority(h.route.history, other.clone()) + .await?; + let options = h.fixture.cell_options(&other).await?; + let pool = h.fixture.appliers[&(0, h.route.history)].0.clone(); + let CellSqlOutcome::Prepared(private) = pool + .stage(options, authority, cell_attempt(239)?, true, |tx| { + tx.execute("UPDATE value SET n=n+1", [])?; + Ok(b"blocking-result".to_vec()) + }) + .await? + else { + anyhow::bail!("private preparation") + }; + pool.mark_submitted(other.clone(), private.proposal()) + .await?; + let original = h.fixture.runtimes[0] + .reconciliation_authority(h.route.history, h.cell.clone()) + .await? + .cell() + .last_mutation; + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + + struct Unblock(std::sync::mpsc::SyncSender<()>); + + impl Drop for Unblock { + fn drop(&mut self) { + let _ = self.0.try_send(()); + } + } + let unblock = Unblock(release); + *faults.before_directory_sync.lock().unwrap() = Some(Box::new(move || { + let _ = entered.send(()); + blocked + .recv_timeout(Duration::from_secs(5)) + .expect("release real manifest sync"); + })); + let accepted = accepted_sql(&h.fixture.runtimes[0], &private).await?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + controller::wait_for(async || { + let status = h.fixture.runtimes[0].status(h.route.history).await?; + Ok(status.committed >= accepted.position.index + && status.applied.index < accepted.position.index) + }) + .await?; + drop(publication_release); + archive_controller::published(&h.controllers[0], h.route.history, original.index).await?; + let mut response = Box::pin(call.response()); + assert!( + tokio::time::timeout(Duration::from_millis(30), response.as_mut()) + .await + .is_err(), + "committed output must wait for fresh authority without revisiting its retired SQL producer" + ); + assert_eq!(h.controllers[0].usage().quarantined_groups, 0); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(unblock); + controller::applied(&h.fixture, h.route.history, accepted.position.index).await?; + let authority = h.fixture.runtimes[0] + .reconciliation_authority(h.route.history, other.clone()) + .await?; + let CellSqlReconciliation::Committed(blocker_receipt) = + pool.reconcile(private.proposal(), authority).await? + else { + anyhow::bail!("blocking command receipt") + }; + assert_eq!( + blocker_receipt.receipt().receipt.result(), + b"blocking-result" + ); + drop(blocker_receipt); + drop(private); + let receipt = response.await?; + assert_eq!(receipt.durability(), Durability::S3); + assert_eq!(receipt.receipt().receipt.result(), b"retained-output"); + assert_eq!(receipt.receipt().version.sequence.get(), 2); + drop(receipt); + let retry = h + .command( + 1, + b"retained-output", + Durability::S3, + Duration::from_secs(5), + ) + .await? + .response() + .await?; + assert_eq!(retry.receipt().receipt.result(), b"retained-output"); + assert_eq!(h.calls.load(Ordering::SeqCst), 1); + drop(retry); + for runtime in &h.fixture.runtimes { + let cell = runtime + .reconciliation_authority(h.route.history, other.clone()) + .await?; + assert_eq!(cell.cell().version.sequence.get(), 2); + } + h.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 18d71033f..ac4329483 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -538,10 +538,10 @@ mod s3_commands; mod s3_response; #[allow(unused_imports)] pub(crate) use s3_commands::{ - archive_dependencies, full_backend_outage, s3_commands, s3_deadline, s3_enrollment, - s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, s3_receipt_deadline, - s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, - s3_remote_outputs, s3_remote_slow_delivery, + archive_dependencies, full_backend_outage, output_during_apply, s3_commands, s3_deadline, + s3_enrollment, s3_enrollment_deadline, s3_missing_sibling, s3_namespace_minimum, + s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, + s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, }; #[allow(unused_imports)] pub(crate) use s3_response::unproved_s3_responses; diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 0b027fdbd..31d1b73b7 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -13,6 +13,19 @@ node resources. A group spans voter nodes; it is not all cells on one node. ## Starting point +Latest checkpoint supersedes the earlier run status below: the full 8a4c5ae0 +twenty-command gate passed (164 execution cases, two RustFS ignores), but its +fully serial RustFS case found a real committed-output fencing race. The current +runtime correction separates buffered S3 receipt/proof ownership from retired +SQL. Its deterministic manifest-sync regression passes both image modes for new +commands and duplicate receipts, without raising the one-item SQL quota or +resetting deadlines. Preserve the valid failing baseline and all constructor +failures under CG-08-output-repro-01 through -04, and passing focused snapshots +under CG-08-output-fixed-01/-02. The runtime correction still requires fresh +all-target/lint/native/archive regressions and serial real RustFS qualification; +CG-08 remains IN_PROGRESS. Both owned backend/VM stop operations returned zero. +Old 8a/e8 closure scripts are obsolete after this production change. + Use branch codex/shared-cell-groups and the existing [PR 160](https://github.com/crabbuild/cellar/pull/160). CG-01 through CG-06 now pass their current source-scoped gates. CG-06 is DONE diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 019f441cd..f0ab1e05b 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -12,6 +12,32 @@ is being added. ## CG-08 node effects and disposable SQL lifetimes +The complete successor gate passes all twenty commands at 8a4c5ae0, including +164 execution cases with zero failures and two explicit RustFS ignores. The +subsequent fully serial real RustFS case exposed a production output-phase race: +SQL had committed and retired, but a later group entry temporarily prevented +fresh serving authority after archive settlement. The driver retried SQL +reconciliation with no remaining origin and returned an ambiguous missing-origin +error. The passing native gate and the serial RustFS failure remain retained; +the owned backend and VM stopped normally, with data and default context intact. + +The correction retains the exact receipt and archive proof in a distinct pending +output phase. Progress retries only current authority under the original deadline; +it cannot reopen SQL, replay a callback or turn completed SQL into quarantine +debt. Both newly committed commands and duplicate receipts use this phase. The +regression holds a real manifest-directory sync on another prepared cell under +the original one-item SQL limit. It establishes committed greater than applied, +requires the first S3 response to remain pending, then checks its exact result, +receipt retry, one callback execution, healthy group and every replica's applied +successor. The valid baseline fails before the correction; the focused corrected +case passes materialized and extent modes for both entry paths in 2.29 seconds. +Its snapshots, executable and real streams are retained under +.cellar/cell-groups-evidence/CG-08-output-repro-04/ and +.cellar/cell-groups-evidence/CG-08-output-fixed-02/. Earlier fixture construction +failures remain recorded separately. No production refusal, deadline or quota +was weakened. The new runtime requires a fresh complete gate and serial RustFS +qualification before CG-08 can become DONE. + At e8edfd12, all ten unchanged required commands pass, along with four extra effects cases and twenty-four library regressions. The same-source/executable real RustFS effects case passes both image modes with all seven signed probes. From 3ce0a934fb2fdeb40bc48894acda41c0a26d9489 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 05:25:24 -0700 Subject: [PATCH 266/568] docs: close shared effects qualification --- docs/design/multi-raft/cell-groups/README.md | 34 ++-- docs/design/multi-raft/cell-groups/handoff.md | 172 +++++++----------- .../cell-groups/implementation-plan.md | 4 +- .../design/multi-raft/cell-groups/progress.md | 67 ++++++- docs/design/multi-raft/cell-groups/tasks.json | 12 +- 5 files changed, 157 insertions(+), 132 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index e9b7879e9..76bfb7bae 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -10,8 +10,8 @@ replace the unavailable historical raw artifacts. CG-06's complete archive, local reclamation and fresh-history recovery boundary also passes both actual Docker/RustFS all-voter-disks-deleted drills with permanent launch revocation, fresh-quorum activation, exact receipts, new writes and retained restart. -CG-08 due effects and safe cache lifetimes are IN_PROGRESS, before CG-07 SDK/fleet -cutover. See the latest [progress](progress.md) and [handoff](handoff.md). +CG-08 due effects and safe cache lifetimes now pass the complete native gate and +same-source/executable RustFS case. CG-07 SDK/fleet cutover has started in the spare checkout and remains unqualified. See the latest [progress](progress.md) and [handoff](handoff.md). The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. @@ -38,7 +38,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package behavior and final release gates; they supersede fixed assignment as the final target. Fixed assignment is an intermediate milestone only. 2. Run the commands below from the repository root. `ready` reports dependency - eligibility; `show CG-06` emits a complete executor brief, source excerpts, + eligibility; `show CG-07` emits a complete executor brief, source excerpts, scope, steps, exact test commands and completion rules. It never runs them. 3. Implement each task in dependency order on the active PR branch or an explicitly assigned branch. Preserve working intermediate commits. Do not expose a second @@ -49,7 +49,7 @@ The plan runner requires Python 3.11 or newer and Git; it has no Python package Current source-scoped records live under the ignored `.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/` directory, with one -`CG-NN-record.json` per completed task. Logs, six copied executables and all source +`CG-NN-record.json` per CG-01–06 task. Logs, six copied executables and all source hashes are retained locally and backed up on the workspace volume. A fresh checkout must receive the same repository-relative evidence paths, or rerun the unchanged exact gates before full package validation. The separately scoped local @@ -58,13 +58,16 @@ Docker/RustFS record is retained under records remain unavailable; their earlier source-scoped results are documented in progress.md and are superseded by the fresh native records. Missing or changed evidence still fails validation. +CG-08 current-source evidence lives under +`.cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/`; same-executable RustFS +evidence lives under the `rustfs-4ea70e48/` effects checkpoint described in progress.md. The design, task briefs and validator unit tests are committed in this package. ```sh python3 docs/design/multi-raft/cell-groups/plan.py validate python3 docs/design/multi-raft/cell-groups/plan.py preflight python3 docs/design/multi-raft/cell-groups/plan.py ready -python3 docs/design/multi-raft/cell-groups/plan.py show CG-06 +python3 docs/design/multi-raft/cell-groups/plan.py show CG-07 python3 -m unittest discover -s docs/design/multi-raft/cell-groups -p 'test_*.py' ``` @@ -83,14 +86,19 @@ qualification belongs only to the harness's recorded, owned disposable resources ### Current executor handoff -Continue from the existing code and retained evidence. CG-01 through CG-05 are -DONE at their declared native boundaries; CG-06 is dependency-ready and requires -implementation. Read [progress](progress.md), then run `plan.py show CG-06` for the -complete archive/reclamation brief. Preserve every group's selected current and -checkpoint closure, exact cell identities and permanent physical claims. Only -verified complete remote coverage may settle S3 outputs or authorize archive-policy -reclamation. A task's current-source gate must pass before marking it DONE; final -release repeats the complete capability/profile matrix at one revision. +CG-01 through CG-06 and CG-08 are DONE at their declared scoped boundaries. +CG-07 is IN_PROGRESS in the spare checkout, with incomplete canonical startup changes. +Read [progress](progress.md) and [handoff](handoff.md), then run `plan.py show CG-07` +for canonical SDK/host/fleet cutover. Keep complete group closure, logical cell +identity, uncertainty, managed admission and physical ownership intact. The public +cutover and all CG-09 through CG-16 capabilities and release profiles remain +required. No intermediate gate proves the final combined runtime. + +### Historical implementation checkpoints + +These checkpoints describe earlier source-scoped work. Current records above +supersede unavailable historical raw artifacts; the current handoff determines +what remains. Earlier “next” or IN_PROGRESS statements below are historical. Physical group genesis, typed replica handles, permanent initial-member claims, complete group inventory and invocation-bounded catalog-confirmed redirects are diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 31d1b73b7..6b99eeddf 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -13,116 +13,68 @@ node resources. A group spans voter nodes; it is not all cells on one node. ## Starting point -Latest checkpoint supersedes the earlier run status below: the full 8a4c5ae0 -twenty-command gate passed (164 execution cases, two RustFS ignores), but its -fully serial RustFS case found a real committed-output fencing race. The current -runtime correction separates buffered S3 receipt/proof ownership from retired -SQL. Its deterministic manifest-sync regression passes both image modes for new -commands and duplicate receipts, without raising the one-item SQL quota or -resetting deadlines. Preserve the valid failing baseline and all constructor -failures under CG-08-output-repro-01 through -04, and passing focused snapshots -under CG-08-output-fixed-01/-02. The runtime correction still requires fresh -all-target/lint/native/archive regressions and serial real RustFS qualification; -CG-08 remains IN_PROGRESS. Both owned backend/VM stop operations returned zero. -Old 8a/e8 closure scripts are obsolete after this production change. - -Use branch codex/shared-cell-groups and the existing -[PR 160](https://github.com/crabbuild/cellar/pull/160). -CG-01 through CG-06 now pass their current source-scoped gates. CG-06 is DONE -at its unmanaged archive/reclamation/disaster-recovery boundary. Native gate -source: e32115aa3218dc6eaa7ed53e134f413f26305938, including origin/main -81cbbc2f72f302751dd471cf06cf51394146ac63. All 123 unchanged gate commands pass: -110 exact native cases, nineteen pure-logic tests, all-target check/Clippy, -format/layout and dependency/documentation checks. Six retained executable hashes -and 3257 source hashes bind the evidence. - -Fresh records replace the unavailable CG-01–05 raw artifacts in tasks.json: -.cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/CG-01-record.json through -CG-06-record.json. They are backed up outside the checkout. Each record passes the -unchanged evidence validator. Full package validation passes with six DONE -tasks and 44 local links. CG-06 record SHA256: -29cfff144bb38f5cda6968e5a8d2cf163ff440df0562ec8948059176ab7827b8. -The reclaim case was rerun with separate real stdout/stderr after its original -passing merged log's diagnostics interrupted the test result line. Both logs -remain; no test/validator expectation was weakened. - -Actual shared-group Docker voters pass permanent fencing and all-voter-disks- -deleted RustFS recovery in both image forms at f8aacd05. Its executable inputs -match the native gate source (only plan documents differ). Pending admission, -fresh activation/quorum, receipts/floors/S3 minima, old-token/launch refusal, new -writes, retained restart and every replica's final state pass. All remaining -processes join normally; the VM is stopped, data retained and default context -preserved. See [progress](progress.md) and tools/multi-raft/README.md. -Scoped Docker record: -.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json -(SHA256 08cf2112d4f91519c00d481a6cea3f8c0e378a242ecef258ac67a182c265e77d). -Keep credentials, signing material and raw tokens private. Docker API config -copies avoid the reproduced virtiofs truncation race; controller keys/launch -ledgers never enter containers. Failed predecessors remain recorded. - -CG-08 is IN_PROGRESS, followed by CG-07 SDK/fleet cutover. The committed due-index -foundation is 783c270b. The isolated effects checkout retains the exact predecessor -five-case execution run. Its expanded compile check and all five native-10 -lifecycle cases pass in both image modes. The preceding native-09 run passes four -cases while an ordinary follower campaign times out under the healthy leader -lease. The test-only correction uses real TimeoutNow transfer inside the same -deadline. Preserve both source/executable/log records. Native-10 is terminal; -source can now be reconciled with the successor checkout. - -The existing PR checkout contains the successor implementation and new -native_cell_groups_effects target: exact cold-cache reopening, cold-cell discovery, -logical delivery/proof identity under two independent physical group bindings, -SQL/dispatch custody, joined shutdown, and normal/explicit RustFS S3 closure cases. -All eight successor native cases now pass in both image modes, including the five -declared cases and S3 coverage through the normal adapter. The exact real RustFS -case also passes in both image modes at e4901835 with seven signed probes and -source/executable/backend binding. Its backend/VM are stopped, data retained. -The extra cleanup-fault case passes actual leader handoff, ambiguous storage -failure, retained one-group quota and receipt retry without callback replay. -Twenty-four library regressions and strict lint pass. Complete successor task -evidence and affected execution regressions remain before closing CG-08. -See the current [progress](progress.md). Local raw records/continuation remain -ignored and backed up outside the checkout. Preserve the user's untracked diagram -and both excluded MR progress basenames. Reconcile both checkouts only after the -source-bound predecessor run terminates; reuse PR 160 and one final runtime. - -The e8edfd12 full gate is now terminal: all ten required commands, four extra -effects cases, twenty-four library regressions and the exact same-binary real -RustFS effects case pass. Its expanded execution target passes 162 cases but -fails two obsolete outbox-projection fixtures; two RustFS cases are ignored. -The test-only correction adds lane/sequence and a blocked later row. Both failing -cases now pass with the earliest runnable head and all pending rows asserted. -Preserve the failed full gate and partial correction records. Rerun the complete -gate at the committed successor before closing CG-08. Its source remains frozen -while a source-bound job runs; poll its exact handle instead of restarting. -The owned RustFS backend and dedicated VM are stopped and default context stays -colima. The separate same-source RustFS record is under rustfs-e8edfd12/ in the -effects checkpoint. The earlier current-source statement above is superseded by -this checkpoint. CG-07's reviewed anchors still match; the spare PR checkout has -a source-bound startup inspection record, without any cutover completion claim. - -1. Finish the due-index validation and wire bounded fair node effect discovery, - exact committed-image scans and sealed private system commands into the shared - controller. Preserve current CellHistory-based identities and receiver dedupe. -2. Drive alarm/outbox work without client requests, retaining unknown delivery and - SQL producer ownership through leadership change, cancellation and shutdown. - Qualify the current close-on-settle/exact-root reopen policy under hot/cold work. -3. Complete the original five CG-08 cases, then cut public SDK/fleet consumers over - in CG-07. Topology, hibernation, managed recovery, load/crash qualification and - removal of old consumers remain later tasks. The full goal stays active. - -The required nullable recovery metadata remains an internal marker-1 hard -cutover. There is no legacy reader, state migration or permanent mode flag. -PreparedCellGroupRecovery is private data preparation; only the sealed physical -seed, recovered-bootstrap and activation contracts admit their respective next -steps. Never reopen owned page/image directories or cast archive ownership into -engine authority. Keep unknown outcomes and physical joins owned. - -CG-08 follows CG-06: implement bounded fair due-work scheduling independent of -client requests and qualify the current close-on-settle/exact-root reopen policy. -Preserve effect identity and unresolved delivery outcomes. Then perform CG-07 -public SDK/fleet cutover. CG-09–16 topology, placement, hibernation, managed -recovery, crash/load qualification and final legacy removal remain required. +Reuse codex/shared-cell-groups and [PR 160](https://github.com/crabbuild/cellar/pull/160). +The isolated effects checkout and PR checkout were reconciled at 4ea70e483e84ca88d77ca062278edb5ff2389646. +Preserve the user's untracked diagram and both excluded MR progress basenames. +The current goal remains incomplete: the production SDK/fleet still uses its +old host and no final combined release or load envelope is qualified. + +CG-01 through CG-06 are DONE at their scoped native boundaries. Their refreshed +records remain under .cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/. +CG-06 also passes the actual Docker-fenced, all-voter-disks-deleted RustFS drill +in both image forms at f8aacd05. Fresh history/genesis/quorum activation, receipt +restoration without replay, S3 minima, old-launch refusal, new writes and retained +restart are qualified at that unmanaged local boundary. Its Docker record is +.cellar/cell-groups-evidence/CG-06-docker-f8aacd05/qualification.json. + +CG-08 is now DONE at 4ea70e483e84ca88d77ca062278edb5ff2389646. All twenty-two current-source commands +pass, including all ten original required commands, ten exact native effects +cases, twenty-four library regressions and the full shared execution target +(164 passed, 2 ignored), and the affected archive-controller suite +(16 passed, 20 explicit RustFS ignores). The same-source/executable real RustFS case passes +both image modes with seven signed probes and independent full archive replay. +The backend and dedicated VM are stopped, data retained and default context +preserved. Source preflight and unchanged task evidence validation pass. +Record: .cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json +SHA256: a072fb6fd8957b6afd5fdf4dffc88c312a0e34c2e1f5c4cfd81b4d2456ea9d0c +Executable SHA256: 85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7cce6 +All source hashes, real streams, original capture record and executable are +retained locally and backed up. See [progress](progress.md) for scope and failures. + +CG-07 is IN_PROGRESS in the spare PR checkout. Its first canonical node service +and startup changes now select CellGroupRuntime and shared transport. They are +uncommitted and intentionally incomplete: the old host/SDK/fleet consumers still +need replacement before that checkout can compile. Preserve both dirty Rust +files and the user's diagrams. Do not restore legacy startup or add a mode flag. +The exact source state and remaining work are recorded in the spare checkout's +.cellar/cell-groups-evidence/CG-07-cutover-progress.json. +Its reviewed source preflight still matches. +Read plan.py show CG-07 and the existing production callers before editing. + +1. Replace canonical host/startup/registration with group placement and cell + enrollment; do not add a legacy/shared toggle. Use the existing shared runtime, + controller, coalesced provisioning, directory and registered dispatch owners. +2. Replace per-cell binding capacity with bounded namespace/class registrations + and separate group/cell occupancy. SDK handles stay cell-addressed and carry + logical cell versions with their original commit witnesses. Preserve typed + ambiguity, caller command identities, strong reads and ingress authorization. +3. Convert managed import, capacity, repair, drain, maintenance, stop/return/resume + and topology to complete group replicas while retaining CAS journals, + permissions and disruption debt. Update native/Axum/admin/CLI/examples together. +4. Qualify all five original CG-07 cases and its complete source/evidence gate. + Occupancy one must use the same path. Feature-off Fleet contracts remain intact. + +The source-bound investigation record in the spare PR checkout is ignored: +.cellar/cell-groups-evidence/CG-07-startup-inspection-e8edfd12.json. +It records integration gaps, not implementation evidence. CG-09 through CG-16 +remain TODO: transfers, topology, placement, hibernation, managed recovery, +workload/crash qualification and deletion of superseded consumers. +The full goal stays active. Final CG-16 repeats every required capability and +profile at one release source; earlier isolated passes cannot qualify it. + +Required nullable recovery metadata stays an internal marker-1 hard cutover. +There is no legacy reader or state migration. Prepared recovery data alone is +never engine/serving authority. Keep unknown outcomes and physical joins owned. The prior whole-group fencing-barrier source is 65c5176975907fb52a53deca58e1ae1f5ba7adec, with 211 native cases and four real RustFS diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index cbc8bc390..2aa267431 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -18,8 +18,8 @@ never executed by the plan runner. Read the [design](design.md), | CG-04 | Execute many independent SQLite cells through one real Raft group | CG-01, CG-02, CG-03 | DONE | 83 | | CG-05 | Snapshot and rebuild complete groups through membership changes | CG-04 | DONE | 5 | | CG-06 | Archive and reclaim the complete group recovery history | CG-04, CG-05 | DONE | 5 | -| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | IN_PROGRESS | 5 | -| CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | TODO | 5 | +| CG-08 | Preserve per-cell effects and bounded idle execution caches | CG-04, CG-05, CG-06 | DONE | 5 | +| CG-07 | Cut public SDK routing and managed fleet over to shared groups | CG-03, CG-04, CG-05, CG-06, CG-08 | IN_PROGRESS | 5 | | CG-09 | Implement irreversible source sealing and destination activation | CG-07, CG-08 | TODO | 7 | | CG-10 | Split hot groups and merge compatible adjacent ranges | CG-09 | TODO | 5 | | CG-11 | Schedule group placement and adaptive partition size | CG-07, CG-10 | TODO | 5 | diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index f0ab1e05b..40b75995a 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -5,12 +5,73 @@ The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). -CG-01 through CG-06 have passed their scoped native gates; -CG-08 is IN_PROGRESS; CG-07 and CG-09 through CG-16 remain TODO. Intermediate contracts +CG-01 through CG-06 and CG-08 have passed their scoped native gates. +CG-07 is IN_PROGRESS; CG-09 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. -## CG-08 node effects and disposable SQL lifetimes +## CG-08 complete effects, cache and RustFS gate + +CG-08 is DONE at its bounded shared-runtime effects/cache boundary. Qualified +source: 4ea70e483e84ca88d77ca062278edb5ff2389646, including origin/main +81cbbc2f72f302751dd471cf06cf51394146ac63. All twenty-two commands pass: the unchanged +ten-command task gate, five additional exact effects cases, twenty-four library +regressions, the complete shared execution target (164 passed, 2 ignored), +the affected archive-controller suite (16 passed, 20 explicit RustFS ignores), +and dependency/documentation checks. The task's reviewed source anchors still +match. Locked all-target check/Clippy, format and layout pass; no source edits +occurred during the run. + +The ten exact native effects cases cover both image forms where applicable: +logical delivery/proof identity across physical bindings, exact cold reopening, +due discovery without client traffic, hot/cold fairness, cancellation and actual +leader transfer, lost receiver replies, held callback/delivery shutdown, S3 claim +and completion coverage, and real materialized cleanup failure. The cleanup-fault +case retains ambiguous outcome and one-group capacity, refuses another group's +registration, and returns the original receipt on the new leader without replay. +No worker, timer or SQL connection is retained for every idle cell. + +Serial RustFS at the preceding 8a source exposed a committed-output race despite +its passing native gate. SQL had retired before fresh output authority became +available; retrying SQL reconciliation then produced missing-origin. The current +pending-output phase retains the exact receipt and archive proof, waits only for +a fresh serving fence under the original deadline, and preserves typed terminal +errors. It cannot resurrect a SQL producer, replay a callback or quarantine a +healthy group. The deterministic physical manifest-sync regression fails before +the fix and passes both image modes for new and duplicate receipt paths. Failed +setup attempts, the valid failing baseline and corrected passing snapshots are +retained under the CG-08-output-repro/fixed checkpoint directories. + +The explicit real RustFS case passes at this same source and exact effects +executable in both image modes. Seven signed S3 probes pass. Claim settlement +precedes delivery; final complete image/receipt closure is independently restored. +This is a host-native TLS fixture with the RustFS container in one local VM, +not a measured workload or independent-domain qualification. The owned backend +and dedicated VM are stopped, data retained; default Docker context stays colima. + +Task record: .cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json +(SHA256 a072fb6fd8957b6afd5fdf4dffc88c312a0e34c2e1f5c4cfd81b4d2456ea9d0c). Effects executable SHA256: +85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7cce6. All 4680 tracked source hashes, stdout/stderr, +the original runner record and executable are retained and backed up on the +workspace volume. Commands with empty stdout reference their real nonempty +stderr as the generic output artifact; exact case outputs remain untouched. +The unchanged task evidence validator passes. Current RustFS evidence is under +.cellar/cell-groups-evidence/CG-08-effects-progress-783c270b/rustfs-4ea70e48/. +Earlier failing fixture runs and predecessor qualifications remain recorded. + +Recorded scope update: shared native fixture helpers under crates/cellar/tests/support/ +and the repeatable tools/multi-raft/ RustFS effects selector support the authorized +local verification. The original archive selector's twenty cases are unchanged. +Acceptance names, oracles, commands, limits and validator rules are unchanged. + +The next critical implementation is CG-07: replace canonical per-cell birth/host +startup and per-cell binding capacity with group placement and replicated +enrollment; carry logical cell versions/witnesses through SDK/ingress; convert +managed lifecycle and accounting to complete groups. Live transfers, topology, +hibernation, managed recovery, load/crash qualification and final legacy removal +remain required. This task gate does not qualify the public SDK/fleet cutover. + +## Earlier CG-08 implementation checkpoints The complete successor gate passes all twenty commands at 8a4c5ae0, including 164 execution cases with zero failures and two explicit RustFS ignores. The diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 484b38979..f4a0fef47 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1893,7 +1893,7 @@ { "id": "CG-07", "title": "Cut public SDK routing and managed fleet over to shared groups", - "status": "TODO", + "status": "IN_PROGRESS", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -2020,7 +2020,7 @@ { "id": "CG-08", "title": "Preserve per-cell effects and bounded idle execution caches", - "status": "IN_PROGRESS", + "status": "DONE", "priority": "P1", "effort": "L", "risk": "HIGH", @@ -2037,7 +2037,9 @@ ], "scope": [ "crates/cellar/durability/raft/", - "crates/cellar-ltx/src/" + "crates/cellar-ltx/src/", + "crates/cellar/tests/support/", + "tools/multi-raft/" ], "creates": [ "crates/cellar/durability/raft/cell_groups/effect_index.rs", @@ -2120,7 +2122,9 @@ "I12", "I13" ], - "evidence": [], + "evidence": [ + ".cellar/cell-groups-evidence/CG-08-gates-4ea70e48-01/record.json" + ], "stop_conditions": [ "A current source excerpt no longer matches: review and record the successor baseline before edits.", "An acknowledged effect, receipt, proof boundary or resource owner would be weakened to make a test pass.", From 05efe653b4db96a074110a6d7fd12ae6415d4a91 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 06:16:27 -0700 Subject: [PATCH 267/568] feat: capture shared commands before polling --- .../raft/cell_groups/controller/commands.rs | 97 +++++++++++++- .../raft/cell_groups/controller/requests.rs | 12 +- .../raft/cell_groups/directory/routing.rs | 19 ++- .../raft/cell_groups/dispatch/command/mod.rs | 121 +++++++++++++++--- .../raft/cell_groups/dispatch/mod.rs | 1 + .../cellar/durability/raft/cell_groups/mod.rs | 1 + .../tests/native_cell_groups_execution.rs | 5 + .../support/cell_group_controller_commands.rs | 100 ++++++++++++++- .../tests/support/cell_group_runtime.rs | 2 +- crates/cellar/tests/support/cell_group_sql.rs | 4 +- docs/design/multi-raft/cell-groups/README.md | 2 + docs/design/multi-raft/cell-groups/handoff.md | 12 +- .../design/multi-raft/cell-groups/progress.md | 49 +++++++ docs/design/multi-raft/cell-groups/tasks.json | 4 +- 14 files changed, 391 insertions(+), 38 deletions(-) diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs index cac797f40..7e98c185b 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/commands.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -5,20 +5,26 @@ use super::calls::Delivery; use super::*; use crate::raft::budget::{Budget, Permit}; use crate::raft::cell_groups::{ - CellCommandError, CellCommandInvocation, CellCommandResponse, CellDispatchError, CellSqlOptions, + CellCommandError, CellCommandInvocation, CellCommandRequest, CellCommandResponse, + CellDispatchError, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, PendingCellCommand, }; use cellar_logic::consensus::CommandId; use tokio::sync::{mpsc, oneshot, watch}; +enum Invocation { + Ready(Box), + Pending(Box), +} + pub(super) struct Request { - invocation: CellCommandInvocation, + invocation: Invocation, options: CellSqlOptions, reply: oneshot::Sender>, permit: Arc, } pub(super) struct Completed { - pub invocation: CellCommandInvocation, + pub invocation: Option, pub error: Option>, _permit: Arc, } @@ -93,6 +99,62 @@ impl CellGroupController { ) -> Result { admit(&self.handle.0, invocation, options) } + + /// Capture and queue a registered cell command before returning its response + /// waiter. Construction, routing, SQL and publication remain node-owned even + /// if that waiter is never polled or is dropped. The timeout starts here. + pub fn dispatch_command( + &self, + dispatcher: &CellGroupDispatcher, + cell: cellar_logic::cell_group::CellHistory, + request: CellCommandRequest, + routing: CellRoutingLimits, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let command = request.id; + if shared.closing.load(Ordering::Acquire) { + return Err(error(command, false, RuntimeError::Closed)); + } + let option_bytes = option_bytes(&mut options).map_err(|e| error(command, false, e))?; + let pending = dispatcher.prepare_command(cell, request, routing, None, None, false)?; + let (_, deadline, bytes) = pending.controller_admission(&shared.domain)?; + let bytes = bytes + .checked_add(option_bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Command(Box::new(Request { + invocation: Invocation::Pending(Box::new(pending)), + options, + reply, + permit: Arc::new(permit), + }))) + .map_err(|e| { + error( + command, + false, + match e { + mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }, + ) + })?; + Ok(CellCommandCall { + inner: calls::Call { + receiver, + identity: command, + deadline, + domain: shared.domain.clone(), + failure: |id, reason| error(id, true, reason), + }, + }) + } } pub(super) fn admit( @@ -127,7 +189,7 @@ pub(super) fn admit( shared .requests .try_send(requests::Request::Command(Box::new(Request { - invocation, + invocation: Invocation::Ready(Box::new(invocation)), options, reply, permit: Arc::new(permit), @@ -140,7 +202,10 @@ pub(super) fn admit( CellCommandAdmissionError { error: error(command, false, reason), invocation: match request { - requests::Request::Command(request) => Box::new(request.invocation), + requests::Request::Command(request) => match request.invocation { + Invocation::Ready(invocation) => invocation, + Invocation::Pending(_) => unreachable!("ready command admission envelope"), + }, _ => unreachable!("command admission envelope"), }, } @@ -164,11 +229,29 @@ pub(super) async fn drive( mut progress: watch::Receiver<()>, ) -> Completed { let Request { - mut invocation, + invocation, options, reply, permit, } = request; + let mut invocation = match invocation { + Invocation::Ready(invocation) => *invocation, + Invocation::Pending(pending) => match (*pending).finish().await { + Ok(invocation) => invocation, + Err(error) => { + let error = Arc::new(error); + let _ = reply.send(Delivery { + result: Err(error.clone()), + _permit: permit.clone(), + }); + return Completed { + invocation: None, + error: Some(error), + _permit: permit, + }; + } + }, + }; let result = invocation .run_owned(&runtime, &sql, options, &shared.groups, &mut progress) .await @@ -179,7 +262,7 @@ pub(super) async fn drive( _permit: permit.clone(), }); Completed { - invocation, + invocation: Some(invocation), error, _permit: permit, } diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index 5919024e4..48beb5ad8 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -21,17 +21,19 @@ pub(super) enum Completed { impl Completed { pub fn group(&self) -> Option { match self { - Self::Command(done) => done.invocation.unresolved_group(), + Self::Command(done) => done.invocation.as_ref().and_then(|i| i.unresolved_group()), Self::Enrollment(done) => done.invocation.unresolved_group(), } } pub fn failure(&self) -> Option { match self { - Self::Command(done) if done.invocation.unresolved() => Some(done.error.clone().map_or( - CellGroupControllerError::Invalid("unresolved successful command"), - CellGroupControllerError::Command, - )), + Self::Command(done) if done.invocation.as_ref().is_some_and(|i| i.unresolved()) => { + Some(done.error.clone().map_or( + CellGroupControllerError::Invalid("unresolved successful command"), + CellGroupControllerError::Command, + )) + } Self::Enrollment(done) if done.invocation.unresolved() => { Some(done.error.clone().map_or( CellGroupControllerError::Invalid("unresolved successful enrollment"), diff --git a/crates/cellar/durability/raft/cell_groups/directory/routing.rs b/crates/cellar/durability/raft/cell_groups/directory/routing.rs index 1d9ab7046..740ec0d31 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/routing.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/routing.rs @@ -34,6 +34,18 @@ pub struct CellRoutingLimits { pub max_attempts: u8, } +impl CellRoutingLimits { + pub(crate) fn deadline_at(self, started_at: u64) -> Option { + if self.timeout.as_millis() == 0 + || self.timeout > Duration::from_secs(60) + || !(1..=16).contains(&self.max_attempts) + { + return None; + } + started_at.checked_add(self.timeout.as_millis() as u64) + } +} + #[derive(Debug)] pub enum CellRoutingFailure { InvalidRequest, @@ -163,9 +175,6 @@ impl RangeDirectory { if request.cell.cluster != config.cluster || request.cell.namespace != config.namespace || request.fingerprint == [0; 32] - || limits.timeout.as_millis() == 0 - || limits.timeout > Duration::from_secs(60) - || !(1..=16).contains(&limits.max_attempts) || cached.as_ref().is_some_and(|cached| { cached.cell != request.cell || !Arc::ptr_eq(&cached.owner, &self.namespace.inner.owner) @@ -179,8 +188,8 @@ impl RangeDirectory { if started_at > now { return Err(error(CellRoutingFailure::InvalidRequest)); } - let deadline = started_at - .checked_add(limits.timeout.as_millis() as u64) + let deadline = limits + .deadline_at(started_at) .ok_or_else(|| error(CellRoutingFailure::InvalidRequest))?; if now >= deadline { return Err(error(CellRoutingFailure::Deadline)); diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs index 6d5f3359c..af1048095 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/command/mod.rs @@ -114,28 +114,48 @@ pub struct CellCommandInvocation { _admission: Permit, } +/// Synchronous request capture for the node queue. No SQL or routing I/O has +/// begun; dropping this preparation releases only its invocation reservation. +pub(crate) struct PendingCellCommand { + dispatcher: CellGroupDispatcher, + cell: CellHistory, + request: CellCommandRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option, + system: bool, + started: u64, + deadline: u64, + admission: Permit, +} + impl CellGroupDispatcher { - pub async fn command( + /// Capture invocation admission and the original deadline immediately. + /// Polling constructs routing state; it never restarts the timeout. + pub fn command( &self, cell: CellHistory, request: CellCommandRequest, limits: CellRoutingLimits, cached: Option, - ) -> Result { + ) -> impl std::future::Future> + + Send + + 'static { self.command_with_inputs(cell, request, limits, cached, None) - .await } - pub(crate) async fn command_with_inputs( + pub(crate) fn command_with_inputs( &self, cell: CellHistory, request: CellCommandRequest, limits: CellRoutingLimits, cached: Option, captured: Option, - ) -> Result { - self.command_compiled(cell, request, limits, cached, captured, false) - .await + ) -> impl std::future::Future> + + Send + + 'static { + let pending = self.prepare_command(cell, request, limits, cached, captured, false); + async move { pending?.finish().await } } pub(crate) async fn system_command( @@ -144,11 +164,12 @@ impl CellGroupDispatcher { request: CellCommandRequest, limits: CellRoutingLimits, ) -> Result { - self.command_compiled(cell, request, limits, None, None, true) + self.prepare_command(cell, request, limits, None, None, true)? + .finish() .await } - async fn command_compiled( + pub(crate) fn prepare_command( &self, cell: CellHistory, mut request: CellCommandRequest, @@ -156,13 +177,19 @@ impl CellGroupDispatcher { cached: Option, captured: Option, system: bool, - ) -> Result { + ) -> Result { let error = |cause| CellCommandError { command: request.id, ambiguous: false, cause: Box::new(cause), }; let started = self.domain.mono_ms().map_err(|_| error(closed()))?; + let deadline = limits.deadline_at(started).ok_or_else(|| { + error(CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::InvalidRequest, + ambiguous_command: None, + })) + })?; request.durability = request .durability .max(self.directory.namespace_config().minimum_durability); @@ -211,10 +238,74 @@ impl CellGroupDispatcher { .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; request.input = request.input.into_boxed_slice().into_vec(); request.contract.operation = request.contract.operation.into_boxed_str().into_string(); + Ok(PendingCellCommand { + dispatcher: self.clone(), + cell, + request, + limits, + cached, + captured, + system, + started, + deadline, + admission, + }) + } +} + +impl PendingCellCommand { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(CommandId, u64, usize), CellCommandError> { + let error = |cause| CellCommandError { + command: self.request.id, + ambiguous: false, + cause: Box::new(cause), + }; + if !self.dispatcher.uses_domain(domain) { + return Err(error(CellDispatchError::Invalid( + "command dispatcher domain", + ))); + } + if domain.mono_ms().map_err(|_| error(closed()))? >= self.deadline { + return Err(error(CellDispatchError::Runtime( + RuntimeError::DeadlineExceeded, + ))); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or_else(|| error(CellDispatchError::Runtime(RuntimeError::Overloaded)))?; + Ok((self.request.id, self.deadline, bytes)) + } + + pub(crate) async fn finish(self) -> Result { + let Self { + dispatcher, + cell, + request, + limits, + cached, + captured, + system, + started, + deadline: _, + admission, + } = self; + let error = |cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + }; let inputs = if let Some(inputs) = captured { inputs } else { - self.domain + dispatcher + .domain .run(async { use rand::RngCore as _; let mut random = [0; 32]; @@ -227,10 +318,10 @@ impl CellGroupDispatcher { .await .map_err(|_| error(closed()))? }; - let session = self + let session = dispatcher .directory .routing_session_in( - self.domain.clone(), + dispatcher.domain.clone(), CellRouteRequest { cell: cell.clone(), operation: CellRouteOperation::Command(request.id), @@ -241,9 +332,9 @@ impl CellGroupDispatcher { started, ) .map_err(|e| error(CellDispatchError::Routing(e)))? - .using_cache(self.routes.clone()); + .using_cache(dispatcher.routes.clone()); Ok(CellCommandInvocation { - dispatcher: self.clone(), + dispatcher, system, cell, request, diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 2bdb87f98..3cd3566fd 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -12,6 +12,7 @@ mod enrollment; pub use enrollment::{CellEnrollmentError, CellEnrollmentInvocation, CellEnrollmentRequest}; mod command; +pub(crate) use command::PendingCellCommand; pub use command::{ CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, CellCommandResponse, CellCommandSettlement, CellCommandSubmission, diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 8a2275515..a7c7e7b38 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -48,6 +48,7 @@ pub use controller::{ }; mod dispatch; pub use apply::{CellGroupApplier, CellGroupApplyImages}; +pub(crate) use dispatch::PendingCellCommand; pub use dispatch::{ CellCommandError, CellCommandInvocation, CellCommandPreparation, CellCommandRequest, CellCommandResponse, CellCommandSettlement, CellCommandSubmission, CellDispatchError, diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index d60d88b69..9d397f488 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -336,6 +336,11 @@ async fn checkpoint_stream_detects_corruption_and_closed_engine() -> Result<()> checkpoint_fixture::corruption().await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn command_construction_reserves_admission_and_original_deadline() -> Result<()> { + Box::pin(runtime_fixture::command_admission_starts_before_poll()).await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn controller_owned_commands_survive_sql_disconnect() -> Result<()> { runtime_fixture::owned_commands(0, 0).await diff --git a/crates/cellar/tests/support/cell_group_controller_commands.rs b/crates/cellar/tests/support/cell_group_controller_commands.rs index c5cb0b52d..150cf2be1 100644 --- a/crates/cellar/tests/support/cell_group_controller_commands.rs +++ b/crates/cellar/tests/support/cell_group_controller_commands.rs @@ -44,6 +44,90 @@ pub(super) async fn invocation( .await?) } +pub(crate) async fn command_admission_starts_before_poll() -> Result<()> { + for rooted in [false, true] { + let calls = Arc::new(AtomicUsize::new(0)); + let counted = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.command( + "Add", + [82; 32], + move |_, _| { + counted.fetch_add(1, Ordering::SeqCst); + Ok(Vec::new()) + }, + )?); + let (mut fixture, route, cell, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let pool = fixture.appliers[&(0, route.history)].0.clone(); + let dispatcher = CellGroupDispatcher::new( + &fixture.cluster.domain, + fixture.cluster.directory.clone(), + class.clone(), + super::remote_queries::keys(89)?, + CellDispatchLimits { + max_cached_routes: 1, + route_cache_bytes: 8192, + max_invocations: 1, + retained_bytes: 64 * 1024, + max_input_bytes: 1024, + }, + )?; + let request = CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"unpolled".to_vec(), + durability: Durability::Quorum, + retry_until_ms: u64::MAX, + }; + let limits = CellRoutingLimits { + timeout: Duration::from_millis(40), + max_attempts: 1, + }; + let pending = dispatcher.command(cell.clone(), request.clone(), limits, None); + assert_eq!( + dispatcher.usage().1, + 1, + "unpolled construction retains admission" + ); + let refused = dispatcher + .command(cell.clone(), request.clone(), limits, None) + .await + .err() + .context("unpolled construction exhausts admission")?; + assert!(!refused.ambiguous); + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + tokio::time::sleep(Duration::from_millis(60)).await; + let expired = pending + .await + .err() + .context("construction keeps original deadline")?; + assert!(!expired.ambiguous); + assert!(matches!( + expired.cause.as_ref(), + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + }) + )); + assert_eq!(dispatcher.usage().1, 0); + let cancelled = dispatcher.command(cell, request, limits, None); + assert_eq!(dispatcher.usage().1, 1); + drop(cancelled); + assert_eq!(dispatcher.usage().1, 0); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(pool.usage()?.active_cells, 0); + for controller in &controllers { + controller.close().await?; + } + drop(controllers); + fixture.close().await?; + } + Ok(()) +} + // phase=0 holds callback execution; phase=1 holds the committed manifest sync. // mode=0 disconnects, mode=1 expires the response, mode=2 cancels node close. pub(crate) async fn owned_commands(phase: u8, mode: u8) -> Result<()> { @@ -189,8 +273,20 @@ pub(crate) async fn owned_commands(phase: u8, mode: u8) -> Result<()> { } else { Duration::from_secs(15) }; - let call = controllers[0].command( - invocation(&dispatcher, &class, &a, timeout, retry_until).await?, + let call = controllers[0].dispatch_command( + &dispatcher, + a.clone(), + CellCommandRequest { + id: CommandId::new(ClientId::new([188; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: b"owned".to_vec(), + durability: Durability::Quorum, + retry_until_ms: retry_until, + }, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, fixture.cell_options(&a).await?, )?; tokio::time::timeout(Duration::from_secs(5), started).await??; diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index b680fb7aa..9c4f0506f 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -823,10 +823,10 @@ pub(super) use sql::{ s3_receipt_deadline, s3_receipt_failures, s3_receipt_minimums, s3_receipt_observations, s3_remote_deadlines, s3_remote_outputs, s3_remote_slow_delivery, }; +pub(super) use sql::{command_admission_starts_before_poll, owned_command_failure, owned_commands}; pub(super) use sql::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; -pub(super) use sql::{owned_command_failure, owned_commands}; pub(super) async fn cell_receipts(rooted: bool) -> Result<()> { sql::cell_receipts(rooted).await diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index ac4329483..473f62322 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -573,7 +573,9 @@ mod controller_commands; pub(crate) use controller::{ controller_apply_faults, controller_close, controller_owners, controller_replication, }; -pub(crate) use controller_commands::{owned_command_failure, owned_commands}; +pub(crate) use controller_commands::{ + command_admission_starts_before_poll, owned_command_failure, owned_commands, +}; pub(super) async fn commands_over_tcp(rooted: bool) -> Result<()> { Box::pin(command_dispatch::commands_with_transport(rooted, true)).await diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 76bfb7bae..8897fc3a5 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -12,6 +12,8 @@ Docker/RustFS all-voter-disks-deleted drills with permanent launch revocation, fresh-quorum activation, exact receipts, new writes and retained restart. CG-08 due effects and safe cache lifetimes now pass the complete native gate and same-source/executable RustFS case. CG-07 SDK/fleet cutover has started in the spare checkout and remains unqualified. See the latest [progress](progress.md) and [handoff](handoff.md). +CG-07 synchronous shared-command admission now passes its focused native and lint +checkpoint, preserving the original deadline and node-owned producer lifetime. The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index 6b99eeddf..e797fb039 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -14,7 +14,7 @@ node resources. A group spans voter nodes; it is not all cells on one node. ## Starting point Reuse codex/shared-cell-groups and [PR 160](https://github.com/crabbuild/cellar/pull/160). -The isolated effects checkout and PR checkout were reconciled at 4ea70e483e84ca88d77ca062278edb5ff2389646. +Keep the isolated effects checkout and PR checkout aligned at each published checkpoint. Preserve the user's untracked diagram and both excluded MR progress basenames. The current goal remains incomplete: the production SDK/fleet still uses its old host and no final combined release or load envelope is qualified. @@ -41,6 +41,16 @@ Executable SHA256: 85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7c All source hashes, real streams, original capture record and executable are retained locally and backed up. See [progress](progress.md) for scope and failures. +CG-07's synchronous command-admission checkpoint now passes all nine focused +commands: the exact new regression, full execution/effects/archive suites, +all-target strict Clippy, format/layout and dependency/docs checks. The record is +.cellar/cell-groups-evidence/CG-07-command-admission-fixed-02/record.json. +Its source patch, 4680 source hashes, three executables and actual streams are +retained and independently backed up. This is a shared-runtime prerequisite; +the public SDK/fleet cutover and all original CG-07 acceptance cases remain open. +Next implement the canonical namespace/class registry and shared host with logical +cell handles, then complete managed whole-group lifecycle and accounting. + CG-07 is IN_PROGRESS in the spare PR checkout. Its first canonical node service and startup changes now select CellGroupRuntime and shared transport. They are uncommitted and intentionally incomplete: the old host/SDK/fleet consumers still diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 40b75995a..1dcdac04f 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,55 @@ CG-07 is IN_PROGRESS; CG-09 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-07 synchronous shared-command admission + +CG-07 remains IN_PROGRESS. The shared controller now captures registered command +admission synchronously and queues construction before returning its response +waiter. Its existing node worker owns preparation, routing, SQL and publication; +an unpolled or dropped response cannot remove that work. Capture retains the +original absolute deadline before queueing. The dispatcher also reserves its +invocation budget when command construction is called, and an unpolled +construction cannot defer its timeout or bypass its configured capacity. + +Preparation failures before SQL have no unresolved producer to quarantine. +Once SQL enters, the existing exact reconciliation, sticky ambiguity, receipt, +physical join and group quarantine rules remain intact. No worker per cell or +second execution queue is introduced. All original limits and lifecycle assertions +remain unchanged. Private effect commands still use their sealed registration. + +All nine focused-checkpoint commands pass with 4680 source hashes and three +retained executables. The new exact regression passes both image modes. The full +execution target passes 165 cases, with zero failures and two ignored child-process +entry points controlled by their parent crash tests. Effects pass ten cases with +one explicit RustFS ignore; archive-controller passes sixteen with twenty explicit +RustFS ignores. All-target locked strict Clippy, format, layout, dependency and +documentation checks pass. These native suites do not establish a new real-backend, +public SDK/fleet, managed lifecycle, load or release qualification. + +Focused record: .cellar/cell-groups-evidence/CG-07-command-admission-fixed-02/record.json +(SHA256 5a44b6f76da74b72eddf45e34cd93c7ec1eb9cf12b67e41acc43fe49331920f6). The record binds the reviewed base revision plus its exact source +patch, all source hashes, actual streams and executables. Artifacts and an +independent complete backup are retained on the workspace volume. + +The valid baseline fails because unpolled construction retains zero invocation +slots instead of one. Its exact source and failing executable are preserved under +CG-07-command-admission-baseline-02. The earlier baseline-01 omitted a key-helper +argument and executed no case. Fixed-01 passed the first mode's constructor +assertions, then its fixture attempted a second SQL owner and correctly received +AlreadyExists. Fixed-02 inspects the existing controller pool with the same +zero-active-cell assertion. Those failures and their analyses remain retained; +no acceptance oracle, timeout, ownership check or resource cap was weakened. + +The critical next vertical slice is canonical shared host/SDK startup: bounded +namespace/class registrations, group range placement and provisioning, replicated +cell enrollment, logical cell handles/versions/witnesses and explicit consistency. +Then convert managed capacity, repair, drain, maintenance, stop/return/resume and +inventory to whole group replicas with separate cell/image occupancy. Preserve +CAS journals, permissions and disruption debt. The spare checkout's two incomplete +canonical node-service/startup edits remain owned work; never restore legacy +startup or add a legacy/shared flag. All five original public CG-07 cases and its +complete gate remain required before DONE. + ## CG-08 complete effects, cache and RustFS gate CG-08 is DONE at its bounded shared-runtime effects/cache boundary. Qualified diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index f4a0fef47..5d230a610 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1919,7 +1919,9 @@ "crates/cellar-axum/", "crates/cellar-admin/", "examples/", - "crates/cellar/sdk.rs" + "crates/cellar/sdk.rs", + "crates/cellar/tests/native_cell_groups_execution.rs", + "crates/cellar/tests/support/" ], "creates": [ "crates/cellar/tests/native_cell_groups_public.rs" From 5e7aadb6f1bd50deb0e32f5f4ddea62460e89db5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 07:28:37 -0700 Subject: [PATCH 268/568] feat: bind cells through shared namespace host --- crates/cellar/application/consensus/class.rs | 2 +- crates/cellar/application/consensus/mod.rs | 1 + .../cell_groups/controller/effects/mod.rs | 29 ++ .../raft/cell_groups/controller/mod.rs | 5 +- .../raft/cell_groups/controller/queries.rs | 111 +++++ .../raft/cell_groups/controller/requests.rs | 7 + .../raft/cell_groups/directory/mod.rs | 4 + .../raft/cell_groups/dispatch/enrollment.rs | 36 +- .../raft/cell_groups/dispatch/mod.rs | 49 ++ .../durability/raft/cell_groups/host/mod.rs | 308 ++++++++++++ .../raft/cell_groups/host/storage.rs | 79 +++ .../cellar/durability/raft/cell_groups/mod.rs | 7 +- .../cellar/tests/native_cell_groups_host.rs | 41 ++ .../support/cell_group_engine_fixture.rs | 3 + .../cellar/tests/support/cell_group_host.rs | 467 ++++++++++++++++++ .../support/cell_group_outbox_effects.rs | 1 + .../tests/support/cell_group_runtime.rs | 2 + .../tests/support/cell_group_s3_commands.rs | 1 + crates/cellar/tests/support/cell_group_sql.rs | 7 + docs/design/multi-raft/cell-groups/README.md | 2 + docs/design/multi-raft/cell-groups/handoff.md | 11 + .../design/multi-raft/cell-groups/progress.md | 63 +++ docs/design/multi-raft/cell-groups/tasks.json | 3 +- 23 files changed, 1230 insertions(+), 9 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/controller/queries.rs create mode 100644 crates/cellar/durability/raft/cell_groups/host/mod.rs create mode 100644 crates/cellar/durability/raft/cell_groups/host/storage.rs create mode 100644 crates/cellar/tests/native_cell_groups_host.rs create mode 100644 crates/cellar/tests/support/cell_group_host.rs diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index bf09bcf30..295de9339 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -15,7 +15,7 @@ use std::{ thread_local! { static CALLBACK_ACTIVE: std::cell::Cell = const { std::cell::Cell::new(false) }; } -pub(super) fn callback_active() -> bool { +pub(crate) fn callback_active() -> bool { CALLBACK_ACTIVE.with(std::cell::Cell::get) } diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index acca8dbc6..0944fd8f5 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -9,6 +9,7 @@ pub(crate) mod admission; mod archive_status; pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; +pub(crate) use class::callback_active; pub(crate) mod client; mod group_contracts; pub(crate) use group_contracts::cell_routing_key; diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs index c5077988e..52b0988c7 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs @@ -68,6 +68,35 @@ pub(super) struct Config { namespaces: BTreeMap<(ClusterId, NamespaceId), Arc>, } +impl CellGroupController { + pub(crate) fn require_host_effects( + &self, + dispatcher: &CellGroupDispatcher, + ) -> Result<(), crate::raft::cell_groups::CellDispatchError> { + use crate::raft::cell_groups::CellDispatchError; + let (_, class, _) = dispatcher.host_binding()?; + if !class.has_system_commands() { + return Ok(()); + } + let effects = self + .handle + .0 + .effects + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))?; + if effects + .as_ref() + .and_then(|config| config.namespaces.get(&dispatcher.namespace())) + .is_none_or(|entry| entry.schema != class.schema_digest()) + { + return Err(CellDispatchError::Execution(ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::UnsupportedCapability, + ))); + } + Ok(()) + } +} + impl CellGroupController { /// Install the node's bounded compiled effect registrations once. All local /// groups in these namespaces, including later registrations, participate. diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index a5a56f516..737179cf5 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -15,6 +15,8 @@ pub use archive::{ mod checkpoints; pub use checkpoints::CellGroupCheckpointPolicy; mod commands; +mod queries; +pub use queries::CellQueryCall; mod effects; mod enrollments; pub use effects::CellGroupEffectRegistration; @@ -51,7 +53,7 @@ pub struct CellGroupControllerLimits { pub max_parallel_apply: usize, /// One node timer backs off transient SQL admission, never one timer per cell. pub retry_interval_ms: u64, - /// Shared by commands, enrollments and registrations, including queued work, resolution + /// Shared by commands, queries, enrollments and registrations, including queued work, resolution /// and unconsumed responses. pub max_commands: usize, /// Invocation/input/options retention. The independent S3 output stage has @@ -106,6 +108,7 @@ pub struct CellGroupControllerUsage { /// Immediate queue/route refusals. The transport tracks later I/O failures. pub send_refusals: u64, pub closing: bool, + /// Ordinary node calls, including queries, enrollments and registrations. pub retained_commands: usize, pub command_bytes: usize, pub retained_preparations: usize, diff --git a/crates/cellar/durability/raft/cell_groups/controller/queries.rs b/crates/cellar/durability/raft/cell_groups/controller/queries.rs new file mode 100644 index 000000000..4c1f6a4ef --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/queries.rs @@ -0,0 +1,111 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Read callbacks share the node queue and SQL owner, independently of waiters. +use super::calls::Delivery; +use super::*; +use crate::raft::budget::Permit; +use crate::raft::cell_groups::{ + CellDispatchError, CellQueryInvocation, CellQueryResponse, CellSqlOptions, +}; +use cellar_logic::consensus::ProposalId; +use tokio::sync::oneshot; + +pub(super) struct Request { + invocation: CellQueryInvocation, + options: CellSqlOptions, + reply: oneshot::Sender>, + permit: Arc, +} + +/// Dropping or expiring the response does not cancel an admitted query callback. +/// Its physical connection and reservation remain owned until the callback exits. +pub struct CellQueryCall { + inner: calls::Call, +} + +impl CellQueryCall { + pub async fn response(self) -> Result> { + self.inner.response().await + } +} + +impl CellGroupController { + /// Queue one fresh registered query synchronously. It retains its original + /// consistency, token and deadline through discovery, read barrier and SQL. + pub fn query( + &self, + invocation: CellQueryInvocation, + mut options: CellSqlOptions, + ) -> Result { + let shared = &self.handle.0; + let (identity, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + if shared.closing.load(Ordering::Acquire) { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + let bytes = bytes + .checked_add(commands::option_bytes(&mut options).map_err(CellDispatchError::Runtime)?) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let permit = shared + .commands + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let (reply, receiver) = oneshot::channel(); + shared + .requests + .try_send(requests::Request::Query(Box::new(Request { + invocation, + options, + reply, + permit: Arc::new(permit), + }))) + .map_err(|error| { + CellDispatchError::Runtime(match error { + tokio::sync::mpsc::error::TrySendError::Closed(_) => RuntimeError::Closed, + tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded, + }) + })?; + Ok(CellQueryCall { + inner: calls::Call { + receiver, + identity, + deadline, + domain: shared.domain.clone(), + failure: |_, reason| CellDispatchError::Runtime(reason), + }, + }) + } +} + +pub(super) async fn drive( + request: Request, + runtime: CellGroupRuntime, + sql: CellGroupSqlPool, + shared: Arc, +) { + let Request { + mut invocation, + options, + reply, + permit, + } = request; + let result = async { + invocation.begin().await?; + let attempt = invocation + .attempt() + .ok_or(CellDispatchError::Runtime(RuntimeError::NotFound))?; + if !shared + .groups + .contains(&attempt.route().history) + .map_err(CellDispatchError::Runtime)? + { + return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); + } + invocation.execute(&runtime, &sql, options).await + } + .await + .map_err(Arc::new); + let _ = reply.send(Delivery { + result, + _permit: permit, + }); +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/requests.rs b/crates/cellar/durability/raft/cell_groups/controller/requests.rs index 48beb5ad8..8b2b583b3 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/requests.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/requests.rs @@ -10,12 +10,14 @@ pub(super) enum Request { Registration(Box), Provision(Box), Command(Box), + Query(Box), Enrollment(Box), } pub(super) enum Completed { Command(Box), Enrollment(Box), + Query, } impl Completed { @@ -23,6 +25,7 @@ impl Completed { match self { Self::Command(done) => done.invocation.as_ref().and_then(|i| i.unresolved_group()), Self::Enrollment(done) => done.invocation.unresolved_group(), + Self::Query => None, } } @@ -62,5 +65,9 @@ pub(super) async fn drive( Request::Enrollment(request) => Completed::Enrollment(Box::new( enrollments::drive(*request, runtime, sql, shared, progress).await, )), + Request::Query(request) => { + queries::drive(*request, runtime, sql, shared).await; + Completed::Query + } } } diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index c1e2a083d..1bb039eda 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -145,6 +145,10 @@ impl DirectoryPublication { } impl RangeDirectory { + pub(crate) fn cache_capacity_bytes(&self) -> usize { + self.limits.max_cache_bytes + } + pub(crate) fn namespace_config(&self) -> &crate::raft::catalog::NamespaceConfig { self.namespace.config() } diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs index df9e3c568..1377b502b 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/enrollment.rs @@ -63,31 +63,57 @@ pub struct CellEnrollmentInvocation { } impl CellGroupDispatcher { - pub async fn enroll( + pub fn enroll( &self, cell: CellHistory, request: CellEnrollmentRequest, limits: CellRoutingLimits, cached: Option, - ) -> Result { + ) -> impl std::future::Future> + + Send + + 'static { self.enroll_with_inputs(cell, request, limits, cached, None) - .await } - pub(crate) async fn enroll_with_inputs( + pub(crate) fn enroll_with_inputs( + &self, + cell: CellHistory, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + cached: Option, + captured: Option<(CommandInputs, ImageId)>, + ) -> impl std::future::Future> + + Send + + 'static { + let dispatcher = self.clone(); + let started = self.domain.mono_ms(); + async move { + let started = started.map_err(|_| CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(closed()), + })?; + dispatcher + .enroll_captured(cell, request, limits, cached, captured, started) + .await + } + } + + async fn enroll_captured( &self, cell: CellHistory, mut request: CellEnrollmentRequest, limits: CellRoutingLimits, cached: Option, captured: Option<(CommandInputs, ImageId)>, + started: u64, ) -> Result { let error = |cause| CellEnrollmentError { creation: request.creation, ambiguous: false, cause: Box::new(cause), }; - let started = self.domain.mono_ms().map_err(|_| error(closed()))?; + request.durability = request .durability .max(self.directory.namespace_config().minimum_durability); diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 3cd3566fd..5bf45504d 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -93,6 +93,32 @@ pub struct CellGroupDispatcher { } impl CellGroupDispatcher { + pub(crate) fn host_binding( + &self, + ) -> Result< + ( + crate::application::consensus::NamespaceConfig, + Arc, + usize, + ), + CellDispatchError, + > { + // Reserve the configured invocation/cache envelopes, rather than only + // their current usage, when admitting a node namespace registration. + let bytes = self + .limits + .retained_bytes + .checked_add(self.limits.route_cache_bytes) + .and_then(|n| n.checked_add(self.directory.cache_capacity_bytes())) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + Ok(( + self.directory.namespace_config().clone(), + self.class.clone(), + bytes, + )) + } + pub(crate) fn uses_domain(&self, domain: &DomainToken) -> bool { self.domain.same_owner(domain) } @@ -287,6 +313,29 @@ pub struct CellQueryInvocation { } impl CellQueryInvocation { + pub(crate) fn controller_admission( + &self, + domain: &DomainToken, + ) -> Result<(ProposalId, u64, usize), CellDispatchError> { + if !self.dispatcher.uses_domain(domain) || self.attempt.is_some() || self.dispatched { + return Err(CellDispatchError::Invalid( + "controller requires a fresh local query", + )); + } + let deadline = self.session.deadline_ms(); + if domain.mono_ms().map_err(|_| closed())? >= deadline { + return Err(CellDispatchError::Runtime(RuntimeError::DeadlineExceeded)); + } + let bytes = self + .request + .input + .len() + .checked_mul(2) + .and_then(|n| n.checked_add(8192)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + Ok((self.request.invocation, deadline, bytes)) + } + pub fn attempt(&self) -> Option<&CellRouteAttempt> { self.attempt.as_ref() } diff --git a/crates/cellar/durability/raft/cell_groups/host/mod.rs b/crates/cellar/durability/raft/cell_groups/host/mod.rs new file mode 100644 index 000000000..e155cdd73 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/host/mod.rs @@ -0,0 +1,308 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace registrations over one shared node controller. Cell handles are +//! disposable values; obtaining or cloning one neither enrolls nor caches a cell. +mod storage; +pub use storage::CellGroupNamespaceStorage; + +use super::*; +use crate::application::consensus::{OperationContract, RaftClass, RegistrationError}; +use crate::raft::{ + budget::{Budget, Permit}, + completion::RuntimeError, + transport::RaftTransport, +}; +use cellar_logic::{ + cell_group::CellHistory, + consensus::{ClusterId, NamespaceId}, +}; +use std::{ + collections::BTreeMap, + sync::{ + atomic::{AtomicBool, Ordering}, + Arc, + }, +}; + +pub struct CellGroupHostRegistration { + pub dispatcher: CellGroupDispatcher, + pub storage: CellGroupNamespaceStorage, +} + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupHostLimits { + pub max_namespaces: usize, + /// Includes each registration's configured invocation, route-cache and + /// directory-cache capacities, not only their current occupancy. + pub namespace_bytes: usize, +} + +struct Binding { + dispatcher: CellGroupDispatcher, + class: Arc, + storage: CellGroupNamespaceStorage, + _admission: Permit, +} + +struct Inner { + controller: CellGroupController, + domain: crate::asyncrt::DomainToken, + namespaces: BTreeMap<(ClusterId, NamespaceId), Arc>, + budget: Arc, + closed: AtomicBool, +} + +/// Bind compiled namespaces to an already initialized shared node controller. +/// Group placement and physical bootstrap remain node-startup responsibilities; +/// these registrations grant no voting, leader or serving authority. +#[derive(Clone)] +pub struct CellGroupHost { + inner: Arc, +} + +#[derive(Clone, Copy, Debug)] +pub struct CellGroupHostUsage { + pub namespaces: usize, + pub namespace_bytes: usize, + pub namespace_byte_limit: usize, + pub controller: CellGroupControllerUsage, +} + +impl CellGroupHost { + pub fn bind( + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + transport: &RaftTransport, + controller: CellGroupController, + registrations: Vec, + limits: CellGroupHostLimits, + ) -> Result { + let domain = runtime.log().domain(); + if !controller.belongs_to(runtime) + || !transport.uses_shared_runtime(runtime) + || !sql.belongs_to_engine(&runtime.log().owner()) + || !sql.uses_domain(&domain) + { + return Err(CellDispatchError::Invalid("shared host resource owner")); + } + if !(1..=4096).contains(&limits.max_namespaces) + || registrations.is_empty() + || registrations.len() > limits.max_namespaces + || limits.namespace_bytes < 8192 + { + return Err(CellDispatchError::Invalid("shared host namespace limits")); + } + let budget = Budget::new(limits.namespace_bytes, limits.max_namespaces); + let mut namespaces = BTreeMap::new(); + for registration in registrations { + let dispatcher = registration.dispatcher; + if !dispatcher.uses_domain(&domain) { + return Err(CellDispatchError::Invalid("shared host namespace domain")); + } + let (config, class, bytes) = dispatcher.host_binding()?; + controller.require_host_effects(&dispatcher)?; + registration.storage.validate_namespace(&config)?; + let bytes = bytes + .checked_add(registration.storage.bytes()) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let admission = budget + .acquire(bytes) + .ok_or(CellDispatchError::Runtime(RuntimeError::Overloaded))?; + let key = (config.cluster, config.namespace); + if namespaces.contains_key(&key) { + return Err(CellDispatchError::Invalid("duplicate host namespace")); + } + namespaces.insert( + key, + Arc::new(Binding { + dispatcher, + class, + storage: registration.storage, + _admission: admission, + }), + ); + } + Ok(Self { + inner: Arc::new(Inner { + controller, + domain, + namespaces, + budget, + closed: AtomicBool::new(false), + }), + }) + } + + /// Selecting a history allocates no permanent binding and opens no SQL. + /// Each invocation still obtains exact runtime/catalog authority. + pub fn cell(&self, history: CellHistory) -> Result { + self.inner.check_open()?; + let binding = self + .inner + .namespaces + .get(&(history.cluster, history.namespace)) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotFound))? + .clone(); + Ok(CellGroupCellHandle { + host: self.inner.clone(), + binding, + history, + }) + } + + pub fn is_closed(&self) -> bool { + self.inner.closed.load(Ordering::Acquire) + || self.inner.domain.mono_ms().is_err() + || self.inner.controller.usage().closing + } + + pub fn usage(&self) -> CellGroupHostUsage { + CellGroupHostUsage { + namespaces: self.inner.namespaces.len(), + namespace_bytes: self.inner.budget.usage().0, + namespace_byte_limit: self.inner.budget.capacity().0, + controller: self.inner.controller.usage(), + } + } + + /// Seal all handles, then join the node controller's physical owners. + /// Cancellation cannot reopen admission or cancel those physical joins. + pub async fn close(&self) -> Result<(), CellGroupControllerError> { + self.inner.closed.store(true, Ordering::Release); + self.inner.controller.close().await + } +} + +impl Inner { + fn check_open(&self) -> Result<(), CellDispatchError> { + if crate::application::consensus::callback_active() { + return Err(CellDispatchError::Execution( + crate::raft::execution::ExecutionError::Refused( + cellar_logic::consensus::CommandFailure::UnsupportedCapability, + ), + )); + } + if self.closed.load(Ordering::Acquire) + || self.domain.mono_ms().is_err() + || self.controller.usage().closing + { + return Err(CellDispatchError::Runtime(RuntimeError::Closed)); + } + Ok(()) + } +} + +/// A logical cell history, independent of its current physical group binding. +#[derive(Clone)] +pub struct CellGroupCellHandle { + host: Arc, + binding: Arc, + history: CellHistory, +} + +impl CellGroupCellHandle { + pub fn history(&self) -> &CellHistory { + &self.history + } + + pub fn contract(&self, operation: &str) -> Result { + if operation.len() > 128 { + return Err(RegistrationError::Invalid("operation bounds")); + } + self.binding + .class + .contract(operation) + .ok_or_else(|| RegistrationError::UnknownOperation(operation.into())) + } + + /// Queue before returning the response waiter. No group ID is retained by + /// this handle, and cancellation cannot release an admitted SQL producer. + pub fn command( + &self, + request: CellCommandRequest, + limits: CellRoutingLimits, + ) -> Result { + self.host.check_open().map_err(|cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + })?; + self.host.controller.dispatch_command( + &self.binding.dispatcher, + self.history.clone(), + request, + limits, + self.binding.storage.for_cell(&self.history), + ) + } + + /// Queue a read with its explicit consistency and original invocation deadline. + pub fn query( + &self, + request: CellQueryRequest, + limits: CellRoutingLimits, + ) -> Result { + self.host.check_open()?; + let invocation = + self.binding + .dispatcher + .query(self.history.clone(), request, limits, None)?; + self.host + .controller + .query(invocation, self.binding.storage.for_cell(&self.history)) + } + + pub fn enroll( + &self, + request: CellEnrollmentRequest, + limits: CellRoutingLimits, + ) -> impl std::future::Future>> + + Send + + 'static { + let opened = self.host.check_open().map_err(|cause| { + Arc::new(CellEnrollmentError { + creation: request.creation, + ambiguous: false, + cause: Box::new(cause), + }) + }); + let invocation = + self.binding + .dispatcher + .enroll(self.history.clone(), request, limits, None); + let host = self.host.clone(); + let options = self.binding.storage.for_cell(&self.history); + async move { + opened?; + let invocation = invocation.await.map_err(Arc::new)?; + let call = host + .controller + .enroll(invocation, options) + .map_err(|e| Arc::new(e.error))?; + call.response().await + } + } + + /// Absence remains ambiguous and never releases an original producer. + pub fn receipt( + &self, + request: CellReceiptRequest, + limits: CellRoutingLimits, + ) -> impl std::future::Future>> + + Send + + 'static { + let invocation = self.host.check_open().and_then(|()| { + self.binding + .dispatcher + .lookup_receipt(self.history.clone(), request, limits, None) + }); + let host = self.host.clone(); + let options = self.binding.storage.for_cell(&self.history); + async move { + host.controller + .lookup_receipt(invocation.map_err(Arc::new)?, options) + .await + .map_err(Arc::new) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/host/storage.rs b/crates/cellar/durability/raft/cell_groups/host/storage.rs new file mode 100644 index 000000000..f011f6535 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/host/storage.rs @@ -0,0 +1,79 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Immutable namespace resources; private paths bind logical history, not placement. +use super::*; +use crate::application::consensus::RaftImageStorage; +use std::path::{Component, Path}; + +/// Namespace-level SQL/image resources. Creating a handle performs no I/O and +/// opens no SQLite connection. Each invocation derives a private history path. +#[derive(Clone)] +pub struct CellGroupNamespaceStorage { + options: CellSqlOptions, + bytes: usize, +} + +impl CellGroupNamespaceStorage { + pub fn new(mut options: CellSqlOptions) -> Result { + options + .limits() + .validate() + .map_err(CellDispatchError::Execution)?; + let (path, vfs) = match &mut options { + CellSqlOptions::Materialized(o) => (&mut o.directory, &mut o.sqlite_vfs), + CellSqlOptions::Extents(o) => (&mut o.cache_directory, &mut o.sqlite_vfs), + }; + validate_path(path)?; + if vfs.as_ref().is_some_and(|name| name.len() > 128) { + return Err(CellDispatchError::Invalid("namespace SQL VFS bounds")); + } + let bytes = 8192 + 2 * (path.as_os_str().len() + vfs.as_ref().map_or(0, String::len)); + *path = std::mem::take(path).into_boxed_path().into_path_buf(); + *vfs = vfs.take().map(|v| v.into_boxed_str().into_string()); + Ok(Self { options, bytes }) + } + + pub(super) fn validate_namespace( + &self, + config: &crate::raft::catalog::NamespaceConfig, + ) -> Result<(), CellDispatchError> { + let (kind, schema) = match &self.options { + CellSqlOptions::Materialized(o) => { + (RaftImageStorage::Materialized, o.application_digest) + } + CellSqlOptions::Extents(o) => (RaftImageStorage::Extents, o.application_digest), + }; + if config.image_storage != kind || config.schema_digest != schema { + return Err(CellDispatchError::Invalid( + "namespace SQL image/schema binding", + )); + } + Ok(()) + } + + pub(super) fn bytes(&self) -> usize { + self.bytes + } + + pub(super) fn for_cell(&self, history: &CellHistory) -> CellSqlOptions { + let key = hex::encode(super::super::cell_image_history_digest(history)); + let mut options = self.options.clone(); + match &mut options { + CellSqlOptions::Materialized(o) => o.directory.push(&key), + CellSqlOptions::Extents(o) => o.cache_directory.push(&key), + } + options + } +} + +fn validate_path(path: &Path) -> Result<(), CellDispatchError> { + // Reserve a separator plus the full 64-byte history digest under the + // existing 4096-byte invocation option bound. Never use cell IDs as paths. + if !path.is_absolute() + || path.as_os_str().len() > 4000 + || path.components().any(|p| matches!(p, Component::ParentDir)) + { + return Err(CellDispatchError::Invalid("namespace SQL directory")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index a7c7e7b38..cb841cf6a 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -44,9 +44,10 @@ pub use controller::{ CellGroupArchiveStatus, CellGroupCheckpointPolicy, CellGroupController, CellGroupControllerError, CellGroupControllerLimits, CellGroupControllerUsage, CellGroupEffectRegistration, CellGroupProvisionCall, CellGroupProvisioning, - CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, + CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, CellQueryCall, }; mod dispatch; +mod host; pub use apply::{CellGroupApplier, CellGroupApplyImages}; pub(crate) use dispatch::PendingCellCommand; pub use dispatch::{ @@ -57,6 +58,10 @@ pub use dispatch::{ CellQueryResponse, CellReceiptInvocation, CellReceiptRequest, CellReceiptResponse, CellReceiptSelection, }; +pub use host::{ + CellGroupCellHandle, CellGroupHost, CellGroupHostLimits, CellGroupHostRegistration, + CellGroupHostUsage, CellGroupNamespaceStorage, +}; mod effect_index; mod execution_authority; diff --git a/crates/cellar/tests/native_cell_groups_host.rs b/crates/cellar/tests/native_cell_groups_host.rs new file mode 100644 index 000000000..6bdc4cf6a --- /dev/null +++ b/crates/cellar/tests/native_cell_groups_host.rs @@ -0,0 +1,41 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared namespace host boundary; canonical SDK/fleet cutover remains separate. +#![allow(clippy::disallowed_methods)] +// Common engine fixture helpers are selected by several independent targets. +#![allow(dead_code, unused_imports)] +#[path = "support/cell_group_archive_store.rs"] +mod archive_store; +#[path = "support/cell_group_checkpoint.rs"] +mod checkpoint_fixture; +#[path = "support/raft_storage_fault.rs"] +mod fault; +#[path = "support/raft_ltx_fixture.rs"] +mod images; +#[path = "support/cell_group_runtime.rs"] +mod runtime_fixture; +include!("support/cell_group_engine_fixture.rs"); + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_many_cells_share_one_physical_group() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_many_cells(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_query_cancellation_retains_physical_join() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_query_shutdown(rooted)).await?; + } + Ok(()) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn namespace_host_bounds_identity_and_enrollment_deadline() -> Result<()> { + for rooted in [false, true] { + Box::pin(runtime_fixture::host_registration_and_deadline(rooted)).await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_engine_fixture.rs b/crates/cellar/tests/support/cell_group_engine_fixture.rs index 55f3b6752..8b20b34bf 100644 --- a/crates/cellar/tests/support/cell_group_engine_fixture.rs +++ b/crates/cellar/tests/support/cell_group_engine_fixture.rs @@ -51,6 +51,7 @@ struct ClusterLogLimits { struct ClusterBirthOptions { group: GroupId, + image_storage: RaftImageStorage, minimum: Durability, log: ClusterLogLimits, } @@ -178,6 +179,7 @@ impl Cluster { store, ClusterBirthOptions { group: GroupId::new([3; 16])?, + image_storage: RaftImageStorage::Materialized, minimum, log: log_limits, }, @@ -210,6 +212,7 @@ impl Cluster { schema, ); config.minimum_durability = minimum; + config.image_storage = birth_options.image_storage; let namespace = domain.run(catalog.open_namespace(config)).await??; let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; let request = directory.prepare_group_birth(CellGroupBirthSpec { diff --git a/crates/cellar/tests/support/cell_group_host.rs b/crates/cellar/tests/support/cell_group_host.rs new file mode 100644 index 000000000..8b36856e2 --- /dev/null +++ b/crates/cellar/tests/support/cell_group_host.rs @@ -0,0 +1,467 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace host over three engines/TLS peers, with production enrollment only. +use super::*; +use std::sync::{ + atomic::{AtomicUsize, Ordering}, + Mutex, +}; + +async fn idle(host: &CellGroupHost) -> Result<()> { + controller::wait_for(async || Ok(host.usage().controller.retained_commands == 0)).await +} + +fn routing() -> CellRoutingLimits { + CellRoutingLimits { + timeout: Duration::from_secs(15), + max_attempts: 4, + } +} + +async fn setup( + rooted: bool, + class: &Arc, +) -> Result<(RuntimeCluster, Vec, GroupRoute)> { + let filesystem = Arc::new(DirectFileSystem); + let cluster = Box::pin(Cluster::with_store_birth( + RaftRuntimeOptions::default(), + tempfile::tempdir()?, + filesystem.clone(), + class.schema_digest(), + Arc::new(object_store::memory::InMemory::new()), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + minimum: Durability::Quorum, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + log: ClusterLogLimits::default(), + }, + )) + .await?; + let mut f = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 32)).await?; + if rooted { + for (node, voter) in f.cluster.voters.iter_mut().enumerate() { + voter.extents = Some(NodeExtentStore::open( + f.cluster._root.path().join(format!("rooted-{node}")), + filesystem.clone(), + Default::default(), + )?); + } + } + let route = f.cluster.birth.route().clone(); + let template = history(&route, "resource-template")?; + for node in 0..3 { + apply::install(&mut f, &template, node).await?; + } + let controllers = controller::start_with_limits( + &mut f, + CellGroupControllerLimits { + max_groups: 1, + max_parallel_apply: 1, + max_commands: 1, + ..controller::limits() + }, + ) + .await?; + f.runtimes[0].campaign(route.history).await?; + controller::wait_for(async || Ok(f.runtimes[0].status(route.history).await?.ready)).await?; + Ok((f, controllers, route)) +} + +async fn registration( + f: &RuntimeCluster, + class: &Arc, + node: usize, +) -> Result { + let template = history(f.cluster.birth.route(), "namespace-options")?; + let mut options = f.cell_options_at(&template, node).await?; + let base = f.cluster._root.path().join(format!("namespace-sql-{node}")); + match &mut options { + CellSqlOptions::Materialized(o) => o.directory = base, + CellSqlOptions::Extents(o) => o.cache_directory = base, + } + Ok(CellGroupHostRegistration { + dispatcher: remote_queries::dispatcher(f, class.clone(), remote_queries::keys(190)?)?, + storage: CellGroupNamespaceStorage::new(options)?, + }) +} + +async fn bind( + f: &RuntimeCluster, + controllers: &[CellGroupController], + class: &Arc, + node: usize, +) -> Result { + Ok(CellGroupHost::bind( + &f.runtimes[node], + &f.appliers[&(node, f.cluster.birth.route().history)].0, + &f.transports[node], + controllers[node].clone(), + vec![registration(f, class, node).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8 * 1024 * 1024, + }, + )?) +} + +fn command(class: &RaftClass, value: u8, retry_until_ms: u64) -> Result { + Ok(CellCommandRequest { + id: CommandId::new(ClientId::new([173; 16])?, 1)?, + contract: class.contract("Add").context("Add")?, + input: vec![value], + durability: Durability::Quorum, + retry_until_ms, + }) +} + +fn query(class: &RaftClass, consistency: CellQueryConsistency) -> Result { + Ok(CellQueryRequest { + invocation: ProposalId::new([174; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![], + consistency, + }) +} + +pub(crate) async fn many_cells(rooted: bool) -> Result<()> { + let initializers = Arc::new(AtomicUsize::new(0)); + let commands = Arc::new(AtomicUsize::new(0)); + let inits = initializers.clone(); + let calls = commands.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, tx| { + inits.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(0);")?; + Ok(()) + })? + .command("Add", [82; 32], move |ctx, input| { + calls.fetch_add(1, Ordering::SeqCst); + let n = *input.first().context("amount")?; + ctx.sql() + .execute("UPDATE value SET n=n+?", [i64::from(n)])?; + Ok(input.to_vec()) + })? + .query("Get", [84; 32], |ctx, _| { + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + })?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let retry_until = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let mut hosts = Vec::new(); + for node in 0..3 { + hosts.push(bind(&f, &controllers, &class, node).await?); + } + let bytes_before = hosts[0].usage().namespace_bytes; + let mut observations = Vec::new(); + let mut tokens = Vec::new(); + for i in 0..8u8 { + let cell = history(&route, &format!("host-{i}"))?; + let handle = hosts[0].cell(cell.clone())?; + let enrollment = handle + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([i + 1; 16])?, + durability: Durability::Quorum, + }, + routing(), + ) + .await?; + assert_eq!(enrollment.record().cell(), &cell); + drop(enrollment); + idle(&hosts[0]).await?; + let call = handle.command(command(&class, i + 1, retry_until)?, routing())?; + // The response remains unpolled while the node enters the callback. + controller::wait_for(async || Ok(commands.load(Ordering::SeqCst) == usize::from(i) + 1)) + .await?; + let response = call.response().await?; + assert_eq!(response.receipt().receipt.result(), &[i + 1]); + assert_eq!(response.receipt().version.history, cell); + assert_eq!(response.receipt().version.sequence.get(), 2); + observations.push(response.receipt().observation()); + tokens.push(response.token().clone()); + drop(response); + idle(&hosts[0]).await?; + } + assert_eq!(initializers.load(Ordering::SeqCst), 8); + assert_eq!(commands.load(Ordering::SeqCst), 8); + for (i, observation) in observations.iter().enumerate() { + for (node, host) in hosts.iter().enumerate() { + let handle = host.cell(observation.version.history.clone())?; + assert_eq!(handle.history(), &observation.version.history); + let response = handle + .query( + query(&class, CellQueryConsistency::AtLeast(tokens[i].clone()))?, + routing(), + )? + .response() + .await?; + assert_eq!(response.result(), (i as i64 + 1).to_be_bytes()); + assert_eq!(response.observation(), observation); + drop(response); + idle(host).await?; + assert_eq!(host.usage().namespaces, 1); + assert_eq!(host.usage().controller.groups, 1); + assert_eq!(f.cluster.voters[node].log.cell_groups()?.len(), 1); + } + } + let first = hosts[0].cell(observations[0].version.history.clone())?; + let retry = first + .command(command(&class, 1, retry_until)?, routing())? + .response() + .await?; + assert_eq!(retry.receipt().observation(), observations[0]); + drop(retry); + idle(&hosts[0]).await?; + assert_eq!(commands.load(Ordering::SeqCst), 8); + let receipt = first + .receipt( + CellReceiptRequest { + id: command(&class, 1, retry_until)?.id, + contract: class.contract("Add").context("Add")?, + input: vec![1], + durability: Durability::Quorum, + }, + routing(), + ) + .await?; + assert_eq!( + receipt.receipt().context("retained receipt")?.result(), + &[1] + ); + drop(receipt); + let wrong_token = first + .query( + query(&class, CellQueryConsistency::AtLeast(tokens[1].clone()))?, + routing(), + )? + .response() + .await; + assert!(wrong_token.is_err()); + let mut tenant = first.history().clone(); + tenant.namespace = NamespaceId::new([99; 16])?; + assert!(hosts[0].cell(tenant).is_err()); + let mut wrong = first.history().clone(); + wrong.incarnation = IncarnationId::new([99; 16])?; + let refused = hosts[0] + .cell(wrong)? + .query(query(&class, CellQueryConsistency::Stale)?, routing())? + .response() + .await; + assert!(refused.is_err()); + for _ in 0..100 { + drop(hosts[0].cell(first.history().clone())?); + } + assert_eq!(hosts[0].usage().namespace_bytes, bytes_before); + for host in &hosts { + host.close().await?; + assert!(host.is_closed()); + } + assert!(first + .command(command(&class, 1, retry_until)?, routing()) + .is_err()); + assert!(first + .query(query(&class, CellQueryConsistency::Stale)?, routing()) + .is_err()); + drop((first, hosts, controllers)); + f.close().await?; + Ok(()) +} + +pub(crate) async fn query_shutdown(rooted: bool) -> Result<()> { + let (entered, started) = tokio::sync::oneshot::channel(); + let entered = Mutex::new(Some(entered)); + let (release, held) = std::sync::mpsc::sync_channel(1); + let held = Mutex::new(held); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], |_, tx| { + tx.execute_batch("CREATE TABLE value(n); INSERT INTO value VALUES(7);")?; + Ok(()) + })? + .command("Add", [82; 32], |_, input| Ok(input.to_vec()))? + .query("Get", [84; 32], move |ctx, _| { + let _ = entered + .lock() + .unwrap() + .take() + .context("one held query")? + .send(()); + held.lock().unwrap().recv_timeout(Duration::from_secs(15))?; + Ok(ctx + .query_row("SELECT n FROM value", [], |r| r.get::<_, i64>(0))? + .to_be_bytes() + .to_vec()) + })?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let retry_until = f + .cluster + .domain + .run(async { cellar::asyncrt::wall_ms().max(0) as u64 + 60_000 }) + .await?; + let host = bind(&f, &controllers, &class, 0).await?; + let handle = host.cell(history(&route, "held-read")?)?; + drop( + handle + .enroll( + CellEnrollmentRequest { + creation: ProposalId::new([29; 16])?, + durability: Durability::Quorum, + }, + routing(), + ) + .await?, + ); + idle(&host).await?; + let pool = f.appliers[&(0, route.history)].0.clone(); + let call = handle.query( + query(&class, CellQueryConsistency::Linearizable)?, + routing(), + )?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + assert_eq!(host.usage().controller.retained_commands, 1); + drop(call); + let refused = handle + .command(command(&class, 1, retry_until)?, routing()) + .err() + .context("query retains global request admission")?; + assert!(matches!( + refused.cause.as_ref(), + CellDispatchError::Runtime(RuntimeError::Overloaded) + )); + assert!( + tokio::time::timeout(Duration::from_millis(30), host.close()) + .await + .is_err() + ); + assert!(host.is_closed()); + assert!(CellGroupRuntime::start( + f.cluster.voters[0].log.clone(), + RaftRuntimeOptions::default() + ) + .await + .is_err()); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), host.close()).await??; + assert_eq!(pool.usage()?.running_workers, 0); + assert_eq!(host.usage().controller.retained_commands, 0); + for c in &controllers { + c.close().await?; + } + drop((host, handle, controllers, pool)); + f.close().await?; + Ok(()) +} + +pub(crate) async fn registration_and_deadline(rooted: bool) -> Result<()> { + let initializers = Arc::new(AtomicUsize::new(0)); + let counted = initializers.clone(); + let class = Arc::new( + RaftClass::new("Counter", 1, [81; 32])? + .initializer([83; 32], move |_, tx| { + counted.fetch_add(1, Ordering::SeqCst); + tx.execute_batch("CREATE TABLE value(n);")?; + Ok(()) + })? + .command("Add", [82; 32], |_, input| Ok(input.to_vec()))?, + ); + let (f, controllers, route) = Box::pin(setup(rooted, &class)).await?; + let pool = &f.appliers[&(0, route.history)].0; + let duplicate = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[0].clone(), + vec![ + registration(&f, &class, 0).await?, + registration(&f, &class, 0).await?, + ], + CellGroupHostLimits { + max_namespaces: 2, + namespace_bytes: 16 * 1024 * 1024, + }, + ); + assert!(matches!( + duplicate, + Err(CellDispatchError::Invalid("duplicate host namespace")) + )); + let capacity = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[0].clone(), + vec![registration(&f, &class, 0).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8192, + }, + ); + assert!(matches!( + capacity, + Err(CellDispatchError::Runtime(RuntimeError::Overloaded)) + )); + let foreign = CellGroupHost::bind( + &f.runtimes[0], + pool, + &f.transports[0], + controllers[1].clone(), + vec![registration(&f, &class, 0).await?], + CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 8 * 1024 * 1024, + }, + ); + assert!(matches!( + foreign, + Err(CellDispatchError::Invalid("shared host resource owner")) + )); + assert!(!controllers[0].usage().closing); + let host = bind(&f, &controllers, &class, 0).await?; + let handle = host.cell(history(&route, "expired-enrollment")?)?; + let request = CellEnrollmentRequest { + creation: ProposalId::new([39; 16])?, + durability: Durability::Quorum, + }; + let pending = handle.enroll( + request, + CellRoutingLimits { + timeout: Duration::from_millis(40), + max_attempts: 1, + }, + ); + tokio::time::sleep(Duration::from_millis(60)).await; + let error = pending + .await + .err() + .context("original enrollment deadline")?; + assert!(!error.ambiguous); + assert!(matches!( + error.cause.as_ref(), + CellDispatchError::Routing(CellRoutingError { + reason: CellRoutingFailure::Deadline, + .. + }) + )); + assert_eq!(initializers.load(Ordering::SeqCst), 0); + assert_eq!(pool.usage()?.active_cells, 0); + drop(handle.enroll(request, routing()).await?); + assert_eq!(initializers.load(Ordering::SeqCst), 1); + for c in &controllers { + c.close().await?; + } + drop((host, handle, controllers)); + f.close().await?; + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_outbox_effects.rs b/crates/cellar/tests/support/cell_group_outbox_effects.rs index bb15c7721..86d1d93f8 100644 --- a/crates/cellar/tests/support/cell_group_outbox_effects.rs +++ b/crates/cellar/tests/support/cell_group_outbox_effects.rs @@ -320,6 +320,7 @@ async fn run_image( class.schema_digest(), store, ClusterBirthOptions { + image_storage: RaftImageStorage::Materialized, group, minimum, log: ClusterLogLimits::default(), diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 9c4f0506f..406a92f44 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -935,3 +935,5 @@ pub(super) async fn retained_log_admission(rooted: bool) -> Result<()> { #[allow(unused_imports)] pub(super) use sql::physical_log_reclamation; + +pub(super) use sql::{host_many_cells, host_query_shutdown, host_registration_and_deadline}; diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index 52d39d958..aca64d51a 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -177,6 +177,7 @@ impl Harness { class.schema_digest(), store, ClusterBirthOptions { + image_storage: RaftImageStorage::Materialized, group: GroupId::new([3; 16])?, minimum, log: ClusterLogLimits::default(), diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index 473f62322..b25fb2127 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -690,3 +690,10 @@ pub(super) use log_retention::retained_log_admission; #[allow(unused_imports)] pub(crate) use archive_controller::physical_log_reclamation; + +#[path = "cell_group_host.rs"] +mod namespace_host; +pub(crate) use namespace_host::{ + many_cells as host_many_cells, query_shutdown as host_query_shutdown, + registration_and_deadline as host_registration_and_deadline, +}; diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 8897fc3a5..4465f92c4 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -14,6 +14,8 @@ CG-08 due effects and safe cache lifetimes now pass the complete native gate and same-source/executable RustFS case. CG-07 SDK/fleet cutover has started in the spare checkout and remains unqualified. See the latest [progress](progress.md) and [handoff](handoff.md). CG-07 synchronous shared-command admission now passes its focused native and lint checkpoint, preserving the original deadline and node-owned producer lifetime. +The namespace-host component also passes its ten-command native/lint checkpoint, +including eight cells in one group and physical query shutdown in both image forms. The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 recovery and deployment/release qualification remain tracked in later tasks. See [implementation progress](progress.md) and the generated task index. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index e797fb039..cf7bbf4ec 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -41,6 +41,17 @@ Executable SHA256: 85ceb50508a519e2b1fd5198f47969185438f77c30d09e9de6ab884fd6a7c All source hashes, real streams, original capture record and executable are retained locally and backed up. See [progress](progress.md) for scope and failures. +The namespace-host component now passes its ten-command checkpoint. It has +bounded namespace/class registrations, history-derived SQL paths, logical cell +handles, controller-owned queries and original enrollment deadlines. All three +new host cases pass both image forms, including eight cells in one physical group +on every voter and held-query shutdown. Execution/archive/effects regressions and +strict checks pass. Evidence is under +.cellar/cell-groups-evidence/CG-07-namespace-host-01/record.json. +This binds already bootstrapped services; canonical startup, public SDK routing, +ingress and managed whole-group lifecycle remain unqualified. Move to those +consumers next, using this component rather than another per-cell binding owner. + CG-07's synchronous command-admission checkpoint now passes all nine focused commands: the exact new regression, full execution/effects/archive suites, all-target strict Clippy, format/layout and dependency/docs checks. The record is diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 1dcdac04f..56238635d 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -10,6 +10,69 @@ CG-07 is IN_PROGRESS; CG-09 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-07 namespace host component + +CG-07 remains IN_PROGRESS. CellGroupHost now binds compiled namespace/class +registrations to an exact existing shared controller, runtime, transport and SQL +engine. Its immutable registry has explicit namespace and byte limits; the byte +reservation includes configured invocation, route-cache and directory-cache +capacities. Cell handles contain logical CellHistory and share that registry. +Obtaining or cloning them neither enrolls cells nor adds permanent cell bindings. +The component does not create another consensus algorithm or a legacy mode. + +Namespace SQL resources validate schema, image backend, execution limits and +absolute private paths. Materialized directories and extent caches derive their +suffix from the full logical history digest, retaining that identity independently +of group placement. Only admitted workers open SQL. The host refuses reentrant +callback host calls and requires configured matching effect ownership for classes +with private background commands. These bindings grant no leader or bootstrap +authority; every operation still obtains runtime/catalog authority. + +Commands and queries enter the existing bounded controller queue synchronously. +Unpolled or disconnected query responses retain admission and physical callback +ownership; close seals all handles and joins the controller's SQL/transport/runtime +owners. Enrollment now captures its original deadline when construction is called, +so delayed polling cannot renew it. Receipt lookup preserves the exact command +binding and returns ambiguous absence without releasing a producer. + +All ten checkpoint commands pass. Three new native host cases pass both image +forms over real three-voter engines/TLS and independent SQLite state. They enroll +and mutate eight cells under a one-group limit, verify every replica's values and +logical observations, reject wrong tenant/history/token bindings, retain exact +receipts without callback replay and keep registry bytes constant after obtaining +more handles. A held query exercises dropped response admission, global overload, +cancelled close and engine-replacement refusal until the physical callback exits. +Duplicate namespace, foreign controller and metadata exhaustion refuse without +closing the healthy controller. Delayed enrollment expires before initialization. + +The full execution target passes 165 cases (two child-process fixture entries +ignored by the ordinary run), archive-controller passes sixteen (twenty explicit +RustFS ignores), and effects pass ten (one explicit RustFS ignore). Locked host +check, all-target strict Clippy, format/layout, dependency and documentation checks +pass. One SQL worker, one queued item and one pending cell per group remain in the +native fixture; its existing node/group byte ceilings are unchanged. The fixture +now explicitly declares its namespace image backend for the new host boundary. + +Record: .cellar/cell-groups-evidence/CG-07-namespace-host-01/record.json +(SHA256 61132e3338e6a729205708ae9cc9043156c51455fecf0ae16df6689939c135d5). It retains all 4685 source hashes, including +five new Rust files, the tracked source patch, copied changed/new files, actual +streams and four executables. An independent complete workspace-volume backup is +retained. No acceptance command, oracle, quota, timeout or validator was weakened. +This is native component qualification, without a new real RustFS, load, managed +lifecycle or complete public SDK/fleet claim. + +The host currently binds already bootstrapped services. The critical next step is +to replace canonical Node startup and SDK consumers: separate namespace/class +registration from group placement/provisioning and replicated cell enrollment; +carry logical versions/witnesses and explicit consistency through local and remote +routing, native/Axum ingress, CLI and examples. Managed capacity, repair, drain, +maintenance, stop/return/resume and inventory must operate on complete group +replicas while preserving CAS journals, permissions and physical debt. Keep the +spare checkout's two canonical startup edits and user diagrams. Do not restore +legacy startup, add a legacy/shared toggle or disguise groups as GroupScope.cell. +All five original native_cell_groups_public cases and the complete CG-07 gate +remain required. CG-09 through CG-16 and the full goal remain unfinished. + ## CG-07 synchronous shared-command admission CG-07 remains IN_PROGRESS. The shared controller now captures registered command diff --git a/docs/design/multi-raft/cell-groups/tasks.json b/docs/design/multi-raft/cell-groups/tasks.json index 5d230a610..201142b2e 100644 --- a/docs/design/multi-raft/cell-groups/tasks.json +++ b/docs/design/multi-raft/cell-groups/tasks.json @@ -1921,7 +1921,8 @@ "examples/", "crates/cellar/sdk.rs", "crates/cellar/tests/native_cell_groups_execution.rs", - "crates/cellar/tests/support/" + "crates/cellar/tests/support/", + "crates/cellar/tests/native_cell_groups_host.rs" ], "creates": [ "crates/cellar/tests/native_cell_groups_public.rs" From 51d136fbea6732cb93553c1a57cd45547ca9aa59 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 09:49:27 -0700 Subject: [PATCH 269/568] fix: release manifest pins on engine close Retained closed log handles kept image stores locked after shutdown. Release derived cache pins on the physical close owner before reporting completion; independent readers keep their own pins until they exit. --- crates/cellar/durability/raft/cache.rs | 25 +++++++++++++++++++ crates/cellar/durability/raft/storage.rs | 24 ++++++++++++++++-- .../support/cell_group_sql_apply_recovery.rs | 9 ++++++- 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/crates/cellar/durability/raft/cache.rs b/crates/cellar/durability/raft/cache.rs index 83e048e79..8c82d8871 100644 --- a/crates/cellar/durability/raft/cache.rs +++ b/crates/cellar/durability/raft/cache.rs @@ -224,6 +224,24 @@ impl std::fmt::Debug for GroupLog { } impl GroupLog { + /// Only the physical engine close owner may detach derived cache pins. + /// Independent readers retain their own pins until their physical work exits. + fn release_closed_pins(&self) -> Result<(), LogError> { + if !self.closed.load(Ordering::Acquire) { + return Err(LogError::Invalid("live manifest-pin release".into())); + } + match self.cache.write() { + Ok(mut cache) => { + cache.manifest_pins.take(); + Ok(()) + } + Err(error) => { + error.into_inner().manifest_pins.take(); + Err(LogError::Corrupt("closed group cache poisoned".into())) + } + } + } + pub(crate) fn log_retention( &self, ) -> Option<&Arc> { @@ -304,6 +322,13 @@ pub(crate) enum StoredGroup { } impl StoredGroup { + pub(crate) fn release_closed_pins(&self) -> Result<(), LogError> { + match self { + Self::Cell(group) => group.release_closed_pins(), + Self::Cells(group) => group.release_closed_pins(), + } + } + pub(crate) fn as_cell(&self) -> Result<&GroupLog, LogError> { match self { Self::Cell(group) => Ok(group), diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 42debf9b9..601fed081 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -159,6 +159,22 @@ pub struct SyncReceipt { } type Catalog = Arc>>; + +fn release_closed_pins(groups: &Catalog) -> Result<(), LogError> { + let (groups, mut result) = match groups.read() { + Ok(groups) => (groups, Ok(())), + Err(error) => ( + error.into_inner(), + Err(LogError::Corrupt("closed log catalog poisoned".into())), + ), + }; + for group in groups.values() { + let released = group.release_closed_pins(); + result = result.and(released); + } + result +} + type Reply = oneshot::Sender>; #[derive(Clone)] @@ -470,11 +486,12 @@ impl ReplicaLog { // if the open waiter or public handle is dropped. let _worker = domain .blocking(move || { - struct ClosedOnDrop(Arc, Arc); + struct ClosedOnDrop(Arc, Arc, Catalog); impl Drop for ClosedOnDrop { fn drop(&mut self) { self.0.store(true, Ordering::Release); + let _ = release_closed_pins(&self.2); // Worker/resources unwind before this outer guard. A // panic may close admission early, but cannot fake join. let mut terminal = self @@ -489,7 +506,7 @@ impl ReplicaLog { self.1.changed.notify_waiters(); } } - let _guard = ClosedOnDrop(closed.clone(), terminal.clone()); + let guard = ClosedOnDrop(closed.clone(), terminal.clone(), groups.clone()); let result = Worker::open( options, files, @@ -522,6 +539,9 @@ impl ReplicaLog { // Engine drop joins its metrics thread; directory locks are owned // by this same worker and cannot outlive its completion. drop(worker); + // Closed public metadata handles may outlive this worker. They + // must not keep image/page-directory locks through cache pins. + let result = result.and(release_closed_pins(&guard.2)); *terminal .result .lock() diff --git a/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs index 46b57de8a..64948ae41 100644 --- a/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs +++ b/crates/cellar/tests/support/cell_group_sql_apply_recovery.rs @@ -66,9 +66,14 @@ pub(crate) async fn recover(rooted: bool) -> Result<()> { let directory = fixture.cluster.directory.clone(); let birth = fixture.cluster.birth.clone(); let disk = std::mem::replace(&mut fixture.cluster._root, tempfile::tempdir()?); + let closed_log = fixture.cluster.voters[0].log.clone(); + let closed_group = closed_log.cell_group(&route.history)?; fixture.close().await?; + assert!(closed_log.cell_groups().is_err()); + assert!(closed_group.group_manifest().is_err()); - // All prior engine, page, image, worker and runtime owners are gone. + // Retained closed metadata must not retain physical image/page owners. + // Keep both handles alive throughout fresh recovery on these same paths. let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let base = disk.path().join("node-1"); let log = domain @@ -95,6 +100,8 @@ pub(crate) async fn recover(rooted: bool) -> Result<()> { base.join("images") }; let config = open_images(&image_directory, rooted)?; + assert!(closed_log.cell_groups().is_err()); + assert!(closed_group.group_manifest().is_err()); let pages = ManifestPageStore::open( &base.join("pages"), Arc::new(DirectFileSystem), From 110b7a4ebc594f144bdd98f3a48978ad39936078 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 10:01:25 -0700 Subject: [PATCH 270/568] docs: review closed-log pin release baseline --- docs/design/multi-raft/cell-groups/baseline.json | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 5581c61a0..274b9485f 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "9a0bad9124f0089055a0ba63b2952e5ef5e77853", - "runtime_revision": "9a0bad9124f0089055a0ba63b2952e5ef5e77853", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability.", + "source_revision": "51d136fbea6732cb93553c1a57cd45547ca9aa59", + "runtime_revision": "51d136fbea6732cb93553c1a57cd45547ca9aa59", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability. Reviewed successor 51d136fbea6732cb93553c1a57cd45547ca9aa59: physical engine close now releases only derived manifest pins after the closed fence and engine join, before publishing its terminal result. Unwind cleanup releases these pins too; external readers retain their own owners. Both retained-closed-handle recovery regressions fail at 5e7aadb6 with image/extent directory-lock refusal, then pass with this fix. Twelve checks pass, including full execution/archive/effects/namespace-host/controller-archive native suites and strict all-target Clippy; no new RustFS qualification is claimed. CachedGroup fields/excerpt are unchanged and move by 25 lines. Canonical SDK/fleet cutover remains uncompiled local work and CG-07 remains IN_PROGRESS.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -19,8 +19,8 @@ }, "manifest": { "path": "crates/cellar/durability/raft/cache.rs", - "sha256": "944ba3c1f181d1af42976d1e19e2c7c0f0d2e2981b121b145acfad9e4fd4e461", - "line": 466, + "sha256": "e97ce93f081b16d24d4b473e3e8549b3050983dddc3144c27a8bae27b156b3ff", + "line": 491, "excerpt": "pub(crate) struct CachedGroup {\n pub archive_required: bool,\n pub hard_state: HardState,\n pub conf_state: ConfState,\n pub membership: GroupMembership,\n pub snapshot: Snapshot,\n pub entries: BTreeMap,\n pub applied: DurableApply,\n pub group_manifest: Option,\n /// Derived from the immutable mapping; never supplied by a write/snapshot.\n pub recovery_genesis: Option>,\n pub manifest_pins: Option,\n pub checkpoint_manifest: Option,\n /// Local receipt; never imported from another node's checkpoint.\n pub group_seed: Option,\n}\n" }, "apply": { From 66d5ea6aa4ff5f2b2753f17c2cf4f7bf3c706136 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 11:21:32 -0700 Subject: [PATCH 271/568] feat: page physical group directory inventory --- crates/cellar/durability/raft/catalog.rs | 3 +- .../raft/cell_groups/directory/inventory.rs | 86 +++++++++++++++++++ .../raft/cell_groups/directory/mod.rs | 2 + .../cellar/durability/raft/cell_groups/mod.rs | 3 +- .../tests/native_cell_groups_catalog.rs | 5 ++ .../tests/support/cell_group_directory.rs | 56 ++++++++++++ 6 files changed, 153 insertions(+), 2 deletions(-) create mode 100644 crates/cellar/durability/raft/cell_groups/directory/inventory.rs diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 56c8016be..214162845 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -28,7 +28,8 @@ pub(crate) use range_directory::RecoverySeedPermit; pub(crate) use range_directory::RouteCache; pub use range_directory::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, diff --git a/crates/cellar/durability/raft/cell_groups/directory/inventory.rs b/crates/cellar/durability/raft/cell_groups/directory/inventory.rs new file mode 100644 index 000000000..b5511de2c --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/inventory.rs @@ -0,0 +1,86 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded physical-range enumeration at one immutable directory observation. +//! A complete traversal is metadata inventory, never serving or release proof. +use super::*; + +/// Continuation for one exact observed tree. It cannot be rebased onto a later +/// root; restart an independent inventory after a topology change. +#[derive(Clone, Debug)] +pub struct CellGroupDirectoryCursor { + root: [u8; 32], + after: CellBound, +} + +pub struct CellGroupDirectoryPage { + pub routes: Vec, + pub next: Option, +} + +impl RangeDirectory { + /// Enumerate physical groups without scanning cells or listing S3 objects. + /// Each call retains at most `limit` routes and one tree path. The cursor + /// binds the immutable view, including its namespace and topology revision. + pub async fn group_page( + &self, + view: &DirectoryView, + after: Option<&CellGroupDirectoryCursor>, + limit: usize, + ) -> Result { + self.check_view(view)?; + if !(1..=128).contains(&limit) { + return Err(CatalogError::Limit); + } + if after.is_some_and(|cursor| cursor.root != view.digest) { + return Err(CatalogError::Conflict("directory inventory cursor")); + } + self.bounded(async { + let mut lower = after.map(|cursor| cursor.after.clone()); + let mut routes = Vec::with_capacity(limit); + for _ in 0..limit { + let Some(route) = self.successor(view, lower.as_ref()).await? else { + return Ok(CellGroupDirectoryPage { routes, next: None }); + }; + lower = Some(route.range.lower().clone()); + routes.push(route); + } + let next = if self.successor(view, lower.as_ref()).await?.is_some() { + Some(CellGroupDirectoryCursor { + root: view.digest, + after: lower.ok_or(CatalogError::Corrupt("empty directory inventory page"))?, + }) + } else { + None + }; + Ok(CellGroupDirectoryPage { routes, next }) + }) + .await + } + + async fn successor( + &self, + view: &DirectoryView, + after: Option<&CellBound>, + ) -> Result, CatalogError> { + let mut next = Some(view.head.tree.clone()); + let mut candidate = None; + for _ in 0..32 { + let Some(reference) = next.take() else { + return Ok(candidate); + }; + let page = self.page(&reference).await?; + let route = page.route.decode(self)?; + if after.is_none_or(|lower| route.range.lower() > lower) { + candidate = Some(route); + next = page.left; + } else { + next = page.right; + } + } + if next.is_none() { + Ok(candidate) + } else { + Err(CatalogError::Limit) + } + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index 1bb039eda..b7f59da82 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -21,6 +21,8 @@ mod archive; pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding}; mod birth; mod bootstrap; +mod inventory; +pub use inventory::{CellGroupDirectoryCursor, CellGroupDirectoryPage}; mod recovery; pub(crate) use recovery::RecoverySeedPermit; pub use recovery::{ diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index cb841cf6a..75addf63f 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -116,7 +116,8 @@ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsag pub use crate::raft::catalog::{ CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, + CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index 35528e81d..e8e4df190 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -15,6 +15,11 @@ mod directory; #[allow(dead_code)] mod fault; +#[tokio::test] +async fn directory_inventory_pages_bind_the_exact_topology_cut() -> anyhow::Result<()> { + directory::inventory_pages().await +} + #[tokio::test] async fn range_directory_has_single_owner() -> anyhow::Result<()> { directory::single_owner().await diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 98b757113..8d9578bc3 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -161,6 +161,62 @@ fn split(parent: &GroupRoute, id: u64) -> Vec { ] } +pub async fn inventory_pages() -> Result<()> { + let f = Fixture::new(DirectoryLimits { + max_cached_pages: 2, + max_cache_bytes: 2 * 64 * 1024, + ..DirectoryLimits::default() + }) + .await?; + let d = &f.directory; + let mut initial = ranges(7); + f.reserve(&mut initial).await?; + let init = d.prepare_initialization(operation(991), &initial)?; + f.run(d.publish(&init)).await?; + let view = f.run(d.observe()).await?; + for limit in [0, 129] { + assert!(f.run(d.group_page(&view, None, limit)).await.is_err()); + } + let page = f.run(d.group_page(&view, None, 2)).await?; + assert_eq!(page.routes, initial[..2]); + let first_cursor = page.next.context("seven groups require a second page")?; + + let mut replacement = split(&initial[0], 992); + f.reserve(&mut replacement).await?; + let mutation = d.prepare_replacement(&view, operation(993), &initial[..1], &replacement)?; + f.run(d.publish(&mutation)).await?; + let current = f.run(d.observe()).await?; + assert_eq!(current.groups(), 8); + assert!(f + .run(d.group_page(¤t, Some(&first_cursor), 2)) + .await + .is_err()); + let reopened = f.reopen().await?; + assert!(f.run(reopened.group_page(&view, None, 2)).await.is_err()); + + let mut routes = page.routes; + let mut cursor = Some(first_cursor); + while let Some(after) = cursor { + let page = f.run(d.group_page(&view, Some(&after), 2)).await?; + assert!(!page.routes.is_empty() && page.routes.len() <= 2); + routes.extend(page.routes); + cursor = page.next; + } + assert_eq!(routes, initial); + let page = f.run(d.group_page(¤t, None, 128)).await?; + let expected: Vec<_> = replacement + .into_iter() + .chain(initial.into_iter().skip(1)) + .collect(); + assert_eq!(page.routes, expected); + assert!(page.next.is_none()); + let usage = d.cache_usage()?; + assert!(usage.0 <= f.limits.max_cached_pages && usage.1 <= f.limits.max_cache_bytes); + assert_eq!(f.store.listing_calls.load(Ordering::Acquire), 0); + f.domain.shutdown().await; + Ok(()) +} + pub async fn single_owner() -> Result<()> { let f = Fixture::new(DirectoryLimits::default()).await?; let d = &f.directory; From c54307c5ca115b9b228e112a91d00424444bfda5 Mon Sep 17 00:00:00 2001 From: forhappy Date: Mon, 2 Nov 2026 11:33:12 -0700 Subject: [PATCH 272/568] docs: review group inventory source baseline --- docs/design/multi-raft/cell-groups/baseline.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/design/multi-raft/cell-groups/baseline.json b/docs/design/multi-raft/cell-groups/baseline.json index 274b9485f..3d601ea4d 100644 --- a/docs/design/multi-raft/cell-groups/baseline.json +++ b/docs/design/multi-raft/cell-groups/baseline.json @@ -1,8 +1,8 @@ { "format": 1, - "source_revision": "51d136fbea6732cb93553c1a57cd45547ca9aa59", - "runtime_revision": "51d136fbea6732cb93553c1a57cd45547ca9aa59", - "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability. Reviewed successor 51d136fbea6732cb93553c1a57cd45547ca9aa59: physical engine close now releases only derived manifest pins after the closed fence and engine join, before publishing its terminal result. Unwind cleanup releases these pins too; external readers retain their own owners. Both retained-closed-handle recovery regressions fail at 5e7aadb6 with image/extent directory-lock refusal, then pass with this fix. Twelve checks pass, including full execution/archive/effects/namespace-host/controller-archive native suites and strict all-target Clippy; no new RustFS qualification is claimed. CachedGroup fields/excerpt are unchanged and move by 25 lines. Canonical SDK/fleet cutover remains uncompiled local work and CG-07 remains IN_PROGRESS.", + "source_revision": "66d5ea6aa4ff5f2b2753f17c2cf4f7bf3c706136", + "runtime_revision": "66d5ea6aa4ff5f2b2753f17c2cf4f7bf3c706136", + "review_note": "2026-10-29: reviewed physical cell-group consensus ownership at 0951440e3748da20573a6de26b7a9e97298fb385. GroupLog clones now share an exclusive consensus lease. GroupWrite retains the lease through queued physical sync/publication even when its waiter is cancelled. The CachedGroup anchor itself is unchanged and moves by three lines. The new CellGroupOwner consumes the existing verified complete root and revalidated catalog permit, persists Ready/LightReady before dependent effects, and advances complete roots only for exact contiguous entry/owner/store completions. Bounded slow-apply backpressure and cancellation/restart fixtures use real Raft Engine and SQLite/LTX. No other source anchor changed. This is the low-level shared owner, not shared SQL scheduling, SDK dispatch, snapshot/configuration/topology integration or completed CG-04 qualification. Successor 51e30eabde1ca76bf0b23963dd06d0e8130f8ee7 boxes large phase/effect payloads after all-target Clippy review and rejects exhausted peer-entry indexes before RawNode mutation. No anchored excerpt or hash changed. Reviewed successor 865f253da0a14d730b62f77b17cefc6d1497890b: the old runtime shares its unchanged terminal completion/wait behavior via a crate-private Terminal constructor with the physical cell-group node runtime; the runtime/ticks excerpts are unchanged apart from line offsets. New group runtime multiplexes owners, bounds node/group/event/completion admission, batches persistence across histories and joins physical writes before releasing runtime ownership. Reserved completion advances existing physical phases under saturated output admission. This is not per-cell SQL scheduling, receipt settlement, SDK serving or full CG-04 qualification. Reviewed successor aec4029e77124848dc6b155c098f8eec9ccdacc4: integrated main documentation consolidation and relocated the package without changing Rust behavior. Attachment now reserves ordinary request admission before catalog revalidation/RawNode construction and retains it through dispatch. The held-vote admission regression exercises the public runtime with one request slot. All anchored excerpts and hashes remain unchanged. Reviewed successor f3d48bdc86186077cff96f66597a65c2c4bd5301: added actual-leader cell execution authority with revocation on role loss, failure and owner drop. A synchronous cell SQL producer reuses the existing receipt transaction, captures either physical image kind without synthetic GroupScope, encodes cell-addressed entries and settles exact committed entry/image receipts. Shared capture and receipt helpers preserve the existing executor path. The SQL pool, registered dispatcher, coherent reads and full CG-04 gate remain unfinished. Anchored files and excerpts remain unchanged. Successor f807b2f70e01e3cd8c741557889e1965fd14ec49 removes two Clippy-rejected fixture wrappers only; runtime and assertions are unchanged. Reviewed successor 51dff0e5abec7c2355a80e84d1b2e3ae2504e3b1: the existing SQL pool now accepts bounded cell-history envelopes on the same fixed blocking workers and fairness queue. The legacy Cell anchor is unchanged except line position; its file gains shared request dispatch and reserved progress capacity. New cell admission retains one physical producer across waiter cancellation, bounds node/group capture and retained payloads, and shrinks transient reservations after capture. Engine identity and an exclusive SQL lease fence foreign/duplicate pools; cleanup owns joins and retained outputs keep the engine admission lease. Extent workspaces are created and dropped only by the admitted worker. Existing production host still uses its old dispatcher until the registered SDK/effect identity cutover; no full CG-04 or release qualification is claimed. Reviewed successor 8a0cf640ae70f3280f02e281f81b55a8823205fd: added receipt reconciliation on the existing SQL workers using cell pins from durable current-term applied roots, including followers. Missing receipts retire a producer only behind a strictly newer applied term; matching receipts may be read from later images. Engine/route/history/proposal checks, stale-pin rechecks, progress admission and retained response budgets remain enforced. Missing producers now report uncertainty. Native tests exercise real leadership transfer, a held new-term apply, later same-cell mutation, absence and receipt retry in both image modes. All anchored files and excerpts remain unchanged. Snapshot/compaction, registered retry dispatch and the complete CG-04 gate remain pending. Reviewed successor 91c3ecfc020e0d42738e4a6395045eb8a44c51f9: split immutable receipt identity inspection from new-client capacity admission. The ordinary command path preserves the same queries, sequence/floor validation and client/receipt ceilings; reconciliation skips only admission capacity. Both physical image fixtures now change the selected cell with another client before reconciling absence at a full identity ceiling, and qualify replicated expiry and conflicting requests through real leader transfer. Unit cases preserve allocation bounds, sequence gaps and floors. Anchored files and excerpts are unchanged. This does not complete CG-04 or production dispatch/compaction qualification. Reviewed successor 6e09bbc72a0ba04d67b9a9d9aa68a0991bf76fbb: actual-leader enrollment authority and current-term applied enrollment state now drive initialization on the existing bounded SQL workers. Name admission excludes competing incarnations; captured initial images are replicated as enrollment entries, and retries retain original creation identity after mutations. Followers resolve exact committed creation records or retire absent/conflicting old producers only behind a newer applied term. Prepared SQL includes its term and the RawNode owner checks it atomically in propose_at_term. Existing root workspace preparation is shared between commands and initialization. Native materialized/extent cases cover cancellation, logical-name contention, three independent cells in one group, creation retry after mutation, follower recovery, and stale prepared proposals after leadership returns. Existing anchored files/excerpts are unchanged. Registered SDK creation/command dispatch, catalog routing, configured group capacity policy, coherent reads, compaction and preview/cache lifecycle remain required; this is not full CG-04 qualification. Reviewed successor 82ae6877c1c727a3a7c68e463e99beefe71fef46: cargo format changes only line wrapping in two authority checks; all anchored source hashes and excerpts remain unchanged. Reviewed successor 446dd9057f10c8221122d0d3a9a21254adeb448f: shared owners now bound pending proposal acceptance by logical cell, exact encoded payload and a separate shared node budget. Creation admission counts applied cells plus accepted pending enrollments before append, mutations validate their selected base, and applied or old-term retries require existing receipt/creation reconciliation. Role loss and terminal failure release only acceptance metadata, not physical SQL ownership. Native fixtures cover capacity, duplicate/conflicting retries, inherited enrollment after all-voter disk restart, and budget sharing across actual runtime groups. The runtime fixture retains overloaded peer frames under existing deadlines and removes an extra async constructor layer. Anchored source files and excerpts are unchanged; registered dispatch, catalog sessions, coherent reads and full CG-04 remain unfinished. Reviewed successor 0cf89c403f4a500d18116499b8fc0f219195cd54: prepared SQL submissions bind engine/group/term and share immutable acceptance or irreversible pre-append rejection across clones. Only actual owner capacity refusal produces an exact preparation proof; generic queue errors preserve uncertainty. SQL cleanup validates that proof on the owning worker, closes the producer before releasing logical name admission and retains output/queued submission budgets. Prepared submission closes raw byte proposal paths on that owner. Ambiguous append observation fails the owner closed. Actual three-voter fixtures exercise initialization and mutation in both image modes, wrong proofs, lost responses, delayed retries and retained-resource accounting. Anchored files and excerpts remain unchanged. Registered dispatch, coherent reads, compaction/retry integration and full CG-04 remain pending. Reviewed successor 88c4717d857fc52fd1d4f04e6d3fac871f8185a1: shared runtime cell reads now correlate fresh ReadIndex to one local invocation and wait for contiguous apply before selecting the exact cell image. Stale/AtLeast use committed local cell versions; histories and equal-sequence images must match. Separate node/group read reservations extend to pinned logical image bytes, without retaining ordinary peer admission while waiting. Immutable SQL queries consume those pins on existing workers, retain output admission, enforce deadlines through delivery and progress hooks, contain panics and join cancelled work. Native cases cover both image modes, slow unrelated apply, future AtLeast versions, older coherent pins, minority refusal, role revocation, capacity and cancellation. Anchored source excerpts and hashes are unchanged. Registered SDK query contexts, catalog routing and production dispatch remain pending; no full CG-04/cutover qualification. Reviewed successor 3da4481e09c3767face30469ffc855d302a7c142: shared read pins use an explicit max_read_pin_bytes limit independent of transport request capacity. The unchanged default preserves 64 MiB retention; validation uses wide arithmetic to bound configured pins by the native image envelope and pending count. A unit case covers full-size native images without raising transport limits, minimum metadata capacity and the upper envelope. The native read fixture uses the execution-domain select macro. Existing source anchors and excerpts remain unchanged. The source-bound read checkpoint passes all 204 selected Rust cases and twelve commands, including all-target Clippy; CG-04 remains incomplete. Reviewed successor ed84df5daa0ae65fdfb09e9556f21f3e1ea12ec2: authenticated cell manifest pages now require a full selected-version CommitWitness at marker 1. Shape validation bounds history/positions and original enrollment proposals; ordinary prepare_update binds current group/position and the owner additionally matches the exact applied proposal. Unchanged and retry roots preserve witnesses. Query and receipt observations retain selected-version witnesses without reading the log entry cache. Native read/token, forged apply and restart/image-collection cases plus marker-one negative codec cases cover the new contract. Existing anchored source excerpts and hashes remain unchanged. Real compaction, registered SDK/catalog serving and full CG-04 qualification remain pending. Reviewed successor 2085cc9697b462401bce1f49d4eda875bf01a7fd: committed cell-group apply now uses a boxed reserved-progress envelope on the existing SQL workers. The legacy Cell excerpt is unchanged; its owning file gains group-keyed progress dispatch. The adapter checks the exact cached committed entry and preceding manifest, transfers the startup page owner, prepares materialized/extent images and bounded effect summaries, and retains independent admission in the completion ticket through physical publication. Materialized retry pins verify exact prepared state without resetting retirement; cold files are independently verified and synced. The old adopt path preserves its retention behavior. Native tests exercise both modes, all voters, duplicate/stale/foreign/forged work, saturated business admission and held publication sync. New image-I/O cancellation/crash cuts, registered SDK/catalog dispatch, membership integration and the complete CG-04 gate remain pending. Successor c9100ec774d66f30c14eae009f9a9c76d2bc0c33: heap-allocate the new physical apply branch in the shared test pump so unrelated nested recovery futures retain their previous inline stack envelope. Both leadership-reconciliation cases pass with the normal test stack; no assertions or stack limits change. Anchored source files and excerpts are unchanged. Reviewed successor 1ec2eeaab2beaa8d8e82c834cbb5dbf692b5e7d1: CellGroupApplier::recover verifies only the engine-selected complete manifest and exact physical image closure, transferring the same page owner into subsequent apply. Materialized names retain their existing derivation; extent roots verify logical history, schema and image geometry. Recovery grants no serving authority. Native tests cover both image modes at real image/manifest directory sync: cancelled preparation and close retain ownership, sync failure preserves the selected root, joined cold reopen ignores unpublished preparation and replays its committed entry without fresh quorum, and missing selected closure refuses. All existing source anchors and excerpts remain unchanged. Whole-process kill/power-loss, registered SDK/catalog dispatch, cache lifecycle, configuration integration and full CG-04 qualification remain pending. Reviewed successor b28078afbd243c18ac7c671a481a3e369c086ab2: registered commands now run through CellGroupSqlPool on the existing fixed workers with bounded owned input and pre-receipt operation/schema/digest/authentication validation. CommandContext and effect targets carry logical CellHistory; system clients, canonical signed effect histories and lane/signature domains exclude physical groups. The class manifest domain changes at marker 1; old group-bearing payloads and old signature domains refuse. The temporary old host projects its actual binding, and its effect-claim path is the only changed source anchor; the anchored Scheduler excerpt is unchanged. The shared path uses authority-owned cell history directly. Native cases cover two cells and duplicate/typed negative requests in both image modes; registered query context, catalog/SDK routing, shared effect scheduling, cache/configuration integration and full CG-04/CG-08 qualification remain pending. Reviewed successor 7a79511d4b8606d11177778a3fe8adf71a8d901a: registered queries use the existing coherent SQL owner with exact CellHistory and CellVersionObservation. Contract/schema checks precede callback execution; bounded operation/input ownership, read-only SQL, original deadline, cancellation and joined shutdown remain enforced. Old query adapters explicitly retain GroupCut during integration; no cell sequence or witness is inferred from a group position. CG-16 requires deleting GroupCut and invoke_group_query. All anchored file hashes and excerpts remain unchanged. Catalog/SDK routing and full CG-04 qualification remain unfinished. Reviewed successor 8723db2df7436478cbc5efc9147d65d3b7651247: CellGroupDispatcher binds a registered class to its namespace and captured execution domain. Query invocations freeze bounded input/consistency, retain shared admission and use existing catalog sessions with a deadline captured before request validation/hashing. Each attempt is consumed before async work; exact route, engine, domain and authenticated cell token checks precede registered SQL execution. Router cancellation preserves existing SQL ownership. The catalog session public constructor delegates to the same captured-start implementation; old constructor behavior and ambiguity rules remain. All anchored hashes and excerpts remain unchanged. This integrates local routed queries, not command/enrollment dispatch, remote transport, public SDK replacement or full CG-04. Reviewed successor f93c690cc70346ae7bdf1cc61aceca0103eb8768: catalog-bound command invocations preserve original command identity, domain-captured inputs, route fingerprint, deadline and producer ownership across cancelled preparation and routing retries. Submission uses existing acceptance/seal proofs; exact receipt settlement and newer applied-term absence retain original authority checks. Response receipts retain their existing boxed admission owner. Request and namespace S3 minima refuse until shared archive proof integration. Reconciliation authority exposes its immutable route for exact binding checks. The leadership fixture now checks the actual caller cell and uses the production applier where installed, preserving the held new-term barrier assertion. All 24 anchored hashes and excerpts remain unchanged. Node-controller ownership after ingress cancellation, enrollment routing, public SDK replacement and full CG-04 qualification remain unfinished. Successor 7d5e01da39408308945ab80dff74d286dadaf14c: box the typed command error cause after Clippy identified a 136-byte result error; command identity, conservative ambiguity and typed error assertions are unchanged. Source anchors remain unchanged. Successor 1f014f42264c73b1d5e3cd387f65ffed45140ecb: use the mandated cellar::asyncrt::select macro in the held command cancellation fixture; assertions and runtime behavior are unchanged. All source anchors remain unchanged. Reviewed successor eb99a2427e8661dc228ea7a1d3d1f56a5e0937aa: physical-group routes now share the existing RaftTransport pools, control/data pipelines, admission and shutdown owners. Shared startup holds an exclusive physical-runtime claim without constructing the legacy runtime. Typed marker-1 peer histories and a new protocol digest refuse old untagged frames. Physical schema, exact replica process, committed member/disk and active configuration checks guard admission and in-flight delivery. Physical routes participate in node-view refresh, fleet pool revocation, pruning, combined limits and query/inspection endpoint conflict checks. Complete-group snapshots refuse. Tests exercise real TLS multi-group replication and registered command cancellation/receipt settlement in both image modes; fixtures still pump runtime events. All 24 anchored file hashes and excerpts remain unchanged. Controller ownership, routed enrollment, public SDK/query-RPC cutover and full CG-04 remain incomplete; CG-16 must remove temporary legacy transport/frame arms. Reviewed successor 61aa775206fdc842848952ffb042a7e1ca0d310e: added a bounded fixed-roster node controller over the existing shared runtime, physical transport and SQL apply pool. Exact event/runtime, transport/runtime and engine/applier identities prevent cross-engine wiring even in one execution domain. One node cleanup task drives transport and ordered per-group apply, retaining ticket admission and backing off transient SQL capacity without fixture pumping. Physical SQL joins precede runtime lease release; fatal apply errors fail closed at node scope. Six native cases cover two groups on shared TLS pools and one SQL worker, both image kinds, exact owner refusal, manifest sync failure, cancelled shutdown and final-handle drop. Caller-independent command ownership, bounded post-response reconciliation, dynamic roster and per-group failure isolation remain incomplete; no full CG-04 or public host cutover is claimed. All 24 anchored source files, excerpts and line positions remain unchanged. Successor ee684c3394ffccbfa84bfc5d94876de281f68b0b reuses the existing physical fault fixture rather than loading it twice, resolving all-target Clippy duplicate_mod. Production code and test assertions are unchanged. The preceding 61aa7752 gate passed 289 Rust cases before the Clippy failure; the affected shared-execution suite and repository checks are repeated at this successor. All 24 source anchors remain unchanged. Reviewed successor 8c615ac91bc3c13e6c25da4f4ab584a73e609207: the shared node controller owns fresh commands under byte/item budgets independently from response waiters. Original deadlines govern responses; exact receipts or newer applied-term absence govern submitted resolution. Never-attempted preparations can be sealed across delayed clones; any attempted error remains uncertain and cannot turn into cancellation or capacity proof. Completed SQL stage errors are distinguished from lost worker replies; unresolved errors retain command ownership through physical SQL/runtime joins. Unconsumed responses remain charged. Seven native cases each run both image modes over real in-process TLS and three submission unit cases cover the new invariants. Routed enrollment, remote command/query routing, public SDK binding, dynamic roster, per-group failure isolation and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Reviewed successor bd3069d7723a8d4ec118bf465fb65633f762e3dd: registered initializers bind an explicit digest into class voting schema and run on the existing shared SQL workers with logical cell history and captured transaction inputs. The old host refuses this registration rather than silently ignoring it. Catalog-routed enrollment transfers immutable creation identity into the shared controller, using the same queue, byte/item budget, response deadline implementation and physical shutdown owner as commands. Exact creation records survive later mutations; conflicts never reinitialize. Never-attempted late preparations can be sealed; submitted uncertainty remains owned until exact resolution or joined shutdown. Initialization admission/received failures are distinguished from lost physical worker replies. Eight native cases cover registration refusal and both-image lifecycle/fault boundaries; a unit case checks schema identity and old-host refusal. Remote RPC, public SDK binding, lazy birth/dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored source files, excerpts and line positions remain unchanged. Successor f77d0d939347ead7129b39e72f76fcff11b99635 removes only a redundant unit expression in the transient enrollment resolution match arm, resolving all-target Clippy unused_unit. The bd3069d7 gate passed all 90 selected Rust cases before this Clippy failure. Runtime behavior and assertions are unchanged; the entire gate is repeated at the successor. All source anchors remain unchanged. Reviewed successor 3c297a06c9863fafb91e2cd8f428fae26915f2b4: shared catalog query attempts now use the existing bounded TLS query lane with authenticated physical group and logical cell identities. Destination catalog epoch/birth and runtime read authority are revalidated; returned tokens authenticate actual cell observations. Shared and legacy queries use one origin-grant/revocation guard and connection exchange. Host-owned weak gateway registration checks the exact runtime/SQL engine and domain, and returned replies retain node/peer admission. Original deadlines/attempt budgets survive explicit retries and catalog redirects. Cancellation or expiry cancels read waiters while physical SQL remains owned until joined cleanup. Three native cases cover both image modes, including stale catalog refusal; three new wire cases cover identity, token and payload boundaries. This checkpoint supports attached-group peers, not nonmember shared discovery or public SDK cutover. Shared mutation RPC/write authorization, dynamic roster, per-group failure isolation, configuration/cache lifecycle and full CG-04 remain incomplete. All 24 anchored files, excerpts and line positions remain unchanged. Successor 201ffaf0465546485f20d26d6e10c23d4867cb09 separates the public gateway registration admission owner from data retained by accepted requests. Dropping the final host registration now refuses new reads even while a prior callback retains Gateway data. Both native lifecycle cases exercise that exact boundary in both image modes. The 3c297a06 gate was interrupted during native testing after Clippy, eight wire cases and 63 transport cases passed; the complete gate is repeated at this successor. All source anchors remain unchanged. Successor da79a69604d58be1c7d9663ebb69c3f6ded173f6 moves query sequence allocation from concurrent admission into the existing single peer sender after dequeueing, preserving the receiver replay check without spurious refusal when callers enqueue out of allocation order. The attached-peer TLS fixture now starts concurrent queries for two cells repeatedly in both image modes. This also repairs the temporary old-host query path through the shared exchange helper. The 201ffaf0 candidate gate was stopped during Clippy; no completed gate is claimed for that source. All source anchors remain unchanged, and the complete gate runs at this final successor. Reviewed c25f0d97: explicit bounded nonmember physical query discovery shares node routes, peer pools, process refresh and endpoint consistency without local replicas or voting authority. Real TLS cases cover both image modes, scoped/process-bound grants, original-budget leader retry, held-SQL revocation and stale replies after route replacement. All existing source-anchor hashes and excerpts remain unchanged; the full checkpoint gate is recorded separately. Reviewed c41d8565: bounded dispatcher range hints are shared across query, command and enrollment without per-cell cache entries. Explicit entry/byte reservations, LRU eviction and verified revision/digest invalidation refuse superseded in-flight refill. RPC destinations retain fresh catalog lookup until replicated transfer seals and admission invalidation are integrated. Three catalog cases measure actual read calls; a native case exercises local reads, commands and enrollment during catalog unavailability in both image modes while remote admission refuses. Only catalog.rs anchors change: re-exports add the private cache and public usage value; anchored excerpt text is unchanged and shifts one line. All other source-anchor hashes and positions remain unchanged. CG-04 remains incomplete; no destination-cache, public SDK, RustFS or load qualification is claimed. Successor 06652e36 changes only the stale-refill test timeout to use its execution domain, resolving all-target Clippy disallowed_methods. The initial c41d8565 full gate stopped at Clippy; no full qualification is claimed for that candidate. Production code and test assertions are unchanged, and the complete gate is repeated at the successor. All source anchors remain unchanged. Reviewed 5fd44437: remote shared commands use the bounded application RPC lane with a mandatory MAC-bound command purpose and independently bounded namespace/process write grants. Read grants and voter membership do not authorize commands. Grant tombstones, endpoint consistency, rooted process refresh, fleet revocation and pool lifecycle retain their existing ownership rules. Host registration binds the exact controller/runtime; captured caller inputs, command identity and budgets reach fresh catalog/leader checks before controller-owned SQL. Admitted commands survive disconnect/revocation; committed duplicate results and signed cell observations retain reply admission. Typed role/conflict/expiry/sequence failures remain explicit. Native both-image cases use a nonmember fourth TLS endpoint; wire/grant cases cover authenticated boundaries and independent bounded permissions. Follower execution-authority requests now report NotLeader instead of NotReady; leader readiness and stale captured-authority checks remain unchanged, and the direct fixture asserts that more precise refusal. All 24 source-anchor hashes, excerpts and lines remain unchanged. This is not remote enrollment, standalone receipt lookup, public SDK cutover, full CG-04, RustFS or load qualification. Reviewed remote enrollment: the existing bounded command exchange, exact controller registration and explicit write grants now carry initialization with a distinct mandatory authenticated purpose. Captured creation identity/inputs/preview, fresh receiver catalog binding, leader authority and response-independent ownership are preserved. Replies bind the exact committed creation record, including its original image after later mutations. Four native TLS cases cover both image modes and three wire cases cover purpose, bounds and identity. Existing source anchors are unchanged; this is a partial CG-04 checkpoint, with standalone receipt lookup and public cutover still incomplete. Reviewed successor fbc15053f8dcfd3b46078abad3cbbae7b68da8d6: added standalone local/remote receipt inspection on catalog-bound quorum images. Command contracts, payload digests and authenticated identity validate before immutable SQL lookup. Explicit command grants and a distinct authenticated purpose protect the existing bounded exchange. Exact results or ambiguous absence carry signed cell observations; lookup never stages callbacks, mutates receipt state or retires unresolved producers. Initial schema absence is distinguished from partial/lost schema and sequence gaps. Read reservations remain bounded with pending producers, cancelled work retains physical ownership, and controller lookup uses a weak SQL handle so closed controllers cannot retain the engine claim. Existing command validation now accepts the common request directly with unchanged validation semantics. All anchored files and excerpts are unchanged. Dynamic controller roster/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; no full CG-04, shared RustFS or load qualification is claimed. Reviewed successor 4326421a94c4f9a257bf6ceb8e0f7e518eacf3de: controller registration now admits pending and active groups under one capacity bound and shared item/byte budget, including empty startup. Selected roots bind the exact engine/history/schema and applier page owner. The applier enters the event roster before attachment is polled; catalog revalidation precedes active membership and actual serving still requires current-term quorum. An owned transport registration refuses adopting existing routes and removes only its exact route on failure. Dropped waiters do not abandon attachment; shutdown joins entered registration before closing transport, SQL and runtime. Failed group slots release before error response; unexpected competing owners or runtime-wide closure remain fail-closed. The existing command/enrollment/receipt paths consult live roster membership under short locks. Native tests cover both image modes, real catalog stalls/refusals and TLS progress, pending/active capacity, cancelled close, and all-voter joined engine/page-store restart into empty controllers with exact receipt replay. All source anchors remain unchanged. Request-driven provisioning/lazy birth, per-group failure isolation, configuration lifecycle and public SDK cutover remain incomplete; this is not full CG-04 or shared RustFS/load qualification. Reviewed successor 9fff8cbaf2a71e0ee8f1c327cc63418e8755bc89: bounded controller group quarantine now revokes runtime execution through reserved control capacity, keeps failed group capacity and retained physical producers/apply tickets, and permits unrelated groups to apply. Late in-flight persistence cannot restore revoked serving. Shared-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Original group failures and active/quarantined counts are observable. Native cases qualify real manifest and engine sync faults separately in both image modes. Source anchors remain unchanged. Request-driven provisioning, configuration lifecycle, public SDK cutover and later CG tasks remain unfinished; no shared RustFS/load qualification is claimed. Qualification successor 41121a7d08f283db6a83982ebfcba2feced036ca: the manifest-failure regression waits for runtime GroupQuarantined before asserting revoked authority, rather than assuming diagnostic publication and runtime revocation occur in one scheduler turn. Reviewed successor 9d8cb9b25e77a97268233f78bd98a0497e5afa42: explicit controller-owned local provisioning materializes already selected initial-member ranges, binds configured peers and permanent genesis claims, verifies complete selected roots on owning blocking workers and attaches through existing runtime/transport ownership. Exact directory/group requests coalesce; independent waiters and retained preparation items/bytes/concurrency are bounded. Attachment refusals reuse recovered page owners. Close joins admitted preparation before physical service shutdown. Missing members refuse before genesis; transport registration refusals preserve healthy groups. Five native cases cover both image modes, real engines/files/TLS, application receipts, pre/post-recovery failures, cancelled close, and joined all-voter engine/page-store reopen with missing-selected-page refusal. All source anchors remain unchanged. Automatic dispatch, follower orchestration, managed fleet/configuration lifecycle and public hard cutover remain unfinished; no full CG-04, independent-process/power-loss or shared RustFS/load qualification is claimed. Qualification successor 69e41e865c267dff1dec5bce6f2b3d0966be09db: reuse the existing catalog fault adapter to resolve all-target duplicate_mod without suppressing its lint. Validate absolute bounded extent cache paths and normalize retained allocation before configuration is admitted, with no-I/O invalid configuration coverage. Clarify independent provisioning admission in controller documentation. The first candidate gate stopped at Clippy; only the successor complete gate can qualify this checkpoint. Source anchors remain unchanged. Scoped CG-04 acceptance completed at 69e41e865c267dff1dec5bce6f2b3d0966be09db: all 83 declared cases passed individually with the required inventory/check/format/layout/Clippy commands, backed by the 238-case regression checkpoint. No acceptance cases or assertions were removed. Task boundary audit preserves membership/snapshot work in CG-05, archival reclamation in CG-06, production SDK/fleet and automatic voter startup in CG-07, and final legacy removal in CG-16. CG-04 completion does not qualify those later capabilities or release profiles. Reviewed successor 394be03daf8f946a040d5f9c9130d98b907e8b80: CG-05 receiver staging verifies bounded ordered chunks, history/range/cut/membership, synced pages and independently sealed materialized/extent images before preparing an engine-local complete root. The cache anchor now records the installed checkpoint separately from the advancing applied root; its descriptor is persisted in existing Raft Snapshot data and validated on load. Engine publication refuses mixed contracts and stale/conflicting bases, retains current/pending/checkpoint closures around the shared sync fence, and retains the older checkpoint after subsequent apply. Recovery reconstructs both closures; owner/applier startup and collection refuse missing checkpoint retention. Three native cases exercise lagging existing voters, byte-complete transfer, empty enrollment, held engine sync, malformed/truncated/corrupt/foreign and quota refusal, subsequent quorum writes and joined cold reopen; deleting a checkpoint-only page refuses recovery despite a complete current tree. Only the manifest anchor changed; all other anchored files remain byte-identical. This is a receiver/storage slice, not Raft Ready snapshot-ticket or RPC integration. Learner/joint membership, receiver crash/sync-error/cancellation and root-slot exhaustion qualification, lost-disk, topology/park, RustFS, load and the full CG-05 gate remain outstanding. Qualification successor cb164e025e129330f576450d214cef0e8fe92aa1: a new corruption regression reproduced that GenerationStore pin_prepared can reuse a cached materialized seal. Receiver reuse now independently opens, inspects and syncs the destination under its pin after checking incoming bytes. The failed regression and intentionally interrupted candidate gate are retained; only the successor complete gate may qualify this slice. All anchors are unchanged from the reviewed receiver candidate. Reviewed successor 096cfb132397d9588ba9138b1cf474210fc062f3: verified group checkpoints now enter RawNode only through explicit step_checkpoint after envelope, destination, history, base, membership, bounds and engine/page-owner checks. Selected snapshots bind to the exact Ready and existing persistence ticket/lease; matching engine completion advances applied state and releases acknowledgement/CheckpointInstalled. Matching retained logs commit/replay without installing received roots. Five new native cases cover both image forms, held sync, shutdown join, invalid/overflow refusal and ignored snapshot replay; all twelve checkpoint cases passed before the final gate. Ordinary peer snapshot ingress remains closed. Local source checkpoint publication, bounded snapshot RPC, membership/disk replacement and full CG-05 qualification remain open. All anchored bytes and excerpt positions are unchanged. Reviewed successor 742b1e6d4e7155e16fb1b4eba0cce3b9056944ae: local group checkpoints use a distinct captured capability and SnapshotChange::CellGroupCheckpoint. Exact applied root/engine/configuration/membership checks reject mixed, stale and foreign writes. Storage compacts only the covered prefix and preserves hard/applied state and the entire suffix; existing retention pins protect current/pending/checkpoint roots through the engine fence. The owner persistence lane emits CheckpointPublished after matching completion; local compaction does not increment applied-state metrics. Exact outbound capture selects the retained Raft descriptor/closure after current advances. Three native cases passed for both image forms, actual RawNode-generated catch-up and suffix replay, cold leader engine/root recovery and shutdown during held publication. Materialized source recovery uses its fixture filename resolver; no RPC, automated policy, membership or full CG-05 claim. The CachedGroup excerpt is unchanged; reviewed cache.rs adds the typed local operation and preserves receiver/legacy branches. Only this anchor hash/line changed. Reviewed successor ceb7ed3149202660855d32df1ec5feed7775e068: dedicated bounded authenticated TLS checkpoint streams bind physical group/member/process identity and exact retained snapshots. Controller registrations revoke endpoints; physical source/receiver owners retain admission through cancellation. Verified receiver closure enters the existing exact Ready/persistence path. Reserved coalesced snapshot status admission prevents ordinary queue pressure from silently dropping feedback; actual failed transfer retry is exercised. Both image forms, suffix/cold recovery, invalid wire input, route revocation, dropped delivery and unrelated-group progress are covered by five native TLS cases. Existing anchored excerpts and hashes are unchanged. Automatic checkpoint policy, full receiver fault/pressure qualification, whole-group membership and disk replacement remain CG-05 work. Reviewed successor 1814b27896c3c21166186fe8fbd1a1a6bdbb721a: whole-group V2 configuration apply now requires an exact committed configuration entry and complete-root preparation in CachedGroup::changed. Pure replay independently validates the resulting ConfState and permanent bindings; the root stays unchanged and its term/membership revision match the entry. Existing legacy configuration/install checks and snapshot paths remain intact. Only the manifest anchor hash changed; its excerpt/line and all other source anchors are unchanged. Shared owner/runtime applies live Raft configuration only after sync, drives automatic joint-leave replication, supports caught-up voter handoff and fences demoted producers. Three new native cases and the 233-case regression gate pass; fresh learner seeding, lost-disk replacement, remaining CG-05 fault/pressure cuts and public cutover remain open. Reviewed successor ab2a308a937f349d3482a825d4b669d042cb03c3: authenticated fresh-group seed exchange binds exact physical identities and fresh ReadIndex source authority. Permanent node/engine/member claims stage a non-serving mapping; only an authenticated complete physical install atomically persists a local receipt with configuration, membership and current/retained roots. The CachedGroup anchor adds this local receipt; the required marker-1 group application record changes to manifest plus seed, with no old-layout reader. Generic snapshot imports cannot mint a receipt. Joined bootstrap revalidates permanent claims, active membership and receipt; same-owner physical recovery reconciles lost results without reopening pinned pages. Engine-queued work retains the grant and lease through actual sync. Joint-leave completion compares validated canonical configurations because live RawNode and pure replay can enumerate equivalent sets in different orders. Native tests delete voter 3 disk, refuse old identity reuse and unproven source/staged/generic startup, hold sync and cancel a waiter, seed/catch up/promote member 4, retire 3, transfer leadership, commit new state and cold recover both closures in both image forms. Test setup is split to bound debug future stack; no stack limit or safety assertion is relaxed. The seed-install fixture is added to CG-05 scope. All other source anchors are unchanged. Full CG-05 fault/pressure cases, managed fleet, RustFS/load/power-loss and zero-quorum qualification remain open. Reviewed successor df0ba1f67c0253a57127a215540f08bb13135279: namespace-owned group archive binding fixes the expected registry identity and canonical storage location before initialization; one-use authority and missing-counter refusal remain intact. Stable runtime checkpoint capture and upload_cut preserve the original committed root/configuration/membership across object-store I/O while quorum advances. The catalog file changes only the public range-directory reexports; the NamespaceConfig and BirthProposal excerpts are unchanged and each moves down one line. No other anchored file, excerpt, case inventory, oracle or admission quota changed. The first final plan unittest detected this unrefreshed catalog anchor after all Rust gates passed; its source-bound failed output is retained. CG-06 remains IN_PROGRESS and controller publication, S3 settlement/outage admission, local Raft file purge and fenced fresh-incarnation serving remain unqualified. Reviewed successor 0247fb8b951a9366bbabef0b0257280d358834e3: node controllers now own one fair, deduplicated dirty-group archive lane, exact-intent disk journals and recovery of uncertain publications. Abort plus second reconciliation fences delayed CAS before reallocation. New heads are journaled before predecessor hold retirement. Completion observes both current applied state and consumed apply events so recovering an old publication cannot lose an idle tail. A sealed execution capability uses captured tokens without shutdown authority and drains orphaned physical closures before verifier/journal/runtime teardown. Checkpoint cut races are typed NotReady; admission overload remains retryable. No anchored file/excerpt/hash changed. Shared S3 output settlement, unarchived-byte/disk admission, local Raft file purge, fenced fresh-incarnation serving, RustFS/load/process-kill qualification and public SDK/fleet cutover remain pending. CG-06 stays IN_PROGRESS; no original case, oracle or existing quota changed. Successor a4f37541cdc979252ea422d3dd5e224eb8b0ac2a removes only a duplicate integration-fixture module declaration caught by all-target Clippy. Runtime, assertions, oracles, anchored source bytes and quotas are unchanged; the failed source-bound gate output is retained. Successor 7914177d6f17e3f6d647a39dc496122157283591 preserves strict public checkpoint refusal after the unchanged prospective/stale-source regression exposed overbroad NotReady classification. Only the owner-private capture path can classify an already sealed root as transient when the engine has published a newer applied root. Prospective, stale public and foreign preparations remain Invalid. The full failed execution gate and executable are retained; its original assertion and all quotas remain unchanged. Reviewed successor dba41bda8d405264af2632c53c2d0d32894bdb8c: local command S3 settlement uses the existing controller archive lane and exact independently replayed receipts, including later cell images. A separate bounded S3 stage acquires retention before releasing resolved SQL output admission; original SQL and command quotas remain unchanged. Responses retain complete physical coverage, archive hold use and the exclusive journal lock, then recheck catalog birth and live leader authority. Original deadlines and callback identity remain fixed. A node budget of one qualifies deadline/cancellation isolation; missing sibling dependencies and foreign runtimes refuse. Receipt-lookup, enrollment and remote S3 output integration, unarchived-byte/disk admission, physical Raft purge, fenced fresh-incarnation serving and the public cutover remain pending. No anchored file/hash/excerpt or original acceptance oracle changed; CG-06 remains IN_PROGRESS. Successor 2b524f2a49978c3ac32acfbc4e08792a69f297fb boxes the retained private S3 proof bundle after Clippy identified large command preparation/settlement enums. Physical ownership, assertions, quotas and anchored source remain unchanged; the failed source-bound Clippy output is retained. Successor db78d771b5f3336bdebff4b5336e0d94067f08d4 shares immutable directory/birth authority per archive context instead of deep-cloning membership records into each response. Per-response S3 metadata remains fixed-size under its existing byte reservation; physical coverage, journal locking, output identity and all quotas/oracles are unchanged. The preceding six-case RustFS qualification and all-target Clippy pass at 2b524f2a remain retained; this successor is requalified separately. Successor 2e04d7509e2f9436d573adb656dfb911e07c20e2 adds positive namespace-minimum S3 qualification in native and real RustFS fixtures. Namespace setup now carries its actual configured minimum through catalog reopening; original Quorum defaults, production source and all original cases/quotas remain unchanged. Successor 2bf0dd2a6992954af41ff68722536064e4465a88 passes namespace minimum explicitly through the integration fixture after Clippy compilation rejected access to a crate-private getter. Production APIs, original defaults, assertions and quotas remain unchanged; the source-bound failed output is retained. Formatting-only successor 3d564f3db2c2eefc87119e7b3395f1bca17b99fa wraps the namespace archive fixture call using cargo format; production source, behavior, anchored bytes and quotas are unchanged. Local enrollment S3 responses now use the existing bounded archive lane and retained complete coverage: exact original creation provenance, namespace minimums, pre-initialization runtime ownership, post-proof catalog/leader/record checks, SQL-to-S3 retention transfer and deadline-safe resolved ownership. Remote enrollment still refuses unsupported S3 transmission. Existing anchor file hashes are unchanged; scoped native enrollment tests pass in both image forms. Full CG-06 acceptance remains pending. The source-bound all-target lint refusal at cc7171d3 retained its output; c2801b65 removes only two redundant fixture literal allocations. Production runtime bytes remain those of cc7171d3; cases, assertions and quotas are unchanged. Local S3 receipt lookup now seals a quorum selection, transfers SQL retention before queueing, reads the actual later archived observation, preserves typed failures/ambiguous absence and reserves payload bytes before fetch by growing the same S3 item. Remote requests/stored-minimum upgrades remain guarded on both ends, including provider-configured TLS gateways. The reproduced direct-caller domain panic at 7e1d93a0 retains its source/executable/backtrace; 214a1a13 scopes the recheck and passes all four new cases. The shared proof recheck now captures its domain and original deadline for all output kinds at 59619e0a. Original node SQL/S3 quotas and required cases/oracles are unchanged; the new expiry case deliberately fills a two-receipt ceiling. Existing anchor hashes are unchanged; full CG-06 acceptance remains pending. The source-bound macro lint refusal at 59619e0a is retained. cde0eea2 changes only the new cancellation fixture: poll the response and observe an archive attempt before cancellation, with unchanged node quotas and retained assertions. Reviewed successor 9eb958eda58841ebbc52c3b9d1df5debdc2cf96f: remote command/enrollment/receipt payloads preserve requested and achieved durability. A bounded owned reply retains the native S3 output through physical TLS flush; stream destruction precedes proof release on every cancellation path. Quorum output release, original deadlines, namespace/stored minima, exact receiver runtime ownership and mutation ambiguity remain enforced. Marker-1 hard cutover adds no legacy reader. New TLS fixtures reduce only real socket buffers and use the original 64 KiB S3 envelope, without changing original task cases/oracles or existing node quotas. Existing anchored files, excerpts and hashes are unchanged. Qualification remains pending. The initial source-bound Clippy refusal at 9eb958ed is retained. Successor 000e737ebc2a7517ee94722a532ea9159ceefd86 fixes only the new reply module logical import paths and waits for resolved ordinary controller admission before asserting same-cell progress with a held wire proof. Quotas, proof assertions and original cases are unchanged; qualification remains pending. The source-bound all-target refusal at 000e737e is retained. Successor 7d433bdbf6639d752aa692e8d0dd3a0bb993c2b1 fixes only two SQL-fixture router module references. Production runtime, assertions, original cases and quotas remain unchanged; qualification remains pending. Real RustFS source 7d433bdb passes remote output semantics and archive-wait deadlines/cancellation in both forms; its physical-delivery case and exact-executable fresh-prefix reproduction refuse before the response header. The independent raw probe encoded its operation string length as u32 while the production codec requires u16, decoding an empty operation. Successor 566e2e06884dfb44b3878f39a277e341800d113e corrects only that probe width. Production code, proof assertions, timeouts and all node quotas remain unchanged; the failed run/executable/backtrace and byte-level diagnosis are retained. Full remote qualification remains pending. The source-bound diagnostic at 79d6d1ae confirms that reading the TLS header prefetched the entire valid 29,226-byte S3 reply: transport query admission was zero and the body validated in 82 microseconds. The source 32bdcd2640f2f71dd38ea6b65b79b064bd324dd8 removes all temporary diagnostics and observes a partial encrypted socket write through the existing performance counter before any client read. Proof, 64 KiB admission, coverage, SQL release, same-cell Quorum progress, overload and cancellation assertions remain required. No production hook, deadline or quota changed; physical delivery qualification remains pending. The retained 32bdcd26 and 7b69250f RustFS probes confirm that a single undrained 30,422-byte encrypted reply fits kernel buffers even without a reader. Successor 25938fd86d13a2755066660797b07aff955ec1ad replaces that ineffective pressure trigger with a bounded window of sequential exact retries, waiting for each prior physical flush/refund before sending the next. A fixed host-lifetime physical query-reply write metric distinguishes write pressure from archive settlement and records success/failure/abandonment without durability authority. Original S3/SQL/command quotas, payload size, timeouts and proof assertions remain unchanged. All temporary DEBUG instrumentation is removed. Original task oracles and anchored source hashes/excerpts remain unchanged; qualification is pending. Runtime source 25938fd8 passes sixteen real RustFS cases in both image forms and 231 native/library regressions. The added exact metrics gate exposes a pre-existing main inventory mismatch: PR #159 added archive_current and archive_unconfirmed, while the test and API comment still expected 111 resource samples. The copied executable reproduces 113 versus 111. Validation successor debdc30c8327aad35757c6b849e029c1bbdab691 changes only the strict expected count to 113, explicitly requires both names, preserves 1-versus-32-group identity equality and corrects that documentation count. No runtime behavior, quota or original oracle changes. Unaffected source-bound passes are carried with their original runtime SHA and hashes; repaired metrics and source checks qualify separately. The remote S3 output runtime at 25938fd86d13a2755066660797b07aff955ec1ad with validation-only inventory successor debdc30c8327aad35757c6b849e029c1bbdab691 passes sixteen exact real RustFS cases in both image forms, seven signed protocol probes, twenty regression/check commands with 232 Rust cases across seven copied executables, strict runtime all-target Clippy, focused fixture Clippy and the independent source/artifact/selection/credential audit. The bounded exact-retry window proves actual held TLS writes without payload, timeout or original quota changes. Original task acceptance cases and oracles are unchanged; four original CG-06 cases and its full gate remain pending. Reviewed successor 66aa95e38fc30b0010bee70e7f6539745f99face: configured archival now persists an irreversible local archive_required bit before object-store I/O. Recovery restores that requirement but no verified frontier; independent complete coverage under the journaled remote hold reopens local compaction. The scheduler skips blocked captures while apply and sibling groups progress. Owner and storage preflight enforce the exact cut, including direct writes. Marker 1 now requires the local field, with no legacy decoder or migration; peer snapshots cannot clear it. The CachedGroup anchor adds that bool and its reviewed file hash/line are updated. No original task case, oracle or quota changes. Explicit unarchived-byte/disk admission, physical file purge, peer-install deployment policy, fenced fresh-quorum recovery and the public cutover remain pending. Qualification is pending. The source-bound all-target compile refusal at 66aa95e3 is retained: its new fixture passed a marker where capture needs the prepared capability and omitted OperationId. Successor ea995319971d023be37e4446c7b87eb467bff2c8 uses runtime capture, imports the explicit identity and orders the sibling checkpoint before retaining the unchanged one-source slot. Archive admission also checks the actual local birth before persisting policy. No original case, quota or assertion is relaxed; qualification remains pending. The retained all-target refusal at ea995319 identifies the new recovery test read guard despite explicit drop; successor 93b51065410952389f9345a4afa795d2d1971151 uses a lexical block before awaits. Review also found activation could retract an already admitted checkpoint while retaining its source slot. Owner and physical-storage admission guards now defer policy activation until that work drains; only this capacity refusal is retried at the activation boundary. The recovery test proves refusal before durable activation and success after the guard drains. Original quotas and cases are unchanged; all qualification remains pending. Native qualification at 93b51065 and a86f29e1 refused the invalid private metadata operation zero; copied reproduction and typed status identify the cause. Corrected private operation is nonzero, with public validation and physical sync sequencing preserved; qualification remains pending. Scoped archive-aware local compaction at 8dfe9ff3a316d0579b656af4818dc57d839a6765 passes seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 260 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes. The durable requirement and fresh verified frontier protect local compaction; admission guards drain earlier physical work before activation. Explicit retention admission, physical file purge, peer-install deployment policy and fenced fresh-quorum/public cutover remain pending. No original task case, oracle or quota was weakened. An earlier native slow-reader timeout remains retained and unexplained; unchanged isolated, ordered-controller and complete reruns pass. Activation fairness under sustained checkpoint traffic needs an adversarial gate before managed capture/admission. Reviewed successor ab0abf26dea4df436b5ced926f28a9ede4f360fd: the deterministic 79323600 regression shows pending activation can be overtaken by another checkpoint admission. Registration now owns an activation barrier across retries; owner and storage share an exact group/cut-bound lease so earlier work can finish. Entered metadata I/O owns activation after waiter loss; only synced cache/root publication makes the requirement permanent. Unentered rejection/shutdown releases intent, while existing durable policy remains irreversible. Frontier reset occurs once before I/O to avoid late duplicate completion discarding newer coverage. The existing 4 KiB checkpoint metadata charge bounds the cut lease. Serialized marker-1 records, original cases/oracles and quotas are unchanged; only the manifest anchor file hash/line changes with private GroupWrite owners. New physical-storage tests cover wrong lease bindings, old work draining, unentered cancellation and a held-sync lost waiter. Qualification is pending. Explicit retained-log admission, full-backend outage, physical purge, fresh-quorum recovery and hard cutover remain incomplete. Owned activation and cut-bound physical checkpoint drain at ab0abf26dea4df436b5ced926f28a9ede4f360fd pass three exact storage ownership cases, seventeen real RustFS cases in both image forms, seven signed S3 probes and twenty-two regression/check commands with 262 Rust cases. Independent source/artifact/selection and credential-exclusion audit passes; the original overtaking failure is retained. Sustained load fairness, explicit retained-log admission, full-backend outage, physical file purge and fenced fresh-quorum/public cutover remain pending. Original task cases/oracles, quotas and marker-1 serialization are unchanged. Reviewed candidate c2759d74fe652fccff52e6f3f5878a89263f4161: shared-group retained-entry admission owns node/group byte and entry reservations before SQL callbacks and Raft appends. A selected physical write atomically transfers exact entry-bound leases into actual durable debt; reopen reconstructs debt. Empty election/automatic joint-leave work has bounded separate progress capacity, while reads and existing completion/checkpoint work retain independent lanes. Callback envelopes shrink after encoding; sealed rejection and authoritative reconciliation retire unused preparation. Marker-1 serialization and original task oracles/quotas are unchanged. The new log limits are explicit and do not establish physical file amplification or full-backend outage authority. The manifest anchor excerpt is unchanged; its file hash/line reflect GroupWrite/GroupLog owners. New storage and native both-image boundary cases are not yet qualified. Full CG06/outage/purge/fresh-quorum/cutover requirements remain pending. The source-bound initial compile at c2759d74 refuses two private tracker-field accesses. Successor 2df640c995f1d9e478792ac64ebebc592909130c uses the public auto_leave() accessor in both checks. No limits, behavior, assertions, original cases or anchored hashes/excerpts change; all qualification remains pending. The retained successor compile at2df640c9 identifies the generated public getter returning &bool. Successor e090757f3c09ad3d90cd50a73609d3f171e81ea6 dereferences that flag in both checks. Limits, assertions and anchored source hashes remain unchanged; qualification is still pending. Retained all-target failure at e090757f identifies the new GroupLog pointer increasing the existing SQL Action enum past its size bound. Successor 1a8970840e2214bb88186e23dbd37effcbe80229 shares the ledger reference through the existing engine-local storage-policy allocation and retains compact GroupLog handles, without boxing unrelated SQL actions or suppressing Clippy. Group/cache ownership, entry binding, limits, original test assertions and serialized records are unchanged. The manifest anchor file hash/line is refreshed; qualification remains pending. Locked all-target Clippy passes at1a897084 with compact handles. Review successor b36ab33d10be7edf150882929c56cf43504660f1 restores execution-fence refresh immediately after four RawNode campaign/tick/step mutations, before new hash/accounting work. This preserves prompt role revocation to concurrent SQL workers; no limit, serialized record, assertion or anchored source hash changes. Runtime qualification and final-source Clippy remain pending. The exact retained-entry storage/reopen case passes atb36ab33d in the actual engine (0.09 s runtime; source-bound build/executable retained). Test successor 4e05c21430216ab12e5341244ec59c7c355f76fc drops every caller lease during held sync and uses Weak to prove physical-only ownership, then verifies consumed-cap refusal and no double debit with a surviving client lease. Both native image scenarios now fill ordinary admission with a valid auto-joint configuration and require automatic leave through the unchanged single progress slot. Production source, original limits and task oracles are unchanged; these stronger cases and final-source qualification remain pending. The retained4e05c214 wrong-term lease negative reproduces from its copied executable: its proposed term2 entry still used hardstate term1, so earlier cache validation refuses before lease checking. Successor 5238b9ac3e7cb07e3fb113642bc8f664d227f37a makes that proposed hardstate coherent, keeps the WrongIdentity assertion and adds selected-state invariance checks plus typed failure context. A new real-engine case isolates group/node byte ceilings with slack entry counts and verifies selected suffix replacement releases bytes. Production source and original task oracles/quotas are unchanged; these cases still need execution. Native fixture successor 462db264cac1a4e3f499b16227b905abc03329e0 replaces the impossible leader-only readiness wait on followers with exact selected checkpoint identity, zero retained-entry debt and released source quota on every voter. The unchanged copied5238b9ac native binary reproduces its deadline. Production code, quotas, deadline and original task oracles are unchanged. Three exact storage cases pass at5238b9ac; successor native and final-source qualification remain pending. Snapshot aggregate successor ad73f03f278f5845b311fe929d184cc947382b2d uses the existing non-inlined fixture constructor for TCP publication and install futures. The broad462db264 run and its unchanged copied binary reproduce a default-stack overflow in snapshot_empty_cell_and_extent_restart, while both constituent scenarios pass in the159-case execution suite. Stack size, quotas, deadlines, production code and every assertion/oracle are unchanged. The six-case focus, final-source Clippy, fresh RustFS and complete broad gate are pending; earlier passing focus/Clippy/RustFS evidence and original failure remain retained. Retained-entry admission at ad73f03f278f5845b311fe929d184cc947382b2d passes three real-engine focus cases, both SQL image-mode focus cases including ordinary-full automatic joint leave, seventeen real RustFS cases in both image forms, seven signed probes and twenty-two regression/check commands with 267 Rust cases. Independent source/artifact/selection/credential audit passes. Copied fixture failures remain retained. Logical byte/count debt admission is qualified at these boundaries; physical file bounds, full-backend outage authority, fenced fresh-quorum serving and public hard cutover remain pending. Original task cases/oracles/evidence array and marker-1 records are unchanged. Physical-reclamation qualification candidate 0ecd34db504104354adc034c2959e268696790b7 adds the original group_reclaim_restart_keeps_history case in both image modes and a real RustFS counterpart. Explicit test geometry uses 4KiB Raft files, 8KiB purge threshold and manual purge only in this scenario; existing fixture defaults/quotas remain unchanged. Actual files and length reduction, an uncheckpointed sibling, retained suffix, old reader, checkpoint streaming, bounded image/page collection, all3 retained-disk reopen/fresh quorum and exact receipt/effect recovery are asserted. No production behavior, marker or original acceptance oracle changes; execution and qualification remain pending. Successor 2d9be1b52d39223f9ec71d85b42fb3c16276c004 corrects test-helper visibility to its enclosing runtime fixture and adds two fresh post-restart TLS commands after all old receipt retries, checking every voter image/effect. The source-bound0ecd34db compile failure is retained; no runtime case executed at that revision. Production code, original oracle and resource limits remain unchanged. Execution and final qualification are pending. Bounded-reader successor fae2f9e341e490de841eed070c68b9080c0b52a9 preserves the complete image oracle but reads materialized images in at-most64KiB chunks. The exact2d9be1b5 case and its copied executable reproduce GenerationError::Limit; GenerationReader::read_exact_at rejects any larger read. No read budget, payload, receipt count, physical assertion, resource limit or production code changed. All qualification remains pending. Checkpoint-target successor 51a846be978c296c491509a997e8a91703b0f838 verifies post-purge retained checkpoint data through the private archive receiver, which grants no engine/vote/serving capability, and explicitly asserts that an ahead live voter still refuses the older cut. Separate one-shot source transfers retain both original one-slot pool limits. The copiedfae2f9e3 binary reproduces the earlier inappropriate live-target refusal before dependency bytes. Original rollback guard, complete data assertions, quotas and production behavior are unchanged; full qualification remains pending. Verification-lifetime successor 58d6559956eb7d37082a7bf066a21cda0df70945 retains independent coverage through actual purge/checkpoint checks, asserts its occupied one-slot archive receiver budget, then drops that finished result before waiting for a new archived cut. The journal retains the remote hold. The exact51a846be case and copied executable reproduce the prior circular wait; ownership types show the held verification permit. No quota, deadline, original assertion, payload or production behavior changes; full qualification remains pending. Selection-parser successor 9e7246b322b45e1d0f2d03a53ba387df4d2a0511 normalizes only the two validated structured observation lines that interrupt libtest progress, preserving exact case matching and the one-pass/zero-failure/zero-ignore summary requirement. The actual58d65599 Rust case passes both image modes, but its wrapper reported selection failure with exitcode0 because those notes split the result line. Raw output, executable and original record stay immutable. No Rust code or test assertion changes. Final qualification is pending. Diagnostic-output successor ea2717b62457f5038fedd6bf4dc54c22f464af16 uses the existing cellar::note API for native qualification observations instead of forbidden stdprintln, following the metrics fixture convention. The exact9e7246b3 native case passes in both modes; its all-target Clippy failure remains source-bound. No physical/recovery behavior, assertion, payload, resource limit or deadline changes. Final-source qualification remains pending. Shared-module successor 37f3def8e06c4c05ff1bd605d77db0062feaa3bb declares the archive-store fixture once in each integration target and reuses it inside the controller harness. Adding runtime support to the archive target had loaded the same file twice; all-target Clippy at ea2717b6 refused duplicate_mod. No lint suppression, runtime/resource change or original assertion changes. The source-bound passing exact ea case and lint failure remain retained; final-source qualification is pending. Owning-module successor ea2f802475aa849a1b00f0990eff6929beb4f207 places the archive-only reclamation scenario under the existing archive harness and reexports its boxed off-frame constructor, preserving target-specific module ownership. All-target Clippy at37f3def8 had refused the unused standalone helper in execution-only targets. No production code, existing target lint policy, original assertion, quota or deadline changes. Final-source qualification remains pending. Visibility successor 7741e4049704f3e1c6519b736e62deaf263c39a2 gives the SQL fixture reexport crate visibility, as required to expose the archive scenario through the runtime fixture to its integration target. The ea2f8024 compiler refusal and exact source/output remain retained. No scenario behavior or production code changed. Final-source qualification remains pending. Physical-reclamation qualification at 7741e4049704f3e1c6519b736e62deaf263c39a2 passes the original native case in both image forms, eighteen real RustFS cases/seven signed probes and twenty-two checks with268 Rust cases, seven copied executables and2575 source hashes. Actual old files disappear and aggregate file length falls while readers, checkpoints, suffixes and siblings survive; all3 retained voters reopen, all24 receipts resolve without callbacks and two new TLS commands commit. Independent source/artifact/selection/credential audit passes. Original task cases/oracles/evidence array, production behavior and marker1 records are unchanged. Three original CG06 cases/full gate, full-backend outage authority, fenced fresh-incarnation serving and public hard cutover remain pending. Reviewed complete-group dependency successor dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8: adds the original CG06 dependency acknowledgement gate and realRustFS counterpart with delayed/corrupt actual B image bytes unique from A, actual local quorum successors, persisted independent recovery holds, complete recovery refusal and exact receipt checks. The recorded CG06 scope extension adds the owning sibling fixture plus existing runtime/SQL/S3 test export paths. No production source, anchored excerpt/hash, marker, original oracle/invariant, quota or deadline changes. Final-source qualification remains pending. Complete-group dependency qualification at dc0ad3c245a2f9e3dd4f261003c868e9a43f19b8 passes the original delayed/corrupt B image oracle in both image modes, nineteen real RustFS cases/seven probes and twenty-two checks with269 Rust cases/seven copied executables/2576 sources. Actual B data differs from A, later A S3 results wait or fail, Quorum progresses, stored selection stays before A, independent held recovery blocks or refuses the exact digest failure, and exact receipts never replay callbacks. Independent source/artifact/selection/credential audit passes. Production behavior, existing quotas/deadlines, original case oracles/invariants and marker1 records remain unchanged. Two original CG06 cases/full task gate, public hard cutover and full end-to-end goal remain pending. Reviewed full-backend outage successor cfbeaab04be0df734128668402e7388bc34b949f: adds the original bounded outage gate in both image modes, a real RustFS stop/start counterpart and an owner-checked Python controller with fixed operations, pinned container/resource/mount checks and joined child/cleanup. Existing fixture constructors retain default settings while the new scenario selects nine entries with one progress slot. Local cached quorum, fresh remote refusal, exact retained invocation inputs/deadline/budget, S3 minima, admission before callbacks/initializers, complete recovery and checkpoint debt retirement are asserted. The recorded scope extension adds the owning outage/control fixtures and runner/controller tests. Production source, anchored excerpts/hashes, marker1, original oracles/invariants and existing quotas/deadlines are unchanged. Final-source qualification remains pending. Inventory successor b0e2dbafec4dab9cdc91345bc9ff847635e4ebf5 replaces an incorrect borrowed four-entry fixture assumption with exact retained indexes2,3,4 and equality to ledger debt on every voter. Registered setup enrolls two cells; the preceding private retention fixture enrolls three. The copied cfbeaab0 executable reproduces the initial mismatch. The scenario now fills six registered commands to the unchanged entry ceiling9 and refuses command7 before callback, preserving all timeouts, quotas, payloads, actual history/data and original oracle. Production behavior is unchanged. Qualification remains pending. Data-readiness successor 1159e9358c106fe1260c3479759e909e413257a4 handles the observed first post-start S3 GET503 after container health200 by retrying only idempotent Generic read failures under the existing retained remote invocation absolute deadline. Missing/auth failures still refuse and exact probe data must match; noRetry outage adapter, mutating retry rules, original deadlines/limits/oracle and production source are unchanged. The original b0e2dbaf realRustFS failure and copied-binary reproduction on a fresh owned backend both retain exact503 output, container transitions and joined cleanup with stopped backend data. Native/Clippy passed atb0; final-source full qualification remains pending. Full-backend outage qualification at 1159e9358c106fe1260c3479759e909e413257a4 passes the original case in both modes, twenty real RustFS cases/seven signed probes/four pinned owner-checked stop-start transitions and twenty-three checks with270 Rust cases/seven executables/2580 source hashes, 37 plan tests and five controller tests. Warm local Quorum reaches the fixed entry9/progress1 boundary; fresh remote admission refuses beforecallback; original inputs/deadline/minima persist; complete recovery permits exact held checkpoints, all-voter debt retirement and resumed writes without replay. Independent source/artifact/selection/credential/backend-transition audit passes. The failed cfbeaab0 inventory assumption and copied reproduction remain retained. Production source, original oracles/invariants, existing quotas/timeouts and marker1 records remain unchanged. Fenced fresh-incarnation serving/fullCG06, public cutover and remaining full goal are incomplete. 2026-11-01: reviewed the 1159e935 runtime and 8e5ca2fc handoff for the next CG-06 recovery boundary. The successor adds a typed whole-group exact intent, mandatory marker-1 nullable recovery lock in the directory head, fleet-import exclusion and independently verified durable fencing. This is a hard record-shape cutover, with no legacy reader. Static unmanaged fencing/data proofs grant no replacement bootstrap or serving authority; fresh history/image/receipt rebinding, installation, activation and Docker deleted-voter qualification remain required. Original task oracles/invariants/status evidence are unchanged. 2026-11-01: reviewed the recovery-fence successor at 65c5176975907fb52a53deca58e1ae1f5ba7adec. Whole-group intent/fence barriers remain data-only, with fresh-genesis serving still required. The broad retained learner drill exposed a deterministic default-stack regression relative to the copied 1159e935 executable. Boxing rare recovery-lock metadata restores its exact default-stack test with identical canonical record bytes. Final broad/RustFS qualification is recorded separately; no task oracle or status evidence is relaxed. 2026-11-01: reviewed 65c51769 runtime / 9f3b5b86 published fence barrier for whole-group fresh-genesis restoration. SQLite receipts/client floors contain no old version authority; CellCommittedReceipt derives its covering version/witness from the selected leaf. The successor must preserve SQLite bytes, create fresh leaf histories and a durable pending/activated recovery lifecycle, and keep every application admission path closed before committed activation. Canonical recovery-intent/source holds retain original witnesses as provenance. Marker-1 root shape is a hard cutover, without legacy readers. Original CG-06 oracle/status and all sixteen tasks remain unchanged. 2026-11-01: Reviewed the candidate recovery lifecycle cache guards from 9f3b5b86. Applied roots preserve the exact recovery lineage; authenticated empty learner installation is the only snapshot acquisition boundary. The anchored CachedGroup shape is unchanged. Mandatory nullable recovery metadata stays at marker 1 with no legacy default. Candidate import/lifecycle qualification remains pending. Recovery import and lifecycle source committed at 7052d218d60181d7b0bb5a4c12686e80e294a61c; focused recovery tests pass. Full scoped native and RustFS qualification is pending. Scoped qualification at 7052d218d60181d7b0bb5a4c12686e80e294a61c now passes: seven native suites (246 cases), 36 recovery-related unit tests, eleven legacy disaster cases, real RustFS import in both image forms and all recorded static checks. Evidence CG-06-recovery-import/qualification/record.json SHA256 1c7ea62c6bd5c1e81357961c4acba22bb2820d89bc3556ec3bcaf82a43e1c0a3. The original fresh-serving oracle/full CG-06 gate and all later tasks remain pending. MVP continuation from 973cad34/7052d218: reviewed node storage/genesis, immutable mappings, physical retention and directory birth reservation before edits. Seed genesis must bind the actual imported root and local node/disk/process/engine through a catalog permit. The global reservation now records mandatory nullable recovery origin, so missing optional seed metadata cannot fall through to ordinary startup. New wire kinds remain marker1 with no legacy reader; candidate qualification is pending. Reviewed fresh-genesis/activation successor from 973cad34: actual sealed engine seed grants, atomic complete mapping/root publication, permanent recovery-origin and physical seed claims, locked topology publication, recovered startup and catalog-bound replicated activation. The CachedGroup anchor now includes immutable mapping-derived recovery genesis shared behind the existing cache; GroupLog handles retain their previous size. Cache/mapping equality preserves this lineage through writes and reopen. Ordinary startup refuses recovery origins; pending reads refuse before wait admission. The owner independently derives activation from the committed entry and preserves lifecycle on other commands. Marker1 envelopes use hard cutovers only. Source/test review and all-target Clippy pass; final-source native/RustFS qualification remains pending. Original task oracles/status/evidence are unchanged. Runtime source pinned at 12944b762631c300ec6da2a8f6528394a19e4eda after all-target Clippy passed. Final-source native/RustFS regression is pending; this is not the original fifth oracle or completed CG06. Reviewed successor 2c4326d5cf29f3935d4859e97a52af0ee08b170a: a deterministic stale-B observation regression on unchanged 12944 runtime showed reconciliation returning terminal Unavailable. Only reconciliation NotReady now returns Uncertain, retaining the original producer and requiring a fresh proof; identity/closed-engine/newer-term absence checks remain intact. The existing stale-proof assertion is strengthened to exact Uncertain plus retained producers. Registered-query qualification joins actual physical/queue cancellation ownership before another one-slot admission, without changing original quotas, deadlines or result assertions. Four reconciliation, two registered-query and one receipt-ingress cases pass at these exact source bytes, as do all-target Clippy, formatting and layout. The two earlier intermittent full-suite failure locations are not causally proven; their copied-executable isolated passes and deterministic red regression remain retained. Final-source broad native/RustFS and regenerated original CG01-05 gates remain pending. No original task oracle, evidence requirement or status is weakened. Reviewed 9a0bad9124f0089055a0ba63b2952e5ef5e77853: physical genesis/activation and registered receipt recovery from archived LTX tails pass the scoped native/RustFS checks; Docker fencing remains pending. Integrated main 81cbbc2f72f302751dd471cf06cf51394146ac63: legacy message-delivery ownership now lasts through transport completion before membership apply, and CI uses macOS 15. The host/runtime/ticks/CI excerpts are unchanged; reviewed file hashes and runtime line offsets are refreshed. No cell-group protocol anchor or task oracle changed. See progress.md for scope and evidence availability. Reviewed successor 51d136fbea6732cb93553c1a57cd45547ca9aa59: physical engine close now releases only derived manifest pins after the closed fence and engine join, before publishing its terminal result. Unwind cleanup releases these pins too; external readers retain their own owners. Both retained-closed-handle recovery regressions fail at 5e7aadb6 with image/extent directory-lock refusal, then pass with this fix. Twelve checks pass, including full execution/archive/effects/namespace-host/controller-archive native suites and strict all-target Clippy; no new RustFS qualification is claimed. CachedGroup fields/excerpt are unchanged and move by 25 lines. Canonical SDK/fleet cutover remains uncompiled local work and CG-07 remains IN_PROGRESS. Reviewed successor 66d5ea6aa4ff5f2b2753f17c2cf4f7bf3c706136: bounded physical range-directory enumeration uses immutable authenticated tree paths and an exact-view cursor, without cell scans, S3 listing, mutation, voter or serving authority. The six-file checkpoint passes its exact native case, all nineteen catalog cases, strict all-target Clippy, formatting/layout, native dependency and docs checks. The missing initial catalog re-export failure is retained before any test execution. The only anchored file change is the public cursor/page re-export in catalog.rs; NamespaceConfig and BirthProposal excerpts remain unchanged and unique, with reviewed line shifts. This is metadata inventory only: no canonical SDK/fleet cutover, managed runtime, new RustFS, load or final release qualification is claimed. Native record SHA256 8061d03d7902d21f291fb82b16bcf04c91d20716465ef5292eecd7c754dcd91a.", "tikv_revision": "b3995dab28308f09cbb7c125bf32f789dbc203fe", "anchors": { "identity": { @@ -61,14 +61,14 @@ }, "catalog": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "19523ded3a4ef95720f9f5ac130fec5b41533e7f6a2dfe627344c5819cfd7d66", - "line": 118, + "sha256": "d373a7003a364b1dd15dcc187b337bb025340762764f57a8d3f5a442cf5389ad", + "line": 119, "excerpt": "pub struct NamespaceConfig {\n pub cluster: ClusterId,\n pub namespace: NamespaceId,\n pub mode: NamespaceMode,\n pub schema_digest: [u8; 32],\n pub minimum_durability: Durability,\n pub voters: VoterPolicy,\n /// Immutable birth-admission owner. Managed namespaces refuse the ordinary\n /// creation API; each birth needs an inventoried fleet reservation." }, "birth": { "path": "crates/cellar/durability/raft/catalog.rs", - "sha256": "19523ded3a4ef95720f9f5ac130fec5b41533e7f6a2dfe627344c5819cfd7d66", - "line": 187, + "sha256": "d373a7003a364b1dd15dcc187b337bb025340762764f57a8d3f5a442cf5389ad", + "line": 188, "excerpt": "pub struct BirthProposal {\n pub scope: GroupScope,\n pub voters: Vec,\n pub learners: Vec,\n}\n\n/// Constructed only after a canonical immutable record and its reservation have\n/// been read back from the selected namespace. Its placement never overrides\n/// committed Raft configuration once a member has initialized." }, "version": { From f50b99af981e8c4f28307fb71e5ac8e5f57b433f Mon Sep 17 00:00:00 2001 From: forhappy Date: Tue, 3 Nov 2026 00:58:23 -0700 Subject: [PATCH 273/568] feat(raft)!: advance shared-group cutover BREAKING CHANGE: canonical SDK and fleet consumers use shared physical cell groups with independent logical cell histories. Retired per-cell APIs have no compatibility path. Publish the cutover checkpoint and its MVP acceptance scope. Public consumer ports, managed S3 activation and complete build, lint and backend gates remain unfinished; keep PR #160 in draft. Refs #160 --- crates/cellar-admin/README.md | 26 +- crates/cellar-logic/raft_fleet/ledger.rs | 12 +- crates/cellar-logic/raft_fleet/mod.rs | 2 +- crates/cellar-logic/raft_fleet/repair.rs | 50 +- .../cellar-logic/raft_fleet/topology/mod.rs | 19 +- .../raft_fleet/topology/schedule.rs | 2 +- .../cellar-logic/tests/raft_fleet_ledger.rs | 26 +- .../cellar-logic/tests/raft_fleet_repair.rs | 12 +- .../cellar-logic/tests/raft_fleet_topology.rs | 49 +- crates/cellar/application/consensus/class.rs | 9 + crates/cellar/application/consensus/client.rs | 169 +- .../application/consensus/client_execution.rs | 122 - .../application/consensus/client_fleet.rs | 22 - .../application/consensus/client_node.rs | 141 - .../consensus/group_client/actions.rs | 297 ++ .../consensus/group_client/close.rs | 227 + .../consensus/group_client/drain.rs | 144 + .../consensus/group_client/fleet.rs | 195 + .../consensus/group_client/learner.rs | 204 + .../consensus/group_client/membership.rs | 301 ++ .../application/consensus/group_client/mod.rs | 562 +++ .../consensus/group_client/node.rs | 367 ++ .../consensus/group_client/repair.rs | 274 ++ .../consensus/group_client/resources.rs | 238 ++ .../consensus/group_client/services.rs | 236 ++ .../consensus/group_client/services_tests.rs | 37 + .../consensus/group_client/startup.rs | 316 ++ .../consensus/group_client/stop.rs | 66 + crates/cellar/application/consensus/mod.rs | 69 +- .../cellar/application/consensus/protocol.rs | 2 +- crates/cellar/durability/raft/README.md | 163 +- .../durability/raft/archive/coverage.rs | 239 -- crates/cellar/durability/raft/archive/mod.rs | 3 - crates/cellar/durability/raft/catalog.rs | 53 +- .../cellar/durability/raft/catalog_birth.rs | 40 - .../raft/catalog_inventory/groups.rs | 417 ++ .../durability/raft/catalog_inventory/mod.rs | 2 +- .../durability/raft/catalog_inventory/wire.rs | 59 + crates/cellar/durability/raft/catalog_wire.rs | 1 + .../durability/raft/cell_groups/README.md | 115 +- .../raft/cell_groups/apply/recovery.rs | 28 + .../raft/cell_groups/archive/fleet.rs | 95 + .../raft/cell_groups/archive/mod.rs | 3 + .../raft/cell_groups/archive/ownership/mod.rs | 56 +- .../cell_groups/archive/ownership/state.rs | 19 + .../archive/ownership/state_tests.rs | 116 + .../raft/cell_groups/checkpoint/header.rs | 15 + .../cell_groups/checkpoint/header_tests.rs | 44 + .../raft/cell_groups/checkpoint/mod.rs | 3 + .../raft/cell_groups/checkpoint/receive.rs | 30 +- .../cell_groups/checkpoint/receive_archive.rs | 42 + .../checkpoint/receive_archive_tests.rs | 52 + .../raft/cell_groups/checkpoint/seed.rs | 77 + .../raft/cell_groups/checkpoint/seed_tests.rs | 148 + .../raft/cell_groups/controller/admission.rs | 122 + .../raft/cell_groups/controller/calls.rs | 10 +- .../raft/cell_groups/controller/commands.rs | 31 +- .../cell_groups/controller/effects/mod.rs | 15 + .../cell_groups/controller/enrollments.rs | 22 +- .../raft/cell_groups/controller/mod.rs | 88 +- .../controller/provision/config.rs | 134 +- .../controller/provision/learner.rs | 330 ++ .../controller/provision/manager.rs | 151 +- .../cell_groups/controller/provision/mod.rs | 172 +- .../controller/provision/prepare.rs | 225 +- .../raft/cell_groups/controller/queries.rs | 25 +- .../cell_groups/controller/registration.rs | 36 + .../raft/cell_groups/controller/worker.rs | 12 +- .../raft/cell_groups/directory/birth.rs | 151 +- .../raft/cell_groups/directory/bootstrap.rs | 167 +- .../raft/cell_groups/directory/import.rs | 315 ++ .../cell_groups/directory/joined_bootstrap.rs | 183 +- .../raft/cell_groups/directory/mod.rs | 46 +- .../raft/cell_groups/directory/wire.rs | 8 + .../raft/cell_groups/dispatch/mod.rs | 77 +- .../raft/cell_groups/effect_index_tests.rs | 58 + .../durability/raft/cell_groups/host/mod.rs | 97 +- .../raft/cell_groups/host/storage.rs | 4 +- .../durability/raft/cell_groups/manifest.rs | 12 +- .../durability/raft/cell_groups/membership.rs | 96 +- .../cell_groups/membership_observation.rs | 102 + .../cellar/durability/raft/cell_groups/mod.rs | 17 +- .../durability/raft/cell_groups/owner.rs | 318 +- .../durability/raft/cell_groups/page_store.rs | 39 +- .../durability/raft/cell_groups/page_wire.rs | 24 + .../raft/cell_groups/read_authority.rs | 18 + .../raft/cell_groups/recovery/tests.rs | 2 + .../durability/raft/cell_groups/retention.rs | 29 + .../durability/raft/cell_groups/root.rs | 13 + .../raft/cell_groups/runtime/mod.rs | 100 +- .../raft/cell_groups/runtime/reads.rs | 112 +- .../raft/cell_groups/runtime/worker.rs | 98 +- .../durability/raft/command_cell_transport.rs | 25 +- crates/cellar/durability/raft/completion.rs | 4 +- .../cellar/durability/raft/effect_delivery.rs | 65 +- .../durability/raft/fleet_action_transport.rs | 4 +- .../raft/fleet_catalog/archive_replay.rs | 208 + .../durability/raft/fleet_catalog/capacity.rs | 4 +- .../fleet_catalog/docker_isolation/mod.rs | 8 +- .../docker_isolation/resources.rs | 9 +- .../raft/fleet_catalog/evacuation.rs | 10 +- .../finalization/archive_holds.rs | 228 + .../fleet_catalog/finalization/fencing.rs | 6 +- .../raft/fleet_catalog/finalization/mod.rs | 61 +- .../fleet_catalog/finalization/receipts.rs | 266 +- .../finalization/receipts_tests.rs | 93 + .../fleet_catalog/finalization/survivors.rs | 10 +- .../fleet_catalog/finalization/terminal.rs | 8 +- .../raft/fleet_catalog/histories.rs | 292 +- .../raft/fleet_catalog/histories_tests.rs | 104 + .../raft/fleet_catalog/imports/README.md | 43 +- .../raft/fleet_catalog/imports/coverage.rs | 206 +- .../raft/fleet_catalog/imports/inventory.rs | 122 +- .../raft/fleet_catalog/imports/mod.rs | 133 +- .../raft/fleet_catalog/imports/records.rs | 20 +- .../fleet_catalog/maintenance/birth_action.rs | 12 +- .../raft/fleet_catalog/maintenance/mod.rs | 4 +- .../maintenance/operator/driver.rs | 4 +- .../fleet_catalog/maintenance/operator/mod.rs | 4 +- .../maintenance/operator/service/mod.rs | 9 +- .../maintenance/operator/service/worker.rs | 50 +- .../maintenance/resume/leadership.rs | 49 +- .../fleet_catalog/maintenance/resume/mod.rs | 13 +- .../maintenance/resume/verification/mod.rs | 6 +- .../runtime/genesis/materialize.rs | 13 +- .../maintenance/runtime/genesis/receive.rs | 70 +- .../maintenance/runtime/handoff/receive.rs | 51 +- .../fleet_catalog/maintenance/runtime/mod.rs | 19 +- .../maintenance/runtime/proof.rs | 10 +- .../maintenance/runtime/reconcile.rs | 4 +- .../raft/fleet_catalog/maintenance/settle.rs | 34 +- .../fleet_catalog/maintenance/settlement.rs | 6 +- .../durability/raft/fleet_catalog/mod.rs | 22 +- .../fleet_catalog/namespaces/inventory.rs | 172 +- .../durability/raft/fleet_catalog/nodes.rs | 13 +- .../durability/raft/fleet_catalog/permits.rs | 93 +- .../raft/fleet_catalog/resources/admission.rs | 15 +- .../fleet_catalog/resources/completion/mod.rs | 3 +- .../resources/completion/proofs.rs | 2 +- .../fleet_catalog/resources/execution/mod.rs | 4 +- .../resources/execution/receive.rs | 186 +- .../resources/execution/service/isolated.rs | 4 +- .../resources/execution/service/mod.rs | 4 +- .../resources/execution/service/recovery.rs | 4 +- .../resources/execution/service/removal.rs | 98 +- .../resources/execution/service/tests.rs | 3 +- .../resources/execution/service/topology.rs | 9 +- .../resources/execution/service/worker.rs | 55 +- .../resources/intent/admission.rs | 2 +- .../fleet_catalog/resources/intent/mod.rs | 4 +- .../resources/intent/progress.rs | 2 +- .../fleet_catalog/resources/membership.rs | 2 +- .../raft/fleet_catalog/resources/mod.rs | 5 +- .../raft/fleet_catalog/resources/movement.rs | 4 +- .../resources/repair/membership.rs | 4 +- .../fleet_catalog/resources/repair/mod.rs | 8 +- .../resources/repair/planning.rs | 12 +- .../resources/topology/inputs.rs | 7 +- .../resources/topology/membership.rs | 13 +- .../fleet_catalog/resources/topology/mod.rs | 9 +- .../raft/fleet_inspection_auth_tests.rs | 3 +- .../raft/fleet_inspection_transport.rs | 4 +- .../durability/raft/fleet_inspection_wire.rs | 388 +- .../raft/fleet_inspection_wire_tests.rs | 393 +- .../durability/raft/fleet_stop/closure.rs | 3 +- .../durability/raft/fleet_stop/records.rs | 7 + .../durability/raft/fleet_stop_transport.rs | 3 + .../durability/raft/group_host/archive.rs | 101 + .../cellar/durability/raft/group_host/mod.rs | 399 ++ .../raft/group_host/registration.rs | 196 + .../durability/raft/group_host/resources.rs | 209 + .../raft/group_host/test_support.rs | 63 + .../raft/group_snapshot_transport/mod.rs | 64 +- .../group_snapshot_transport/route_tests.rs | 211 + .../group_snapshot_transport/seed/client.rs | 115 +- .../group_snapshot_transport/seed/grant.rs | 101 +- .../seed/grant_tests.rs | 77 + .../group_snapshot_transport/seed/install.rs | 54 +- .../raft/group_snapshot_transport/seed/mod.rs | 1 + crates/cellar/durability/raft/host.rs | 3735 +---------------- .../cellar/durability/raft/host_admission.rs | 126 - .../durability/raft/host_admission_tests.rs | 23 +- .../cellar/durability/raft/host_controller.rs | 421 +- .../cellar/durability/raft/host_directory.rs | 441 +- crates/cellar/durability/raft/host_effects.rs | 2 +- .../cellar/durability/raft/host_enrollment.rs | 83 - crates/cellar/durability/raft/host_extents.rs | 142 +- .../cellar/durability/raft/host_fleet_cut.rs | 174 +- .../durability/raft/host_fleet_cut_tests.rs | 161 +- .../durability/raft/host_fleet_execution.rs | 187 - .../durability/raft/host_fleet_inspection.rs | 237 -- .../durability/raft/host_fleet_observation.rs | 274 +- .../durability/raft/host_image_limits.rs | 5 +- .../raft/host_image_limits_tests.rs | 42 +- .../cellar/durability/raft/host_inspection.rs | 342 +- .../durability/raft/host_leader_load.rs | 154 +- crates/cellar/durability/raft/host_node.rs | 191 +- .../cellar/durability/raft/host_readiness.rs | 44 +- crates/cellar/durability/raft/host_seed.rs | 18 +- .../raft/host_sql/cell_groups/admission.rs | 4 + .../raft/host_sql/cell_groups/mod.rs | 8 +- .../raft/host_sql/cell_groups/query.rs | 62 +- .../durability/raft/host_startup_tests.rs | 233 +- crates/cellar/durability/raft/inspection.rs | 13 + .../durability/raft/inspection_transport.rs | 153 - crates/cellar/durability/raft/member_fleet.rs | 60 +- .../durability/raft/member_fleet_tests.rs | 56 +- crates/cellar/durability/raft/member_state.rs | 16 +- crates/cellar/durability/raft/mod.rs | 3 +- .../raft/node_inspection_transport.rs | 48 +- .../durability/raft/node_inspection_wire.rs | 2 +- .../raft/node_inspection_wire_tests.rs | 26 +- crates/cellar/durability/raft/operations.rs | 64 +- .../durability/raft/physical_inventory.rs | 2 +- crates/cellar/durability/raft/query.rs | 107 +- .../durability/raft/query_cell_transport.rs | 120 +- .../cellar/durability/raft/query_cell_wire.rs | 14 + .../cellar/durability/raft/query_identity.rs | 54 +- crates/cellar/durability/raft/query_route.rs | 194 - .../cellar/durability/raft/query_transport.rs | 293 +- crates/cellar/durability/raft/query_wire.rs | 176 +- .../durability/raft/query_wire_tests.rs | 340 +- crates/cellar/durability/raft/records.rs | 2 +- .../durability/raft/replica_leadership.rs | 78 +- .../durability/raft/restart_leadership.rs | 107 + .../raft/restart_leadership_tests.rs | 240 ++ crates/cellar/durability/raft/runtime.rs | 30 +- .../cellar/durability/raft/seed_transport.rs | 112 - crates/cellar/durability/raft/storage.rs | 127 +- .../durability/raft/storage_genesis_tests.rs | 31 +- .../raft/storage_lifecycle_tests.rs | 10 + crates/cellar/durability/raft/transport.rs | 40 +- .../cellar/durability/raft/transport_fleet.rs | 44 +- .../durability/raft/transport_query_tests.rs | 248 ++ .../durability/raft/transport_shared.rs | 210 +- .../raft/transport_shared_fixture.rs | 85 + .../cellar/durability/raft/transport_tests.rs | 231 +- .../fleet/management/raft/connection.rs | 6 +- .../fleet/management/raft/lifecycle/mod.rs | 25 +- .../management/raft/lifecycle/records.rs | 1 - .../management/raft/lifecycle/verification.rs | 161 +- crates/cellar/fleet/management/raft/mod.rs | 7 +- .../cellar/fleet/management/raft/records.rs | 1 - crates/cellar/fleet/management/raft_stop.rs | 24 +- .../fleet/management/raft_stop_tests.rs | 45 +- crates/cellar/node/lifecycle.rs | 2 +- crates/cellar/node/mod.rs | 15 +- crates/cellar/node/raft_startup.rs | 214 +- .../tests/native_cell_groups_catalog.rs | 5 + .../tests/native_cell_groups_execution.rs | 40 + .../support/cell_group_controller_archive.rs | 1 + .../tests/support/cell_group_crash_fixture.rs | 21 +- .../tests/support/cell_group_directory.rs | 112 + .../cell_group_dispatch_cache_execution.rs | 2 +- .../cellar/tests/support/cell_group_host.rs | 47 +- .../tests/support/cell_group_log_reclaim.rs | 14 +- .../tests/support/cell_group_membership.rs | 237 ++ .../support/cell_group_membership_crash.rs | 5 +- .../support/cell_group_outbox_effects.rs | 6 +- .../tests/support/cell_group_provision.rs | 93 +- .../tests/support/cell_group_query_origins.rs | 50 +- .../support/cell_group_receipt_lookup.rs | 8 +- .../support/cell_group_remote_commands.rs | 14 +- .../support/cell_group_remote_enrollment.rs | 12 +- .../support/cell_group_remote_queries.rs | 108 +- .../tests/support/cell_group_runtime.rs | 8 +- .../tests/support/cell_group_s3_commands.rs | 23 +- .../tests/support/cell_group_s3_receipts.rs | 4 +- .../tests/support/cell_group_s3_remote.rs | 4 +- .../tests/support/cell_group_seed_crash.rs | 5 +- .../support/cell_group_seed_persistence.rs | 57 +- .../tests/support/cell_group_seed_receiver.rs | 15 +- .../tests/support/cell_group_seed_retry.rs | 9 +- crates/cellar/tests/support/cell_group_sql.rs | 11 +- .../tests/support/cell_group_sql_reads.rs | 184 + .../support/cell_group_sql_registered.rs | 35 +- .../tests/support/raft_docker_binding.rs | 11 +- docs/catalog.json | 187 +- .../design/multi-raft/application-contract.md | 48 +- docs/design/multi-raft/cell-groups/README.md | 32 +- docs/design/multi-raft/cell-groups/design.md | 2 + docs/design/multi-raft/cell-groups/handoff.md | 417 +- .../cell-groups/implementation-plan.md | 2 + .../design/multi-raft/cell-groups/progress.md | 1139 +++++ .../multi-raft/cell-groups/protocols.md | 2 + .../multi-raft/cell-groups/qualification.md | 2 + .../multi-raft/cell-groups/remaining-work.md | 92 + docs/design/multi-raft/density-decision.md | 2 + docs/index.html | 2 +- examples/raft-effects/README.md | 13 +- examples/raft-effects/configuration.rs | 175 + examples/raft-effects/effects.rs | 32 +- examples/raft-effects/main.rs | 86 +- 293 files changed, 17782 insertions(+), 11451 deletions(-) delete mode 100644 crates/cellar/application/consensus/client_execution.rs create mode 100644 crates/cellar/application/consensus/group_client/actions.rs create mode 100644 crates/cellar/application/consensus/group_client/close.rs create mode 100644 crates/cellar/application/consensus/group_client/drain.rs create mode 100644 crates/cellar/application/consensus/group_client/fleet.rs create mode 100644 crates/cellar/application/consensus/group_client/learner.rs create mode 100644 crates/cellar/application/consensus/group_client/membership.rs create mode 100644 crates/cellar/application/consensus/group_client/mod.rs create mode 100644 crates/cellar/application/consensus/group_client/node.rs create mode 100644 crates/cellar/application/consensus/group_client/repair.rs create mode 100644 crates/cellar/application/consensus/group_client/resources.rs create mode 100644 crates/cellar/application/consensus/group_client/services.rs create mode 100644 crates/cellar/application/consensus/group_client/services_tests.rs create mode 100644 crates/cellar/application/consensus/group_client/startup.rs create mode 100644 crates/cellar/application/consensus/group_client/stop.rs delete mode 100644 crates/cellar/durability/raft/archive/coverage.rs create mode 100644 crates/cellar/durability/raft/catalog_inventory/groups.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/fleet.rs create mode 100644 crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs create mode 100644 crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/admission.rs create mode 100644 crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs create mode 100644 crates/cellar/durability/raft/cell_groups/directory/import.rs create mode 100644 crates/cellar/durability/raft/cell_groups/membership_observation.rs create mode 100644 crates/cellar/durability/raft/fleet_catalog/archive_replay.rs create mode 100644 crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs create mode 100644 crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs create mode 100644 crates/cellar/durability/raft/fleet_catalog/histories_tests.rs create mode 100644 crates/cellar/durability/raft/group_host/archive.rs create mode 100644 crates/cellar/durability/raft/group_host/mod.rs create mode 100644 crates/cellar/durability/raft/group_host/registration.rs create mode 100644 crates/cellar/durability/raft/group_host/resources.rs create mode 100644 crates/cellar/durability/raft/group_host/test_support.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs create mode 100644 crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs delete mode 100644 crates/cellar/durability/raft/host_fleet_execution.rs delete mode 100644 crates/cellar/durability/raft/host_fleet_inspection.rs delete mode 100644 crates/cellar/durability/raft/query_route.rs create mode 100644 crates/cellar/durability/raft/restart_leadership.rs create mode 100644 crates/cellar/durability/raft/restart_leadership_tests.rs create mode 100644 crates/cellar/durability/raft/transport_query_tests.rs create mode 100644 crates/cellar/durability/raft/transport_shared_fixture.rs create mode 100644 docs/design/multi-raft/cell-groups/remaining-work.md create mode 100644 examples/raft-effects/configuration.rs diff --git a/crates/cellar-admin/README.md b/crates/cellar-admin/README.md index 1665b6144..9b727e454 100644 --- a/crates/cellar-admin/README.md +++ b/crates/cellar-admin/README.md @@ -99,7 +99,14 @@ For an existing managed fleet, add `raft` to its private dashboard configuration "catalog_prefix": "raft/", "scope": "bucket", "managed": true, - "image_storage": "Materialized" + "image_storage": "Materialized", + "group_directory": { + "max_groups": 1000000, + "max_mutations": 1000000, + "max_cached_pages": 64, + "max_cache_bytes": 4194304, + "operation_timeout_ms": 5000 + } } ] } @@ -145,13 +152,16 @@ Graceful node departure adds independent `fleet_evacuate`, `fleet_remove` and then exclude it, prepare the complete-inventory finalization fence, explicitly request the exact process stop, and consume its signed joined closure in terminal removal. The embedding server must additionally configure -`RaftFleetControl::with_removal_verifier` with its private bounded replay store; -extent namespaces additionally require `ArchiveVerifier::with_extent_store` -with a separate exclusively owned, initially empty `NodeExtentStore`. Never -supply a live node's inventory. Each proof freshly downloads and seals the exact -remote extent closure in this scratch owner. Entered physical work retains the -owner after cancellation; a subsequent attempt reconciles scratch only after that -work exits. Physical inventory quarantine still refuses verification. +`RaftFleetControl::with_removal_verifier` with a `FleetGroupArchiveVerifier`. +Its explicit replay options supply private bounded materialized/extent inventories, +page limits and node-shared archive, checkpoint, tail and scratch pools. Never +supply a live node's inventory. Each proof downloads the complete group checkpoint +and replays its contiguous committed tail. An exact acquire intent is journaled +before S3 mutation, and restart restores its original fixed cut and allocation. +Entered physical work retains page/image ownership and memory admission after +cancellation; physical inventory quarantine still refuses verification. Persistent +holds remain retained until an explicit owner can fence users and join their work; +dropping a controller or timing out does not release remote retention. Standalone existing-root connections cannot perform live proof or dispatch stop. Status and exact committed receipt reconciliation remain metadata operations. See the [departure workflow](../../docs/design/multi-raft/fleet-operations/operator-workflows.md#evacuate-and-permanently-remove-a-node) diff --git a/crates/cellar-logic/raft_fleet/ledger.rs b/crates/cellar-logic/raft_fleet/ledger.rs index 69f765073..d41ee4b01 100644 --- a/crates/cellar-logic/raft_fleet/ledger.rs +++ b/crates/cellar-logic/raft_fleet/ledger.rs @@ -3,9 +3,8 @@ //! Conservative accounting before a membership move. Serving roles are not //! physical resources: a retired source still occupies its engine and disk. -use crate::consensus::{ - ClusterId, EngineInstanceId, GroupScope, IncarnationId, NodeIdentity, ProposalId, -}; +use crate::cell_group::GroupHistory; +use crate::consensus::{ClusterId, EngineInstanceId, IncarnationId, NodeIdentity, ProposalId}; use std::collections::{BTreeMap, BTreeSet}; const MAX_RESERVATIONS: usize = 4096; @@ -153,7 +152,9 @@ pub struct Materialization { #[derive(Clone, Debug, PartialEq, Eq)] pub struct Reservation { pub id: ProposalId, - pub group: GroupScope, + /// One physical replica, regardless of its cell count. All of its retained + /// images and staging work remain charged in peak/materialized resources. + pub group: GroupHistory, pub scope: CapacityScope, /// Increment beyond starting physical occupancy, including temporary work. pub peak: Resources, @@ -163,9 +164,6 @@ pub struct Reservation { impl Reservation { fn validate(&self, scope: CapacityScope) -> Result<(), LedgerError> { self.scope.validate()?; - self.group - .validate() - .map_err(|_| LedgerError::InvalidReservation)?; if !scope.same_engine(self.scope) || self.group.cluster != scope.cluster || self.peak == Resources::default() diff --git a/crates/cellar-logic/raft_fleet/mod.rs b/crates/cellar-logic/raft_fleet/mod.rs index 4b036c3a1..094f70c21 100644 --- a/crates/cellar-logic/raft_fleet/mod.rs +++ b/crates/cellar-logic/raft_fleet/mod.rs @@ -14,7 +14,7 @@ pub use ledger::{ CapacityScope, LedgerError, LedgerPolicy, LedgerSnapshot, Materialization, PhysicalSample, Reservation, ResourceCharge, ResourceKind, ResourceLedger, Resources, SampleAttribution, }; -pub use repair::{select_isolated_repair, IsolatedRepairView}; +pub use repair::{select_isolated_repair, GroupRepairPlacement, IsolatedRepairView}; pub use topology::{ balance_topology, preview_topology, NodeRecovery, ReplicaMove, TopologyError, TopologyGroup, TopologyNode, TopologyPeak, TopologyPolicy, TopologyPreview, TopologyTarget, TopologyView, diff --git a/crates/cellar-logic/raft_fleet/repair.rs b/crates/cellar-logic/raft_fleet/repair.rs index 53d89df37..408f3e895 100644 --- a/crates/cellar-logic/raft_fleet/repair.rs +++ b/crates/cellar-logic/raft_fleet/repair.rs @@ -3,12 +3,42 @@ //! One required replacement after the adapter proves exact-boot isolation. //! Selection grants no membership, health, reservation or release authority. use super::{ResourceLedger, Resources, TopologyError, TopologyNode, TopologyPolicy}; -use crate::consensus::{GroupScope, LogIndex, MemberId, NodeId}; -use crate::raft_placement::{PlacedMember, RepairPlacement}; +use crate::cell_group::GroupHistory; +use crate::consensus::{LogIndex, MemberId, NodeId}; +use crate::raft_placement::PlacedMember; use std::collections::BTreeSet; +/// A physical-group replacement proposal. It grants no membership or isolation +/// authority. Every cell in this history moves under the same revision fence. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupRepairPlacement { + pub scope: GroupHistory, + pub policy_revision: u64, + pub expected_revision: LogIndex, + pub obsolete: MemberId, + pub replacement: PlacedMember, +} + +// Stable load tie-break for a complete physical history. Cell identities never +// enter placement scoring; the adapter accounts their combined resource peak. +fn rank_group(history: &GroupHistory, node: NodeId) -> u64 { + let mut hash = 0xcbf29ce484222325_u64; + for bytes in [ + history.cluster.bytes(), + history.namespace.bytes(), + history.group.bytes(), + history.incarnation.bytes(), + node.bytes(), + ] { + for byte in bytes { + hash = (hash ^ u64::from(byte)).wrapping_mul(0x100000001b3); + } + } + hash +} + pub struct IsolatedRepairView { - pub scope: GroupScope, + pub scope: GroupHistory, pub revision: LogIndex, pub high_water: u64, pub voters: [PlacedMember; 3], @@ -49,7 +79,7 @@ pub fn select_isolated_repair( nodes: &[TopologyNode], policy: TopologyPolicy, now_ms: u64, -) -> Result { +) -> Result { if policy.revision == 0 || policy.max_work == 0 || policy.max_work > 50_000_000 { return Err(TopologyError::InvalidPolicy); } @@ -59,8 +89,7 @@ pub fn select_isolated_repair( if policy.max_work < nodes.len() as u64 * 2 { return Err(TopologyError::SearchExhausted); } - if view.scope.validate().is_err() - || view.revision.get() == 0 + if view.revision.get() == 0 || view.revision.get() >= u64::MAX - 2 || !label(&view.failed_host) || view.peak.engine_slots != 1 @@ -191,9 +220,8 @@ pub fn select_isolated_repair( (an * bd) .cmp(&(bn * ad)) .then_with(|| { - crate::raft_placement::rank(&view.scope, b.ledger.scope.node.node).cmp( - &crate::raft_placement::rank(&view.scope, a.ledger.scope.node.node), - ) + rank_group(&view.scope, b.ledger.scope.node.node) + .cmp(&rank_group(&view.scope, a.ledger.scope.node.node)) }) .then_with(|| a.ledger.scope.node.node.cmp(&b.ledger.scope.node.node)) }); @@ -206,8 +234,8 @@ pub fn select_isolated_repair( .iter() .find(|v| v.node == view.failed) .ok_or(TopologyError::InvalidInput)?; - Ok(RepairPlacement { - scope: view.scope.clone(), + Ok(GroupRepairPlacement { + scope: view.scope, policy_revision: policy.revision, expected_revision: view.revision, obsolete: obsolete.member, diff --git a/crates/cellar-logic/raft_fleet/topology/mod.rs b/crates/cellar-logic/raft_fleet/topology/mod.rs index dead60fb1..0bb22b63a 100644 --- a/crates/cellar-logic/raft_fleet/topology/mod.rs +++ b/crates/cellar-logic/raft_fleet/topology/mod.rs @@ -10,7 +10,8 @@ mod schedule; pub use balance::balance_topology; use super::{CapacityScope, LedgerError, LedgerSnapshot, ResourceLedger, Resources}; -use crate::consensus::{GroupScope, LogIndex, NodeId}; +use crate::cell_group::GroupHistory; +use crate::consensus::{LogIndex, NodeId}; use std::collections::{BTreeMap, BTreeSet}; #[derive(Clone, Debug)] @@ -30,7 +31,7 @@ pub struct TopologyNode { #[derive(Clone, Debug)] pub struct TopologyGroup { - pub scope: GroupScope, + pub scope: GroupHistory, /// Initial stable RF3 revision. Re-read before each actual membership permit. pub revision: LogIndex, pub voters: [NodeId; 3], @@ -44,7 +45,7 @@ pub struct TopologyGroup { #[derive(Clone, Debug, PartialEq, Eq)] pub struct TopologyTarget { - pub group: GroupScope, + pub group: GroupHistory, pub voters: [NodeId; 3], } @@ -69,7 +70,7 @@ pub struct TopologyPolicy { #[derive(Clone, Debug, PartialEq, Eq)] pub struct ReplicaMove { - pub group: GroupScope, + pub group: GroupHistory, pub source: CapacityScope, pub target: CapacityScope, /// Serial dependency: complete the preceding move's terminal proof first. @@ -214,7 +215,7 @@ fn verify(model: &Model<'_>, work: &mut Work) -> Result Model<'a> { }; model.groups.sort_by_key(|group| &group.scope); let mut histories = BTreeSet::new(); - let mut cells = BTreeSet::new(); let mut occupancy = vec![0_u64; view.nodes.len()]; for group in &model.groups { - group - .scope - .validate() - .map_err(|_| TopologyError::InvalidInput)?; if group.scope.cluster != first.cluster || group.revision.get() == 0 || group.revision.get() >= u64::MAX - 2 - || !histories.insert(group.scope.group) - || !cells.insert((group.scope.namespace, &group.scope.cell)) + || !histories.insert((group.scope.namespace, group.scope.group)) || group.resident.engine_slots != 1 || group.resident.durable_bytes == 0 || group.resident.memory_bytes == 0 diff --git a/crates/cellar-logic/raft_fleet/topology/schedule.rs b/crates/cellar-logic/raft_fleet/topology/schedule.rs index 8bf60ce90..7d1069fb5 100644 --- a/crates/cellar-logic/raft_fleet/topology/schedule.rs +++ b/crates/cellar-logic/raft_fleet/topology/schedule.rs @@ -145,7 +145,7 @@ fn materialize( peaks[a.target].high_water(add(retained[a.target], model.groups[a.group].peak)?); retained[a.target] = add(retained[a.target], model.groups[a.group].resident)?; moves.push(ReplicaMove { - group: model.groups[a.group].scope.clone(), + group: model.groups[a.group].scope, source: model.view.nodes[a.source].ledger.scope, target: model.view.nodes[a.target].ledger.scope, after: i.checked_sub(1), diff --git a/crates/cellar-logic/tests/raft_fleet_ledger.rs b/crates/cellar-logic/tests/raft_fleet_ledger.rs index 8d6406c03..4d2bdb814 100644 --- a/crates/cellar-logic/tests/raft_fleet_ledger.rs +++ b/crates/cellar-logic/tests/raft_fleet_ledger.rs @@ -3,6 +3,7 @@ //! Pure resource-accounting boundary. These tests do not qualify catalog CAS, //! physical observations, membership execution or fleet performance. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*}; fn resources(slots: u64, disk: u64, memory: u64) -> Resources { @@ -41,10 +42,10 @@ fn sample(sequence: u64, physical: Resources) -> PhysicalSample { fn reservation(id: u8, peak: Resources) -> Reservation { Reservation { id: ProposalId::new([id; 16]).unwrap(), - group: GroupScope { + group: GroupHistory { cluster: scope().cluster, namespace: NamespaceId::new([9; 16]).unwrap(), - cell: format!("Counter:{id}"), + group: GroupId::new([id; 16]).unwrap(), incarnation: IncarnationId::new([10; 16]).unwrap(), }, @@ -438,3 +439,24 @@ fn malformed_inputs_and_bounded_inventory_refuse() { Err(LedgerError::Limit) )); } + +#[test] +fn group_incarnation_cannot_reuse_a_reservation_or_erase_old_replica_debt() { + let mut ledger = ResourceLedger::restore(snapshot()).unwrap(); + let first = reservation(11, resources(1, 100, 50)); + ledger.reserve(first.clone(), 100).unwrap(); + let mut fresh = first.clone(); + fresh.group.incarnation = IncarnationId::new([79; 16]).unwrap(); + assert_eq!( + ledger.reserve(fresh.clone(), 100), + Err(LedgerError::PayloadConflict) + ); + assert_eq!(ledger.charge(100).unwrap().unmaterialized, first.peak); + fresh.id = ProposalId::new([12; 16]).unwrap(); + ledger.reserve(fresh, 100).unwrap(); + assert_eq!( + ledger.charge(100).unwrap().unmaterialized, + resources(2, 200, 100) + ); + assert_eq!(ledger.snapshot().reservations.len(), 2); +} diff --git a/crates/cellar-logic/tests/raft_fleet_repair.rs b/crates/cellar-logic/tests/raft_fleet_repair.rs index 4411c494b..793e47240 100644 --- a/crates/cellar-logic/tests/raft_fleet_repair.rs +++ b/crates/cellar-logic/tests/raft_fleet_repair.rs @@ -1,6 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Required repair selection; the host owns isolation and quorum provenance. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*, raft_placement::PlacedMember}; fn id(n: u8) -> NodeId { @@ -22,10 +23,10 @@ fn fixture() -> (IsolatedRepairView, Vec, TopologyPolicy) { node: id(n), disk: DiskLineage::new([n + 10; 16]).unwrap(), }; - let group = GroupScope { + let group = GroupHistory { cluster, namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:repair".into(), + group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; @@ -183,7 +184,7 @@ fn isolated_repair_refuses_unsettled_and_incomplete_ledgers() { let scope = n[2].ledger.scope; n[2].ledger.reservations.push(Reservation { id: ProposalId::new([6; 16]).unwrap(), - group: v.scope.clone(), + group: v.scope, scope, peak: bytes(1), materialized: None, @@ -244,9 +245,8 @@ fn isolated_repair_refuses_overflow_instead_of_saturating_capacity() { ); } -fn unrelated(node: &TopologyNode, group: &GroupScope, id: u8) -> Reservation { - let mut other = group.clone(); - other.cell = format!("Counter:unrelated-{id}"); +fn unrelated(node: &TopologyNode, group: &GroupHistory, id: u8) -> Reservation { + let mut other = *group; other.group = GroupId::new([id; 16]).unwrap(); Reservation { id: ProposalId::new([id; 16]).unwrap(), diff --git a/crates/cellar-logic/tests/raft_fleet_topology.rs b/crates/cellar-logic/tests/raft_fleet_topology.rs index 28c6a3ed1..db42fd718 100644 --- a/crates/cellar-logic/tests/raft_fleet_topology.rs +++ b/crates/cellar-logic/tests/raft_fleet_topology.rs @@ -3,8 +3,9 @@ //! Executable topology witnesses. No catalog, deployed fleet or performance //! qualification is implied by these pure decision-core cases. +use cellar_logic::cell_group::GroupHistory; use cellar_logic::{consensus::*, raft_fleet::*}; -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; fn node(n: usize) -> NodeId { NodeId::new([n as u8 + 1; 16]).unwrap() @@ -79,10 +80,10 @@ fn fixture(count: usize, triples: &[[usize; 3]]) -> (TopologyView, Vec (TopologyView, Vec Vec<[usize; 3]> { } fn replay(view: &TopologyView, targets: &[TopologyTarget], moves: &[ReplicaMove]) { - let mut voters: BTreeMap<_, _> = view - .groups - .iter() - .map(|g| (g.scope.clone(), g.voters)) - .collect(); + let mut voters: BTreeMap<_, _> = view.groups.iter().map(|g| (g.scope, g.voters)).collect(); for (i, m) in moves.iter().enumerate() { assert_eq!(m.after, i.checked_sub(1)); let set = voters.get_mut(&m.group).unwrap(); @@ -360,7 +357,7 @@ fn incomplete_stale_and_unsettled_inputs_cannot_authorize_preview() { let scope = view.nodes[3].ledger.scope; view.nodes[3].ledger.reservations.push(Reservation { id: ProposalId::new([5; 16]).unwrap(), - group: view.groups[0].scope.clone(), + group: view.groups[0].scope, scope, peak: resources(1, 10, 10), materialized: None, @@ -651,7 +648,7 @@ fn automatic_balance_requires_complete_stable_fresh_inputs() { let scope = view.nodes[3].ledger.scope; view.nodes[3].ledger.reservations.push(Reservation { id: ProposalId::new([5; 16]).unwrap(), - group: view.groups[0].scope.clone(), + group: view.groups[0].scope, scope, peak: resources(1, 10, 10), materialized: None, @@ -745,3 +742,33 @@ fn automatic_balance_all_small_two_group_layouts_preserve_rf3() { } } } + +#[test] +fn topology_refuses_two_current_incarnations_of_one_physical_group() { + let (mut view, mut targets) = fixture(4, &[[0, 1, 2], [1, 2, 3]]); + view.groups[1].scope = view.groups[0].scope; + view.groups[1].scope.incarnation = IncarnationId::new([77; 16]).unwrap(); + targets[1].group = view.groups[1].scope; + assert_eq!( + preview_topology(&view, policy(), &targets, 100), + Err(TopologyError::InvalidInput) + ); +} + +#[test] +fn topology_binds_opaque_group_ids_to_their_namespace() { + let (mut view, mut targets) = fixture(4, &[[0, 1, 2], [1, 2, 3]]); + view.groups[1].scope.group = view.groups[0].scope.group; + view.groups[1].scope.namespace = NamespaceId::new([78; 16]).unwrap(); + targets[1].group = view.groups[1].scope; + let plan = preview_topology(&view, policy(), &targets, 100).unwrap(); + assert!(plan.moves.is_empty()); + assert_eq!(plan.assignments.len(), 2); + assert_eq!( + plan.assignments + .iter() + .map(|a| a.group) + .collect::>(), + view.groups.iter().map(|g| g.scope).collect::>() + ); +} diff --git a/crates/cellar/application/consensus/class.rs b/crates/cellar/application/consensus/class.rs index 295de9339..21fa83321 100644 --- a/crates/cellar/application/consensus/class.rs +++ b/crates/cellar/application/consensus/class.rs @@ -526,6 +526,15 @@ impl RaftClass { self.initializer.is_none(), "registered initializers require the shared cell-group controller" ); + self.validate_shared_host(host) + } + + /// Shared groups support initializers. All other declared capabilities + /// retain the same execution, receipt, scheduler and archive bounds. + pub(crate) fn validate_shared_host( + &self, + host: &crate::raft::host::RaftHostOptions, + ) -> anyhow::Result<()> { if let Some(outbox) = &self.outbox { anyhow::ensure!(host.effects.is_some_and(|options|options.metadata_bytes>=512*1024) && host.max_input_bytes>=32*1024 diff --git a/crates/cellar/application/consensus/client.rs b/crates/cellar/application/consensus/client.rs index 7f4016cd5..280d2bb4d 100644 --- a/crates/cellar/application/consensus/client.rs +++ b/crates/cellar/application/consensus/client.rs @@ -23,9 +23,6 @@ use tokio::sync::{mpsc, oneshot, Notify}; #[path = "client_fleet.rs"] mod fleet; pub(crate) use fleet::FleetRequest; -#[path = "client_execution.rs"] -mod execution; -pub(crate) use execution::{LocalFleetActionCall, PeerFleetActionCall}; #[path = "client_bindings.rs"] mod bindings; @@ -33,8 +30,7 @@ mod bindings; mod node; pub(crate) use bindings::Bindings; use node::send_failure; -pub(crate) use node::{ControlRequest, NodeKind, NodeRequest}; -pub(crate) use node::{PeerFleetRequest, PeerStopRequest}; +pub(crate) use node::{ControlRequest, NodeKind}; /// A compare-and-set admission request for a preconfigured fresh learner. #[derive(Clone, Debug)] @@ -93,64 +89,6 @@ pub struct CommandReceipt { pub durability: Durability, } -#[derive(Clone, Debug, PartialEq, Eq)] -pub enum QueryConsistency { - Stale, - AtLeast(VersionToken), - Linearizable, -} - -#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] -pub enum StaleReadPolicy { - /// Refuse after the configured deadline for observing a current source. - #[default] - RecentSource, - /// Permit a verified historical image during disconnection. This supplies - /// no wall-clock staleness bound and does not strengthen its consistency. - DisconnectedHistorical, -} - -#[derive(Clone, Debug, Default)] -pub enum QueryRouting { - #[default] - Local, - /// Try at most eight distinct members in priority order, then optionally - /// one untried leader hint. Each attempt retains the same consistency, - /// token and captured input under the original invocation deadline. - Replicas { - members: Vec, - attempt_timeout: Duration, - leader_fallback: bool, - }, -} - -#[derive(Clone, Debug)] -pub struct QueryOptions { - pub consistency: QueryConsistency, - pub timeout: Duration, - pub stale_policy: StaleReadPolicy, - pub routing: QueryRouting, -} - -impl Default for QueryOptions { - fn default() -> Self { - Self { - consistency: QueryConsistency::Stale, - timeout: Duration::from_secs(5), - stale_policy: StaleReadPolicy::RecentSource, - routing: QueryRouting::Local, - } - } -} - -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct QueryResult { - pub result: Vec, - pub version: CommittedVersion, - pub version_token: VersionToken, - pub served_by: MemberId, -} - /// Discovery/readiness observation, never a grant of mutation authority. #[derive(Clone, Debug, PartialEq, Eq)] pub struct CellStatus { @@ -747,31 +685,6 @@ impl RaftCellHandle { } } - /// Queue a read immediately; the returned version names its pinned image. - /// Dropping the returned future cancels the waiter. - pub fn query( - &self, - contract: OperationContract, - input: Vec, - options: QueryOptions, - ) -> impl Future> + Send + 'static { - let timeout = options.timeout; - let call = self.call( - Kind::Query { - contract, - input, - options, - }, - timeout, - ); - async move { - match call.await? { - Response::Query(result) => Ok(result), - _ => Err(CommandFailure::CorruptState), - } - } - } - /// Queue a readiness observation immediately. Readiness may change before /// a subsequent command; the observation grants no write authority. pub fn status( @@ -897,50 +810,6 @@ impl RaftCellHandle { return Err(CommandFailure::Overloaded); } } - Kind::Query { - contract, - input, - options, - } => { - if matches!(options.routing, QueryRouting::Local) { - if self.binding.member().is_none() { - return Err(CommandFailure::UnsupportedCapability); - } - self.validate(contract, OperationKind::Query, input)?; - } else { - // A voter may forward a query whose implementation is - // only installed on another replica. Destination code - // compatibility remains mandatory before execution. - if contract.kind != OperationKind::Query - || contract.digest == [0; 32] - || super::class::validate_name(&contract.operation).is_err() - { - return Err(CommandFailure::UnsupportedCapability); - } - if input.len() > self.handle.max_input_bytes { - return Err(CommandFailure::Overloaded); - } - } - if let QueryRouting::Replicas { - members, - attempt_timeout, - .. - } = &options.routing - { - if members.is_empty() - || members.len() > 8 - || members - .iter() - .collect::>() - .len() - != members.len() - || attempt_timeout.as_millis() == 0 - || *attempt_timeout > Duration::from_secs(60) - { - return Err(CommandFailure::UnsupportedConsistency); - } - } - } Kind::Inspect => (), Kind::InspectPeer(_) => { if timeout > Duration::from_secs(60) { @@ -987,14 +856,6 @@ impl RaftCellHandle { return Err(CommandFailure::UnsupportedCapability); } } - Kind::ObserveFleet(authorization) => { - if self.binding.member().is_none() - || timeout > Duration::from_secs(60) - || !authorization.matches_scope(&self.binding.scope) - { - return Err(CommandFailure::UnsupportedCapability); - } - } Kind::AdvanceRepair { expected_revision, .. } => { @@ -1108,7 +969,6 @@ pub(crate) enum Kind { ActivateInitial(Box), Fleet(Box), FleetStatus, - ObserveFleet(Box), ActivateManaged { authorization: Box, activation: LearnerActivation, @@ -1148,24 +1008,17 @@ pub(crate) enum Kind { input: Vec, options: CommandOptions, }, - Query { - contract: OperationContract, - input: Vec, - options: QueryOptions, - }, Status, } impl Kind { fn is_control(&self) -> bool { - !matches!(self, Self::Command { .. } | Self::Query { .. }) + !matches!(self, Self::Command { .. }) } fn bytes(&self) -> usize { 4096 + match self { - Self::Command { input, .. } - | Self::SystemCommand { input, .. } - | Self::Query { input, .. } => input.len(), + Self::Command { input, .. } | Self::SystemCommand { input, .. } => input.len(), Self::Inspect | Self::InspectPeer(_) | Self::Status @@ -1181,7 +1034,6 @@ impl Kind { | Self::ActivateInitial(_) | Self::Fleet(_) | Self::FleetStatus - | Self::ObserveFleet(_) | Self::RepairStatus => 0, } } @@ -1191,7 +1043,7 @@ impl Kind { Self::Command { options, .. } | Self::SystemCommand { options, .. } => { CancelFailure::Command(options.id) } - Self::Query { .. } | Self::Seed { .. } => CancelFailure::Query, + Self::Seed { .. } => CancelFailure::Seed, Self::Inspect | Self::InspectPeer(_) | Self::Status @@ -1206,7 +1058,6 @@ impl Kind { | Self::ActivateInitial(_) | Self::Fleet(_) | Self::FleetStatus - | Self::ObserveFleet(_) | Self::RepairStatus => CancelFailure::Status, } } @@ -1214,7 +1065,7 @@ impl Kind { enum CancelFailure { Command(CommandId), - Query, + Seed, Status, } @@ -1223,30 +1074,23 @@ impl CancelFailure { let submitted = control.cancel(); match self { Self::Command(id) if submitted => CommandFailure::Ambiguous { command: *id }, - Self::Query => CommandFailure::VersionUnavailable, + Self::Seed => CommandFailure::VersionUnavailable, _ => CommandFailure::Unavailable, } } } pub(crate) enum Response { - FleetExecutionAction(Result<(), crate::raft::fleet_catalog::FleetCatalogError>), PeerFleetStop( Result< Box, crate::raft::fleet_catalog::FleetCatalogError, >, ), - PeerFleetObservation( - Result, super::InspectionError>, - ), - LocalFleetObservation(Box), PeerNodeInspection(Result, super::InspectionError>), PhysicalNode(Box), - HistoryRegistered, Node(Box), Fleet(Option>), - FleetObservation(Box), Drained(super::RaftDrainReport), Controller(super::RaftControllerStatus), PeerInspection( @@ -1259,7 +1103,6 @@ pub(crate) enum Response { Seed(Box), Learner(LearnerReceipt), Command(CommandReceipt), - Query(QueryResult), Status(CellStatus), } diff --git a/crates/cellar/application/consensus/client_execution.rs b/crates/cellar/application/consensus/client_execution.rs deleted file mode 100644 index 450d1dffc..000000000 --- a/crates/cellar/application/consensus/client_execution.rs +++ /dev/null @@ -1,122 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Internal fleet actions preserve one original bounded waiter. -use super::*; -use crate::raft::fleet_catalog::{FleetCatalogError, FleetExecutionCommand, MovementCommand}; -use crate::raft::fleet_stop::BootBinding; - -pub(crate) struct PeerFleetActionCall { - pub command: FleetExecutionCommand, - pub minimum_revision: u64, -} - -pub(crate) struct LocalFleetActionCall { - pub command: FleetExecutionCommand, - pub origin: BootBinding, - pub minimum_revision: u64, -} - -impl RaftCells { - pub(crate) async fn start_restart_worker( - &self, - work: crate::raft::fleet_catalog::RestartServiceWork, - timeout: Duration, - ) -> Result<(), FleetCatalogError> { - match self - .node_call(NodeKind::StartRestartService(Box::new(work)), timeout) - .await - { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - } - - pub(crate) async fn start_movement_worker( - &self, - work: crate::raft::fleet_catalog::MovementServiceWork, - timeout: Duration, - ) -> Result<(), FleetCatalogError> { - match self - .node_call(NodeKind::StartMovementService(Box::new(work)), timeout) - .await - { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - } - - pub(crate) fn request_peer_movement( - &self, - command: MovementCommand, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - self.request_peer_fleet_action( - FleetExecutionCommand::Movement(command), - minimum_revision, - timeout, - ) - } - - pub(crate) fn request_peer_fleet_action( - &self, - command: FleetExecutionCommand, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - if command.validate().is_err() { - return Box::pin(async { Err(FleetCatalogError::Invalid("fleet execution command")) }); - } - self.fleet_action_call( - NodeKind::PeerFleetAction(Box::new(PeerFleetActionCall { - command, - minimum_revision, - })), - minimum_revision, - timeout, - ) - } - - pub(crate) fn execute_local_fleet_action( - &self, - command: FleetExecutionCommand, - origin: BootBinding, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - self.fleet_action_call( - NodeKind::LocalFleetAction(Box::new(LocalFleetActionCall { - command, - origin, - minimum_revision, - })), - minimum_revision, - timeout, - ) - } - - fn fleet_action_call( - &self, - kind: NodeKind, - minimum_revision: u64, - timeout: Duration, - ) -> Pin> + Send>> { - if minimum_revision == 0 - || minimum_revision >= u64::MAX - 2 - || timeout.as_millis() == 0 - || timeout > Duration::from_secs(60) - { - return Box::pin(async { Err(FleetCatalogError::Invalid("fleet action bounds")) }); - } - let call = self.node_call(kind, timeout); - Box::pin(async move { - match call.await { - Ok(Response::FleetExecutionAction(result)) => result, - Err(CommandFailure::Overloaded) => Err(FleetCatalogError::Overloaded), - _ => Err(FleetCatalogError::Unconfirmed), - } - }) - } -} diff --git a/crates/cellar/application/consensus/client_fleet.rs b/crates/cellar/application/consensus/client_fleet.rs index ed0b8c0b1..ea79a37c1 100644 --- a/crates/cellar/application/consensus/client_fleet.rs +++ b/crates/cellar/application/consensus/client_fleet.rs @@ -1,7 +1,6 @@ //! Managed configuration calls retain opaque catalog authorizations. use super::*; use crate::raft::fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}; -use crate::raft::host::FleetMembershipObservation; use crate::raft::member_state::{FleetMembershipStatus, RepairPlan, RepairStatus, RepairStep}; pub(crate) enum FleetRequest { @@ -48,27 +47,6 @@ impl FleetRequest { } impl RaftCellHandle { - /// Read the enrolled configuration after a new quorum ReadIndex barrier. - /// The result is one observation during this invocation, not a lease or a - /// certificate to release reservations or remove a node. Cached local fleet - /// status, an old observation and a successful enrollment retry cannot - /// substitute for this call's barrier. No application query or image is - /// required. The original deadline also bounds barrier retries. - pub fn observe_fleet_membership( - &self, - authorization: FleetGroupAuthorization, - timeout: Duration, - ) -> impl Future> + Send + 'static - { - let call = self.call(Kind::ObserveFleet(Box::new(authorization)), timeout); - async move { - match call.await? { - Response::FleetObservation(observation) => Ok(*observation), - _ => Err(CommandFailure::CorruptState), - } - } - } - /// Initialize an exact reserved initial voter inside this existing host. /// Enrollment and genesis share their first sync before attachment. The /// receipt is local durable state, never a live-quorum/write-ack proof. diff --git a/crates/cellar/application/consensus/client_node.rs b/crates/cellar/application/consensus/client_node.rs index abd6289a8..c14743be3 100644 --- a/crates/cellar/application/consensus/client_node.rs +++ b/crates/cellar/application/consensus/client_node.rs @@ -3,10 +3,6 @@ use super::*; pub(crate) enum NodeKind { - StartRestartService(Box), - StartMovementService(Box), - PeerFleetAction(Box), - LocalFleetAction(Box), Drain, Observe, ObservePhysical, @@ -15,23 +11,11 @@ pub(crate) enum NodeKind { minimum_revision: u64, physical: bool, }, - InspectPeerFleet(Box), PeerFleetStop(Box), - ObserveFleet { - history: Box, - minimum_revision: u64, - }, - RegisterHistory(Box), ControllerStatus, SetControllerEnabled(bool), } -pub(crate) struct PeerFleetRequest { - pub node: cellar_logic::consensus::NodeId, - pub minimum_revision: u64, - pub authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, -} - pub(crate) struct PeerStopRequest { pub request: crate::raft::fleet_stop::FleetStopRequest, pub minimum_revision: u64, @@ -117,99 +101,6 @@ impl RaftCells { }) } - /// Obtain a fresh safe-ReadIndex observation from an enrolled group. A local - /// target uses the same guarded quorum observer without a loopback RPC. - /// The origin may have zero local groups. Both endpoints must install the - /// requested catalog cut; the target independently authorizes this exact - /// history. This creates no query, seeding or membership-mutation grant. - /// Retaining a response does not establish current absence or removal proof. - pub fn observe_peer_fleet_membership( - &self, - node: cellar_logic::consensus::NodeId, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - minimum_revision: u64, - timeout: Duration, - ) -> Pin< - Box< - dyn Future< - Output = Result< - super::super::RaftPeerFleetMembershipObservation, - super::super::InspectionError, - >, - > + Send, - >, - > { - use super::super::InspectionError; - if minimum_revision == 0 - || minimum_revision >= u64::MAX - 2 - || timeout.is_zero() - || timeout > Duration::from_secs(60) - { - return Box::pin(async { Err(InspectionError::Refused) }); - } - let domain = self.handle.domain.clone(); - let started = match domain.mono_ms() { - Ok(started) => started, - Err(_) => return Box::pin(async { Err(InspectionError::Unavailable) }), - }; - let call = self.node_call( - NodeKind::InspectPeerFleet(Box::new(PeerFleetRequest { - node, - minimum_revision, - authorization, - })), - timeout, - ); - Box::pin(async move { - let response = call.await.map_err(|error| match error { - CommandFailure::Overloaded => InspectionError::Overloaded, - CommandFailure::UnsupportedCapability => InspectionError::Refused, - _ => InspectionError::Unavailable, - })?; - let Response::PeerFleetObservation(result) = response else { - return Err(InspectionError::Unavailable); - }; - let mut observed = result?; - let age = domain - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(started)) - .ok_or(InspectionError::Unavailable)?; - observed.record_delivery_age(age)?; - Ok(*observed) - }) - } - - pub(crate) fn observe_local_fleet_membership( - &self, - history: crate::raft::fleet_inspection_wire::History, - minimum_revision: u64, - timeout: Duration, - ) -> Pin< - Box< - dyn Future< - Output = Result< - Box, - CommandFailure, - >, - > + Send, - >, - > { - let call = self.node_call( - NodeKind::ObserveFleet { - history: Box::new(history), - minimum_revision, - }, - timeout, - ); - Box::pin(async move { - match call.await? { - Response::LocalFleetObservation(result) => Ok(result), - _ => Err(CommandFailure::CorruptState), - } - }) - } - /// Inspect an enrolled process without a cell. Both endpoints must /// acknowledge at least this trust revision; stale or missing metadata is /// not evidence of node failure. No placement or membership is activated. @@ -334,38 +225,6 @@ impl RaftCells { }) } - /// Register dormant metadata for an already enrolled history under this - /// host's declared class. This creates no member, image or voting authority. - /// A lost response can race publication; retry the same authorization. - pub fn register_managed_history( - &self, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - timeout: Duration, - ) -> Pin> + Send>> { - if timeout > Duration::from_secs(60) { - return Box::pin(async { Err(CommandFailure::UnsupportedCapability) }); - } - let scope = authorization - .binding() - .scope() - .map_err(|_| CommandFailure::CommandConflict); - let response = self.node_call(NodeKind::RegisterHistory(Box::new(authorization)), timeout); - let host = self.clone(); - Box::pin(async move { - let scope = scope?; - match response.await? { - Response::HistoryRegistered => { - let cell = host.cell(scope.namespace, &scope.cell)?; - if cell.binding.scope != scope { - return Err(CommandFailure::CommandConflict); - } - Ok(cell) - } - _ => Err(CommandFailure::CorruptState), - } - }) - } - /// Point-in-time local physical inventory with startup and installed current /// enrollment. This grants no capacity activation, quorum or removal proof. pub fn observe_node( diff --git a/crates/cellar/application/consensus/group_client/actions.rs b/crates/cellar/application/consensus/group_client/actions.rs new file mode 100644 index 000000000..65241f263 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/actions.rs @@ -0,0 +1,297 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical-group fleet controls share the installed node's real runtime and +//! authenticated transport. Lost responses do not establish mutation rollback. +use super::*; +use crate::raft::{ + completion::{ReplicationStatus, RestartLeadershipCheck, RuntimeError}, + fleet_catalog::{FleetCatalogError, FleetExecutionCommand, MovementCommand}, + fleet_stop::BootBinding, + transport::fleet_actions::Delivery, +}; +use std::pin::Pin; + +const ACTION_BYTES: usize = 128 * 1024; + +struct Local { + command: FleetExecutionCommand, + origin: BootBinding, +} + +enum Dispatch { + Local(Box), + Remote(Delivery), +} + +impl RaftCells { + pub(crate) fn request_peer_movement( + &self, + command: MovementCommand, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + self.request_peer_fleet_action( + FleetExecutionCommand::Movement(command), + minimum_revision, + timeout, + ) + } + + pub(crate) fn request_peer_fleet_action( + &self, + command: FleetExecutionCommand, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + // Admission and the original deadline precede first polling. The + // remote transport also starts its owned request at invocation. + let admitted: Result<_, FleetCatalogError> = (|| { + command.validate()?; + let (deadline, memory) = self.action_admission(minimum_revision, timeout)?; + let dispatch = if command.target().node == self.handle.core.node.node.bytes() { + let origin = self.action_boot(command.target())?; + Dispatch::Local(Box::new(Local { command, origin })) + } else { + Dispatch::Remote(self.handle.core.transport.request_fleet_action( + command, + minimum_revision, + deadline, + )?) + }; + Ok((deadline, memory, dispatch)) + })(); + let owner = self.internal_observer(); + Box::pin(async move { + let (deadline, _memory, dispatch) = admitted?; + owner.action_live(deadline)?; + let result = match dispatch { + Dispatch::Local(local) => { + owner + .local_action_at(local.command, local.origin, minimum_revision, deadline) + .await + } + Dispatch::Remote(delivery) => delivery.receive().await, + }; + // Expiry/close after dispatch is uncertain even when the command + // may already have reached its serialized owner. + owner + .action_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + result + }) + } + + pub(crate) fn execute_local_fleet_action( + &self, + command: FleetExecutionCommand, + origin: BootBinding, + minimum_revision: u64, + timeout: Duration, + ) -> Pin> + Send>> { + let admitted: Result<_, FleetCatalogError> = (|| { + command.validate()?; + origin + .validate() + .map_err(|_| FleetCatalogError::Invalid("fleet action origin"))?; + self.action_boot(command.target())?; + self.action_admission(minimum_revision, timeout) + })(); + let owner = self.internal_observer(); + Box::pin(async move { + let (deadline, _memory) = admitted?; + owner + .local_action_at(command, origin, minimum_revision, deadline) + .await + }) + } + + fn action_admission( + &self, + minimum_revision: u64, + timeout: Duration, + ) -> Result<(u64, crate::raft::budget::Permit), FleetCatalogError> { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + if self.handle.core.managed.is_none() { + return Err(FleetCatalogError::Invalid("no managed fleet action owner")); + } + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout.is_zero() + || timeout > Duration::from_secs(60) + { + return Err(FleetCatalogError::Invalid("fleet action bounds")); + } + self.metadata_admission(timeout, ACTION_BYTES) + .map_err(|error| match error { + cellar_logic::consensus::CommandFailure::Overloaded => { + FleetCatalogError::Overloaded + } + cellar_logic::consensus::CommandFailure::UnsupportedCapability => { + FleetCatalogError::Invalid("fleet action callback context") + } + _ => FleetCatalogError::Unconfirmed, + }) + } + + fn action_live(&self, deadline: u64) -> Result<(), FleetCatalogError> { + self.metadata_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed) + } + + fn action_boot(&self, target: &BootBinding) -> Result { + let core = &self.handle.core; + let guard = core + .log + .lifecycle_guard() + .map_err(|_| FleetCatalogError::Closed)? + .ok_or(FleetCatalogError::Closed)?; + let installed = core + .managed + .as_ref() + .ok_or(FleetCatalogError::Invalid("no managed fleet receiver"))?; + if &guard.boot != target + || installed.boot_binding()? != guard.boot + || target.node != core.node.node.bytes() + || target.disk != core.node.disk.bytes() + || target.process != core.node.process.bytes() + || target.engine != core.engine.bytes() + || !guard.admits_group_start(target) && guard.quiescing().is_none() + { + return Err(FleetCatalogError::Conflict); + } + Ok(guard.boot) + } + + async fn local_action_at( + &self, + command: FleetExecutionCommand, + origin: BootBinding, + minimum_revision: u64, + deadline: u64, + ) -> Result<(), FleetCatalogError> { + self.action_live(deadline)?; + let target = command.target().clone(); + self.action_boot(&target)?; + let host = self + .handle + .core + .managed + .as_ref() + .ok_or(FleetCatalogError::Closed)?; + let remaining = deadline + .checked_sub( + self.handle + .domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)?, + ) + .filter(|ms| *ms > 0) + .map(Duration::from_millis) + .ok_or(FleetCatalogError::Unconfirmed)?; + let timer = self + .handle + .domain + .sleep_until(deadline) + .map_err(|_| FleetCatalogError::Closed)?; + let work = self.handle.domain.run(host.execute_fleet_command( + command, + origin, + minimum_revision, + self, + remaining, + )); + let result = crate::asyncrt::select! { + _ = timer => Err(FleetCatalogError::Unconfirmed), + value = work => value.unwrap_or(Err(FleetCatalogError::Unconfirmed)), + }; + self.action_live(deadline) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + self.action_boot(&target) + .map_err(|_| FleetCatalogError::Unconfirmed)?; + result + } +} + +impl RaftGroupHandle { + /// Leader-local metadata, without a quorum, applied image or health lease. + pub fn replication_status( + &self, + timeout: Duration, + ) -> impl Future, RuntimeError>> + Send + 'static + { + self.replication_status_for(timeout) + } + + pub(crate) fn replication_status_for( + &self, + timeout: Duration, + ) -> impl Future, RuntimeError>> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?; + group.control_live(deadline)?; + let result = super::drain::within( + &group.owner.handle.domain, + deadline, + group + .owner + .handle + .core + .runtime + .replication_status(group.history), + ) + .await?; + group.control_live(deadline)?; + Ok(result) + } + } + + /// An exact restart observation or transfer, rechecked by the real group + /// owner after queuing. Acceptance never proves election completion. + pub(crate) fn restart_leadership( + &self, + mut check: RestartLeadershipCheck, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let deadline = self.control_deadline(timeout); + let group = self.clone(); + async move { + let deadline = deadline?.min(check.deadline); + check.deadline = deadline; + group.control_live(deadline)?; + super::drain::within( + &group.owner.handle.domain, + deadline, + group + .owner + .handle + .core + .runtime + .restart_leadership(group.history, check), + ) + .await?; + group.control_live(deadline) + } + } + + fn control_live(&self, deadline: u64) -> Result<(), RuntimeError> { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if self + .owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + Ok(()) + } +} diff --git a/crates/cellar/application/consensus/group_client/close.rs b/crates/cellar/application/consensus/group_client/close.rs new file mode 100644 index 000000000..ee6c483ba --- /dev/null +++ b/crates/cellar/application/consensus/group_client/close.rs @@ -0,0 +1,227 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Joined closure records complete physical group counts. A sampled empty queue +//! is accepted only behind sealed admission and idle physical SQL ownership. +use super::*; +use std::sync::atomic::{AtomicBool, Ordering}; +use tokio::sync::Notify; + +/// Local producer quiescence does not prove that callers received outcomes, +/// archive coverage is complete, or old deployments can be restarted/deleted. +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum MutationOutcomes { + Unobserved, + NoUnresolvedLocalProducers, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RaftGroupCloseReport { + pub history: GroupHistory, + pub member: MemberId, + pub committed: cellar_logic::consensus::LogIndex, + pub applied: cellar_logic::consensus::LogPosition, + /// None is unavailable coverage, never a measured zero-lag archive. + pub archived: Option, + /// Activation metadata in the final synced manifest, not a live quorum or + /// serving permission after shutdown. + pub active: bool, + pub failed: bool, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RaftCloseReport { + pub node: NodeIdentity, + pub storage_instance: EngineInstanceId, + pub groups: Vec, + pub groups_total: usize, + pub omitted_groups: usize, + pub groups_with_archive_lag: usize, + pub groups_with_unknown_archive: usize, + pub retained_calls_at_shutdown: usize, + pub mutation_outcomes: MutationOutcomes, +} + +#[derive(Clone, Debug)] +pub struct RaftCloseError { + pub controller: Option>, + pub log: Option>, + pub inventory: Option>, +} + +impl std::fmt::Display for RaftCloseError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + write!(f, "shared Raft closure: {self:?}") + } +} + +impl std::error::Error for RaftCloseError {} + +type Closed = Result>; + +pub(super) struct Close { + pub closing: AtomicBool, + requested: Notify, + terminal: Mutex>, + finished: Notify, + admission: Arc, +} + +impl Close { + pub fn new(admission: Arc) -> Arc { + Arc::new(Self { + closing: AtomicBool::new(false), + requested: Notify::new(), + terminal: Mutex::new(None), + finished: Notify::new(), + admission, + }) + } + + pub fn request(&self) { + self.closing.store(true, Ordering::Release); + let _ = self.admission.seal(); + self.requested.notify_one(); + } + + pub async fn wait(&self) -> Closed { + loop { + let notified = self.finished.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + if let Some(result) = self + .terminal + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clone() + { + return result; + } + notified.await; + } + } +} + +pub(super) fn start( + core: Arc, + close: Arc, + domain: crate::asyncrt::DomainToken, +) -> Result<(), CellGroupControllerError> { + domain + .spawn_cleanup(async move { + let requested = close.requested.notified(); + tokio::pin!(requested); + requested.as_mut().enable(); + if !close.closing.load(Ordering::Acquire) { + crate::asyncrt::select! { _ = requested => (), _ = core.controller.wait() => () } + } + close.request(); + core.services.close().await; + core.gateways + .lock() + .unwrap_or_else(|e| e.into_inner()) + .clear(); + let usage = core.controller.usage(); + // Acquire no new writer after sealing. Accepted physical work cannot + // disappear from these counters when its response times out or drops. + let quiesced = !close.admission.is_poisoned() + && core.controller.local_producers_quiesced() + && core.resources.sql.usage().is_ok_and(|sql| { + sql.active_cells == 0 + && sql.retained_operations == 0 + && sql.queued_or_running == 0 + }); + let controller = core.host.close().await.err().map(Arc::new); + // Read after persistence/SQL/runtime joins, before closing the engine. + // Include every durable group even if its detail row exceeds the cap. + let report = capture(&core, usage.retained_commands, quiesced); + let log = core.log.close().await.err().map(Arc::new); + let result = match (controller, log, report) { + (None, None, Ok(report)) => Ok(report), + (controller, log, report) => Err(Arc::new(RaftCloseError { + controller, + log, + inventory: report.err().map(Arc::new), + })), + }; + *close.terminal.lock().unwrap_or_else(|e| e.into_inner()) = Some(result); + close.finished.notify_waiters(); + }) + .map_err(|_| CellGroupControllerError::Closed)? + .detach(); + Ok(()) +} + +fn capture( + core: &Core, + retained_calls: usize, + quiesced: bool, +) -> Result { + let groups = core.log.cell_groups()?; + let mut report = RaftCloseReport { + node: core.node, + storage_instance: core.engine, + groups: Vec::new(), + groups_total: groups.len(), + omitted_groups: 0, + groups_with_archive_lag: 0, + groups_with_unknown_archive: 0, + retained_calls_at_shutdown: retained_calls, + mutation_outcomes: if quiesced { + MutationOutcomes::NoUnresolvedLocalProducers + } else { + MutationOutcomes::Unobserved + }, + }; + // Fixed-size domain values only. Charge generously for vector/allocation + // overhead; counts remain complete even when all detail rows are omitted. + let details = core + .close_limits + .max_groups + .min(core.close_limits.max_bytes / 512); + for group in groups { + let history = group.identity().scope; + let published = core + .controller + .archive_status(history) + .ok() + .flatten() + .and_then(|status| status.published); + let failed = core + .controller + .group_failure(history) + .map_or(true, |failure| failure.is_some()); + if failed { + report.mutation_outcomes = MutationOutcomes::Unobserved; + } + let cache = group.read()?; + let marker = cache + .group_manifest + .as_ref() + .ok_or(LogError::Corrupt("closed group manifest missing".into()))?; + let archived = published + .filter(|published| published.route() == marker.route()) + .map(|published| published.position()); + match archived { + None => report.groups_with_unknown_archive += 1, + Some(position) if position.index.get() < cache.hard_state.commit => { + report.groups_with_archive_lag += 1 + } + Some(_) => (), + } + if report.groups.len() < details { + report.groups.push(RaftGroupCloseReport { + history, + member: group.identity().member, + committed: cellar_logic::consensus::LogIndex::new(cache.hard_state.commit), + applied: marker.position(), + archived, + active: marker.is_active(), + failed, + }); + } else { + report.omitted_groups += 1; + } + } + Ok(report) +} diff --git a/crates/cellar/application/consensus/group_client/drain.rs b/crates/cellar/application/consensus/group_client/drain.rs new file mode 100644 index 000000000..2ed78cf8a --- /dev/null +++ b/crates/cellar/application/consensus/group_client/drain.rs @@ -0,0 +1,144 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One bounded turn per physical group. The actual Raft owner rechecks term, +//! membership revision, recent activity and catch-up before admitting transfer. +use super::*; +use crate::raft::{ + completion::{LeadershipTransfer, RuntimeError}, + operations::RaftDrainReport, +}; +use cellar_logic::consensus::CommandFailure; + +impl RaftCells { + pub(crate) async fn drain_node( + &self, + deadline: u64, + ) -> Result { + self.begin_node_drain(); + let core = &self.handle.core; + let domain = &self.handle.domain; + let groups = core + .log + .cell_groups() + .map_err(|_| CommandFailure::Unavailable)?; + let mut report = RaftDrainReport::default(); + for group in groups { + let now = domain.mono_ms().map_err(|_| CommandFailure::Unavailable)?; + if now >= deadline { + report.deadline_elapsed = true; + break; + } + report.groups_examined += 1; + let through = deadline.min(now.saturating_add(1000)); + let history = group.identity().scope; + let local = group.identity().member.get(); + let mut turn = Turn::default(); + let outcome = within(domain, through, async { + loop { + let status = core.runtime.status(history).await?; + if status.leader != Some(local) { + return Ok(()); + } + turn.needed = true; + if status.ready && status.applied.index == status.committed { + break; + } + pause(domain, through).await?; + } + let progress = core.runtime.replication_status(history).await?; + for member in progress + .members + .iter() + .filter(|p| { + p.voter + && p.member.get() != local + && p.recent_active + && p.matched >= progress.committed + }) + .take(8) + { + turn.attempts += 1; + match core + .runtime + .transfer_leadership( + history, + LeadershipTransfer { + target: member.member, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }, + ) + .await + { + Ok(()) => { + // Admission remains a separate observation if this + // deadline expires before a successor is visible. + turn.accepted = true; + turn.released = within(domain, through, async { + loop { + let status = core.runtime.status(history).await?; + if status.term >= progress.term + && status.leader.is_some_and(|id| id != local) + { + return Ok(()); + } + pause(domain, through).await?; + } + }) + .await + .is_ok(); + return Ok(()); + } + Err(RuntimeError::NotReady) => (), + Err(error) => return Err(error), + } + } + Ok(()) + }) + .await; + report.handoff_attempts += turn.attempts; + report.handoffs_accepted += usize::from(turn.accepted); + report.leadership_released += usize::from(turn.released); + if outcome.is_err() || turn.needed && !turn.accepted { + report.unavailable_groups += 1; + } + } + report.deadline_elapsed |= + domain.mono_ms().map_err(|_| CommandFailure::Unavailable)? >= deadline; + Ok(report) + } +} + +#[derive(Default)] +struct Turn { + needed: bool, + attempts: usize, + accepted: bool, + released: bool, +} + +pub(super) async fn within( + domain: &crate::asyncrt::DomainToken, + deadline: u64, + future: impl Future>, +) -> Result { + if domain.mono_ms().map_err(|_| RuntimeError::Closed)? >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + let timer = domain + .sleep_until(deadline) + .map_err(|_| RuntimeError::Closed)?; + crate::asyncrt::select! { result = future => result, _ = timer => Err(RuntimeError::DeadlineExceeded) } +} + +async fn pause(domain: &crate::asyncrt::DomainToken, deadline: u64) -> Result<(), RuntimeError> { + let now = domain.mono_ms().map_err(|_| RuntimeError::Closed)?; + if now >= deadline { + return Err(RuntimeError::DeadlineExceeded); + } + domain + .sleep_until(deadline.min(now.saturating_add(25))) + .map_err(|_| RuntimeError::Closed)? + .await; + Ok(()) +} diff --git a/crates/cellar/application/consensus/group_client/fleet.rs b/crates/cellar/application/consensus/group_client/fleet.rs new file mode 100644 index 000000000..570a2be96 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/fleet.rs @@ -0,0 +1,195 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fleet reads are metadata-only and require the exact rooted group admission. +//! The receiver independently reauthorizes it; node transport trust is not enough. +use super::*; +use crate::raft::{ + fleet_catalog::FleetGroupAuthorization, + fleet_inspection_wire::{ + GroupSnapshot, History, LocalObservation, RaftPeerFleetMembershipObservation, + RESPONSE_BYTES, + }, + inspection::InspectionError, +}; +use cellar_logic::consensus::{CommandFailure, NodeId}; + +impl RaftCells { + /// Fresh complete-group membership/publication from an exact enrolled boot. + /// Followers use safe ReadIndex forwarding and their own durable apply; + /// this grants no leader, seed, mutation, archive or physical-release right. + pub fn observe_peer_fleet_membership( + &self, + node: NodeId, + authorization: FleetGroupAuthorization, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || self.handle.core.managed.is_none() + { + return Err(InspectionError::Refused); + } + let local = node == self.handle.core.node.node; + let (deadline, memory) = self + .metadata_admission( + timeout, + if local { + crate::raft::physical_inventory::RESPONSE_BYTES + } else { + RESPONSE_BYTES + }, + ) + .map_err(inspection_error)?; + let started = deadline + .checked_sub(timeout.as_millis() as u64) + .ok_or(InspectionError::Unavailable)?; + let remote = if local { + None + } else { + Some(self.handle.core.transport.inspect_fleet_membership( + node, + &authorization, + minimum_revision, + deadline, + )?) + }; + Ok((deadline, started, memory, remote)) + })(); + let owner = self.internal_observer(); + async move { + let (deadline, started, _memory, remote) = admitted?; + owner.metadata_live(deadline).map_err(inspection_error)?; + let local = if let Some(delivery) = remote { + delivery.receive().await?.observation + } else { + owner + .local_fleet_at(History::new(&authorization), minimum_revision, deadline) + .await + .map_err(inspection_error)? + }; + owner.metadata_live(deadline).map_err(inspection_error)?; + owner + .handle + .core + .transport + .validate_fleet_node_observation(&local.node, minimum_revision)?; + let age = owner + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_sub(started)) + .ok_or(InspectionError::Unavailable)?; + local + .into_peer(&authorization, age) + .map_err(|_| InspectionError::Refused) + } + } + + pub(crate) fn observe_local_fleet_membership( + &self, + history: History, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let admitted = + self.metadata_admission(timeout, crate::raft::physical_inventory::RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _memory) = admitted?; + owner + .local_fleet_at(history, minimum_revision, deadline) + .await + } + } + + async fn local_fleet_at( + &self, + history: History, + minimum_revision: u64, + deadline: u64, + ) -> Result { + self.metadata_live(deadline)?; + if minimum_revision == 0 || minimum_revision >= u64::MAX - 2 { + return Err(CommandFailure::CommandConflict); + } + let scope = history + .scope() + .map_err(|_| CommandFailure::CommandConflict)?; + let host = self + .handle + .core + .managed + .as_ref() + .ok_or(CommandFailure::UnsupportedCapability)?; + let remaining = deadline + .checked_sub( + self.handle + .domain + .mono_ms() + .map_err(|_| CommandFailure::Unavailable)?, + ) + .filter(|n| *n > 0) + .map(Duration::from_millis) + .ok_or(CommandFailure::VersionUnavailable)?; + let authorization = host + .group_authorization_at_least(&scope, minimum_revision, remaining) + .await + .map_err(|error| match error { + crate::raft::fleet_catalog::FleetCatalogError::Overloaded + | crate::raft::fleet_catalog::FleetCatalogError::Limit => { + CommandFailure::Overloaded + } + _ => CommandFailure::Unavailable, + })?; + if !history.matches(&authorization) { + return Err(CommandFailure::CommandConflict); + } + let selected = self + .handle + .core + .runtime + .quorum_membership(scope, deadline) + .await + .map_err(|error| match error { + crate::raft::completion::RuntimeError::Overloaded => CommandFailure::Overloaded, + _ => CommandFailure::Unavailable, + })?; + let snapshot = GroupSnapshot::capture(&selected); + // This descriptor contains no image/page pins. Release the temporary + // metadata read reservation before sampling unrelated physical work. + drop(selected); + let (node, _) = self.capture_physical_node()?; + self.handle + .core + .transport + .validate_fleet_node_observation(&node, minimum_revision) + .map_err(|_| CommandFailure::Unavailable)?; + self.metadata_live(deadline)?; + let local = LocalObservation { + node: Box::new(node), + snapshot, + }; + local.validate().map_err(|_| CommandFailure::CorruptState)?; + // The same constructor checks binding, immutable birth/schema and the + // actual local replica before anything reaches the transport encoder. + crate::raft::host::FleetMembershipObservation::check_quorum_group( + &local.snapshot, + &authorization, + )?; + Ok(local) + } +} + +fn inspection_error(error: CommandFailure) -> InspectionError { + match error { + CommandFailure::Overloaded => InspectionError::Overloaded, + CommandFailure::CommandConflict | CommandFailure::UnsupportedCapability => { + InspectionError::Refused + } + _ => InspectionError::Unavailable, + } +} diff --git a/crates/cellar/application/consensus/group_client/learner.rs b/crates/cellar/application/consensus/group_client/learner.rs new file mode 100644 index 000000000..8a367276e --- /dev/null +++ b/crates/cellar/application/consensus/group_client/learner.rs @@ -0,0 +1,204 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed learner startup belongs to the node's retained provisioning owner. +use super::startup::{admission_error, catalog_error}; +use super::*; +use crate::application::consensus::client::LearnerActivation; +use crate::raft::{ + catalog::CatalogError, + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::{FleetMembershipStatus, FleetPermitPhase}, +}; + +impl RaftCells { + /// Install and attach one catalog-selected physical learner. Success is a + /// local durable receipt; it cannot promote the learner or grant serving. + /// Once submitted, cancellation retains preparation and joins entered I/O. + pub fn activate_managed_learner_group( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + activation: LearnerActivation, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + let (deadline, memory) = self + .metadata_admission(activation.timeout, 4 * 1024 * 1024) + .map_err(admission_error)?; + let host = self + .handle + .core + .managed + .clone() + .ok_or(CellGroupControllerError::Invalid("managed learner owner"))?; + let history = group + .binding() + .scope() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + if group.binding() != authorization.permit().binding() + || group.binding().fleet() != host.fleet() + || activation.from_s3 + { + return Err(CellGroupControllerError::Invalid( + "managed learner capability", + )); + } + let directory = + self.handle + .core + .host + .directory(history) + .map_err(|error| match error { + CellDispatchError::Runtime(error) => error.into(), + _ => CellGroupControllerError::Invalid("learner namespace capability"), + })?; + Ok((deadline, memory, host, history, directory)) + })(); + let owner = self.internal_observer(); + async move { + use CellGroupStartupError::{Refused, Unconfirmed}; + let (deadline, _memory, host, history, directory) = admitted.map_err(Refused)?; + let mut submitted = false; + let work = async { + owner.startup_live(&host, deadline).map_err(Refused)?; + let birth = directory + .materialize_selected_history(history) + .await + .map_err(catalog_error) + .map_err(Refused)?; + if !group.check_birth(&directory, &birth) { + return Err(Refused(CellGroupControllerError::Invalid("learner birth"))); + } + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Refused)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + if fresh.binding() != group.binding() || !fresh.check_birth(&directory, &birth) { + return Err(Refused(CellGroupControllerError::Invalid( + "learner history changed", + ))); + } + owner.startup_live(&host, deadline).map_err(Refused)?; + let member = activation.member; + let call = owner + .handle + .core + .controller + .provision_managed_learner( + directory.clone(), + birth.route().clone(), + host.clone(), + authorization.clone(), + activation, + deadline, + ) + .map_err(Refused)?; + submitted = true; + call.response() + .await + .map_err(|e| Unconfirmed((*e).clone()))?; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Unconfirmed)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Unconfirmed)?; + if fresh.binding() != group.binding() || !fresh.check_birth(&directory, &birth) { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "learner history changed", + ))); + } + let core = &owner.handle.core; + if core + .controller + .group_failure(history) + .map_err(CellGroupControllerError::from) + .map_err(Unconfirmed)? + .is_some() + { + return Err(Unconfirmed(RuntimeError::GroupQuarantined.into())); + } + let local = core + .log + .cell_group(&history) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let status = { + let cache = local + .read() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let joined = cache + .membership + .joined(member) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let status = cache.membership.fleet_status().ok_or(Unconfirmed( + CellGroupControllerError::Invalid("learner fleet guard"), + ))?; + if local.identity().member != member + || !fresh.matches_replica(local.identity(), &cache.membership) + || cache + .group_manifest + .as_ref() + .is_none_or(|m| m.route() != birth.route() || !m.is_active()) + || joined.is_none_or(|(binding, at)| { + binding.node != core.node.node + || binding.disk != core.node.disk + || cache.group_seed.is_none_or(|seed| seed < at.get()) + }) + || status.permit.as_ref() != Some(authorization.permit()) + || &status.binding != authorization.permit().binding() + || status.consumed_sequence != authorization.permit().sequence() + || !matches!( + status.phase, + Some( + FleetPermitPhase::Seeding + | FleetPermitPhase::Joint + | FleetPermitPhase::Complete + ) + ) + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "learner installed receipt", + ))); + } + status + }; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + Ok(status) + }; + match owner + .handle + .core + .transport + .seed_control_until(deadline, async { + owner + .handle + .domain + .run(work) + .await + .map_err(|_| crate::raft::transport::TransportError::Closed) + }) + .await + { + Ok(result) => result, + Err(error) => { + let error = CellGroupControllerError::Transport(error); + Err(if submitted { + Unconfirmed(error) + } else { + Refused(error) + }) + } + } + } + } +} diff --git a/crates/cellar/application/consensus/group_client/membership.rs b/crates/cellar/application/consensus/group_client/membership.rs new file mode 100644 index 000000000..9ba21971b --- /dev/null +++ b/crates/cellar/application/consensus/group_client/membership.rs @@ -0,0 +1,301 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Administrative commands address a physical group. Durable apply and a fresh +//! quorum barrier, rather than proposal admission, establish their result. +use super::*; +use crate::raft::{ + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::FleetPermitPhase, +}; +use cellar_logic::consensus::LogIndex; + +/// Refused means this invocation submitted no configuration change. It does not +/// resolve an earlier lost response. Unconfirmed may follow actual commitment; +/// reconcile the same authorization instead of inventing another operation. +#[derive(Clone, Debug)] +pub enum CellGroupControlError { + Refused(RuntimeError), + Unconfirmed(RuntimeError), +} + +impl std::fmt::Display for CellGroupControlError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Refused(error) => write!(f, "group control refused: {error}"), + Self::Unconfirmed(error) => write!(f, "group control outcome unconfirmed: {error}"), + } + } +} + +impl std::error::Error for CellGroupControlError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Refused(error) | Self::Unconfirmed(error) => error, + }) + } +} + +pub(super) enum Change { + Enroll { + authorization: FleetGroupAuthorization, + expected: LogIndex, + }, + Fence(FleetPermitAuthorization), + Repair(super::repair::Change), +} + +impl Change { + fn satisfied(&self, observed: &CellGroupQuorumMembership) -> Result { + let status = observed.membership().fleet_status(); + match self { + Self::Enroll { authorization, .. } => { + if authorization.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !authorization.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid("fleet enrollment authority".into())); + } + let Some(status) = status else { + return Ok(false); + }; + if &status.binding != authorization.binding() { + return Err(RuntimeError::Invalid( + "group enrolled in another fleet".into(), + )); + } + Ok(true) + } + Self::Fence(authorization) => { + let status = + status.ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + let permit = authorization.permit(); + if &status.binding != permit.binding() { + return Err(RuntimeError::Invalid("foreign group fence".into())); + } + if status.consumed_sequence < permit.sequence() { + return Ok(false); + } + if status.consumed_sequence == permit.sequence() + && status.permit.as_ref() == Some(permit) + && status.phase == Some(FleetPermitPhase::Fenced) + { + return Ok(true); + } + // A later sequence or another consumed phase cannot prove that + // this exact cancellation fence committed, nor release its debt. + Err(RuntimeError::StaleCompletion) + } + Self::Repair(change) => change.satisfied(observed), + } + } + + fn prepare( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let route = observed.root().route(); + let membership = observed.membership(); + let configuration = observed.configuration(); + Ok(match self { + Self::Enroll { + authorization, + expected, + } => { + if membership.revision() != *expected { + return Err(RuntimeError::StaleCompletion); + } + CellGroupMembershipChange::fleet_enrollment( + route, + membership, + configuration, + authorization, + )? + } + Self::Fence(authorization) => CellGroupMembershipChange::fleet_fence( + route, + membership, + configuration, + authorization, + )?, + Self::Repair(change) => change.prepare(observed)?, + }) + } +} + +impl RaftGroupHandle { + pub(super) fn control_deadline(&self, timeout: Duration) -> Result { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if timeout.is_zero() || timeout > Duration::from_secs(60) { + return Err(RuntimeError::Invalid("group control timeout".into())); + } + self.owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(RuntimeError::Overloaded) + } + + /// A fresh quorum observation behind a durable physical apply floor. Time + /// before polling consumes the original invocation deadline. + pub fn quorum_membership_after( + &self, + minimum: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.owner.clone(); + let history = self.history; + async move { + owner + .handle + .core + .runtime + .quorum_membership_after(history, minimum, deadline?) + .await + } + } + + /// The exact rooted fleet binding must already be committed by this group. + /// The result contains no cell alias, SQL image or physical-release lease. + pub fn observe_fleet_membership( + &self, + authorization: FleetGroupAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.owner.clone(); + let history = self.history; + async move { + if !authorization.matches_scope(&history) { + return Err(RuntimeError::Invalid("fleet observation history".into())); + } + let observed = owner + .handle + .core + .runtime + .quorum_membership(history, deadline?) + .await?; + let status = observed + .membership() + .fleet_status() + .ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + if &status.binding != authorization.binding() + || authorization.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !authorization.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid("fleet observation authority".into())); + } + Ok(observed) + } + } + + /// Commit this physical history's exact fleet guard. An exact retry can + /// return a fresh observation of the already committed guard. + pub fn enroll_fleet( + &self, + authorization: FleetGroupAuthorization, + expected: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.clone(); + async move { + if !authorization.matches_scope(&owner.history) { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "fleet enrollment history".into(), + ))); + } + owner + .change_membership( + Change::Enroll { + authorization, + expected, + }, + deadline.map_err(CellGroupControlError::Refused)?, + ) + .await + } + } + + /// Commit the exact unused permit's cancellation fence. Admission alone + /// never supplies a fence receipt or permits reservation release. + pub fn fence_fleet_permit( + &self, + authorization: FleetPermitAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let deadline = self.control_deadline(timeout); + let owner = self.clone(); + async move { + if !authorization.matches_scope(&owner.history) { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "fleet fence history".into(), + ))); + } + owner + .change_membership( + Change::Fence(authorization), + deadline.map_err(CellGroupControlError::Refused)?, + ) + .await + } + } + + pub(super) async fn change_membership( + &self, + change: Change, + deadline: u64, + ) -> Result { + use CellGroupControlError::{Refused, Unconfirmed}; + let core = &self.owner.handle.core; + let observed = core + .runtime + .quorum_membership(self.history, deadline) + .await + .map_err(Refused)?; + if change.satisfied(&observed).map_err(Refused)? { + return Ok(observed); + } + if let Change::Repair(repair) = &change { + repair + .check_placement(&core.transport, &observed) + .map_err(Refused)?; + } + let proposal = change.prepare(&observed).map_err(Refused)?; + drop(observed); + // Runtime owns an accepted configuration independently of this waiter. + // A timed-out send/response is conservatively uncertain, never aborted. + let admitted = super::drain::within( + &self.owner.handle.domain, + deadline, + core.runtime.configure(self.history, proposal), + ) + .await + .map_err(Unconfirmed)?; + let observed = core + .runtime + .quorum_membership_after(self.history, admitted.position.index, deadline) + .await + .map_err(Unconfirmed)?; + if !change.satisfied(&observed).map_err(Unconfirmed)? { + return Err(Unconfirmed(RuntimeError::StaleCompletion)); + } + Ok(observed) + } +} diff --git a/crates/cellar/application/consensus/group_client/mod.rs b/crates/cellar/application/consensus/group_client/mod.rs new file mode 100644 index 000000000..00a3e02ec --- /dev/null +++ b/crates/cellar/application/consensus/group_client/mod.rs @@ -0,0 +1,562 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical SDK handles bind logical cell histories. Physical group controls +//! are separate values; no cell handle owns a group, worker or permanent map entry. +use super::{CommandId, Durability, OperationContract}; +use crate::raft::{ + cell_groups::*, + storage::{LogError, ReplicaLog}, + version::{CellVersionObservation, CellVersionToken}, +}; +use cellar_logic::{ + cell_group::{CellHistory, CellVersion, CommitWitness, GroupHistory}, + consensus::{EngineInstanceId, MemberId, NodeIdentity}, +}; +use std::sync::atomic::Ordering; +use std::{ + future::Future, + sync::{Arc, Mutex}, + time::Duration, +}; + +pub use super::client::CommandOptions; + +#[derive(Clone, Debug)] +pub enum QueryConsistency { + Stale, + AtLeast(CellVersionToken), + Linearizable, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum StaleReadPolicy { + RecentSource, + DisconnectedHistorical, +} + +#[derive(Clone, Debug)] +pub struct QueryOptions { + pub consistency: QueryConsistency, + pub stale_policy: StaleReadPolicy, + pub timeout: Duration, + pub max_attempts: u8, +} + +impl Default for QueryOptions { + fn default() -> Self { + Self { + consistency: QueryConsistency::Stale, + stale_policy: StaleReadPolicy::RecentSource, + timeout: Duration::from_secs(5), + max_attempts: 4, + } + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CommandReceipt { + pub id: CommandId, + pub result: Vec, + /// Selected logical cell version covering this exact stored receipt. + pub version: CellVersion, + pub witness: CommitWitness, + pub version_token: CellVersionToken, + pub durability: Durability, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct QueryResult { + pub result: Vec, + pub version: CellVersion, + pub witness: CommitWitness, + pub version_token: CellVersionToken, + pub served_by: MemberId, + pub served_by_node: NodeIdentity, +} + +mod actions; +mod close; +mod drain; +mod fleet; +mod learner; +mod membership; +mod node; +mod repair; +mod services; +mod startup; +mod stop; +pub use membership::CellGroupControlError; +pub(crate) use services::{ControlServiceCompletion, ControlServiceKind, ControlServiceLease}; +pub use startup::CellGroupStartupError; +pub(crate) mod resources; +use close::Close; +pub use close::{MutationOutcomes, RaftCloseError, RaftCloseReport, RaftGroupCloseReport}; +pub use resources::{CellGroupImageUsage, CellGroupResourceUsage}; + +struct Core { + host: CellGroupHost, + controller: CellGroupController, + runtime: CellGroupRuntime, + log: ReplicaLog, + node: NodeIdentity, + engine: EngineInstanceId, + gateways: Mutex>, + resources: resources::Resources, + close_limits: crate::raft::operations::RaftCloseReportLimits, + services: Arc, + managed: Option, + transport: crate::raft::transport::RaftTransport, + node_profile: Option, + startup_bounds: crate::raft::host::RaftHostStartupBounds, + stale_source_timeout_ms: u64, +} + +pub(crate) struct Handle { + pub(crate) domain: crate::asyncrt::DomainToken, + pub(crate) admission: Arc, + core: Arc, + close: Arc, +} + +impl Drop for Handle { + fn drop(&mut self) { + self.close.request(); + } +} + +struct ExternalLifetime { + close: Arc, +} + +impl Drop for ExternalLifetime { + fn drop(&mut self) { + self.close.request(); + } +} + +pub struct RaftCells { + pub(crate) handle: Arc, + // Internal service observers retain metadata without keeping the last + // external SDK owner alive. They cannot indefinitely self-own this node. + lifetime: Option>, +} + +impl Clone for RaftCells { + fn clone(&self) -> Self { + Self { + handle: self.handle.clone(), + lifetime: self.lifetime.clone(), + } + } +} + +impl RaftCells { + pub(crate) fn from_shared( + host: CellGroupHost, + controller: CellGroupController, + runtime: CellGroupRuntime, + log: ReplicaLog, + admission: Arc, + resources: resources::Resources, + transport: crate::raft::transport::RaftTransport, + options: &crate::raft::host::RaftHostOptions, + ) -> Result { + let domain = log.domain(); + let node = NodeIdentity { + node: log.limits().node, + disk: log.limits().disk, + process: log.limits().process, + }; + let engine = log + .storage_instance() + .map_err(|_| CellGroupControllerError::Invalid("SDK engine identity"))?; + controller.bind_public_admission( + admission.clone(), + crate::asyncrt::services().application_work_registry(), + )?; + let close = Close::new(admission.clone()); + let startup_bounds = node::startup_bounds(&controller, &runtime, &log, options)?; + let core = Arc::new(Core { + host, + controller, + runtime, + log, + node, + engine, + gateways: Mutex::new(Vec::new()), + resources, + close_limits: options.close_report, + services: services::Services::new( + options.max_control_bytes, + options.max_control_requests, + ), + managed: options + .managed + .as_ref() + .map(|managed| managed.authorization.clone()), + transport, + node_profile: options.node_profile.clone(), + startup_bounds, + stale_source_timeout_ms: options.stale_source_timeout_ms, + }); + let service = Self { + handle: Arc::new(Handle { + domain: domain.clone(), + admission, + core: core.clone(), + close: close.clone(), + }), + lifetime: Some(Arc::new(ExternalLifetime { + close: close.clone(), + })), + }; + close::start(core, close, domain)?; + service + .handle + .core + .transport + .install_group_fleet_gateway(&service) + .map_err(|_| CellGroupControllerError::Invalid("fleet metadata gateway"))?; + Ok(service) + } + + pub(crate) fn from_internal_handle(handle: Arc) -> Self { + Self { + handle, + lifetime: None, + } + } + + pub(crate) fn uses_transport(&self, transport: &crate::raft::transport::RaftTransport) -> bool { + transport.uses_shared_runtime(&self.handle.core.runtime) + } + + pub(crate) fn retain_gateways( + &self, + query: CellQueryGateway, + command: CellCommandGateway, + ) -> Result<(), CellGroupControllerError> { + self.handle + .core + .gateways + .lock() + .map_err(|_| CellGroupControllerError::Closed)? + .push((query, command)); + Ok(()) + } + + pub fn cell(&self, history: CellHistory) -> Result { + self.check_open()?; + let cell = self.handle.core.host.cell(history)?; + Ok(RaftCellHandle { + owner: self.clone(), + cell, + }) + } + + /// Administrative identity is an entire physical group, never a cell alias. + pub fn group( + &self, + history: GroupHistory, + ) -> Result { + self.handle + .core + .log + .cell_group(&history) + .map_err(|_| crate::raft::completion::RuntimeError::NotFound)?; + Ok(RaftGroupHandle { + owner: self.clone(), + history, + }) + } + + fn check_open(&self) -> Result<(), CellDispatchError> { + if self.is_closed() { + return Err(CellDispatchError::Runtime( + crate::raft::completion::RuntimeError::Closed, + )); + } + Ok(()) + } + + pub fn is_closed(&self) -> bool { + self.handle.close.closing.load(Ordering::Acquire) || self.handle.core.host.is_closed() + } + + pub fn resource_usage(&self) -> CellGroupResourceUsage { + self.handle.core.resources.capture(&self.handle.core) + } + + pub fn performance(&self) -> super::RaftPerformanceSnapshot { + self.handle.core.log.performance_snapshot() + } + + pub(crate) fn begin_node_drain(&self) { + let _ = self.handle.admission.seal(); + } + + /// Request joined closure before returning the waiter. Cancellation cannot + /// abandon the physical close or turn unknown mutations into failed writes. + pub fn close( + &self, + ) -> impl Future>> + Send + 'static { + self.handle.close.request(); + let close = self.handle.close.clone(); + async move { close.wait().await } + } +} + +#[derive(Clone)] +pub struct RaftCellHandle { + owner: RaftCells, + cell: CellGroupCellHandle, +} + +/// Effect routing cannot keep its own host alive. Recreate a disposable logical +/// handle from the same immutable namespace registry while that owner exists. +#[derive(Clone)] +pub(crate) struct WeakCellHandle { + owner: std::sync::Weak, + history: CellHistory, +} + +impl WeakCellHandle { + pub(crate) fn upgrade(&self) -> Result, CellDispatchError> { + let Some(handle) = self.owner.upgrade() else { + return Ok(None); + }; + RaftCells { + handle, + lifetime: None, + } + .cell(self.history.clone()) + .map(Some) + } +} + +impl RaftCellHandle { + pub(crate) fn downgrade(&self) -> WeakCellHandle { + WeakCellHandle { + owner: Arc::downgrade(&self.owner.handle), + history: self.history().clone(), + } + } + + pub(crate) fn definition_digest(&self) -> [u8; 32] { + self.cell.definition_digest() + } + + pub fn history(&self) -> &CellHistory { + self.cell.history() + } + + pub fn contract(&self, operation: &str) -> Result { + self.cell.contract(operation) + } + + pub fn command( + &self, + contract: &OperationContract, + input: Vec, + options: CommandOptions, + ) -> impl Future>> + Send + 'static { + let id = options.id; + let accepted = self + .owner + .check_open() + .map_err(|cause| CellCommandError { + command: id, + ambiguous: false, + cause: Box::new(cause), + }) + .and_then(|()| { + self.cell.command( + CellCommandRequest { + id, + contract: contract.clone(), + input, + durability: options.durability, + retry_until_ms: options.retry_until_ms, + }, + CellRoutingLimits { + timeout: options.timeout, + max_attempts: 4, + }, + ) + }); + let owner = self.owner.clone(); + async move { + let _owner = owner; + let response = accepted.map_err(Arc::new)?.response().await?; + let CellVersionObservation { version, witness } = response.receipt().observation(); + Ok(CommandReceipt { + id, + result: response.receipt().receipt.result().to_vec(), + version, + witness, + version_token: response.token().clone(), + durability: response.durability(), + }) + } + } + + pub fn query( + &self, + invocation: super::ProposalId, + contract: &OperationContract, + input: Vec, + options: QueryOptions, + ) -> impl Future>> + Send + 'static { + let accepted = self.owner.check_open().and_then(|()| { + if !matches!(options.consistency, QueryConsistency::Stale) + && options.stale_policy != StaleReadPolicy::RecentSource + { + return Err(CellDispatchError::Invalid( + "stale policy requires stale consistency", + )); + } + let consistency = match options.consistency { + QueryConsistency::Stale => match options.stale_policy { + StaleReadPolicy::RecentSource => CellQueryConsistency::RecentSource { + max_age_ms: self.owner.handle.core.stale_source_timeout_ms, + }, + StaleReadPolicy::DisconnectedHistorical => CellQueryConsistency::Stale, + }, + QueryConsistency::AtLeast(token) => CellQueryConsistency::AtLeast(token), + QueryConsistency::Linearizable => CellQueryConsistency::Linearizable, + }; + self.cell.query( + CellQueryRequest { + invocation, + contract: contract.clone(), + input, + consistency, + }, + CellRoutingLimits { + timeout: options.timeout, + max_attempts: options.max_attempts, + }, + ) + }); + let owner = self.owner.clone(); + async move { + let _owner = owner; + let response = accepted.map_err(Arc::new)?.response().await?; + let CellVersionObservation { version, witness } = response.observation().clone(); + Ok(QueryResult { + result: response.result().to_vec(), + version, + witness, + version_token: response.token().clone(), + served_by: response.replica().member, + served_by_node: response.replica().node_identity(), + }) + } + } + + pub fn enroll( + &self, + request: CellEnrollmentRequest, + timeout: Duration, + ) -> impl Future>> + Send + 'static + { + let response = self.cell.enroll( + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + ); + let owner = self.owner.clone(); + async move { + let _owner = owner; + response.await + } + } + + pub fn receipt( + &self, + request: CellReceiptRequest, + timeout: Duration, + ) -> impl Future>> + Send + 'static + { + let response = self.cell.receipt( + request, + CellRoutingLimits { + timeout, + max_attempts: 4, + }, + ); + let owner = self.owner.clone(); + async move { + let _owner = owner; + response.await + } + } +} + +#[derive(Clone)] +pub struct RaftGroupHandle { + owner: RaftCells, + history: GroupHistory, +} + +impl RaftGroupHandle { + pub fn history(&self) -> GroupHistory { + self.history + } + + pub async fn status(&self) -> Result { + self.owner.handle.core.runtime.status(self.history).await + } + + /// A fresh quorum observation of the complete physical configuration. The + /// deadline starts at invocation, including time before this future polls. + /// The returned metadata is not a lease or permission to release fleet debt. + pub fn quorum_membership( + &self, + timeout: Duration, + ) -> impl Future> + + Send + + 'static { + use crate::raft::completion::RuntimeError; + let deadline = (|| { + if self.owner.is_closed() { + return Err(RuntimeError::Closed); + } + if timeout.is_zero() || timeout > Duration::from_secs(60) { + return Err(RuntimeError::Invalid("group observation timeout".into())); + } + self.owner + .handle + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(RuntimeError::Overloaded) + })(); + let owner = self.owner.clone(); + let history = self.history; + async move { + owner + .handle + .core + .runtime + .quorum_membership(history, deadline?) + .await + } + } + + /// Admission only; observe subsequent election status independently. + pub async fn transfer_leader( + &self, + member: MemberId, + ) -> Result<(), crate::raft::completion::RuntimeError> { + self.owner + .handle + .core + .runtime + .transfer_leader(self.history, member.get()) + .await + } +} diff --git a/crates/cellar/application/consensus/group_client/node.rs b/crates/cellar/application/consensus/group_client/node.rs new file mode 100644 index 000000000..5d8c1818e --- /dev/null +++ b/crates/cellar/application/consensus/group_client/node.rs @@ -0,0 +1,367 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical metadata uses the shared engine/roster, independently of cell count. +//! Installed transport trust, not a separate root refresh, acknowledges routing. +use super::*; +use crate::raft::{ + host::{ + RaftHostEnrollmentObservation, RaftHostObservation, RaftHostStartupBounds, + RaftPeerNodeObservation, + }, + inspection::{InspectionError, NodeInspection, RESPONSE_RESERVATION}, + physical_inventory::{PhysicalNodeObservation, RESPONSE_BYTES}, +}; +use cellar_logic::consensus::{CommandFailure, NodeId}; + +impl RaftCells { + /// A point-in-time local physical cut. No quorum, capacity activation, + /// physical absence or release permission follows from retaining it. + pub fn observe_node( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static { + let admitted = self.metadata_admission(timeout, RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _bytes) = admitted?; + owner.metadata_live(deadline)?; + let (view, _) = owner.capture_physical_node()?; + owner.metadata_live(deadline)?; + Ok(view) + } + } + + pub(crate) fn observe_physical_node( + &self, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self.metadata_admission(timeout, RESPONSE_BYTES); + let owner = self.internal_observer(); + async move { + let (deadline, _bytes) = admitted?; + owner.metadata_live(deadline)?; + let observed = owner.capture_physical_node()?; + owner.metadata_live(deadline)?; + Ok(observed) + } + } + + /// Inspect the exact enrolled process over the authenticated control lane. + /// A local target uses the same installed-trust checks without loopback I/O. + pub fn inspect_peer_node( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.inspect_peer_node_mode(node, minimum_revision, timeout, false) + } + + /// Include every durable engine slot, even if unattached or quarantined. + /// An omitted inventory is never interpreted as an empty physical node. + pub fn inspect_peer_physical_inventory( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.inspect_peer_node_mode(node, minimum_revision, timeout, true) + } + + fn inspect_peer_node_mode( + &self, + node: NodeId, + minimum_revision: u64, + timeout: Duration, + physical: bool, + ) -> impl Future> + Send + 'static + { + // Starting the remote transport at invocation preserves its original + // deadline even if the response future is retained without polling. + let admitted = (|| { + if minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout > Duration::from_secs(60) + || self.handle.core.managed.is_none() + { + return Err(InspectionError::Refused); + } + let (deadline, memory) = self + .metadata_admission( + timeout, + if physical || node == self.handle.core.node.node { + RESPONSE_BYTES + } else { + RESPONSE_RESERVATION + }, + ) + .map_err(inspection_error)?; + let started = deadline + .checked_sub(timeout.as_millis() as u64) + .ok_or(InspectionError::Unavailable)?; + let remote = if node != self.handle.core.node.node { + Some(self.handle.core.transport.inspect_fleet_node( + node, + minimum_revision, + deadline, + physical, + )?) + } else { + None + }; + Ok((deadline, started, memory, remote)) + })(); + let owner = self.internal_observer(); + async move { + let (deadline, started, _memory, remote) = admitted?; + owner.metadata_live(deadline).map_err(inspection_error)?; + let (view, inventory) = if let Some(delivery) = remote { + let received = delivery.receive().await?; + if received.inventory.is_some() != physical { + return Err(InspectionError::Refused); + } + (*received.view, received.inventory) + } else { + let (view, inventory) = owner.capture_physical_node().map_err(inspection_error)?; + (view, physical.then(|| Box::new(inventory))) + }; + owner.metadata_live(deadline).map_err(inspection_error)?; + owner + .handle + .core + .transport + .validate_fleet_node_observation(&view, minimum_revision)?; + let sample_age_ms = owner + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_sub(started)) + .ok_or(InspectionError::Unavailable)?; + Ok(RaftPeerNodeObservation { + view, + inventory, + sample_age_ms, + }) + } + } + + pub(super) fn metadata_admission( + &self, + timeout: Duration, + bytes: usize, + ) -> Result<(u64, crate::raft::budget::Permit), CommandFailure> { + if crate::application::consensus::class::callback_active() { + return Err(CommandFailure::UnsupportedCapability); + } + let _gate = self + .handle + .admission + .registration(super::super::admission::Access::Cleanup)?; + if self.is_closed() { + return Err(CommandFailure::Unavailable); + } + if timeout.as_millis() == 0 || timeout > Duration::from_secs(60) { + return Err(CommandFailure::VersionUnavailable); + } + let deadline = self + .handle + .domain + .mono_ms() + .ok() + .and_then(|now| now.checked_add(timeout.as_millis() as u64)) + .ok_or(CommandFailure::Unavailable)?; + let memory = self + .control_budget() + .acquire(bytes) + .ok_or(CommandFailure::Overloaded)?; + Ok((deadline, memory)) + } + + pub(super) fn metadata_live(&self, deadline: u64) -> Result<(), CommandFailure> { + if self.is_closed() { + return Err(CommandFailure::Unavailable); + } + if !self.handle.domain.mono_ms().is_ok_and(|now| now < deadline) { + return Err(CommandFailure::VersionUnavailable); + } + Ok(()) + } + + pub(super) fn capture_physical_node(&self) -> Result { + let core = &self.handle.core; + let roster = core + .controller + .registration_inventory() + .map_err(|_| CommandFailure::Unavailable)?; + let inventory = core + .log + .cell_group_inventory(None) + .map_err(|_| CommandFailure::Unavailable)?; + if roster + .0 + .iter() + .any(|history| inventory.group_history(history).is_none()) + { + return Err(CommandFailure::Unavailable); + } + let usage = core.controller.usage(); + let mut profile = core + .node_profile + .clone() + .ok_or(CommandFailure::UnsupportedCapability)?; + profile.draining |= self.handle.admission.application_paused(); + let count = |n| u32::try_from(n).map_err(|_| CommandFailure::Overloaded); + let groups = count(inventory.histories().len())?; + let reserved_bytes = profile + .bytes_per_group + .checked_mul(u64::from(groups)) + .ok_or(CommandFailure::Overloaded)?; + let current = core + .managed + .as_ref() + .map(|startup| { + let current = core + .transport + .installed_fleet_node(core.node.node) + .map_err(|_| CommandFailure::Unavailable)?; + let boot = current + .boot_binding() + .map_err(|_| CommandFailure::CommandConflict)?; + let expected = startup + .boot_binding() + .map_err(|_| CommandFailure::CommandConflict)?; + if boot != expected + || boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || current.disk() != core.node.disk + || current.storage_instance() != Some(core.engine) + { + return Err(CommandFailure::CommandConflict); + } + Ok(RaftHostEnrollmentObservation::capture(¤t)) + }) + .transpose()?; + let transport = core + .transport + .resource_observer() + .capture() + .ok_or(CommandFailure::Unavailable)?; + let seed_workers = transport + .snapshot_source + .items + .checked_add(usage.active_checkpoints) + .and_then(|n| n.checked_add(usage.retained_preparations)) + .and_then(|n| n.checked_add(core.runtime.usage().read_pins.items)) + .ok_or(CommandFailure::Overloaded)?; + let view = RaftHostObservation { + node: NodeInspection { + identity: core.node, + storage_instance: core.engine, + profile, + groups, + reserved_bytes, + }, + fleet_boot: core + .log + .lifecycle_guard() + .map_err(|_| CommandFailure::Unavailable)? + .map(|guard| guard.boot), + startup_bounds: core.startup_bounds.clone(), + startup_enrollment: core + .managed + .as_ref() + .map(RaftHostEnrollmentObservation::capture), + fleet_trust_revision: current.as_ref().map(|n| n.revision), + current_enrollment: current, + declared_namespaces: count(core.host.usage().namespaces)?, + registered_histories: count(roster.0.len())?, + dormant_histories: count(roster.2)?, + pending_history_registrations: count( + roster + .1 + .checked_add(usage.provision_waiters) + .ok_or(CommandFailure::Overloaded)?, + )?, + unattached_engine_histories: count( + inventory + .histories() + .len() + .checked_sub(roster.0.len()) + .ok_or(CommandFailure::CorruptState)?, + )?, + seed_workers: count(seed_workers)?, + }; + // The catalog is append-only for this engine. Refuse a changed slot + // count or roster rather than combine unrelated physical cuts. + if core + .log + .group_count() + .map_err(|_| CommandFailure::Unavailable)? + != inventory.histories().len() + || core + .controller + .registration_inventory() + .map_err(|_| CommandFailure::Unavailable)? + != roster + { + return Err(CommandFailure::Unavailable); + } + if core.managed.is_some() { + inventory + .validate(&view) + .map_err(|_| CommandFailure::CorruptState)?; + } + Ok((view, inventory)) + } +} + +pub(super) fn startup_bounds( + controller: &CellGroupController, + runtime: &CellGroupRuntime, + log: &ReplicaLog, + options: &crate::raft::host::RaftHostOptions, +) -> Result { + let c = controller.limits(); + let r = runtime.limits(); + let l = log.limits(); + let bytes = [ + c.max_command_bytes, + c.max_command_bytes, + options.max_control_bytes, + r.max_request_bytes, + r.max_event_bytes, + r.max_ready_bytes, + r.max_apply_bytes, + r.max_read_pin_bytes, + l.max_pending_bytes, + l.max_cache_bytes, + ]; + let bounded_buffer_bytes = bytes + .into_iter() + .try_fold(0u64, |sum, n| sum.checked_add(u64::try_from(n).ok()?)) + .ok_or(CellGroupControllerError::Invalid("node buffer capacities"))?; + let count = + |n| u32::try_from(n).map_err(|_| CellGroupControllerError::Invalid("node group capacity")); + Ok(RaftHostStartupBounds { + host_groups: count(c.max_groups)?, + consensus_groups: count(r.max_groups)?, + engine_groups: count(l.max_groups)?, + bounded_buffer_bytes, + }) +} + +fn inspection_error(error: CommandFailure) -> InspectionError { + match error { + CommandFailure::Overloaded => InspectionError::Overloaded, + CommandFailure::UnsupportedCapability | CommandFailure::CommandConflict => { + InspectionError::Refused + } + _ => InspectionError::Unavailable, + } +} diff --git a/crates/cellar/application/consensus/group_client/repair.rs b/crates/cellar/application/consensus/group_client/repair.rs new file mode 100644 index 000000000..229257348 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/repair.rs @@ -0,0 +1,274 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Whole-group repair controls use exact permits and fresh post-apply quorum. +use super::membership::{CellGroupControlError, Change as MembershipChange}; +use super::*; +use crate::raft::{ + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetPermitAuthorization}, + member_state::{FleetPermitPhase, RepairPhase, RepairPlan, RepairStep}, + transport::{RaftTransport, TransportError}, +}; +use cellar_logic::consensus::{CommandFailure, LogIndex}; + +pub(super) enum Change { + Start { + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + plan: RepairPlan, + }, + Advance { + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + step: RepairStep, + expected: LogIndex, + }, +} + +impl Change { + fn authority(&self) -> (&FleetGroupAuthorization, &FleetPermitAuthorization) { + match self { + Self::Start { + group, + authorization, + .. + } + | Self::Advance { + group, + authorization, + .. + } => (group, authorization), + } + } + + pub(super) fn satisfied( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let (group, authorization) = self.authority(); + let permit = authorization.permit(); + if group.binding() != permit.binding() + || group.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + || !group.matches_replica(observed.replica(), observed.membership()) + { + return Err(RuntimeError::Invalid( + "managed group repair authority".into(), + )); + } + let fleet = observed + .membership() + .fleet_status() + .ok_or_else(|| RuntimeError::Invalid("group is not enrolled".into()))?; + if &fleet.binding != permit.binding() { + return Err(RuntimeError::Invalid("foreign repair permit".into())); + } + if let Self::Start { plan, .. } = self { + permit.check_plan(plan)?; + } + if fleet.consumed_sequence < permit.sequence() { + return match self { + Self::Start { .. } => Ok(false), + Self::Advance { .. } => Err(RuntimeError::StaleCompletion), + }; + } + if fleet.consumed_sequence != permit.sequence() + || fleet.permit.as_ref() != Some(permit) + || fleet.phase == Some(FleetPermitPhase::Fenced) + { + return Err(RuntimeError::StaleCompletion); + } + let repair = observed + .membership() + .repair_status()? + .ok_or(RuntimeError::StaleCompletion)?; + permit.check_plan(&repair.plan)?; + let wanted = match self { + Self::Start { plan, .. } => { + if &repair.plan != plan { + return Err(RuntimeError::StaleCompletion); + } + RepairPhase::Seeding + } + Self::Advance { step, .. } => match step { + RepairStep::PromoteAndRemove => RepairPhase::Joint, + RepairStep::FinishJoint => RepairPhase::Complete, + RepairStep::AbortLearner => RepairPhase::Aborted, + }, + }; + if repair.phase == RepairPhase::Aborted && wanted != RepairPhase::Aborted + || wanted == RepairPhase::Aborted + && repair.phase != RepairPhase::Seeding + && repair.phase != RepairPhase::Aborted + { + return Err(RuntimeError::StaleCompletion); + } + Ok(repair.phase == wanted + || wanted == RepairPhase::Seeding + || wanted == RepairPhase::Joint && repair.phase == RepairPhase::Complete) + } + + pub(super) fn prepare( + &self, + observed: &CellGroupQuorumMembership, + ) -> Result { + let (group, authorization) = self.authority(); + if !group.matches_scope(&observed.replica().scope) + || !authorization.matches_scope(&observed.replica().scope) + { + return Err(RuntimeError::Invalid("group repair history".into())); + } + Ok(match self { + Self::Start { plan, .. } => CellGroupMembershipChange::managed_repair_start( + observed.root().route(), + observed.membership(), + observed.configuration(), + plan, + authorization, + )?, + Self::Advance { step, expected, .. } => { + if observed.membership().revision() != *expected { + return Err(RuntimeError::StaleCompletion); + } + CellGroupMembershipChange::managed_repair_step( + observed.root().route(), + observed.membership(), + observed.configuration(), + authorization.permit().repair_id()?, + *step, + authorization, + )? + } + }) + } + + pub(super) fn check_placement( + &self, + transport: &RaftTransport, + observed: &CellGroupQuorumMembership, + ) -> Result<(), RuntimeError> { + if let Self::Start { + group, + authorization, + plan, + } = self + { + let nodes = group.placement_nodes(observed)?; + transport + .check_fleet_placement( + &observed.replica().scope, + &nodes, + &plan.replacement, + authorization, + ) + .map_err(|error| match error { + TransportError::Overloaded => RuntimeError::Overloaded, + TransportError::StalePeer | TransportError::Refused => RuntimeError::NotReady, + TransportError::Runtime(error) => error, + TransportError::Closed => RuntimeError::Closed, + _ => RuntimeError::Invalid("managed repair placement".into()), + })?; + } + Ok(()) + } +} + +fn admission_error(error: CommandFailure) -> RuntimeError { + match error { + CommandFailure::Overloaded => RuntimeError::Overloaded, + CommandFailure::VersionUnavailable => RuntimeError::DeadlineExceeded, + CommandFailure::UnsupportedCapability => { + RuntimeError::Invalid("group repair callback context".into()) + } + _ => RuntimeError::Closed, + } +} + +impl RaftGroupHandle { + /// Start the exact catalog-selected learner repair. Admission is followed by + /// actual durable apply and a fresh quorum observation of the same permit. + pub fn begin_managed_repair( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + plan: RepairPlan, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.repair_change( + Change::Start { + group, + authorization, + plan, + }, + timeout, + ) + } + + /// Advance or abort the consumed plan at this observed revision. Promotion + /// still requires actual RawNode progress; a route or seed receipt is insufficient. + pub fn advance_managed_repair( + &self, + group: FleetGroupAuthorization, + authorization: FleetPermitAuthorization, + step: RepairStep, + expected: LogIndex, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + self.repair_change( + Change::Advance { + group, + authorization, + step, + expected, + }, + timeout, + ) + } + + fn repair_change( + &self, + change: Change, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = self + .owner + .metadata_admission(timeout, 64 * 1024) + .map_err(admission_error); + let owner = self.clone(); + async move { + let (deadline, _memory) = admitted.map_err(CellGroupControlError::Refused)?; + let (group, authorization) = change.authority(); + if !group.matches_scope(&owner.history) || !authorization.matches_scope(&owner.history) + { + return Err(CellGroupControlError::Refused(RuntimeError::Invalid( + "group repair history".into(), + ))); + } + let result = owner + .change_membership(MembershipChange::Repair(change), deadline) + .await?; + if owner.owner.is_closed() { + return Err(CellGroupControlError::Unconfirmed(RuntimeError::Closed)); + } + if owner + .owner + .handle + .domain + .mono_ms() + .map_err(|_| CellGroupControlError::Unconfirmed(RuntimeError::Closed))? + >= deadline + { + return Err(CellGroupControlError::Unconfirmed( + RuntimeError::DeadlineExceeded, + )); + } + Ok(result) + } + } +} diff --git a/crates/cellar/application/consensus/group_client/resources.rs b/crates/cellar/application/consensus/group_client/resources.rs new file mode 100644 index 000000000..0c0aa9593 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/resources.rs @@ -0,0 +1,238 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Shared-group observations use the actual shared admissions. Missing physical +//! observers remain unavailable, including after their owners have exited. +use super::*; +use crate::application::consensus::{RaftExtentUsage, RaftImageUsage, RaftTransportResources}; +use cellar_ltx::{generation::GenerationAccounting, internal::NodeExtentObserver}; +use std::fmt::Write; + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupImageUsage { + pub materialized: Option, + pub extents: Option, +} + +#[derive(Clone, Copy, Debug, serde::Serialize)] +pub struct CellGroupResourceUsage { + pub host: CellGroupHostUsage, + pub runtime: CellGroupRuntimeUsage, + pub control: crate::application::consensus::RaftBudgetUsage, + pub sql: Option, + pub serving_images: CellGroupImageUsage, + pub archive_verifier_images: Option, + pub transport: Option, + /// Cached manifest metadata over the bounded durable group inventory. + /// This counts retained cell histories, not current serving authority. + pub retained_manifest_cells: Option, +} + +#[derive(Clone)] +pub(crate) struct Images { + pub materialized: Option, + pub extents: Option, +} + +impl Images { + fn capture(&self) -> CellGroupImageUsage { + CellGroupImageUsage { + materialized: self + .materialized + .as_ref() + .and_then(|v| v.snapshot()) + .map(Into::into), + extents: self + .extents + .as_ref() + .and_then(|v| v.capture()) + .map(Into::into), + } + } +} + +pub(crate) struct Resources { + pub(super) sql: CellGroupSqlPool, + transport: crate::raft::transport::ResourceObserver, + serving: Images, + archive: Option, +} + +impl Resources { + pub(crate) fn new( + sql: CellGroupSqlPool, + transport: &crate::raft::transport::RaftTransport, + serving: Images, + archive: Option, + ) -> Self { + Self { + sql, + transport: transport.resource_observer(), + serving, + archive, + } + } + + pub(super) fn capture(&self, core: &Core) -> CellGroupResourceUsage { + let cells = core.log.cell_groups().ok().and_then(|groups| { + groups.iter().try_fold(0u64, |sum, group| { + let manifest = group.group_manifest().ok()??; + sum.checked_add(u64::from(manifest.root().cells())) + }) + }); + CellGroupResourceUsage { + host: core.host.usage(), + runtime: core.runtime.usage(), + control: crate::application::consensus::RaftBudgetUsage::capture(&core.services.memory), + sql: self.sql.usage().ok(), + serving_images: self.serving.capture(), + archive_verifier_images: self.archive.as_ref().map(Images::capture), + transport: self.transport.capture(), + retained_manifest_cells: cells, + } + } +} + +impl CellGroupResourceUsage { + /// Fixed vocabularies only. Group, cell, tenant and peer identities never + /// enter labels. Unavailable observers emit availability without zero usage. + pub fn prometheus(&self) -> String { + let mut out = String::new(); + let c = self.host.controller; + for (name, value) in [ + ("namespaces", self.host.namespaces), + ("namespace_bytes", self.host.namespace_bytes), + ("namespace_capacity_bytes", self.host.namespace_byte_limit), + ("registered_groups", c.groups), + ("quarantined_groups", c.quarantined_groups), + ("retained_calls", c.retained_commands), + ("call_bytes", c.command_bytes), + ("retained_s3", c.retained_s3), + ("s3_bytes", c.s3_bytes), + ("active_apply", c.active_apply), + ( + "provision_configuration_bytes", + c.provision_configuration_bytes, + ), + ("pending_apply", c.pending_apply), + ("active_effects", c.active_effects), + ] { + sample(&mut out, name, "", value); + } + for (lane, usage) in [ + ("ordinary", c.ordinary), + ("archive_settlement", c.archive_settlement), + ] { + let labels = format!("{{lane=\"{lane}\"}}"); + for (name, value) in [ + ("call_admission_items", usage.items), + ("call_admission_bytes", usage.bytes), + ("call_admission_capacity_items", usage.max_items), + ("call_admission_capacity_bytes", usage.max_bytes), + ] { + sample(&mut out, name, &labels, value); + } + } + for (lane, usage) in [ + ("ordinary", self.runtime.ordinary), + ("completion", self.runtime.completions), + ("snapshot_feedback", self.runtime.snapshot_feedback), + ("read_pin", self.runtime.read_pins), + ] { + let labels = format!("{{lane=\"{lane}\"}}"); + for (name, value) in [ + ("runtime_items", usage.items), + ("runtime_bytes", usage.bytes), + ("runtime_capacity_items", usage.max_items), + ("runtime_capacity_bytes", usage.max_bytes), + ] { + sample(&mut out, name, &labels, value); + } + } + sample( + &mut out, + "sql_sample_available", + "", + usize::from(self.sql.is_some()), + ); + if let Some(sql) = self.sql { + for (name, value) in [ + ("sql_workers", sql.running_workers), + ("sql_worker_capacity", sql.worker_capacity), + ("sql_active_cells", sql.active_cells), + ("sql_retained_operations", sql.retained_operations), + ("sql_retained_bytes", sql.retained_bytes), + ("sql_queued_or_running", sql.queued_or_running), + ("sql_apply_operations", sql.apply_retained_operations), + ("sql_apply_bytes", sql.apply_retained_bytes), + ("sql_completion_reservations", sql.completion_reservations), + ("sql_capacity_bytes", sql.limits.node_bytes), + ("sql_group_capacity_bytes", sql.limits.group_bytes), + ("sql_capacity_cells", sql.limits.max_pending_cells), + ("sql_group_capacity_cells", sql.limits.group_pending_cells), + ("sql_queue_closed", usize::from(sql.closing)), + ] { + sample(&mut out, name, "", value); + } + } + for (kind, images) in [ + ("serving", Some(self.serving_images)), + ("archive_verifier", self.archive_verifier_images), + ] { + let labels = format!("{{kind=\"{kind}\"}}"); + sample( + &mut out, + "images_sample_available", + &labels, + usize::from(images.and_then(|i| i.materialized).is_some()), + ); + sample( + &mut out, + "extents_sample_available", + &labels, + usize::from(images.and_then(|i| i.extents).is_some()), + ); + if let Some(i) = images.and_then(|i| i.materialized) { + for (name, value) in [ + ("image_bytes", i.bytes), + ("image_capacity_bytes", i.max_bytes), + ("image_generations", i.generations as u64), + ("image_retained", i.retained as u64), + ] { + sample(&mut out, name, &labels, value); + } + } + if let Some(i) = images.and_then(|i| i.extents) { + for (name, value) in [ + ("extent_bytes", i.bytes), + ("extent_capacity_bytes", i.max_bytes), + ("extent_objects", i.objects as u64), + ("extent_roots", i.roots as u64), + ("extent_workspace_bytes", i.workspace_bytes), + ("extent_quarantined", u64::from(i.quarantined)), + ] { + sample(&mut out, name, &labels, value); + } + } + } + sample( + &mut out, + "manifest_cell_sample_available", + "", + usize::from(self.retained_manifest_cells.is_some()), + ); + if let Some(cells) = self.retained_manifest_cells { + sample(&mut out, "retained_manifest_cells", "", cells); + } + sample( + &mut out, + "transport_sample_available", + "", + usize::from(self.transport.is_some()), + ); + out + } +} + +fn sample(out: &mut String, name: &str, labels: &str, value: impl std::fmt::Display) { + let _ = writeln!(out, "cellar_cell_groups_{name}{labels} {value}"); +} diff --git a/crates/cellar/application/consensus/group_client/services.rs b/crates/cellar/application/consensus/group_client/services.rs new file mode 100644 index 000000000..9966a8454 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/services.rs @@ -0,0 +1,236 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bounded per-node control services. Closing signals every admitted worker, +//! then joins its actual lifetime before consensus/storage shutdown proceeds. +use super::*; +use crate::raft::{ + budget::{Budget, Permit}, + fleet_catalog::FleetCatalogError, +}; +use tokio::sync::{watch, Notify}; + +#[derive(Clone, Copy)] +pub(crate) enum ControlServiceKind { + Movement, + Restart, +} + +impl ControlServiceKind { + fn slot(self) -> usize { + match self { + Self::Movement => 0, + Self::Restart => 1, + } + } +} + +struct Entry { + id: u64, + stop: watch::Sender, +} + +#[derive(Default)] +struct State { + closing: bool, + next: u64, + slots: [Option; 2], +} + +pub(super) struct Services { + state: Mutex, + finished: Notify, + pub(super) memory: Arc, +} + +pub(crate) struct ControlServiceLease { + owner: Arc, + kind: ControlServiceKind, + id: u64, + stop: watch::Receiver, + memory: Option, +} + +pub(crate) struct ControlServiceCompletion(watch::Sender); + +impl ControlServiceCompletion { + pub(crate) fn new(done: watch::Sender) -> Self { + Self(done) + } +} + +impl Drop for ControlServiceCompletion { + fn drop(&mut self) { + self.0.send_replace(true); + } +} + +impl ControlServiceLease { + pub(crate) fn stop_receiver(&self) -> watch::Receiver { + self.stop.clone() + } +} + +impl Drop for ControlServiceLease { + fn drop(&mut self) { + // Release the worker's memory before advertising completed physical + // ownership. A cancelled close waiter does not remove this entry. + self.memory.take(); + let mut state = self.owner.state.lock().unwrap_or_else(|p| p.into_inner()); + if state.slots[self.kind.slot()] + .as_ref() + .is_some_and(|e| e.id == self.id) + { + state.slots[self.kind.slot()] = None; + } + drop(state); + self.owner.finished.notify_waiters(); + } +} + +impl Services { + pub(super) fn new(bytes: usize, items: usize) -> Arc { + Arc::new(Self { + state: Mutex::new(State::default()), + finished: Notify::new(), + memory: Budget::new(bytes, items), + }) + } + + fn register( + self: &Arc, + kind: ControlServiceKind, + bytes: usize, + ) -> Result { + let mut state = self.state.lock().map_err(|_| FleetCatalogError::Closed)?; + if state.closing { + return Err(FleetCatalogError::Closed); + } + if state.slots[kind.slot()].is_some() { + return Err(FleetCatalogError::Overloaded); + } + let memory = self + .memory + .acquire(bytes) + .ok_or(FleetCatalogError::Overloaded)?; + state.next = state.next.checked_add(1).ok_or(FleetCatalogError::Limit)?; + let id = state.next; + let (stop, receiver) = watch::channel(false); + state.slots[kind.slot()] = Some(Entry { id, stop }); + Ok(ControlServiceLease { + owner: self.clone(), + kind, + id, + stop: receiver, + memory: Some(memory), + }) + } + + pub(super) async fn close(&self) { + loop { + let notified = self.finished.notified(); + tokio::pin!(notified); + notified.as_mut().enable(); + { + let mut state = self.state.lock().unwrap_or_else(|p| p.into_inner()); + state.closing = true; + for entry in state.slots.iter().flatten() { + entry.stop.send_replace(true); + } + if state.slots.iter().all(Option::is_none) { + return; + } + } + notified.await; + } + } +} + +impl RaftCells { + // Internal loops must not retain the last external SDK lifetime. An active + // service is joined by the registry, independently of its public handle. + pub(crate) fn internal_observer(&self) -> Self { + Self { + handle: self.handle.clone(), + lifetime: None, + } + } + + pub(crate) fn control_budget(&self) -> &Arc { + &self.handle.core.services.memory + } + + pub(crate) fn register_control_service( + &self, + kind: ControlServiceKind, + bytes: usize, + ) -> Result { + if self.is_closed() || self.handle.admission.is_draining() { + return Err(FleetCatalogError::Closed); + } + self.handle.core.services.register(kind, bytes) + } + + async fn installed_host( + &self, + timeout: Duration, + ) -> Result { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + let core = &self.handle.core; + let host = core.managed.as_ref().ok_or(FleetCatalogError::Invalid( + "control service requires managed host", + ))?; + let expected = host.boot_binding()?; + let current = host.refresh(timeout).await?; + let boot = current.boot_binding()?; + if boot != expected + || boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || core + .log + .lifecycle_guard() + .map_err(|_| FleetCatalogError::Closed)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(FleetCatalogError::Conflict); + } + Ok(current) + } + + pub(crate) async fn start_movement_worker( + &self, + mut work: crate::raft::fleet_catalog::MovementServiceWork, + timeout: Duration, + ) -> Result<(), FleetCatalogError> { + let host = self.installed_host(timeout).await?; + work.admit(&host)?; + self.handle + .domain + .spawn_cleanup(async move { work.run().await }) + .map_err(|_| FleetCatalogError::Closed)? + .detach(); + Ok(()) + } + + pub(crate) async fn start_restart_worker( + &self, + mut work: crate::raft::fleet_catalog::RestartServiceWork, + timeout: Duration, + ) -> Result<(), FleetCatalogError> { + let host = self.installed_host(timeout).await?; + work.admit(&host)?; + self.handle + .domain + .spawn_cleanup(async move { work.run().await }) + .map_err(|_| FleetCatalogError::Closed)? + .detach(); + Ok(()) + } +} + +#[cfg(test)] +#[path = "services_tests.rs"] +mod tests; diff --git a/crates/cellar/application/consensus/group_client/services_tests.rs b/crates/cellar/application/consensus/group_client/services_tests.rs new file mode 100644 index 000000000..ce4b1b784 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/services_tests.rs @@ -0,0 +1,37 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +#[tokio::test] +async fn close_retains_service_admission_until_the_actual_worker_exits() { + let services = Services::new(128, 1); + let worker = services.register(ControlServiceKind::Movement, 64).unwrap(); + let stop = worker.stop_receiver(); + assert!(matches!( + services.register(ControlServiceKind::Movement, 64), + Err(FleetCatalogError::Overloaded) + )); + assert!(matches!( + services.register(ControlServiceKind::Restart, 64), + Err(FleetCatalogError::Overloaded) + )); + let mut closing = Box::pin(services.close()); + assert!( + tokio::time::timeout(Duration::from_millis(30), &mut closing) + .await + .is_err() + ); + assert!(*stop.borrow()); + assert_eq!(services.memory.usage(), (64, 1)); + drop(closing); + // Cancelling a close waiter neither removes the live worker nor reopens + // admission. The independently owned cleanup can resume the same join. + assert!(matches!( + services.register(ControlServiceKind::Restart, 1), + Err(FleetCatalogError::Closed) + )); + assert_eq!(services.memory.usage(), (64, 1)); + drop(worker); + services.close().await; + assert_eq!(services.memory.usage(), (0, 0)); +} diff --git a/crates/cellar/application/consensus/group_client/startup.rs b/crates/cellar/application/consensus/group_client/startup.rs new file mode 100644 index 000000000..37b0b52ba --- /dev/null +++ b/crates/cellar/application/consensus/group_client/startup.rs @@ -0,0 +1,316 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Initial voters use the controller's retained whole-group preparation owner. +//! Attachment receipts grant neither a fresh quorum nor application admission. +use super::*; +use crate::raft::{ + catalog::CatalogError, + completion::RuntimeError, + fleet_catalog::{FleetGroupAuthorization, FleetHostAuthorization}, + member_state::FleetMembershipStatus, +}; +use cellar_logic::consensus::CommandFailure; + +const STARTUP_BYTES: usize = 4 * 1024 * 1024; + +/// A refusal submitted no preparation in this invocation. An unconfirmed result +/// can follow durable genesis or attachment; retry the same physical history. +#[derive(Clone, Debug)] +pub enum CellGroupStartupError { + Refused(CellGroupControllerError), + Unconfirmed(CellGroupControllerError), +} + +impl std::fmt::Display for CellGroupStartupError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Refused(error) => write!(f, "group startup refused: {error}"), + Self::Unconfirmed(error) => write!(f, "group startup unconfirmed: {error}"), + } + } +} + +impl std::error::Error for CellGroupStartupError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Refused(error) | Self::Unconfirmed(error) => error, + }) + } +} + +pub(super) fn admission_error(error: CommandFailure) -> CellGroupControllerError { + match error { + CommandFailure::Overloaded => RuntimeError::Overloaded.into(), + CommandFailure::VersionUnavailable => RuntimeError::DeadlineExceeded.into(), + CommandFailure::UnsupportedCapability => { + CellGroupControllerError::Invalid("group startup callback context") + } + _ => CellGroupControllerError::Closed, + } +} + +pub(super) fn catalog_error(error: CatalogError) -> CellGroupControllerError { + CellGroupControllerError::Catalog(Arc::new(error)) +} + +impl RaftCells { + /// Recover or initialize this exact admitted initial voter. Namespace lookup + /// allocates no cell/group identity; the rooted directory selects its birth. + /// Success describes a durable local attachment, never quorum readiness. + pub fn activate_managed_initial_group( + &self, + authorization: FleetGroupAuthorization, + timeout: Duration, + ) -> impl Future> + Send + 'static + { + let admitted = (|| { + let (deadline, memory) = self + .metadata_admission(timeout, STARTUP_BYTES) + .map_err(admission_error)?; + let host = + self.handle + .core + .managed + .clone() + .ok_or(CellGroupControllerError::Invalid( + "managed group startup owner", + ))?; + let history = authorization + .binding() + .scope() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + if !authorization.allows_managed_genesis() + || authorization.binding().fleet() != host.fleet() + { + return Err(CellGroupControllerError::Invalid( + "managed genesis authority", + )); + } + let directory = + self.handle + .core + .host + .directory(history) + .map_err(|error| match error { + CellDispatchError::Runtime(error) => error.into(), + _ => { + CellGroupControllerError::Invalid("group startup namespace capability") + } + })?; + Ok((deadline, memory, host, history, directory)) + })(); + let owner = self.internal_observer(); + async move { + use CellGroupStartupError::{Refused, Unconfirmed}; + let (deadline, _memory, host, history, directory) = admitted.map_err(Refused)?; + let mut submitted = false; + let work = async { + owner.startup_live(&host, deadline).map_err(Refused)?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + let namespace_owner = directory + .namespace() + .cell_group_owner() + .await + .map_err(catalog_error) + .map_err(Refused)?; + if namespace_owner + .is_none_or(|o| o.fleet != host.fleet() || o.catalog != boot.catalog) + { + return Err(Refused(catalog_error( + CatalogError::ManagedAdmissionRequired, + ))); + } + let birth = directory + .materialize_selected_history(history) + .await + .map_err(catalog_error) + .map_err(Refused)?; + if !authorization.check_birth(&directory, &birth) + || !birth + .voters() + .iter() + .any(|m| m.node == owner.handle.core.node.node) + { + return Err(Refused(CellGroupControllerError::Invalid( + "initial group voter birth", + ))); + } + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Refused)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Refused)?; + if fresh.binding() != authorization.binding() + || !fresh.check_birth(&directory, &birth) + || !fresh.allows_managed_genesis() + { + return Err(Refused(CellGroupControllerError::Invalid( + "initial group authorization changed", + ))); + } + owner.startup_live(&host, deadline).map_err(Refused)?; + let call = owner + .handle + .core + .controller + .provision_managed_initial( + directory.clone(), + birth.route().clone(), + host.clone(), + deadline, + ) + .map_err(Refused)?; + submitted = true; + call.response() + .await + .map_err(|error| Unconfirmed((*error).clone()))?; + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + // Reauthorize after attachment. The saved admission and a successful + // controller reply cannot substitute for this current root check. + let fresh = host + .group_authorization( + &history, + owner.startup_remaining(deadline).map_err(Unconfirmed)?, + ) + .await + .map_err(CatalogError::from) + .map_err(catalog_error) + .map_err(Unconfirmed)?; + if fresh.binding() != authorization.binding() + || !fresh.check_birth(&directory, &birth) + || !fresh.allows_managed_genesis() + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group authorization changed", + ))); + } + let core = &owner.handle.core; + if core + .controller + .group_failure(history) + .map_err(CellGroupControllerError::from) + .map_err(Unconfirmed)? + .is_some() + { + return Err(Unconfirmed(RuntimeError::GroupQuarantined.into())); + } + let group = core + .log + .cell_group(&history) + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + let marker = group + .group_manifest() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))? + .ok_or(Unconfirmed(CellGroupControllerError::Invalid( + "missing attached group root", + )))?; + if marker.route() != birth.route() || !marker.is_active() { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group root", + ))); + } + let status = { + let cache = group + .read() + .map_err(|e| Unconfirmed(RuntimeError::from(e).into()))?; + if !crate::raft::member_state::members(&cache.conf_state) + .contains(&group.identity().member.get()) + || !fresh.matches_replica(group.identity(), &cache.membership) + { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial group membership", + ))); + } + cache.membership.fleet_status().ok_or(Unconfirmed( + CellGroupControllerError::Invalid("missing initial fleet guard"), + ))? + }; + if &status.binding != authorization.binding() { + return Err(Unconfirmed(CellGroupControllerError::Invalid( + "initial fleet binding", + ))); + } + owner.startup_live(&host, deadline).map_err(Unconfirmed)?; + Ok(status) + }; + match owner + .handle + .core + .transport + .seed_control_until(deadline, async { + // Public callers may poll on another runtime. Catalog + // deadlines and I/O must still enter this installed owner. + owner + .handle + .domain + .run(work) + .await + .map_err(|_| crate::raft::transport::TransportError::Closed) + }) + .await + { + Ok(result) => result, + Err(error) => { + let error = CellGroupControllerError::Transport(error); + Err(if submitted { + Unconfirmed(error) + } else { + Refused(error) + }) + } + } + } + } + + pub(super) fn startup_remaining( + &self, + deadline: u64, + ) -> Result { + let now = self + .handle + .domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + let remaining = deadline + .checked_sub(now) + .filter(|ms| *ms > 0 && *ms <= 60_000) + .ok_or(CellGroupControllerError::Runtime( + RuntimeError::DeadlineExceeded, + ))?; + Ok(Duration::from_millis(remaining)) + } + + pub(super) fn startup_live( + &self, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result<(), CellGroupControllerError> { + self.metadata_live(deadline).map_err(admission_error)?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let core = &self.handle.core; + if boot.node != core.node.node.bytes() + || boot.disk != core.node.disk.bytes() + || boot.process != core.node.process.bytes() + || boot.engine != core.engine.bytes() + || core + .log + .lifecycle_guard() + .map_err(RuntimeError::from)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(catalog_error(CatalogError::Fenced)); + } + Ok(()) + } +} diff --git a/crates/cellar/application/consensus/group_client/stop.rs b/crates/cellar/application/consensus/group_client/stop.rs new file mode 100644 index 000000000..3dc3eba11 --- /dev/null +++ b/crates/cellar/application/consensus/group_client/stop.rs @@ -0,0 +1,66 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Exact-boot lifecycle RPCs use the node's authenticated transport. Its receiver +//! owns admitted mutations; losing the origin waiter cannot establish rollback. +use super::*; +use crate::raft::{ + fleet_catalog::FleetCatalogError, + fleet_stop::{FleetStopAcceptance, FleetStopRequest}, + transport::fleet_stops::STOP_REQUEST_BYTES, +}; + +impl RaftCells { + /// Submit the exact persisted stop/quiesce/fence request. No business cell + /// or local group is required. Acceptance proves historical guard admission, + /// independently of physical closure and terminal removal certificates. + /// A lost or expired response is Unconfirmed; reconcile the same request. + pub fn request_peer_fleet_stop( + &self, + request: FleetStopRequest, + minimum_revision: u64, + timeout: Duration, + ) -> impl Future> + Send + 'static { + // Capture deadline and admission before returning the waiter. The + // transport starts owned bounded work at invocation, not at first poll. + let domain = self.handle.domain.clone(); + let admitted = (|| { + if self.is_closed() { + return Err(FleetCatalogError::Closed); + } + if self.handle.core.managed.is_none() { + return Err(FleetCatalogError::Invalid("no managed fleet origin")); + } + if request.selector.validate().is_err() + || minimum_revision == 0 + || minimum_revision >= u64::MAX - 2 + || timeout.as_millis() == 0 + || timeout > Duration::from_secs(60) + { + return Err(FleetCatalogError::Invalid("peer stop request")); + } + let deadline = domain + .mono_ms() + .map_err(|_| FleetCatalogError::Closed)? + .checked_add(timeout.as_millis() as u64) + .ok_or(FleetCatalogError::Limit)?; + let memory = self + .control_budget() + .acquire(STOP_REQUEST_BYTES) + .ok_or(FleetCatalogError::Overloaded)?; + let delivery = self.handle.core.transport.request_fleet_stop( + request, + minimum_revision, + deadline, + )?; + Ok((deadline, memory, delivery)) + })(); + async move { + let (deadline, _memory, delivery) = admitted?; + let received = delivery.receive().await?; + if !domain.mono_ms().is_ok_and(|now| now < deadline) { + return Err(FleetCatalogError::Unconfirmed); + } + Ok(*received.acceptance) + } + } +} diff --git a/crates/cellar/application/consensus/mod.rs b/crates/cellar/application/consensus/mod.rs index 0944fd8f5..fbcabf68a 100644 --- a/crates/cellar/application/consensus/mod.rs +++ b/crates/cellar/application/consensus/mod.rs @@ -11,6 +11,7 @@ pub use archive_status::{ArchiveFailure, ArchiveReadiness, ArchiveStatus}; mod class; pub(crate) use class::callback_active; pub(crate) mod client; +pub(crate) mod group_client; mod group_contracts; pub(crate) use group_contracts::cell_routing_key; pub(crate) mod group_protocol; @@ -30,7 +31,9 @@ pub use resources::{ pub(crate) mod protocol; pub use crate::raft::archive::ArchiveLimits; -pub use crate::raft::completion::{LeadershipTransfer, MemberProgress, ReplicationStatus}; +pub use crate::raft::completion::{ + LeadershipTransfer, MemberProgress, ReplicationStatus, RuntimeError, +}; pub use crate::raft::effects::{ RaftAlarmLimits, RaftBlobLimits, RaftBlobReference, RaftDeliveryRequest, RaftEffectOptions, RaftEffectSigningKey, RaftEffectSink, RaftEffectSourceGrant, RaftEffectTarget, @@ -51,17 +54,26 @@ pub use crate::raft::disaster::S3DisasterRecovery; pub use cellar_logic::raft_leadership::LeaderPolicy; pub use cellar_logic::raft_placement::PlacementPolicy; +pub use crate::raft::cell_groups::{ + CellCommandError, CellDispatchError, CellDispatchLimits, CellEnrollmentError, + CellEnrollmentRequest, CellGroupArchiveLimits, CellGroupArchiveTailLimits, + CellGroupArchiveUploadLimits, CellGroupBirthSpec, CellGroupCheckpointLimits, + CellGroupCheckpointPolicy, CellGroupControllerLimits, CellGroupHostLimits, + CellGroupProvisionLimits, CellGroupQuorumMembership, CellReceiptRequest, CellReceiptResponse, + CellSqlLimits, DirectoryLimits, ManifestLimits, +}; pub use crate::raft::fleet_catalog::{ fleet_capacity_profile_digest, FleetAbortedMovementCompletion, FleetCapacityAction, FleetCapacityAdmission, FleetCapacityAttestation, FleetCapacityAuthorization, - FleetCapacityRequest, FleetFencedMovementCompletion, FleetGroupAuthorization, - FleetHostAuthorization, FleetIsolationAttestation, FleetMovementAdmission, - FleetMovementCompletion, FleetMovementCompletionRequest, FleetMovementExecutionRequest, - FleetMovementPhase, FleetMovementProgress, FleetMovementRequest, FleetNodeEligibility, - FleetNodeIsolation, FleetNodeIsolationRequest, FleetNodeSpec, FleetNodeState, - FleetPendingMovement, FleetPendingReservation, FleetPermitAuthorization, - FleetReplicaSeparation, FleetRequestIdentity, FleetResourceAdmission, FleetResourceAttestation, - FleetResourceBounds, FleetResourceRequest, + FleetCapacityRequest, FleetFencedMovementCompletion, FleetGroupArchiveReplayOptions, + FleetGroupArchiveVerifier, FleetGroupAuthorization, FleetHostAuthorization, + FleetIsolationAttestation, FleetMovementAdmission, FleetMovementCompletion, + FleetMovementCompletionRequest, FleetMovementExecutionRequest, FleetMovementPhase, + FleetMovementProgress, FleetMovementRequest, FleetNodeEligibility, FleetNodeIsolation, + FleetNodeIsolationRequest, FleetNodeSpec, FleetNodeState, FleetPendingMovement, + FleetPendingReservation, FleetPermitAuthorization, FleetReplicaSeparation, + FleetRequestIdentity, FleetResourceAdmission, FleetResourceAttestation, FleetResourceBounds, + FleetResourceRequest, }; pub use crate::raft::fleet_catalog::{ FleetDockerIsolation, FleetDockerResourcePolicy, FleetDockerResources, @@ -86,25 +98,29 @@ pub use crate::raft::fleet_catalog::{ pub use crate::raft::fleet_stop::{ FleetBootBinding, FleetStopAcceptance, FleetStopAction, FleetStopRequest, VerifiedFleetClosure, }; +pub use crate::raft::group_host::{ + CellGroupNodeArchiveOptions, CellGroupNodeLimits, RaftCellsConfig, RaftGroupAssignment, + RaftGroupPlacement, RaftNamespaceConfig, RaftNamespaceInitialization, +}; pub use crate::raft::host::{ FleetImageState, FleetMembershipConfiguration, FleetMembershipObservation, FleetPublishedCut, - FleetPublishedImage, RaftArchiveOptions, RaftCellPlacement, RaftCellRegistration, - RaftCellsConfig, RaftCellsServices, RaftControllerOptions, RaftControllerStatus, - RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions, RaftHostEnrollmentObservation, - RaftHostObservation, RaftHostOptions, RaftHostStartupBounds, RaftInspectionScanOptions, - RaftManagedHostOptions, RaftNamespaceRegistration, RaftNodeDirectoryOptions, - RaftPeerFleetMembershipObservation, RaftPeerNode, RaftPeerNodeObservation, + FleetPublishedImage, RaftArchiveOptions, RaftCellsServices, RaftControllerOptions, + RaftControllerStatus, RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions, + RaftHostEnrollmentObservation, RaftHostObservation, RaftHostOptions, RaftHostStartupBounds, + RaftInspectionScanOptions, RaftManagedHostOptions, RaftNamespaceRegistration, + RaftNodeDirectoryOptions, RaftPeerFleetMembershipObservation, RaftPeerNode, + RaftPeerNodeObservation, }; pub use crate::raft::inspection::{ - CellInspection, InspectionError, LeaderLoad, MemberInspection, NodeInspection, RaftNodeProfile, - ReplicaOccupancy, + CellInspection, GroupInspection, InspectionError, LeaderLoad, MemberInspection, NodeInspection, + RaftNodeProfile, ReplicaOccupancy, }; pub use crate::raft::member_state::{ FleetGroupBinding, FleetMembershipPermit, FleetMembershipStatus, FleetPermitPhase, JoiningMember, RepairPhase, RepairPlan, RepairStatus, RepairStep, }; pub use crate::raft::operations::{ - RaftCloseReportLimits, RaftDrainReport, RaftGroupCloseReport, RaftNodeShutdownReport, + RaftCloseReportLimits, RaftDrainReport, RaftNodeShutdownError, RaftNodeShutdownReport, }; pub use crate::raft::options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}; pub use crate::raft::transport_options::{ @@ -115,14 +131,23 @@ pub use crate::raft::version::{ VersionError, VersionKeys, VersionSigningKey, VersionToken, VersionVerificationKey, }; pub use crate::storage::{CommandInputs, CommandTransaction}; +pub use cellar_logic::cell_group::{CellBound, CellRange, RouteEpoch}; pub use class::{ command_request_digest, CommandContext, OperationContract, OperationKind, QueryContext, QueryVersion, RaftClass, RegistrationError, }; pub use client::{ - CellStatus, CommandOptions, CommandReceipt, LearnerActivation, LearnerOptions, LearnerReceipt, - QueryConsistency, QueryOptions, QueryResult, QueryRouting, RaftCellHandle, RaftCells, - RaftCloseReport, StaleReadPolicy, UnfinishedCommand, + CellStatus, LearnerActivation, LearnerOptions, LearnerReceipt, UnfinishedCommand, +}; +pub use group_protocol::GroupEntryLimits; + +pub use crate::raft::version::CellVersionToken; +pub use cellar_logic::cell_group::{CommitWitness, GroupHistory}; +pub use group_client::{ + CellGroupControlError, CellGroupImageUsage, CellGroupResourceUsage, CellGroupStartupError, + CommandOptions, CommandReceipt, MutationOutcomes, QueryConsistency, QueryOptions, QueryResult, + RaftCellHandle, RaftCells, RaftCloseError, RaftCloseReport, RaftGroupCloseReport, + RaftGroupHandle, StaleReadPolicy, }; pub use crate::raft::version::CellVersionObservation; @@ -145,7 +170,7 @@ pub mod verification { ArchiveDependency, ArchiveError, ArchiveLimits, ArchiveMetadata, ArchiveRecoverySet, ArchiveReference, ArchiveVerifier, CellArchive, CommittedArchiveCut, RootAllocation, RootArchiveReference, RootArchiveSnapshot, RootDescriptor, RootIntent, RootProtection, - RootReference, RootRegistry, RootRegistryId, RootReservation, VerifiedArchiveCoverage, + RootReference, RootRegistry, RootRegistryId, RootReservation, }; pub use crate::raft::cache::{ CommittedImage, DurableApply, GroupLog, GroupWrite, SnapshotChange, diff --git a/crates/cellar/application/consensus/protocol.rs b/crates/cellar/application/consensus/protocol.rs index e38488e8f..c49dc7f15 100644 --- a/crates/cellar/application/consensus/protocol.rs +++ b/crates/cellar/application/consensus/protocol.rs @@ -18,7 +18,7 @@ const DIGEST_LEN: usize = 32; // Wire markers remain 1. The immutable namespace identity also binds member // metadata so incompatible binaries cannot attach to the same namespace. const PROTOCOL: &str = - "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1"; + "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1/cell-group-fleet-1/group-import-1"; /// Bounds apply before decoding the header or copying any payload. The LTX /// decoder must additionally enforce these bounds on actual decoded pages; diff --git a/crates/cellar/durability/raft/README.md b/crates/cellar/durability/raft/README.md index 6310a96b4..bacc0d640 100644 --- a/crates/cellar/durability/raft/README.md +++ b/crates/cellar/durability/raft/README.md @@ -727,7 +727,9 @@ pinned until the callback joins and its private mutations are discarded. A new term and applied startup barrier are required to reconcile an unresolved submission. Closing records unfinished identity and refuses further output. `Node::start_raft_cells` installs registered groups in the node's execution -domain. `host.rs` owns serial command producers and effect dispatch; +domain. `group_host/mod.rs` owns canonical namespace/group startup; the shared controller +owns serial group apply and bounded cell command producers. `host.rs` retains +physical configuration and observation values; `host_sql/mod.rs` owns SQLite/image resources in a fixed node-wide blocking pool. Its private `images` module selects physical command/read adapters beneath the shared controller. The extent branch is exercised by the worker integration @@ -893,70 +895,44 @@ After submission, the producer remains owned through consensus and publication. A newer committed/published term barrier can retire old uncertainty even on a follower, after disposing of its private shadow; that grants no write authority. -MR-08's local read controller holds each request's original deadline and byte/item -permit while waiting for an image. `AtLeast` verifies an opaque signed token -before comparing history positions. `Linearizable` uses fresh safe ReadIndex -contexts, waits for actual image publication, and never treats a runtime status -or cached commit position as a barrier. A checked role/term change or lost reply -can resubmit the invocation's barrier under the same deadline. Retries wait four -host polling intervals, clamped to 100–1000 ms, and still require a safe ReadIndex -result. Status and submission generations prevent late replies from regressing -an executing query. - -The SQL owner captures the durable applied marker with its exact immutable pin. -Protocol-only entries can cover that pin; an unrelated newer image cannot relabel -it. Queries retain the pin through callback completion, and their result metadata -names that captured cut. The default Stale policy requires a recent observed -source; `DisconnectedHistorical` explicitly permits an older verified image -without a time bound. This policy observation is never used for strong reads. -SQL progress hooks observe an atomic cancellation flag raised by the node's -deadline controller, including for an unpolled waiter. Native Rust callbacks -remain trusted/cooperative and are joined before resource release. -Completed responses retain their byte admission until consumed or dropped; -unpolled response channels cannot reuse capacity merely because SQL finished. - -`RaftCellsServices::versions` injects managed signing/verification keys. Ed25519 -tokens use bounded canonical marker-1 framing and bind full history, schema and -log position. Namespace startup reconciles public-key retention metadata through -conditional updates and exact reads; configuration cannot silently drop a live -verifier on restart. The catalog supplies no new trust roots or consensus -authority. Key rotation and token lifetime are described in the +The shared cell-group query controller fixes each invocation's original deadline +and bounded attempt count. `Stale`, `AtLeast` and `Linearizable` are distinct +contracts. `AtLeast` authenticates a CellVersionToken and compares the logical +cell's version, rather than an unrelated group's log index. `Linearizable` uses +a fresh safe ReadIndex barrier and waits for its actual applied root. Runtime +status and cached commit positions cannot substitute for that barrier. + +The SQL owner retains the selected cell's exact immutable image through callback +completion. Returned observations separate CellVersion from its physical +CommitWitness, and protocol-only entries cannot relabel a different image. +The public SDK's QueryResult names the serving NodeIdentity. The lower transport +retains the authenticated responding member as part of its own delivery result. +Completed responses retain their byte admission until consumed or dropped. +Native callbacks remain cooperative and are joined before resource release. + +RaftCellsServices::versions injects managed Ed25519 signing/verification keys. +The bounded canonical marker-1 cell token binds logical history, schema, version, +commit witness, issuance and expiry. Namespace startup reconciles verifier +retention through conditional updates and exact reads. The registry supplies no +new trust roots or consensus authority. See the [application contract](../../../../docs/design/multi-raft/application-contract.md). -`QueryRouting::Replicas` tries up to eight distinct configured members in priority -order, with one optional untried leader hint. The caller supplies a bounded -per-attempt timeout; every attempt keeps the same contract, captured input, -consistency/token and stale policy under the invocation's original deadline. -Forwarding does not require query code on the origin voter. The destination -still requires a matching query definition before execution. `served_by` names -the actual responding member, independently of the history token. - -`NodeRaftListeners::query` and `RaftPeer::query_address` provide the separate -query endpoint. A peer has one bounded query queue/connection alongside its -control and LTX connections. A shared pool owns the physical RaftPeerNode -endpoint; it retains no group or member identity. Each consensus request still -carries and validates its own full member route. Query MAC domains, exporter -label and marker-1 framing are independent of consensus frames. Requests/replies bind full route, -schema, sequence, request tag and payload digest. The receiver forces local -query execution; it cannot recursively route. The sender deducts queued and -handshake time before sending a remaining timeout, while the origin separately -enforces its deadline. Closing the connection cancels the destination waiter. -Request/response representations remain charged through delivery. - -Query headers use a distinct route representation: the caller carries its -node/disk/process identity and label, with no Raft member ID. The target carries -its complete replica/history identity. The response reverses that exact route; -its orientation cannot be reused as a request. Consensus headers retain their -separate member-bearing routes and validators. This is a marker-1 hard cutover -for experimental query peers; incompatible headers are refused without an old -reader or translation path. - -Each `RaftCellRegistration` selects `RaftCellPlacement::Replica` or -`QueryRouter { leader_hint }`. A query router reads an existing namespace and -birth record, verifies the expected history/initial placement, and installs -discovery without creating a local member or SQLite image. It cannot reinterpret -existing member storage as a query-only binding. Its SDK handle rejects commands, -local reads and local status; use explicit `QueryRouting::Replicas`. The optional -hint is discovery only and cannot supply a read proof. + +The separate peer query listener has its own bounded queue, connection and byte +banks alongside control and LTX traffic. The same authenticated framing carries +cell queries, commands, enrollments and receipt lookups, with distinct purposes. +MAC domains, TLS exporter, schema, sequence, exact reversed response route, +request tag and payload digest remain checked. A route binds the physical group, +logical cell and exact node/disk/process endpoints. Namespace read permission +never grants command, enrollment, receipt mutation or consensus authority. + +The receiver executes a selected cell query locally. It cannot recursively route +the request. Queued/handshake time consumes the sender's original deadline before +it transmits the remaining timeout. Connection closure cancels the waiter while +entered physical work keeps its owner and admission until it completes. +The retired QueryRouting/QueryRouter SDK path, per-cell query wire bodies and +Cell history reader are removed. Marker 1 remains unchanged; old peers refuse. +Public SDK/fleet startup qualification is tracked in the +[cell-group progress record](../../../../docs/design/multi-raft/cell-groups/progress.md). `RaftCellsConfig::query_origins` supplies trusted `NamespaceQueryOrigin` grants for exact cluster/namespace/node/disk/process endpoints. A receiving replica @@ -975,10 +951,9 @@ supplies current trusted grants on restart; durable control-plane distribution belongs to the placement/operations controllers. Routing-only views and peer pools are bounded and never initialize a voter or learner. The [read audit](../../../../docs/design/multi-raft/read-audit.md) retains qualification -scope and remaining delivery work. The counter's native -and Axum HTTP paths expose consistency, tokens and bounded member options, and -their executable restart drill includes remote strong reads and cross-cell refusal. -These paths do not claim completion of MR-08 or the full delivery scope. +scope and remaining delivery work for its captured source. Older counter/Axum +and restart-drill fixtures still need the shared SDK cutover; their former member +routing APIs no longer describe the canonical query contract. Fresh status checks precede staging, submission and successful output. Transient protocol Ready work defers preflight or duplicate-receipt settlement under the @@ -1918,3 +1893,55 @@ after that join. Complete-group snapshot streaming, shared query RPC routing and production controller ownership of apply/command settlement remain unfinished. The TLS fixtures still drive runtime events themselves; they do not qualify controller ownership after an ingress caller disconnects. + +### Managed complete-group peer seeding + +`RaftTransport::prepare_managed_group_seed` requires the destination's rooted +FleetHostAuthorization and a fresh active physical learner reservation. It uses +the existing complete-group TLS seed exchange, receiver pools and original +invocation deadline. The authenticated source checkpoint must carry the exact +consumed permit in Seeding phase and the same repair plan. The source must be a +catalog-selected seed owner with the exact current disk/process boot and a voter +in the authenticated checkpoint. Revalidation cannot regress the catalog revision. +Catalog-read and handshake time are deducted before transmitting the remaining +source timeout. Catalog preflight, staging and joined-permit delivery share the +original transport deadline and process/shutdown cancellation. Queued engine +work retains its grant independently of a timed-out metadata waiter. +Unmanaged seed calls +refuse managed source metadata and managed namespace ownership. + +Staging retains the permanent member/node/disk/engine claim. The staged learner +is absent from the birth configuration and has no startup or serving authority. +Its base preserves the source's actual enrollment origin: a guard installed at +genesis remains a genesis guard; an imported group retains its later committed +enrollment through ordinary snapshot successor validation. Neither path can +retrofit a different origin or drop an installed fleet guard. + +The seed grant carries the exact local boot. The serialized engine worker checks +its last synced lifecycle guard when staging and immediately before the install +fence. Quiescence, stop acceptance and fencing refuse further installation. +Actual accepted engine work keeps its owner through sync; late/closed delivery +cannot erase an already persisted install receipt or report late startup success. + +`RangeDirectory::bootstrap_managed_joined_group_member` requires that durable +receipt, its permanent engine claim, the current rooted Live group and the exact +running local boot. This supports reconciliation after a lost install response +and restart after repair completion without replaying a saved active permit. +The returned permit repeats these checks before attachment. Fresh quorum and +application activation remain separate requirements. SDK/fleet startup, S3-based +managed learner recovery and native/backend qualification remain tracked in the +[cell-group progress record](../../../../docs/design/multi-raft/cell-groups/progress.md). + +Canonical startup prepares every selected local group root before publishing +physical routes. An explicit startup error joins controller/transport/SQL/runtime +and engine cleanup before returning, including actual storage-lock release. +Cancellation schedules that same cleanup on the owning execution domain. The +predecessor per-cell dispatcher is no longer compiled. Its used option/observation +contracts and enabled storage-policy regressions remain while the final legacy +verification sweep is incomplete. + +Old inbound per-cell seed and per-cell inspection RPC dispatch is removed. New +shared checkpoint seeding and physical-node/group inspection keep their existing +deadline, process, shutdown and bounded socket helpers. Retained outbound test +utilities do not provide an inbound compatibility reader. The library compiles; +unit/native and complete managed startup qualification are still in progress. diff --git a/crates/cellar/durability/raft/archive/coverage.rs b/crates/cellar/durability/raft/archive/coverage.rs deleted file mode 100644 index 8c5f7d909..000000000 --- a/crates/cellar/durability/raft/archive/coverage.rs +++ /dev/null @@ -1,239 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Physical verification of one exact immutable archive and fixed publication. -use super::*; -use crate::raft::host::{FleetImageState, FleetPublishedCut}; -use crate::raft::member_state::FleetGroupBinding; -use cellar_logic::consensus::LogIndex; -use std::time::Duration; - -/// Historical, physically replayed coverage of one selected immutable archive. -/// This supplies no current health, quorum, placement or removal authority. -/// A lifecycle owner must obtain fresh survivor observations and recheck its -/// rooted operation before committing a terminal receipt or releasing resources. -#[derive(Clone, Debug)] -pub struct VerifiedArchiveCoverage { - reference: ArchiveReference, - binding: FleetGroupBinding, - birth: [u8; 32], - schema: [u8; 32], - revision: LogIndex, - position: LogPosition, - image_position: Option, - image: Option, -} - -impl VerifiedArchiveCoverage { - fn capture(set: &ArchiveRecoverySet) -> Result { - let membership = set.membership(); - let status = membership - .fleet_status() - .ok_or(ArchiveError::Invalid("archive has no fleet enrollment"))?; - let published = set.published_image()?; - let image = published.as_ref().map(|image| image.state()); - // A full snapshot's applied position can include configuration-only - // progress. Preserve the producer's exact publication origin, including - // legitimate intact-restore aliases; never infer it from that snapshot. - let image_position = published.as_ref().map(|image| image.position()); - Ok(Self { - reference: set.reference(), - binding: status.binding, - birth: set.birth_digest(), - schema: set.schema_digest(), - revision: membership.revision(), - position: set.position(), - image_position, - image, - }) - } - - pub fn reference(&self) -> &ArchiveReference { - &self.reference - } - - pub fn binding(&self) -> &FleetGroupBinding { - &self.binding - } - - pub fn birth_digest(&self) -> [u8; 32] { - self.birth - } - - pub fn schema_digest(&self) -> [u8; 32] { - self.schema - } - - pub fn membership_revision(&self) -> LogIndex { - self.revision - } - - pub fn position(&self) -> LogPosition { - self.position - } - - /// None denotes a physically verified canonical-empty archive, never a - /// missing payload or a pending SQL publication. - pub fn image_state(&self) -> Option { - self.image - } - - pub fn image_position(&self) -> Option { - self.image_position - } - - /// Compare complete prefix coverage at the same exact fleet/history and - /// membership revision. A configuration change requires a new attempt. - pub fn covers(&self, minimum: &FleetPublishedCut) -> Result { - if &self.binding != minimum.binding() - || self.birth != minimum.birth_digest() - || self.schema != minimum.schema_digest() - || self.revision != minimum.membership_revision() - { - return Err(ArchiveError::Conflict); - } - if self - .image - .as_ref() - .zip(minimum.image()) - .is_some_and(|(a, b)| !a.same_representation(&b.state())) - { - return Err(ArchiveError::Corrupt("archive image commitment domain")); - } - if self.position.index < minimum.applied().index { - return Ok(false); - } - if !consistent(self.position, minimum.applied()) { - return Err(ArchiveError::Corrupt("archive publication prefix term")); - } - match (self.image, minimum.image()) { - (None, None) => Ok(true), - (None, Some(_)) => Err(ArchiveError::Corrupt("archive erased published image")), - (Some(state), prior) => { - let position = self - .image_position - .ok_or(ArchiveError::Corrupt("archive image position"))?; - if !consistent(position, minimum.applied()) - || prior.is_some_and(|image| !consistent(position, image.position())) - { - return Err(ArchiveError::Corrupt("archive data prefix term")); - } - if prior.is_some_and(|image| state == image.state()) { - return Ok(true); - } - if position.index <= minimum.applied().index { - return Err(ArchiveError::Corrupt("archive conflicting published image")); - } - Ok(true) - } - } - } -} - -fn consistent(a: LogPosition, b: LogPosition) -> bool { - match a.index.cmp(&b.index) { - std::cmp::Ordering::Equal => a.term == b.term, - std::cmp::Ordering::Less => a.term <= b.term, - std::cmp::Ordering::Greater => a.term >= b.term, - } -} - -impl CellArchive { - /// Download and physically replay exactly `reference`, then prove coverage - /// of a fixed quorum-backed publication cut. A missing selected object - /// refuses; this operation never substitutes a newer mutable head. - /// - /// The original deadline covers every object read, workspace wait and - /// replay. Blocking work retains its inputs until joined; expiry/cancellation - /// suppresses the proof even if that work finishes later. The caller owns - /// scratch storage and admission for the configured recovery-memory bound. - pub async fn verify_published_at( - &self, - reference: &ArchiveReference, - minimum: &FleetPublishedCut, - verifier: &ArchiveVerifier, - timeout: Duration, - ) -> Result { - let millis = u64::try_from(timeout.as_millis()).map_err(|_| ArchiveError::Limit)?; - if !(1..=60_000).contains(&millis) || timeout > Duration::from_secs(60) { - return Err(ArchiveError::Invalid("archive verification timeout")); - } - let mut archive = self.clone(); - let now = archive - .io_domain - .as_ref() - .ok_or(ArchiveError::Stopped)? - .mono_ms() - .map_err(|_| ArchiveError::Stopped)?; - let deadline = now.checked_add(millis).ok_or(ArchiveError::Limit)?; - archive.verification_deadline = Some( - archive - .verification_deadline - .map_or(deadline, |previous| previous.min(deadline)), - ); - archive.check_verification()?; - let metadata = archive.load_metadata(&reference.0).await?; - let lease = if matches!(&metadata.inner, super::representation::Metadata::Rooted(_)) { - let lease = verifier.begin_extent()?; - archive.root_store = Some(lease.store()); - archive.root_verification = Some(lease.clone()); - if lease.needs_recovery() { - let worker = archive.clone(); - let owner = lease.clone(); - archive - .host - .run_blocking(move || owner.reconcile(|| worker.check_verification())) - .await - .map_err(ArchiveError::Worker)??; - } - Some(lease) - } else { - None - }; - let set = archive.load_recovery_material(metadata).await?; - let worker = archive.clone(); - let verifier = verifier.clone(); - let minimum = minimum.clone(); - let proof = archive - .host - .run_blocking(move || { - worker.check_verification()?; - let proof = VerifiedArchiveCoverage::capture(&set)?; - let coverage = proof.covers(&minimum); - match set.inner { - super::representation::Recovery::Materialized(set) => { - if !coverage? { - return Err(ArchiveError::Unconfirmed); - } - verifier.verify_checked(&set, || worker.check_verification())?; - } - super::representation::Recovery::Rooted(set) => { - lease - .ok_or(ArchiveError::Invalid("extent verifier lease required"))? - .finish(set, || worker.check_verification())?; - if !coverage? { - return Err(ArchiveError::Unconfirmed); - } - } - } - worker.check_verification()?; - Ok(proof) - }) - .await - .map_err(ArchiveError::Worker)??; - // Delivery can be delayed after the worker completed, including by an - // unpolled caller. A completed replay is no exception to this deadline. - archive.check_verification()?; - Ok(proof) - } - - fn check_verification(&self) -> Result<(), ArchiveError> { - if self - .cancellation - .as_ref() - .is_some_and(|cancel| *cancel.borrow() || cancel.has_changed().is_err()) - { - return Err(ArchiveError::Stopped); - } - self.deadline().map(|_| ()) - } -} diff --git a/crates/cellar/durability/raft/archive/mod.rs b/crates/cellar/durability/raft/archive/mod.rs index 543482c15..2a296c68e 100644 --- a/crates/cellar/durability/raft/archive/mod.rs +++ b/crates/cellar/durability/raft/archive/mod.rs @@ -23,7 +23,6 @@ use std::{collections::BTreeMap, fmt, sync::Arc}; pub(crate) mod bundle; mod configuration; -mod coverage; mod physical; mod publication; mod replay; @@ -41,8 +40,6 @@ pub use retention::{ pub use rooted::{RootArchiveReference, RootArchiveSnapshot}; pub use verifier::ArchiveVerifier; -pub use coverage::VerifiedArchiveCoverage; - const READ_WINDOW: usize = 4; pub(crate) const fn physical_read_bytes() -> usize { diff --git a/crates/cellar/durability/raft/catalog.rs b/crates/cellar/durability/raft/catalog.rs index 214162845..ce1cf1c6e 100644 --- a/crates/cellar/durability/raft/catalog.rs +++ b/crates/cellar/durability/raft/catalog.rs @@ -27,8 +27,8 @@ mod range_directory; pub(crate) use range_directory::RecoverySeedPermit; pub(crate) use range_directory::RouteCache; pub use range_directory::{ - CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthDisposition, CellGroupBirthRequest, + CellGroupBirthSpec, CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, @@ -39,7 +39,9 @@ pub use range_directory::{ #[path = "catalog_inventory/mod.rs"] mod inventory; -pub(crate) use inventory::{BirthImportCut, ManagedBirthInventory}; +pub(crate) use inventory::groups::group_key as cell_group_import_key; +pub(crate) use inventory::groups::CellGroupImportCut; +pub use inventory::groups::FrozenCellGroupInventory; pub use inventory::{ BirthInventoryCursor, BirthInventoryEntry, BirthInventoryPage, FrozenBirthInventory, }; @@ -63,6 +65,8 @@ pub enum CatalogError { BirthCancelled, BirthsFrozen, ManagedAdmissionRequired, + /// Preserve the owning fleet's conflict, pressure and uncertain outcome. + FleetAuthority(Box), LostStableState, Unconfirmed, Storage(object_store::Error), @@ -80,7 +84,20 @@ impl fmt::Display for CatalogError { } } -impl std::error::Error for CatalogError {} +impl std::error::Error for CatalogError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + match self { + Self::FleetAuthority(error) => Some(error.as_ref()), + _ => None, + } + } +} + +impl From for CatalogError { + fn from(error: super::fleet_catalog::FleetCatalogError) -> Self { + Self::FleetAuthority(Box::new(error)) + } +} #[derive(Clone, Debug)] pub struct CatalogOptions { @@ -496,34 +513,6 @@ impl NamespaceCatalog { self.bootstrap_initial(birth, log, None).await } - pub(crate) async fn bootstrap_managed_member( - &self, - birth: &GroupBirth, - log: &ReplicaLog, - authorization: &super::fleet_catalog::FleetGroupAuthorization, - ) -> Result { - if !birth - .managed_owner() - .is_some_and(|owner| owner.fleet == authorization.binding().fleet()) - || !authorization.check_birth(self, birth) - || birth.voters.len() != 3 - || !birth.learners.is_empty() - { - return Err(CatalogError::Conflict( - "managed initial birth authorization", - )); - } - if !authorization.allows_managed_genesis() { - match log.group(birth.scope()) { - Ok(_) => (), - Err(LogError::NotFound) => return Err(CatalogError::ManagedAdmissionRequired), - Err(error) => return Err(CatalogError::Log(error)), - } - } - self.bootstrap_initial(birth, log, Some(authorization.binding())) - .await - } - async fn bootstrap_initial( &self, birth: &GroupBirth, diff --git a/crates/cellar/durability/raft/catalog_birth.rs b/crates/cellar/durability/raft/catalog_birth.rs index fcea2cd92..106f087ac 100644 --- a/crates/cellar/durability/raft/catalog_birth.rs +++ b/crates/cellar/durability/raft/catalog_birth.rs @@ -62,46 +62,6 @@ impl NamespaceCatalog { .await } - /// A reservation remains inventoried even if drain wins a later revision. - /// Cancellation contends on the same birth key and never erases a live birth. - pub async fn cancel_managed_birth( - &self, - authorization: &super::super::fleet_catalog::FleetBirthAuthorization, - operation: ProposalId, - timeout: Duration, - ) -> Result { - authorization.check_namespace(self)?; - self.cancel_selected_birth( - authorization.proposal(), - operation, - timeout, - Some(authorization.owner()), - ) - .await - } - - pub async fn resolve_managed_birth( - &self, - authorization: &super::super::fleet_catalog::FleetBirthAuthorization, - timeout: Duration, - ) -> Result { - authorization.check_namespace(self)?; - if timeout.is_zero() || timeout > Duration::from_secs(60) { - return Err(CatalogError::Invalid("managed birth timeout")); - } - let birth = crate::asyncrt::timeout( - timeout, - self.select_birth(authorization.proposal(), None, Some(authorization.owner())), - ) - .await - .map_err(|_| CatalogError::Unconfirmed)?? - .live()?; - if !authorization.check_birth(self, &birth) { - return Err(CatalogError::Conflict("reserved birth winner")); - } - Ok(birth) - } - async fn cancel_selected_birth( &self, proposal: BirthProposal, diff --git a/crates/cellar/durability/raft/catalog_inventory/groups.rs b/crates/cellar/durability/raft/catalog_inventory/groups.rs new file mode 100644 index 000000000..6c87defc5 --- /dev/null +++ b/crates/cellar/durability/raft/catalog_inventory/groups.rs @@ -0,0 +1,417 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Physical-group admission shares the namespace CAS with disaster exclusion. +//! Frozen intents include unpublished groups; listing never defines coverage. +use super::*; +use crate::raft::catalog::ManagedBirthOwner; +use crate::raft::cell_groups::{CellGroupBirthRequest, RangeDirectory}; +use serde::{Deserialize, Serialize}; + +const CUT_KIND: u8 = 196; + +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub(crate) struct CellGroupImportCut { + pub root: [u8; 32], + pub operation: [u8; 16], + pub revision: u64, + pub count: u64, + #[serde(deserialize_with = "required_index")] + pub index: Option<[u8; 32]>, + pub directory: [u8; 32], +} + +fn required_index<'de, D: serde::Deserializer<'de>>(d: D) -> Result, D::Error> { + Option::deserialize(d) +} + +impl CellGroupImportCut { + pub(crate) fn validate(&self, namespace: &NamespaceCatalog) -> Result<(), CatalogError> { + if self.root != namespace.root_digest + || self.revision <= self.count + || self.revision < 3 + || self.index.is_none() != (self.count == 0) + || self.directory == [0; 32] + || namespace.config.managed_fleet.is_some() + || ProposalId::new(self.operation).is_err() + { + return Err(CatalogError::Corrupt("group import cut")); + } + Ok(()) + } + + pub(crate) fn digest(&self) -> Result<[u8; 32], CatalogError> { + Ok(record::digest(&record::encode(CUT_KIND, self, 4096)?)) + } +} + +/// A frozen, authoritative admission set. This is neither complete enrollment +/// nor permission to start, activate, remove or release a physical replica. +#[derive(Clone)] +pub struct FrozenCellGroupInventory { + owner: Arc, + cut: CellGroupImportCut, +} + +impl FrozenCellGroupInventory { + pub fn len(&self) -> u64 { + self.cut.count + } + + pub fn is_empty(&self) -> bool { + self.cut.count == 0 + } + + pub fn operation(&self) -> ProposalId { + // Construction validates the durable cut. Retain no unchecked public + // constructor, including for a resumed inventory. + ProposalId::new(self.cut.operation).expect("validated group inventory operation") + } + + pub(crate) fn import_cut( + &self, + directory: &RangeDirectory, + ) -> Result { + if !Arc::ptr_eq(&self.owner, &directory.namespace().inner) { + return Err(CatalogError::Conflict("foreign frozen group inventory")); + } + self.cut.validate(directory.namespace())?; + Ok(self.cut.clone()) + } +} + +pub(crate) struct ImportCellGroupStep { + pub entry: Option, + pub next: Option, +} + +pub(crate) fn group_key(request: &CellGroupBirthRequest) -> String { + hex::encode(record::digest(&request.route().history.group.bytes())) +} + +impl NamespaceCatalog { + pub(crate) async fn cell_group_owner(&self) -> Result, CatalogError> { + let (head, _) = Store(self).current().await?; + Ok(match head.mode { + Mode::Owned { fleet, catalog } + | Mode::GroupImporting { fleet, catalog, .. } + | Mode::GroupAdopted { fleet, catalog, .. } => { + Some(ManagedBirthOwner { fleet, catalog }) + } + Mode::Open | Mode::GroupFreezing { .. } | Mode::GroupFrozen { .. } => None, + _ => return Err(CatalogError::Conflict("not a shared-group namespace")), + }) + } + + pub(crate) async fn require_cell_group_unmanaged(&self) -> Result<(), CatalogError> { + if self.cell_group_owner().await?.is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(()) + } + + /// Called after the permanent global claim, before any topology publication + /// or materialization. A losing CAS can retain a harmless identity claim; + /// without this admission it can never become a group birth. + pub(crate) async fn admit_cell_group( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + let store = Store(self); + let index = Index::new(&store); + let key = group_key(request); + let id = request.route().birth_digest; + if request.route().history.cluster != self.config.cluster + || request.route().history.namespace != self.config.namespace + || request.route().schema_digest != self.config.schema_digest + || record::digest(request.encode()) != id + { + return Err(CatalogError::Conflict("group admission namespace")); + } + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if let Some(prior) = index.lookup(head.index, &key).await? { + if pointer(&prior)? != id { + return Err(CatalogError::Conflict("group admission identity")); + } + return self.verify_cell_group_intent(request).await; + } + match head.mode { + Mode::Open => (), + Mode::GroupFreezing { .. } + | Mode::GroupFrozen { .. } + | Mode::GroupImporting { .. } + | Mode::Frozen { .. } + | Mode::Importing { .. } => return Err(CatalogError::BirthsFrozen), + _ => return Err(CatalogError::ManagedAdmissionRequired), + } + let intent_key = store.key(&format!("group-intents/{}", hex::encode(id))); + if self + .inner + .create_and_observe_bounded(&intent_key, request.encode(), INDEX_PAGE_LIMIT) + .await? + != request.encode() + { + return Err(CatalogError::Corrupt("group admission intent collision")); + } + head.index = index.update(head.index, &key, Some(id.to_vec()), 0).await?; + head.count = head.count.checked_add(1).ok_or(CatalogError::Limit)?; + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + let (observed, _) = store.current().await?; + if index.lookup(observed.index, &key).await?.as_deref() == Some(id.as_slice()) { + return self.verify_cell_group_intent(request).await; + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn verify_cell_group_intent( + &self, + request: &CellGroupBirthRequest, + ) -> Result<(), CatalogError> { + let store = Store(self); + let (head, _) = store.current().await?; + let value = Index::new(&store) + .lookup(head.index, &group_key(request)) + .await? + .ok_or(CatalogError::Corrupt("untracked group birth"))?; + let id = pointer(&value)?; + let bytes = self + .inner + .read_versioned_bounded( + &store.key(&format!("group-intents/{}", hex::encode(id))), + INDEX_PAGE_LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing group intent"))? + .0; + if id != request.route().birth_digest + || record::digest(&bytes) != id + || bytes != request.encode() + { + return Err(CatalogError::Corrupt("group intent identity")); + } + Ok(()) + } + + pub(crate) async fn begin_cell_group_freeze( + &self, + operation: ProposalId, + ) -> Result<(), CatalogError> { + let store = Store(self); + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if head.disaster { + return Err(CatalogError::Conflict( + "disaster namespace cannot be imported", + )); + } + match &head.mode { + Mode::GroupFreezing { operation: prior } if *prior == operation.bytes() => { + return Ok(()); + } + Mode::GroupFrozen { cut } if cut.operation == operation.bytes() => return Ok(()), + Mode::Open => (), + _ => return Err(CatalogError::Conflict("group freeze owner/operation")), + } + head.mode = Mode::GroupFreezing { + operation: operation.bytes(), + }; + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode == head.mode { + return Ok(()); + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn finish_cell_group_freeze( + &self, + operation: ProposalId, + directory: [u8; 32], + ) -> Result { + let store = Store(self); + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if let Mode::GroupFrozen { cut } = &head.mode { + if cut.operation == operation.bytes() && cut.directory == directory { + cut.validate(self)?; + return Ok(FrozenCellGroupInventory { + owner: self.inner.clone(), + cut: cut.clone(), + }); + } + } + if head.mode + != (Mode::GroupFreezing { + operation: operation.bytes(), + }) + { + return Err(CatalogError::Conflict("group freeze continuation")); + } + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + let cut = CellGroupImportCut { + root: self.root_digest, + operation: operation.bytes(), + revision: head.revision, + count: head.count, + index: head.index, + directory, + }; + cut.validate(self)?; + head.mode = Mode::GroupFrozen { cut }; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode != head.mode { + result?; + } + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn change_cell_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + cut.validate(self)?; + let store = Store(self); + let desired = if adopted { + Mode::GroupAdopted { + fleet: owner.fleet, + catalog: owner.catalog, + cut: cut.clone(), + } + } else { + Mode::GroupImporting { + fleet: owner.fleet, + catalog: owner.catalog, + cut: cut.clone(), + } + }; + for _ in 0..MAX_ATTEMPTS { + let (mut head, token) = store.current().await?; + if head.mode == desired + || !adopted + && matches!(&head.mode, + Mode::GroupAdopted { fleet, catalog, cut: prior } + if *fleet == owner.fleet && *catalog == owner.catalog && prior == cut) + { + return Ok(()); + } + let allowed = if adopted { + matches!(&head.mode, Mode::GroupImporting { fleet, catalog, cut: prior } + if *fleet == owner.fleet && *catalog == owner.catalog && prior == cut) + } else { + matches!(&head.mode, Mode::GroupFrozen { cut: prior } if prior == cut) + }; + if !allowed { + return Err(CatalogError::Conflict("group import owner/cut")); + } + head.mode = desired.clone(); + head.revision = head.revision.checked_add(1).ok_or(CatalogError::Limit)?; + head.validate(self)?; + let result = store.cas(&head, token).await; + if store.current().await?.0.mode == desired { + return Ok(()); + } + result?; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) async fn check_cell_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + cut.validate(self)?; + let valid = match Store(self).current().await?.0.mode { + Mode::GroupAdopted { + fleet, + catalog, + cut: prior, + } => fleet == owner.fleet && catalog == owner.catalog && prior == *cut, + Mode::GroupImporting { + fleet, + catalog, + cut: prior, + } if !adopted => fleet == owner.fleet && catalog == owner.catalog && prior == *cut, + _ => false, + }; + if !valid { + return Err(CatalogError::Conflict("group import marker")); + } + Ok(()) + } + + pub(crate) async fn cell_group_import_entry( + &self, + directory: &RangeDirectory, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + key: &str, + ) -> Result { + self.check_cell_group_import(cut, owner, false).await?; + let store = Store(self); + let pointer = Index::new(&store) + .lookup(cut.index, key) + .await? + .ok_or(CatalogError::Corrupt("untracked group import witness"))?; + let id = super::pointer(&pointer)?; + let bytes = self + .inner + .read_versioned_bounded( + &store.key(&format!("group-intents/{}", hex::encode(id))), + INDEX_PAGE_LIMIT, + ) + .await? + .ok_or(CatalogError::Corrupt("missing group import intent"))? + .0; + if record::digest(&bytes) != id { + return Err(CatalogError::Corrupt("group import intent digest")); + } + let request = directory.decode_group_birth(&bytes)?; + if request.route().birth_digest != id || group_key(&request) != key { + return Err(CatalogError::Corrupt("group import index identity")); + } + Ok(request) + } + + pub(crate) async fn cell_group_import_step( + &self, + directory: &RangeDirectory, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + after: Option<&str>, + ) -> Result { + self.check_cell_group_import(cut, owner, false).await?; + let cursor = after + .map(|key| IndexCursor::after(cut.index, key.to_owned())) + .transpose()?; + let page = Index::new(&Store(self)) + .scan(cut.index, cursor.as_ref(), 2) + .await?; + let more = page.entries.len() > 1; + let Some(first) = page.entries.into_iter().next() else { + return Ok(ImportCellGroupStep { + entry: None, + next: None, + }); + }; + let entry = self + .cell_group_import_entry(directory, cut, owner, &first.key) + .await?; + Ok(ImportCellGroupStep { + entry: Some(entry), + next: more.then_some(first.key), + }) + } +} diff --git a/crates/cellar/durability/raft/catalog_inventory/mod.rs b/crates/cellar/durability/raft/catalog_inventory/mod.rs index 21f649bd6..fbe2de42b 100644 --- a/crates/cellar/durability/raft/catalog_inventory/mod.rs +++ b/crates/cellar/durability/raft/catalog_inventory/mod.rs @@ -10,9 +10,9 @@ use object_store::{path::Path, PutMode, PutOptions, UpdateVersion}; use std::{sync::Arc, time::Duration}; mod current; +pub(crate) mod groups; mod import; mod wire; -pub(crate) use current::ManagedBirthInventory; pub(crate) use import::BirthImportCut; use wire::{Head, Mode, Pin}; diff --git a/crates/cellar/durability/raft/catalog_inventory/wire.rs b/crates/cellar/durability/raft/catalog_inventory/wire.rs index 9f33b072a..25273573d 100644 --- a/crates/cellar/durability/raft/catalog_inventory/wire.rs +++ b/crates/cellar/durability/raft/catalog_inventory/wire.rs @@ -22,6 +22,22 @@ pub(super) enum Mode { catalog: [u8; 32], cut: BirthImportCut, }, + GroupFreezing { + operation: [u8; 16], + }, + GroupFrozen { + cut: groups::CellGroupImportCut, + }, + GroupImporting { + fleet: [u8; 16], + catalog: [u8; 32], + cut: groups::CellGroupImportCut, + }, + GroupAdopted { + fleet: [u8; 16], + catalog: [u8; 32], + cut: groups::CellGroupImportCut, + }, } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -68,6 +84,49 @@ impl Head { return Err(CatalogError::Corrupt("birth inventory identity/count")); } match (&self.mode, &namespace.config.managed_fleet) { + (Mode::GroupFreezing { operation }, None) if self.revision > 1 => { + ProposalId::new(*operation) + .map_err(|_| CatalogError::Corrupt("group freeze operation"))?; + } + (Mode::GroupFrozen { cut }, None) => { + cut.validate(namespace)?; + if self.revision != cut.revision + || self.count != cut.count + || self.index != cut.index + { + return Err(CatalogError::Corrupt("changed frozen group set")); + } + } + ( + Mode::GroupImporting { + fleet, + catalog, + cut, + }, + None, + ) + | ( + Mode::GroupAdopted { + fleet, + catalog, + cut, + }, + None, + ) => { + cut.validate(namespace)?; + validate_owner(*fleet, *catalog)?; + let delta = if matches!(self.mode, Mode::GroupAdopted { .. }) { + 2 + } else { + 1 + }; + if self.revision != cut.revision.checked_add(delta).ok_or(CatalogError::Limit)? + || self.count != cut.count + || self.index != cut.index + { + return Err(CatalogError::Corrupt("changed imported group set")); + } + } (Mode::Open, None) => (), (Mode::Owned { fleet, catalog }, Some(owner)) => { if fleet != &owner.fleet || catalog != &owner.catalog { diff --git a/crates/cellar/durability/raft/catalog_wire.rs b/crates/cellar/durability/raft/catalog_wire.rs index 6fe705b88..154efc89d 100644 --- a/crates/cellar/durability/raft/catalog_wire.rs +++ b/crates/cellar/durability/raft/catalog_wire.rs @@ -220,6 +220,7 @@ mod tests { let mut root = Root::new(&config, "test".into()); root.validate().unwrap(); for protocol in [ + "cellar-raft-ltx-1/members-1/seeds-1/s3-1/repairs-1/nodes-1/bundles-1/fleet-1/enrollment-1/node-probes-1/birth-admission-1/managed-genesis-1/group-reservations-1/birth-inventory-1/namespace-import-1/fleet-namespaces-1/catalog-trust-1/fleet-admission-1/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1/fleet-stop-1/fleet-stop-rpc-1/capacity-readiness-1/resource-reservations-1/physical-inventory-1/fleet-removal-1/resource-movement-1/movement-completion-1/node-isolation-1/isolated-removal-1/images-1/fenced-movement-1/fleet-execution-1/aborted-movement-1/topology-bindings-1/topology-intent-1/config-tails-1/restart-stop-1/restart-return-1/restart-resume-1/maintenance-operator-1/stop-consent-refresh-1/extent-owners-1/operator-lifecycle-1/extent-fleet-proofs-1/failure-recovery-1", "cellar-raft-ltx-1", "cellar-raft-ltx-1/members-1", "cellar-raft-ltx-1/members-1/seeds-1", diff --git a/crates/cellar/durability/raft/cell_groups/README.md b/crates/cellar/durability/raft/cell_groups/README.md index d11e05dbc..737248c0b 100644 --- a/crates/cellar/durability/raft/cell_groups/README.md +++ b/crates/cellar/durability/raft/cell_groups/README.md @@ -495,8 +495,58 @@ output resources without quarantining a group whose SQL producer already closed. Shared log-engine I/O/corruption/closure and lost shared SQL worker replies still close the node. Unknown events and controller ownership violations fail closed. -Request-driven group provisioning, configuration lifecycle and public SDK -integration remain unfinished. See the delivery package for qualification status. +Initial-member provisioning and canonical startup use the same physical page +inventory under the configured node directory: +`groups/////pages`. Their common path +belongs to `ManifestPageStore`; both take its exclusive lock and recover the +engine-selected root. Independent archive replay uses its own node base with the +same layout. A restart must refuse a missing selected page rather than open a +second empty inventory. Provisioning fault/restart fixtures target this canonical +path for both image backends. No previous flat-path reader or migration is added. + +Provisioning binds immutable `CellGroupProvisionNamespace` profiles to exact +`RangeDirectory` owners and the declared image representation. Canonical startup +clones its node image inventories into these profiles. Duplicate namespaces, +foreign directory owners and mismatched backends refuse; a request cannot select +another image store. Profile caches and fixed peer metadata count once against +the configured preparation-byte total. Every namespace then shares the remaining +preparation bank and parallel worker limit. `provision_configuration_bytes` +reports retained configuration separately from active preparations. + +`activate_managed_initial_group` uses the rooted group birth and actual initial +voter identity. It refreshes catalog authorization and checks the exact synced +boot before submission and again after local attachment. The controller retains +preparation, selected pages, roster and transport ownership when a waiter leaves; +retries reuse prepared owners. Entered physical work is joined even after its +invocation expires. An attached retry revalidates under the same shared budget +without reopening pages or attaching consensus again; it does not adopt another +registration's physical lifecycle. Shared-engine uncertainty still closes the +node. An attachment receipt describes the persisted fleet guard; it supplies no +fresh quorum, seed, serving or physical-release authority. Quorum readiness is +checked independently by the existing Raft owner. + +Managed group repair controls take both the rooted physical history and its +exact admitted permit. They inspect a fresh quorum, preserve the complete plan, +consume/advance only the matching permit and revision, and observe durable apply +behind a new quorum before reporting success. Beginning placement checks the +installed trust revision, original target boot and physical-host separation over +all current occupants, including members omitted from advisory routing. Promotion +still uses actual RawNode catch-up; seed receipts and route discovery cannot +substitute for progress. Lost configuration replies remain unconfirmed. + +After committed configuration changes, managed routing can resolve newly joined +member/disk bindings through rooted node views. Missing discovery leaves known +voters usable; it grants no initialization, vote, quorum or release authority. +Ordinary cell writes do not scan routes. Ordinary frames retry discovery once +only for an unknown destination; checkpoint sends also reconcile before capture. +Advisory refresh preserves the physical registration and checkpoint factory, +while each transfer pins an exact route generation. Old stream generations still +refuse. Per-chunk validation allocates no new endpoint. Explicit replacement or +removal revokes the old factory; an old registration cannot remove its successor. +All work retains the existing peer queues, admission and physical join owners. + +Managed learner activation and full managed canonical startup remain unfinished. +See the delivery package for qualification status. ### Retained entry admission @@ -594,3 +644,64 @@ before send, retains the receiver's S3 policy, and independently restores the fi complete image/receipt closure. It has an explicit local RustFS variant without a memory fallback. Current results and remaining qualification are recorded in the [implementation progress](../../../../../docs/design/multi-raft/cell-groups/progress.md). + +Managed peer learner activation enters the same retained provisioning manager as +initial group startup. An exact original-boot permit selects the joining member; +the authenticated seed exchange independently checks that same permit and obtains +a fresh source quorum grant. The manager saves the recovered page/applier owner +before starting installation and retains it after an unknown result. A durable +seed receipt resumes through the permanent joined claim and selected physical +closure; a staged mapping cannot bootstrap genesis. The parallel preparation +permit spans recovery and installation. Attachment remains separate from learner +promotion and application serving. + +The SDK and movement receiver now use this whole-group peer path. S3 learner +activation currently refuses before submission: exact-reference archive transfer +has not yet been integrated. These new managed paths have no executed native or +RustFS qualification while the ordinary library still has legacy consumers to +remove. + +The canonical group host now joins physical cleanup before an explicit startup +error returns and withholds physical routes until all local group roots have +attached. Its cancellation guard retains the same cleanup on the execution +domain. The two existing startup regressions now use real shared groups, engines +and TLS endpoints; their original lock-release and incomplete-inventory traffic +oracles remain. Their execution is not yet qualified. + +Canonical archive startup creates and syncs its node-owned archive spool directory +on the execution domain before registering an archive job. Materialized checkpoint +receivers open anonymous files there under the existing shared scratch admission; +they do not create a missing caller-supplied parent. Missing image data and failed +scratch preparation still refuse S3 admission rather than changing durability. + +A provisioning retry may reuse its retained, completely verified selected root +without scanning or syncing it again. Reuse checks the same live engine and page +owners, exact selected marker, settled publication and complete checkpoint pins. +Pending or quarantined retention refuses; marker equality cannot resolve an +uncertain write. Changed durable selection recovers on the same page owner. + +A peer seed requires a live source grant through the engine's pre-fence admission +check. After actual engine persistence succeeds, local completion checks the +receiving engine/member/disk/process, managed boot, transport shutdown, original +deadline and fresh catalog bootstrap. A later source restart does not revoke that +completed installation. Neither its receipt nor bootstrap establishes fresh +quorum or serving authority; an expired completion is refused even when cold +recovery later finds the installed root. + +Shared linearizable cell reads may be selected on followers after their own safe +ReadIndex round. Their private forwarded context binds the authenticated origin +member, process-local nonce and sequence, separately from membership metadata and +leader-local seed/execution contexts. Selection waits for complete local apply +through the correlated barrier. SQL rechecks the selected history, live member, +term and source generation; a later term/leader change or shutdown revokes the +retained proof. This changes read availability, not leader-only mutation or seed +admission. SQL results describe the actual selecting replica. + +Recent-source reads use an explicit age bound for the observed current source. +A stale-term message or catalog refresh cannot renew that observation. The selected +proof retains its original source expiry through queued and entered SQL work; +fresh later traffic cannot extend it. SDK stale reads use the injected host source +age by default; disconnected historical reads require explicit selection. Neither +policy supplies a quorum proof. The default request deadline and physical query +retention still apply. Public replica-candidate routing and counter/performance +entrypoint conversion remain separate, unfinished CG-07 work. diff --git a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs index d8d6ca031..8b776b3e0 100644 --- a/crates/cellar/durability/raft/cell_groups/apply/recovery.rs +++ b/crates/cellar/durability/raft/cell_groups/apply/recovery.rs @@ -47,6 +47,34 @@ impl CellGroupApplier { Ok(self.recover_roots(&mut owner)?.0) } + /// A retry may retain the exact complete closure. Reuse only with the same + /// live page/engine owners and settled selected/checkpoint retention. A + /// changed durable selection is recovered on this owner as before. + pub(in crate::raft::cell_groups) fn recover_or_reuse_selected( + &self, + previous: &PreparedGroupManifest, + ) -> Result { + let mut owner = self + .storage + .lock() + .map_err(|_| ExecutionError::Invalid("poisoned recovery owner"))?; + owner.pages.require_live().map_err(storage)?; + if !self.belongs_to_engine(&previous.engine_owner) + || self.history() != previous.marker().route().history + || !Arc::ptr_eq(&owner.pages.owner, &previous.page_owner) + { + return Err(ExecutionError::Invalid("retained recovery owner")); + } + if self.group.group_manifest().map_err(storage)?.as_ref() == Some(previous.marker()) { + previous.require_selected(&self.group).map_err(storage)?; + if self.group.group_manifest().map_err(storage)?.as_ref() != Some(previous.marker()) { + return Err(ExecutionError::Busy); + } + return Ok(previous.clone()); + } + Ok(self.recover_roots(&mut owner)?.0) + } + pub(in crate::raft::cell_groups) fn recover_roots( &self, owner: &mut Storage, diff --git a/crates/cellar/durability/raft/cell_groups/archive/fleet.rs b/crates/cellar/durability/raft/cell_groups/archive/fleet.rs new file mode 100644 index 000000000..8767e9d90 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/fleet.rs @@ -0,0 +1,95 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Fleet coverage consumes complete independently recovered group closures. +//! A selected object, uploaded bytes or a single cell image is insufficient. +use super::*; +use crate::raft::{host::FleetPublishedCut, member_state::FleetGroupBinding}; +use cellar_logic::consensus::{LogIndex, LogPosition}; + +#[derive(Clone)] +pub struct VerifiedFleetGroupArchiveCoverage { + coverage: CellGroupArchiveCoverage, + binding: FleetGroupBinding, + configuration_digest: [u8; 32], +} + +impl VerifiedFleetGroupArchiveCoverage { + pub fn hold(&self) -> &CellGroupArchiveHold { + self.coverage.hold() + } + + pub fn root(&self) -> &GroupManifest { + self.coverage.marker() + } + + pub fn binding(&self) -> &FleetGroupBinding { + &self.binding + } + + pub fn position(&self) -> LogPosition { + self.root().position() + } + + pub fn membership_revision(&self) -> LogIndex { + LogIndex::new(self.root().membership_revision()) + } + + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest + } + + pub fn covers(&self, minimum: &FleetPublishedCut) -> Result { + if self.binding() != minimum.binding() + || self.configuration_digest != minimum.configuration_digest() + { + return Err(CellGroupArchiveError::Conflict); + } + crate::raft::host::group_prefix(self.root(), minimum.root()) + .map_err(|_| CellGroupArchiveError::Conflict) + } +} + +impl CellGroupArchiveRegistry { + /// Re-read the exact persisted hold, independently restore every checkpoint + /// dependency and replay the contiguous tail, then compare the full root. + /// The execution owner retains entered physical work after cancellation. + /// The caller must enforce its original deadline around this read/replay and + /// suppress late delivery; a saved proof is historical, never a health lease. + pub async fn verify_fleet_published_at( + &self, + domain: &impl CellGroupArchiveExecution, + tails: &CellGroupArchiveTailPool, + hold: &CellGroupArchiveHold, + minimum: &FleetPublishedCut, + receiver: CellGroupArchiveCheckpointReceiver, + ) -> Result { + let coverage = self.verify_hold(domain, hold, tails, receiver).await?; + let membership = coverage.recovery().membership(); + let status = membership + .fleet_status() + .ok_or(CellGroupArchiveError::Invalid( + "archive has no fleet enrollment", + ))?; + if membership.revision().get() != coverage.marker().membership_revision() + || !status + .binding + .matches_scope(&coverage.marker().route().history) + { + return Err(CellGroupArchiveError::Invalid("fleet archive membership")); + } + let configuration_digest = crate::raft::host::group_membership_digest( + membership, + coverage.recovery().configuration(), + ) + .map_err(|_| CellGroupArchiveError::Invalid("fleet archive configuration commitment"))?; + let proof = VerifiedFleetGroupArchiveCoverage { + coverage, + binding: status.binding, + configuration_digest, + }; + if !proof.covers(minimum)? { + return Err(CellGroupArchiveError::Unconfirmed); + } + Ok(proof) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/archive/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/mod.rs index 1477d606d..7b717c570 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/mod.rs @@ -21,7 +21,9 @@ pub(super) use execution::ControllerExecution; mod checkpoint; mod data_store; +mod fleet; pub use data_store::CellGroupArchiveStore; +pub use fleet::VerifiedFleetGroupArchiveCoverage; mod collection; mod graph; mod objects; @@ -34,6 +36,7 @@ pub use upload::{ CellGroupArchiveUpload, CellGroupArchiveUploadLimits, CellGroupArchiveUploadPool, }; mod ownership; +pub(crate) use ownership::CellGroupArchiveAcquireOutcome; pub use ownership::{ CellGroupArchiveCreation, CellGroupArchiveHold, CellGroupArchiveIntent, CellGroupArchiveRegistry, CellGroupArchiveUploadTicket, PreparedCellGroupArchiveCreation, diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs index 135ab3ae7..d2ab28e62 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/mod.rs @@ -122,6 +122,29 @@ impl CellGroupArchiveHold { pub fn marker(&self) -> &GroupManifest { &self.root.marker } + + /// Descriptor shape only. Persistent registry membership must still be + /// checked by the owning registry before and after physical verification. + pub(crate) fn check_reference(&self) -> Result<()> { + self.root + .marker + .check() + .map_err(CellGroupCheckpointError::from)?; + if self.registry == [0; 32] + || self.allocation == [0; 32] + || self.id == 0 + || self.id >= u64::MAX - 2 + || self.root.object.registry != self.registry + || self.root.object.digest == [0; 32] + || self.root.object.generation < 2 + || self.root.object.generation >= u64::MAX - 2 + || self.root.object.length == 0 + || self.root.object.length > 8192 + { + return Err(CellGroupArchiveError::Invalid("archive hold descriptor")); + } + Ok(()) + } } #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] @@ -152,6 +175,14 @@ pub struct CellGroupArchiveIntent { } impl CellGroupArchiveIntent { + pub(crate) fn acquire_marker(&self) -> Result<&GroupManifest> { + self.digest()?; + match &self.action { + Action::Acquire { root } => Ok(&root.marker), + _ => Err(CellGroupArchiveError::Invalid("acquire intent purpose")), + } + } + fn digest(&self) -> Result<[u8; 32]> { if self.domain != INTENT_DOMAIN || self.registry == [0; 32] @@ -177,6 +208,14 @@ pub(super) enum Allocation { Hold(Box), } +/// A fresh registry observation, distinct from an unsuccessful mutation. Only +/// Superseded proves that this allocation cannot still commit later. +pub(crate) enum CellGroupArchiveAcquireOutcome { + Held(Box), + Pending, + Superseded, +} + pub(super) struct Protection { pub revision: u64, pub roots: Vec, @@ -416,7 +455,7 @@ impl CellGroupArchiveRegistry { let root = state .selected .clone() - .ok_or(CellGroupArchiveError::Invalid("no selected archive"))?; + .ok_or(CellGroupArchiveError::Unconfirmed)?; self.intent(&state, Action::Acquire { root }) }) .await @@ -443,6 +482,21 @@ impl CellGroupArchiveRegistry { .map_err(|_| CellGroupArchiveError::Stopped)? } + pub(crate) async fn acquire_outcome( + &self, + domain: &impl CellGroupArchiveExecution, + intent: &CellGroupArchiveIntent, + ) -> Result { + intent.acquire_marker()?; + domain + .run(async { + let _io = self.archive.admit()?; + self.load().await?.0.acquire_outcome(intent) + }) + .await + .map_err(|_| CellGroupArchiveError::Stopped)? + } + fn intent(&self, state: &State, action: Action) -> Result { let mut nonce = [0; 32]; crate::asyncrt::rng("cell-group-archive-intent").fill_bytes(&mut nonce); diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs index cca6ef83a..1635beb0d 100644 --- a/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/state.rs @@ -3,6 +3,10 @@ //! Pure bounded transitions. All allocating mutations reconcile their exact //! intent digest; retired revisions never reappear and generations never repeat. use super::*; + +#[cfg(test)] +#[path = "state_tests.rs"] +mod tests; use std::collections::{BTreeMap, BTreeSet}; #[derive(Clone, Debug, Serialize, Deserialize)] @@ -17,6 +21,21 @@ pub(super) struct State { } impl State { + pub fn acquire_outcome( + &self, + intent: &CellGroupArchiveIntent, + ) -> Result { + intent.acquire_marker()?; + if let Some(Allocation::Hold(hold)) = self.reconcile(intent)? { + return Ok(CellGroupArchiveAcquireOutcome::Held(hold)); + } + Ok(if self.revision >= intent.revision { + CellGroupArchiveAcquireOutcome::Superseded + } else { + CellGroupArchiveAcquireOutcome::Pending + }) + } + pub fn empty(identity: CellGroupArchiveCreation) -> Self { Self { domain: STATE_DOMAIN.into(), diff --git a/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs b/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs new file mode 100644 index 000000000..dfcefdec4 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/archive/ownership/state_tests.rs @@ -0,0 +1,116 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Registry metadata tests grant no physical archive coverage. +use super::*; +use cellar_logic::cell_group::{CellRange, GroupRoute, RouteEpoch}; +use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}; + +fn selected() -> (State, CellGroupArchiveIntent) { + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + }; + let identity = CellGroupArchiveCreation { + domain: IDENTITY_DOMAIN.into(), + id: [7; 32], + route: route.clone(), + }; + let root = Descriptor { + object: Object { + registry: identity.id, + generation: 2, + digest: [8; 32], + length: 128, + }, + marker: GroupManifest::genesis(route).unwrap(), + }; + let mut state = State::empty(identity.clone()); + state.revision = 3; + state.selected = Some(root.clone()); + state.check(&identity).unwrap(); + let intent = CellGroupArchiveIntent { + domain: INTENT_DOMAIN.into(), + registry: identity.id, + revision: 4, + nonce: [9; 32], + action: Action::Acquire { root }, + }; + (state, intent) +} + +#[test] +fn acquire_outcome_reconciles_a_lost_response_after_unrelated_allocations() { + let (mut state, intent) = selected(); + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Pending + )); + let Allocation::Hold(hold) = state.allocate(&intent).unwrap() else { + panic!("acquire did not allocate a hold"); + }; + let unrelated = CellGroupArchiveIntent { + revision: 5, + nonce: [10; 32], + action: Action::Reserve { + parent: state.selected.clone(), + }, + ..intent.clone() + }; + state.allocate(&unrelated).unwrap(); + state.check(&state.identity).unwrap(); + let CellGroupArchiveAcquireOutcome::Held(observed) = state.acquire_outcome(&intent).unwrap() + else { + panic!("lost response did not reconcile"); + }; + assert_eq!(*observed, *hold); + assert_eq!(state.holds.len(), 1); +} + +#[test] +fn acquire_outcome_requires_a_consumed_revision_before_reallocation() { + let (mut state, intent) = selected(); + let competing = CellGroupArchiveIntent { + nonce: [11; 32], + action: Action::Reserve { + parent: state.selected.clone(), + }, + ..intent.clone() + }; + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Pending + )); + state.allocate(&competing).unwrap(); + assert!(matches!( + state.acquire_outcome(&intent).unwrap(), + CellGroupArchiveAcquireOutcome::Superseded + )); + assert!(matches!( + state.allocate(&intent), + Err(CellGroupArchiveError::Expired) + )); + assert!(state.holds.is_empty()); +} + +#[test] +fn acquire_outcome_refuses_foreign_registry_and_other_allocation_purposes() { + let (state, mut intent) = selected(); + intent.registry = [12; 32]; + assert!(matches!( + state.acquire_outcome(&intent), + Err(CellGroupArchiveError::Conflict) + )); + intent.registry = state.identity.id; + intent.action = Action::Reserve { + parent: state.selected.clone(), + }; + assert!(state.acquire_outcome(&intent).is_err()); +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs index 154c7c8a7..8f4998c83 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/header.rs @@ -6,6 +6,10 @@ use serde::{Deserialize, Serialize}; const DOMAIN: &str = "cellar-cell-group-checkpoint-1"; +#[cfg(test)] +#[path = "header_tests.rs"] +mod tests; + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub(super) enum Transition { @@ -28,6 +32,14 @@ impl Header { conf: &ConfState, membership: GroupMembership, ) -> Result { + root.check()?; + membership.validate(conf, root.position().index.get())?; + membership.validate_scope(&root.route().history)?; + if root.membership_revision() != membership.revision().get() { + return Err(CellGroupCheckpointError::Invalid( + "checkpoint capture membership", + )); + } Ok(Self { domain: DOMAIN.into(), root, @@ -55,6 +67,9 @@ impl Header { header .membership .validate(&header.configuration()?, header.root.position().index.get())?; + header + .membership + .validate_scope(&header.root.route().history)?; Ok(header) } diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs new file mode 100644 index 000000000..80f628bf0 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/header_tests.rs @@ -0,0 +1,44 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use crate::raft::member_state::FleetGroupBinding; +use cellar_logic::{ + cell_group::{CellRange, GroupHistory, GroupRoute, RouteEpoch}, + consensus::{ClusterId, GroupId, IncarnationId, NamespaceId}, +}; + +#[test] +fn checkpoint_refuses_a_foreign_managed_group_guard() { + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [5; 32], + schema_digest: [6; 32], + }; + let root = GroupManifest::genesis(route.clone()).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let binding = |history: &GroupHistory| { + FleetGroupBinding::new(IncarnationId::new([7; 16]).unwrap(), history, [8; 32]).unwrap() + }; + let mut foreign = route.history; + foreign.group = GroupId::new([9; 16]).unwrap(); + let foreign_membership = GroupMembership::managed_initial(&conf, binding(&foreign)).unwrap(); + assert!(Header::capture(root.clone(), &conf, foreign_membership.clone()).is_err()); + let membership = GroupMembership::managed_initial(&conf, binding(&route.history)).unwrap(); + let header = Header::capture(root, &conf, membership).unwrap(); + Header::decode(&header.encode().unwrap()).unwrap(); + let malformed = Header { + membership: foreign_membership, + ..header + }; + assert!(Header::decode(&malformed.encode().unwrap()).is_err()); +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs index 757c2f685..b254d7c83 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/mod.rs @@ -16,12 +16,15 @@ mod local; mod prepared; mod receive; mod receive_archive; +mod seed; +pub(crate) use seed::{seed_initial_membership, seed_profile, seed_source}; pub(crate) mod wire; use header::Header; pub use local::PreparedLocalCellGroupCheckpoint; pub(crate) use prepared::persisted_marker; pub use prepared::PreparedCellGroupCheckpoint; pub use receive::CellGroupCheckpointReceiver; +pub(crate) use receive_archive::PreparedArchiveReceive; pub use receive_archive::{CellGroupArchiveCheckpointReceiver, PreparedCellGroupArchiveCheckpoint}; use sha2::{Digest, Sha256}; use std::sync::Arc; diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs index 0ca021e1a..6468cb1dd 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive.rs @@ -200,11 +200,11 @@ impl State { scratch: ScratchPool, directory: PathBuf, ) -> Result { - if !directory.is_absolute() || directory.as_os_str().len() > 4096 { - return Err(CellGroupCheckpointError::Invalid( - "receive scratch directory", - )); - } + let reservation = Self::reserve(pool, 0)?; + Self::with_reservation(pool, target, scratch, directory, reservation) + } + + pub(super) fn reserve(pool: &CellGroupCheckpointPool, extra_bytes: usize) -> Result { let cells = pool.limits.max_cells as usize; let extents = usize::try_from(pool.limits.max_transfer_bytes.div_ceil(EXTENT_BYTES)) .map_err(|_| CellGroupCheckpointError::Limit)?; @@ -213,11 +213,25 @@ impl State { .and_then(|n| n.checked_add(cells.checked_mul(4096)?)) .and_then(|n| n.checked_add(extents.checked_mul(2048)?)) .and_then(|n| n.checked_add(RECEIVE_WORKSPACE)) + .and_then(|n| n.checked_add(extra_bytes)) .ok_or(CellGroupCheckpointError::Limit)?; - let reservation = pool - .budget + pool.budget .acquire(bytes) - .ok_or(CellGroupCheckpointError::Busy)?; + .ok_or(CellGroupCheckpointError::Busy) + } + + pub(super) fn with_reservation( + pool: &CellGroupCheckpointPool, + target: T, + scratch: ScratchPool, + directory: PathBuf, + reservation: Permit, + ) -> Result { + if !directory.is_absolute() || directory.as_os_str().len() > 4096 { + return Err(CellGroupCheckpointError::Invalid( + "receive scratch directory", + )); + } target.require_live()?; target .storage() diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs index 4ae18ddd4..f2d3ef4a9 100644 --- a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive.rs @@ -9,6 +9,10 @@ use cellar_logic::cell_group::GroupRoute; use cellar_ltx::stream_snapshot::ScratchPool; use std::{path::PathBuf, sync::Mutex}; +#[cfg(test)] +#[path = "receive_archive_tests.rs"] +mod tests; + impl Target for CellGroupArchiveStore { fn storage(&self) -> &Arc> { &self.storage @@ -44,6 +48,22 @@ impl Target for CellGroupArchiveStore { } impl CellGroupCheckpointPool { + /// Reserve before opening/scanning an independently owned page inventory. + /// Includes worst-case inventory and directory metadata in this same bank. + pub(crate) fn prepare_archive_receive( + &self, + max_pages: usize, + ) -> Result { + let bytes = max_pages + .checked_mul(512) + .and_then(|n| n.checked_add(16384)) + .ok_or(CellGroupCheckpointError::Limit)?; + Ok(PreparedArchiveReceive { + pool: self.clone(), + reservation: State::::reserve(self, bytes)?, + }) + } + /// Receive into private data stores, without a Raft mapping, surviving voter /// disk or applied-cut observation. Shares the node's source/receiver quota. pub fn receive_archive( @@ -58,6 +78,28 @@ impl CellGroupCheckpointPool { } } +pub(crate) struct PreparedArchiveReceive { + pool: CellGroupCheckpointPool, + reservation: Permit, +} + +impl PreparedArchiveReceive { + pub(crate) fn receive( + self, + store: CellGroupArchiveStore, + scratch: ScratchPool, + directory: PathBuf, + ) -> Result { + Ok(CellGroupArchiveCheckpointReceiver(State::with_reservation( + &self.pool, + store, + scratch, + directory, + self.reservation, + )?)) + } +} + /// Physical archive receiver. It cannot produce the engine-bound checkpoint /// capability accepted by SnapshotChange or the Raft owner. pub struct CellGroupArchiveCheckpointReceiver(State); diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs new file mode 100644 index 000000000..aaf4dc250 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/receive_archive_tests.rs @@ -0,0 +1,52 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; + +fn pool() -> CellGroupCheckpointPool { + CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 64 * 1024 * 1024, + max_cells: 4, + max_transfer_bytes: 4 * 1024 * 1024, + max_chunk_bytes: 64 * 1024, + }) + .unwrap() +} + +#[test] +fn archive_inventory_admission_is_retained_before_storage_opens() { + let pool = pool(); + let pending = pool.prepare_archive_receive(1024).unwrap(); + assert_eq!(pool.usage().sources, 1); + let reserved = pool.usage().retained_bytes; + assert!(reserved >= 1024 * 512 + 16384); + assert!(matches!( + pool.prepare_archive_receive(1024), + Err(CellGroupCheckpointError::Busy) + )); + assert_eq!(pool.usage().retained_bytes, reserved); + drop(pending); + assert_eq!( + pool.usage(), + CellGroupCheckpointUsage { + sources: 0, + retained_bytes: 0 + } + ); + assert!(pool.prepare_archive_receive(1024).is_ok()); +} + +#[test] +fn archive_inventory_cannot_bypass_existing_receiver_byte_bank() { + let pool = pool(); + assert!(matches!( + pool.prepare_archive_receive(1_000_000), + Err(CellGroupCheckpointError::Busy) + )); + assert!(matches!( + pool.prepare_archive_receive(usize::MAX), + Err(CellGroupCheckpointError::Limit) + )); + assert_eq!(pool.usage().sources, 0); + assert_eq!(pool.usage().retained_bytes, 0); +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs new file mode 100644 index 000000000..5b539be44 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/seed.rs @@ -0,0 +1,77 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! A complete authenticated descriptor must carry the exact consumed fleet +//! repair permit. These metadata checks mint no transport or catalog authority. +use super::*; +use crate::raft::member_state::{FleetMembershipPermit, FleetPermitPhase, RepairPhase}; + +#[cfg(test)] +#[path = "seed_tests.rs"] +mod tests; + +pub(super) fn check_membership( + membership: &GroupMembership, + permit: Option<&FleetMembershipPermit>, +) -> Result<()> { + match (membership.fleet_status(), permit) { + (None, None) => Ok(()), + (Some(status), Some(permit)) => { + let repair = membership + .repair_status()? + .ok_or(CellGroupCheckpointError::Invalid( + "seed has no managed repair", + ))?; + if &status.binding != permit.binding() + || status.consumed_sequence != permit.sequence() + || status.permit.as_ref() != Some(permit) + || status.phase != Some(FleetPermitPhase::Seeding) + || repair.phase != RepairPhase::Seeding + { + return Err(CellGroupCheckpointError::Invalid("seed fleet permit")); + } + permit.check_plan(&repair.plan)?; + Ok(()) + } + _ => Err(CellGroupCheckpointError::Invalid("seed managed profile")), + } +} + +pub(crate) fn seed_profile( + snapshot: &raft::prelude::Snapshot, + permit: Option<&FleetMembershipPermit>, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + if header.snapshot()? != *snapshot { + return Err(CellGroupCheckpointError::Invalid("seed snapshot encoding")); + } + check_membership(&header.membership, permit) +} + +pub(crate) fn seed_initial_membership( + snapshot: &raft::prelude::Snapshot, + conf: &raft::prelude::ConfState, +) -> Result { + let header = Header::decode(&snapshot.data)?; + Ok(header.membership.seed_initial(conf)?) +} + +/// The responding process must also be a current, physically bound voter in +/// the authenticated checkpoint, independently of the joining target. +pub(crate) fn seed_source( + snapshot: &raft::prelude::Snapshot, + source: &cellar_logic::consensus::ReplicaIdentity, +) -> Result<()> { + let header = Header::decode(&snapshot.data)?; + let conf = header.configuration()?; + if header.root.route().history != source.scope + || !header.membership.permits_group(source) + || !conf + .voters + .iter() + .chain(&conf.voters_outgoing) + .any(|id| *id == source.member.get()) + { + return Err(CellGroupCheckpointError::Invalid("seed source member")); + } + Ok(()) +} diff --git a/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs b/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs new file mode 100644 index 000000000..e232b7129 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/checkpoint/seed_tests.rs @@ -0,0 +1,148 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Descriptive membership fixtures exercise profile checks. These values grant +//! no catalog, network, physical-install, startup or serving authority. +use super::*; +use crate::raft::member_state::{FleetGroupBinding, JoiningMember, RepairPlan, RepairStep}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{ + ClusterId, DiskLineage, GroupId, IncarnationId, LogIndex, MemberId, NamespaceId, NodeId, + ProposalId, + }, +}; +use raft::prelude::{ConfChangeV2, ConfState}; + +fn fixture(genesis: bool) -> (GroupMembership, ConfState, FleetMembershipPermit) { + let history = GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }; + let binding = + FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &history, [6; 32]).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let mut state = if genesis { + GroupMembership::managed_initial(&conf, binding.clone()).unwrap() + } else { + GroupMembership::initial(&conf) + }; + if !genesis { + let mut change = ConfChangeV2::default(); + state + .bind_fleet_enrollment(&mut change, &conf, &binding) + .unwrap(); + state = state + .applied(&change, &conf, &conf, LogIndex::new(2)) + .unwrap(); + } + let plan = RepairPlan { + id: ProposalId::new([7; 16]).unwrap(), + policy_revision: 1, + expected_revision: state.revision(), + obsolete: MemberId::new(3).unwrap(), + replacement: JoiningMember { + member: MemberId::new(4).unwrap(), + node: NodeId::new([8; 16]).unwrap(), + disk: DiskLineage::new([9; 16]).unwrap(), + }, + }; + let permit = FleetMembershipPermit::new( + binding, + 1, + ProposalId::new([10; 16]).unwrap(), + ProposalId::new([11; 16]).unwrap(), + &plan, + ) + .unwrap(); + let mut change = ConfChangeV2::default(); + state + .bind_managed_repair_start(&mut change, &conf, &plan, &permit) + .unwrap(); + let mut next = conf.clone(); + next.learners.push(4); + let state = state + .applied( + &change, + &conf, + &next, + LogIndex::new(state.revision().get() + 1), + ) + .unwrap(); + state.validate(&next, state.revision().get()).unwrap(); + (state, next, permit) +} + +#[test] +fn managed_seed_profile_requires_exact_consumed_permit_and_profile() { + for genesis in [false, true] { + let (state, conf, permit) = fixture(genesis); + check_membership(&state, Some(&permit)).unwrap(); + assert!(check_membership(&state, None).is_err()); + assert!(check_membership(&GroupMembership::initial(&conf), Some(&permit)).is_err()); + let plan = state.repair_status().unwrap().unwrap().plan; + let other = FleetMembershipPermit::new( + permit.binding().clone(), + 2, + ProposalId::new([12; 16]).unwrap(), + ProposalId::new([13; 16]).unwrap(), + &plan, + ) + .unwrap(); + assert!(check_membership(&state, Some(&other)).is_err()); + } +} + +#[test] +fn managed_seed_profile_refuses_aborted_repair() { + let (state, conf, permit) = fixture(true); + let mut change = ConfChangeV2::default(); + state + .bind_managed_repair_step( + &mut change, + &conf, + permit.repair_id().unwrap(), + RepairStep::AbortLearner, + &permit, + ) + .unwrap(); + let mut next = conf.clone(); + next.learners.clear(); + let aborted = state + .applied( + &change, + &conf, + &next, + LogIndex::new(state.revision().get() + 1), + ) + .unwrap(); + aborted.validate(&next, aborted.revision().get()).unwrap(); + assert!(check_membership(&aborted, Some(&permit)).is_err()); +} + +#[test] +fn managed_seed_base_preserves_genesis_and_committed_enrollment_origins() { + for genesis in [false, true] { + let (source, _, _) = fixture(genesis); + let birth = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; + let staged = source.seed_initial(&birth).unwrap(); + staged.validate(&birth, 1).unwrap(); + assert_eq!(staged.revision(), LogIndex::new(1)); + assert!(staged.joined(MemberId::new(4).unwrap()).unwrap().is_none()); + assert_eq!(staged.fleet_status().is_some(), genesis); + source.succeeds(&staged).unwrap(); + assert!(staged.succeeds(&source).is_err()); + let wrong = ConfState { + voters: vec![1, 2, 5], + ..Default::default() + }; + assert!(source.seed_initial(&wrong).is_err()); + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/admission.rs b/crates/cellar/durability/raft/cell_groups/controller/admission.rs new file mode 100644 index 000000000..b78c87b98 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/admission.rs @@ -0,0 +1,122 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Node maintenance and work accounting become visible in one short admission +//! section. Physical producers and undelivered responses share the work lease. +use super::*; +use crate::application::consensus::admission::{Access, Admission}; +use crate::host_services::{ApplicationWork, ApplicationWorkGuard}; +use std::sync::MutexGuard; + +pub(super) struct Public { + admission: Arc, + work: Arc, +} + +pub(super) struct Registration<'a> { + // Serializes initial binding with standalone admissions. Never crosses an + // await, filesystem operation or application callback. + _configuration: MutexGuard<'a, ()>, + _gate: Option>, + pub work: Option, +} + +pub(super) struct Retention { + pub _permit: crate::raft::budget::Permit, + pub _work: Option, +} + +impl Shared { + pub(super) fn registration(&self) -> Result, RuntimeError> { + let configuration = self + .admission_configuration + .lock() + .map_err(|_| RuntimeError::Closed)?; + if self.closing.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + let (gate, work) = match self.public.get() { + Some(public) => { + let gate = public + .admission + .registration(Access::Application) + .map_err(|_| RuntimeError::Closed)?; + (Some(gate), Some(public.work.enter())) + } + None => (None, None), + }; + Ok(Registration { + _configuration: configuration, + _gate: gate, + work, + }) + } + + pub(super) fn application_paused(&self) -> bool { + self.public + .get() + .is_some_and(|public| public.admission.application_paused()) + } +} + +impl CellGroupController { + /// Conservative shutdown observation behind the sealed public barrier. + /// Poison, an unbound owner, held responses, pending provisioning or any + /// accepted physical application work make this proof unavailable. + pub(crate) fn local_producers_quiesced(&self) -> bool { + let shared = &self.handle.0; + let Ok(_configuration) = shared.admission_configuration.lock() else { + return false; + }; + let Some(public) = shared.public.get() else { + return false; + }; + if !public.admission.is_draining() + || public.admission.is_poisoned() + || public.work.snapshot().pending_events != 0 + || shared.commands.usage().1 != 0 + || shared.active_effects.load(Ordering::Acquire) != 0 + || !shared.groups.producers_quiesced() + { + return false; + } + let Ok(provisioning) = shared.provisioning.lock() else { + return false; + }; + provisioning.as_ref().is_none_or(|config| { + config.preparations.usage().1 == 0 && config.waiters.usage().1 == 0 + }) + } + + /// Install the canonical SDK's node admission before exposing ingress. A + /// different owner, or previously admitted untracked work, cannot be rebound. + pub(crate) fn bind_public_admission( + &self, + admission: Arc, + work: Arc, + ) -> Result<(), CellGroupControllerError> { + let shared = &self.handle.0; + let _configuration = shared + .admission_configuration + .lock() + .map_err(|_| CellGroupControllerError::Closed)?; + if let Some(old) = shared.public.get() { + return if Arc::ptr_eq(&old.admission, &admission) && Arc::ptr_eq(&old.work, &work) { + Ok(()) + } else { + Err(CellGroupControllerError::Invalid("public admission owner")) + }; + } + if shared.closing.load(Ordering::Acquire) + || shared.commands.usage().1 != 0 + || shared.active_effects.load(Ordering::Acquire) != 0 + { + return Err(CellGroupControllerError::Invalid( + "public admission already active/closed", + )); + } + shared + .public + .set(Public { admission, work }) + .map_err(|_| CellGroupControllerError::Invalid("public admission already bound")) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/calls.rs b/crates/cellar/durability/raft/cell_groups/controller/calls.rs index 66d4fe6d5..84ebf54ea 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/calls.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/calls.rs @@ -5,20 +5,20 @@ use super::*; use crate::raft::budget::Permit; use tokio::sync::oneshot; -pub(super) struct Delivery { +pub(super) struct Delivery { pub result: Result>, - pub _permit: Arc, + pub _permit: Arc

, } -pub(super) struct Call { - pub receiver: oneshot::Receiver>, +pub(super) struct Call { + pub receiver: oneshot::Receiver>, pub identity: I, pub deadline: u64, pub domain: crate::asyncrt::DomainToken, pub failure: fn(I, RuntimeError) -> E, } -impl Call { +impl Call { pub async fn response(self) -> Result> { let failure = |reason| Arc::new((self.failure)(self.identity, reason)); let now = self diff --git a/crates/cellar/durability/raft/cell_groups/controller/commands.rs b/crates/cellar/durability/raft/cell_groups/controller/commands.rs index 7e98c185b..0b88e1713 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/commands.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/commands.rs @@ -1,9 +1,10 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Bounded invocation ownership is independent of response-waiter lifetime. +use super::admission::Retention; use super::calls::Delivery; use super::*; -use crate::raft::budget::{Budget, Permit}; +use crate::raft::budget::Budget; use crate::raft::cell_groups::{ CellCommandError, CellCommandInvocation, CellCommandRequest, CellCommandResponse, CellDispatchError, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, PendingCellCommand, @@ -19,20 +20,20 @@ enum Invocation { pub(super) struct Request { invocation: Invocation, options: CellSqlOptions, - reply: oneshot::Sender>, - permit: Arc, + reply: oneshot::Sender>, + permit: Arc, } pub(super) struct Completed { pub invocation: Option, pub error: Option>, - _permit: Arc, + _permit: Arc, } /// Dropping or timing out this response never cancels the admitted command. /// An unpolled reply retains node admission, including error-only replies. pub struct CellCommandCall { - inner: calls::Call, + inner: calls::Call, } /// Admission did not transfer ownership. In particular, a rejected non-fresh @@ -113,6 +114,9 @@ impl CellGroupController { ) -> Result { let shared = &self.handle.0; let command = request.id; + let mut registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; if shared.closing.load(Ordering::Acquire) { return Err(error(command, false, RuntimeError::Closed)); } @@ -133,7 +137,10 @@ impl CellGroupController { invocation: Invocation::Pending(Box::new(pending)), options, reply, - permit: Arc::new(permit), + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), }))) .map_err(|e| { error( @@ -164,6 +171,9 @@ pub(super) fn admit( ) -> Result { let admitted = (|| { let (command, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + let registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; if shared.closing.load(Ordering::Acquire) { return Err(error(command, false, RuntimeError::Closed)); } @@ -174,9 +184,9 @@ pub(super) fn admit( .commands .acquire(bytes) .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; - Ok((command, deadline, permit)) + Ok((command, deadline, permit, registration)) })(); - let (command, deadline, permit) = match admitted { + let (command, deadline, permit, mut registration) = match admitted { Ok(admitted) => admitted, Err(error) => { return Err(CellCommandAdmissionError { @@ -192,7 +202,10 @@ pub(super) fn admit( invocation: Invocation::Ready(Box::new(invocation)), options, reply, - permit: Arc::new(permit), + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), }))) .map_err(|e| { let (request, reason) = match e { diff --git a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs index 52b0988c7..306e2cc35 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/effects/mod.rs @@ -289,12 +289,27 @@ impl Manager { else { continue; }; + // Serialize work visibility with the public maintenance/drain + // barrier before scheduling a scan or invoking an effect adapter. + let mut admission = match shared.registration() { + Ok(admission) => admission, + Err(_) if shared.application_paused() || shared.closing.load(Ordering::Acquire) => { + return Ok(()) + } + Err(error) => return Err(error.into()), + }; + if shared.application_paused() { + return Ok(()); + } + let work = admission.work.take(); let group = *group; let after = self.cells.get(&group).cloned(); let registration = registration.clone(); + drop(admission); let shared = shared.clone(); shared.active_effects.store(1, Ordering::Release); self.jobs.push(Box::pin(async move { + let _work = work; let mut cell = None; let result = std::panic::AssertUnwindSafe(turn::run( group, diff --git a/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs index 509af5910..263df493c 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/enrollments.rs @@ -1,9 +1,9 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Bounded invocation ownership is independent of response-waiter lifetime. +use super::admission::Retention; use super::calls::Delivery; use super::*; -use crate::raft::budget::Permit; use crate::raft::cell_groups::{ CellDispatchError, CellEnrollmentError, CellEnrollmentInvocation, CellSqlEnrollment, CellSqlOptions, @@ -14,20 +14,20 @@ use tokio::sync::{mpsc, oneshot, watch}; pub(super) struct Request { invocation: CellEnrollmentInvocation, options: CellSqlOptions, - reply: oneshot::Sender>, - permit: Arc, + reply: oneshot::Sender>, + permit: Arc, } pub(super) struct Completed { pub invocation: CellEnrollmentInvocation, pub error: Option>, - _permit: Arc, + _permit: Arc, } /// Dropping or timing out this response never cancels the admitted enrollment. /// An unpolled reply retains node admission, including error-only replies. pub struct CellEnrollmentCall { - inner: calls::Call, + inner: calls::Call, } /// Admission did not transfer ownership. In particular, a rejected non-fresh @@ -82,6 +82,9 @@ impl CellGroupController { let shared = &self.handle.0; let admitted = (|| { let (command, deadline) = invocation.controller_admission(&shared.domain)?; + let registration = shared + .registration() + .map_err(|reason| error(command, false, reason))?; if shared.closing.load(Ordering::Acquire) { return Err(error(command, false, RuntimeError::Closed)); } @@ -94,9 +97,9 @@ impl CellGroupController { .commands .acquire(bytes) .ok_or_else(|| error(command, false, RuntimeError::Overloaded))?; - Ok((command, deadline, permit)) + Ok((command, deadline, permit, registration)) })(); - let (command, deadline, permit) = match admitted { + let (command, deadline, permit, mut registration) = match admitted { Ok(admitted) => admitted, Err(error) => { return Err(CellEnrollmentAdmissionError { @@ -112,7 +115,10 @@ impl CellGroupController { invocation, options, reply, - permit: Arc::new(permit), + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), }))) .map_err(|e| { let (request, reason) = match e { diff --git a/crates/cellar/durability/raft/cell_groups/controller/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/mod.rs index 737179cf5..0bc97d094 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/mod.rs @@ -3,6 +3,7 @@ //! Owns registered groups' consensus events, transport dispatch and physical apply. //! Admitted command ownership survives response cancellation and expiry. +mod admission; mod archive; pub(crate) use archive::settlement::{ Client as S3Client, LookupPrepared as S3LookupPrepared, OutputProof as S3OutputProof, @@ -23,7 +24,10 @@ pub use effects::CellGroupEffectRegistration; mod failure; mod provision; mod registration; -pub use provision::{CellGroupProvisionCall, CellGroupProvisioning}; +pub use provision::{ + CellGroupProvisionCall, CellGroupProvisionLimits, CellGroupProvisionNamespace, + CellGroupProvisioning, +}; mod requests; pub use enrollments::{CellEnrollmentAdmissionError, CellEnrollmentCall}; pub(crate) use registration::GroupRoster; @@ -43,7 +47,7 @@ use futures_util::FutureExt; use std::collections::BTreeMap; use std::sync::{ atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}, - Arc, Mutex, + Arc, Mutex, OnceLock, }; use tokio::sync::Notify; @@ -92,8 +96,10 @@ impl From for CellGroupControllerError { } } -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Copy, Debug, serde::Serialize)] pub struct CellGroupControllerUsage { + pub ordinary: super::CellGroupAdmissionUsage, + pub archive_settlement: super::CellGroupAdmissionUsage, pub groups: usize, pub pending_registrations: usize, /// Retained failed groups still consume the registration capacity. @@ -113,6 +119,8 @@ pub struct CellGroupControllerUsage { pub command_bytes: usize, pub retained_preparations: usize, pub preparation_bytes: usize, + /// Immutable namespace/directory metadata charged to provisioning's total. + pub provision_configuration_bytes: usize, pub provision_waiters: usize, /// Independent archive-settlement admission, bounded by max_commands and /// max_command_bytes. This lane cannot recursively consume command slots. @@ -126,6 +134,11 @@ pub struct CellGroupControllerUsage { } struct Shared { + admission_configuration: Mutex<()>, + public: OnceLock, + // One-shot physical release is opaque to break the ownership type graph: + // bindings contain weak archive clients back to this same shared owner. + host_resources: Mutex Result<(), CellGroupControllerError> + Send>>>, runtime: CellGroupRuntime, sql: crate::raft::host_sql::cell_groups::WeakCellGroupSqlPool, groups: Arc, @@ -206,10 +219,34 @@ pub struct CellGroupController { } impl CellGroupController { + pub(crate) fn limits(&self) -> CellGroupControllerLimits { + self.handle.0.limits + } + pub(crate) fn belongs_to(&self, runtime: &CellGroupRuntime) -> bool { self.handle.0.runtime.same_owner(runtime) } + pub(in crate::raft::cell_groups) fn own_host_resources( + &self, + resources: Arc, + ) -> Result<(), RuntimeError> { + let mut slot = self + .handle + .0 + .host_resources + .lock() + .map_err(|_| RuntimeError::Closed)?; + if self.handle.0.closing.load(Ordering::Acquire) { + return Err(RuntimeError::Closed); + } + if slot.is_some() { + return Err(RuntimeError::AlreadyAttached); + } + *slot = Some(Box::new(move || resources.release())); + Ok(()) + } + /// Consume the exclusive event receiver after validating exact engine and /// runtime ownership. Initial groups must already be attached. Later groups /// enter through register() or provision(); unknown events still fail closed. @@ -256,6 +293,9 @@ impl CellGroupController { let domain = runtime.log().domain(); let (requests, receiver) = tokio::sync::mpsc::channel(limits.max_commands); let shared = Arc::new(Shared { + admission_configuration: Mutex::new(()), + public: OnceLock::new(), + host_resources: Mutex::new(None), runtime: runtime.clone(), sql: sql.downgrade(), groups: GroupRoster::new(roster.keys().copied(), limits.max_groups), @@ -324,7 +364,20 @@ impl CellGroupController { shared.active_checkpoints.store(0, Ordering::Release); shared.active_apply.store(0, Ordering::Release); shared.pending_apply.store(0, Ordering::Release); - shared.complete(result.and(joined)); + // Resource release belongs to this joined owner, never the + // lifetime of an async close waiter or a retained cell handle. + let released = match shared.host_resources.lock() { + Ok(mut slot) => slot.take().map_or(Ok(()), |release| release()), + Err(error) => { + if let Some(release) = error.into_inner().take() { + let _ = release(); + } + Err(CellGroupControllerError::Invalid( + "host resource owner poisoned", + )) + } + }; + shared.complete(result.and(joined).and(released)); }) .map_err(|_| CellGroupControllerError::Closed)?; Ok(handle) @@ -343,22 +396,35 @@ impl CellGroupController { let shared = &self.handle.0; let (command_bytes, retained_commands) = shared.commands.usage(); let (s3_bytes, retained_s3) = shared.s3.usage(); - let (preparation_bytes, retained_preparations, provision_waiters) = shared + let ( + preparation_bytes, + retained_preparations, + provision_waiters, + provision_configuration_bytes, + ) = shared .provisioning .lock() .ok() .and_then(|slot| { slot.as_ref().map(|config| { let (bytes, items) = config.preparations.usage(); - (bytes, items, config.waiters.usage().1) + ( + bytes, + items, + config.waiters.usage().1, + config.configuration_bytes, + ) }) }) .unwrap_or_default(); CellGroupControllerUsage { + ordinary: super::CellGroupAdmissionUsage::read(&shared.commands), + archive_settlement: super::CellGroupAdmissionUsage::read(&shared.s3), active_effects: shared.active_effects.load(Ordering::Acquire), effect_turns: shared.effect_turns.load(Ordering::Relaxed), effect_retries: shared.effect_retries.load(Ordering::Relaxed), preparation_bytes, + provision_configuration_bytes, retained_preparations, provision_waiters, groups: shared.groups.len(), @@ -399,9 +465,13 @@ impl CellGroupController { options: super::CellSqlOptions, ) -> Result { let shared = &self.handle.0; - if shared.closing.load(Ordering::Acquire) { - return Err(super::CellDispatchError::Runtime(RuntimeError::Closed)); - } + let work = { + let mut registration = shared + .registration() + .map_err(super::CellDispatchError::Runtime)?; + registration.work.take() + }; + let _work = work; invocation.begin().await?; let attempt = invocation .attempt() diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs index 7c87666af..3891873c2 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/config.rs @@ -7,13 +7,48 @@ use crate::raft::cell_groups::{CellGroupApplyImages, ManifestLimits, ManifestPin use crate::raft::{budget::Budget, transport_options::RaftPeerNode}; use std::path::PathBuf; +/// Node-shared dynamic preparation ceilings; no namespace gets its own bank. +#[derive(Clone, Copy, Debug)] +pub struct CellGroupProvisionLimits { + pub max_preparations: usize, + pub max_preparation_bytes: usize, + pub max_parallel_preparations: usize, + pub max_waiters: usize, + pub peer_revision: u64, +} + +impl CellGroupProvisionLimits { + pub(crate) fn validate(&self, max_groups: usize) -> Result<(), CellGroupControllerError> { + if self.max_preparations == 0 + || self.max_preparations > max_groups + || self.max_preparation_bytes == 0 + || self.max_parallel_preparations == 0 + || self.max_parallel_preparations > self.max_preparations.min(256) + || !(1..=65_536).contains(&self.max_waiters) + || self.peer_revision == 0 + { + return Err(CellGroupControllerError::Invalid( + "group provisioning limits", + )); + } + Ok(()) + } +} + +/// Immutable namespace image resources. Directory identity is checked before +/// catalog reads or physical preparation; a matching name is not ownership. +pub struct CellGroupProvisionNamespace { + pub directory: RangeDirectory, + pub images: CellGroupApplyImages, +} + /// Resources for initial members of already assigned ranges. Managed fleet /// admission is not inferred from this configuration. Image stores retain their /// own quotas; page inventory and recovery work have additional bounds here. pub struct CellGroupProvisioning { pub directory: PathBuf, pub filesystem: Arc, - pub images: CellGroupApplyImages, + pub namespaces: Vec, pub manifest: ManifestLimits, pub entries: GroupEntryLimits, pub max_effect_rows: u32, @@ -36,6 +71,7 @@ pub(in super::super) struct Config { pub waiters: Arc, pub parallel: Arc, pub bytes: usize, + pub configuration_bytes: usize, } impl Config { @@ -46,17 +82,59 @@ impl Config { let invalid = || CellGroupControllerError::Invalid("group provisioning configuration"); settings.manifest.validate().map_err(|_| invalid())?; settings.entries.validate().map_err(|_| invalid())?; - settings.images.validate().map_err(apply_error)?; - if let CellGroupApplyImages::Extents { - cache_directory, .. - } = &mut settings.images - { - if !cache_directory.is_absolute() || cache_directory.as_os_str().len() > 4096 { + if settings.namespaces.is_empty() || settings.namespaces.len() > 4096 { + return Err(invalid()); + } + let mut namespaces = std::collections::BTreeSet::new(); + // Trusted endpoints and fixed configuration metadata remain retained + // even when no preparation is active. Keep them in the same total. + let mut configuration_bytes = 2 * 1024 * 1024_usize; + let mut image_bytes = 0; + for profile in &mut settings.namespaces { + let namespace = profile.directory.namespace_config(); + if !namespaces.insert((namespace.cluster, namespace.namespace)) + || !matches!( + (&profile.images, namespace.image_storage), + ( + CellGroupApplyImages::Materialized { .. }, + crate::raft::catalog::RaftImageStorage::Materialized + ) | ( + CellGroupApplyImages::Extents { .. }, + crate::raft::catalog::RaftImageStorage::Extents + ) + ) + { return Err(invalid()); } - *cache_directory = std::mem::take(cache_directory) - .into_boxed_path() - .into_path_buf(); + profile.images.validate().map_err(apply_error)?; + let mut metadata = profile + .directory + .cache_capacity_bytes() + .checked_add(8192) + .ok_or_else(invalid)?; + if let CellGroupApplyImages::Extents { + cache_directory, .. + } = &mut profile.images + { + if !cache_directory.is_absolute() + || cache_directory.as_os_str().len() > 4096 + || cache_directory + .components() + .any(|p| matches!(p, std::path::Component::ParentDir)) + { + return Err(invalid()); + } + metadata = metadata + .checked_add(cache_directory.as_os_str().len() * 2) + .ok_or_else(invalid)?; + *cache_directory = std::mem::take(cache_directory) + .into_boxed_path() + .into_path_buf(); + } + configuration_bytes = configuration_bytes + .checked_add(metadata) + .ok_or_else(invalid)?; + image_bytes = image_bytes.max(profile.images.memory_bound()); } if !settings.directory.is_absolute() || settings.directory.as_os_str().len() > 4096 @@ -119,22 +197,52 @@ impl Config { * crate::raft::cell_groups::page_wire::MAX_PAGE_BYTES, ) }) - .and_then(|n| n.checked_add(settings.images.memory_bound())) + .and_then(|n| n.checked_add(image_bytes)) .and_then(|n| n.checked_add(2 * 1024 * 1024)) .ok_or_else(invalid)?; - if settings.max_preparation_bytes < bytes { + // Immutable namespace handles keep their advertised directory caches alive. + // Charge those once against the existing total, leaving one shared bank + // for every namespace's physical preparations. No per-namespace banks. + let available = settings + .max_preparation_bytes + .checked_sub(configuration_bytes) + .ok_or_else(invalid)?; + if available < bytes { return Err(invalid()); } settings.directory = settings.directory.into_boxed_path().into_path_buf(); settings.peers = settings.peers.into_boxed_slice().into_vec(); + settings.namespaces = settings.namespaces.into_boxed_slice().into_vec(); Ok(Self { - preparations: Budget::new(settings.max_preparation_bytes, settings.max_preparations), + preparations: Budget::new(available, settings.max_preparations), waiters: Budget::new(settings.max_waiters * WAITER_BYTES, settings.max_waiters), parallel: Arc::new(tokio::sync::Semaphore::new( settings.max_parallel_preparations, )), bytes, + configuration_bytes, settings, }) } + + pub fn images( + &self, + directory: &RangeDirectory, + ) -> Result { + self.profile(directory) + .map(|profile| profile.images.clone()) + } + + pub fn profile( + &self, + directory: &RangeDirectory, + ) -> Result<&CellGroupProvisionNamespace, CellGroupControllerError> { + self.settings + .namespaces + .iter() + .find(|profile| profile.directory.same_owner(directory)) + .ok_or(CellGroupControllerError::Invalid( + "unconfigured namespace owner", + )) + } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs new file mode 100644 index 000000000..9a83957fe --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/learner.rs @@ -0,0 +1,330 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Managed learners keep one physical preparation through grant, install and +//! attachment. Neither a reservation nor a staged mapping can create genesis. +use super::*; +use crate::application::consensus::client::LearnerActivation; +use crate::raft::{ + cache::GroupLog, + cell_groups::CellGroupBirth, + fleet_catalog::{FleetHostAuthorization, FleetPermitAuthorization}, + member_state::{members, FleetPermitPhase}, + storage::LogError, + transport::{CellGroupSeedError, CellGroupSeedSession, GroupRoute as PhysicalRoute, RaftPeer}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{MemberId, NodeId, ReplicaIdentity}, +}; + +#[derive(Clone)] +pub(super) struct Admission { + authorization: FleetPermitAuthorization, + pub member: MemberId, + source: MemberId, + source_node: NodeId, +} + +impl Admission { + pub fn new( + route: &GroupRoute, + host: &FleetHostAuthorization, + authorization: FleetPermitAuthorization, + activation: LearnerActivation, + ) -> Result { + let replacement = authorization + .permit() + .replacement() + .map_err(|e| catalog_error(CatalogError::Log(e)))?; + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + if !authorization.matches_scope(&route.history) + || authorization.permit().binding().fleet() != host.fleet() + || authorization.target_boot() != Some(&boot) + || replacement.member != activation.member + || replacement.node != host.node() + || replacement.disk != host.disk() + || activation.member == activation.source + || activation.source_node == host.node() + { + return Err(CellGroupControllerError::Invalid( + "managed learner admission", + )); + } + Ok(Self { + authorization, + member: activation.member, + source: activation.source, + source_node: activation.source_node, + }) + } + + pub(super) fn check_installed( + &self, + group: &GroupLog, + ) -> Result<(), CellGroupControllerError> { + let cache = group.read().map_err(RuntimeError::from)?; + let status = cache + .membership + .fleet_status() + .ok_or(CellGroupControllerError::Invalid( + "missing learner fleet guard", + ))?; + let permit = self.authorization.permit(); + if group.identity().member != self.member + || status.permit.as_ref() != Some(permit) + || &status.binding != permit.binding() + || status.consumed_sequence != permit.sequence() + || !matches!( + status.phase, + Some( + FleetPermitPhase::Seeding + | FleetPermitPhase::Joint + | FleetPermitPhase::Complete + ) + ) + { + return Err(CellGroupControllerError::Invalid( + "installed learner permit", + )); + } + Ok(()) + } +} + +pub(super) fn same_admission(a: Option<&Admission>, b: Option<&Admission>) -> bool { + match (a, b) { + (None, None) => true, + (Some(a), Some(b)) => { + a.authorization.permit() == b.authorization.permit() + && a.authorization.target_boot() == b.authorization.target_boot() + && a.member == b.member + && a.source == b.source + && a.source_node == b.source_node + } + _ => false, + } +} + +pub(super) struct Staged { + pub prepared: prepare::Prepared, + session: CellGroupSeedSession, + birth: CellGroupBirth, + config: Arc, + transport: RaftTransport, + admission: Admission, + _parallel: tokio::sync::OwnedSemaphorePermit, +} + +impl Staged { + pub async fn install(self) -> Result { + let (permit, current) = self + .session + .install(self.prepared.applier.clone()) + .await + .map_err(seed_error)?; + self.admission.check_installed(permit.group())?; + let route = physical_route( + &self.birth, + permit.group(), + &self.config, + &self.transport, + self.admission.authorization.minimum_trust_revision(), + )?; + Ok(CellGroupRegistration { + permit, + current, + applier: self.prepared.applier, + transport: route, + }) + } +} + +pub(super) async fn recover( + directory: &RangeDirectory, + birth: &CellGroupBirth, + config: &Arc, + runtime: &CellGroupRuntime, + transport: &RaftTransport, + host: &FleetHostAuthorization, + admission: &Admission, + deadline: u64, + existing: Option, + parallel: tokio::sync::OwnedSemaphorePermit, +) -> Result { + let log = runtime.log(); + let installed = match log.cell_group(&birth.route().history) { + Ok(group) => group + .read() + .map_err(RuntimeError::from)? + .group_seed + .is_some(), + Err(LogError::NotFound) => false, + Err(error) => return Err(RuntimeError::from(error).into()), + }; + let images = config.images(directory)?; + if installed { + // A lost install response cannot authorize downloading/reinitializing a + // different cut. The durable receipt and permanent claim select restart. + let permit = directory + .bootstrap_managed_joined_group_member(birth, admission.member, log, host) + .await + .map_err(catalog_error)?; + admission.check_installed(permit.group())?; + let prepared = + prepare::recover_pages(config, runtime, permit.group().clone(), images, existing) + .await?; + let route = physical_route( + birth, + permit.group(), + config, + transport, + admission.authorization.minimum_trust_revision(), + )?; + return Ok(prepare::Recovery::Ready(CellGroupRegistration { + permit, + current: prepared.current, + applier: prepared.applier, + transport: route, + })); + } + let peer = transport + .managed_group_peer( + &birth.route().history, + admission.source_node, + admission.authorization.minimum_trust_revision(), + ) + .map_err(CellGroupControllerError::Transport)?; + let local = log.limits(); + let target = ReplicaIdentity { + scope: birth.route().history, + member: admission.member, + node: local.node, + disk: local.disk, + process: local.process, + }; + let source = RaftPeer { + replica: ReplicaIdentity { + scope: target.scope, + member: admission.source, + node: peer.node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label, + server_name: peer.server_name, + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + }; + let session = transport + .prepare_managed_group_seed( + target, + source, + directory.clone(), + birth.clone(), + deadline, + host.clone(), + admission.authorization.clone(), + ) + .await + .map_err(seed_error)?; + let prepared = + prepare::recover_pages(config, runtime, session.group().clone(), images, existing).await?; + Ok(prepare::Recovery::Seed(Staged { + prepared, + session, + birth: birth.clone(), + config: config.clone(), + transport: transport.clone(), + admission: admission.clone(), + _parallel: parallel, + })) +} + +fn physical_route( + birth: &CellGroupBirth, + group: &GroupLog, + config: &Config, + transport: &RaftTransport, + minimum_revision: u64, +) -> Result, CellGroupControllerError> { + let bindings = { + let cache = group.read().map_err(RuntimeError::from)?; + members(&cache.conf_state) + .into_iter() + .map(|id| { + let member = MemberId::new(id) + .map_err(|_| CellGroupControllerError::Invalid("member id"))?; + let binding = cache + .membership + .joined(member) + .map_err(RuntimeError::from)? + .map(|(b, _)| (b.node, b.disk)); + let node = binding + .map(|b| b.0) + .or_else(|| { + birth + .voters() + .iter() + .chain(birth.learners()) + .find(|b| b.member == member) + .map(|b| b.node) + }) + .ok_or(CellGroupControllerError::Invalid("member node binding"))?; + Ok((member, node, binding.map(|b| b.1))) + }) + .collect::, CellGroupControllerError>>()? + }; + // Resolve after the group read guard drops. Each endpoint is independently + // checked against rooted trust again by physical registration. + let mut peers = Vec::new(); + for (member, node, disk) in bindings { + let peer = + match transport.managed_group_peer(&birth.route().history, node, minimum_revision) { + Ok(peer) => peer, + Err(TransportError::StalePeer | TransportError::Refused) + if node != group.identity().node => + { + continue + } + Err(error) => return Err(CellGroupControllerError::Transport(error)), + }; + if disk.is_some_and(|disk| disk != peer.disk) { + return Err(CellGroupControllerError::Invalid( + "joined member disk changed", + )); + } + peers.push(RaftPeer { + replica: ReplicaIdentity { + scope: birth.route().history, + member, + node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label, + server_name: peer.server_name, + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + }); + } + Ok(PhysicalRoute { + scope: birth.route().history, + schema_digest: birth.route().schema_digest, + revision: config.settings.peer_revision, + peers, + }) +} + +fn seed_error(error: CellGroupSeedError) -> CellGroupControllerError { + match error { + CellGroupSeedError::Transport(e) => CellGroupControllerError::Transport(e), + CellGroupSeedError::Catalog(e) => catalog_error(e), + CellGroupSeedError::Log(e) => RuntimeError::from(e).into(), + CellGroupSeedError::Checkpoint(e) => CellGroupControllerError::Checkpoint(Arc::new(e)), + } +} diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs index b79eb75fd..b6280e494 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/manager.rs @@ -10,6 +10,7 @@ struct Entry { directory: RangeDirectory, route: GroupRoute, config: Arc, + authority: Authority, reservation: registration::Reservation, permit: Option, prepared: Option, @@ -21,18 +22,26 @@ struct Entry { pub(in super::super) enum Completed { Prepared( GroupHistory, - Result, CellGroupControllerError>, + Result, CellGroupControllerError>, ), Attached( GroupHistory, Result, ), + Checked( + GroupHistory, + Result<(), CellGroupControllerError>, + Waiter, + Permit, + ), } impl Completed { pub(in super::super) fn group(&self) -> GroupHistory { match self { - Self::Prepared(group, _) | Self::Attached(group, _) => *group, + Self::Prepared(group, _) | Self::Attached(group, _) | Self::Checked(group, _, _, _) => { + *group + } } } } @@ -52,11 +61,12 @@ impl Manager { self.jobs.next().await } - pub fn request(&mut self, request: Request, shared: &Shared) { + pub fn request(&mut self, request: Request, shared: &Shared, transport: &RaftTransport) { let Request { directory, route, config, + authority, waiter, } = request; let group = route.history; @@ -68,7 +78,10 @@ impl Manager { } }; if let Some(entry) = self.entries.get_mut(&group) { - if !entry.directory.same_owner(&directory) || entry.route != route { + if !entry.directory.same_owner(&directory) + || entry.route != route + || !entry.authority.same_owner(&authority) + { waiter.finish(Err(CellGroupControllerError::Invalid( "conflicting provision binding", ))); @@ -77,19 +90,32 @@ impl Manager { if entry.active { // This owner already attached the exact assignment. No network // read or fresh quorum claim is implied by this idempotent reply. - waiter.finish(if ready { - Ok(()) + if !ready { + waiter.finish(Err(RuntimeError::NotFound.into())); + return; + } + if matches!(authority, Authority::Initial) { + waiter.finish(Ok(())); } else { - Err(RuntimeError::NotFound.into()) - }); + self.validate_attached(directory, route, config, authority, waiter, shared); + } return; } entry.waiters.push(waiter); if !entry.running { - self.launch(group, shared.runtime.clone()); + // A completed attempt may be retried with a fresh invocation + // deadline. A still-entered attempt never extends its deadline. + entry.authority = authority; + self.launch(group, shared.runtime.clone(), transport.clone()); } return; } + if ready && matches!(authority, Authority::Managed { .. }) { + // Observe an existing controller-owned attachment without adopting + // its applier, roster reservation or physical route lifecycle. + self.validate_attached(directory, route, config, authority, waiter, shared); + return; + } let admitted = (|| { // Manual registrations cannot be adopted by a different lifecycle. if ready { @@ -115,6 +141,7 @@ impl Manager { directory, route, config, + authority, reservation, permit: Some(permit), prepared: None, @@ -123,22 +150,54 @@ impl Manager { active: false, }, ); - self.launch(group, shared.runtime.clone()); + self.launch(group, shared.runtime.clone(), transport.clone()); } - fn launch(&mut self, group: GroupHistory, runtime: CellGroupRuntime) { + fn validate_attached( + &mut self, + directory: RangeDirectory, + route: GroupRoute, + config: Arc, + authority: Authority, + waiter: Waiter, + shared: &Shared, + ) { + let permit = config.profile(&directory).and_then(|_| { + config + .preparations + .acquire(config.bytes) + .ok_or(RuntimeError::Overloaded.into()) + }); + let permit = match permit { + Ok(permit) => permit, + Err(error) => { + waiter.finish(Err(error)); + return; + } + }; + let runtime = shared.runtime.clone(); + self.jobs.push(Box::pin(async move { + let result = prepare::validate(directory, &route, config, runtime, authority).await; + Completed::Checked(route.history, result, waiter, permit) + })); + } + + fn launch(&mut self, group: GroupHistory, runtime: CellGroupRuntime, transport: RaftTransport) { let entry = self.entries.get_mut(&group).expect("provision entry"); entry.running = true; let directory = entry.directory.clone(); let route = entry.route.clone(); let config = entry.config.clone(); let prepared = entry.prepared.clone(); + let authority = entry.authority.clone(); self.jobs.push(Box::pin(async move { Completed::Prepared( group, - prepare::recover(directory, route, config, runtime, prepared) - .await - .map(Box::new), + prepare::recover( + directory, route, config, runtime, transport, authority, prepared, + ) + .await + .map(Box::new), ) })); } @@ -153,6 +212,29 @@ impl Manager { ) -> Result<(), CellGroupControllerError> { match done { Completed::Prepared(group, Ok(input)) => { + let input = match *input { + prepare::Recovery::Seed(staged) => { + let entry = self.entries.get_mut(&group).expect("staged entry"); + // Save the physical owner before entering transfer. Even + // a lost/expired install reply must retain these pins. + entry.prepared = Some(staged.prepared.clone()); + if let Err(error) = entry.authority.check_live(&shared.runtime) { + return self.failed(group, error); + } + self.jobs.push(Box::pin(async move { + Completed::Prepared( + group, + staged + .install() + .await + .map(prepare::Recovery::Ready) + .map(Box::new), + ) + })); + return Ok(()); + } + prepare::Recovery::Ready(input) => input, + }; let entry = self.entries.get_mut(&group).expect("prepared entry"); // Save the owners before polling attachment: early Raft events // must see the applier, and a catalog refusal must retain pins. @@ -160,13 +242,16 @@ impl Manager { applier: input.applier.clone(), current: input.current.clone(), }); + if let Err(error) = entry.authority.check_live(&shared.runtime) { + return self.failed(group, error); + } roster.insert(group, input.applier.clone()); let runtime = shared.runtime.clone(); let transport = transport.clone(); self.jobs.push(Box::pin(async move { Completed::Attached( group, - registration::attach(*input, runtime, transport).await, + registration::attach(input, runtime, transport).await, ) })); Ok(()) @@ -186,13 +271,35 @@ impl Manager { entry.running = false; entry.prepared = None; entry.permit = None; + // Attachment can have finished after the caller expired. + // Keep the real active slot/route, but suppress late success. + let result = entry.authority.check_live(&shared.runtime); for waiter in std::mem::take(&mut entry.waiters) { - waiter.finish(Ok(())); + waiter.finish(result.clone()); } Ok(()) } Err(error) => self.failed(group, error), }, + Completed::Checked(group, result, waiter, _permit) => { + let result = result.and_then(|()| { + if shared.groups.contains(&group)? { + Ok(()) + } else { + Err(RuntimeError::NotFound.into()) + } + }); + let fatal = result + .as_ref() + .err() + .filter(|error| failure::node_wide(error)) + .cloned(); + waiter.finish(result); + match fatal { + Some(error) => Err(error), + None => Ok(()), + } + } } } @@ -250,13 +357,17 @@ impl Manager { // blocking recovery and attachment before releasing dependencies. while let Some(done) = self.jobs.next().await { if let Completed::Prepared(group, Ok(input)) = done { + let prepared = match *input { + prepare::Recovery::Ready(input) => prepare::Prepared { + applier: input.applier, + current: input.current, + }, + prepare::Recovery::Seed(staged) => staged.prepared, + }; self.entries .get_mut(&group) .expect("closing entry") - .prepared = Some(prepare::Prepared { - applier: input.applier, - current: input.current, - }); + .prepared = Some(prepared); } } for entry in self.entries.values_mut() { diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs index 27a236c01..a5cfde466 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/mod.rs @@ -2,10 +2,11 @@ //! One preparation per physical group, owned and joined by the node controller. mod config; +mod learner; mod manager; mod prepare; -pub use config::CellGroupProvisioning; pub(super) use config::Config; +pub use config::{CellGroupProvisionLimits, CellGroupProvisionNamespace, CellGroupProvisioning}; pub(super) use manager::Manager; use super::{calls::Delivery, *}; @@ -29,24 +30,29 @@ fn catalog_error(error: CatalogError) -> CellGroupControllerError { /// locally attached, not elected, caught up, or authorized to serve a request. pub struct CellGroupProvisionCall { receiver: oneshot::Receiver>, + deadline: Option<(crate::asyncrt::DomainToken, u64)>, } impl CellGroupProvisionCall { pub async fn response(self) -> Result<(), Arc> { - self.receiver + let delivery = self + .receiver .await - .map_err(|_| Arc::new(CellGroupControllerError::Closed))? - .result + .map_err(|_| Arc::new(CellGroupControllerError::Closed))?; + delivery.result?; + check_delivery_deadline(&self.deadline).map_err(Arc::new) } } -struct Waiter { +pub(super) struct Waiter { reply: oneshot::Sender>, permit: Arc, + deadline: Option<(crate::asyncrt::DomainToken, u64)>, } impl Waiter { fn finish(self, result: Result<(), CellGroupControllerError>) { + let result = result.and_then(|()| check_delivery_deadline(&self.deadline)); let _ = self.reply.send(Delivery { result: result.map_err(Arc::new), _permit: self.permit, @@ -54,13 +60,100 @@ impl Waiter { } } +fn check_delivery_deadline( + deadline: &Option<(crate::asyncrt::DomainToken, u64)>, +) -> Result<(), CellGroupControllerError> { + if let Some((domain, deadline)) = deadline { + if domain + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)? + >= *deadline + { + return Err(RuntimeError::DeadlineExceeded.into()); + } + } + Ok(()) +} + pub(super) struct Request { directory: RangeDirectory, route: GroupRoute, config: Arc, + authority: Authority, waiter: Waiter, } +#[derive(Clone)] +enum Authority { + Initial, + Managed { + learner: Option, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + deadline: u64, + }, +} + +impl Authority { + fn same_owner(&self, other: &Self) -> bool { + match (self, other) { + (Self::Initial, Self::Initial) => true, + ( + Self::Managed { + host: a, + learner: x, + .. + }, + Self::Managed { + host: b, + learner: y, + .. + }, + ) => a.same_boot_owner(b) && learner::same_admission(x.as_ref(), y.as_ref()), + _ => false, + } + } + + fn check_live(&self, runtime: &CellGroupRuntime) -> Result<(), CellGroupControllerError> { + if let Self::Managed { host, deadline, .. } = self { + let now = runtime + .log() + .domain() + .mono_ms() + .map_err(|_| CellGroupControllerError::Closed)?; + let remaining = deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .ok_or(RuntimeError::DeadlineExceeded)?; + if remaining > 60_000 { + return Err(CellGroupControllerError::Invalid( + "group startup deadline bound", + )); + } + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let log = runtime.log(); + if boot.node != log.limits().node.bytes() + || boot.disk != log.limits().disk.bytes() + || boot.process != log.limits().process.bytes() + || boot.engine + != log + .storage_instance() + .map_err(|e| catalog_error(CatalogError::Log(e)))? + .bytes() + || log + .lifecycle_guard() + .map_err(|e| catalog_error(CatalogError::Log(e)))? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(catalog_error(CatalogError::Fenced)); + } + } + Ok(()) + } +} + impl CellGroupController { /// Install immutable node resources once. Reconfiguration requires a joined /// restart so pending recovery cannot change filesystem or image owners. @@ -93,11 +186,72 @@ impl CellGroupController { &self, directory: RangeDirectory, route: GroupRoute, + ) -> Result { + self.provision_authorized(directory, route, Authority::Initial) + } + + /// Initial members require their exact managed boot and fresh rooted birth. + /// The caller's deadline controls admission and delivery; accepted physical + /// work remains owned and joined after response loss or deadline expiry. + pub(crate) fn provision_managed_initial( + &self, + directory: RangeDirectory, + route: GroupRoute, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + deadline: u64, + ) -> Result { + self.provision_authorized( + directory, + route, + Authority::Managed { + host, + deadline, + learner: None, + }, + ) + } + + /// A learner must retain the exact repair admission and complete authenticated + /// seed. This path never creates genesis from a placement reservation. + pub(crate) fn provision_managed_learner( + &self, + directory: RangeDirectory, + route: GroupRoute, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + authorization: crate::raft::fleet_catalog::FleetPermitAuthorization, + activation: crate::application::consensus::client::LearnerActivation, + deadline: u64, + ) -> Result { + if activation.from_s3 { + // Exact-reference archive seeding is a separate transfer. Never + // interpret an unavailable S3 path as permission to fetch a peer. + return Err(CellGroupControllerError::Invalid( + "S3 learner transfer unavailable", + )); + } + let learner = learner::Admission::new(&route, &host, authorization, activation)?; + self.provision_authorized( + directory, + route, + Authority::Managed { + host, + deadline, + learner: Some(learner), + }, + ) + } + + fn provision_authorized( + &self, + directory: RangeDirectory, + route: GroupRoute, + authority: Authority, ) -> Result { let shared = &self.handle.0; if shared.closing.load(Ordering::Acquire) { return Err(CellGroupControllerError::Closed); } + authority.check_live(&shared.runtime)?; let config = shared .provisioning .lock() @@ -111,15 +265,21 @@ impl CellGroupController { .acquire(WAITER_BYTES) .ok_or(RuntimeError::Overloaded)?; let (reply, receiver) = oneshot::channel(); + let deadline = match &authority { + Authority::Initial => None, + Authority::Managed { deadline, .. } => Some((shared.domain.clone(), *deadline)), + }; shared .requests .try_send(requests::Request::Provision(Box::new(Request { directory, route, config, + authority, waiter: Waiter { reply, permit: Arc::new(permit), + deadline: deadline.clone(), }, }))) .map_err(|e| match e { @@ -128,6 +288,6 @@ impl CellGroupController { } tokio::sync::mpsc::error::TrySendError::Full(_) => RuntimeError::Overloaded.into(), })?; - Ok(CellGroupProvisionCall { receiver }) + Ok(CellGroupProvisionCall { receiver, deadline }) } } diff --git a/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs index 7ba87057c..228e4e9d3 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/provision/prepare.rs @@ -11,22 +11,41 @@ pub(super) struct Prepared { pub current: PreparedGroupManifest, } +pub(super) enum Recovery { + Ready(CellGroupRegistration), + Seed(learner::Staged), +} + pub(super) async fn recover( directory: RangeDirectory, route: GroupRoute, config: Arc, runtime: CellGroupRuntime, + transport: RaftTransport, + authority: Authority, existing: Option, -) -> Result { +) -> Result { + authority.check_live(&runtime)?; + let images = config.images(&directory)?; let _parallel = config .parallel - .acquire() + .clone() + .acquire_owned() .await .map_err(|_| CellGroupControllerError::Closed)?; - let birth = directory - .materialize_selected_group(&route) - .await - .map_err(catalog_error)?; + let birth = resolve(&directory, &route, &runtime, &authority).await?; + if let Authority::Managed { + host, + deadline, + learner: Some(admission), + } = &authority + { + return learner::recover( + &directory, &birth, &config, &runtime, &transport, host, admission, *deadline, + existing, _parallel, + ) + .await; + } // Check complete trusted placement before creating genesis or page files. let mut peers = Vec::new(); for member in birth.voters().iter().chain(birth.learners()) { @@ -53,53 +72,10 @@ pub(super) async fn recover( query_address: node.query_address, }); } - let permit = directory - .bootstrap_group_member(&birth, runtime.log()) - .await - .map_err(catalog_error)?; - let prepared = match existing { - Some(prepared) => prepared, - None => { - let config = config.clone(); - let group = permit.group().clone(); - runtime - .log() - .domain() - .blocking(move || { - let settings = &config.settings; - let history = group.identity().scope; - let path = settings.directory.join(format!( - "{}-{}-{}-{}", - hex::encode(history.cluster.bytes()), - hex::encode(history.namespace.bytes()), - hex::encode(history.group.bytes()), - hex::encode(history.incarnation.bytes()) - )); - let pages = ManifestPageStore::open( - &path, - settings.filesystem.clone(), - settings.max_pages, - settings.max_page_bytes, - settings.max_retained_roots, - ) - .map_err(crate::raft::execution::storage)?; - let (applier, current) = CellGroupApplier::recover( - group, - pages, - settings.images.clone(), - settings.manifest, - settings.entries, - settings.max_effect_rows, - )?; - Ok::<_, ExecutionError>(Prepared { applier, current }) - }) - .map_err(|_| CellGroupControllerError::Closed)? - .await - .map_err(|_| CellGroupControllerError::Closed)? - .map_err(apply_error)? - } - }; - Ok(CellGroupRegistration { + let permit = bootstrap(&directory, &birth, &runtime, &authority).await?; + let prepared = + recover_pages(&config, &runtime, permit.group().clone(), images, existing).await?; + Ok(Recovery::Ready(CellGroupRegistration { permit, current: prepared.current, applier: prepared.applier, @@ -109,5 +85,146 @@ pub(super) async fn recover( revision: config.settings.peer_revision, peers, }, - }) + })) +} + +async fn resolve( + directory: &RangeDirectory, + route: &GroupRoute, + runtime: &CellGroupRuntime, + authority: &Authority, +) -> Result { + authority.check_live(runtime)?; + if let Authority::Managed { host, .. } = authority { + let boot = host + .boot_binding() + .map_err(CatalogError::from) + .map_err(catalog_error)?; + let owner = directory + .namespace() + .cell_group_owner() + .await + .map_err(catalog_error)?; + if owner.is_none_or(|o| o.fleet != host.fleet() || o.catalog != boot.catalog) { + return Err(catalog_error(CatalogError::ManagedAdmissionRequired)); + } + authority.check_live(runtime)?; + } + let birth = match authority { + Authority::Initial => directory.materialize_selected_group(route).await, + Authority::Managed { .. } => directory.materialize_selected_history(route.history).await, + } + .map_err(catalog_error)?; + if birth.route() != route { + return Err(CellGroupControllerError::Invalid("provision birth route")); + } + authority.check_live(runtime)?; + Ok(birth) +} + +async fn bootstrap( + directory: &RangeDirectory, + birth: &crate::raft::cell_groups::CellGroupBirth, + runtime: &CellGroupRuntime, + authority: &Authority, +) -> Result { + authority.check_live(runtime)?; + let permit = match authority { + Authority::Initial => directory.bootstrap_group_member(birth, runtime.log()).await, + Authority::Managed { + host, + learner: Some(admission), + .. + } => { + directory + .bootstrap_managed_joined_group_member(birth, admission.member, runtime.log(), host) + .await + } + Authority::Managed { + host, + learner: None, + .. + } => { + directory + .bootstrap_managed_group_member(birth, runtime.log(), host) + .await + } + } + .map_err(catalog_error)?; + authority.check_live(runtime)?; + Ok(permit) +} + +pub(super) async fn recover_pages( + config: &Arc, + runtime: &CellGroupRuntime, + group: crate::raft::cache::GroupLog, + images: crate::raft::cell_groups::CellGroupApplyImages, + existing: Option, +) -> Result { + let config = config.clone(); + runtime + .log() + .domain() + .blocking(move || { + if let Some(mut prepared) = existing { + // Reuse the retained complete closure only while its exact + // selection/owners are live and publication is settled. A lost + // response with a changed selection recovers on this same owner. + prepared.current = prepared + .applier + .recover_or_reuse_selected(&prepared.current)?; + return Ok(prepared); + } + let settings = &config.settings; + let path = + ManifestPageStore::group_pages_path(&settings.directory, group.identity().scope); + let pages = ManifestPageStore::open( + &path, + settings.filesystem.clone(), + settings.max_pages, + settings.max_page_bytes, + settings.max_retained_roots, + ) + .map_err(crate::raft::execution::storage)?; + let (applier, current) = CellGroupApplier::recover( + group, + pages, + images, + settings.manifest, + settings.entries, + settings.max_effect_rows, + )?; + Ok::<_, ExecutionError>(Prepared { applier, current }) + }) + .map_err(|_| CellGroupControllerError::Closed)? + .await + .map_err(|_| CellGroupControllerError::Closed)? + .map_err(apply_error) +} + +/// An attached retry rechecks catalog/boot authority without opening a second +/// page owner or trying to attach its already claimed consensus slot again. +pub(super) async fn validate( + directory: RangeDirectory, + route: &GroupRoute, + config: Arc, + runtime: CellGroupRuntime, + authority: Authority, +) -> Result<(), CellGroupControllerError> { + let _parallel = config + .parallel + .acquire() + .await + .map_err(|_| CellGroupControllerError::Closed)?; + let birth = resolve(&directory, route, &runtime, &authority).await?; + let permit = bootstrap(&directory, &birth, &runtime, &authority).await?; + if let Authority::Managed { + learner: Some(admission), + .. + } = &authority + { + admission.check_installed(permit.group())?; + } + Ok(()) } diff --git a/crates/cellar/durability/raft/cell_groups/controller/queries.rs b/crates/cellar/durability/raft/cell_groups/controller/queries.rs index 4c1f6a4ef..bd2c7a65d 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/queries.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/queries.rs @@ -1,9 +1,9 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Read callbacks share the node queue and SQL owner, independently of waiters. +use super::admission::Retention; use super::calls::Delivery; use super::*; -use crate::raft::budget::Permit; use crate::raft::cell_groups::{ CellDispatchError, CellQueryInvocation, CellQueryResponse, CellSqlOptions, }; @@ -13,14 +13,14 @@ use tokio::sync::oneshot; pub(super) struct Request { invocation: CellQueryInvocation, options: CellSqlOptions, - reply: oneshot::Sender>, - permit: Arc, + reply: oneshot::Sender>, + permit: Arc, } /// Dropping or expiring the response does not cancel an admitted query callback. /// Its physical connection and reservation remain owned until the callback exits. pub struct CellQueryCall { - inner: calls::Call, + inner: calls::Call, } impl CellQueryCall { @@ -39,6 +39,7 @@ impl CellGroupController { ) -> Result { let shared = &self.handle.0; let (identity, deadline, bytes) = invocation.controller_admission(&shared.domain)?; + let mut registration = shared.registration().map_err(CellDispatchError::Runtime)?; if shared.closing.load(Ordering::Acquire) { return Err(CellDispatchError::Runtime(RuntimeError::Closed)); } @@ -56,7 +57,10 @@ impl CellGroupController { invocation, options, reply, - permit: Arc::new(permit), + permit: Arc::new(Retention { + _permit: permit, + _work: registration.work.take(), + }), }))) .map_err(|error| { CellDispatchError::Runtime(match error { @@ -100,7 +104,16 @@ pub(super) async fn drive( { return Err(CellDispatchError::Runtime(RuntimeError::NotFound)); } - invocation.execute(&runtime, &sql, options).await + invocation + .execute_retained( + &runtime, + &sql, + options, + Some(crate::raft::host_sql::cell_groups::ReadRetention::new( + permit.clone(), + )), + ) + .await } .await .map_err(Arc::new); diff --git a/crates/cellar/durability/raft/cell_groups/controller/registration.rs b/crates/cellar/durability/raft/cell_groups/controller/registration.rs index 30fa9cb38..dc588b67e 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/registration.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/registration.rs @@ -21,6 +21,14 @@ pub(crate) struct GroupRoster { } impl GroupRoster { + pub(super) fn producers_quiesced(&self) -> bool { + self.groups.lock().is_ok_and(|groups| { + groups + .values() + .all(|state| matches!(state, GroupState::Active)) + }) + } + pub(super) fn new(groups: impl Iterator, limit: usize) -> Arc { Arc::new(Self { groups: Mutex::new(groups.map(|g| (g, GroupState::Active)).collect()), @@ -220,6 +228,34 @@ pub(super) struct Completed { } impl CellGroupController { + /// One locked roster cut. Failed attachments remain registered physical + /// debt; pending slots are separate from attached histories. + pub(crate) fn registration_inventory( + &self, + ) -> Result<(Vec, usize, usize), RuntimeError> { + let groups = self + .handle + .0 + .groups + .groups + .lock() + .map_err(|_| RuntimeError::Closed)?; + let mut attached = Vec::with_capacity(groups.len()); + let mut pending = 0; + let mut quarantined = 0; + for (history, state) in groups.iter() { + match state { + GroupState::Pending => pending += 1, + GroupState::Active => attached.push(*history), + GroupState::Quarantined(_) => { + attached.push(*history); + quarantined += 1; + } + } + } + Ok((attached, pending, quarantined)) + } + /// Atomically reserve the controller slot and transfer attachment ownership. /// Admission failure returns all prepared owners without starting attachment. pub fn register( diff --git a/crates/cellar/durability/raft/cell_groups/controller/worker.rs b/crates/cellar/durability/raft/cell_groups/controller/worker.rs index 7a09a9ef2..0368a7de2 100644 --- a/crates/cellar/durability/raft/cell_groups/controller/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/controller/worker.rs @@ -204,7 +204,7 @@ impl Worker { match request { requests::Request::Archive(request) => self.archives.request(*request, &self.shared), requests::Request::S3(request) => self.archives.settle(*request), - requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared), + requests::Request::Provision(request) => self.provisioning.request(*request, &self.shared, &self.transport), requests::Request::Registration(request) => { let group = request.registration.applier.history(); if self.roster.insert(group, request.registration.applier.clone()).is_some() { @@ -305,6 +305,7 @@ impl Worker { continue; } }; + let installed_checkpoint = matches!(&effect, CellGroupEffect::CheckpointInstalled { .. }); match effect { CellGroupEffect::Message(message) => { if let Err(error) = self.transport.submit_physical(&group, message.clone()) { @@ -342,6 +343,15 @@ impl Worker { self.pending.push_back(Pending { group, work: Work::Prepare(Box::new(work)), retry_at: now }); } CellGroupEffect::Applied { manifest, .. } | CellGroupEffect::CheckpointInstalled { manifest } => { + // Configuration apply sets its revision to this + // entry's index. Ordinary cell writes do not scan routes. + if installed_checkpoint || manifest.membership_revision() == manifest.position().index.get() { + if let Err(error) = self.transport.reconcile_managed_physical(&group) { + if !matches!(error, TransportError::Overloaded | TransportError::UnknownGroup | TransportError::StalePeer | TransportError::Refused) { + return Err(CellGroupControllerError::Transport(error)); + } + } + } self.archives.applied(group, manifest.position().index); self.effects.observe(group, manifest.position().index, manifest.root().next_due_unix_millis())?; // Only this post-sync event advances the published count. diff --git a/crates/cellar/durability/raft/cell_groups/directory/birth.rs b/crates/cellar/durability/raft/cell_groups/directory/birth.rs index 65acc29a1..45c610ce4 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/birth.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/birth.rs @@ -17,7 +17,7 @@ pub struct CellGroupBirthSpec { /// Persist the canonical request before reserving its global group ID. Retrying /// must keep these bytes, including placement, creation identity and incarnation. -#[derive(Clone)] +#[derive(Clone, Debug)] pub struct CellGroupBirthRequest { bytes: Vec, route: GroupRoute, @@ -57,6 +57,10 @@ pub struct CellGroupBirth { } impl CellGroupBirth { + pub fn encode(&self) -> &[u8] { + self.request.encode() + } + pub fn route(&self) -> &GroupRoute { self.request.route() } @@ -75,6 +79,55 @@ impl CellGroupBirth { } impl RangeDirectory { + /// Resolve a physical placement by its complete history. The permanent + /// reservation only supplies candidate metadata: directory selection is + /// rechecked before materializing it, and bootstrap still requires a permit. + pub async fn materialize_selected_history( + &self, + history: GroupHistory, + ) -> Result { + if history.cluster != self.namespace.config.cluster + || history.namespace != self.namespace.config.namespace + { + return Err(CatalogError::Conflict("group placement namespace")); + } + let request = self + .bounded(async { + let key = Path::from(format!( + "{}groups/{}", + self.namespace.inner.prefix, + hex::encode(history.group.bytes()) + )); + let (bytes, _) = self + .namespace + .inner + .read_versioned_bounded(&key, wire::LIMIT) + .await? + .ok_or(CatalogError::Corrupt("missing group birth reservation"))?; + let reservation: wire::BirthReservation = + record::decode(wire::GROUP_BIRTH_RESERVATION, &bytes, wire::LIMIT)?; + if reservation.recovery == Some([0; 32]) { + return Err(CatalogError::Corrupt("recovery reservation identity")); + } + let request = self.decode_group_birth(&reservation.birth)?; + if request.route().history != history { + return Err(CatalogError::Fenced); + } + Ok(request) + }) + .await?; + if self.namespace.cell_group_owner().await?.is_some() { + let birth = CellGroupBirth { + request, + owner: self.namespace.inner.owner.clone(), + }; + self.require_current_group_birth(&birth).await?; + Ok(birth) + } else { + self.materialize_group_birth(&request).await + } + } + /// Recover the immutable reservation for an already selected range. This /// never chooses placement or allocates another group in response to a cell. pub async fn materialize_selected_group( @@ -103,9 +156,33 @@ impl RangeDirectory { pub fn prepare_group_birth( &self, - mut spec: CellGroupBirthSpec, + spec: CellGroupBirthSpec, ) -> Result { self.unmanaged_birth()?; + self.prepare_birth(spec) + } + + pub(crate) fn prepare_fleet_group_birth( + &self, + spec: CellGroupBirthSpec, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result { + if self + .namespace + .config + .managed_fleet + .as_ref() + .is_some_and(|configured| configured != owner) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.prepare_birth(spec) + } + + fn prepare_birth( + &self, + mut spec: CellGroupBirthSpec, + ) -> Result { if spec.history.cluster != self.namespace.config.cluster || spec.history.namespace != self.namespace.config.namespace { @@ -145,7 +222,6 @@ impl RangeDirectory { } pub fn decode_group_birth(&self, bytes: &[u8]) -> Result { - self.unmanaged_birth()?; let value: wire::Birth = record::decode(wire::GROUP_BIRTH, bytes, wire::LIMIT)?; if value.namespace != self.namespace.root_digest { return Err(CatalogError::Conflict("group birth namespace")); @@ -205,6 +281,29 @@ impl RangeDirectory { &self, request: &CellGroupBirthRequest, ) -> Result<(), CatalogError> { + self.unmanaged_birth()?; + self.reserve_birth_record( + request, + &wire::BirthReservation { + birth: request.encode().to_vec(), + recovery: None, + }, + ) + .await?; + self.bounded(self.namespace.admit_cell_group(request)).await + } + + /// A provisional permanent identity claim precedes the fleet root CAS. + /// Losing that CAS retains this claim; it grants no materialization, + /// bootstrap or quorum authority without the rooted birth capability. + pub(crate) async fn reserve_fleet_group_identity( + &self, + request: &CellGroupBirthRequest, + owner: &crate::raft::catalog::ManagedBirthOwner, + ) -> Result<(), CatalogError> { + if self.namespace.cell_group_owner().await?.as_ref() != Some(owner) { + return Err(CatalogError::ManagedAdmissionRequired); + } self.reserve_birth_record( request, &wire::BirthReservation { @@ -269,6 +368,11 @@ impl RangeDirectory { if recovery.is_some() { return Err(CatalogError::Fenced); } + if self.namespace.config.managed_fleet.is_none() + && self.namespace.cell_group_owner().await?.is_none() + { + self.namespace.verify_cell_group_intent(&request).await?; + } Ok(request) } @@ -335,6 +439,40 @@ impl RangeDirectory { pub async fn materialize_group_birth( &self, request: &CellGroupBirthRequest, + ) -> Result { + self.unmanaged_birth()?; + self.namespace.require_cell_group_unmanaged().await?; + self.materialize_birth(request).await + } + + /// Only a rooted fleet reservation can create managed birth metadata. + /// Materialization supplies no voter, quorum or application authority. + pub async fn materialize_fleet_group_birth( + &self, + authorization: &crate::raft::fleet_catalog::FleetBirthAuthorization, + ) -> Result { + authorization.check_namespace(&self.namespace)?; + if self.namespace.cell_group_owner().await?.as_ref() != Some(authorization.owner()) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let request = authorization.proposal(); + let decoded = self.decode_group_birth(request.encode())?; + if decoded.encode() != request.encode() { + return Err(CatalogError::Conflict("managed group birth request")); + } + self.materialize_birth(&request).await + } + + pub(super) async fn materialize_import_birth( + &self, + request: &CellGroupBirthRequest, + ) -> Result { + self.materialize_birth(request).await + } + + async fn materialize_birth( + &self, + request: &CellGroupBirthRequest, ) -> Result { self.decode_group_birth(request.encode())?; self.bounded(async { @@ -349,8 +487,11 @@ impl RangeDirectory { wire::LIMIT, ) .await?; - if observed != request.encode() { - return Err(CatalogError::Conflict("selected group birth")); + match self.decode_group_disposition(request, &observed)? { + CellGroupBirthDisposition::Live(_) => (), + CellGroupBirthDisposition::Cancelled { .. } => { + return Err(CatalogError::BirthCancelled) + } } self.selected_birth_route(request.route()).await?; Ok(CellGroupBirth { diff --git a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs index 584b81502..51d00647a 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/bootstrap.rs @@ -27,7 +27,9 @@ pub struct CellGroupBootstrapPermit { enum BootstrapOrigin { Initial, + Managed(Box), Joined, + ManagedJoined(Box), Recovery(Box), } @@ -67,11 +69,26 @@ impl CellGroupBootstrapPermit { .bootstrap_group_member(&self.birth, &self.log) .await .map(|_| ()), + BootstrapOrigin::Managed(host) => self + .directory + .bootstrap_managed_group_member(&self.birth, &self.log, host) + .await + .map(|_| ()), BootstrapOrigin::Joined => self .directory .bootstrap_joined_group_member(&self.birth, self.group.identity().member, &self.log) .await .map(|_| ()), + BootstrapOrigin::ManagedJoined(host) => self + .directory + .bootstrap_managed_joined_group_member( + &self.birth, + self.group.identity().member, + &self.log, + host, + ) + .await + .map(|_| ()), BootstrapOrigin::Recovery(startup) => self .directory .bootstrap_recovered_group_member( @@ -110,6 +127,35 @@ pub(super) fn check_member(group: &GroupLog) -> Result<(), Catalog } impl RangeDirectory { + pub(crate) async fn verify_group_import_member_claim( + &self, + birth: &CellGroupBirth, + member: MemberId, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result<(), CatalogError> { + let history = birth.route().history; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_claim_key(history, member), wire::LIMIT) + .await? + .ok_or(CatalogError::Conflict("missing group import member claim"))? + .0; + let claim: Claim = record::decode(wire::GROUP_CLAIM, &bytes, wire::LIMIT)?; + if claim.namespace != self.namespace.root_digest + || claim.group != history.group.bytes() + || claim.incarnation != history.incarnation.bytes() + || claim.birth != birth.route().birth_digest + || claim.member != member.get() + || claim.node != host.node().bytes() + || claim.disk != host.disk().bytes() + || Some(claim.engine) != host.storage_instance().map(|id| id.bytes()) + { + return Err(CatalogError::Conflict("group import member claim identity")); + } + Ok(()) + } + fn group_claim_key(&self, history: GroupHistory, member: MemberId) -> Path { self.key(&format!( "claims/{}/{}/{:016x}", @@ -129,6 +175,78 @@ impl RangeDirectory { ) -> Result { self.unmanaged_birth()?; self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.bootstrap_initial_group_member(birth, log, None).await + }) + .await + } + + /// Recheck the exact running node/engine boot and the rooted Live group on + /// every startup and attachment. A saved birth or binding is insufficient. + pub async fn bootstrap_managed_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + self.bounded(async { + self.require_managed_group_boot(log, host).await?; + self.bootstrap_initial_group_member(birth, log, Some(host)) + .await + }) + .await + } + + pub(super) async fn require_managed_group_boot( + &self, + log: &ReplicaLog, + host: &crate::raft::fleet_catalog::FleetHostAuthorization, + ) -> Result { + let owner = self + .namespace + .cell_group_owner() + .await? + .ok_or(CatalogError::ManagedAdmissionRequired)?; + let boot = host.boot_binding().map_err(CatalogError::from)?; + if owner.fleet != host.fleet() + || owner.catalog != boot.catalog + || host.cluster() != self.namespace.config.cluster + || host.node() != log.limits().node + || host.disk() != log.limits().disk + || host.storage_instance() != Some(log.storage_instance().map_err(CatalogError::Log)?) + || boot.process != log.limits().process.bytes() + || log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(CatalogError::ManagedAdmissionRequired); + } + Ok(boot) + } + + async fn bootstrap_initial_group_member( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + host: Option<&crate::raft::fleet_catalog::FleetHostAuthorization>, + ) -> Result { + self.bounded(async { + let authorization = if let Some(host) = host { + let authorization = host + .group_authorization( + &birth.route().history, + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + if !authorization.check_birth(self, birth) { + return Err(CatalogError::ManagedAdmissionRequired); + } + Some(authorization) + } else { + None + }; self.require_current_group_birth(birth).await?; self.require_ordinary_birth_reservation(birth.route()) .await?; @@ -173,18 +291,34 @@ impl RangeDirectory { if remote.is_some() && local.is_none() { return Err(CatalogError::LostStableState); } + if local.is_none() + && authorization + .as_ref() + .is_some_and(|authority| !authority.allows_managed_genesis()) + { + return Err(CatalogError::ManagedAdmissionRequired); + } // The worker creates the complete empty root with genesis and only // returns after actual engine sync. Existing mappings must match // the original route, membership and birth, even after later apply. - let group = log - .create_cell_group(CellGroupBootstrap { - route: birth.route().clone(), - member: selected.member, - voters: birth.voters().iter().map(|m| m.member).collect(), - learners: birth.learners().iter().map(|m| m.member).collect(), - }) - .await - .map_err(CatalogError::Log)?; + let bootstrap = CellGroupBootstrap { + route: birth.route().clone(), + member: selected.member, + voters: birth.voters().iter().map(|m| m.member).collect(), + learners: birth.learners().iter().map(|m| m.member).collect(), + }; + let group = match &authorization { + Some(authorization) => { + let boot = host + .ok_or(CatalogError::ManagedAdmissionRequired)? + .boot_binding() + .map_err(CatalogError::from)?; + log.create_managed_cell_group(bootstrap, authorization, &boot) + .await + } + None => log.create_cell_group(bootstrap).await, + } + .map_err(CatalogError::Log)?; check_member(&group)?; let observed = self .namespace @@ -196,12 +330,25 @@ impl RangeDirectory { } self.require_current_group_birth(birth).await?; check_member(&group)?; + if let Some(host) = host { + let boot = host.boot_binding().map_err(CatalogError::from)?; + if log + .lifecycle_guard() + .map_err(CatalogError::Log)? + .is_none_or(|guard| !guard.admits_group_start(&boot)) + { + return Err(CatalogError::Fenced); + } + } Ok(CellGroupBootstrapPermit { group, birth: birth.clone(), directory: self.clone(), log: log.clone(), - origin: BootstrapOrigin::Initial, + origin: match host { + Some(host) => BootstrapOrigin::Managed(Box::new(host.clone())), + None => BootstrapOrigin::Initial, + }, }) }) .await diff --git a/crates/cellar/durability/raft/cell_groups/directory/import.rs b/crates/cellar/durability/raft/cell_groups/directory/import.rs new file mode 100644 index 000000000..eafa21bc5 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/directory/import.rs @@ -0,0 +1,315 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Namespace import first closes admission, then CAS-locks topology. Delayed +//! topology writers lose their exact expected head; neither effect auto-expires. +use super::*; +use crate::raft::catalog::ManagedBirthOwner; +use crate::raft::catalog::{CellGroupImportCut, FrozenCellGroupInventory}; +use serde::{Deserialize, Serialize}; + +const CANCELLED: u8 = 197; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Cancelled { + namespace: [u8; 32], + birth: [u8; 32], + operation: [u8; 16], +} + +/// An immutable exact-key decision. Cancellation retains the admission and +/// permanent identity; it grants no removal or physical-release permission. +pub enum CellGroupBirthDisposition { + Live(CellGroupBirth), + Cancelled { + history: GroupHistory, + operation: ProposalId, + }, +} + +impl RangeDirectory { + /// Settle an unselected managed reservation at the same immutable decision + /// key as materialization. A selected or already live birth remains Live; + /// cancellation never erases a possible replica or releases its resources. + pub async fn cancel_fleet_group_birth( + &self, + authorization: &crate::raft::fleet_catalog::FleetBirthAuthorization, + operation: ProposalId, + ) -> Result { + authorization.check_namespace(&self.namespace)?; + self.bounded(async { + if self.namespace.cell_group_owner().await?.as_ref() != Some(authorization.owner()) { + return Err(CatalogError::ManagedAdmissionRequired); + } + let request = authorization.proposal(); + self.verify_birth_reservation(request.route()).await?; + if let Some(winner) = self.group_birth_disposition(&request).await? { + return Ok(winner); + } + let (view, _) = self.current().await?; + match self.selected_route_in(&view, request.route()).await { + Ok(()) => { + return self + .materialize_import_birth(&request) + .await + .map(CellGroupBirthDisposition::Live) + } + Err(CatalogError::Fenced) => (), + Err(error) => return Err(error), + } + let cancelled = record::encode( + CANCELLED, + &Cancelled { + namespace: self.namespace.root_digest, + birth: request.route().birth_digest, + operation: operation.bytes(), + }, + wire::LIMIT, + )?; + let winner = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + &cancelled, + wire::LIMIT, + ) + .await?; + self.decode_group_disposition(&request, &winner) + }) + .await + } + + /// Freeze the complete admitted group set, including unselected and lazy + /// births. Retry the same operation after loss. Already running groups keep + /// their Raft write path; new topology and birth admission remain closed. + pub async fn freeze_group_births( + &self, + operation: ProposalId, + ) -> Result { + self.unmanaged_birth()?; + self.bounded(async { + self.namespace.begin_cell_group_freeze(operation).await?; + for _ in 0..8 { + let (mut view, version) = self.current().await?; + if view.head.recovery.is_some() { + return Err(CatalogError::Fenced); + } + if let Some(prior) = view.head.import { + if prior != operation.bytes() { + return Err(CatalogError::Conflict("directory import operation")); + } + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let digest = record::digest(&bytes); + self.immutable("import-cuts", digest, &bytes).await?; + return self + .namespace + .finish_cell_group_freeze(operation, digest) + .await; + } + view.head.import = Some(operation.bytes()); + view.head.revision = view + .head + .revision + .checked_add(1) + .ok_or(CatalogError::Limit)?; + self.check_head(&view.head)?; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + // Read back even after a lost CAS response. A missing head is + // ambiguous; it never reopens admission or chooses another ID. + let (observed, _) = self.current().await?; + if observed + .head + .import + .is_some_and(|prior| prior != operation.bytes()) + { + return Err(CatalogError::Conflict("directory import winner")); + } + } + Err(CatalogError::Unconfirmed) + }) + .await + } + + pub(crate) async fn import_directory_view( + &self, + cut: &CellGroupImportCut, + ) -> Result { + cut.validate(&self.namespace)?; + let bytes = self + .read(&format!("import-cuts/{}", hex::encode(cut.directory))) + .await? + .ok_or(CatalogError::Corrupt("missing frozen group directory"))?; + if record::digest(&bytes) != cut.directory { + return Err(CatalogError::Corrupt("frozen group directory digest")); + } + let head: wire::Head = record::decode(wire::HEAD, &bytes, wire::LIMIT)?; + self.check_head(&head)?; + if head.import != Some(cut.operation) || head.recovery.is_some() { + return Err(CatalogError::Corrupt("frozen group directory operation")); + } + Ok(DirectoryView { + head, + digest: cut.directory, + owner: self.namespace.inner.owner.clone(), + }) + } + + pub(crate) async fn check_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + adopted: bool, + ) -> Result<(), CatalogError> { + self.namespace + .check_cell_group_import(cut, owner, adopted) + .await?; + self.import_directory_view(cut).await?; + if !adopted { + let (current, _) = self.current().await?; + if current.digest != cut.directory || current.head.import != Some(cut.operation) { + // A Handoff retry may observe the already adopted and unlocked + // directory after losing the final fleet-root response. + if current.head.import.is_some() { + return Err(CatalogError::Conflict("changed import topology")); + } + self.namespace + .check_cell_group_import(cut, owner, true) + .await?; + } + } + Ok(()) + } + + pub(crate) async fn finish_group_import( + &self, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + ) -> Result<(), CatalogError> { + self.namespace + .change_cell_group_import(cut, owner, true) + .await?; + for _ in 0..8 { + let (mut view, version) = self.current().await?; + if view.head.import.is_none() { + return self.check_group_import(cut, owner, true).await; + } + if view.digest != cut.directory || view.head.import != Some(cut.operation) { + return Err(CatalogError::Conflict("import handoff directory")); + } + view.head.import = None; + view.head.revision = view + .head + .revision + .checked_add(1) + .ok_or(CatalogError::Limit)?; + self.check_head(&view.head)?; + let bytes = record::encode(wire::HEAD, &view.head, wire::LIMIT)?; + let _ = self + .namespace + .inner + .store + .put_opts( + &self.key("head"), + bytes.into(), + PutOptions::from(PutMode::Update(version)), + ) + .await; + } + Err(CatalogError::Unconfirmed) + } + + pub(crate) fn decode_group_disposition( + &self, + request: &CellGroupBirthRequest, + bytes: &[u8], + ) -> Result { + if bytes == request.encode() { + return Ok(CellGroupBirthDisposition::Live(CellGroupBirth { + request: request.clone(), + owner: self.namespace.inner.owner.clone(), + })); + } + let cancelled: Cancelled = record::decode(CANCELLED, bytes, wire::LIMIT)?; + if cancelled.namespace != self.namespace.root_digest + || cancelled.birth != request.route().birth_digest + { + return Err(CatalogError::Corrupt("cancelled group birth identity")); + } + let operation = ProposalId::new(cancelled.operation) + .map_err(|_| CatalogError::Corrupt("group cancellation operation"))?; + Ok(CellGroupBirthDisposition::Cancelled { + history: request.route().history, + operation, + }) + } + + pub(crate) async fn group_birth_disposition( + &self, + request: &CellGroupBirthRequest, + ) -> Result, CatalogError> { + self.verify_birth_reservation(request.route()).await?; + let bytes = self + .namespace + .inner + .read_versioned_bounded(&self.group_birth_key(request.route()), wire::LIMIT) + .await?; + bytes + .map(|(bytes, _)| self.decode_group_disposition(request, &bytes)) + .transpose() + } + + pub(crate) async fn settle_import_group( + &self, + request: &CellGroupBirthRequest, + cut: &CellGroupImportCut, + owner: &ManagedBirthOwner, + ) -> Result { + self.check_group_import(cut, owner, false).await?; + self.namespace.verify_cell_group_intent(request).await?; + let view = self.import_directory_view(cut).await?; + // Every selected range must remain live. A lazy selected birth is + // materialized, never confused with an unselected cancelled intent. + match self.selected_route_in(&view, request.route()).await { + Ok(()) => { + return self + .materialize_import_birth(request) + .await + .map(CellGroupBirthDisposition::Live) + } + Err(CatalogError::Fenced) => (), + Err(error) => return Err(error), + } + let cancelled = record::encode( + CANCELLED, + &Cancelled { + namespace: self.namespace.root_digest, + birth: request.route().birth_digest, + operation: cut.operation, + }, + wire::LIMIT, + )?; + let bytes = self + .namespace + .inner + .create_and_observe_bounded( + &self.group_birth_key(request.route()), + &cancelled, + wire::LIMIT, + ) + .await?; + // A materializer that won before the topology lock keeps its original + // immutable live decision. It must be covered too, even if unselected. + self.decode_group_disposition(request, &bytes) + } +} diff --git a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs index 283bb4568..2b9a444a9 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/joined_bootstrap.rs @@ -3,7 +3,10 @@ //! A staging mapping never grants startup. Only a locally fenced authenticated //! install plus the exact permanent remote claim can resume a joined member. use super::*; -use crate::raft::transport::CellGroupSeedGrant; +use crate::raft::{ + fleet_catalog::{FleetHostAuthorization, FleetPermitAuthorization}, + transport::CellGroupSeedGrant, +}; impl RangeDirectory { pub(crate) async fn stage_group_learner( @@ -13,8 +16,109 @@ impl RangeDirectory { grant: Arc, ) -> Result, CatalogError> { self.unmanaged_birth()?; + self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.stage_joined_group(birth, log, grant, None).await + }) + .await + } + + pub(crate) async fn stage_managed_group_learner( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + host: &FleetHostAuthorization, + ) -> Result, CatalogError> { + self.stage_joined_group(birth, log, grant, Some(host)).await + } + + /// The same active physical reservation must still name this boot. The + /// returned capability does not substitute for the authenticated seed grant. + pub(crate) async fn authorize_group_learner_seed( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + target: &cellar_logic::consensus::ReplicaIdentity, + source: &cellar_logic::consensus::ReplicaIdentity, + host: &FleetHostAuthorization, + deadline: u64, + ) -> Result { + let remaining = || -> Result { + let now = log.domain().mono_ms().map_err(|_| CatalogError::Fenced)?; + deadline + .checked_sub(now) + .filter(|ms| *ms > 0) + .map(|ms| Duration::from_millis(ms.min(self.limits.operation_timeout_ms))) + .ok_or(CatalogError::Fenced) + }; + crate::asyncrt::timeout(remaining()?, async { + let boot = self.require_managed_group_boot(log, host).await?; + let authorization = host + .admitted_learner_from(&birth.route().history, source, remaining()?) + .await + .map_err(CatalogError::from)?; + let group = host + .group_authorization_at_least( + &birth.route().history, + authorization.minimum_trust_revision(), + remaining()?, + ) + .await + .map_err(CatalogError::from)?; + let replacement = authorization + .permit() + .replacement() + .map_err(CatalogError::Log)?; + if !group.check_birth(self, birth) + || !authorization.matches_scope(&target.scope) + || target.scope != birth.route().history + || authorization.target_boot() != Some(&boot) + || replacement.member != target.member + || replacement.node != target.node + || replacement.disk != target.disk + || target.node != log.limits().node + || target.disk != log.limits().disk + || target.process != log.limits().process + { + return Err(CatalogError::ManagedAdmissionRequired); + } + self.require_current_group_birth(birth).await?; + remaining()?; + self.require_managed_group_boot(log, host).await?; + remaining()?; + Ok(authorization) + }) + .await + .map_err(|_| CatalogError::Unconfirmed)? + } + + async fn stage_joined_group( + &self, + birth: &CellGroupBirth, + log: &ReplicaLog, + grant: Arc, + host: Option<&FleetHostAuthorization>, + ) -> Result, CatalogError> { self.bounded(async { grant.check_log(log).map_err(CatalogError::Log)?; + if let Some(host) = host { + let current = self + .authorize_group_learner_seed( + birth, + log, + grant.local(), + grant.source(), + host, + grant.deadline(), + ) + .await?; + grant + .check_authorization(¤t) + .map_err(CatalogError::Log)?; + } else if grant.managed_binding().is_some() { + return Err(CatalogError::ManagedAdmissionRequired); + } self.require_current_group_birth(birth).await?; let _node = self.namespace.require_node_storage(log).await?; let member = grant.local().member; @@ -60,6 +164,21 @@ impl RangeDirectory { } self.require_current_group_birth(birth).await?; grant.check_log(log).map_err(CatalogError::Log)?; + if let Some(host) = host { + let current = self + .authorize_group_learner_seed( + birth, + log, + grant.local(), + grant.source(), + host, + grant.deadline(), + ) + .await?; + grant + .check_authorization(¤t) + .map_err(CatalogError::Log)?; + } Ok(group) }) .await @@ -100,6 +219,49 @@ impl RangeDirectory { ) -> Result { self.unmanaged_birth()?; self.bounded(async { + self.namespace.require_cell_group_unmanaged().await?; + self.bootstrap_joined_member(birth, member, log, None).await + }) + .await + } + + /// A retained install receipt and permanent engine claim are mandatory even + /// after a repair completes. A saved active permit is not restart authority. + pub async fn bootstrap_managed_joined_group_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + host: &FleetHostAuthorization, + ) -> Result { + self.bootstrap_joined_member(birth, member, log, Some(host)) + .await + } + + async fn bootstrap_joined_member( + &self, + birth: &CellGroupBirth, + member: MemberId, + log: &ReplicaLog, + host: Option<&FleetHostAuthorization>, + ) -> Result { + self.bounded(async { + let authorization = if let Some(host) = host { + self.require_managed_group_boot(log, host).await?; + let group = host + .group_authorization( + &birth.route().history, + Duration::from_millis(self.limits.operation_timeout_ms), + ) + .await + .map_err(CatalogError::from)?; + if !group.check_birth(self, birth) { + return Err(CatalogError::ManagedAdmissionRequired); + } + Some(group) + } else { + None + }; self.require_current_group_birth(birth).await?; let _node = self.namespace.require_node_storage(log).await?; let group = log @@ -123,6 +285,17 @@ impl RangeDirectory { "joined group has no durable seed receipt", )); } + match (&authorization, cache.membership.fleet_status()) { + (Some(authorization), Some(status)) + if &status.binding == authorization.binding() + && authorization + .matches_replica(group.identity(), &cache.membership) => + { + () + } + (None, None) => (), + _ => return Err(CatalogError::ManagedAdmissionRequired), + } drop(cache); check_member(&group) }; @@ -139,12 +312,18 @@ impl RangeDirectory { } self.require_current_group_birth(birth).await?; check()?; + if let Some(host) = host { + self.require_managed_group_boot(log, host).await?; + } Ok(CellGroupBootstrapPermit { group, birth: birth.clone(), directory: self.clone(), log: log.clone(), - origin: BootstrapOrigin::Joined, + origin: match host { + Some(host) => BootstrapOrigin::ManagedJoined(Box::new(host.clone())), + None => BootstrapOrigin::Joined, + }, }) }) .await diff --git a/crates/cellar/durability/raft/cell_groups/directory/mod.rs b/crates/cellar/durability/raft/cell_groups/directory/mod.rs index b7f59da82..337c63415 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/mod.rs @@ -21,6 +21,8 @@ mod archive; pub use archive::{CellGroupArchiveBindingError, PreparedCellGroupArchiveBinding}; mod birth; mod bootstrap; +mod import; +pub use import::CellGroupBirthDisposition; mod inventory; pub use inventory::{CellGroupDirectoryCursor, CellGroupDirectoryPage}; mod recovery; @@ -45,7 +47,8 @@ pub use routing::{ CellRoutingLimits, CellRoutingSession, ResolvedCellRoute, }; -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Copy, Debug, serde::Serialize, serde::Deserialize)] +#[serde(deny_unknown_fields)] pub struct DirectoryLimits { pub max_groups: u32, pub max_mutations: u64, @@ -73,6 +76,14 @@ pub struct RangeDirectory { cache: Arc>, } +impl RangeDirectory { + /// Namespace identity for the owning fleet's rooted declarations. This + /// exposes metadata inside the host, never an application-cell birth API. + pub(crate) fn namespace(&self) -> &NamespaceCatalog { + &self.namespace + } +} + /// A pinned catalog observation, not authority or an automatic freshness claim. #[derive(Clone)] pub struct DirectoryView { @@ -151,7 +162,9 @@ impl RangeDirectory { self.limits.max_cache_bytes } - pub(crate) fn namespace_config(&self) -> &crate::raft::catalog::NamespaceConfig { + /// Immutable namespace properties for binding schema and image resources. + /// This does not refresh routing metadata or grant member admission. + pub fn namespace_config(&self) -> &crate::raft::catalog::NamespaceConfig { self.namespace.config() } @@ -308,6 +321,11 @@ impl RangeDirectory { { return Err(CatalogError::Corrupt("directory head")); } + if head.import.is_some_and(|id| ProposalId::new(id).is_err()) + || (head.import.is_some() && head.recovery.is_some()) + { + return Err(CatalogError::Corrupt("directory import fence")); + } if let Some(recovery) = &head.recovery { recovery.check(self)?; } @@ -369,16 +387,20 @@ impl RangeDirectory { Ok((view, version)) } - async fn bounded( + fn bounded( &self, work: impl std::future::Future>, - ) -> Result { - crate::asyncrt::timeout( - Duration::from_millis(self.limits.operation_timeout_ms), - work, - ) - .await - .map_err(|_| CatalogError::Unconfirmed)? + ) -> impl std::future::Future> { + let duration = Duration::from_millis(self.limits.operation_timeout_ms); + // Bootstrap revalidation nests catalog operations during attachment. + // Box before constructing the timeout future so each wrapper retains + // only a pointer instead of copying the complete nested operation. + let work = Box::pin(work); + async move { + crate::asyncrt::timeout(duration, work) + .await + .map_err(|_| CatalogError::Unconfirmed)? + } } pub async fn observe(&self) -> Result { @@ -490,6 +512,9 @@ impl RangeDirectory { head }; if let Some((view, _)) = ¤t { + if view.head.import.is_some() { + return Err(CatalogError::BirthsFrozen); + } if let Some(publication) = self.confirmed(&view.head, mutation).await? { if request.expected.is_some() { return Ok(publication); @@ -576,6 +601,7 @@ impl RangeDirectory { groups, journal: id, recovery: recovery.map(|grant| grant.published_lock(id)).transpose()?, + import: None, }; self.check_head(&head)?; let bytes = record::encode(wire::HEAD, &head, wire::LIMIT)?; diff --git a/crates/cellar/durability/raft/cell_groups/directory/wire.rs b/crates/cellar/durability/raft/cell_groups/directory/wire.rs index b47364786..cd6e4c01a 100644 --- a/crates/cellar/durability/raft/cell_groups/directory/wire.rs +++ b/crates/cellar/durability/raft/cell_groups/directory/wire.rs @@ -228,6 +228,14 @@ pub(super) struct Head { pub journal: [u8; 32], #[serde(deserialize_with = "required_recovery")] pub recovery: Option>, + #[serde(deserialize_with = "required_import")] + pub import: Option<[u8; 16]>, +} + +fn required_import<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) } // Marker 1 has one record shape. Missing fields are not an older-reader mode. diff --git a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs index 5bf45504d..7a0692b22 100644 --- a/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/dispatch/mod.rs @@ -46,6 +46,7 @@ pub struct CellDispatchLimits { #[derive(Clone, Debug)] pub enum CellQueryConsistency { Stale, + RecentSource { max_age_ms: u64 }, AtLeast(CellVersionToken), Linearizable, } @@ -93,6 +94,10 @@ pub struct CellGroupDispatcher { } impl CellGroupDispatcher { + pub(in crate::raft::cell_groups) fn directory(&self) -> RangeDirectory { + self.directory.clone() + } + pub(crate) fn host_binding( &self, ) -> Result< @@ -139,6 +144,17 @@ impl CellGroupDispatcher { class: Arc, keys: VersionKeys, limits: CellDispatchLimits, + ) -> Result { + let domain = domain.services().domain_token().ok_or_else(closed)?; + Self::from_domain(domain, directory, class, keys, limits) + } + + pub(crate) fn from_domain( + domain: DomainToken, + directory: RangeDirectory, + class: Arc, + keys: VersionKeys, + limits: CellDispatchLimits, ) -> Result { if class.schema_digest() != directory.namespace_config().schema_digest { return Err(CellDispatchError::Registration( @@ -155,7 +171,6 @@ impl CellGroupDispatcher { { return Err(CellDispatchError::Invalid("invocation limits")); } - let domain = domain.services().domain_token().ok_or_else(closed)?; let routes = Arc::new( crate::raft::catalog::RouteCache::new( directory.clone(), @@ -255,6 +270,11 @@ impl CellGroupDispatcher { { return Err(CellDispatchError::Runtime(RuntimeError::Overloaded)); } + if matches!(&request.consistency, CellQueryConsistency::RecentSource { max_age_ms } + if !(1..=3_600_000).contains(max_age_ms)) + { + return Err(CellDispatchError::Invalid("stale source age")); + } if request.contract.kind != OperationKind::Query { return Err(CellDispatchError::Registration( RegistrationError::KindMismatch, @@ -295,6 +315,7 @@ impl CellGroupDispatcher { request, session, attempt: None, + remote_route: None, dispatched: false, _admission: admission, }) @@ -308,6 +329,7 @@ pub struct CellQueryInvocation { request: CellQueryRequest, session: CellRoutingSession, attempt: Option, + remote_route: Option, dispatched: bool, _admission: Permit, } @@ -336,17 +358,29 @@ impl CellQueryInvocation { Ok((self.request.invocation, deadline, bytes)) } + pub(crate) fn bind_remote_route(&mut self, route: RemoteMutationRoute) { + self.remote_route = Some(route); + } + pub fn attempt(&self) -> Option<&CellRouteAttempt> { self.attempt.as_ref() } pub async fn begin(&mut self) -> Result<(), CellDispatchError> { self.dispatched = true; - let attempt = self - .session - .begin() - .await - .map_err(CellDispatchError::Routing)?; + let attempt = if self.remote_route.is_some() { + self.session.begin_current().await + } else { + self.session.begin().await + } + .map_err(CellDispatchError::Routing)?; + if self + .remote_route + .as_ref() + .is_some_and(|route| !route.matches(attempt.route())) + { + return Err(CellDispatchError::Invalid("remote query destination route")); + } self.attempt = Some(attempt); self.dispatched = false; Ok(()) @@ -406,6 +440,16 @@ impl CellQueryInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, + ) -> Result { + self.execute_retained(runtime, sql, options, None).await + } + + pub(crate) async fn execute_retained( + &mut self, + runtime: &CellGroupRuntime, + sql: &CellGroupSqlPool, + options: CellSqlOptions, + retention: Option, ) -> Result { let attempt = self .attempt @@ -426,7 +470,7 @@ impl CellQueryInvocation { .sleep_until(attempt.deadline_ms()) .map_err(|_| closed())?; crate::asyncrt::select! { - result = self.execute_at(attempt, runtime, sql, options) => { + result = self.execute_at(attempt, runtime, sql, options, retention) => { let result = result?; self.session.validate_attempt(attempt).map_err(CellDispatchError::Routing)?; Ok(result) @@ -526,11 +570,17 @@ impl CellQueryInvocation { runtime: &CellGroupRuntime, sql: &CellGroupSqlPool, options: CellSqlOptions, + retention: Option, ) -> Result { let cell = &attempt.request().cell; let schema = self.dispatcher.class.schema_digest(); let consistency = match &self.request.consistency { CellQueryConsistency::Stale => CellReadConsistency::Stale, + CellQueryConsistency::RecentSource { max_age_ms } => { + CellReadConsistency::RecentSource { + max_age_ms: *max_age_ms, + } + } CellQueryConsistency::Linearizable => CellReadConsistency::Linearizable, CellQueryConsistency::AtLeast(token) => { let observation = self @@ -554,12 +604,13 @@ impl CellQueryInvocation { return Err(CellDispatchError::Invalid("destination route binding")); } let selected = sql - .query_registered( + .query_registered_retained( options, authority, self.dispatcher.class.clone(), self.request.contract.clone(), self.request.input.clone(), + retention, ) .await .map_err(CellDispatchError::Execution)?; @@ -597,6 +648,12 @@ impl CellQueryResponse { pub fn token(&self) -> &CellVersionToken { &self.token } + + pub fn replica( + &self, + ) -> &cellar_logic::consensus::ReplicaIdentity { + self.selected.replica() + } } fn closed() -> CellDispatchError { @@ -621,6 +678,10 @@ fn fingerprint( hash.update(&request.input); match &request.consistency { CellQueryConsistency::Stale => hash.update([0]), + CellQueryConsistency::RecentSource { max_age_ms } => { + hash.update([3]); + hash.update(max_age_ms.to_be_bytes()); + } CellQueryConsistency::Linearizable => hash.update([1]), CellQueryConsistency::AtLeast(token) => { hash.update([2]); diff --git a/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs index e81d08a01..c8d7d2f02 100644 --- a/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs +++ b/crates/cellar/durability/raft/cell_groups/effect_index_tests.rs @@ -240,6 +240,8 @@ fn marker_one_references_require_nullable_due_summary() { digest: [1; 32], height: 1, cells: 1, + last_mutation_index: 1, + last_mutation_term: 1, next_due_unix_millis: None, }; let mut value = serde_json::to_value(reference).unwrap(); @@ -253,12 +255,68 @@ fn marker_one_references_require_nullable_due_summary() { .unwrap() .remove("next_due_unix_millis"); assert!(serde_json::from_value::(value).is_err()); + for field in ["last_mutation_index", "last_mutation_term"] { + let mut value = serde_json::to_value(reference).unwrap(); + value.as_object_mut().unwrap().remove(field); + assert!(serde_json::from_value::(value).is_err()); + } let mut invalid = f.leaves[3].clone(); invalid.effects.next_outbox_unix_millis = Some(i64::MAX as u64 + 1); invalid.effects.pending_outbox = 1; assert!(invalid.validate(f.tree.route()).is_err()); } +#[test] +fn complete_manifest_authenticates_latest_mutation_across_other_cell_updates() { + let mut f = fixture(false); + let before = f.root; + assert_eq!( + before.last_mutation(), + Some(LogPosition { + index: LogIndex::new(2), + term: Term::new(1) + }) + ); + let mut changed = f.leaves[7].clone(); + let old = changed.version.clone(); + changed.version.sequence = MutationSequence::new(2).unwrap(); + changed.version.image.txid = 2; + changed.version.image.commitment = CellImageCommitment::Materialized([11; 32]); + changed.last_mutation = LogPosition { + index: LogIndex::new(40), + term: Term::new(2), + }; + changed.witness = CommitWitness { + group: f.tree.route().history, + position: changed.last_mutation, + proposal: ProposalId::new([12; 16]).unwrap(), + }; + let after = f + .tree + .update(&mut f.store, before, changed.clone(), Some(&old)) + .unwrap(); + assert_eq!(after.cells(), 64); + assert_eq!(after.last_mutation(), Some(changed.last_mutation)); + assert_eq!(before.last_mutation().unwrap().index.get(), 2); + let applied = LogPosition { + index: LogIndex::new(60), + term: Term::new(3), + }; + f.tree.verify(&f.store, after, applied, |_| Ok(())).unwrap(); + let mut forged = after; + forged.0.as_mut().unwrap().last_mutation_index = 39; + assert!(f + .tree + .verify(&f.store, forged, applied, |_| Ok(())) + .is_err()); + let mut forged = after; + forged.0.as_mut().unwrap().last_mutation_term = 1; + assert!(f + .tree + .verify(&f.store, forged, applied, |_| Ok(())) + .is_err()); +} + #[test] fn verified_pinned_index_matches_page_lookup_without_opening_images() { let mut f = fixture(true); diff --git a/crates/cellar/durability/raft/cell_groups/host/mod.rs b/crates/cellar/durability/raft/cell_groups/host/mod.rs index e155cdd73..3ae096925 100644 --- a/crates/cellar/durability/raft/cell_groups/host/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/host/mod.rs @@ -20,7 +20,7 @@ use std::{ collections::BTreeMap, sync::{ atomic::{AtomicBool, Ordering}, - Arc, + Arc, Mutex, }, }; @@ -40,14 +40,14 @@ pub struct CellGroupHostLimits { struct Binding { dispatcher: CellGroupDispatcher, class: Arc, - storage: CellGroupNamespaceStorage, + storage: Mutex>, _admission: Permit, } struct Inner { controller: CellGroupController, domain: crate::asyncrt::DomainToken, - namespaces: BTreeMap<(ClusterId, NamespaceId), Arc>, + namespaces: Arc>>, budget: Arc, closed: AtomicBool, } @@ -60,7 +60,7 @@ pub struct CellGroupHost { inner: Arc, } -#[derive(Clone, Copy, Debug)] +#[derive(Clone, Copy, Debug, serde::Serialize)] pub struct CellGroupHostUsage { pub namespaces: usize, pub namespace_bytes: usize, @@ -117,11 +117,17 @@ impl CellGroupHost { Arc::new(Binding { dispatcher, class, - storage: registration.storage, + storage: Mutex::new(Some(registration.storage)), _admission: admission, }), ); } + let namespaces = Arc::new(namespaces); + controller + .own_host_resources(Arc::new(HostResources { + namespaces: namespaces.clone(), + })) + .map_err(CellDispatchError::Runtime)?; Ok(Self { inner: Arc::new(Inner { controller, @@ -150,6 +156,19 @@ impl CellGroupHost { }) } + /// Namespace capability lookup only; supplies no group placement or startup authority. + pub(crate) fn directory( + &self, + history: cellar_logic::cell_group::GroupHistory, + ) -> Result { + self.inner.check_open()?; + self.inner + .namespaces + .get(&(history.cluster, history.namespace)) + .map(|binding| binding.dispatcher.directory()) + .ok_or(CellDispatchError::Runtime(RuntimeError::NotFound)) + } + pub fn is_closed(&self) -> bool { self.inner.closed.load(Ordering::Acquire) || self.inner.domain.mono_ms().is_err() @@ -173,6 +192,43 @@ impl CellGroupHost { } } +/// Registered exactly once with the controller's physical join owner. This +/// retains no public controller handle and cannot prevent last-handle shutdown. +pub(in crate::raft::cell_groups) struct HostResources { + namespaces: Arc>>, +} + +impl HostResources { + pub(in crate::raft::cell_groups) fn release(&self) -> Result<(), CellGroupControllerError> { + let mut released = Ok(()); + for binding in self.namespaces.values() { + match binding.storage.lock() { + Ok(mut storage) => { + storage.take(); + } + Err(error) => { + error.into_inner().take(); + released = Err(CellGroupControllerError::Invalid( + "namespace resource owner poisoned", + )); + } + } + } + released + } +} + +impl Binding { + fn options(&self, history: &CellHistory) -> Result { + self.storage + .lock() + .map_err(|_| CellDispatchError::Runtime(RuntimeError::Closed))? + .as_ref() + .map(|storage| storage.for_cell(history)) + .ok_or(CellDispatchError::Runtime(RuntimeError::Closed)) + } +} + impl Inner { fn check_open(&self) -> Result<(), CellDispatchError> { if crate::application::consensus::callback_active() { @@ -201,6 +257,10 @@ pub struct CellGroupCellHandle { } impl CellGroupCellHandle { + pub(crate) fn definition_digest(&self) -> [u8; 32] { + self.binding.class.definition_digest() + } + pub fn history(&self) -> &CellHistory { &self.history } @@ -227,12 +287,20 @@ impl CellGroupCellHandle { ambiguous: false, cause: Box::new(cause), })?; + let options = self + .binding + .options(&self.history) + .map_err(|cause| CellCommandError { + command: request.id, + ambiguous: false, + cause: Box::new(cause), + })?; self.host.controller.dispatch_command( &self.binding.dispatcher, self.history.clone(), request, limits, - self.binding.storage.for_cell(&self.history), + options, ) } @@ -249,7 +317,7 @@ impl CellGroupCellHandle { .query(self.history.clone(), request, limits, None)?; self.host .controller - .query(invocation, self.binding.storage.for_cell(&self.history)) + .query(invocation, self.binding.options(&self.history)?) } pub fn enroll( @@ -266,15 +334,24 @@ impl CellGroupCellHandle { cause: Box::new(cause), }) }); + let creation = request.creation; let invocation = self.binding .dispatcher .enroll(self.history.clone(), request, limits, None); let host = self.host.clone(); - let options = self.binding.storage.for_cell(&self.history); + let binding = self.binding.clone(); + let history = self.history.clone(); async move { opened?; let invocation = invocation.await.map_err(Arc::new)?; + let options = binding.options(&history).map_err(|cause| { + Arc::new(CellEnrollmentError { + creation, + ambiguous: false, + cause: Box::new(cause), + }) + })?; let call = host .controller .enroll(invocation, options) @@ -297,8 +374,10 @@ impl CellGroupCellHandle { .lookup_receipt(self.history.clone(), request, limits, None) }); let host = self.host.clone(); - let options = self.binding.storage.for_cell(&self.history); + let binding = self.binding.clone(); + let history = self.history.clone(); async move { + let options = binding.options(&history).map_err(Arc::new)?; host.controller .lookup_receipt(invocation.map_err(Arc::new)?, options) .await diff --git a/crates/cellar/durability/raft/cell_groups/host/storage.rs b/crates/cellar/durability/raft/cell_groups/host/storage.rs index f011f6535..6ba4b9962 100644 --- a/crates/cellar/durability/raft/cell_groups/host/storage.rs +++ b/crates/cellar/durability/raft/cell_groups/host/storage.rs @@ -33,7 +33,7 @@ impl CellGroupNamespaceStorage { Ok(Self { options, bytes }) } - pub(super) fn validate_namespace( + pub(crate) fn validate_namespace( &self, config: &crate::raft::catalog::NamespaceConfig, ) -> Result<(), CellDispatchError> { @@ -55,7 +55,7 @@ impl CellGroupNamespaceStorage { self.bytes } - pub(super) fn for_cell(&self, history: &CellHistory) -> CellSqlOptions { + pub(crate) fn for_cell(&self, history: &CellHistory) -> CellSqlOptions { let key = hex::encode(super::super::cell_image_history_digest(history)); let mut options = self.options.clone(); match &mut options { diff --git a/crates/cellar/durability/raft/cell_groups/manifest.rs b/crates/cellar/durability/raft/cell_groups/manifest.rs index 426875204..7455303d1 100644 --- a/crates/cellar/durability/raft/cell_groups/manifest.rs +++ b/crates/cellar/durability/raft/cell_groups/manifest.rs @@ -64,7 +64,7 @@ impl Default for ManifestLimits { } impl ManifestLimits { - pub(super) fn validate(self) -> Result<(), ManifestError> { + pub(crate) fn validate(self) -> Result<(), ManifestError> { if self.max_cells == 0 || self.max_cells > 1_000_000 || self.max_depth == 0 @@ -218,6 +218,16 @@ impl ManifestRoot { pub fn digest(self) -> Option<[u8; 32]> { self.0.map(|p| p.digest) } + + /// Maximum physical installation position across the authenticated subtree. + /// Metadata-only apply leaves this unchanged. Complete-root verification + /// checks it against the actual pages and each cell's original mutation. + pub fn last_mutation(self) -> Option { + self.0.map(|p| LogPosition { + index: cellar_logic::consensus::LogIndex::new(p.last_mutation_index), + term: cellar_logic::consensus::Term::new(p.last_mutation_term), + }) + } } /// Synchronous operations belong on the owning blocking execution domain. diff --git a/crates/cellar/durability/raft/cell_groups/membership.rs b/crates/cellar/durability/raft/cell_groups/membership.rs index 43016f22c..2366c8e10 100644 --- a/crates/cellar/durability/raft/cell_groups/membership.rs +++ b/crates/cellar/durability/raft/cell_groups/membership.rs @@ -47,6 +47,17 @@ pub struct CellGroupMembershipChange { } impl CellGroupMembershipChange { + fn wrap(route: &GroupRoute, mut change: ConfChangeV2) -> Result { + let intent = Intent { + domain: "cellar-cell-group-membership-1".into(), + route: binding(route)?, + intent: String::from_utf8(change.context.to_vec()).map_err(|_| invalid())?, + }; + change.context = serde_json::to_vec(&intent).map_err(|_| invalid())?.into(); + validate(&change, route)?; + Ok(Self { change }) + } + pub fn new( route: &GroupRoute, membership: &GroupMembership, @@ -57,14 +68,81 @@ impl CellGroupMembershipChange { return Err(invalid()); } membership.bind_change(&mut change, joins)?; - let intent = Intent { - domain: "cellar-cell-group-membership-1".into(), - route: binding(route)?, - intent: String::from_utf8(change.context.to_vec()).map_err(|_| invalid())?, - }; - change.context = serde_json::to_vec(&intent).map_err(|_| invalid())?.into(); - validate(&change, route)?; - Ok(Self { change }) + Self::wrap(route, change) + } + + /// Enrollment requires the exact catalog capability. A serialized binding + /// is descriptive and cannot authorize an unmanaged group to join a fleet. + pub fn fleet_enrollment( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + authorization: &crate::raft::fleet_catalog::FleetGroupAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) + || authorization.history_digests() != (route.birth_digest, route.schema_digest) + { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_fleet_enrollment(&mut change, conf, authorization.binding())?; + Self::wrap(route, change) + } + + /// Consume the next catalog-selected permit without changing membership. + /// Completion must be observed after durable apply; admission is not a fence. + pub fn fleet_fence( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_fleet_fence(&mut change, conf, authorization.permit())?; + Self::wrap(route, change) + } + + /// Start one whole-group learner repair under its rooted physical debt. + /// The owning RawNode additionally checks readiness and actual catch-up. + pub fn managed_repair_start( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + plan: &crate::raft::member_state::RepairPlan, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) + || authorization.target_boot().is_none_or(|boot| { + boot.node != plan.replacement.node.bytes() + || boot.disk != plan.replacement.disk.bytes() + || boot.validate().is_err() + }) + { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_managed_repair_start(&mut change, conf, plan, authorization.permit())?; + Self::wrap(route, change) + } + + /// Continue the same consumed permit through joint consensus or abort. + pub fn managed_repair_step( + route: &GroupRoute, + membership: &GroupMembership, + conf: &ConfState, + id: cellar_logic::consensus::ProposalId, + step: crate::raft::member_state::RepairStep, + authorization: &crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + if !authorization.matches_scope(&route.history) { + return Err(invalid()); + } + let mut change = ConfChangeV2::default(); + membership.bind_managed_repair_step(&mut change, conf, id, step, authorization.permit())?; + Self::wrap(route, change) } pub(crate) fn bytes(&self) -> usize { @@ -227,7 +305,7 @@ pub(crate) fn transition( if change.context.is_empty() && (before.voters_outgoing.is_empty() || !before.auto_leave) { return Err(invalid()); } - membership.admit_group(&change, before)?; + membership.admit_group(&change, before, &route.history)?; let mut tracker = ProgressTracker::new(1); restore_configuration(&mut tracker, 1, before).map_err(|_| invalid())?; let mut changer = Changer::new(&tracker); diff --git a/crates/cellar/durability/raft/cell_groups/membership_observation.rs b/crates/cellar/durability/raft/cell_groups/membership_observation.rs new file mode 100644 index 000000000..404a22f05 --- /dev/null +++ b/crates/cellar/durability/raft/cell_groups/membership_observation.rs @@ -0,0 +1,102 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Invocation-owned safe-ReadIndex observations for whole physical groups. +//! These contain no application image and grant no mutation or release lease. +use super::GroupManifest; +use crate::raft::{budget::Permit, completion::RuntimeError, member_state::GroupMembership}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, LogPosition, ReplicaIdentity}, +}; +use raft::prelude::ConfState; + +// Charged to the existing node/group read banks before sending ReadIndex or +// cloning membership metadata. No admission ceiling is enlarged for this lane. +pub(super) const MEMBERSHIP_BYTES: usize = 64 * 1024; + +pub(super) struct MembershipSelection { + pub replica: ReplicaIdentity, + pub engine: EngineInstanceId, + pub root: GroupManifest, + pub barrier: LogPosition, + pub membership: GroupMembership, + pub configuration: ConfState, + pub status: super::CellGroupStatus, +} + +/// Exact durable configuration observed behind this invocation's fresh quorum +/// barrier. Retaining this snapshot supplies no current quorum, fleet eligibility, +/// complete inventory, archive coverage or permission to reclaim physical debt. +pub struct CellGroupQuorumMembership { + selected: MembershipSelection, + _node: Permit, + _group: Permit, +} + +impl CellGroupQuorumMembership { + pub(super) fn new( + selected: MembershipSelection, + node: Permit, + group: Permit, + ) -> Result { + selected.membership.validate( + &selected.configuration, + selected.root.position().index.get(), + )?; + selected + .membership + .validate_scope(&selected.replica.scope)?; + if selected.root.route().history != selected.replica.scope + || selected.root.membership_revision() != selected.membership.revision().get() + || selected.barrier.index.get() == 0 + || selected.barrier.term.get() == 0 + || selected.root.position().index < selected.barrier.index + || selected.root.position().term != selected.barrier.term + || selected.status.term != selected.barrier.term + || selected.status.applied != selected.root.position() + || selected.status.committed < selected.root.position().index + || selected.status.leader.is_none() + || selected.membership.revision() > selected.barrier.index + || !selected.membership.permits_group(&selected.replica) + { + return Err(RuntimeError::Invalid( + "quorum group membership observation".into(), + )); + } + Ok(Self { + selected, + _node: node, + _group: group, + }) + } + + pub fn replica(&self) -> &ReplicaIdentity { + &self.selected.replica + } + + pub fn engine(&self) -> EngineInstanceId { + self.selected.engine + } + + pub fn root(&self) -> &GroupManifest { + &self.selected.root + } + + pub fn barrier(&self) -> LogPosition { + self.selected.barrier + } + + pub fn membership(&self) -> &GroupMembership { + &self.selected.membership + } + + pub fn configuration(&self) -> &ConfState { + &self.selected.configuration + } + + /// Same-owner progress at selection. A follower observation establishes no + /// local leader, execution, transfer or application-serving authority. + pub fn status(&self) -> &super::CellGroupStatus { + &self.selected.status + } +} diff --git a/crates/cellar/durability/raft/cell_groups/mod.rs b/crates/cellar/durability/raft/cell_groups/mod.rs index 75addf63f..921459877 100644 --- a/crates/cellar/durability/raft/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/mod.rs @@ -16,6 +16,7 @@ pub use crate::raft::transport::{ mod apply; mod archive; +pub(crate) use archive::CellGroupArchiveAcquireOutcome; pub(crate) mod archive_compaction; pub use archive::{ CellGroupArchive, CellGroupArchiveCheckpointReceiver, CellGroupArchiveCollectionLimits, @@ -28,6 +29,7 @@ pub use archive::{ CellGroupArchiveUploadedTail, CellGroupArchiveUsage, PreparedCellGroupArchiveCheckpoint, PreparedCellGroupArchiveCreation, PreparedCellGroupArchivePublication, VerifiedCellGroupArchiveCheckpoint, VerifiedCellGroupArchiveRecovery, + VerifiedFleetGroupArchiveCoverage, }; pub(crate) mod checkpoint; pub use checkpoint::{ @@ -43,8 +45,9 @@ pub use controller::{ CellGroupArchiveCall, CellGroupArchiveControlError, CellGroupArchiveRegistration, CellGroupArchiveStatus, CellGroupCheckpointPolicy, CellGroupController, CellGroupControllerError, CellGroupControllerLimits, CellGroupControllerUsage, - CellGroupEffectRegistration, CellGroupProvisionCall, CellGroupProvisioning, - CellGroupRegistration, CellGroupRegistrationCall, CellGroupRegistrationError, CellQueryCall, + CellGroupEffectRegistration, CellGroupProvisionCall, CellGroupProvisionLimits, + CellGroupProvisionNamespace, CellGroupProvisioning, CellGroupRegistration, + CellGroupRegistrationCall, CellGroupRegistrationError, CellQueryCall, }; mod dispatch; mod host; @@ -79,7 +82,9 @@ pub(crate) mod membership; pub use membership::CellGroupMembershipChange; mod log_admission; pub(crate) mod log_retention; +mod membership_observation; mod owner; +pub use membership_observation::CellGroupQuorumMembership; mod page_store; mod page_wire; mod pins; @@ -115,14 +120,14 @@ pub use manifest::{ pub use page_store::{ManifestPageStore, ManifestReclaimReport, ManifestStoreUsage}; pub use crate::raft::catalog::{ - CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, - CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, + CellGroupArchiveBindingError, CellGroupBirth, CellGroupBirthDisposition, CellGroupBirthRequest, + CellGroupBirthSpec, CellGroupBootstrapPermit, CellGroupDirectoryCursor, CellGroupDirectoryPage, CellGroupDisasterFenceRequest, CellGroupRecoveryActivationAck, CellGroupRecoveryActivationPermit, CellGroupRecoveryError, CellGroupRecoveryRequest, CellRouteAttempt, CellRouteCacheUsage, CellRouteOperation, CellRouteRequest, CellRoutingError, CellRoutingFailure, CellRoutingLimits, CellRoutingSession, DirectoryLimits, DirectoryMutation, - DirectoryPublication, DirectoryView, FencedCellGroupRecovery, PreparedCellGroupArchiveBinding, - RangeDirectory, ResolvedCellRoute, + DirectoryPublication, DirectoryView, FencedCellGroupRecovery, FrozenCellGroupInventory, + PreparedCellGroupArchiveBinding, RangeDirectory, ResolvedCellRoute, }; pub use enrollment::{ decide_enrollment, EnrollmentDecision, EnrollmentError, EnrollmentPhase, EnrollmentRecord, diff --git a/crates/cellar/durability/raft/cell_groups/owner.rs b/crates/cellar/durability/raft/cell_groups/owner.rs index e1b6a9f8e..2ad5ae497 100644 --- a/crates/cellar/durability/raft/cell_groups/owner.rs +++ b/crates/cellar/durability/raft/cell_groups/owner.rs @@ -19,7 +19,7 @@ use crate::raft::{ use ::raft::{prelude::*, Config, LightReady, RawNode, Ready, StateRole, Storage}; use cellar_logic::{ cell_group::*, - consensus::{LogIndex, LogPosition, OperationId, ReplicaIdentity, Term}, + consensus::{LogIndex, LogPosition, MemberId, OperationId, ReplicaIdentity, Term}, }; use protobuf::Message as _; use std::{ @@ -180,6 +180,9 @@ enum Phase { pub struct CellGroupOwner { log_admission: super::log_admission::LogAdmission, leader_term: Arc, + source_epoch: Arc, + source_identity: (u64, u64), + source_observed: Option<(u64, u64, u64)>, lease: Arc, log: ReplicaLog, group: GroupLog, @@ -269,6 +272,9 @@ impl CellGroupOwner { Ok(Self { log_admission: Default::default(), leader_term: Arc::new(AtomicU64::new(0)), + source_epoch: Arc::new(AtomicU64::new(1)), + source_identity: (0, 0), + source_observed: None, lease, log: permit.log().clone(), group, @@ -309,11 +315,27 @@ impl CellGroupOwner { pub(super) fn revoke_execution(&mut self) { self.leader_term.store(0, Ordering::Release); + self.source_epoch.store(0, Ordering::Release); self.proposals.clear(); self.recovery_activation = None; } fn refresh_execution_fence(&mut self) { + let source = (self.node.raft.term, self.node.raft.leader_id); + if source != self.source_identity { + self.source_identity = source; + self.source_observed = None; + if self + .source_epoch + .fetch_update(Ordering::AcqRel, Ordering::Acquire, |epoch| { + (epoch != 0).then(|| epoch.checked_add(1)).flatten() + }) + .is_err() + { + // Exhaustion or revocation never rearms an old read capability. + self.source_epoch.store(0, Ordering::Release); + } + } let term = if self.node.raft.state == StateRole::Leader && self.node.raft.lead_transferee.is_none() && self @@ -404,18 +426,59 @@ impl CellGroupOwner { ) -> Result { self.require_application_active()?; let status = self.status()?; + let mut source = None; + let mut source_deadline = None; let leader = if matches!(consistency, super::CellReadConsistency::Linearizable) { - if self.node.raft.state != StateRole::Leader || Some(status.term) != term { + if term.is_none() { + return Err(RuntimeError::NotReady); + } + if Some(status.term) != term { return Err(RuntimeError::NotLeader); } let index = barrier.ok_or(RuntimeError::NotReady)?; - if !status.ready || status.applied.index < index { + if status.applied.index < index { return Err(RuntimeError::NotReady); } - Some((status.term, self.leader_term.clone())) + if self.node.raft.state == StateRole::Leader { + if !status.ready { + return Err(RuntimeError::NotReady); + } + Some((status.term, self.leader_term.clone())) + } else if self.node.raft.state == StateRole::Follower && self.membership_read_ready()? { + source = Some(self.read_source_epoch(status.term)?); + None + } else { + return Err(RuntimeError::NotLeader); + } } else { None }; + if let super::CellReadConsistency::RecentSource { max_age_ms } = consistency { + if !(1..=3_600_000).contains(max_age_ms) { + return Err(RuntimeError::Invalid("stale source age".into())); + } + let now = self + .log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?; + let observed = if self.node.raft.state == StateRole::Leader + && self.current.marker().position().term == status.term + { + Some(now) + } else { + self.source_observed + .filter(|(member, term, _)| { + *member == self.node.raft.leader_id && *term == status.term.get() + }) + .map(|(_, _, at)| at) + }; + source_deadline = Some( + crate::raft::query::recent_source_deadline(observed, now, *max_age_ms) + .ok_or(RuntimeError::NotReady)?, + ); + source = Some(self.read_source_epoch(status.term)?); + } let (leaf, image) = self .current .pin_cell(&cell.cell) @@ -442,9 +505,19 @@ impl CellGroupOwner { .zip(barrier) .map(|(term, index)| LogPosition { term, index }), leader, + source, + source_deadline, }) } + fn read_source_epoch(&self, term: Term) -> Result<(Term, u64, Arc), RuntimeError> { + let epoch = self.source_epoch.load(Ordering::Acquire); + if epoch == 0 || self.group.read()?.hard_state.term != term.get() { + return Err(RuntimeError::NotReady); + } + Ok((term, epoch, self.source_epoch.clone())) + } + pub(super) fn check_seed_target( &self, target: &ReplicaIdentity, @@ -453,6 +526,60 @@ impl CellGroupOwner { super::seed_authority::check_target(&self.group, target).map(|_| ()) } + pub(super) fn select_membership( + &self, + minimum: LogIndex, + term: Option, + barrier: Option, + ) -> Result { + self.require_application_active()?; + let status = self.status()?; + if Some(status.term) != term { + return Err(RuntimeError::NotLeader); + } + let index = barrier.ok_or(RuntimeError::NotReady)?; + if !self.membership_read_ready()? + || status.applied.index < index + || status.applied.index < minimum + { + return Err(RuntimeError::NotReady); + } + let cache = self.group.read()?; + let root = self.current.marker(); + if cache.group_manifest.as_ref() != Some(root) + || root.membership_revision() != cache.membership.revision().get() + || cache.membership.revision() > index + { + return Err(RuntimeError::NotReady); + } + cache + .membership + .validate(&cache.conf_state, root.position().index.get())?; + cache.membership.validate_scope(&root.route().history)?; + let bytes = cache + .membership + .reserved_bytes() + .checked_add(cache.conf_state.compute_size() as usize) + .and_then(|n| n.checked_add(serde_json::to_vec(root).ok()?.len())) + .and_then(|n| n.checked_add(4096)) + .ok_or(RuntimeError::Overloaded)?; + if bytes > super::membership_observation::MEMBERSHIP_BYTES { + return Err(RuntimeError::Overloaded); + } + Ok(super::membership_observation::MembershipSelection { + replica: self.group.identity().clone(), + engine: self.log.storage_instance()?, + root: root.clone(), + barrier: LogPosition { + term: status.term, + index, + }, + membership: cache.membership.clone(), + configuration: cache.conf_state.clone(), + status, + }) + } + pub(super) fn select_seed( &self, target: &ReplicaIdentity, @@ -526,6 +653,42 @@ impl CellGroupOwner { }) } + /// Observe the real leader's progress tracker. This is advisory; transfer + /// admission rechecks the same owner against the caller's term and revision. + pub fn replication_status( + &self, + ) -> Result, RuntimeError> { + use crate::raft::completion::{MemberProgress, ReplicationStatus}; + self.live()?; + if self.node.raft.state != StateRole::Leader { + return Err(RuntimeError::NotLeader); + } + let cache = self.group.read()?; + let mut members = self + .node + .raft + .prs() + .iter() + .map(|(&id, progress)| { + Ok(MemberProgress { + member: MemberId::new(id).map_err(|e| RuntimeError::Protocol(e.to_string()))?, + matched: LogIndex::new(progress.matched), + voter: cache.conf_state.voters.contains(&id) + || cache.conf_state.voters_outgoing.contains(&id), + recent_active: progress.recent_active, + }) + }) + .collect::, RuntimeError>>()?; + members.sort_by_key(|member| member.member); + Ok(ReplicationStatus { + replica: self.group.identity().clone(), + term: Term::new(self.node.raft.term), + membership_revision: cache.membership.revision(), + committed: LogIndex::new(self.node.raft.raft_log.committed), + members, + }) + } + fn protocol_ready(&self, cache: &crate::raft::cache::CachedGroup) -> bool { self.node.raft.state == StateRole::Leader && self.node.raft.lead_transferee.is_none() @@ -648,28 +811,73 @@ impl CellGroupOwner { /// Admit a handoff to a caught-up incoming voter. Completion is observed /// through leadership status; admission alone cannot prove an election. pub fn transfer_leader(&mut self, member: u64) -> Result<(), RuntimeError> { + let progress = self.replication_status()?; + self.transfer_leadership(crate::raft::completion::LeadershipTransfer { + target: MemberId::new(member).map_err(|e| RuntimeError::Invalid(e.to_string()))?, + expected_term: progress.term, + expected_revision: progress.membership_revision, + }) + } + + /// Recheck whole-group matched progress immediately before restart + /// handoff/observation. The runtime separately checks the synced stop gate. + pub(crate) fn restart_leadership( + &mut self, + check: crate::raft::completion::RestartLeadershipCheck, + ) -> Result<(), RuntimeError> { + let progress = self.replication_status()?; + let cache = self.group.read()?; + let transfer = crate::raft::restart_leadership::check_progress( + &progress, + &cache.conf_state, + &check, + self.node.raft.pending_conf_index > self.node.raft.raft_log.applied, + self.node.raft.lead_transferee.is_some(), + )?; + drop(cache); + if let Some(transfer) = transfer { + self.transfer_leadership(transfer)?; + } + Ok(()) + } + + pub fn transfer_leadership( + &mut self, + transfer: crate::raft::completion::LeadershipTransfer, + ) -> Result<(), RuntimeError> { self.live()?; if self.node.raft.state != StateRole::Leader { return Err(RuntimeError::NotLeader); } let status = self.status()?; let cache = self.group.read()?; - if member == self.node.raft.id || !cache.conf_state.voters.contains(&member) { + let member = transfer.target.get(); + if transfer.expected_term != status.term + || transfer.expected_revision != cache.membership.revision() + || member == self.node.raft.id + || !cache.conf_state.voters.contains(&member) + { return Err(RuntimeError::Invalid( "leadership target is not another incoming voter".into(), )); } if !status.ready + || !cache.conf_state.voters_outgoing.is_empty() + || self.node.raft.lead_transferee.is_some() + || cache.membership.repair_status()?.is_some_and(|repair| { + matches!( + repair.phase, + crate::raft::member_state::RepairPhase::Seeding + | crate::raft::member_state::RepairPhase::Joint + ) + }) || status.applied.index != status.committed || !self.apply.is_empty() || self.applying.is_some() || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied - || self - .node - .raft - .prs() - .get(member) - .is_none_or(|p| p.matched < self.node.raft.raft_log.last_index()) + || self.node.raft.prs().get(member).is_none_or(|p| { + !p.recent_active || p.matched < self.node.raft.raft_log.last_index() + }) { return Err(RuntimeError::NotReady); } @@ -894,10 +1102,24 @@ impl CellGroupOwner { .log_admission .election(&self.group, &self.node, admission)?; } + let (from, term, kind) = (message.from, message.term, message.get_msg_type()); let result = self.node.step(message); self.refresh_execution_fence(); self.finish_log_admission(admission)?; result?; + if matches!(kind, MessageType::MsgAppend | MessageType::MsgHeartbeat) + && from == self.node.raft.leader_id + && term == self.node.raft.term + { + self.source_observed = Some(( + from, + term, + self.log + .domain() + .mono_ms() + .map_err(|_| RuntimeError::Closed)?, + )); + } Ok(()) } @@ -1164,6 +1386,63 @@ impl CellGroupOwner { Ok(()) } + /// Business reads have their own authenticated forwarding context; seed + /// contexts remain leader-local and cannot enter this path. + pub(super) fn read_forwarded_cell_index( + &mut self, + context: Vec, + ) -> Result<(), RuntimeError> { + if super::runtime::reads::cell_origin(&context) != Some(self.membership_read_origin()) { + return Err(RuntimeError::Invalid("cell read origin".into())); + } + if !self.membership_read_ready()? + || (self.node.raft.state == StateRole::Leader && !self.status()?.ready) + { + return Err(RuntimeError::NotReady); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + + pub(super) fn membership_read_origin(&self) -> u64 { + self.group.identity().member.get() + } + + /// Safe forwarding requires an active coherent member and known leader. + /// This does not change leader-only mutation or seed readiness. + pub(super) fn membership_read_ready(&self) -> Result { + self.require_application_active()?; + let cache = self.group.read()?; + Ok(matches!( + self.node.raft.state, + StateRole::Leader | StateRole::Follower + ) && self.node.raft.leader_id != 0 + && self.node.raft.term > 0 + && self.phase.is_none() + && self.checkpoint.is_none() + && cache.hard_state.term == self.node.raft.term + && cache.group_manifest.as_ref() == Some(self.current.marker()) + && cache.membership.permits_group(self.group.identity())) + } + + pub(super) fn read_membership_index(&mut self, context: Vec) -> Result<(), RuntimeError> { + if super::runtime::reads::membership_origin(&context) != Some(self.membership_read_origin()) + { + return Err(RuntimeError::Invalid("membership read origin".into())); + } + if !self.membership_read_ready()? { + return Err(RuntimeError::NotReady); + } + if self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads { + return Err(RuntimeError::Overloaded); + } + self.node.read_index(context); + Ok(()) + } + pub fn complete_persist(&mut self, completed: CellGroupPersisted) -> Result<(), RuntimeError> { self.live()?; if self @@ -1892,7 +2171,6 @@ impl CellGroupOwner { let entry = &message.entries[0]; if entry.data.is_empty() || entry.data.len() > 4096 - || super::runtime::reads::reserved_context(&entry.data) || entry.index != 0 || entry.term != 0 || !entry.context.is_empty() @@ -1900,6 +2178,21 @@ impl CellGroupOwner { { return Err(invalid()); } + if super::runtime::reads::reserved_context(&entry.data) { + let origin = super::runtime::reads::membership_origin(&entry.data) + .or_else(|| super::runtime::reads::cell_origin(&entry.data)) + .ok_or_else(invalid)?; + // Seed/private execution contexts remain local. Forwarded + // cell/metadata requests bind their authenticated origin; + // replies come only from this follower's current leader. + if (kind == MessageType::MsgReadIndex && origin != message.from) + || (kind == MessageType::MsgReadIndexResp + && (origin != self.group.identity().member.get() + || message.from != self.node.raft.leader_id)) + { + return Err(invalid()); + } + } if kind == MessageType::MsgReadIndex && self.node.raft.read_only.pending_read_count() >= self.options.max_pending_reads @@ -1917,5 +2210,6 @@ impl CellGroupOwner { impl Drop for CellGroupOwner { fn drop(&mut self) { self.leader_term.store(0, Ordering::Release); + self.source_epoch.store(0, Ordering::Release); } } diff --git a/crates/cellar/durability/raft/cell_groups/page_store.rs b/crates/cellar/durability/raft/cell_groups/page_store.rs index 3fd4f3c8d..d5d0d43bd 100644 --- a/crates/cellar/durability/raft/cell_groups/page_store.rs +++ b/crates/cellar/durability/raft/cell_groups/page_store.rs @@ -60,6 +60,36 @@ pub struct ManifestReclaimReport { } impl ManifestPageStore { + /// Startup and controller provisioning must lock the same page inventory. + /// Archive replay uses its independent node base with this same layout. + pub(crate) fn group_pages_path( + base: &Path, + history: cellar_logic::cell_group::GroupHistory, + ) -> PathBuf { + base.join("groups") + .join(hex::encode(history.cluster.bytes())) + .join(hex::encode(history.namespace.bytes())) + .join(hex::encode(history.group.bytes())) + .join(hex::encode(history.incarnation.bytes())) + .join("pages") + } + + pub(crate) fn validate_limits( + max_pages: usize, + max_bytes: u64, + max_retained_roots: usize, + ) -> Result<(), ManifestError> { + if max_pages == 0 + || max_pages > 4_000_000 + || max_bytes == 0 + || max_bytes > 64 * 1024 * 1024 * 1024 + || !(2..=4096).contains(&max_retained_roots) + { + return Err(ManifestError::Limit); + } + Ok(()) + } + pub(super) fn require_live(&self) -> Result<(), ManifestError> { if self.quarantined { return Err(ManifestError::Quarantined); @@ -74,14 +104,7 @@ impl ManifestPageStore { max_bytes: u64, max_retained_roots: usize, ) -> Result { - if max_pages == 0 - || max_pages > 4_000_000 - || max_bytes == 0 - || max_bytes > 64 * 1024 * 1024 * 1024 - || !(2..=4096).contains(&max_retained_roots) - { - return Err(ManifestError::Limit); - } + Self::validate_limits(max_pages, max_bytes, max_retained_roots)?; cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), directory)?; let lock = filesystem.lock_file(&directory.join("LOCK"))?; let mut store = Self { diff --git a/crates/cellar/durability/raft/cell_groups/page_wire.rs b/crates/cellar/durability/raft/cell_groups/page_wire.rs index 156983f7a..e68844633 100644 --- a/crates/cellar/durability/raft/cell_groups/page_wire.rs +++ b/crates/cellar/durability/raft/cell_groups/page_wire.rs @@ -21,6 +21,8 @@ pub(super) struct PageRef { pub digest: [u8; 32], pub height: u8, pub cells: u32, + pub last_mutation_index: u64, + pub last_mutation_term: u64, // Required even when null: internal marker-1 hard cutover, no old reader. #[serde(deserialize_with = "required_due")] pub next_due_unix_millis: Option, @@ -51,6 +53,10 @@ impl Page { || child.height > limits.max_depth || child.cells == 0 || child.cells > limits.max_cells + || child.last_mutation_index == 0 + || child.last_mutation_term == 0 + || child.last_mutation_index >= u64::MAX - 2 + || child.last_mutation_term >= u64::MAX - 2 || child .next_due_unix_millis .is_some_and(|due| due > i64::MAX as u64) @@ -72,10 +78,28 @@ impl Page { if height > limits.max_depth || cells > limits.max_cells { return Err(ManifestError::Limit); } + let mut latest = self.leaf.last_mutation; + for child in [self.left, self.right].into_iter().flatten() { + let next = LogPosition { + index: LogIndex::new(child.last_mutation_index), + term: Term::new(child.last_mutation_term), + }; + if (next.index < latest.index && next.term > latest.term) + || (next.index > latest.index && next.term < latest.term) + || (next.index == latest.index && next.term != latest.term) + { + return Err(ManifestError::Invalid("subtree mutation order")); + } + if next.index > latest.index { + latest = next; + } + } Ok(PageRef { digest: digest(bytes), height, cells, + last_mutation_index: latest.index.get(), + last_mutation_term: latest.term.get(), next_due_unix_millis: [ self.leaf.effects.next_due_unix_millis(), self.left.and_then(|p| p.next_due_unix_millis), diff --git a/crates/cellar/durability/raft/cell_groups/read_authority.rs b/crates/cellar/durability/raft/cell_groups/read_authority.rs index 22103896d..0940256b9 100644 --- a/crates/cellar/durability/raft/cell_groups/read_authority.rs +++ b/crates/cellar/durability/raft/cell_groups/read_authority.rs @@ -17,6 +17,10 @@ use std::sync::{ pub enum CellReadConsistency { /// Explicitly allows a locally committed older image, including on followers. Stale, + /// A bounded-age observation of the current source; never a quorum proof. + RecentSource { + max_age_ms: u64, + }, /// The dispatcher must first authenticate the version and its schema/lineage. AtLeast(CellVersion), Linearizable, @@ -30,6 +34,8 @@ pub(super) struct ReadSelection { pub applied: LogPosition, pub barrier: Option, pub leader: Option<(Term, Arc)>, + pub source: Option<(Term, u64, Arc)>, + pub source_deadline: Option, } /// Consumed by one SQL query. The image and its admission cannot be cloned out @@ -56,6 +62,9 @@ impl CellReadAuthority { if !node.grow_to(bytes) || !group.grow_to(bytes) { return Err(RuntimeError::Overloaded); } + let deadline = selection + .source_deadline + .map_or(deadline, |source| deadline.min(source)); Ok(Self { selection, deadline, @@ -68,6 +77,10 @@ impl CellReadAuthority { &self.selection.cell } + pub fn replica(&self) -> &cellar_logic::consensus::ReplicaIdentity { + self.selection.group.identity() + } + pub fn route(&self) -> &GroupRoute { &self.selection.route } @@ -103,6 +116,11 @@ impl CellReadAuthority { .as_ref() .is_none_or(|m| m.route() != &s.route || !m.is_active()) || !cache.membership.permits_group(s.group.identity()) + || s.source.as_ref().is_some_and(|(term, epoch, source)| { + cache.hard_state.term != term.get() + || *epoch == 0 + || source.load(Ordering::Acquire) != *epoch + }) || s.leader.as_ref().is_some_and(|(term, leader)| { cache.hard_state.term != term.get() || leader.load(Ordering::Acquire) != term.get() diff --git a/crates/cellar/durability/raft/cell_groups/recovery/tests.rs b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs index 03d6bd04d..d32965253 100644 --- a/crates/cellar/durability/raft/cell_groups/recovery/tests.rs +++ b/crates/cellar/durability/raft/cell_groups/recovery/tests.rs @@ -364,6 +364,8 @@ async fn recovery_pending_root_denies_stale_and_execution_authorities() -> anyho applied: marker.position(), barrier: None, leader: None, + source: None, + source_deadline: None, }, 100, node.acquire(4096).unwrap(), diff --git a/crates/cellar/durability/raft/cell_groups/retention.rs b/crates/cellar/durability/raft/cell_groups/retention.rs index 2a986939b..9e38c56d4 100644 --- a/crates/cellar/durability/raft/cell_groups/retention.rs +++ b/crates/cellar/durability/raft/cell_groups/retention.rs @@ -251,6 +251,35 @@ impl RootRetentions { }) } + /// Validate an already verified physical closure without rebuilding it. + /// Marker equality alone cannot clear pending or uncertain publication. + pub(super) fn require_selected( + &self, + group: &GroupLog, + marker: &GroupManifest, + ) -> Result<(), ManifestError> { + let checkpoint = group + .checkpoint_manifest() + .map_err(|_| ManifestError::Invalid("closed engine"))?; + let state = self.state()?; + let selected = state + .groups + .get(&key(marker)) + .ok_or(ManifestError::Invalid("selected root not recovered"))?; + if !Arc::ptr_eq(&selected.engine, &group.owner) + || group.closed.load(Ordering::Acquire) + || selected.closed.load(Ordering::Acquire) + || selected.pending.is_some() + || selected.pending_checkpoint + || selected.current.as_ref().map(|root| &root.marker) != Some(marker) + || selected.expected_checkpoint != checkpoint + || selected.checkpoint.as_ref().map(|root| &root.marker) != checkpoint.as_ref() + { + return Err(ManifestError::Conflict); + } + Ok(()) + } + pub(super) fn require_checkpoint(&self, group: &GroupLog) -> Result<(), ManifestError> { let expected = group .checkpoint_manifest() diff --git a/crates/cellar/durability/raft/cell_groups/root.rs b/crates/cellar/durability/raft/cell_groups/root.rs index 790f975c9..6decb40d0 100644 --- a/crates/cellar/durability/raft/cell_groups/root.rs +++ b/crates/cellar/durability/raft/cell_groups/root.rs @@ -107,6 +107,12 @@ impl GroupManifest { || root.cells > 1_000_000 || root.height == 0 || root.height > 32 + || root.last_mutation_index == 0 + || root.last_mutation_term == 0 + || root.last_mutation_index > self.position.index.get() + || root.last_mutation_term > self.position.term.get() + || root.last_mutation_index == self.position.index.get() + && root.last_mutation_term != self.position.term.get() || root .next_due_unix_millis .is_some_and(|due| due > i64::MAX as u64) @@ -233,6 +239,13 @@ impl PreparedGroupManifest { self.images.next_due(now_unix_millis, after.as_ref()) } + pub(super) fn require_selected( + &self, + group: &crate::raft::cache::GroupLog, + ) -> Result<(), ManifestError> { + self.retention.require_selected(group, &self.marker) + } + pub(crate) fn require_checkpoint( &self, group: &crate::raft::cache::GroupLog, diff --git a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs index 60d1c9cb3..d5d6cc437 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/mod.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/mod.rs @@ -30,7 +30,7 @@ const SNAPSHOT_HINT_SLOTS: usize = 256; /// Advisory admission counters. Different lanes may be observed at different /// moments; these values neither reserve capacity nor establish durability or /// serving authority. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)] pub struct CellGroupAdmissionUsage { pub items: usize, pub bytes: usize, @@ -39,7 +39,7 @@ pub struct CellGroupAdmissionUsage { } impl CellGroupAdmissionUsage { - fn read(budget: &Budget) -> Self { + pub(crate) fn read(budget: &Budget) -> Self { let (bytes, items) = budget.usage(); let (max_bytes, max_items) = budget.capacity(); Self { @@ -52,7 +52,7 @@ impl CellGroupAdmissionUsage { } /// Node-wide request/pin admission, retained through each lane's owned work. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[derive(Clone, Copy, Debug, PartialEq, Eq, serde::Serialize)] pub struct CellGroupRuntimeUsage { pub ordinary: CellGroupAdmissionUsage, /// Eight slots and max_ready_bytes of request metadata, independently of @@ -132,6 +132,8 @@ enum Command { Quarantine, Campaign, TransferLeader(u64), + TransferLeadership(crate::raft::completion::LeadershipTransfer), + RestartLeadership(Box), Configure(CellGroupMembershipChange), ActivateRecovery(Box), Execution(cellar_logic::cell_group::CellHistory), @@ -160,6 +162,7 @@ enum Command { Apply(CellGroupTicket, Box), CaptureCheckpoint(CellGroupCheckpointPool), Status, + ReplicationStatus, Tick, } @@ -175,8 +178,10 @@ enum Response { EnrollmentState(Box), Reconciliation(Box), Status(CellGroupStatus), + ReplicationStatus(crate::raft::completion::ReplicationStatus), Read(Box), Seed(Box), + Membership(Box), CheckpointSource(Box>), } @@ -217,6 +222,10 @@ impl CellGroupRuntime { &self.inner.log } + pub(crate) fn limits(&self) -> &RaftRuntimeOptions { + &self.inner.options + } + pub(crate) fn claim_transport(&self) -> Result { self.admit()?; self.inner @@ -536,10 +545,11 @@ impl CellGroupRuntime { { return Err(RuntimeError::DeadlineExceeded); } + let bytes = kind.reservation_bytes(); let reservation = self .inner .read_pins - .acquire(4096) + .acquire(bytes) .ok_or(RuntimeError::Overloaded)?; let request = reads::ReadRequest { kind, @@ -552,13 +562,54 @@ impl CellGroupRuntime { .sleep_until(deadline) .map_err(|_| RuntimeError::Closed)?; crate::asyncrt::select! { - result = self.call(Some(history), Command::Read(Box::new(request)), 4096) => { + result = self.call(Some(history), Command::Read(Box::new(request)), bytes) => { result } _ = timer => Err(RuntimeError::DeadlineExceeded), } } + /// Observe one actual physical configuration after a fresh safe ReadIndex + /// and contiguous durable apply, without selecting an application cell. + /// A follower forwards the metadata request to its current leader and waits + /// for its own durable apply. No leader/execution authority is returned. + pub async fn quorum_membership( + &self, + history: GroupHistory, + deadline: u64, + ) -> Result { + self.quorum_membership_after(history, cellar_logic::consensus::LogIndex::new(0), deadline) + .await + } + + /// Wait for this durable apply floor before issuing the fresh safe ReadIndex. + /// Admission of a configuration proposal alone cannot satisfy this read. + pub async fn quorum_membership_after( + &self, + history: GroupHistory, + minimum: cellar_logic::consensus::LogIndex, + deadline: u64, + ) -> Result { + let response = self + .select_authority(history, reads::ReadKind::Membership { minimum }, deadline) + .await?; + if self + .inner + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed)? + >= deadline + { + return Err(RuntimeError::DeadlineExceeded); + } + match response { + Response::Membership(observation) => Ok(*observation), + _ => Err(RuntimeError::Protocol( + "group quorum membership reply".into(), + )), + } + } + pub async fn execution_authority( &self, history: GroupHistory, @@ -894,6 +945,45 @@ impl CellGroupRuntime { } } + pub async fn replication_status( + &self, + history: GroupHistory, + ) -> Result, RuntimeError> { + match self + .call(Some(history), Command::ReplicationStatus, 4096) + .await? + { + Response::ReplicationStatus(status) => Ok(status), + _ => Err(RuntimeError::Protocol( + "group replication status reply".into(), + )), + } + } + + pub async fn transfer_leadership( + &self, + history: GroupHistory, + transfer: crate::raft::completion::LeadershipTransfer, + ) -> Result<(), RuntimeError> { + self.call(Some(history), Command::TransferLeadership(transfer), 4096) + .await + .map(|_| ()) + } + + pub(crate) async fn restart_leadership( + &self, + history: GroupHistory, + check: crate::raft::completion::RestartLeadershipCheck, + ) -> Result<(), RuntimeError> { + self.call( + Some(history), + Command::RestartLeadership(Box::new(check)), + 4096, + ) + .await + .map(|_| ()) + } + /// Capture the owner's exact durable applied cut on the existing worker. /// This performs bounded metadata work only; streaming and archive I/O run /// outside the worker. Concurrent persistence may transiently invalidate a diff --git a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs index a715e6950..5ddb0af18 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/reads.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/reads.rs @@ -12,9 +12,34 @@ use cellar_logic::{ use std::{collections::BTreeMap, sync::Weak}; const PREFIX: &[u8] = b"cellar-cell-read-1\0"; +const CELL_FORWARD_PREFIX: &[u8] = b"cellar-cell-forward-read-1\0"; +const MEMBERSHIP_PREFIX: &[u8] = b"cellar-group-membership-read-1\0"; pub(in crate::raft::cell_groups) fn reserved_context(context: &[u8]) -> bool { context.starts_with(PREFIX) + || context.starts_with(MEMBERSHIP_PREFIX) + || context.starts_with(CELL_FORWARD_PREFIX) +} + +/// Forwarded metadata reads have a separate namespace from local execution +/// authority. Bind the exact origin member as well as this invocation's nonce. +pub(in crate::raft::cell_groups) fn membership_origin(context: &[u8]) -> Option { + forwarded_origin(context, MEMBERSHIP_PREFIX) +} + +pub(in crate::raft::cell_groups) fn cell_origin(context: &[u8]) -> Option { + forwarded_origin(context, CELL_FORWARD_PREFIX) +} + +fn forwarded_origin(context: &[u8], prefix: &[u8]) -> Option { + let rest = context.strip_prefix(prefix)?; + if rest.len() != 32 { + return None; + } + let member = u64::from_be_bytes(rest[..8].try_into().ok()?); + let sequence = u64::from_be_bytes(rest[24..].try_into().ok()?); + cellar_logic::consensus::MemberId::new(member).ok()?; + (sequence > 0).then_some(member) } pub(super) struct ReadRequest { @@ -26,13 +51,23 @@ pub(super) struct ReadRequest { pub(super) enum ReadKind { Cell(CellHistory, CellReadConsistency), Seed(cellar_logic::consensus::ReplicaIdentity), + Membership { minimum: LogIndex }, } impl ReadKind { + pub fn reservation_bytes(&self) -> usize { + match self { + Self::Membership { .. } => super::super::membership_observation::MEMBERSHIP_BYTES, + _ => 4096, + } + } + pub fn linearizable(&self) -> bool { matches!( self, - Self::Seed(_) | Self::Cell(_, CellReadConsistency::Linearizable) + Self::Seed(_) + | Self::Membership { .. } + | Self::Cell(_, CellReadConsistency::Linearizable) ) } } @@ -40,6 +75,7 @@ impl ReadKind { pub(super) enum Selection { Cell(ReadSelection), Seed(SeedSelection), + Membership(super::super::membership_observation::MembershipSelection), } struct Pending { @@ -102,6 +138,7 @@ impl Reads { } } ReadKind::Seed(target) => owner.check_seed_target(target)?, + ReadKind::Membership { .. } => (), } // A pending recovery is an admission refusal, not a read barrier // that may occupy the ordinary wait queue until its deadline. @@ -112,22 +149,61 @@ impl Reads { .get(&history) .and_then(Weak::upgrade) .unwrap_or_else(|| Budget::new(self.max_bytes, self.max_items)); - let group = budget.acquire(4096).ok_or(RuntimeError::Overloaded)?; + let group = budget + .acquire(request.kind.reservation_bytes()) + .ok_or(RuntimeError::Overloaded)?; self.groups.insert(history, Arc::downgrade(&budget)); self.sequence = self .sequence .checked_add(1) .ok_or(RuntimeError::Overloaded)?; - let mut context = PREFIX.to_vec(); + let membership = matches!(request.kind, ReadKind::Membership { .. }); + let cell_read = matches!( + request.kind, + ReadKind::Cell(_, CellReadConsistency::Linearizable) + ); + let mut context = if membership || cell_read { + let mut context = if membership { + MEMBERSHIP_PREFIX + } else { + CELL_FORWARD_PREFIX + } + .to_vec(); + context.extend_from_slice(&owner.membership_read_origin().to_be_bytes()); + context + } else { + PREFIX.to_vec() + }; context.extend_from_slice(&self.nonce); context.extend_from_slice(&self.sequence.to_be_bytes()); let term = if request.kind.linearizable() { let status = owner.status()?; - if !status.ready { - return Err(RuntimeError::NotReady); + if matches!(request.kind, ReadKind::Membership { minimum } if status.applied.index < minimum) + { + // The configuration proposal is only admitted. Start this + // invocation's quorum barrier after its durable apply floor. + None + } else { + if membership { + match owner.read_membership_index(context.clone()) { + Ok(()) => Some(status.term), + Err(RuntimeError::NotReady) => None, + Err(error) => return Err(error), + } + } else if cell_read { + match owner.read_forwarded_cell_index(context.clone()) { + Ok(()) => Some(status.term), + Err(RuntimeError::NotReady) => None, + Err(error) => return Err(error), + } + } else { + if !status.ready { + return Err(RuntimeError::NotReady); + } + owner.read_cell_index(context.clone())?; + Some(status.term) + } } - owner.read_cell_index(context.clone())?; - Some(status.term) } else { None }; @@ -172,8 +248,9 @@ impl Reads { mut select: impl FnMut( GroupHistory, &ReadRequest, - Option, - Option, + &mut Option, + &mut Option, + &[u8], ) -> Result, ) -> bool { let mut made = false; @@ -187,17 +264,22 @@ impl Reads { } }); let mut finished = Vec::new(); - for (context, pending) in &self.pending { + for (context, pending) in &mut self.pending { + let before = (pending.term, pending.barrier); let result = if now >= pending.request.deadline { Err(RuntimeError::DeadlineExceeded) } else { select( pending.history, &pending.request, - pending.term, - pending.barrier, + &mut pending.term, + &mut pending.barrier, + context, ) }; + // A newly issued barrier must drive the owner's Ready work even + // when selection still waits. Waiting without a state change sleeps. + made |= before != (pending.term, pending.barrier); if !matches!(result, Err(RuntimeError::NotReady)) { finished.push((context.clone(), result)); } @@ -223,6 +305,12 @@ impl Reads { pending.group, ) .map(|authority| Response::Seed(Box::new(authority))), + Selection::Membership(selection) => CellGroupQuorumMembership::new( + selection, + pending.request.reservation, + pending.group, + ) + .map(|observation| Response::Membership(Box::new(observation))), }); let _ = pending.reply.send(result); made = true; diff --git a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs index b9fa29293..ce58af5f1 100644 --- a/crates/cellar/durability/raft/cell_groups/runtime/worker.rs +++ b/crates/cellar/durability/raft/cell_groups/runtime/worker.rs @@ -232,6 +232,16 @@ impl Worker { reply(request, group.owner.status().map(Response::Status)); return; } + if matches!(request.command, Command::ReplicationStatus) { + reply( + request, + group + .owner + .replication_status() + .map(Response::ReplicationStatus), + ); + return; + } if let Command::CaptureCheckpoint(pool) = &request.command { let source = group.owner.capture_checkpoint(pool); reply(request, Ok(Response::CheckpointSource(Box::new(source)))); @@ -430,6 +440,21 @@ impl Worker { Command::TransferLeader(member) => { group.owner.transfer_leader(member).map(|()| Response::Unit) } + Command::TransferLeadership(transfer) => group + .owner + .transfer_leadership(transfer) + .map(|()| Response::Unit), + Command::RestartLeadership(check) => self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed) + .and_then(|now| { + crate::raft::restart_leadership::check_gate( + &self.log, now, &check, + ) + }) + .and_then(|()| group.owner.restart_leadership(*check)) + .map(|()| Response::Unit), Command::Campaign => { group.owner.campaign().map(|()| Response::Unit) } @@ -712,24 +737,67 @@ impl Worker { } let mut made = self.progress()? | self.start_batch()? | self.flush(); let now = self.domain.mono_ms().map_err(|_| RuntimeError::Closed)?; - made |= - self.cell_reads - .progress(now, &self.domain, |history, request, term, barrier| { - let group = self.groups.get(&history).ok_or(RuntimeError::NotFound)?; - if let Some(error) = &group.failed { - return Err(error.clone()); - } - match &request.kind { - reads::ReadKind::Cell(cell, consistency) => group + made |= self.cell_reads.progress( + now, + &self.domain, + |history, request, term, barrier, context| { + let group = self + .groups + .get_mut(&history) + .ok_or(RuntimeError::NotFound)?; + if let Some(error) = &group.failed { + return Err(error.clone()); + } + match &request.kind { + reads::ReadKind::Cell(cell, consistency) => { + if matches!(consistency, CellReadConsistency::Linearizable) + && term.is_none() + { + group.owner.read_forwarded_cell_index(context.to_vec())?; + *term = Some(group.owner.status()?.term); + *barrier = None; + } + group .owner - .select_read(cell, consistency, term, barrier) - .map(reads::Selection::Cell), - reads::ReadKind::Seed(target) => group + .select_read(cell, consistency, *term, *barrier) + .map(reads::Selection::Cell) + } + reads::ReadKind::Seed(target) => group + .owner + .select_seed(target, *term, *barrier) + .map(reads::Selection::Seed), + reads::ReadKind::Membership { minimum } => { + let status = group.owner.status()?; + if term.is_some_and(|observed| observed != status.term) { + return Err(RuntimeError::NotLeader); + } + if status.applied.index < *minimum + || !group.owner.membership_read_ready()? + { + return Err(RuntimeError::NotReady); + } + if barrier.is_some_and(|index| { + group.owner.manifest().marker().membership_revision() > index.get() + }) { + // A configuration overtook this read's barrier. + // Re-arm within the same invocation, reservation + // and original deadline; never spin a new call. + *term = None; + *barrier = None; + } + if term.is_none() { + group.owner.read_membership_index(context.to_vec())?; + *term = Some(status.term); + return Err(RuntimeError::NotReady); + } + group .owner - .select_seed(target, term, barrier) - .map(reads::Selection::Seed), + .select_membership(*minimum, *term, *barrier) + .map(reads::Selection::Membership) } - }); + } + }, + ); let deadline = if self.retry_at > now { tick_at.min(self.retry_at) } else { diff --git a/crates/cellar/durability/raft/command_cell_transport.rs b/crates/cellar/durability/raft/command_cell_transport.rs index 3332b7d79..8861a0b56 100644 --- a/crates/cellar/durability/raft/command_cell_transport.rs +++ b/crates/cellar/durability/raft/command_cell_transport.rs @@ -4,8 +4,8 @@ use super::*; use crate::application::consensus::CommandFailure; use crate::raft::cell_groups::{ - CellDispatchError, CellGroupController, CellGroupDispatcher, CellRoutingLimits, CellSqlOptions, - EnrollmentRecord, + CellDispatchError, CellGroupController, CellGroupDispatcher, CellGroupNamespaceStorage, + CellRoutingLimits, EnrollmentRecord, }; use crate::raft::query_identity::QueryRoute; use crate::raft::query_wire::{ @@ -45,7 +45,7 @@ pub(super) struct Gateway { accepting: AtomicBool, controller: CellGroupController, dispatcher: CellGroupDispatcher, - options: CellSqlOptions, + storage: CellGroupNamespaceStorage, } /// Retains the same node/peer query admission through consumption of the result. @@ -227,13 +227,19 @@ impl RaftTransport { &self, dispatcher: CellGroupDispatcher, controller: CellGroupController, - options: CellSqlOptions, + storage: CellGroupNamespaceStorage, ) -> Result { let state = &self.handle.0; let runtime = state.runtime.shared()?; if !dispatcher.uses_domain(&state.domain) || !controller.belongs_to(runtime) { return Err(TransportError::Identity); } + let (namespace, _, _) = dispatcher + .host_binding() + .map_err(|_| TransportError::Identity)?; + storage + .validate_namespace(&namespace) + .map_err(|_| TransportError::Identity)?; let key = dispatcher.namespace(); let mut gateways = state .cell_command_gateways @@ -252,7 +258,7 @@ impl RaftTransport { accepting: AtomicBool::new(true), controller, dispatcher, - options, + storage, }); gateways.insert(key, Arc::downgrade(&inner)); Ok(CellCommandGateway { @@ -614,6 +620,7 @@ impl State { if !gateway.accepting.load(Ordering::Acquire) { return Err(TransportError::Refused); } + let options = gateway.storage.for_cell(&cell); let binding = crate::raft::cell_groups::RemoteMutationRoute { group: target.scope, schema: route.schema(), @@ -638,7 +645,7 @@ impl State { .map_err(|_| CommandFailure::Unavailable)?; gateway .controller - .lookup_receipt(invocation, gateway.options.clone()) + .lookup_receipt(invocation, options) .await .map_err(|e| command_failure(&e)) } @@ -662,6 +669,7 @@ impl State { if !gateway.accepting.load(Ordering::Acquire) { return Err(TransportError::Refused); } + let options = gateway.storage.for_cell(&cell); let binding = crate::raft::cell_groups::RemoteMutationRoute { group: target.scope, schema: route.schema(), @@ -691,7 +699,7 @@ impl State { .map_err(|_| CommandFailure::Unavailable)?; let call = gateway .controller - .enroll(invocation, gateway.options.clone()) + .enroll(invocation, options) .map_err(|error| command_failure(&error.error.cause))?; call.response() .await @@ -717,6 +725,7 @@ impl State { if !gateway.accepting.load(Ordering::Acquire) { return Err(TransportError::Refused); } + let options = gateway.storage.for_cell(&cell); let binding = crate::raft::cell_groups::RemoteMutationRoute { group: target.scope, schema: route.schema(), @@ -748,7 +757,7 @@ impl State { .map_err(|_| CommandFailure::Unavailable)?; let call = gateway .controller - .command(invocation, gateway.options.clone()) + .command(invocation, options) .map_err(|error| command_failure(&error.error.cause))?; call.response() .await diff --git a/crates/cellar/durability/raft/completion.rs b/crates/cellar/durability/raft/completion.rs index 68ed84a4a..40bc0e0d2 100644 --- a/crates/cellar/durability/raft/completion.rs +++ b/crates/cellar/durability/raft/completion.rs @@ -147,8 +147,8 @@ pub(crate) struct RestartLeadershipCheck { /// Leader-local replication observations. These are neither apply nor quorum /// proofs; the owner rechecks actual progress when accepting a transfer. #[derive(Clone, Debug, PartialEq, Eq)] -pub struct ReplicationStatus { - pub replica: ReplicaIdentity, +pub struct ReplicationStatus { + pub replica: ReplicaIdentity, pub term: Term, pub membership_revision: LogIndex, pub committed: LogIndex, diff --git a/crates/cellar/durability/raft/effect_delivery.rs b/crates/cellar/durability/raft/effect_delivery.rs index a25c016aa..434c76228 100644 --- a/crates/cellar/durability/raft/effect_delivery.rs +++ b/crates/cellar/durability/raft/effect_delivery.rs @@ -15,11 +15,7 @@ use ed25519_dalek::{Signature, Signer, SigningKey, VerifyingKey}; use futures_util::future::BoxFuture; use serde::{de::DeserializeOwned, Deserialize, Serialize}; use sha2::{Digest, Sha256}; -use std::{ - fmt, - sync::{Arc, Weak}, - time::Duration, -}; +use std::{fmt, sync::Arc, time::Duration}; #[cfg(all(test, not(cellar_internal_tests)))] #[path = "effect_delivery_tests.rs"] @@ -569,8 +565,7 @@ pub struct RaftNativeEffectSink { #[derive(Clone)] struct NativeEndpoint { - handle: Weak, - scope: CellHistory, + handle: crate::application::consensus::group_client::WeakCellHandle, } impl RaftNativeEffectSink { @@ -591,13 +586,10 @@ impl RaftNativeEffectSink { Ok(Arc::new(Self { endpoints: endpoints .into_iter() - .map(|endpoint| { - Ok(NativeEndpoint { - handle: Arc::downgrade(&endpoint.handle), - scope: CellHistory::try_from(endpoint.scope())?, - }) + .map(|endpoint| NativeEndpoint { + handle: endpoint.downgrade(), }) - .collect::>>()?, + .collect(), timeout, retry_window_ms, })) @@ -619,21 +611,12 @@ impl RaftEffectSink for RaftNativeEffectSink { .ok_or(CommandFailure::Unavailable)?; let mut last = CommandFailure::Unavailable; for hint in endpoints { - let Some(handle) = hint.handle.upgrade() else { - continue; + let endpoint = match hint.handle.upgrade() { + Ok(Some(endpoint)) => endpoint, + Ok(None) | Err(_) => continue, }; - let binding = match handle - .bindings - .get(&(hint.scope.namespace, hint.scope.cell.as_str().to_owned())) - { - Ok(binding) => binding, - Err(CommandFailure::Unavailable) => continue, - Err(error) => return Err(error), - }; - let endpoint = RaftCellHandle { handle, binding }; - if CellHistory::try_from(endpoint.scope()).as_ref().ok() - != Some(request.inner.target.scope()) - || endpoint.binding.class.definition_digest() != request.inner.target.definition + if endpoint.history() != request.inner.target.scope() + || endpoint.definition_digest() != request.inner.target.definition { continue; } @@ -645,11 +628,12 @@ impl RaftEffectSink for RaftNativeEffectSink { options.durability = request.inner.target.durability; match endpoint .command( - request.inner.target.contract.clone(), + &request.inner.target.contract, request.inner.bytes.clone(), options, ) .await + .map_err(|error| native_command_failure(&error)) { Ok(receipt) => { request @@ -673,3 +657,28 @@ impl RaftEffectSink for RaftNativeEffectSink { }) } } + +fn native_command_failure(error: &crate::raft::cell_groups::CellCommandError) -> CommandFailure { + use crate::raft::{ + cell_groups::CellDispatchError, completion::RuntimeError, execution::ExecutionError, + }; + if let CellDispatchError::Execution(ExecutionError::Refused(failure)) = error.cause.as_ref() { + return failure.clone(); + } + if error.ambiguous { + return CommandFailure::Ambiguous { + command: error.command, + }; + } + match error.cause.as_ref() { + CellDispatchError::Runtime(RuntimeError::NotLeader) => { + CommandFailure::NotLeader { leader: None } + } + CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, + CellDispatchError::Registration(_) => CommandFailure::UnsupportedCapability, + CellDispatchError::Version(crate::raft::version::VersionError::WrongHistory) => { + CommandFailure::WrongIncarnation + } + _ => CommandFailure::Unavailable, + } +} diff --git a/crates/cellar/durability/raft/fleet_action_transport.rs b/crates/cellar/durability/raft/fleet_action_transport.rs index 356b328b3..1781f0570 100644 --- a/crates/cellar/durability/raft/fleet_action_transport.rs +++ b/crates/cellar/durability/raft/fleet_action_transport.rs @@ -387,11 +387,11 @@ impl State { stream.read_exact(&mut bytes).await?; let command = request.decode_body(&bytes)?; self.authorize_node_inspection(&request.route)?; - let handle = self.query_gateway.lock().map_err(|_| TransportError::Closed)? + let handle = self.fleet_gateway.lock().map_err(|_| TransportError::Closed)? .as_ref().and_then(|handle| handle.upgrade()).ok_or(TransportError::Refused)?; let remaining = deadline.checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) .filter(|ms| *ms > 0).ok_or(TransportError::Timeout)?; - let work = RaftCells { handle }.execute_local_fleet_action(command, request.origin.clone(), + let work = RaftCells::from_internal_handle(handle).execute_local_fleet_action(command, request.origin.clone(), request.route.minimum_revision()?, std::time::Duration::from_millis(remaining)); let mut extra = [0; 1]; let result = crate::asyncrt::select_biased! { "disconnect cancels the waiter, not committed movement"; diff --git a/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs b/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs new file mode 100644 index 000000000..c12e8c5ff --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/archive_replay.rs @@ -0,0 +1,208 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Complete physical group replay for managed lifecycle verification. The +//! supplied inventories belong to the verifier, separately from serving data. +use super::*; +use crate::application::consensus::group_protocol::GroupEntryLimits; +use crate::raft::catalog::{CellGroupArchiveBindingError, RaftImageStorage, RangeDirectory}; +use crate::raft::cell_groups::{ + CellGroupApplyImages, CellGroupArchiveCheckpointReceiver, CellGroupArchivePool, + CellGroupArchiveRegistry, CellGroupArchiveStore, CellGroupArchiveTailPool, + CellGroupCheckpointError, CellGroupCheckpointPool, ManifestLimits, ManifestPageStore, +}; +use cellar_logic::cell_group::GroupRoute; +use cellar_ltx::{stream_snapshot::ScratchPool, FileSystem}; +use sha2::{Digest, Sha256}; +use std::path::{Component, PathBuf}; + +/// Explicit verifier-owned physical stores and existing per-group page bounds. +/// Image, checkpoint, tail and scratch pools are shared across all histories. +/// Use private inventories: serving stores cannot substitute for S3 replay. +pub struct FleetGroupArchiveReplayOptions { + pub directory: PathBuf, + pub filesystem: Arc, + pub materialized: Option, + pub extents: Option, + pub manifest: ManifestLimits, + pub entries: GroupEntryLimits, + pub max_pages: usize, + pub max_page_bytes: u64, + pub max_retained_roots: usize, + pub max_effect_rows: u32, +} + +/// Opens independent receivers on the caller's execution domain. This owns no +/// Raft log, application callback, admission authority or remote retention lease. +#[derive(Clone)] +pub struct FleetGroupArchiveVerifier { + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, + scratch: ScratchPool, + options: Arc, +} + +impl FleetGroupArchiveVerifier { + pub fn new( + archives: CellGroupArchivePool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, + scratch: ScratchPool, + options: FleetGroupArchiveReplayOptions, + ) -> Result { + if !options.directory.is_absolute() + || options.directory.as_os_str().len() > 3800 + || options + .directory + .components() + .any(|p| matches!(p, Component::ParentDir)) + || options.materialized.is_none() && options.extents.is_none() + || options + .materialized + .as_ref() + .is_some_and(|i| !matches!(i, CellGroupApplyImages::Materialized { .. })) + || options + .extents + .as_ref() + .is_some_and(|i| !matches!(i, CellGroupApplyImages::Extents { .. })) + || !(1..=1_000_000).contains(&options.max_effect_rows) + { + return Err(FleetCatalogError::Invalid("group archive replay storage")); + } + ManifestPageStore::validate_limits( + options.max_pages, + options.max_page_bytes, + options.max_retained_roots, + ) + .and_then(|()| options.manifest.validate()) + .map_err(|_| FleetCatalogError::Limit)?; + options + .entries + .validate() + .map_err(|_| FleetCatalogError::Invalid("group archive entry limits"))?; + for images in options.materialized.iter().chain(options.extents.iter()) { + images + .validate() + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + } + Ok(Self { + archives, + tails, + checkpoints, + scratch, + options: Arc::new(options), + }) + } + + pub(crate) fn tails(&self) -> &CellGroupArchiveTailPool { + &self.tails + } + + /// Reopen the immutable namespace-owned registry for this exact rooted + /// physical history. Never initialize counters or select a cell image. + pub(crate) async fn open( + &self, + domain: &DomainToken, + namespace: &RangeDirectory, + history: &FleetHistory, + ) -> Result { + domain + .run(async { + let scope = history + .binding() + .scope() + .map_err(|_| FleetCatalogError::Corrupt("archive history"))?; + let birth = namespace.materialize_selected_history(scope).await?; + if birth.route().history != scope + || birth.route().birth_digest != history.birth_digest() + || birth.route().schema_digest != history.schema_digest() + { + return Err(FleetCatalogError::Conflict); + } + namespace + .open_group_archive(domain, &self.archives, &birth) + .await + .map_err(|e| match e { + CellGroupArchiveBindingError::Catalog(e) => FleetCatalogError::from(e), + CellGroupArchiveBindingError::Archive(e) => FleetCatalogError::from(e), + }) + }) + .await + .map_err(|_| FleetCatalogError::Closed)? + } + + /// Allocate a complete receiver under the existing checkpoint/scratch + /// admissions. Deterministic group paths keep retries bounded; a prior + /// entered worker retains its directory lock until physical work exits. + pub(crate) async fn receiver( + &self, + domain: &DomainToken, + route: &GroupRoute, + storage: RaftImageStorage, + ) -> Result { + let route = route.clone(); + let mut images = match storage { + RaftImageStorage::Materialized => self.options.materialized.as_ref(), + RaftImageStorage::Extents => self.options.extents.as_ref(), + } + .ok_or(FleetCatalogError::Invalid( + "missing group archive image inventory", + ))? + .clone(); + let mut digest = Sha256::new(); + digest.update(b"cellar-fleet-group-replay-directory-1"); + digest.update(route.history.cluster.bytes()); + digest.update(route.history.namespace.bytes()); + digest.update(route.history.group.bytes()); + digest.update(route.history.incarnation.bytes()); + let key = hex::encode(digest.finalize()); + if let CellGroupApplyImages::Extents { + cache_directory, .. + } = &mut images + { + *cache_directory = self.options.directory.join("cache").join(&key); + } + let options = self.options.clone(); + let pages = options.directory.join("groups").join(key).join("pages"); + let admission = self + .checkpoints + .prepare_archive_receive(options.max_pages) + .map_err(checkpoint_error)?; + let scratch = self.scratch.clone(); + domain + .blocking(move || { + let pages = ManifestPageStore::open( + &pages, + options.filesystem.clone(), + options.max_pages, + options.max_page_bytes, + options.max_retained_roots, + ) + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + let store = CellGroupArchiveStore::new( + route, + pages, + images, + options.manifest, + options.entries, + options.max_effect_rows, + ) + .map_err(|e| FleetCatalogError::Storage(e.into()))?; + admission + .receive(store, scratch, options.directory.join("scratch")) + .map_err(checkpoint_error) + }) + .map_err(|_| FleetCatalogError::Closed)? + .await + .map_err(|e| FleetCatalogError::Storage(e.into()))? + } +} + +fn checkpoint_error(error: CellGroupCheckpointError) -> FleetCatalogError { + match error { + CellGroupCheckpointError::Invalid(reason) => FleetCatalogError::Invalid(reason), + CellGroupCheckpointError::Limit => FleetCatalogError::Limit, + CellGroupCheckpointError::Busy => FleetCatalogError::Overloaded, + error => FleetCatalogError::Storage(error.into()), + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/capacity.rs b/crates/cellar/durability/raft/fleet_catalog/capacity.rs index e54ef9476..eae2e1a64 100644 --- a/crates/cellar/durability/raft/fleet_catalog/capacity.rs +++ b/crates/cellar/durability/raft/fleet_catalog/capacity.rs @@ -6,7 +6,7 @@ use super::histories::validate_request; use super::nodes::NODE_KIND; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use crate::raft::host::RaftPeerNodeObservation; use crate::raft::inspection::RaftNodeProfile; @@ -203,7 +203,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetCapacityRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs index e1d333d9a..4416060ba 100644 --- a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/mod.rs @@ -5,7 +5,7 @@ //! Immutable IDs never authorize deletion or replacement-container discovery. use super::*; use crate::docker::{ContainerId, Docker}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use cellar_logic::consensus::{NodeId, ProposalId}; use std::pin::Pin; @@ -42,7 +42,7 @@ pub struct FleetDockerIsolation { struct Adapter { catalog: FleetCatalog, engine: Engine, - namespaces: Vec, + namespaces: Vec, turn: Mutex<()>, } @@ -54,7 +54,7 @@ impl FleetDockerIsolation { catalog: FleetCatalog, docker: Docker, owner: &str, - namespaces: Vec, + namespaces: Vec, ) -> Result { if namespaces.len() > super::namespaces::MAX_NAMESPACES { return Err(FleetCatalogError::Limit); @@ -176,7 +176,7 @@ impl FleetFailureRecoveryPlatform for FleetDockerIsolation { fn namespaces<'a>( &'a self, timeout: Duration, - ) -> Pin>> + Send + 'a>> { + ) -> Pin>> + Send + 'a>> { Box::pin(async move { self.run(timeout, async { Ok(self.inner.namespaces.clone()) }) .await diff --git a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs index 336f2f81b..34c1cf859 100644 --- a/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs +++ b/crates/cellar/durability/raft/fleet_catalog/docker_isolation/resources.rs @@ -201,10 +201,6 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { let timeout = timeout.min(Duration::from_secs(5)); self.isolation .run(timeout, async { - context - .group - .validate() - .map_err(|_| FleetCatalogError::Invalid("Docker repair scope"))?; context .failed_boot .validate() @@ -212,6 +208,9 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { let catalog = &self.isolation.inner.catalog; let deadline = Deadline::new(catalog, timeout)?; let cut = catalog.read(deadline.remaining()?).await?; + catalog + .authorize_group(&cut, &context.group, deadline.remaining()?) + .await?; let nodes = catalog .run(deadline.remaining()?, catalog.node_records(&cut)) .await?; @@ -234,7 +233,7 @@ impl FleetIsolatedRepairEvidence for FleetDockerResources { } Ok(FleetIsolatedRepairInputs { history: FleetTopologyHistoryResources { - group: context.group.clone(), + group: context.group, resident: self.policy.resident, peak: self.policy.peak, }, diff --git a/crates/cellar/durability/raft/fleet_catalog/evacuation.rs b/crates/cellar/durability/raft/fleet_catalog/evacuation.rs index 88852e77e..abfac4f4b 100644 --- a/crates/cellar/durability/raft/fleet_catalog/evacuation.rs +++ b/crates/cellar/durability/raft/fleet_catalog/evacuation.rs @@ -3,7 +3,7 @@ use super::histories::{history_mutation, validate_request}; use super::nodes::{NodeRecord, NODE_KIND}; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_stop::BootBinding; use cellar_logic::consensus::{DiskLineage, EngineInstanceId, NodeId, ProposalId}; use serde::Deserialize; @@ -291,7 +291,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -323,7 +323,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -352,7 +352,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, failure: (BootBinding, u64), - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -384,7 +384,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, node: NodeId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], mode: AdmissionMode, ) -> Result { let current = self.inner.current().await?; diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs new file mode 100644 index 000000000..c42f53d9b --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/archive_holds.rs @@ -0,0 +1,228 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One durable acquire allocation for an exact verification basis and fixed cut. +//! A timeout leaves the allocation untouched. These holds outlive local handles; +//! terminal cleanup must fence all users and join physical work before release. +use super::*; +use crate::raft::cell_groups::{ + CellGroupArchiveAcquireOutcome, CellGroupArchiveCreation, CellGroupArchiveError, + CellGroupArchiveIntent, +}; +use receipts::Publication; +use serde::Deserialize; + +const KIND: u8 = 101; + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Record { + basis: Basis, + fixed: Publication, + registry: CellGroupArchiveCreation, + attempt: u64, + intent: CellGroupArchiveIntent, +} + +impl Record { + fn check( + &self, + basis: &Basis, + fixed: &FleetPublishedCut, + registry: &CellGroupArchiveRegistry, + ) -> Result<()> { + if &self.basis != basis || &self.registry != registry.creation() || self.attempt == u64::MAX + { + return Err(FleetCatalogError::PayloadConflict); + } + if self.fixed != Publication::from(fixed) { + return Err(FleetCatalogError::Conflict); + } + check_marker(self.intent.acquire_marker()?, fixed) + } +} + +fn check_marker( + marker: &crate::raft::cell_groups::GroupManifest, + fixed: &FleetPublishedCut, +) -> Result<()> { + if !crate::raft::host::group_prefix(marker, fixed.root()) + .map_err(|_| FleetCatalogError::Conflict)? + { + return Err(FleetCatalogError::Unconfirmed); + } + Ok(()) +} + +impl FleetEvacuationVerification { + fn archive_hold_key(&self) -> Result { + let pending = &self + .pending + .as_ref() + .ok_or(FleetCatalogError::InventoryIncomplete)? + .0; + Ok(format!( + "removal-archive-acquires/{}/{}", + hex::encode(hash(&encode(KIND, &self.book.basis, PAGE_LIMIT)?)), + pending.key + )) + } + + /// Recover a fixed cut even if the last controller disappeared between + /// acquiring retention and appending a live receipt. A new observation must + /// cover the stored cut; it cannot silently select a new acquire journal. + pub(super) async fn archive_fixed( + &self, + current: &FleetPublishedCut, + ) -> Result> { + let Some((record, _)) = self + .catalog + .inner + .read::(&self.archive_hold_key()?, KIND, PAGE_LIMIT) + .await? + else { + return Ok(None); + }; + let fixed = record.fixed.restore(current)?; + if record.basis != self.book.basis + || record.registry.route() != fixed.root().route() + || record.attempt == u64::MAX + { + return Err(FleetCatalogError::PayloadConflict); + } + check_marker(record.intent.acquire_marker()?, &fixed)?; + Ok(Some(fixed)) + } + + /// Acquire this walk's own remote retention. Call freeze_live first. The + /// exact intent is stored in the fleet catalog before any registry mutation. + /// Unknown outcomes reconcile the same allocation after controller loss. + /// No publisher-owned hold, current-health lease or terminal authority is + /// borrowed. Local cancellation/drop never retires this persistent owner. + pub async fn acquire_archive_hold( + &mut self, + registry: &CellGroupArchiveRegistry, + timeout: Duration, + ) -> Result { + let catalog = &self.catalog; + let deadline = Deadline::new(catalog, timeout)?; + let fixed = &self + .frozen + .as_ref() + .ok_or(FleetCatalogError::Unconfirmed)? + .fixed; + let basis = &self.book.basis; + if registry.route() != fixed.root().route() { + return Err(FleetCatalogError::Conflict); + } + let key = self.archive_hold_key()?; + let result = catalog + .run(deadline.remaining()?, async { + basis.check_current(catalog).await?; + let (record, token) = + match catalog.inner.read::(&key, KIND, PAGE_LIMIT).await? { + Some(stored) => stored, + None => { + let candidate = prepare(catalog, registry, basis, fixed, 0).await?; + persist(catalog, &key, &candidate, None).await? + } + }; + record.check(basis, fixed, registry)?; + basis.check_current(catalog).await?; + match registry + .acquire(&catalog.inner.domain, &record.intent) + .await + { + Ok(hold) => Ok(hold), + Err( + error @ (CellGroupArchiveError::Expired | CellGroupArchiveError::Conflict), + ) => { + match registry + .acquire_outcome(&catalog.inner.domain, &record.intent) + .await? + { + CellGroupArchiveAcquireOutcome::Held(hold) => Ok(*hold), + CellGroupArchiveAcquireOutcome::Pending => Err(error.into()), + CellGroupArchiveAcquireOutcome::Superseded => { + // A fresh consistent registry read proved that the + // old revision is consumed and its nonce owns no + // hold. Its delayed conditional write cannot win. + let next = record + .attempt + .checked_add(1) + .ok_or(FleetCatalogError::Limit)?; + let candidate = + prepare(catalog, registry, basis, fixed, next).await?; + basis.check_current(catalog).await?; + let (winner, _) = + persist(catalog, &key, &candidate, Some(&token)).await?; + winner.check(basis, fixed, registry)?; + // Reconcile this persisted winner on a later turn; + // never hide another allocation retry in this call. + Err(FleetCatalogError::Unconfirmed) + } + } + } + Err(error) => Err(error.into()), + } + }) + .await; + let hold = match result { + Ok(hold) => hold, + Err(error) => { + // Another controller may have persisted an earlier fixed cut + // since this one's quorum capture. Re-enter fresh quorum proof + // and restore that winner; do not move its journal or nonce. + self.frozen = None; + return Err(error); + } + }; + check_marker(hold.marker(), fixed)?; + catalog + .run(deadline.remaining()?, basis.check_current(catalog)) + .await?; + deadline.remaining()?; + Ok(hold) + } +} + +async fn prepare( + catalog: &FleetCatalog, + registry: &CellGroupArchiveRegistry, + basis: &Basis, + fixed: &FleetPublishedCut, + attempt: u64, +) -> Result { + let intent = registry.prepare_acquire(&catalog.inner.domain).await?; + let record = Record { + basis: basis.clone(), + fixed: fixed.into(), + registry: registry.creation().clone(), + attempt, + intent, + }; + record.check(basis, fixed, registry)?; + Ok(record) +} + +/// Conditional journal storage chooses one candidate even when simultaneous +/// controllers prepare different nonces. Only the observed winner may be sent. +async fn persist( + catalog: &FleetCatalog, + key: &str, + candidate: &Record, + token: Option<&str>, +) -> Result<(Record, String)> { + let bytes = encode(KIND, candidate, PAGE_LIMIT)?; + let outcome = catalog + .inner + .bucket + .put_cas(&catalog.inner.key(key), bytes, token) + .await; + match catalog.inner.read(key, KIND, PAGE_LIMIT).await? { + Some(stored) => Ok(stored), + None => match outcome { + Err(error) => Err(FleetCatalogError::Storage(error)), + Ok(_) => Err(FleetCatalogError::Unconfirmed), + }, + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs index 2af4f1fae..50ac7e155 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/fencing.rs @@ -262,7 +262,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.begin_finalization_inner( @@ -284,7 +284,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, isolation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.begin_finalization_inner( @@ -306,7 +306,7 @@ impl FleetEvacuationCandidate { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, source: FinalizationSource<'_>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let catalog = &self.catalog; diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs index f5ea5c5e6..0db58511b 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/mod.rs @@ -2,6 +2,7 @@ //! Bounded receipt material for inventory-consuming terminal verification. //! Stored descriptors grant no completion or disruption-release authority. +mod archive_holds; mod cancellation; mod fencing; pub(super) mod receipts; @@ -19,8 +20,11 @@ pub use terminal::{ use super::*; use crate::application::consensus::RaftCells; -use crate::raft::archive::{ArchiveReference, ArchiveVerifier, CellArchive}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; +use crate::raft::cell_groups::{ + CellGroupArchiveCheckpointReceiver, CellGroupArchiveHold, CellGroupArchiveRegistry, + CellGroupArchiveTailPool, +}; use crate::raft::host::FleetPublishedCut; use cellar_logic::consensus::{NodeId, ProposalId}; use receipts::{Basis, Book, Candidate, Entry, Evidence, LiveEvidence}; @@ -28,9 +32,10 @@ use receipts::{Basis, Book, Candidate, Entry, Evidence, LiveEvidence}; /// Services and the exact immutable archive selected for one history. Scratch /// storage and recovery-memory admission remain owned by the embedding runtime. pub struct FleetEvacuationArchive<'a> { - pub archive: &'a CellArchive, - pub reference: &'a ArchiveReference, - pub verifier: &'a ArchiveVerifier, + pub registry: &'a CellGroupArchiveRegistry, + pub hold: &'a CellGroupArchiveHold, + pub tails: &'a CellGroupArchiveTailPool, + pub receiver: CellGroupArchiveCheckpointReceiver, } /// A bounded, complete-inventory walk that owns its authenticated quorum calls @@ -107,7 +112,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -327,13 +332,12 @@ impl FleetEvacuationVerification { .try_into() .map_err(|_| FleetCatalogError::Corrupt("receipt observation count"))?; let observations: [crate::raft::host::RaftPeerFleetMembershipObservation; 3] = observations; - let current = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let current = observations[0].published_cut(); let (fixed, survivors) = catalog .run(deadline.remaining()?, async { self.book.basis.check_current(&catalog).await?; let mut fixed = current.clone(); + let mut from_prior = false; if let Some(prior) = &self.prior { let (pending, _) = self .pending @@ -342,13 +346,25 @@ impl FleetEvacuationVerification { if let Some(entry) = prior.entry(&catalog, &pending.key).await? { if entry.record == hash(&pending.value) { if let Evidence::Live(live) = entry.evidence { - if live.fixed.revision == current.membership_revision().get() { + if live.fixed.configuration_revision()? + == current.membership_revision() + { fixed = live.fixed.restore(current)?; + from_prior = true; } } } } } + if let Some(saved) = self.archive_fixed(current).await? { + if from_prior + && (saved.root() != fixed.root() + || saved.configuration_digest() != fixed.configuration_digest()) + { + return Err(FleetCatalogError::Conflict); + } + fixed = saved; + } let survivors = survivors::verify(&catalog, &self.snapshot, history, &observations, &fixed) .await?; @@ -383,17 +399,20 @@ impl FleetEvacuationVerification { .freeze_live(observer, nodes, deadline.remaining()?) .await?; let coverage = catalog - .inner - .domain - .run(archive.archive.verify_published_at( - archive.reference, - &fixed, - archive.verifier, - deadline.remaining()?, - )) - .await - .map_err(|_| FleetCatalogError::Closed)? - .map_err(|error| FleetCatalogError::Storage(error.into()))?; + .run(deadline.remaining()?, async { + archive + .registry + .verify_fleet_published_at( + &catalog.inner.domain, + archive.tails, + archive.hold, + &fixed, + archive.receiver, + ) + .await + .map_err(FleetCatalogError::from) + }) + .await?; deadline.remaining()?; if !coverage .covers(&fixed) diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs index e5f6fa769..115ea7a00 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts.rs @@ -4,8 +4,10 @@ //! from these records; the runtime must independently establish every proof. use super::super::evacuation::ActiveEvacuation; use super::super::*; -use crate::raft::archive::{ArchiveReference, VerifiedArchiveCoverage}; -use crate::raft::host::{FleetImageState, FleetPublishedCut}; +use crate::raft::cell_groups::{ + CellGroupArchiveHold, GroupManifest, VerifiedFleetGroupArchiveCoverage, +}; +use crate::raft::host::FleetPublishedCut; use crate::raft::manifest::Position; use cellar_logic::consensus::{DiskLineage, EngineInstanceId, NodeId, ProcessGeneration}; use serde::Deserialize; @@ -13,6 +15,11 @@ use std::collections::BTreeSet; const CANDIDATE_KIND: u8 = 55; const ENTRY_KIND: u8 = 56; +const LIVE_KIND: u8 = 59; +#[cfg(test)] +#[path = "receipts_tests.rs"] +mod tests; + const ENTRY_LIMIT: usize = 4096; const APPEND_LIMIT: usize = 16; const BASIS_TABLES: [FleetTable; 4] = [ @@ -218,21 +225,15 @@ impl Basis { #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(in crate::raft::fleet_catalog) struct Publication { - pub(super) applied: Position, - pub(super) revision: u64, - #[serde(deserialize_with = "nullable")] - image_position: Option, - #[serde(deserialize_with = "nullable")] - image: Option, + root: GroupManifest, + configuration_digest: [u8; 32], } impl From<&FleetPublishedCut> for Publication { fn from(cut: &FleetPublishedCut) -> Self { Self { - applied: cut.applied().into(), - revision: cut.membership_revision().get(), - image_position: cut.image().map(|image| image.position().into()), - image: cut.image().map(|image| image.state()), + root: cut.root().clone(), + configuration_digest: cut.configuration_digest(), } } } @@ -241,94 +242,42 @@ impl Publication { pub(in crate::raft::fleet_catalog) fn applied_position( &self, ) -> Result { - self.applied - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication position")) + self.root + .check() + .map_err(|_| FleetCatalogError::Corrupt("receipt group root"))?; + Ok(self.root.position()) } pub(in crate::raft::fleet_catalog) fn configuration_revision( &self, ) -> Result { - if self.revision == 0 || self.revision >= u64::MAX - 2 { - return Err(FleetCatalogError::Corrupt("receipt configuration revision")); - } - Ok(cellar_logic::consensus::LogIndex::new(self.revision)) + self.root + .check() + .map_err(|_| FleetCatalogError::Corrupt("receipt configuration root"))?; + Ok(cellar_logic::consensus::LogIndex::new( + self.root.membership_revision(), + )) } pub(super) fn restore(&self, current: &FleetPublishedCut) -> Result { - validate_image(self.applied, self.image_position, self.image)?; - let image = match (self.image_position, self.image) { - (Some(position), Some(image)) => Some( - crate::raft::host::FleetPublishedImage::new_state( - position - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication"))?, - image, - ) - .map_err(|_| FleetCatalogError::Corrupt("receipt image"))?, - ), - (None, None) => None, - _ => return Err(FleetCatalogError::Corrupt("receipt image binding")), - }; current - .checked_prefix( - self.applied - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt position"))?, - cellar_logic::consensus::LogIndex::new(self.revision), - image, - ) + .checked_prefix(self.root.clone(), self.configuration_digest) .map_err(|_| FleetCatalogError::Unconfirmed) } } -fn validate_image( - position: Position, - origin: Option, - image: Option, -) -> Result<()> { - position - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt position"))?; - match (origin, image) { - (None, None) => Ok(()), - (Some(origin), Some(image)) => { - origin - .decode() - .map_err(|_| FleetCatalogError::Corrupt("receipt publication"))?; - image - .validate() - .map_err(|_| FleetCatalogError::Corrupt("receipt image"))?; - if origin.index > position.index - || origin.term > position.term - || origin.index == position.index && origin.term != position.term - { - return Err(FleetCatalogError::Corrupt("receipt publication prefix")); - } - Ok(()) - } - _ => Err(FleetCatalogError::Corrupt("receipt image binding")), - } -} - #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub(super) struct Archive { - reference: ArchiveReference, - position: Position, - #[serde(deserialize_with = "nullable")] - image_position: Option, - #[serde(deserialize_with = "nullable")] - image: Option, + hold: CellGroupArchiveHold, + configuration_digest: [u8; 32], } -impl From<&VerifiedArchiveCoverage> for Archive { - fn from(proof: &VerifiedArchiveCoverage) -> Self { +impl From<&VerifiedFleetGroupArchiveCoverage> for Archive { + fn from(proof: &VerifiedFleetGroupArchiveCoverage) -> Self { Self { - reference: proof.reference().clone(), - position: proof.position().into(), - image_position: proof.image_position().map(Into::into), - image: proof.image_state(), + hold: proof.hold().clone(), + configuration_digest: proof.configuration_digest(), } } } @@ -384,40 +333,24 @@ impl Entry { archive, survivors, } = live.as_ref(); - validate_image(fixed.applied, fixed.image_position, fixed.image)?; - validate_image(archive.position, archive.image_position, archive.image)?; - let selected = archive - .reference - .position() - .map_err(|_| FleetCatalogError::Corrupt("receipt archive reference"))?; - if Position::from(selected) != archive.position - || fixed.revision == 0 - || archive.position.index < fixed.applied.index - || archive.position.term < fixed.applied.term - || archive.position.index == fixed.applied.index - && archive.position.term != fixed.applied.term + fixed + .root + .check() + .and_then(|()| archive.hold.marker().check()) + .map_err(|_| FleetCatalogError::Corrupt("receipt complete group root"))?; + archive + .hold + .check_reference() + .map_err(|_| FleetCatalogError::Corrupt("receipt archive hold"))?; + if fixed.configuration_digest == [0; 32] + || fixed.configuration_digest != archive.configuration_digest + || fixed.root.route().history.cluster.bytes() != basis.cluster + || !crate::raft::host::group_prefix(archive.hold.marker(), &fixed.root) + .map_err(|_| FleetCatalogError::Corrupt("receipt archive group prefix"))? { - return Err(FleetCatalogError::Corrupt("receipt archive cut")); - } - if fixed - .image - .as_ref() - .zip(archive.image.as_ref()) - .is_some_and(|(a, b)| !a.same_representation(b)) - { - return Err(FleetCatalogError::Corrupt( - "receipt image commitment domain", - )); - } - if fixed.image.is_some() && archive.image.is_none() - || archive.image.is_some() - && archive.image != fixed.image - && archive - .image_position - .is_none_or(|origin| origin.index <= fixed.applied.index) - { - return Err(FleetCatalogError::Corrupt("receipt archive publication")); + return Err(FleetCatalogError::Corrupt("receipt archive group cut")); } + let fixed_position = fixed.root.position(); let mut nodes = BTreeSet::new(); let mut members = BTreeSet::new(); let mut hosts = BTreeSet::new(); @@ -442,10 +375,10 @@ impl Entry { || survivor.publication_digest == [0; 32] || !nodes.insert(survivor.node) || !members.insert(survivor.member) - || survivor.applied.index < fixed.applied.index - || survivor.applied.term < fixed.applied.term - || survivor.applied.index == fixed.applied.index - && survivor.applied.term != fixed.applied.term + || survivor.applied.index < fixed_position.index.get() + || survivor.applied.term < fixed_position.term.get() + || survivor.applied.index == fixed_position.index.get() + && survivor.applied.term != fixed_position.term.get() { return Err(FleetCatalogError::Corrupt("receipt survivor")); } @@ -460,6 +393,50 @@ impl Entry { } } +// Index values stay within the original 4 KiB bound. Complete roots and +// persistent archive holds live in one independently hashed bounded object. +#[derive(Serialize, Deserialize)] +#[serde(tag = "state", rename_all = "snake_case", deny_unknown_fields)] +enum StoredEvidence { + Live { object: [u8; 32] }, + Cancelled { operation: [u8; 16] }, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct StoredEntry { + history: [u8; 32], + record: [u8; 32], + evidence: StoredEvidence, +} + +impl Entry { + fn stored(&self) -> Result<(Vec, Option<([u8; 32], Vec)>)> { + let (evidence, object) = match &self.evidence { + Evidence::Cancelled { operation } => ( + StoredEvidence::Cancelled { + operation: *operation, + }, + None, + ), + Evidence::Live(live) => { + let bytes = encode(LIVE_KIND, live, PAGE_LIMIT)?; + let digest = hash(&bytes); + ( + StoredEvidence::Live { object: digest }, + Some((digest, bytes)), + ) + } + }; + let stored = StoredEntry { + history: self.history, + record: self.record, + evidence, + }; + Ok((encode(ENTRY_KIND, &stored, ENTRY_LIMIT)?, object)) + } +} + /// An incremental immutable index, not an inventory or verification capability. #[derive(Clone)] pub(super) struct Book { @@ -493,12 +470,31 @@ impl Book { if entries.is_empty() || entries.len() > APPEND_LIMIT { return Err(FleetCatalogError::Limit); } - let mut index = self.index; - let mut count = self.count; + let mut prepared = Vec::with_capacity(entries.len()); + let mut bytes_total = 0usize; for entry in entries { entry.validate(&self.basis)?; - let key = hex::encode(entry.history); - let bytes = encode(ENTRY_KIND, entry, ENTRY_LIMIT)?; + let (stored, object) = entry.stored()?; + bytes_total = bytes_total + .checked_add(stored.len()) + .and_then(|n| n.checked_add(object.as_ref().map_or(0, |(_, bytes)| bytes.len()))) + .filter(|n| *n <= APPEND_LIMIT * ENTRY_LIMIT) + .ok_or(FleetCatalogError::Limit)?; + prepared.push((hex::encode(entry.history), stored, object)); + } + let mut index = self.index; + let mut count = self.count; + for (key, bytes, object) in prepared { + if let Some((digest, payload)) = object { + catalog + .inner + .create_exact( + &format!("receipt-live/{}", hex::encode(digest)), + &payload, + PAGE_LIMIT, + ) + .await?; + } if let Some(previous) = catalog.inner.lookup(index, &key).await? { if previous != bytes { return Err(FleetCatalogError::PayloadConflict); @@ -549,7 +545,33 @@ impl Candidate { let Some(bytes) = catalog.inner.lookup(self.index, key).await? else { return Ok(None); }; - let entry: Entry = decode(ENTRY_KIND, &bytes, ENTRY_LIMIT)?; + let stored: StoredEntry = decode(ENTRY_KIND, &bytes, ENTRY_LIMIT)?; + let evidence = match stored.evidence { + StoredEvidence::Cancelled { operation } => Evidence::Cancelled { operation }, + StoredEvidence::Live { object } => { + if object == [0; 32] { + return Err(FleetCatalogError::Corrupt("receipt live object")); + } + let (live, _) = catalog + .inner + .read::>( + &format!("receipt-live/{}", hex::encode(object)), + LIVE_KIND, + PAGE_LIMIT, + ) + .await? + .ok_or(FleetCatalogError::Corrupt("missing receipt live object"))?; + if hash(&encode(LIVE_KIND, &live, PAGE_LIMIT)?) != object { + return Err(FleetCatalogError::Corrupt("receipt live object digest")); + } + Evidence::Live(live) + } + }; + let entry = Entry { + history: stored.history, + record: stored.record, + evidence, + }; entry.validate(&self.basis)?; if hex::encode(entry.history) != key { return Err(FleetCatalogError::Corrupt("receipt entry key")); diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs new file mode 100644 index 000000000..107ecab7f --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/receipts_tests.rs @@ -0,0 +1,93 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Storage-shape tests; decoded receipt material cannot mint runtime proofs. +use super::*; +use crate::application::consensus::cell_routing_key; +use cellar_logic::{cell_group::*, consensus::*}; + +fn large_entry() -> Entry { + let history = GroupHistory { + cluster: ClusterId::new([240; 16]).unwrap(), + namespace: NamespaceId::new([241; 16]).unwrap(), + group: GroupId::new([242; 16]).unwrap(), + incarnation: IncarnationId::new([243; 16]).unwrap(), + }; + let route = GroupRoute { + history, + range: CellRange::new( + CellBound::Key(cell_routing_key( + history.namespace, + &CellId::new("\\\"".repeat(500)).unwrap(), + )), + CellBound::PositiveInfinity, + ) + .unwrap(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [244; 32], + schema_digest: [245; 32], + }; + let root = GroupManifest::genesis(route).unwrap(); + let hold:CellGroupArchiveHold=serde_json::from_value(serde_json::json!({"registry":vec![246;32],"id":3,"allocation":vec![247;32],"root":{"object":{"registry":vec![246;32],"generation":2,"digest":vec![248;32],"length":128},"marker":root}})).unwrap(); + hold.check_reference().unwrap(); + let survivors = std::array::from_fn(|i| Survivor { + node: [i as u8 + 1; 16], + disk: [i as u8 + 10; 16], + engine: [i as u8 + 20; 16], + process: [i as u8 + 30; 16], + physical_host: [i as u8 + 40; 16], + epoch: 1, + member: i as u64 + 1, + applied: root.position().into(), + publication_digest: [249; 32], + }); + Entry { + history: [250; 32], + record: [251; 32], + evidence: Evidence::Live(Box::new(LiveEvidence { + fixed: Publication { + root, + configuration_digest: [252; 32], + }, + archive: Archive { + hold, + configuration_digest: [252; 32], + }, + survivors, + })), + } +} + +#[test] +fn complete_group_receipt_keeps_original_index_row_bound() { + let entry = large_entry(); + assert!(encode(ENTRY_KIND, &entry, ENTRY_LIMIT).is_err()); + let (row, object) = entry.stored().unwrap(); + assert!(row.len() <= ENTRY_LIMIT); + let (digest, payload) = object.unwrap(); + assert_eq!(digest, hash(&payload)); + assert!(payload.len() <= PAGE_LIMIT); + let stored: StoredEntry = decode(ENTRY_KIND, &row, ENTRY_LIMIT).unwrap(); + assert!(matches!(stored.evidence,StoredEvidence::Live {object} if object==digest)); + let live: Box = decode(LIVE_KIND, &payload, PAGE_LIMIT).unwrap(); + assert_eq!(Evidence::Live(live), entry.evidence); + assert!(row.len() + payload.len() <= APPEND_LIMIT * ENTRY_LIMIT); +} + +#[test] +fn cancelled_receipt_uses_no_external_live_payload() { + let entry = Entry { + history: [1; 32], + record: [2; 32], + evidence: Evidence::Cancelled { operation: [3; 16] }, + }; + let (row, object) = entry.stored().unwrap(); + assert!(object.is_none()); + let stored: StoredEntry = decode(ENTRY_KIND, &row, ENTRY_LIMIT).unwrap(); + assert!(matches!(stored.evidence,StoredEvidence::Cancelled {operation} if operation==[3;16])); + assert!(decode::( + ENTRY_KIND, + &encode(ENTRY_KIND, &large_entry(), PAGE_LIMIT).unwrap(), + ENTRY_LIMIT + ) + .is_err()); +} diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs index 7d830fa76..23ee4ea22 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/survivors.rs @@ -137,9 +137,7 @@ pub(super) async fn refresh_terminal( let observations: [RaftPeerFleetMembershipObservation; 3] = observations .try_into() .map_err(|_| FleetCatalogError::Corrupt("terminal survivor count"))?; - let current = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let current = observations[0].published_cut(); let fixed = live.fixed.restore(current)?; catalog .run(deadline.remaining()?, async { @@ -212,9 +210,7 @@ pub(super) async fn verify( let proof = observed.membership(); let replica = proof.replica(); let node = observed.node(); - let cut = observed - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let cut = observed.published_cut(); let record = records .iter() .find(|record| record.node == replica.node.bytes()) @@ -295,7 +291,7 @@ pub(super) async fn verify( epoch: auth.epoch(), member: replica.member.get(), applied: proof.applied().into(), - publication_digest: hash(&encode(57, &Publication::from(cut), 4096)?), + publication_digest: hash(&encode(57, &Publication::from(cut), PAGE_LIMIT)?), }); } if snapshot.replica_separation() == FleetReplicaSeparation::PhysicalHosts && hosts.len() != 3 { diff --git a/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs b/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs index a1a5c33d9..a9d7414e9 100644 --- a/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs +++ b/crates/cellar/durability/raft/fleet_catalog/finalization/terminal.rs @@ -388,7 +388,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.complete_evacuation_attempt(snapshot, request, verification, namespaces, timeout) @@ -401,7 +401,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { match self @@ -437,7 +437,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetIsolatedRemovalRequest, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let completed = self @@ -466,7 +466,7 @@ impl FleetCatalog { request: &FleetRequestIdentity, proof: RemovalProof, verification: Option<&mut FleetEvacuationVerification>, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/histories.rs b/crates/cellar/durability/raft/fleet_catalog/histories.rs index 5a126e587..899629e93 100644 --- a/crates/cellar/durability/raft/fleet_catalog/histories.rs +++ b/crates/cellar/durability/raft/fleet_catalog/histories.rs @@ -3,16 +3,38 @@ //! empty runtime scan proves that those references can be released. use super::*; use crate::raft::catalog::{ - BirthDisposition, BirthProposal, CatalogError, GroupBirth, InitialMember, ManagedBirthOwner, - NamespaceCatalog, NamespaceConfig, VoterPolicy, + CatalogError, InitialMember, ManagedBirthOwner, NamespaceCatalog, NamespaceConfig, VoterPolicy, +}; +use crate::raft::cell_groups::{ + CellGroupBirth, CellGroupBirthRequest, CellGroupBirthSpec, RangeDirectory, }; use crate::raft::member_state::FleetGroupBinding; -use cellar_logic::consensus::{GroupScope, MemberId, NodeId, ProposalId}; +use cellar_logic::{ + cell_group::{CellBound, CellId, CellRange, GroupHistory, RouteEpoch}, + consensus::{MemberId, NodeId, ProposalId}, +}; use serde::Deserialize; use std::collections::BTreeSet; const HISTORY_KIND: u8 = 48; +#[cfg(test)] +#[path = "histories_tests.rs"] +mod tests; + +fn bound_key(bound: &CellBound) -> Option { + match bound { + CellBound::Key(key) => Some(key.cell().as_str().into()), + CellBound::NegativeInfinity | CellBound::PositiveInfinity => None, + } +} + +fn required_bound<'de, D: serde::Deserializer<'de>>( + decoder: D, +) -> std::result::Result, D::Error> { + Option::deserialize(decoder) +} + #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] pub enum FleetHistoryState { Reserved, @@ -43,6 +65,12 @@ pub struct FleetHistory { pub(super) schema_digest: [u8; 32], pub(super) namespace_root: [u8; 32], pub(super) birth_digest: [u8; 32], + #[serde(deserialize_with = "required_bound")] + lower: Option, + #[serde(deserialize_with = "required_bound")] + upper: Option, + epoch: u64, + creation: [u8; 16], pub(super) admission: [u8; 16], pub(super) operation: [u8; 16], pub(super) voters: Vec, @@ -52,21 +80,61 @@ pub struct FleetHistory { } impl FleetHistory { + pub(super) fn cancelled_group( + binding: FleetGroupBinding, + directory: &RangeDirectory, + request: &CellGroupBirthRequest, + admission: ProposalId, + operation: ProposalId, + cancellation: ProposalId, + ) -> Result { + let decoded = directory + .decode_group_birth(request.encode()) + .map_err(catalog_error)?; + if !binding.matches_scope(&decoded.route().history) { + return Err(FleetCatalogError::Conflict); + } + let value = Self { + binding, + schema_digest: directory.namespace_config().schema_digest, + namespace_root: directory.namespace().root_digest(), + birth_digest: decoded.route().birth_digest, + lower: bound_key(decoded.route().range.lower()), + upper: bound_key(decoded.route().range.upper()), + epoch: decoded.route().epoch.get(), + creation: decoded.creation().bytes(), + admission: admission.bytes(), + operation: operation.bytes(), + voters: decoded.voters().iter().copied().map(Member::from).collect(), + learners: decoded + .learners() + .iter() + .copied() + .map(Member::from) + .collect(), + state: FleetHistoryState::Cancelled { + operation: cancellation.bytes(), + }, + origin: Origin::Observed, + }; + value.validate()?; + Ok(value) + } + /// An exact observed birth is import material, not proof that the namespace /// inventory is complete or that legacy creation has been quiesced. pub fn observed( binding: FleetGroupBinding, - namespace: &NamespaceCatalog, - birth: &GroupBirth, + directory: &RangeDirectory, + birth: &CellGroupBirth, admission: ProposalId, operation: ProposalId, ) -> Result { - namespace.check_birth(birth).map_err(catalog_error)?; - if !binding.matches_scope(birth.scope()) - || birth.managed_owner().is_some_and(|owner| { - owner.fleet != binding.fleet() || owner.catalog != binding.catalog() - }) - { + let namespace = directory.namespace(); + let decoded = directory + .decode_group_birth(birth.encode()) + .map_err(catalog_error)?; + if !binding.matches_scope(&birth.route().history) || decoded.route() != birth.route() { return Err(FleetCatalogError::Conflict); } if namespace @@ -83,7 +151,11 @@ impl FleetHistory { binding, schema_digest: namespace.config().schema_digest, namespace_root: namespace.root_digest(), - birth_digest: birth.digest(), + birth_digest: birth.route().birth_digest, + lower: bound_key(birth.route().range.lower()), + upper: bound_key(birth.route().range.upper()), + epoch: birth.route().epoch.get(), + creation: birth.creation().bytes(), admission: admission.bytes(), operation: operation.bytes(), voters: birth.voters().iter().copied().map(Member::from).collect(), @@ -127,6 +199,7 @@ impl FleetHistory { } pub(super) fn validate(&self) -> Result<()> { + self.initial_placement()?; self.binding .scope() .map_err(|_| FleetCatalogError::Corrupt("history binding"))?; @@ -163,12 +236,39 @@ impl FleetHistory { } /// Recorded genesis references, never a current-membership or absence proof. - pub fn initial_placement(&self) -> Result { - Ok(BirthProposal { - scope: self - .binding - .scope() - .map_err(|_| FleetCatalogError::Corrupt("history scope"))?, + pub fn initial_placement(&self) -> Result { + let history = self + .binding + .scope() + .map_err(|_| FleetCatalogError::Corrupt("history scope"))?; + let key = |name: &str| { + CellId::new(name) + .map(|cell| { + crate::application::consensus::cell_routing_key(history.namespace, &cell) + }) + .map(CellBound::Key) + .map_err(|_| FleetCatalogError::Corrupt("history range")) + }; + let range = CellRange::new( + self.lower + .as_deref() + .map(key) + .transpose()? + .unwrap_or(CellBound::NegativeInfinity), + self.upper + .as_deref() + .map(key) + .transpose()? + .unwrap_or(CellBound::PositiveInfinity), + ) + .map_err(|_| FleetCatalogError::Corrupt("history range"))?; + Ok(CellGroupBirthSpec { + history, + range, + epoch: RouteEpoch::new(self.epoch) + .map_err(|_| FleetCatalogError::Corrupt("history route epoch"))?, + creation: ProposalId::new(self.creation) + .map_err(|_| FleetCatalogError::Corrupt("history creation"))?, voters: self .voters .iter() @@ -182,15 +282,26 @@ impl FleetHistory { }) } + // Multiple admitted candidates can share a range/epoch before the directory + // selects one. Every losing physical identity must remain inventoried too. pub(super) fn name_key(&self) -> Result { let scope = self .binding .scope() .map_err(|_| FleetCatalogError::Corrupt("history scope"))?; Ok(format!( - "birth/{}/{}", + "range/{}/{}", hex::encode(scope.namespace.bytes()), - hex::encode(hash(scope.cell.as_bytes())) + hex::encode(hash( + &serde_json::to_vec(&( + &self.lower, + &self.upper, + self.epoch, + scope.group.bytes(), + scope.incarnation.bytes(), + )) + .map_err(|_| FleetCatalogError::Corrupt("history range key"))? + )) )) } @@ -202,17 +313,34 @@ impl FleetHistory { Ok(format!("group/{}", hex::encode(scope.group.bytes()))) } - pub(crate) fn check_birth(&self, namespace: &NamespaceCatalog, birth: &GroupBirth) -> bool { - namespace.check_birth(birth).is_ok() + pub(crate) fn check_birth(&self, directory: &RangeDirectory, birth: &CellGroupBirth) -> bool { + let namespace = directory.namespace(); + directory.decode_group_birth(birth.encode()).is_ok() && namespace .config() .managed_fleet .as_ref() .is_none_or(|owner| owner.fleet == self.binding.fleet()) - && self.binding.matches_scope(birth.scope()) + && self.binding.matches_scope(&birth.route().history) && self.namespace_root == namespace.root_digest() && self.schema_digest == namespace.config().schema_digest - && self.birth_digest == birth.digest() + && self.birth_digest == birth.route().birth_digest + && self.lower == bound_key(birth.route().range.lower()) + && self.upper == bound_key(birth.route().range.upper()) + && self.epoch == birth.route().epoch.get() + && self.creation == birth.creation().bytes() + && self.voters.len() == birth.voters().len() + && self.learners.len() == birth.learners().len() + && self + .voters + .iter() + .zip(birth.voters()) + .all(|(a, b)| a.member == b.member.get() && a.node == b.node.bytes()) + && self + .learners + .iter() + .zip(birth.learners()) + .all(|(a, b)| a.member == b.member.get() && a.node == b.node.bytes()) } } @@ -241,20 +369,20 @@ impl Member { #[derive(Clone, Debug)] pub struct FleetBirthAuthorization { history: FleetHistory, - proposal: BirthProposal, + proposal: CellGroupBirthRequest, owner: ManagedBirthOwner, } impl FleetBirthAuthorization { - pub(crate) fn check_birth(&self, namespace: &NamespaceCatalog, birth: &GroupBirth) -> bool { - self.history.check_birth(namespace, birth) + pub(crate) fn check_birth(&self, directory: &RangeDirectory, birth: &CellGroupBirth) -> bool { + self.history.check_birth(directory, birth) } - pub fn scope(&self) -> &GroupScope { - &self.proposal.scope + pub fn scope(&self) -> &GroupHistory { + &self.proposal.route().history } - pub(crate) fn proposal(&self) -> BirthProposal { + pub(crate) fn proposal(&self) -> CellGroupBirthRequest { self.proposal.clone() } @@ -330,25 +458,22 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - birth: &GroupBirth, + directory: &RangeDirectory, + birth: &CellGroupBirth, + class: &str, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; - if let Some(owner) = birth.managed_owner() { - if self.birth_owner()? != *owner { - return Err(FleetCatalogError::Conflict); - } - } + let namespace = directory.namespace(); if namespace.config().managed_fleet.is_some() && self.bind_namespace(namespace.config().clone())? != *namespace.config() { return Err(FleetCatalogError::Conflict); } let history = FleetHistory::observed( - self.binding(birth.scope())?, - namespace, + self.binding(&birth.route().history)?, + directory, birth, ProposalId::new(request.id).map_err(|_| FleetCatalogError::Invalid("admission"))?, ProposalId::new(request.operation) @@ -356,7 +481,7 @@ impl FleetCatalog { )?; self.run( timeout, - self.insert_history(snapshot, request, &history, namespace, false), + self.insert_history(snapshot, request, &history, directory, class, false), ) .await } @@ -365,12 +490,14 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - proposal: BirthProposal, + directory: &RangeDirectory, + proposal: CellGroupBirthSpec, + class: &str, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; + let namespace = directory.namespace(); if namespace.config().cluster.bytes() != self.inner.cluster || namespace.config().voters != VoterPolicy::ThreeVoters || proposal.voters.len() != 3 @@ -379,25 +506,34 @@ impl FleetCatalog { return Err(FleetCatalogError::Invalid("managed initial placement")); } self.run(timeout, async { - let owner = self.managed_owner_for_birth(snapshot, namespace).await?; - let (proposal, digest) = namespace - .prepare_birth_with_owner(proposal, Some(&owner)) + let owner = self.managed_owner_for_birth(snapshot, directory).await?; + let proposal = directory + .prepare_fleet_group_birth(proposal, &owner) .map_err(catalog_error)?; let history = FleetHistory { - binding: self.binding(&proposal.scope)?, + binding: self.binding(&proposal.route().history)?, schema_digest: namespace.config().schema_digest, namespace_root: namespace.root_digest(), - birth_digest: digest, + birth_digest: proposal.route().birth_digest, + lower: bound_key(proposal.route().range.lower()), + upper: bound_key(proposal.route().range.upper()), + epoch: proposal.route().epoch.get(), + creation: proposal.creation().bytes(), admission: request.id, operation: request.operation, - voters: proposal.voters.iter().copied().map(Member::from).collect(), + voters: proposal + .voters() + .iter() + .copied() + .map(Member::from) + .collect(), learners: vec![], state: FleetHistoryState::Reserved, origin: Origin::Managed, }; history.validate()?; let commit = self - .insert_history(snapshot, request, &history, namespace, true) + .insert_history(snapshot, request, &history, directory, class, true) .await?; Ok(FleetBirthReservation { commit, @@ -418,11 +554,12 @@ impl FleetCatalog { pub async fn recover_birth_reservation( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, - scope: &GroupScope, + directory: &RangeDirectory, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; + let namespace = directory.namespace(); let binding = self.binding(scope)?; let key = binding .key() @@ -491,10 +628,11 @@ impl FleetCatalog { if applied.is_none() || applied != request.expected_revision.checked_add(1) { return Err(FleetCatalogError::Corrupt("unadmitted birth reservation")); } - let owner = self.managed_owner_for_birth(snapshot, namespace).await?; - let (proposal, digest) = namespace - .prepare_birth_with_owner(history.initial_placement()?, Some(&owner)) + let owner = self.managed_owner_for_birth(snapshot, directory).await?; + let proposal = directory + .prepare_fleet_group_birth(history.initial_placement()?, &owner) .map_err(catalog_error)?; + let digest = proposal.route().birth_digest; let authorization = FleetBirthAuthorization { history, proposal, @@ -517,12 +655,13 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, authorization: &FleetBirthAuthorization, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; + let namespace = directory.namespace(); authorization .check_namespace(namespace) .map_err(catalog_error)?; @@ -530,27 +669,18 @@ impl FleetCatalog { return Err(FleetCatalogError::Conflict); } self.run(timeout, async { - if self.managed_owner_for_birth(snapshot, namespace).await? != authorization.owner { + if self.managed_owner_for_birth(snapshot, directory).await? != authorization.owner { return Err(FleetCatalogError::Conflict); } let mut history = authorization.history.clone(); - match namespace - .birth_disposition(&authorization.scope().cell) + let birth = directory + .materialize_fleet_group_birth(authorization) .await - .map_err(catalog_error)? - { - Some(BirthDisposition::Live(birth)) if history.check_birth(namespace, &birth) => { - history.state = FleetHistoryState::Live - } - Some(BirthDisposition::Cancelled { scope, operation }) - if scope == *authorization.scope() => - { - history.state = FleetHistoryState::Cancelled { - operation: operation.bytes(), - } - } - _ => return Err(FleetCatalogError::Conflict), + .map_err(catalog_error)?; + if !history.check_birth(directory, &birth) { + return Err(FleetCatalogError::Conflict); } + history.state = FleetHistoryState::Live; let key = history .binding .key() @@ -585,7 +715,7 @@ impl FleetCatalog { pub async fn history( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -638,9 +768,11 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetRequestIdentity, history: &FleetHistory, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, + class: &str, placement: bool, ) -> Result { + let namespace = directory.namespace(); let key = history .binding .key() @@ -681,11 +813,6 @@ impl FleetCatalog { .await? .is_none() { - let class = scope - .cell - .split_once(':') - .ok_or(FleetCatalogError::Invalid("history class"))? - .0; self.require_namespace_declaration(snapshot, namespace, Some(class)) .await?; if current.token != snapshot.token || request.expected_revision != snapshot.revision() { @@ -739,8 +866,12 @@ impl FleetCatalog { // This claim grants no birth authority. Confirming it before // the root CAS closes races with other namespace catalogs that // share the global group-ID key, including unmanaged imports. - namespace - .reserve_group_identity(&scope) + let owner = self.birth_owner()?; + let proposal = directory + .prepare_fleet_group_birth(history.initial_placement()?, &owner) + .map_err(catalog_error)?; + directory + .reserve_fleet_group_identity(&proposal, &owner) .await .map_err(catalog_error)?; } @@ -782,6 +913,7 @@ pub(super) fn catalog_error(error: CatalogError) -> FleetCatalogError { CatalogError::Limit => FleetCatalogError::Limit, CatalogError::Unconfirmed => FleetCatalogError::Unconfirmed, CatalogError::Storage(error) => FleetCatalogError::Storage(error.into()), + CatalogError::FleetAuthority(error) => *error, _ => FleetCatalogError::Conflict, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs b/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs new file mode 100644 index 000000000..27b4347be --- /dev/null +++ b/crates/cellar/durability/raft/fleet_catalog/histories_tests.rs @@ -0,0 +1,104 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +use super::*; +use cellar_logic::consensus::{ClusterId, GroupId, NamespaceId}; + +#[test] +fn whole_group_history_retains_range_and_original_creation() { + let history = fixture(); + let restored = FleetHistory::decode(&history.encode().unwrap()).unwrap(); + let placement = restored.initial_placement().unwrap(); + assert_eq!(placement.history, history.binding.scope().unwrap()); + assert_eq!(placement.creation.bytes(), history.creation); + assert_eq!(placement.epoch.get(), 7); + assert_eq!(placement.voters.len(), 3); + assert_eq!( + bound_key(placement.range.lower()).as_deref(), + history.lower.as_deref() + ); + assert_eq!( + bound_key(placement.range.upper()).as_deref(), + history.upper.as_deref() + ); + let mut changed = restored; + changed.upper = Some("Counter:zz".into()); + assert_ne!(history.name_key().unwrap(), changed.name_key().unwrap()); + assert_eq!(history.group_key().unwrap(), changed.group_key().unwrap()); +} + +#[test] +fn pending_groups_with_the_same_range_keep_distinct_inventory_keys() { + let first = fixture(); + let mut second = first.clone(); + let mut history = second.binding.scope().unwrap(); + history.group = GroupId::new([13; 16]).unwrap(); + history.incarnation = IncarnationId::new([14; 16]).unwrap(); + second.binding = FleetGroupBinding::new( + IncarnationId::new(first.binding.fleet()).unwrap(), + &history, + first.binding.catalog(), + ) + .unwrap(); + assert_eq!(first.lower, second.lower); + assert_eq!(first.upper, second.upper); + assert_eq!(first.epoch, second.epoch); + assert_ne!(first.name_key().unwrap(), second.name_key().unwrap()); + assert_ne!(first.group_key().unwrap(), second.group_key().unwrap()); +} + +#[test] +fn history_refuses_missing_bounds_and_invalid_ranges() { + let history = fixture(); + for field in ["lower", "upper", "epoch", "creation"] { + let mut value = serde_json::to_value(&history).unwrap(); + value.as_object_mut().unwrap().remove(field); + let bytes = encode(HISTORY_KIND, &value, 4096).unwrap(); + assert!( + FleetHistory::decode(&bytes).is_err(), + "accepted missing {field}" + ); + } + let mut inverted = history.clone(); + inverted.lower = history.upper; + inverted.upper = history.lower; + assert!(inverted.encode().is_err()); +} + +fn fixture() -> FleetHistory { + let namespace = NamespaceId::new([2; 16]).unwrap(); + let mut names = ["Counter:a", "Counter:z"]; + names.sort_by_key(|name| { + crate::application::consensus::cell_routing_key(namespace, &CellId::new(name).unwrap()) + }); + FleetHistory { + binding: FleetGroupBinding::new( + IncarnationId::new([5; 16]).unwrap(), + &GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace, + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), + }, + [6; 32], + ) + .unwrap(), + schema_digest: [7; 32], + namespace_root: [8; 32], + birth_digest: [9; 32], + lower: Some(names[0].into()), + upper: Some(names[1].into()), + epoch: 7, + creation: [10; 16], + admission: [11; 16], + operation: [12; 16], + voters: (1..=3) + .map(|member| Member { + member, + node: [member as u8; 16], + }) + .collect(), + learners: vec![], + state: FleetHistoryState::Reserved, + origin: Origin::Managed, + } +} diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/README.md b/crates/cellar/durability/raft/fleet_catalog/imports/README.md index 1340cc905..381cee1dd 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/README.md +++ b/crates/cellar/durability/raft/fleet_catalog/imports/README.md @@ -1,24 +1,34 @@ # Namespace import runtime -This component imports one authoritative frozen namespace admission set. Fleet +This component imports one authoritative frozen physical-group admission set. Fleet capacity activation, global namespace coverage, maintenance/removal certificates, automatic reconciliation and operator surfaces remain separate delivery work. Keep automatic fleet management default-off until its qualification gates pass. -The namespace's immutable configuration and schema stay fixed. The mutable birth -inventory is claimed by one exact fleet/catalog scope, then adopted only after -every frozen intent is settled and every live group has committed enrollment. -Business writes continue through their existing Raft acknowledgement path. +The namespace's immutable configuration and schema stay fixed. Call +`RangeDirectory::freeze_group_births` with a persisted operation identity. The +namespace admission CAS first closes new group intents and arbitrates against +disaster recovery. The directory-head CAS then locks topology; a writer holding +an earlier expected head loses. The exact locked directory is retained by digest. +Retrying that operation recovers the same cut after a lost reply or reopen. + +The cut includes every admitted physical request, including never-served, +unselected reservations. An unadmitted global identity claim remains permanently +reserved and grants no birth or voter permission. Fleet import claims this exact +cut under one catalog scope, then adopts only after every intent is settled and +every live group has committed enrollment. Business writes continue through their +existing Raft acknowledgement path. This does not import predecessor per-cell +inventories or create a compatibility path. ## Durable steps | Root phase | Effect and admission boundary | | --- | --- | | `Preparing` | `begin_namespace_import` roots the exact cut, operation and catalog owner. The next step conditionally claims that same frozen namespace. A competing catalog can retain a harmless Preparing intent but cannot replace the winner's claim. | -| `Inventory` | Traverse the rooted cursor, one canonical intent at a time. Reconcile the immutable birth decision; conditionally cancel a still-pending decision. Root the history and permanent name/group references with progress in one CAS. Existing exact histories retain their original admission identity. | -| `Coverage` | Traverse the same cut again. Cancelled histories retain their tombstones. Each live history requires its own opaque safe-ReadIndex observation with the exact catalog binding, birth/schema, stable RF3 configuration, committed guard and no active repair. Verify current voter enrollment, engine pins and permanent member/disk claims. | -| `Handoff` | The last coverage CAS admits irreversible handoff. Before changing namespace ownership, read and validate the complete retained witness set against the canonical cut, actual histories and immutable decisions/claims. Unresolved namespace admissions block handoff. | -| `Complete` | After namespace adoption is confirmed, CAS the terminal root state. Only this state plus the exact namespace adoption marker permits new managed birth reservations in the imported namespace. | +| `Inventory` | Traverse the rooted cursor, one canonical intent at a time. Reconcile the immutable birth decision; conditionally cancel an unselected pending group at the same exact key that materialization uses. A selected lazy birth stays live. Root the history and permanent range/group references with progress in one CAS. Existing exact histories retain their original admission identity. | +| `Coverage` | Traverse the same cut again. Cancelled histories retain their tombstones. Each live physical history requires its own opaque `CellGroupQuorumMembership` safe-ReadIndex observation with the exact catalog binding, birth/schema, stable RF3 configuration, committed guard and no active repair. Verify current voter enrollment, engine pins and permanent member/disk claims. | +| `Handoff` | The last coverage CAS admits irreversible handoff. Before changing namespace ownership, read and validate the complete retained witness set against the canonical cut, actual histories and immutable decisions/claims. Independently traverse every frozen selected range and require its exact admission and coverage witness. Unresolved namespace admissions block handoff. | +| `Complete` | After namespace adoption is confirmed, unlock topology with another CAS, then CAS the terminal fleet-root state. Both head changes advance revision, so an old expected head cannot become current again. Only this state plus the exact namespace adoption marker permits new managed birth reservations in the imported namespace. | Progress comes from the fleet root, never a caller-provided manifest, cursor or reachable-node scan. Every turn is deadline-bound and owns one bounded catalog @@ -48,16 +58,19 @@ admission. An unadmitted stale candidate cannot overwrite a later root revision. Imported original births retain their original unmanaged birth identity. Their enrollment is a committed Raft configuration entry, which intact original engines -can replay during recovery. New births after adoption carry an explicit managed -owner in their canonical birth record. They require an inventoried reservation -and initialize with their fleet guard in the first engine sync, even though the -namespace's immutable `managed_fleet` configuration remains absent. +can replay during recovery. New births after adoption require a rooted fleet reservation and the exact +adopted catalog owner; ordinary birth and bootstrap calls refuse that namespace. +They initialize with their configuration guard in the first engine sync, even +though the immutable `managed_fleet` configuration remains absent. Observed +histories may only resume an existing matching local guard: import does not turn +a missing disk into permission to recreate genesis. A selected lazy group without +an enrolled quorum therefore blocks completion; timeout does not cancel it. Group bindings now include the catalog-scope fingerprint as well as fleet and -cell-history identity. Use `FleetCatalog::group_binding` for that descriptor. +physical group/incarnation identity. Use `FleetCatalog::group_binding` for that descriptor. Constructing a descriptor grants no authority. Changing catalog prefixes cannot reuse a cached authorization or permit to mutate the already-enrolled group. -The experimental protocol hard-cuts to `/fleet-quorum-1/fleet-cuts-1/empty-archives-1/archive-cuts-1/fleet-finalization-1`; earlier namespace, +The experimental protocol includes `/cell-group-fleet-1/group-import-1`; earlier namespace, fleet and group records require fresh experimental prefixes and matching binaries. Numeric format markers remain 1, with no migration or compatibility reader. diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs b/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs index 15bdb4624..08300e517 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/coverage.rs @@ -1,29 +1,30 @@ use super::*; -use crate::raft::catalog::BirthDisposition; +use crate::raft::cell_groups::CellGroupBirthDisposition; +use crate::raft::host::FleetMembershipConfiguration; use crate::raft::member_state::FleetMembershipPermit; use records::{Decision, Voter, Witness}; use std::collections::BTreeSet; impl FleetCatalog { /// Confirm that every retained proof and history is still readable before - /// the irreversible namespace effect. Missing pages are never empty sets. + /// the irreversible directory effect. Missing pages are never empty sets. pub(super) async fn verify_complete_import( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &Record, ) -> Result<()> { - self.verify_import_coverage(snapshot, namespace, record, true) + self.verify_import_coverage(snapshot, directory, record, true) .await } pub(in crate::raft::fleet_catalog) async fn verify_imported_inventory( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result<()> { let record = self - .import_record(snapshot, namespace) + .import_record(snapshot, directory) .await .map_err(|e| match e { FleetCatalogError::Conflict => FleetCatalogError::InventoryIncomplete, @@ -32,21 +33,63 @@ impl FleetCatalog { if record.phase != records::FleetImportPhase::Complete || record.stopped { return Err(FleetCatalogError::InventoryIncomplete); } - namespace - .check_birth_import(&record.cut, &self.birth_owner()?, true) + directory + .check_group_import(&record.cut, &self.birth_owner()?, true) .await .map_err(catalog_error)?; - self.verify_import_coverage(snapshot, namespace, &record, false) + self.verify_import_coverage(snapshot, directory, &record, false) .await } async fn verify_import_coverage( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &Record, handoff: bool, ) -> Result<()> { + directory + .check_group_import(&record.cut, &self.birth_owner()?, !handoff) + .await + .map_err(catalog_error)?; + let frozen = directory + .import_directory_view(&record.cut) + .await + .map_err(catalog_error)?; + let mut cursor = None; + let mut selected = 0u64; + loop { + let page = directory + .group_page(&frozen, cursor.as_ref(), 16) + .await + .map_err(catalog_error)?; + for route in page.routes { + selected = selected + .checked_add(1) + .filter(|n| *n <= record.cut.count) + .ok_or(FleetCatalogError::Corrupt( + "untracked selected import group", + ))?; + let key = hex::encode(hash(&route.history.group.bytes())); + let admitted = directory + .namespace() + .cell_group_import_entry(directory, &record.cut, &self.birth_owner()?, &key) + .await + .map_err(catalog_error)?; + if admitted.route() != &route + || self.inner.lookup(record.coverage, &key).await?.is_none() + { + return Err(FleetCatalogError::Corrupt("selected import group coverage")); + } + } + cursor = page.next; + if cursor.is_none() { + break; + } + } + if selected != u64::from(frozen.groups()) { + return Err(FleetCatalogError::Corrupt("missing selected import group")); + } let mut after = None; let mut seen = 0u64; loop { @@ -60,18 +103,24 @@ impl FleetCatalog { .filter(|n| *n <= record.cut.count) .ok_or(FleetCatalogError::Corrupt("extra import coverage"))?; let witness: Witness = decode(records::WITNESS_KIND, &value.value, 4096)?; - let entry = namespace - .import_birth_entry(&record.cut, &self.birth_owner()?, &value.key) + let entry = directory + .namespace() + .cell_group_import_entry( + directory, + &record.cut, + &self.birth_owner()?, + &value.key, + ) .await .map_err(catalog_error)?; - let scope = &entry.proposal().scope; + let scope = &entry.route().history; let binding = self.binding(scope)?; let history_key = binding .key() .map_err(|_| FleetCatalogError::Corrupt("import witness binding"))?; let history = self.history_inner(snapshot, &history_key).await?; if witness.history != history_key - || witness.birth != entry.expected_digest() + || witness.birth != entry.route().birth_digest || history.birth_digest != witness.birth || history.namespace_root != record.cut.root || history.schema_digest != record.schema @@ -80,15 +129,15 @@ impl FleetCatalog { } match ( witness.decision, - namespace - .birth_disposition(&scope.cell) + directory + .group_birth_disposition(&entry) .await .map_err(catalog_error)?, ) { ( Decision::Cancelled { operation }, - Some(BirthDisposition::Cancelled { - scope: actual, + Some(CellGroupBirthDisposition::Cancelled { + history: actual, operation: op, }), ) => { @@ -113,9 +162,9 @@ impl FleetCatalog { phase, voters, }, - Some(BirthDisposition::Live(birth)), + Some(CellGroupBirthDisposition::Live(birth)), ) if history.state == FleetHistoryState::Live - && history.check_birth(namespace, &birth) => + && history.check_birth(directory, &birth) => { let valid_id = |id| cellar_logic::consensus::IncarnationId::new(id).is_ok(); if !valid_id(node) @@ -164,19 +213,18 @@ impl FleetCatalog { { return Err(FleetCatalogError::Conflict); } - namespace + directory + .namespace() .verify_fleet_storage_pin(&host) .await .map_err(catalog_error)?; - namespace - .verify_import_member_claim( + directory + .verify_group_import_member_claim( &birth, cellar_logic::consensus::MemberId::new(voter.member).map_err( |_| FleetCatalogError::Corrupt("import witness member"), )?, - cellar_logic::consensus::DiskLineage::new(voter.disk).map_err( - |_| FleetCatalogError::Corrupt("import witness disk"), - )?, + &host, ) .await .map_err(catalog_error)?; @@ -200,7 +248,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Corrupt("missing import coverage")); } if handoff { - self.no_namespace_admissions(snapshot, namespace.config().namespace) + self.no_namespace_admissions(snapshot, directory.namespace_config().namespace) .await } else { Ok(()) @@ -210,61 +258,77 @@ impl FleetCatalog { pub(super) async fn import_coverage_step( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &mut Record, - observation: Option<&FleetMembershipObservation>, + observation: Option<&CellGroupQuorumMembership>, ) -> Result> { - let step = namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + directory + .check_group_import(&record.cut, &self.birth_owner()?, false) + .await + .map_err(catalog_error)?; + let step = directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) .await .map_err(catalog_error)?; if let Some(entry) = step.entry { - let proposal = entry.proposal(); - let binding = self.binding(&proposal.scope)?; + let proposal = &entry; + let binding = self.binding(&proposal.route().history)?; let history_key = binding .key() .map_err(|_| FleetCatalogError::Corrupt("import binding key"))?; let history = self.history_inner(snapshot, &history_key).await?; if history.namespace_root != record.cut.root || history.schema_digest != record.schema - || history.birth_digest != entry.expected_digest() + || history.birth_digest != entry.route().birth_digest { return Err(FleetCatalogError::Conflict); } - let decision = match namespace - .birth_disposition(&proposal.scope.cell) + let decision = match directory + .group_birth_disposition(proposal) .await .map_err(catalog_error)? { - Some(BirthDisposition::Cancelled { scope, operation }) - if scope == proposal.scope - && history.state - == (FleetHistoryState::Cancelled { - operation: operation.bytes(), - }) - && observation.is_none() => + Some(CellGroupBirthDisposition::Cancelled { + history: scope, + operation, + }) if scope == proposal.route().history + && history.state + == (FleetHistoryState::Cancelled { + operation: operation.bytes(), + }) + && observation.is_none() => { Decision::Cancelled { operation: operation.bytes(), } } - Some(BirthDisposition::Live(birth)) + Some(CellGroupBirthDisposition::Live(birth)) if history.state == FleetHistoryState::Live - && history.check_birth(namespace, &birth) => + && history.check_birth(directory, &birth) => { let observed = observation.ok_or(FleetCatalogError::Invalid( "missing live import observation", ))?; - let status = observed.status(); - let conf = observed.configuration(); - if observed.replica().scope != proposal.scope + let status = observed + .membership() + .fleet_status() + .ok_or(FleetCatalogError::Conflict)?; + let conf = FleetMembershipConfiguration::capture(observed.configuration()) + .map_err(|_| FleetCatalogError::Corrupt("import configuration"))?; + if observed.replica().scope != proposal.route().history || status.binding != binding - || observed.birth_digest() != history.birth_digest - || observed.schema_digest() != record.schema - || observed.barrier().get() == 0 - || observed.applied().index < observed.barrier() - || observed.membership_revision() > observed.applied().index - || observed.membership_revision() < status.changed_at + || observed.root().route().birth_digest != history.birth_digest + || observed.root().route().schema_digest != record.schema + || observed.barrier().index.get() == 0 + || observed.root().position().index < observed.barrier().index + || observed.membership().revision() > observed.root().position().index + || observed.membership().revision() < status.changed_at || conf.voters().len() != 3 || !conf.outgoing_voters().is_empty() || !conf.learners().is_empty() @@ -272,6 +336,7 @@ impl FleetCatalog { || conf.auto_leave() || status.phase.is_some_and(|phase| phase.active()) || observed + .membership() .repair_status() .map_err(|_| FleetCatalogError::Corrupt("import repair status"))? .is_some_and(|repair| repair.phase.active()) @@ -283,15 +348,16 @@ impl FleetCatalog { let mut nodes = BTreeSet::new(); for member in conf.voters() { let (node, disk) = if let Some((joined, _)) = observed - .joined_member(*member) + .membership() + .joined(*member) .map_err(|_| FleetCatalogError::Corrupt("import joined member"))? { (joined.node, Some(joined.disk)) } else { let initial = proposal - .voters + .voters() .iter() - .chain(&proposal.learners) + .chain(proposal.learners()) .find(|m| m.member == *member) .ok_or(FleetCatalogError::Conflict)?; (initial.node, None) @@ -308,12 +374,13 @@ impl FleetCatalog { { return Err(FleetCatalogError::Conflict); } - namespace + directory + .namespace() .verify_fleet_storage_pin(&host) .await .map_err(catalog_error)?; - namespace - .verify_import_member_claim(&birth, *member, host.disk()) + directory + .verify_group_import_member_claim(&birth, *member, &host) .await .map_err(catalog_error)?; voters.push(Voter { @@ -328,6 +395,7 @@ impl FleetCatalog { } let source = self.host_from(snapshot, observed.replica().node).await?; if source.disk() != observed.replica().disk + || source.storage_instance() != Some(observed.engine()) || !voters.iter().any(|v| { v.member == observed.replica().member.get() && v.node == observed.replica().node.bytes() @@ -341,10 +409,10 @@ impl FleetCatalog { disk: observed.replica().disk.bytes(), process: observed.replica().process.bytes(), member: observed.replica().member.get(), - barrier: observed.barrier().get(), - applied: observed.applied().index.get(), - term: observed.applied().term.get(), - membership_revision: observed.membership_revision().get(), + barrier: observed.barrier().index.get(), + applied: observed.root().position().index.get(), + term: observed.root().position().term.get(), + membership_revision: observed.membership().revision().get(), consumed_sequence: status.consumed_sequence, phase: status.phase, voters, @@ -352,7 +420,7 @@ impl FleetCatalog { } _ => return Err(FleetCatalogError::Conflict), }; - let name = hex::encode(hash(proposal.scope.cell.as_bytes())); + let name = crate::raft::catalog::cell_group_import_key(proposal); if self.inner.lookup(record.coverage, &name).await?.is_some() { return Err(FleetCatalogError::Corrupt("duplicate import coverage")); } @@ -360,7 +428,7 @@ impl FleetCatalog { records::WITNESS_KIND, &Witness { history: history_key, - birth: entry.expected_digest(), + birth: entry.route().birth_digest, decision, }, 4096, @@ -389,7 +457,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Corrupt("incomplete import coverage")); } if record.covered == record.cut.count { - self.no_namespace_admissions(snapshot, namespace.config().namespace) + self.no_namespace_admissions(snapshot, directory.namespace_config().namespace) .await?; record.phase = FleetImportPhase::Handoff; } @@ -399,7 +467,7 @@ impl FleetCatalog { async fn no_namespace_admissions( &self, snapshot: &FleetSnapshot, - namespace: NamespaceId, + directory: NamespaceId, ) -> Result<()> { let index = snapshot.head.tables[FleetTable::Admissions.slot()]; let mut after = None; @@ -417,7 +485,7 @@ impl FleetCatalog { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("import permit scope"))?; - if scope.namespace == namespace { + if scope.namespace == directory { return Err(FleetCatalogError::Conflict); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs b/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs index b5821146f..693a8c6eb 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/inventory.rs @@ -1,98 +1,80 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One rooted physical intent per step, including cancelled pending groups. use super::*; -use crate::raft::catalog::{BirthDisposition, GroupBirth}; -use crate::raft::fleet_catalog::histories::{Member, Origin}; +use crate::raft::cell_groups::{CellGroupBirth, CellGroupBirthDisposition}; impl FleetCatalog { pub(super) async fn import_inventory_step( &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: &mut Record, - timeout: Duration, ) -> Result> { - let step = namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + directory + .check_group_import(&record.cut, &self.birth_owner()?, false) .await .map_err(catalog_error)?; - let Some(entry) = step.entry else { + let step = directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) + .await + .map_err(catalog_error)?; + let Some(proposal) = step.entry else { if record.cut.count != 0 || record.inventoried != 0 || step.next.is_some() { - return Err(FleetCatalogError::Corrupt("incomplete import inventory")); + return Err(FleetCatalogError::Corrupt( + "incomplete group import inventory", + )); } record.phase = FleetImportPhase::Coverage; return Ok(Vec::new()); }; - let proposal = entry.proposal(); - let class = proposal - .scope - .cell - .split_once(':') - .ok_or(FleetCatalogError::Invalid("import history class"))? - .0; - self.require_namespace_declaration(snapshot, namespace, Some(class)) + self.require_namespace_declaration(snapshot, directory.namespace(), None) .await?; - let decision = match namespace - .birth_disposition(&proposal.scope.cell) + let decision = directory + .settle_import_group(&proposal, &record.cut, &self.birth_owner()?) .await - .map_err(catalog_error)? - { - Some(decision) => decision, - None => namespace - .cancel_birth( - proposal.clone(), - ProposalId::new(record.cut.operation) - .map_err(|_| FleetCatalogError::Corrupt("import cancellation"))?, - timeout, - ) - .await - .map_err(catalog_error)?, - }; + .map_err(catalog_error)?; + let admission = ProposalId::new(request.id) + .map_err(|_| FleetCatalogError::Invalid("group import admission"))?; + let operation = ProposalId::new(request.operation) + .map_err(|_| FleetCatalogError::Invalid("group import operation"))?; let history = match &decision { - BirthDisposition::Live(birth) - if birth.scope() == &proposal.scope - && birth.digest() == entry.expected_digest() => - { + CellGroupBirthDisposition::Live(birth) if birth.route() == proposal.route() => { FleetHistory::observed( - self.binding(birth.scope())?, - namespace, + self.binding(&proposal.route().history)?, + directory, birth, - ProposalId::new(request.id) - .map_err(|_| FleetCatalogError::Invalid("import admission"))?, - ProposalId::new(request.operation) - .map_err(|_| FleetCatalogError::Invalid("import operation"))?, + admission, + operation, )? } - BirthDisposition::Cancelled { scope, operation } if scope == &proposal.scope => { - FleetHistory { - binding: self.binding(scope)?, - schema_digest: namespace.config().schema_digest, - namespace_root: namespace.root_digest(), - birth_digest: entry.expected_digest(), - admission: request.id, - operation: request.operation, - voters: proposal.voters.iter().copied().map(Member::from).collect(), - learners: proposal - .learners - .iter() - .copied() - .map(Member::from) - .collect(), - state: FleetHistoryState::Cancelled { - operation: operation.bytes(), - }, - origin: Origin::Observed, - } - } + CellGroupBirthDisposition::Cancelled { + history, + operation: cancellation, + } if *history == proposal.route().history => FleetHistory::cancelled_group( + self.binding(history)?, + directory, + &proposal, + admission, + operation, + *cancellation, + )?, _ => return Err(FleetCatalogError::Conflict), }; - history.validate()?; let changes = self .import_history_changes( snapshot, - namespace, + directory, &history, match &decision { - BirthDisposition::Live(birth) => Some(birth), + CellGroupBirthDisposition::Live(birth) => Some(birth), _ => None, }, ) @@ -102,10 +84,10 @@ impl FleetCatalog { .checked_add(1) .ok_or(FleetCatalogError::Limit)?; if record.inventoried > record.cut.count - || (step.next.is_none() != (record.inventoried == record.cut.count)) + || step.next.is_none() != (record.inventoried == record.cut.count) { return Err(FleetCatalogError::Corrupt( - "import inventory count/continuation", + "group import count/continuation", )); } record.cursor = step.next; @@ -118,9 +100,9 @@ impl FleetCatalog { async fn import_history_changes( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, history: &FleetHistory, - birth: Option<&GroupBirth>, + birth: Option<&CellGroupBirth>, ) -> Result> { let binding_key = history .binding @@ -143,7 +125,7 @@ impl FleetCatalog { || saved.voters != history.voters || saved.learners != history.learners || saved.state != history.state - || birth.is_some_and(|birth| !saved.check_birth(namespace, birth)) + || birth.is_some_and(|birth| !saved.check_birth(directory, birth)) { return Err(FleetCatalogError::Conflict); } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs b/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs index ba037b269..a459891a3 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/mod.rs @@ -1,12 +1,11 @@ -//! Resumable namespace import. The fleet root owns progress; namespace effects +//! Resumable directory import. The fleet root owns progress; directory effects //! retain frozen references until every canonical history has a committed guard. use super::histories::{catalog_error, history_mutation, validate_request}; use super::*; use crate::raft::catalog::{ - BirthImportCut, BirthInventoryEntry, FrozenBirthInventory, ManagedBirthOwner, NamespaceCatalog, - VoterPolicy, + CellGroupImportCut, FrozenCellGroupInventory, ManagedBirthOwner, VoterPolicy, }; -use crate::raft::host::FleetMembershipObservation; +use crate::raft::cell_groups::{CellGroupBirthRequest, CellGroupQuorumMembership, RangeDirectory}; use cellar_logic::consensus::{NamespaceId, ProposalId}; mod coverage; @@ -55,34 +54,34 @@ impl FleetNamespaceImport { } } -fn key(namespace: NamespaceId) -> String { - format!("namespace/{}", hex::encode(namespace.bytes())) +fn key(directory: NamespaceId) -> String { + format!("namespace/{}", hex::encode(directory.bytes())) } impl FleetCatalog { - /// Root the exact frozen cut before attempting its exclusive namespace + /// Root the exact frozen cut before attempting its exclusive directory /// claim. Preparing/stopped imports grant no managed birth or maintenance. pub async fn begin_namespace_import( &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - frozen: &FrozenBirthInventory, + directory: &RangeDirectory, + frozen: &FrozenCellGroupInventory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; - let cut = frozen.import_cut(namespace).map_err(catalog_error)?; + let cut = frozen.import_cut(directory).map_err(catalog_error)?; if request.operation != cut.operation || cut.count > HISTORY_LIMIT - || namespace.config().cluster.bytes() != self.inner.cluster - || namespace.config().voters != VoterPolicy::ThreeVoters + || directory.namespace_config().cluster.bytes() != self.inner.cluster + || directory.namespace_config().voters != VoterPolicy::ThreeVoters { - return Err(FleetCatalogError::Invalid("namespace import cut/scope")); + return Err(FleetCatalogError::Invalid("directory import cut/scope")); } let record = Record { - namespace: namespace.config().namespace.bytes(), - schema: namespace.config().schema_digest, + namespace: directory.namespace_config().namespace.bytes(), + schema: directory.namespace_config().schema_digest, catalog: self.birth_owner()?.catalog, cut, phase: FleetImportPhase::Preparing, @@ -92,15 +91,19 @@ impl FleetCatalog { cursor: None, coverage: None, }; - record.validate(self, namespace)?; + record.validate(self, directory)?; self.run(timeout, async { - self.require_namespace_declaration(snapshot, namespace, None) + directory + .import_directory_view(&record.cut) + .await + .map_err(catalog_error)?; + self.require_namespace_declaration(snapshot, directory.namespace(), None) .await?; let mutation = history_mutation( request, vec![FleetChange { table: FleetTable::Policy, - key: key(namespace.config().namespace), + key: key(directory.namespace_config().namespace), value: Some(record.encode()?), }], ); @@ -116,7 +119,7 @@ impl FleetCatalog { .inner .lookup( snapshot.head.tables[FleetTable::Policy.slot()], - &key(namespace.config().namespace), + &key(directory.namespace_config().namespace), ) .await? .is_some() @@ -131,13 +134,13 @@ impl FleetCatalog { pub async fn namespace_import( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; self.run(timeout, async { Ok(FleetNamespaceImport { - record: self.import_record(snapshot, namespace).await?, + record: self.import_record(snapshot, directory).await?, }) }) .await @@ -148,14 +151,14 @@ impl FleetCatalog { pub async fn next_namespace_import_birth( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, - ) -> Result> { + ) -> Result> { self.check_snapshot(snapshot)?; self.run_import( timeout, Box::pin(async { - let record = self.import_record(snapshot, namespace).await?; + let record = self.import_record(snapshot, directory).await?; if record.stopped { return Err(FleetCatalogError::Conflict); } @@ -165,8 +168,14 @@ impl FleetCatalog { ) { return Ok(None); } - Ok(namespace - .import_birth_step(&record.cut, &self.birth_owner()?, record.cursor.as_deref()) + Ok(directory + .namespace() + .cell_group_import_step( + directory, + &record.cut, + &self.birth_owner()?, + record.cursor.as_deref(), + ) .await .map_err(catalog_error)? .entry) @@ -182,8 +191,8 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, - observation: Option<&FleetMembershipObservation>, + directory: &RangeDirectory, + observation: Option<&CellGroupQuorumMembership>, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -191,7 +200,7 @@ impl FleetCatalog { self.run_import( timeout, Box::pin(async { - let mut record = self.import_record(snapshot, namespace).await?; + let mut record = self.import_record(snapshot, directory).await?; if request.operation != record.cut.operation || record.stopped { return Err(FleetCatalogError::Conflict); } @@ -200,32 +209,27 @@ impl FleetCatalog { } let changes = match record.phase { FleetImportPhase::Preparing => { - namespace - .claim_birth_import(&record.cut, &self.birth_owner()?) + directory + .namespace() + .change_cell_group_import(&record.cut, &self.birth_owner()?, false) .await .map_err(catalog_error)?; record.phase = FleetImportPhase::Inventory; Vec::new() } FleetImportPhase::Inventory => { - self.import_inventory_step( - snapshot, - request, - namespace, - &mut record, - timeout, - ) - .await? + self.import_inventory_step(snapshot, request, directory, &mut record) + .await? } FleetImportPhase::Coverage => { - self.import_coverage_step(snapshot, namespace, &mut record, observation) + self.import_coverage_step(snapshot, directory, &mut record, observation) .await? } FleetImportPhase::Handoff => { - self.verify_complete_import(snapshot, namespace, &record) + self.verify_complete_import(snapshot, directory, &record) .await?; - namespace - .finish_birth_import(&record.cut, &self.birth_owner()?) + directory + .finish_group_import(&record.cut, &self.birth_owner()?) .await .map_err(catalog_error)?; record.phase = FleetImportPhase::Complete; @@ -233,7 +237,7 @@ impl FleetCatalog { } FleetImportPhase::Complete => return Err(FleetCatalogError::Conflict), }; - self.commit_import_record(snapshot, request, namespace, record, changes) + self.commit_import_record(snapshot, request, directory, record, changes) .await }), ) @@ -247,14 +251,14 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, stopped: bool, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; validate_request(request)?; self.run(timeout, async { - let mut record = self.import_record(snapshot, namespace).await?; + let mut record = self.import_record(snapshot, directory).await?; if request.operation != record.cut.operation || matches!( record.phase, @@ -264,7 +268,7 @@ impl FleetCatalog { return Err(FleetCatalogError::Conflict); } record.stopped = stopped; - self.commit_import_record(snapshot, request, namespace, record, Vec::new()) + self.commit_import_record(snapshot, request, directory, record, Vec::new()) .await }) .await @@ -276,7 +280,7 @@ impl FleetCatalog { pub async fn retry_namespace_import_request( &self, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, timeout: Duration, ) -> Result { validate_request(request)?; @@ -304,7 +308,8 @@ impl FleetCatalog { .changes .last() .ok_or(FleetCatalogError::Corrupt("empty import retry"))?; - if last.table != FleetTable::Policy || last.key != key(namespace.config().namespace) + if last.table != FleetTable::Policy + || last.key != key(directory.namespace_config().namespace) { return Err(FleetCatalogError::Conflict); } @@ -313,7 +318,7 @@ impl FleetCatalog { last.value.as_deref().ok_or(FleetCatalogError::Conflict)?, 4096, )?; - record.validate(self, namespace)?; + record.validate(self, directory)?; if record.cut.operation != request.operation || mutation.changes.len() > 4 || mutation @@ -332,7 +337,7 @@ impl FleetCatalog { .await? .is_none() { - self.verify_complete_import(¤t, namespace, &record) + self.verify_complete_import(¤t, directory, &record) .await?; } self.commit_inner(¤t, &mutation).await @@ -344,18 +349,18 @@ impl FleetCatalog { async fn import_record( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result { let bytes = self .inner .lookup( snapshot.head.tables[FleetTable::Policy.slot()], - &key(namespace.config().namespace), + &key(directory.namespace_config().namespace), ) .await? .ok_or(FleetCatalogError::Conflict)?; let record: Record = decode(records::KIND, &bytes, 4096)?; - record.validate(self, namespace)?; + record.validate(self, directory)?; Ok(record) } @@ -363,14 +368,14 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRequestIdentity, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, record: Record, mut changes: Vec, ) -> Result { - record.validate(self, namespace)?; + record.validate(self, directory)?; changes.push(FleetChange { table: FleetTable::Policy, - key: key(namespace.config().namespace), + key: key(directory.namespace_config().namespace), value: Some(record.encode()?), }); let mutation = history_mutation(request, changes); @@ -399,20 +404,22 @@ impl FleetCatalog { pub(super) async fn managed_owner_for_birth( &self, snapshot: &FleetSnapshot, - namespace: &NamespaceCatalog, + directory: &RangeDirectory, ) -> Result { let owner = self.birth_owner()?; - if namespace.config().managed_fleet.is_some() { - if self.bind_namespace(namespace.config().clone())? != *namespace.config() { + if directory.namespace_config().managed_fleet.is_some() { + if self.bind_namespace(directory.namespace_config().clone())? + != *directory.namespace_config() + { return Err(FleetCatalogError::Conflict); } } else { - let record = self.import_record(snapshot, namespace).await?; + let record = self.import_record(snapshot, directory).await?; if record.phase != FleetImportPhase::Complete || record.stopped { return Err(FleetCatalogError::Conflict); } - namespace - .check_birth_import(&record.cut, &owner, true) + directory + .check_group_import(&record.cut, &owner, true) .await .map_err(catalog_error)?; } diff --git a/crates/cellar/durability/raft/fleet_catalog/imports/records.rs b/crates/cellar/durability/raft/fleet_catalog/imports/records.rs index 6a461a5a2..2257f2098 100644 --- a/crates/cellar/durability/raft/fleet_catalog/imports/records.rs +++ b/crates/cellar/durability/raft/fleet_catalog/imports/records.rs @@ -19,7 +19,7 @@ pub(super) struct Record { pub namespace: [u8; 16], pub schema: [u8; 32], pub catalog: [u8; 32], - pub cut: BirthImportCut, + pub cut: CellGroupImportCut, pub phase: FleetImportPhase, pub stopped: bool, pub inventoried: u64, @@ -37,12 +37,14 @@ fn required_nullable<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>( } impl Record { - pub fn validate(&self, catalog: &FleetCatalog, namespace: &NamespaceCatalog) -> Result<()> { - self.cut.validate(namespace).map_err(catalog_error)?; - if self.namespace != namespace.config().namespace.bytes() - || namespace.config().cluster.bytes() != catalog.inner.cluster - || namespace.config().voters != VoterPolicy::ThreeVoters - || self.schema != namespace.config().schema_digest + pub fn validate(&self, catalog: &FleetCatalog, directory: &RangeDirectory) -> Result<()> { + self.cut + .validate(directory.namespace()) + .map_err(catalog_error)?; + if self.namespace != directory.namespace_config().namespace.bytes() + || directory.namespace_config().cluster.bytes() != catalog.inner.cluster + || directory.namespace_config().voters != VoterPolicy::ThreeVoters + || self.schema != directory.namespace_config().schema_digest || self.catalog != catalog.birth_owner()?.catalog || self.cut.count > HISTORY_LIMIT || self.inventoried > self.cut.count @@ -55,7 +57,7 @@ impl Record { .all(|b| b.is_ascii_hexdigit() && !b.is_ascii_uppercase()) }) { - return Err(FleetCatalogError::Corrupt("namespace import record")); + return Err(FleetCatalogError::Corrupt("directory import record")); } let valid = match self.phase { FleetImportPhase::Preparing => { @@ -84,7 +86,7 @@ impl Record { }; if !valid { return Err(FleetCatalogError::Corrupt( - "namespace import phase/progress", + "directory import phase/progress", )); } Ok(()) diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs index 0052400d0..723d114fa 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/birth_action.rs @@ -344,7 +344,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, resolution: FleetRestartBirthResolution, timeout: Duration, ) -> Result<(Action, FleetSnapshot)> { @@ -395,8 +395,8 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, parent: &Action, - namespace: &NamespaceCatalog, - winner: crate::raft::catalog::BirthDisposition, + namespace: &RangeDirectory, + winner: crate::raft::catalog::CellGroupBirthDisposition, timeout: Duration, ) -> Result { self.run(timeout, async { @@ -423,13 +423,13 @@ impl FleetCatalog { action.check_history(&history)?; let scope = history.binding().scope().map_err(|_| corrupt())?; history.state = match winner { - crate::raft::catalog::BirthDisposition::Live(birth) + crate::raft::catalog::CellGroupBirthDisposition::Live(birth) if history.check_birth(namespace, &birth) => { FleetHistoryState::Live } - crate::raft::catalog::BirthDisposition::Cancelled { - scope: actual, + crate::raft::catalog::CellGroupBirthDisposition::Cancelled { + history: actual, operation, } if actual == scope => FleetHistoryState::Cancelled { operation: operation.bytes(), diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs index 042cde44a..15c1ab227 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/mod.rs @@ -17,7 +17,7 @@ mod stop; use super::evacuation::CanonicalRequest; use super::histories::validate_request; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::{NodeId, ProcessGeneration, ProposalId}; pub use gate::{FleetRestartGateAdmission, FleetRestartGateRequest}; pub use gate_cancel::FleetRestartGateCancelRequest; @@ -125,7 +125,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetRestartRequest, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs index 0f21c00e2..56b9ddbc1 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/driver.rs @@ -52,7 +52,7 @@ pub struct FleetRestartOperatorProgress { /// controller retains Root work; a recreated driver repeats ephemeral proof. pub struct FleetRestartOperatorDriver { pub(super) catalog: FleetCatalog, - pub(super) namespaces: Vec, + pub(super) namespaces: Vec, pub(super) observer: RaftCells, pub(super) evidence: Option>, session: Mutex>, @@ -107,7 +107,7 @@ impl Stage { impl FleetRestartOperatorDriver { pub fn new( catalog: FleetCatalog, - namespaces: Vec, + namespaces: Vec, observer: RaftCells, evidence: Option>, ) -> Result { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs index 08b89eee8..7ce05d70e 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/mod.rs @@ -14,7 +14,7 @@ mod service; mod status; use super::super::resources::Authority; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; pub use cancellation::{ FleetRestartOperatorCancelAdmission, FleetRestartOperatorCancelRequest, FleetRestartOperatorCancelled, @@ -80,7 +80,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, request: &'a FleetRestartOperatorRequest, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], timeout: Duration, ) -> BoxFuture<'a, Result> { Box::pin(async move { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs index d4502f822..807d9665a 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/mod.rs @@ -93,7 +93,7 @@ impl FleetCatalog { pub async fn start_restart_service( &self, observer: crate::application::consensus::RaftCells, - namespaces: Vec, + namespaces: Vec, evidence: Option>, options: FleetRestartServiceOptions, ) -> Result { @@ -115,7 +115,12 @@ impl FleetCatalog { }; let timeout = options.turn_timeout; let work = RestartServiceWork::new( - FleetRestartOperatorDriver::new(self.clone(), namespaces, observer.clone(), evidence)?, + FleetRestartOperatorDriver::new( + self.clone(), + namespaces, + observer.internal_observer(), + evidence, + )?, options, wait, done, diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs index 8823dca4a..f89853b58 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/operator/service/worker.rs @@ -2,19 +2,19 @@ //! Node job-set lifetime, bounded collection and independent shutdown signals. use super::*; -use crate::raft::budget::Permit; +use crate::application::consensus::group_client::{ + ControlServiceCompletion, ControlServiceKind, ControlServiceLease, +}; use crate::raft::fleet_catalog::deadline::Deadline; -use std::sync::atomic::Ordering; pub(crate) struct RestartServiceWork { driver: FleetRestartOperatorDriver, options: FleetRestartServiceOptions, stop: watch::Receiver, - done: watch::Sender, status: watch::Sender, node_stop: Option>, - memory: Option, - admitted: bool, + _registration: Option, + _completion: ControlServiceCompletion, } impl RestartServiceWork { @@ -29,11 +29,10 @@ impl RestartServiceWork { driver, options, stop, - done, status, node_stop: None, - memory: None, - admitted: false, + _registration: None, + _completion: ControlServiceCompletion::new(done), } } @@ -42,23 +41,14 @@ impl RestartServiceWork { return Err(FleetCatalogError::Invalid("restart service fleet")); } let handle = &self.driver.observer.handle; - if handle.closing.load(Ordering::Acquire) - || handle.admission.is_draining() - || *self.stop.borrow() + if self.driver.observer.is_closed() || handle.admission.is_draining() || *self.stop.borrow() { return Err(FleetCatalogError::Closed); } - let mut slot = handle - .fleet_restart_service - .lock() - .map_err(|_| FleetCatalogError::Closed)?; - if slot.is_some() { - return Err(FleetCatalogError::Overloaded); - } - let memory = handle - .control_budget - .acquire(2 * ROOT_LIMIT) - .ok_or(FleetCatalogError::Overloaded)?; + let registration = self + .driver + .observer + .register_control_service(ControlServiceKind::Restart, 2 * ROOT_LIMIT)?; // Select the installed backend and Node domain. A caller cannot replace // either by constructing another client with the same descriptive ID. let source = &host.catalog.inner; @@ -72,11 +62,8 @@ impl RestartServiceWork { turn: Mutex::new(()), }), }; - let (stop, wait) = watch::channel(false); - *slot = Some(stop); - self.node_stop = Some(wait); - self.memory = Some(memory); - self.admitted = true; + self.node_stop = Some(registration.stop_receiver()); + self._registration = Some(registration); self.status.send_modify(|s| s.running = true); Ok(()) } @@ -99,7 +86,7 @@ impl RestartServiceWork { async fn poll(&mut self) { loop { let handle = &self.driver.observer.handle; - if handle.closing.load(Ordering::Acquire) || handle.admission.is_draining() { + if self.driver.observer.is_closed() || handle.admission.is_draining() { return; } self.status @@ -193,13 +180,6 @@ impl RestartServiceWork { impl Drop for RestartServiceWork { fn drop(&mut self) { - if self.admitted { - if let Ok(mut slot) = self.driver.observer.handle.fleet_restart_service.lock() { - slot.take(); - } - } - self.memory.take(); self.status.send_modify(|s| s.running = false); - self.done.send_replace(true); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs index 5e061fae4..b87606143 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/leadership.rs @@ -168,13 +168,8 @@ impl FleetHostAuthorization { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resume scope"))?; - let cell = gateway - .cell(scope.namespace, &scope.cell) - .map_err(failure)?; - if cell.scope() != &scope { - return Err(FleetCatalogError::Conflict); - } - let progress = cell + let group = gateway.group(scope).map_err(failure)?; + let progress = group .replication_status_for(deadline.remaining()?) .await .map_err(failure)?; @@ -310,20 +305,21 @@ impl FleetHostAuthorization { .checked_add(remaining) .ok_or(FleetCatalogError::Limit)? .min(owner_limit); - cell.restart_leadership( - RestartLeadershipCheck { - deadline: expires, - expected_term: Term::new(command.term), - expected_revision: LogIndex::new(command.revision), - through: progress.committed, - survivors, - all_voters: true, - transfer, - }, - deadline.remaining()?, - ) - .await - .map_err(failure)?; + group + .restart_leadership( + RestartLeadershipCheck { + deadline: expires, + expected_term: Term::new(command.term), + expected_revision: LogIndex::new(command.revision), + through: progress.committed, + survivors, + all_voters: true, + transfer, + }, + deadline.remaining()?, + ) + .await + .map_err(failure)?; catalog .run(deadline.remaining()?, input.basis.check_current(catalog)) .await?; @@ -438,12 +434,13 @@ pub(super) async fn reconcile( ) } -fn failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn failure(error: crate::raft::completion::RuntimeError) -> FleetCatalogError { + use crate::raft::completion::RuntimeError as R; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("resume leader state"), + R::Overloaded => FleetCatalogError::Overloaded, + R::Invalid(_) | R::NotFound | R::StaleCompletion => FleetCatalogError::Conflict, + R::Protocol(_) | R::GroupQuarantined => FleetCatalogError::Corrupt("resume leader state"), + R::Closed => FleetCatalogError::Closed, _ => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs index c0596eca5..b83847b1d 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/mod.rs @@ -10,14 +10,15 @@ use super::rejoin::ReturnedCut; use super::runtime::{proof, records as runtime_records}; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::resources::Authority; use crate::raft::fleet_stop::{ FleetStopAction, FleetStopRequest, LocalResume, ResumeMode, StopSelector, }; use crate::raft::physical_inventory::RaftPhysicalInventory; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::MemberId; use futures_util::future::BoxFuture; pub(crate) use leadership::ResumeLeadershipCommand; use records::{Completion, Intent}; @@ -60,16 +61,16 @@ impl From for FleetRestartResumePhase { #[derive(Clone, Debug)] pub enum FleetRestartResumeProgress { Unrelated { - group: GroupScope, + group: GroupHistory, }, Verified { - group: GroupScope, + group: GroupHistory, current_member: Option, retained_member: Option, replaced: bool, }, HandoffRequested { - group: GroupScope, + group: GroupHistory, from: MemberId, to: MemberId, }, @@ -161,7 +162,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, request: &'a FleetRestartResumeRequest, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: &'a RaftCells, timeout: Duration, ) -> BoxFuture<'a, Result> { diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs index 548cbce88..a7522648f 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/resume/verification/mod.rs @@ -187,11 +187,7 @@ impl FleetRestartResumeVerification { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resume scope"))?; - let physical = self - .inventory - .history(&group) - .map_err(|_| FleetCatalogError::Conflict)? - .copied(); + let physical = self.inventory.group_history(&group).copied(); let target = NodeId::new(self.cut.node.node) .map_err(|_| FleetCatalogError::Corrupt("resume target"))?; let key = history diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs index 6bf68b423..d8eba18ab 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/materialize.rs @@ -59,9 +59,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( .as_ref(), None => verification.inventory.as_ref(), }; - let slot = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)?; + let slot = inventory.group_history(&scope); if let Some(slot) = slot { check_slot(history, member, *slot)?; let unregistered = physical @@ -121,8 +119,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( ) .await?; let slot = actual - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .ok_or(FleetCatalogError::Unconfirmed)?; check_slot(history, member, *slot)?; refresh_target(verification, history, &voters, actual)?; @@ -132,8 +129,7 @@ pub(in crate::raft::fleet_catalog::maintenance::runtime) async fn next( .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)? - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .copied() .ok_or(FleetCatalogError::Unconfirmed)?; check_slot(history, member, slot)?; @@ -184,8 +180,7 @@ fn refresh_target( .map_err(|_| FleetCatalogError::Corrupt("genesis maintenance node"))?; let member = *voters.get(&main).ok_or(FleetCatalogError::Conflict)?; let slot = actual - .history(&scope) - .map_err(|_| FleetCatalogError::Conflict)? + .group_history(&scope) .copied() .ok_or(FleetCatalogError::Conflict)?; check_slot(history, member, slot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs index a3545e1df..3ef823cb4 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/genesis/receive.rs @@ -79,21 +79,12 @@ impl FleetHostAuthorization { let authorization = catalog .authorize_group(&snapshot, &scope, deadline.remaining()?) .await?; - let cell = match gateway.cell(scope.namespace, &scope.cell) { - Ok(cell) if cell.scope() == &scope => cell, - Ok(_) => return Err(FleetCatalogError::Conflict), - Err(_) => gateway - .register_managed_history(authorization.clone(), deadline.remaining()?) - .await - .map_err(command_failure)?, - }; - // This pipeline owns initialization after submission, even when the - // control waiter disconnects. It refuses an already-claimed blank - // engine, occupied/retired slot, wrong schema or substituted boot. - let receipt = cell - .activate_managed_initial_member(authorization, deadline.remaining()?) + // The controller owns complete-group recovery and attachment after + // admission. A receipt is descriptive; fresh quorum is checked later. + let receipt = gateway + .activate_managed_initial_group(authorization, deadline.remaining()?) .await - .map_err(command_failure)?; + .map_err(startup_failure)?; if &receipt.binding != history.binding() { return Err(FleetCatalogError::Conflict); } @@ -110,13 +101,50 @@ impl FleetHostAuthorization { } } -fn command_failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn startup_failure( + error: crate::application::consensus::CellGroupStartupError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupStartupError; + use crate::raft::cell_groups::CellGroupControllerError as C; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("genesis host state"), - C::UnsupportedCapability => FleetCatalogError::Invalid("genesis host capability"), - _ => FleetCatalogError::Unconfirmed, + // Entered recovery may have synced or attached even if its delivery failed. + CellGroupStartupError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupStartupError::Refused(C::Runtime( + crate::raft::completion::RuntimeError::Overloaded, + )) => FleetCatalogError::Overloaded, + CellGroupStartupError::Refused(C::Catalog(error)) => { + use crate::raft::catalog::CatalogError as E; + match error.as_ref() { + E::Invalid(reason) => FleetCatalogError::Invalid(reason), + E::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + E::Limit => FleetCatalogError::Limit, + E::Log(crate::raft::storage::LogError::Corrupt(_)) | E::LostStableState => { + FleetCatalogError::Corrupt("genesis group state") + } + E::Conflict(_) | E::ManagedAdmissionRequired | E::NotInitialMember | E::Fenced => { + FleetCatalogError::Conflict + } + E::FleetAuthority(authority) => match authority.as_ref() { + FleetCatalogError::Invalid(reason) => FleetCatalogError::Invalid(reason), + FleetCatalogError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + FleetCatalogError::Overloaded => FleetCatalogError::Overloaded, + FleetCatalogError::Limit => FleetCatalogError::Limit, + FleetCatalogError::Conflict => FleetCatalogError::Conflict, + FleetCatalogError::Closed => FleetCatalogError::Closed, + // Preserve the complete typed cause for local diagnosis; + // transport delivery still classifies unknown outcomes. + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + }, + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + } + } + CellGroupStartupError::Refused(C::Invalid(_)) => { + FleetCatalogError::Invalid("genesis group capability") + } + CellGroupStartupError::Refused(_) => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs index 7085d96c4..e79b221f0 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/handoff/receive.rs @@ -103,13 +103,8 @@ impl FleetHostAuthorization { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("restart scope"))?; - let cell = gateway - .cell(scope.namespace, &scope.cell) - .map_err(failure)?; - if cell.scope() != &scope { - return Err(FleetCatalogError::Conflict); - } - let progress = cell + let group = gateway.group(scope).map_err(failure)?; + let progress = group .replication_status_for(deadline.remaining()?) .await .map_err(failure)?; @@ -243,20 +238,21 @@ impl FleetHostAuthorization { .checked_add(budget) .ok_or(FleetCatalogError::Limit)?, ); - cell.restart_leadership( - RestartLeadershipCheck { - deadline: evidence_deadline, - expected_term: Term::new(command.term), - expected_revision: LogIndex::new(command.revision), - through: progress.committed, - survivors, - all_voters: false, - transfer, - }, - deadline.remaining()?, - ) - .await - .map_err(failure)?; + group + .restart_leadership( + RestartLeadershipCheck { + deadline: evidence_deadline, + expected_term: Term::new(command.term), + expected_revision: LogIndex::new(command.revision), + through: progress.committed, + survivors, + all_voters: false, + transfer, + }, + deadline.remaining()?, + ) + .await + .map_err(failure)?; catalog .run(deadline.remaining()?, basis.check_current(catalog)) .await?; @@ -267,12 +263,15 @@ impl FleetHostAuthorization { } } -fn failure(error: crate::application::consensus::CommandFailure) -> FleetCatalogError { - use crate::application::consensus::CommandFailure as C; +fn failure(error: crate::raft::completion::RuntimeError) -> FleetCatalogError { + use crate::raft::completion::RuntimeError as R; match error { - C::Overloaded => FleetCatalogError::Overloaded, - C::CommandConflict => FleetCatalogError::Conflict, - C::CorruptState => FleetCatalogError::Corrupt("restart leadership host state"), + R::Overloaded => FleetCatalogError::Overloaded, + R::Invalid(_) | R::NotFound | R::StaleCompletion => FleetCatalogError::Conflict, + R::Protocol(_) | R::GroupQuarantined => { + FleetCatalogError::Corrupt("restart leadership host state") + } + R::Closed => FleetCatalogError::Closed, _ => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs index 4cdf1bcd7..2f3c0d7c6 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/mod.rs @@ -19,7 +19,8 @@ use crate::raft::fleet_catalog::finalization::receipts::Publication; use crate::raft::physical_inventory::{RaftPhysicalHistory, RaftPhysicalInventory}; pub(crate) use action::RestartActionContext; use basis::Basis; -use cellar_logic::consensus::{GroupScope, MemberId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::MemberId; use futures_util::future::BoxFuture; pub(super) use handoff::LeadershipAction; pub(crate) use handoff::RestartLeadershipCommand; @@ -50,17 +51,17 @@ pub struct FleetRestartVerification { #[derive(Clone, Debug)] pub enum FleetRestartHistoryProgress { Cancelled { - group: GroupScope, + group: GroupHistory, }, Unrelated { - group: GroupScope, + group: GroupHistory, }, Retained { - group: GroupScope, + group: GroupHistory, member: MemberId, }, Voting { - group: GroupScope, + group: GroupHistory, member: MemberId, survivors: [NodeId; 2], }, @@ -117,7 +118,7 @@ impl FleetCatalog { &'a self, snapshot: &'a FleetSnapshot, operation: ProposalId, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: &'a RaftCells, timeout: Duration, ) -> BoxFuture<'a, Result> { @@ -263,11 +264,7 @@ impl FleetRestartVerification { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("restart scope"))?; - let physical = self - .inventory - .history(&group) - .map_err(|_| FleetCatalogError::Conflict)? - .copied(); + let physical = self.inventory.group_history(&group).copied(); let target = NodeId::new(self.gate.node.node) .map_err(|_| FleetCatalogError::Corrupt("restart target"))?; let voters = catalog diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs index da12b2623..def3e433d 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/proof.rs @@ -380,9 +380,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( if observations.len() != 3 { return Err(FleetCatalogError::InventoryIncomplete); } - let fixed = observations[0] - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let fixed = observations[0].published_cut(); let result = catalog .run(deadline.remaining()?, async { let first = observations[0].membership(); @@ -415,9 +413,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( state, replica.node == target && paused, )?; - let cut = observed - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let cut = observed.published_cut(); let member = observed .progress() .member @@ -476,7 +472,7 @@ pub(in crate::raft::fleet_catalog::maintenance) fn observe_current_voting<'a>( ), epoch: auth.epoch(), member: replica.member.get(), - publication: hash(&encode(57, &Publication::from(cut), records::LIMIT)?), + publication: hash(&encode(57, &Publication::from(cut), PAGE_LIMIT)?), leader_load: observed .progress() .leader_load diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs index 03f69b0cd..5278cb8b8 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/runtime/reconcile.rs @@ -9,13 +9,13 @@ use super::*; pub enum FleetRestartRuntimeProgress { /// Transfer admission only. A later turn must observe the survivor leader. HandoffRequested { - group: GroupScope, + group: GroupHistory, from: MemberId, to: MemberId, term: cellar_logic::consensus::Term, }, Materialized { - group: GroupScope, + group: GroupHistory, node: NodeId, member: MemberId, previously_present: bool, diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs index 08f972647..0c14c730a 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/settle.rs @@ -3,8 +3,8 @@ //! Explicit bounded controller turns; durable roots own ambiguous work. use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; -use cellar_logic::consensus::GroupScope; +use crate::raft::catalog::RangeDirectory; +use cellar_logic::cell_group::GroupHistory; use rand::RngCore; use std::pin::Pin; @@ -22,7 +22,7 @@ pub enum FleetRestartBirthResolution { #[derive(Clone, Debug, PartialEq, Eq)] pub struct FleetRestartBirthAction { pub admission: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub resolution: FleetRestartBirthResolution, } @@ -31,11 +31,11 @@ pub struct FleetRestartBirthAction { pub enum FleetRestartSettlementBlocker { ResourceOnly { reservation: [u8; 16], - group: GroupScope, + group: GroupHistory, }, MovementServicesUnavailable { reservation: [u8; 16], - group: GroupScope, + group: GroupHistory, }, } @@ -46,7 +46,7 @@ pub enum FleetRestartSettlementProgress { Birth { admission: [u8; 16], resolution: FleetRestartBirthResolution, - group: GroupScope, + group: GroupHistory, state: FleetHistoryState, applied_revision: u64, observed_revision: u64, @@ -86,7 +86,7 @@ impl FleetCatalog { &'a self, operation: ProposalId, resolution: FleetRestartBirthResolution, - namespaces: &'a [NamespaceCatalog], + namespaces: &'a [RangeDirectory], observer: Option<&'a RaftCells>, evidence: Option<&'a dyn FleetMovementEvidence>, timeout: Duration, @@ -115,7 +115,7 @@ impl FleetCatalog { } else if let Some(scope) = work.next_birth { let namespace = namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; self.recover_birth_reservation(&snapshot, namespace, &scope, deadline.remaining()?) .await?; @@ -177,36 +177,38 @@ impl FleetCatalog { .await?; let namespace = namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; let authorization = self .recover_birth_reservation(&snapshot, namespace, &scope, deadline.remaining()?) .await?; let namespace_timeout = deadline.remaining()?; let decision = async { - use crate::raft::catalog::{BirthDisposition, CatalogError}; + use crate::raft::catalog::{CatalogError, CellGroupBirthDisposition}; match action.resolution { FleetRestartBirthResolution::Cancel => { namespace - .cancel_managed_birth( + .cancel_fleet_group_birth( &authorization, ProposalId::new(action.request.id) .map_err(|_| CatalogError::Corrupt("restart birth ID"))?, - namespace_timeout, ) .await } FleetRestartBirthResolution::Finish => { - if let Some(winner) = namespace.birth_disposition(&scope.cell).await? { + if let Some(winner) = namespace + .group_birth_disposition(&authorization.proposal()) + .await? + { return Ok(winner); } match namespace - .resolve_managed_birth(&authorization, namespace_timeout) + .materialize_fleet_group_birth(&authorization) .await { - Ok(birth) => Ok(BirthDisposition::Live(birth)), + Ok(birth) => Ok(CellGroupBirthDisposition::Live(birth)), Err(CatalogError::BirthCancelled) => namespace - .birth_disposition(&scope.cell) + .group_birth_disposition(&authorization.proposal()) .await? .ok_or(CatalogError::Unconfirmed), Err(error) => Err(error), diff --git a/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs b/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs index feaa86bdd..ba0f67096 100644 --- a/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs +++ b/crates/cellar/durability/raft/fleet_catalog/maintenance/settlement.rs @@ -2,7 +2,7 @@ //! Complete rooted pending work, distinct from live survivor/stop readiness. use super::*; -use cellar_logic::consensus::GroupScope; +use cellar_logic::cell_group::GroupHistory; /// Descriptive work at one current root. Memory is bounded to the next birth, /// one owned birth action and the active resource book (at most two rows). An empty result proves no @@ -11,7 +11,7 @@ use cellar_logic::consensus::GroupScope; pub struct FleetRestartPendingWork { pub observed_revision: u64, pub pending_births: u64, - pub next_birth: Option, + pub next_birth: Option, pub active_birth: Option, pub movements: Vec, pub reservations: Vec, @@ -25,7 +25,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, operation: ProposalId, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/mod.rs b/crates/cellar/durability/raft/fleet_catalog/mod.rs index 5cef81800..2d335c233 100644 --- a/crates/cellar/durability/raft/fleet_catalog/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/mod.rs @@ -1,7 +1,9 @@ //! Bounded CAS coordination. These records never grant Raft membership or votes. //! A retained mutation journal resolves ambiguous writes after later revisions. +mod archive_replay; mod boots; +pub use archive_replay::{FleetGroupArchiveReplayOptions, FleetGroupArchiveVerifier}; mod capacity; mod deadline; mod disruption; @@ -65,11 +67,11 @@ pub(crate) use maintenance::{ RestartActionContext, RestartLeadershipCommand, ResumeLeadershipCommand, }; pub use namespaces::FleetInventory; +pub(crate) use nodes::FleetTrustSnapshot; pub use nodes::{ FleetHostAuthorization, FleetNodeEligibility, FleetNodeSpec, FleetNodeState, FleetRequestIdentity, }; -pub(crate) use nodes::{FleetTrustReader, FleetTrustSnapshot}; pub use permits::{FleetGroupAuthorization, FleetPermitAuthorization}; pub use placement::FleetReplicaSeparation; pub(crate) use resources::MovementServiceWork; @@ -148,11 +150,29 @@ impl From for FleetCatalogError { super::catalog::CatalogError::Corrupt(reason) => Self::Corrupt(reason), super::catalog::CatalogError::Limit => Self::Limit, super::catalog::CatalogError::Storage(error) => Self::Storage(error.into()), + super::catalog::CatalogError::FleetAuthority(error) => *error, _ => Self::Corrupt("index storage contract"), } } } +impl From for FleetCatalogError { + fn from(error: crate::raft::cell_groups::CellGroupArchiveError) -> Self { + use crate::raft::cell_groups::CellGroupArchiveError; + match error { + CellGroupArchiveError::Invalid(reason) => Self::Invalid(reason), + CellGroupArchiveError::Limit => Self::Limit, + CellGroupArchiveError::Busy => Self::Overloaded, + CellGroupArchiveError::Conflict => Self::Conflict, + CellGroupArchiveError::Timeout + | CellGroupArchiveError::Expired + | CellGroupArchiveError::Unconfirmed => Self::Unconfirmed, + CellGroupArchiveError::Stopped => Self::Closed, + error => Self::Storage(error.into()), + } + } +} + impl fmt::Display for FleetCatalogError { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "fleet catalog: {self:?}") diff --git a/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs b/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs index b535f9b57..6053d7735 100644 --- a/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs +++ b/crates/cellar/durability/raft/fleet_catalog/namespaces/inventory.rs @@ -1,26 +1,29 @@ -use super::super::histories::Origin; +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Bidirectional physical-group inventory. Selected ranges must be rooted in +//! the fleet, and all rooted reservations keep their node/debt references. use super::*; -use crate::raft::catalog::{BirthDisposition, ManagedBirthInventory}; +use crate::raft::cell_groups::{CellGroupBirthDisposition, CellGroupBirthRequest, RangeDirectory}; use std::collections::BTreeMap; impl FleetCatalog { - /// Verify the entire rooted namespace set and all histories. Caller handles - /// select adapters, never the coverage set. Missing/offline namespaces refuse - /// success; pending births retain references and do not become absence proof. + /// Verify every declared namespace and every retained physical history at + /// this fleet cut. Caller adapters select neither the coverage set nor its + /// count. An inventory is not a current eligibility or release certificate. pub async fn check_inventory( &self, snapshot: &FleetSnapshot, - namespaces: &[NamespaceCatalog], + directories: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; - if namespaces.len() > MAX_NAMESPACES { + if directories.len() > MAX_NAMESPACES { return Err(FleetCatalogError::Limit); } self.run(timeout, async { self.inner .domain - .run(Box::pin(self.check_inventory_inner(snapshot, namespaces))) + .run(Box::pin(self.check_inventory_inner(snapshot, directories))) .await .map_err(|_| FleetCatalogError::Closed)? }) @@ -30,13 +33,13 @@ impl FleetCatalog { pub(in super::super) async fn check_inventory_inner( &self, snapshot: &FleetSnapshot, - namespaces: &[NamespaceCatalog], + directories: &[RangeDirectory], ) -> Result { let declarations = self.declarations(snapshot).await?; let mut catalogs = BTreeMap::new(); - for namespace in namespaces { + for directory in directories { if catalogs - .insert(namespace.config().namespace.bytes(), namespace) + .insert(directory.namespace_config().namespace.bytes(), directory) .is_some() { return Err(FleetCatalogError::Invalid("duplicate inventory namespace")); @@ -45,53 +48,56 @@ impl FleetCatalog { if catalogs.len() != declarations.len() { return Err(FleetCatalogError::InventoryIncomplete); } + let owner = self.birth_owner()?; for declaration in &declarations { - let namespace = catalogs + let directory = *catalogs .get(&declaration.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; - declaration.check(self, namespace, None)?; - if namespace.config().managed_fleet.is_none() { - self.verify_imported_inventory(snapshot, namespace).await?; + declaration.check(self, directory.namespace(), None)?; + // Import must pin and certify the complete range directory before + // it can acquire this owner. The predecessor cell-birth inventory + // cannot certify a shared namespace. + if self.managed_owner_for_birth(snapshot, directory).await? != owner { + return Err(FleetCatalogError::InventoryIncomplete); } - let cut: ManagedBirthInventory = namespace - .managed_birth_inventory(&self.birth_owner()?) - .await - .map_err(catalog_error)?; - if cut.count > HISTORY_LIMIT { + if directory.namespace_config().managed_fleet.is_none() { + self.verify_imported_inventory(snapshot, directory).await?; + } + let view = directory.observe().await.map_err(catalog_error)?; + if u64::from(view.groups()) > HISTORY_LIMIT { return Err(FleetCatalogError::Limit); } let mut cursor = None; let mut count = 0u64; loop { - let page = namespace - .managed_birth_page(&cut, cursor.as_ref()) + let page = directory + .group_page(&view, cursor.as_ref(), 16) .await .map_err(catalog_error)?; - for entry in page.entries { + for route in page.routes { count = count .checked_add(1) - .filter(|n| *n <= cut.count) - .ok_or(FleetCatalogError::Corrupt("extra managed birth intent"))?; - let binding = self.binding(&entry.proposal().scope)?; - let history = self - .history_inner( - snapshot, - &binding - .key() - .map_err(|_| FleetCatalogError::Corrupt("inventory binding"))?, - ) - .await - .map_err(|error| match error { - FleetCatalogError::Conflict => FleetCatalogError::InventoryIncomplete, - error => error, - })?; - let initial = history.initial_placement()?; - if history.birth_digest != entry.expected_digest() - || initial.scope != entry.proposal().scope - || initial.voters != entry.proposal().voters - || initial.learners != entry.proposal().learners + .filter(|n| *n <= u64::from(view.groups())) + .ok_or(FleetCatalogError::Corrupt("directory inventory count"))?; + let binding = self.binding(&route.history)?; + let key = binding + .key() + .map_err(|_| FleetCatalogError::Corrupt("inventory binding"))?; + let history = + self.history_inner(snapshot, &key) + .await + .map_err(|error| match error { + FleetCatalogError::Conflict => { + FleetCatalogError::InventoryIncomplete + } + error => error, + })?; + let expected = self.inventory_birth(directory, &history)?; + if history.binding != binding + || expected.route() != &route + || matches!(history.state, FleetHistoryState::Cancelled { .. }) { - return Err(FleetCatalogError::Corrupt("uninventoried canonical birth")); + return Err(FleetCatalogError::Corrupt("uninventoried selected group")); } } cursor = page.next; @@ -99,8 +105,8 @@ impl FleetCatalog { break; } } - if count != cut.count { - return Err(FleetCatalogError::Corrupt("missing managed birth intent")); + if count != u64::from(view.groups()) { + return Err(FleetCatalogError::Corrupt("missing selected group")); } } let nodes = self.node_records(snapshot).await?; @@ -129,7 +135,7 @@ impl FleetCatalog { .iter() .find(|d| d.namespace == scope.namespace.bytes()) .ok_or(FleetCatalogError::InventoryIncomplete)?; - let namespace = catalogs[&declaration.namespace]; + let directory = catalogs[&declaration.namespace]; if entry.key != history .binding @@ -138,8 +144,6 @@ impl FleetCatalog { || history.binding != self.binding(&scope)? || history.namespace_root != declaration.root || history.schema_digest != declaration.schema - || scope.cell.split_once(':').map(|(class, _)| class) - != Some(declaration.class.as_str()) { return Err(FleetCatalogError::Corrupt("inventory history declaration")); } @@ -155,48 +159,31 @@ impl FleetCatalog { } } for member in history.voters.iter().chain(&history.learners) { - if !nodes.iter().any(|n| n.node == member.node) { + if !nodes.iter().any(|node| node.node == member.node) { return Err(FleetCatalogError::InventoryIncomplete); } } - namespace - .verify_group_identity(&scope) - .await - .map_err(catalog_error)?; - let decision = namespace - .birth_disposition(&scope.cell) + let request = self.inventory_birth(directory, &history)?; + let birth = directory + .group_birth_disposition(&request) .await .map_err(catalog_error)?; - match (&history.state, decision) { - (FleetHistoryState::Reserved, decision) => { - let (_, digest) = namespace - .prepare_birth_with_owner( - history.initial_placement()?, - Some(&self.birth_owner()?), - ) - .map_err(catalog_error)?; - if history.origin != Origin::Managed - || digest != history.birth_digest - || decision.as_ref().is_some_and(|d| match d { - BirthDisposition::Live(b) => !history.check_birth(namespace, b), - BirthDisposition::Cancelled { scope: actual, .. } => { - actual != &scope - } - }) - { - return Err(FleetCatalogError::Corrupt("reserved inventory history")); - } - } - (FleetHistoryState::Live, Some(BirthDisposition::Live(birth))) - if history.check_birth(namespace, &birth) => {} + match (&history.state, birth) { + (FleetHistoryState::Reserved, None) if history.allows_managed_genesis() => (), + ( + FleetHistoryState::Reserved | FleetHistoryState::Live, + Some(CellGroupBirthDisposition::Live(birth)), + ) if history.check_birth(directory, &birth) => (), ( FleetHistoryState::Cancelled { operation }, - Some(BirthDisposition::Cancelled { - scope: actual, - operation: actual_op, + Some(CellGroupBirthDisposition::Cancelled { + history: actual, + operation: decision, }), - ) if actual == scope && actual_op.bytes() == *operation => {} - _ => return Err(FleetCatalogError::Corrupt("inventory birth disposition")), + ) if actual == scope && decision.bytes() == *operation => (), + // Cancellation is a terminal directory decision, not a + // caller-supplied enum value or an observation of absence. + _ => return Err(FleetCatalogError::InventoryIncomplete), } } cursor = page.next; @@ -213,4 +200,23 @@ impl FleetCatalog { histories, }) } + + fn inventory_birth( + &self, + directory: &RangeDirectory, + history: &FleetHistory, + ) -> Result { + let request = directory + .prepare_fleet_group_birth(history.initial_placement()?, &self.birth_owner()?) + .map_err(catalog_error)?; + if request.route().birth_digest != history.birth_digest + || directory.namespace().root_digest() != history.namespace_root + || request.route().schema_digest != history.schema_digest + { + return Err(FleetCatalogError::Corrupt( + "inventory immutable group birth", + )); + } + Ok(request) + } } diff --git a/crates/cellar/durability/raft/fleet_catalog/nodes.rs b/crates/cellar/durability/raft/fleet_catalog/nodes.rs index 45aedd073..323d36845 100644 --- a/crates/cellar/durability/raft/fleet_catalog/nodes.rs +++ b/crates/cellar/durability/raft/fleet_catalog/nodes.rs @@ -360,6 +360,15 @@ impl fmt::Debug for FleetHostAuthorization { } impl FleetHostAuthorization { + pub(crate) fn same_boot_owner(&self, other: &Self) -> bool { + Arc::ptr_eq(&self.catalog.inner, &other.catalog.inner) + && self + .boot_binding() + .ok() + .zip(other.boot_binding().ok()) + .is_some_and(|(left, right)| left == right) + } + pub(crate) fn is_isolated(&self) -> bool { self.record.isolation.is_some() } @@ -555,7 +564,7 @@ impl FleetHostAuthorization { pub(crate) async fn group_authorization( &self, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, timeout: Duration, ) -> Result { self.group_authorization_at_least(scope, 0, timeout).await @@ -563,7 +572,7 @@ impl FleetHostAuthorization { pub(crate) async fn group_authorization_at_least( &self, - scope: &cellar_logic::consensus::GroupScope, + scope: &cellar_logic::cell_group::GroupHistory, minimum_revision: u64, timeout: Duration, ) -> Result { diff --git a/crates/cellar/durability/raft/fleet_catalog/permits.rs b/crates/cellar/durability/raft/fleet_catalog/permits.rs index a3289e331..3f6457c60 100644 --- a/crates/cellar/durability/raft/fleet_catalog/permits.rs +++ b/crates/cellar/durability/raft/fleet_catalog/permits.rs @@ -2,7 +2,7 @@ //! A capability never replaces the group's current quorum or consumed sequence. use super::*; use crate::raft::member_state::{FleetGroupBinding, FleetMembershipPermit}; -use cellar_logic::consensus::GroupScope; +use cellar_logic::{cell_group::GroupHistory, consensus::NodeId}; #[derive(Clone, Debug)] pub struct FleetGroupAuthorization { @@ -12,11 +12,14 @@ pub struct FleetGroupAuthorization { } impl FleetGroupAuthorization { - pub(crate) fn matches_replica( + pub(crate) fn matches_replica( &self, - replica: &cellar_logic::consensus::ReplicaIdentity, + replica: &cellar_logic::consensus::ReplicaIdentity, membership: &crate::raft::member_state::GroupMembership, - ) -> bool { + ) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { if !self.matches_scope(&replica.scope) || !membership.permits(replica) { return false; } @@ -34,6 +37,42 @@ impl FleetGroupAuthorization { } } + /// Resolve all current physical occupants from the immutable birth or the + /// committed joining bindings. An advisory route may omit an isolated peer. + pub(crate) fn placement_nodes( + &self, + observed: &crate::raft::cell_groups::CellGroupQuorumMembership, + ) -> std::result::Result, crate::raft::storage::LogError> { + let invalid = + || crate::raft::storage::LogError::Invalid("group placement authority".into()); + if !self.matches_replica(observed.replica(), observed.membership()) + || self.history_digests() + != ( + observed.root().route().birth_digest, + observed.root().route().schema_digest, + ) + { + return Err(invalid()); + } + crate::raft::member_state::members(observed.configuration()) + .into_iter() + .map(|id| { + let member = cellar_logic::consensus::MemberId::new(id).map_err(|_| invalid())?; + if let Some((binding, _)) = observed.membership().joined(member)? { + return Ok(binding.node); + } + let initial = self + .history + .voters + .iter() + .chain(&self.history.learners) + .find(|v| v.member == id) + .ok_or_else(invalid)?; + NodeId::new(initial.node).map_err(|_| invalid()) + }) + .collect() + } + pub(crate) fn history_digests(&self) -> ([u8; 32], [u8; 32]) { (self.history.birth_digest(), self.history.schema_digest()) } @@ -44,33 +83,36 @@ impl FleetGroupAuthorization { pub(crate) fn check_birth( &self, - namespace: &crate::raft::catalog::NamespaceCatalog, - birth: &crate::raft::catalog::GroupBirth, + directory: &crate::raft::cell_groups::RangeDirectory, + birth: &crate::raft::cell_groups::CellGroupBirth, ) -> bool { - self.history.check_birth(namespace, birth) - && birth - .managed_owner() + self.history.check_birth(directory, birth) + && directory + .namespace() + .config() + .managed_fleet + .as_ref() .is_none_or(|owner| owner == &self.owner) } pub(crate) fn matches_birth( &self, - birth: &crate::raft::catalog::GroupBirth, + birth: &crate::raft::cell_groups::CellGroupBirth, schema: [u8; 32], ) -> bool { - self.matches_scope(birth.scope()) - && self.history.birth_digest() == birth.digest() + self.matches_scope(&birth.route().history) + && self.history.birth_digest() == birth.route().birth_digest && self.history.schema_digest() == schema - && birth - .managed_owner() - .is_none_or(|owner| owner == &self.owner) } pub(crate) fn binding(&self) -> &FleetGroupBinding { &self.binding } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { self.binding.matches_scope(scope) } } @@ -109,7 +151,10 @@ impl FleetPermitAuthorization { &self.permit } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { self.permit.binding().matches_scope(scope) } } @@ -120,7 +165,7 @@ impl FleetCatalog { pub async fn authorize_group( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -156,7 +201,7 @@ impl FleetCatalog { pub async fn authorize_membership( &self, snapshot: &FleetSnapshot, - scope: &GroupScope, + scope: &GroupHistory, timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -189,21 +234,25 @@ impl FleetCatalog { .await } - pub(super) fn binding(&self, scope: &GroupScope) -> Result { + pub(super) fn binding(&self, scope: &S) -> Result + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); if scope.cluster.bytes() != self.inner.cluster { return Err(FleetCatalogError::Conflict); } FleetGroupBinding::new( IncarnationId::new(self.inner.incarnation) .map_err(|_| FleetCatalogError::Corrupt("fleet incarnation"))?, - scope, + &scope, self.birth_owner()?.catalog, ) .map_err(|_| FleetCatalogError::Invalid("group scope")) } /// Exact catalog-scoped descriptor. The value itself grants no authority. - pub fn group_binding(&self, scope: &GroupScope) -> Result { + pub fn group_binding(&self, scope: &GroupHistory) -> Result { self.binding(scope) } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs b/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs index d6c5219c8..e698ae174 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/admission.rs @@ -5,7 +5,7 @@ use super::records::{Book, Record, Reference, Request}; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::host::RaftPeerNodeObservation; use cellar_logic::consensus::{ ClusterId, DiskLineage, EngineInstanceId, IncarnationId, NodeIdentity, ProcessGeneration, @@ -51,7 +51,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetResourceRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.reserve_resource_request( @@ -70,7 +70,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, canonical: Request, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], options: AdmissionOptions, timeout: Duration, ) -> Result { @@ -156,11 +156,7 @@ impl FleetCatalog { .binding() .scope() .map_err(|_| FleetCatalogError::Corrupt("resource scope"))?; - if inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("resource destination scope"))? - .is_some() - { + if inventory.group_history(&scope).is_some() { return Err(FleetCatalogError::Invalid("occupied resource destination")); } if *history.state() != FleetHistoryState::Live { @@ -431,8 +427,7 @@ impl FleetCatalog { .scope() .map_err(|_| FleetCatalogError::Corrupt("resource genesis scope"))?; let occupied = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("resource genesis scope"))? + .group_history(&scope) .ok_or(resource_error(LedgerError::Incomplete))?; if occupied.birth_digest() != history.birth_digest() || !history.voters.iter().chain(&history.learners).any(|m| { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs index 251bd3f69..bc7090415 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/completion/mod.rs @@ -471,8 +471,7 @@ impl FleetCatalog { .inventory .as_ref() .ok_or(FleetCatalogError::Unconfirmed)? - .history(&scope) - .map_err(|_| invalid())? + .group_history(&scope) .is_some(), membership_revision: quorum.membership().membership_revision().get(), barrier: quorum.membership().barrier().get(), diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs b/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs index 4b25a33af..358c2f473 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/completion/proofs.rs @@ -298,7 +298,7 @@ impl FleetCatalog { .ok_or(FleetCatalogError::Unconfirmed)?; let history = self.history_inner(snapshot, &row.request.group).await?; let scope = history.binding().scope().map_err(|_| invalid())?; - let physical = inventory.history(&scope).map_err(|_| invalid())?; + let physical = inventory.group_history(&scope); let matches = match (role, physical) { (PhysicalRole::EmptyTarget, None) => true, (PhysicalRole::AbortedTarget { .. }, None) => true, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs index acbd56223..20ce4b19f 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/mod.rs @@ -60,7 +60,7 @@ pub struct FleetMovementProgress { #[derive(Clone, Debug)] pub struct FleetPendingMovement { pub reservation: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub plan: crate::raft::member_state::RepairPlan, } @@ -69,7 +69,7 @@ pub struct FleetPendingMovement { #[derive(Clone, Debug)] pub struct FleetPendingReservation { pub reservation: [u8; 16], - pub group: cellar_logic::consensus::GroupScope, + pub group: cellar_logic::cell_group::GroupHistory, pub target: cellar_logic::consensus::NodeId, pub boot: crate::raft::fleet_stop::FleetBootBinding, pub peak: FleetResourceBounds, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs index d9156adbf..75fba7098 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/receive.rs @@ -23,7 +23,25 @@ impl FleetHostAuthorization { /// exclusion. A history descriptor or old member permit alone is insufficient. pub(crate) async fn admitted_learner( &self, - scope: &GroupScope, + scope: &GroupHistory, + timeout: Duration, + ) -> Result { + self.admitted_seed(scope, None, timeout).await + } + + pub(crate) async fn admitted_learner_from( + &self, + scope: &GroupHistory, + source: &cellar_logic::consensus::ReplicaIdentity, + timeout: Duration, + ) -> Result { + self.admitted_seed(scope, Some(source), timeout).await + } + + async fn admitted_seed( + &self, + scope: &GroupHistory, + source: Option<&cellar_logic::consensus::ReplicaIdentity>, timeout: Duration, ) -> Result { let catalog = &self.catalog; @@ -50,6 +68,22 @@ impl FleetHostAuthorization { { return Err(FleetCatalogError::Conflict); } + if let Some(source) = source { + if source.scope != *scope + || !row.seed_owners.as_ref().is_some_and(|owners| { + owners.iter().any(|owner| { + owner.member == source.member.get() + && owner.node == source.node.bytes() + }) + }) + { + return Err(FleetCatalogError::Conflict); + } + let boot = catalog.movement_boot(&snapshot, &row, source.node).await?; + if boot.disk != source.disk.bytes() || boot.process != source.process.bytes() { + return Err(FleetCatalogError::Conflict); + } + } Ok(FleetPermitAuthorization::admitted( permit, snapshot.revision(), @@ -131,35 +165,56 @@ impl FleetHostAuthorization { let authorization = catalog .authorize_membership(&snapshot, &scope, deadline.remaining()?) .await?; - let cell = match gateway.cell(scope.namespace, &scope.cell) { - Ok(cell) if cell.scope() == &scope => cell, - _ if matches!(command.action, MovementAction::Activate { .. }) => { - let group = catalog - .authorize_group(&snapshot, &scope, deadline.remaining()?) - .await?; - gateway - .register_managed_history(group, deadline.remaining()?) + if !matches!(command.action, MovementAction::Activate { .. }) { + let group = gateway + .group(scope) + .map_err(|_| FleetCatalogError::Conflict)?; + let history = catalog + .authorize_group(&snapshot, &scope, deadline.remaining()?) + .await?; + return match command.action { + MovementAction::Begin => group + .begin_managed_repair( + history, + authorization, + row.request.movement.as_ref().ok_or_else(invalid)?.plan()?, + deadline.remaining()?, + ) .await - .map_err(command_failure)? - } - _ => return Err(FleetCatalogError::Conflict), - }; + .map(|_| ()) + .map_err(control_failure), + MovementAction::Promote { revision } | MovementAction::Finish { revision } => { + let step = if matches!(command.action, MovementAction::Promote { .. }) { + RepairStep::PromoteAndRemove + } else { + RepairStep::FinishJoint + }; + group + .advance_managed_repair( + history, + authorization, + step, + LogIndex::new(revision), + deadline.remaining()?, + ) + .await + .map(|_| ()) + .map_err(control_failure) + } + MovementAction::Activate { .. } => Err(invalid()), + }; + } + let history = catalog + .authorize_group(&snapshot, &scope, deadline.remaining()?) + .await?; match command.action { - MovementAction::Begin => cell - .begin_managed_repair( - authorization, - row.request.movement.as_ref().ok_or_else(invalid)?.plan()?, - deadline.remaining()?, - ) - .await - .map(|_| ()) - .map_err(command_failure), MovementAction::Activate { source, source_node, from_s3, - } => cell - .activate_managed_learner( + } => gateway + .activate_managed_learner_group( + history, authorization, LearnerActivation { member: permit.replacement().map_err(|_| invalid())?.member, @@ -171,23 +226,74 @@ impl FleetHostAuthorization { ) .await .map(|_| ()) - .map_err(command_failure), - MovementAction::Promote { revision } | MovementAction::Finish { revision } => { - let step = if matches!(command.action, MovementAction::Promote { .. }) { - RepairStep::PromoteAndRemove - } else { - RepairStep::FinishJoint - }; - cell.advance_managed_repair( - authorization, - step, - LogIndex::new(revision), - deadline.remaining()?, - ) - .await - .map(|_| ()) - .map_err(command_failure) + .map_err(startup_failure), + _ => Err(invalid()), + } + } +} + +fn control_failure( + error: crate::application::consensus::CellGroupControlError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupControlError; + use crate::raft::completion::RuntimeError; + match error { + CellGroupControlError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupControlError::Refused(RuntimeError::Overloaded) => FleetCatalogError::Overloaded, + CellGroupControlError::Refused( + RuntimeError::Invalid(_) | RuntimeError::StaleCompletion, + ) => FleetCatalogError::Conflict, + CellGroupControlError::Refused(RuntimeError::Storage( + crate::raft::storage::LogError::Corrupt(_), + )) => FleetCatalogError::Corrupt("group repair state"), + CellGroupControlError::Refused(_) => FleetCatalogError::Unconfirmed, + } +} + +fn startup_failure( + error: crate::application::consensus::CellGroupStartupError, +) -> FleetCatalogError { + use crate::application::consensus::CellGroupStartupError; + use crate::raft::cell_groups::CellGroupControllerError as C; + match error { + // Entered recovery may have synced or attached even if its delivery failed. + CellGroupStartupError::Unconfirmed(_) => FleetCatalogError::Unconfirmed, + CellGroupStartupError::Refused(C::Runtime( + crate::raft::completion::RuntimeError::Overloaded, + )) => FleetCatalogError::Overloaded, + CellGroupStartupError::Refused(C::Catalog(error)) => { + use crate::raft::catalog::CatalogError as E; + match error.as_ref() { + E::Invalid(reason) => FleetCatalogError::Invalid(reason), + E::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + E::Limit => FleetCatalogError::Limit, + E::Log(crate::raft::storage::LogError::Corrupt(_)) | E::LostStableState => { + FleetCatalogError::Corrupt("learner group state") + } + E::Conflict(_) | E::ManagedAdmissionRequired | E::NotInitialMember | E::Fenced => { + FleetCatalogError::Conflict + } + E::FleetAuthority(authority) => match authority.as_ref() { + FleetCatalogError::Invalid(reason) => FleetCatalogError::Invalid(reason), + FleetCatalogError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), + FleetCatalogError::Overloaded => FleetCatalogError::Overloaded, + FleetCatalogError::Limit => FleetCatalogError::Limit, + FleetCatalogError::Conflict => FleetCatalogError::Conflict, + FleetCatalogError::Closed => FleetCatalogError::Closed, + // Preserve the complete typed cause for local diagnosis; + // transport delivery still classifies unknown outcomes. + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), + }, + _ => FleetCatalogError::Storage( + CellGroupStartupError::Refused(C::Catalog(error)).into(), + ), } } + CellGroupStartupError::Refused(C::Invalid(_)) => { + FleetCatalogError::Invalid("learner group capability") + } + CellGroupStartupError::Refused(_) => FleetCatalogError::Unconfirmed, } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs index d8226de47..1d971f9c3 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/isolated.rs @@ -316,7 +316,7 @@ impl IsolatedRepairService { .and_then(|n| n.checked_add(nodes.checked_mul(512 * 1024)?)) .ok_or(FleetCatalogError::Limit)?; let _memory = - super::topology::reserve_collection_memory(&observer.handle.control_budget, bytes, 0)?; + super::topology::reserve_collection_memory(observer.control_budget(), bytes, 0)?; let group = Group { history, voters, @@ -366,7 +366,7 @@ impl IsolatedRepairService { fn block( &self, - group: GroupScope, + group: GroupHistory, error: &FleetCatalogError, status: &watch::Sender, ) { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs index 146540a65..b467729fc 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/mod.rs @@ -49,7 +49,7 @@ pub trait FleetTopologyEvidence: Send + Sync { pub struct FleetTopologyServiceInputs { pub request: FleetTopologyRequest, - pub namespaces: Vec, + pub namespaces: Vec, } /// Local diagnostics, not a convergence, membership or physical-release proof. @@ -162,7 +162,7 @@ pub struct FleetIsolatedRepairServiceStatus { pub histories_checked: u64, pub blocked_histories: u64, pub inventory_exhausted: bool, - pub last_blocked_group: Option, + pub last_blocked_group: Option, pub last_blocked_failure: Option, } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs index 50f976894..dd3a105de 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/recovery.rs @@ -2,7 +2,7 @@ //! Durable failure exclusion precedes a retryable trusted platform effect. use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::disruption::Disruption; use crate::raft::fleet_stop::BootBinding; @@ -42,7 +42,7 @@ pub trait FleetFailureRecoveryPlatform: Send + Sync { fn namespaces<'a>( &'a self, timeout: Duration, - ) -> Pin>> + Send + 'a>>; + ) -> Pin>> + Send + 'a>>; fn isolate<'a>( &'a self, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs index d08513d01..9e12f0135 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/removal.rs @@ -4,30 +4,25 @@ //! the slot. A restarted worker repeats replay and current survivor checks. use super::recovery::Target; use super::*; -use crate::raft::archive::{ArchiveError, ArchiveLimits, ArchiveVerifier}; use crate::raft::fleet_catalog::deadline::Deadline; use crate::raft::fleet_catalog::{ - FleetEvacuationArchive, FleetEvacuationVerification, FleetHistoryState, - FleetIsolatedRemovalRequest, + FleetEvacuationArchive, FleetEvacuationVerification, FleetGroupArchiveVerifier, + FleetHistoryState, FleetIsolatedRemovalRequest, }; use cellar_logic::consensus::{NodeId, ProposalId}; -use cellar_ltx::LtxHost; use sha2::{Digest, Sha256}; /// Explicit replay ownership for automatic repair and terminal removal. The -/// embedding runtime owns the scratch store and its disk/memory limits. These -/// services grant no archive proof until physical verification succeeds. +/// embedding runtime owns shared private image inventories and replay pools. +/// These services grant no archive proof until complete physical replay succeeds. pub struct FleetFailureRecoveryOptions { pub detection: FleetFailureDetectionOptions, - pub archive_host: LtxHost, - pub archive_verifier: ArchiveVerifier, - pub archive_limits: ArchiveLimits, + pub archive: FleetGroupArchiveVerifier, } impl FleetFailureRecoveryOptions { pub(super) fn validate(&self) -> Result<()> { - self.detection.validate()?; - self.archive_limits.validate().map_err(archive_error) + self.detection.validate() } } @@ -39,9 +34,7 @@ struct Session { } pub(super) struct Removal { - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, + archive: FleetGroupArchiveVerifier, platform: Arc, session: Option, } @@ -52,9 +45,7 @@ impl Removal { platform: Arc, ) -> Self { Self { - host: options.archive_host, - verifier: options.archive_verifier, - limits: options.archive_limits, + archive: options.archive, platform, session: None, } @@ -135,50 +126,43 @@ impl Removal { .map_err(|_| FleetCatalogError::Corrupt("recovery removal scope"))?; let namespace = namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; - let birth = catalog - .inner - .domain - .run(namespace.birth(&scope.cell)) - .await - .map_err(|_| FleetCatalogError::Closed)?? - .ok_or(FleetCatalogError::InventoryIncomplete)?; - if birth.scope() != &scope || birth.digest() != history.birth_digest() { - return Err(FleetCatalogError::Conflict); - } - let archive = catalog - .inner - .domain - .run(async { - namespace.cell_archive(&birth, self.host.clone(), self.limits) - }) - .await - .map_err(|_| FleetCatalogError::Closed)??; - session + let fixed = session .walk .freeze_live(observer, nodes, deadline.remaining()?) .await?; - let metadata = catalog - .inner - .domain - .run(archive.recovery_metadata()) - .await - .map_err(|_| FleetCatalogError::Closed)? - .map_err(archive_error)? - .ok_or(FleetCatalogError::Unconfirmed)?; - let reference = metadata.reference(); - // The selected immutable reference must cover the fixed - // cut; archive publication may catch up on a later turn. + let registry = catalog + .run( + deadline.remaining()?, + self.archive + .open(&catalog.inner.domain, namespace, &history), + ) + .await?; + let hold = session + .walk + .acquire_archive_hold(®istry, deadline.remaining()?) + .await?; + let receiver = catalog + .run( + deadline.remaining()?, + self.archive.receiver( + &catalog.inner.domain, + fixed.root().route(), + namespace.namespace_config().image_storage, + ), + ) + .await?; session .walk .verify_live( observer, nodes, FleetEvacuationArchive { - archive: &archive, - reference: &reference, - verifier: &self.verifier, + registry: ®istry, + hold: &hold, + tails: self.archive.tails(), + receiver, }, deadline.remaining()?, ) @@ -263,15 +247,3 @@ fn request(target: &Target, revision: u64, candidate: [u8; 32]) -> FleetRequestI principal: "fleet-failure-recovery".into(), } } - -fn archive_error(error: ArchiveError) -> FleetCatalogError { - match error { - ArchiveError::Invalid(reason) => FleetCatalogError::Invalid(reason), - ArchiveError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), - ArchiveError::Conflict => FleetCatalogError::Conflict, - ArchiveError::Limit => FleetCatalogError::Limit, - ArchiveError::Unconfirmed | ArchiveError::Timeout => FleetCatalogError::Unconfirmed, - ArchiveError::Stopped => FleetCatalogError::Closed, - error => FleetCatalogError::Storage(error.into()), - } -} diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs index 80542b72b..dcec2be36 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/tests.rs @@ -10,10 +10,9 @@ use cellar_logic::consensus::{ fn pending(id: u8) -> FleetPendingMovement { FleetPendingMovement { reservation: [id; 16], - group: GroupScope { + group: GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: format!("Counter:{id}"), group: GroupId::new([id; 16]).unwrap(), incarnation: IncarnationId::new([3; 16]).unwrap(), }, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs index 6acfb5347..eba74a495 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/topology.rs @@ -245,7 +245,7 @@ impl TopologyService { ) .and_then(|n| n.checked_add(histories.checked_mul(4096)?)) .ok_or(FleetCatalogError::Limit)?; - let input = reserve_collection_memory(&observer.handle.control_budget, input_bytes, 0)?; + let input = reserve_collection_memory(observer.control_budget(), input_bytes, 0)?; let inputs = self .evidence .refresh(&operation, deadline.remaining()?) @@ -263,11 +263,8 @@ impl TopologyService { .checked_mul(28 * 1024) .and_then(|n| n.checked_add(actions.checked_mul(48)?)) .ok_or(FleetCatalogError::Limit)?; - let planning = reserve_collection_memory( - &observer.handle.control_budget, - planning_bytes, - input_bytes, - )?; + let planning = + reserve_collection_memory(observer.control_budget(), planning_bytes, input_bytes)?; let memory = Arc::new(vec![input, planning]); let verification = catalog .prepare_topology_verification( diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs index bfe84a8e0..bacbd7bc3 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/execution/service/worker.rs @@ -2,9 +2,10 @@ //! The node job set polls and joins this worker, including on physical closure. use super::*; -use crate::raft::budget::Permit; +use crate::application::consensus::group_client::{ + ControlServiceCompletion, ControlServiceKind, ControlServiceLease, +}; use crate::raft::fleet_catalog::deadline::Deadline; -use std::sync::atomic::Ordering; pub(crate) struct MovementServiceWork { catalog: FleetCatalog, @@ -12,16 +13,15 @@ pub(crate) struct MovementServiceWork { evidence: Arc, options: FleetMovementServiceOptions, stop: watch::Receiver, - done: watch::Sender, status: watch::Sender, schedule: schedule::Schedule, - admitted: bool, - memory: Option, node_stop: Option>, topology: Option, detection: Option, isolated: Option, recovery: Option, + _registration: Option, + _completion: ControlServiceCompletion, } impl MovementServiceWork { @@ -36,20 +36,19 @@ impl MovementServiceWork { ) -> Self { Self { catalog, - observer, + observer: observer.internal_observer(), evidence, options, stop, - done, status, schedule: schedule::Schedule::default(), - admitted: false, - memory: None, node_stop: None, topology: None, detection: None, isolated: None, recovery: None, + _registration: None, + _completion: ControlServiceCompletion::new(done), } } @@ -93,29 +92,15 @@ impl MovementServiceWork { if host.catalog.birth_owner()? != self.catalog.birth_owner()? { return Err(FleetCatalogError::Invalid("movement service fleet")); } - if self.observer.handle.closing.load(Ordering::Acquire) + if self.observer.is_closed() || self.observer.handle.admission.is_draining() || *self.stop.borrow() { return Err(FleetCatalogError::Closed); } - let mut slot = self + let registration = self .observer - .handle - .fleet_movement_service - .lock() - .map_err(|_| FleetCatalogError::Closed)?; - if slot.is_some() { - return Err(FleetCatalogError::Overloaded); - } - // A retained service plus one bounded catalog turn. Transport/physical - // work separately acquires its existing request, seed and image pools. - let memory = self - .observer - .handle - .control_budget - .acquire(2 * ROOT_LIMIT) - .ok_or(FleetCatalogError::Overloaded)?; + .register_control_service(ControlServiceKind::Movement, 2 * ROOT_LIMIT)?; // Use the enrolled node's installed client, never a caller's alternate // backend with the same descriptive fleet identity. Periodic I/O has a // private turn under the node's domain, so it cannot occupy foreground @@ -131,20 +116,17 @@ impl MovementServiceWork { turn: Mutex::new(()), }), }; - let (stop, wait) = watch::channel(false); - *slot = Some(stop); - self.memory = Some(memory); + self.node_stop = Some(registration.stop_receiver()); + self._registration = Some(registration); if let Some(detection) = &mut self.detection { detection.set_origin(host.node()); } if let Some(recovery) = &mut self.recovery { recovery.detection.set_origin(host.node()); } - self.admitted = true; self.status.send_modify(|status| status.running = true); // Keep public cancellation and node drain independent. The node-owned // receiver is selected for the worker's entire lifetime, including I/O. - self.node_stop = Some(wait); Ok(()) } @@ -169,9 +151,7 @@ impl MovementServiceWork { Ok(now) => now, Err(_) => return, }; - if self.observer.handle.closing.load(Ordering::Acquire) - || self.observer.handle.admission.is_draining() - { + if self.observer.is_closed() || self.observer.handle.admission.is_draining() { return; } self.status @@ -398,13 +378,6 @@ impl MovementServiceWork { impl Drop for MovementServiceWork { fn drop(&mut self) { - if self.admitted { - if let Ok(mut slot) = self.observer.handle.fleet_movement_service.lock() { - slot.take(); - } - } - self.memory.take(); self.status.send_modify(|status| status.running = false); - self.done.send_replace(true); } } diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs index edc75cece..e8d369d32 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/admission.rs @@ -12,7 +12,7 @@ impl FleetCatalog { &self, snapshot: &FleetSnapshot, request: &FleetTopologyIntentRequest, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs index f9916df4a..8e0cb8365 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/mod.rs @@ -10,7 +10,7 @@ mod records; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::ProposalId; use records::{Definition, IntentRecord, Policy}; use std::collections::BTreeSet; @@ -60,7 +60,7 @@ pub struct FleetTopologyOperation { #[derive(Clone, Debug)] pub struct FleetTopologyMovementEntry { pub reservation: [u8; 16], - pub group: GroupScope, + pub group: GroupHistory, pub source: NodeId, pub target: NodeId, pub admitted_revision: u64, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs b/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs index 9f7a0bfd0..41e9fbd56 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/intent/progress.rs @@ -222,7 +222,7 @@ impl FleetCatalog { preview: &FleetTopologyPreview, identity: FleetRequestIdentity, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result> { self.check_snapshot(&preview.snapshot)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs index 27a00bc6a..993aa9794 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/membership.rs @@ -43,7 +43,7 @@ impl FleetCatalog { // Called within an owned catalog turn. Return only the configuration pinned // by immutable birth/permit/settlement journals, never a current-health lease. - pub(in crate::raft::fleet_catalog) async fn settled_group_voters( + pub(crate) async fn settled_group_voters( &self, snapshot: &FleetSnapshot, history: &super::super::histories::FleetHistory, diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs index b174d374f..c21d19db2 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/mod.rs @@ -68,7 +68,8 @@ pub use movement::{FleetMovementAdmission, FleetMovementRequest}; use super::*; use crate::raft::fleet_stop::BootBinding; -use cellar_logic::consensus::{GroupScope, NodeId}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::NodeId; use cellar_logic::raft_fleet::{LedgerError, Resources}; use serde::Deserialize; @@ -140,7 +141,7 @@ impl FleetResourceAttestation { #[derive(Clone, Debug, PartialEq, Eq)] pub struct FleetResourceRequest { pub identity: FleetRequestIdentity, - pub group: GroupScope, + pub group: GroupHistory, pub attestation: FleetResourceAttestation, /// Conservative incremental learner bound, including seed, image, tail /// and worker obligations. Actual host admission can still refuse it. diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs b/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs index 955de3873..81076a500 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/movement.rs @@ -172,7 +172,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetMovementRequest, observer: &RaftCells, - namespaces: &[crate::raft::catalog::NamespaceCatalog], + namespaces: &[crate::raft::catalog::RangeDirectory], timeout: Duration, ) -> Result { self.reserve_movement_inner(snapshot, request, observer, namespaces, false, timeout) @@ -184,7 +184,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetMovementRequest, observer: &RaftCells, - namespaces: &[crate::raft::catalog::NamespaceCatalog], + namespaces: &[crate::raft::catalog::RangeDirectory], topology: bool, timeout: Duration, ) -> Result { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs index 4b5546441..789a5f191 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/membership.rs @@ -62,9 +62,7 @@ impl FleetCatalog { .current_enrollment .as_ref() .ok_or(FleetCatalogError::Unconfirmed)?; - let published = sample - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let published = sample.published_cut(); let expected = group.voters.values().copied().collect::>(); let settled = match &group.latest { None => { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs index 039e02c38..f49d388fa 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/mod.rs @@ -5,7 +5,7 @@ mod membership; mod planning; use super::*; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use cellar_logic::consensus::ProposalId; pub(super) use planning::Planning; use std::pin::Pin; @@ -14,7 +14,7 @@ use std::pin::Pin; pub struct FleetIsolatedRepairContext { pub operation: ProposalId, pub failed_boot: BootBinding, - pub group: GroupScope, + pub group: GroupHistory, } /// Conservative deployment inputs for one history and at most 64 candidate or @@ -23,7 +23,7 @@ pub struct FleetIsolatedRepairContext { pub struct FleetIsolatedRepairInputs { pub history: FleetTopologyHistoryResources, pub nodes: Vec, - pub namespaces: Vec, + pub namespaces: Vec, } /// The owned service cancels/joins the callback future with its original round @@ -54,7 +54,7 @@ impl FleetCatalog { cut: &FleetSnapshot, selected: &Selected, observer: &crate::application::consensus::RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { let mut canonical = super::records::Request::new(self, &selected.request.resources)?; diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs b/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs index d20cf1333..8ede92802 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/repair/planning.rs @@ -170,11 +170,7 @@ impl Planning<'_, '_> { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)?; - if inventory - .history(&context.group) - .map_err(|_| FleetCatalogError::Corrupt("isolated repair physical scope"))? - .is_some() - { + if inventory.group_history(&context.group).is_some() { occupied.insert(node); } let observed_ms = catalog @@ -203,10 +199,8 @@ impl Planning<'_, '_> { if let Some(initializing) = inventory.initializing() { let mut attributed = false; for reservation in &charged.reservations { - attributed |= - crate::raft::physical_inventory::scope_digest(&reservation.group).map_err( - |_| FleetCatalogError::Corrupt("isolated repair initializer scope"), - )? == initializing; + attributed |= crate::raft::physical_inventory::group_digest(&reservation.group) + == initializing; } if !attributed { if group.voters.contains_key(&node) { diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs index c6763d161..85c4d1333 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/inputs.rs @@ -115,11 +115,8 @@ impl FleetTopologyVerification { let live = self .groups .iter() - .map(|group| { - crate::raft::physical_inventory::scope_digest(&group.resources.group) - .map_err(|_| FleetCatalogError::Corrupt("topology scope digest")) - }) - .collect::>>()?; + .map(|group| crate::raft::physical_inventory::group_digest(&group.resources.group)) + .collect::>(); for node in &self.nodes { let original = node .physical diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs index 787c2ba14..a3011b04d 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/membership.rs @@ -43,10 +43,7 @@ impl FleetTopologyVerification { .chain(&history.learners) .any(|m| m.node == input.host.node().bytes()) || voters.contains_key(&input.host.node())) - && inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology genesis"))? - .is_none() + && inventory.group_history(&scope).is_none() { return Err(FleetCatalogError::InventoryIncomplete); } @@ -83,9 +80,7 @@ impl FleetTopologyVerification { )?; let proof = sample.membership(); let actual = proof.configuration(); - let published = sample - .published_cut() - .ok_or(FleetCatalogError::Unconfirmed)?; + let published = sample.published_cut(); let current = sample .node() .current_enrollment @@ -165,9 +160,7 @@ impl FleetTopologyVerification { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)?; - let physical = inventory - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology physical binding"))?; + let physical = inventory.group_history(&scope); if (history .voters .iter() diff --git a/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs b/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs index 1c7d77152..5ca515325 100644 --- a/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs +++ b/crates/cellar/durability/raft/fleet_catalog/resources/topology/mod.rs @@ -7,7 +7,7 @@ mod membership; use super::*; use crate::application::consensus::RaftCells; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::host::RaftPeerNodeObservation; use crate::raft::member_state::{JoiningMember, RepairPlan}; use cellar_logic::consensus::{MemberId, ProposalId}; @@ -40,7 +40,7 @@ pub struct FleetTopologyNodeResources { /// values are not measurements of image size or estimated command throughput. #[derive(Clone, Debug)] pub struct FleetTopologyHistoryResources { - pub group: GroupScope, + pub group: GroupHistory, pub resident: FleetResourceBounds, pub peak: FleetResourceBounds, } @@ -183,7 +183,7 @@ impl FleetCatalog { snapshot: &FleetSnapshot, request: &FleetTopologyRequest, observer: &RaftCells, - namespaces: &[NamespaceCatalog], + namespaces: &[RangeDirectory], timeout: Duration, ) -> Result { self.check_snapshot(snapshot)?; @@ -499,8 +499,7 @@ impl FleetTopologyVerification { .inventory .as_ref() .ok_or(FleetCatalogError::InventoryIncomplete)? - .history(&scope) - .map_err(|_| FleetCatalogError::Corrupt("topology physical scope"))? + .group_history(&scope) .is_some() { occupied.push(node.host.node()); diff --git a/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs b/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs index 11546b960..58f3259b7 100644 --- a/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs +++ b/crates/cellar/durability/raft/fleet_inspection_auth_tests.rs @@ -9,12 +9,11 @@ fn request() -> RequestHead { route.identities().unwrap(); let binding = crate::raft::member_state::FleetGroupBinding::new( cellar_logic::consensus::IncarnationId::new([80; 16]).unwrap(), - &cellar_logic::consensus::GroupScope { + &cellar_logic::cell_group::GroupHistory { cluster: cellar_logic::consensus::ClusterId::new([1; 16]).unwrap(), namespace: cellar_logic::consensus::NamespaceId::new([2; 16]).unwrap(), group: cellar_logic::consensus::GroupId::new([3; 16]).unwrap(), incarnation: cellar_logic::consensus::IncarnationId::new([4; 16]).unwrap(), - cell: "Counter:quorum".into(), }, [81; 32], ) diff --git a/crates/cellar/durability/raft/fleet_inspection_transport.rs b/crates/cellar/durability/raft/fleet_inspection_transport.rs index 667990ba3..c7f18cf26 100644 --- a/crates/cellar/durability/raft/fleet_inspection_transport.rs +++ b/crates/cellar/durability/raft/fleet_inspection_transport.rs @@ -279,7 +279,7 @@ impl State { .checked_add(request.timeout_ms) .ok_or(TransportError::Limit)?; let handle = self - .query_gateway + .fleet_gateway .lock() .map_err(|_| TransportError::Closed)? .as_ref() @@ -290,7 +290,7 @@ impl State { .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) .filter(|ms| *ms > 0) .ok_or(TransportError::Timeout)?; - let work = RaftCells { handle }.observe_local_fleet_membership( + let work = RaftCells::from_internal_handle(handle).observe_local_fleet_membership( request.history.clone(), request.route.minimum_revision()?, std::time::Duration::from_millis(remaining), ); let mut extra = [0; 1]; diff --git a/crates/cellar/durability/raft/fleet_inspection_wire.rs b/crates/cellar/durability/raft/fleet_inspection_wire.rs index 402b4686c..67d08f07b 100644 --- a/crates/cellar/durability/raft/fleet_inspection_wire.rs +++ b/crates/cellar/durability/raft/fleet_inspection_wire.rs @@ -1,24 +1,24 @@ -//! Quorum metadata binds one rooted history to one authenticated node route. -//! The body reuses the bounded node and membership inspection codecs. +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Authenticated fresh quorum metadata for a complete physical group. The body +//! never chooses a cell image and cannot be decoded as an ordinary inspection. +use super::cell_groups::{CellGroupQuorumMembership, CellGroupStatus, GroupManifest}; use super::fleet_catalog::FleetGroupAuthorization; use super::host::{FleetMembershipObservation, FleetPublishedCut, RaftHostObservation}; -use super::inspection::{CellInspection, InspectionSnapshot}; -use super::member_state::FleetGroupBinding; +use super::inspection::{GroupInspection, LeaderLoad, MemberInspection}; +use super::member_state::{FleetGroupBinding, GroupMembership}; use super::transport::TransportError; use super::{inspection_wire, node_inspection_wire}; -use cellar_logic::consensus::{GroupScope, LogIndex}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{EngineInstanceId, LogIndex, LogPosition, MemberId, ReplicaIdentity, Term}, +}; +use raft::prelude::ConfState; use serde::{Deserialize, Serialize}; -#[path = "fleet_cut_wire.rs"] -mod publication; - -pub(crate) const MAX_BODY: usize = - 20 + node_inspection_wire::MAX_BODY + inspection_wire::MAX_BODY + publication::MAX_BODY; -pub(crate) const RESPONSE_BYTES: usize = 128 * 1024; - -#[cfg(test)] -#[path = "fleet_inspection_wire_tests.rs"] -mod tests; +pub(crate) const MAX_BODY: usize = 12 + node_inspection_wire::MAX_BODY + inspection_wire::MAX_BODY; +pub(crate) const RESPONSE_BYTES: usize = super::physical_inventory::RESPONSE_BYTES; +const DOMAIN: &str = "cellar-fleet-group-quorum-1"; #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -38,7 +38,7 @@ impl History { } } - pub(crate) fn scope(&self) -> Result { + pub(crate) fn scope(&self) -> Result { if self.birth == [0; 32] || self.schema == [0; 32] { return Err(TransportError::Identity); } @@ -50,33 +50,90 @@ impl History { } } +#[derive(Clone)] +pub(crate) struct GroupSnapshot { + pub replica: ReplicaIdentity, + pub engine: EngineInstanceId, + pub root: GroupManifest, + pub barrier: LogPosition, + pub membership: GroupMembership, + pub configuration: ConfState, + pub status: CellGroupStatus, + pub load: Option, +} + +impl GroupSnapshot { + pub(crate) fn capture(selected: &CellGroupQuorumMembership) -> Self { + Self { + replica: selected.replica().clone(), + engine: selected.engine(), + root: selected.root().clone(), + barrier: selected.barrier(), + membership: selected.membership().clone(), + configuration: selected.configuration().clone(), + status: selected.status().clone(), + load: None, + } + } + + pub(crate) fn validate(&self) -> Result<(), TransportError> { + self.root.check().map_err(|_| TransportError::Protocol)?; + self.membership + .validate(&self.configuration, self.root.position().index.get()) + .map_err(|_| TransportError::Protocol)?; + self.membership + .validate_scope(&self.replica.scope) + .map_err(|_| TransportError::Identity)?; + let fleet = self + .membership + .fleet_status() + .ok_or(TransportError::Identity)?; + let leader = self + .status + .leader + .and_then(|id| MemberId::new(id).ok()) + .ok_or(TransportError::Protocol)?; + if self.root.route().history != self.replica.scope + || !fleet.binding.matches_scope(&self.replica.scope) + || self.root.membership_revision() != self.membership.revision().get() + || self.barrier.index.get() == 0 + || self.barrier.term.get() == 0 + || self.barrier.index < self.membership.revision() + || self.root.position().index < self.barrier.index + || self.root.position().term != self.barrier.term + || self.status.term != self.barrier.term + || self.status.applied != self.root.position() + || self.status.committed < self.root.position().index + || !self.membership.permits_group(&self.replica) + || !(self.configuration.voters.contains(&leader.get()) + || self.configuration.voters_outgoing.contains(&leader.get())) + || self.status.ready && self.status.leader != Some(self.replica.member.get()) + || !self.root.is_active() + { + return Err(TransportError::Identity); + } + Ok(()) + } +} + pub(crate) struct LocalObservation { pub node: Box, - pub snapshot: Box, - pub barrier: LogIndex, + pub snapshot: GroupSnapshot, } impl LocalObservation { - fn validate(&self) -> Result<(), TransportError> { + pub(crate) fn validate(&self) -> Result<(), TransportError> { node_inspection_wire::validate(&self.node)?; - self.snapshot - .validate() - .map_err(|_| TransportError::Protocol)?; - let view = &self.snapshot.view; - if view.node.identity != self.node.node.identity - || view.node.storage_instance != self.node.node.storage_instance - || self.barrier.get() == 0 - || self.snapshot.publication.is_none() - || view - .member - .as_ref() - .is_none_or(|m| m.applied.index < self.barrier) - || self - .snapshot - .membership - .as_ref() - .and_then(|m| m.fleet_status()) - .is_none() + self.snapshot.validate()?; + let s = &self.snapshot; + if s.replica.node != self.node.node.identity.node + || s.replica.disk != self.node.node.identity.disk + || s.replica.process != self.node.node.identity.process + || s.engine != self.node.node.storage_instance + || s.load.as_ref().is_some_and(|load| { + load.capacity != self.node.node.profile.leader_capacity + || load.leaders > self.node.node.groups + }) { return Err(TransportError::Identity); } @@ -87,97 +144,192 @@ impl LocalObservation { self, authorization: &FleetGroupAuthorization, sample_age_ms: u64, - ) -> Result { + ) -> Result { self.validate()?; - let membership = FleetMembershipObservation::from_quorum_snapshot( - &self.snapshot, - authorization, - self.barrier, - ) - .map_err(|_| TransportError::Identity)?; - let published = FleetPublishedCut::capture( - &membership, - self.node.node.storage_instance, - self.snapshot - .publication - .as_ref() - .ok_or(TransportError::Protocol)?, - ); - Ok(super::host::RaftPeerFleetMembershipObservation::new( - *self.node, + let s = self.snapshot; + let progress = GroupInspection { + history: s.replica.scope, + schema_digest: s.root.route().schema_digest, + birth_digest: s.root.route().birth_digest, + node: self.node.node.clone(), + member: Some(MemberInspection { + member: s.replica.member, + term: s.status.term, + leader: s + .status + .leader + .map(|id| MemberId::new(id).map_err(|_| TransportError::Protocol)) + .transpose()?, + is_leader: s.status.leader == Some(s.replica.member.get()), + membership_revision: s.membership.revision(), + member_high_water: s.membership.high_water(), + committed: s.status.committed, + applied: s.status.applied, + published: Some(s.root.position()), + repair: s + .membership + .repair_status() + .map_err(|_| TransportError::Protocol)?, + sample_age_ms: 0, + }), + leader_load: s.load.clone(), + }; + let engine = s.engine; + let membership = FleetMembershipObservation::from_quorum_group(s, authorization) + .map_err(|_| TransportError::Identity)?; + let published = FleetPublishedCut::capture_group(&membership, engine); + let mut observed = RaftPeerFleetMembershipObservation { + node: *self.node, membership, - self.snapshot.view, + progress, published, - sample_age_ms, - )) + sample_age_ms: 0, + }; + observed + .record_delivery_age(sample_age_ms) + .map_err(|_| TransportError::Limit)?; + Ok(observed) } } +fn nullable<'de, T: Deserialize<'de>, D: serde::Deserializer<'de>>( + decoder: D, +) -> Result, D::Error> { + Option::deserialize(decoder) +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Body { + domain: String, + root: GroupManifest, + member: String, + barrier_index: String, + barrier_term: String, + committed: String, + #[serde(deserialize_with = "nullable")] + leader: Option, + ready: bool, + membership: GroupMembership, + configuration: Vec, + #[serde(deserialize_with = "nullable")] + load: Option, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct Load { + leaders: u32, + capacity: u32, + age: u64, +} + pub(crate) fn encode(observation: &LocalObservation) -> Result, TransportError> { observation.validate()?; let node = node_inspection_wire::encode(&observation.node)?; - let group = inspection_wire::encode(&observation.snapshot)?; - let mut bytes = Vec::with_capacity(20 + node.len() + group.len()); - bytes.extend_from_slice(&1_u32.to_be_bytes()); - bytes.extend_from_slice(&(node.len() as u32).to_be_bytes()); - bytes.extend_from_slice(&(group.len() as u32).to_be_bytes()); - bytes.extend_from_slice(&observation.barrier.get().to_be_bytes()); + let s = &observation.snapshot; + let group = serde_json::to_vec(&Body { + domain: DOMAIN.into(), + root: s.root.clone(), + member: s.replica.member.get().to_string(), + barrier_index: s.barrier.index.get().to_string(), + barrier_term: s.barrier.term.get().to_string(), + committed: s.status.committed.get().to_string(), + leader: s.status.leader.map(|id| id.to_string()), + ready: s.status.ready, + membership: s.membership.clone(), + configuration: super::manifest::encode_conf(&s.configuration) + .map_err(|_| TransportError::Protocol)?, + load: s.load.as_ref().map(|l| Load { + leaders: l.leaders, + capacity: l.capacity, + age: l.sample_age_ms, + }), + }) + .map_err(|_| TransportError::Protocol)?; + if group.len() > inspection_wire::MAX_BODY { + return Err(TransportError::Limit); + } + let mut bytes = Vec::with_capacity(12 + node.len() + group.len()); + for word in [1, node.len() as u32, group.len() as u32] { + bytes.extend_from_slice(&word.to_be_bytes()); + } bytes.extend_from_slice(&node); bytes.extend_from_slice(&group); - bytes.extend_from_slice(&publication::encode( - observation - .snapshot - .publication - .as_ref() - .ok_or(TransportError::Protocol)?, - )); Ok(bytes) } pub(crate) fn decode(bytes: &[u8]) -> Result { - if bytes.len() < 20 || bytes.len() > MAX_BODY { + if bytes.len() < 12 || bytes.len() > MAX_BODY { return Err(TransportError::Limit); } - // The fixed prefix is checked above before reading any word or lengths. - let word = |start| { - u32::from_be_bytes([ - bytes[start], - bytes[start + 1], - bytes[start + 2], - bytes[start + 3], - ]) - }; - let node = word(4) as usize; - let group = word(8) as usize; + let word = |i| u32::from_be_bytes([bytes[i], bytes[i + 1], bytes[i + 2], bytes[i + 3]]); + let (node_len, group_len) = (word(4) as usize, word(8) as usize); if word(0) != 1 - || node > node_inspection_wire::MAX_BODY - || group > inspection_wire::MAX_BODY - || 20 + node + group >= bytes.len() - || bytes.len() - (20 + node + group) > publication::MAX_BODY + || node_len > node_inspection_wire::MAX_BODY + || group_len > inspection_wire::MAX_BODY + || 12 + node_len + group_len != bytes.len() { return Err(TransportError::Protocol); } - let mut snapshot = inspection_wire::decode(&bytes[20 + node..20 + node + group])?; - snapshot.publication = Some(publication::decode(&bytes[20 + node + group..])?); - let observation = LocalObservation { - node: node_inspection_wire::decode(&bytes[20..20 + node])?, - snapshot, - barrier: LogIndex::new(u64::from_be_bytes([ - bytes[12], bytes[13], bytes[14], bytes[15], bytes[16], bytes[17], bytes[18], bytes[19], - ])), + let node = node_inspection_wire::decode(&bytes[12..12 + node_len])?; + let body: Body = + serde_json::from_slice(&bytes[12 + node_len..]).map_err(|_| TransportError::Protocol)?; + if body.domain != DOMAIN || body.configuration.len() > 4096 { + return Err(TransportError::Protocol); + } + let positive = |value: &str| { + let n = super::transport_frame::decimal(value)?; + if n == 0 || n >= u64::MAX - 2 { + return Err(TransportError::Protocol); + } + Ok(n) + }; + let barrier = LogPosition { + index: LogIndex::new(positive(&body.barrier_index)?), + term: Term::new(positive(&body.barrier_term)?), + }; + let leader = body.leader.as_deref().map(positive).transpose()?; + let snapshot = GroupSnapshot { + replica: ReplicaIdentity { + scope: body.root.route().history, + member: MemberId::new(positive(&body.member)?).map_err(|_| TransportError::Identity)?, + node: node.node.identity.node, + disk: node.node.identity.disk, + process: node.node.identity.process, + }, + engine: node.node.storage_instance, + status: CellGroupStatus { + term: barrier.term, + leader, + applied: body.root.position(), + committed: LogIndex::new(positive(&body.committed)?), + ready: body.ready, + }, + root: body.root, + barrier, + membership: body.membership, + configuration: super::manifest::decode_conf(&body.configuration) + .map_err(|_| TransportError::Protocol)?, + load: body.load.map(|l| LeaderLoad { + leaders: l.leaders, + capacity: l.capacity, + sample_age_ms: l.age, + }), }; + let observation = LocalObservation { node, snapshot }; observation.validate()?; Ok(observation) } -/// Opaque authenticated result. Keeping it does not extend its freshness or -/// confer placement, reservation-release or removal authority. +/// Opaque result from the purpose-specific authenticated safe-ReadIndex lane. +/// Retaining it neither extends freshness nor permits mutation or reclamation. #[derive(Clone, Debug)] pub struct RaftPeerFleetMembershipObservation { node: RaftHostObservation, membership: FleetMembershipObservation, - progress: CellInspection, - published: Option, + progress: GroupInspection, + published: FleetPublishedCut, sample_age_ms: u64, } @@ -187,38 +339,24 @@ impl RaftPeerFleetMembershipObservation { age: u64, ) -> Result<(), super::inspection::InspectionError> { use super::inspection::InspectionError; - self.sample_age_ms = self.sample_age_ms.max(age); + let next = self.sample_age_ms.max(age); + let additional = next - self.sample_age_ms; if let Some(member) = &mut self.progress.member { member.sample_age_ms = member .sample_age_ms - .checked_add(self.sample_age_ms) + .checked_add(additional) .ok_or(InspectionError::Unavailable)?; } if let Some(load) = &mut self.progress.leader_load { load.sample_age_ms = load .sample_age_ms - .checked_add(self.sample_age_ms) + .checked_add(additional) .ok_or(InspectionError::Unavailable)?; } + self.sample_age_ms = next; Ok(()) } - pub(crate) fn new( - node: RaftHostObservation, - membership: FleetMembershipObservation, - progress: CellInspection, - published: Option, - sample_age_ms: u64, - ) -> Self { - Self { - node, - membership, - progress, - published, - sample_age_ms, - } - } - pub fn node(&self) -> &RaftHostObservation { &self.node } @@ -227,20 +365,20 @@ impl RaftPeerFleetMembershipObservation { &self.membership } - pub fn progress(&self) -> &CellInspection { + pub fn progress(&self) -> &GroupInspection { &self.progress } - /// A fixed data/configuration cut only when publication matched durable - /// apply at capture. None means publication is pending; an empty cut is an - /// explicit Some value whose image() is None. Neither supplies archive or - /// current-health proof. - pub fn published_cut(&self) -> Option<&FleetPublishedCut> { - self.published.as_ref() + /// Atomic complete-root publication. The zero-cell root is explicit. + pub fn published_cut(&self) -> &FleetPublishedCut { + &self.published } - /// Whole invocation through SDK delivery, including retained reply time. pub fn sample_age_ms(&self) -> u64 { self.sample_age_ms } } + +#[cfg(test)] +#[path = "fleet_inspection_wire_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs b/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs index 946dd8740..25bef2ecf 100644 --- a/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs +++ b/crates/cellar/durability/raft/fleet_inspection_wire_tests.rs @@ -1,281 +1,180 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; -use crate::raft::fleet_catalog::FleetNodeState; -use crate::raft::host::RaftHostEnrollmentObservation; -use crate::raft::inspection::*; -use crate::raft::member_state::GroupMembership; -use cellar_logic::consensus::*; +use cellar_logic::{ + cell_group::{CellRange, GroupRoute, RouteEpoch}, + consensus::*, +}; -fn snapshot() -> InspectionSnapshot { - let conf = ::raft::prelude::ConfState { - voters: vec![1, 2, 3], - ..Default::default() - }; - let scope = GroupScope { +fn observation(cells: bool) -> LocalObservation { + let mut node = super::super::node_inspection_wire::tests::view(); + node.node.groups = 1; + node.node.reserved_bytes = node.node.profile.bytes_per_group; + node.registered_histories = 1; + let history = GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:inspection".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; + let route = GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [6; 32], + schema_digest: [5; 32], + }; + let mut root = serde_json::to_value(GroupManifest::genesis(route).unwrap()).unwrap(); + root["index"] = serde_json::json!(5); + root["term"] = serde_json::json!(2); + if cells { + root["root"] = serde_json::json!({"digest":vec![7;32],"height":2,"cells":2,"last_mutation_index":4,"last_mutation_term":2,"next_due_unix_millis":null}); + } + let root: GroupManifest = serde_json::from_value(root).unwrap(); + let conf = ConfState { + voters: vec![1, 2, 3], + ..Default::default() + }; let binding = - FleetGroupBinding::new(IncarnationId::new([80; 16]).unwrap(), &scope, [81; 32]).unwrap(); + FleetGroupBinding::new(IncarnationId::new([80; 16]).unwrap(), &history, [81; 32]).unwrap(); let membership = GroupMembership::managed_initial(&conf, binding).unwrap(); - InspectionSnapshot { - view: CellInspection { - scope: cellar_logic::consensus::GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:inspection".into(), - group: GroupId::new([3; 16]).unwrap(), - incarnation: IncarnationId::new([4; 16]).unwrap(), - }, - schema_digest: [5; 32], - birth_digest: [6; 32], - node: NodeInspection { - identity: NodeIdentity { - node: NodeId::new([7; 16]).unwrap(), - disk: DiskLineage::new([8; 16]).unwrap(), - process: ProcessGeneration::new([9; 16]).unwrap(), - }, - storage_instance: EngineInstanceId::new([10; 16]).unwrap(), - profile: RaftNodeProfile { - pool: "native".into(), - region: "local".into(), - zone: "a".into(), - group_capacity: 8, - byte_capacity: 8192, - bytes_per_group: 1024, - leader_capacity: 8, - draining: false, - }, - groups: 1, - reserved_bytes: 1024, - }, - occupancy: ReplicaOccupancy::Member(MemberId::new(1).unwrap()), - member: Some(MemberInspection { - member: MemberId::new(1).unwrap(), - term: Term::new(2), - leader: Some(MemberId::new(1).unwrap()), - is_leader: true, - membership_revision: LogIndex::new(1), - member_high_water: 3, - committed: LogIndex::new(3), - applied: LogPosition { - index: LogIndex::new(2), - term: Term::new(1), - }, - published: Some(LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }), - repair: None, - sample_age_ms: 17, - }), - leader_load: None, + let snapshot = GroupSnapshot { + replica: ReplicaIdentity { + scope: history, + member: MemberId::new(1).unwrap(), + node: node.node.identity.node, + disk: node.node.identity.disk, + process: node.node.identity.process, }, - membership: Some(membership), - configuration: Some(conf), - publication: Some(crate::raft::host::Publication::Image( - crate::raft::host::FleetPublishedImage::new( - LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }, - cellar_ltx::apply::ImageState { - epoch: 1, - txid: cellar_ltx::TXID(1), - checksum: 0, - page_size: 4096, - length: 4096, - digest: [7; 32], - }, - ) - .unwrap(), - )), - } -} - -fn observation() -> LocalObservation { - let snapshot = Box::new(snapshot()); - let enrollment = RaftHostEnrollmentObservation { - fleet: [80; 16], - revision: 3, - epoch: 2, - physical_host: IncarnationId::new([90; 16]).unwrap(), - state: FleetNodeState::Draining, - capability_digest: [9; 32], - }; - let node = Box::new(RaftHostObservation { - fleet_boot: Some(super::super::fleet_stop::BootBinding { - catalog: [81; 32], - fleet: [80; 16], - node: snapshot.view.node.identity.node.bytes(), - disk: snapshot.view.node.identity.disk.bytes(), - process: snapshot.view.node.identity.process.bytes(), - engine: snapshot.view.node.storage_instance.bytes(), - boot: [41; 16], - epoch: 1, - join_key: crate::peer_probe::PeerProbeSigner::from_secret([4; 32]).public_key_bytes(), - }), - startup_bounds: crate::raft::host::RaftHostStartupBounds { - host_groups: 4096, - consensus_groups: 4096, - engine_groups: 4096, - bounded_buffer_bytes: 1024, + engine: node.node.storage_instance, + barrier: LogPosition { + index: LogIndex::new(5), + term: Term::new(2), + }, + status: CellGroupStatus { + term: Term::new(2), + leader: Some(1), + applied: root.position(), + committed: LogIndex::new(5), + ready: true, }, - node: snapshot.view.node.clone(), - startup_enrollment: Some(enrollment.clone()), - current_enrollment: Some(enrollment), - fleet_trust_revision: Some(3), - declared_namespaces: 1, - registered_histories: 1, - dormant_histories: 0, - pending_history_registrations: 0, - unattached_engine_histories: 0, - seed_workers: 0, - }); + root, + membership, + configuration: conf, + load: None, + }; LocalObservation { - node, + node: Box::new(node), snapshot, - barrier: LogIndex::new(2), } } #[test] -fn fleet_quorum_wire_retains_exact_configuration_and_publication() { - let view = observation(); - let restored = decode(&encode(&view).unwrap()).unwrap(); - assert_eq!(restored.node, view.node); - assert_eq!(restored.barrier, view.barrier); - assert_eq!(restored.snapshot.view, view.snapshot.view); - assert_eq!(restored.snapshot.membership, view.snapshot.membership); - assert_eq!(restored.snapshot.configuration, view.snapshot.configuration); - assert_eq!(restored.snapshot.publication, view.snapshot.publication); - assert!( - restored - .snapshot - .view - .member - .unwrap() - .published - .unwrap() - .index - < restored.barrier - ); +fn fleet_quorum_body_roundtrips_complete_group_roots_and_explicit_empty() { + for cells in [false, true] { + let observed = observation(cells); + let bytes = encode(&observed).unwrap(); + let decoded = decode(&bytes).unwrap(); + assert_eq!(*decoded.node, *observed.node); + assert_eq!(decoded.snapshot.root, observed.snapshot.root); + assert_eq!( + decoded.snapshot.root.root().cells(), + if cells { 2 } else { 0 } + ); + assert_eq!(decoded.snapshot.membership, observed.snapshot.membership); + assert_eq!( + decoded.snapshot.configuration, + observed.snapshot.configuration + ); + assert_eq!(decoded.snapshot.barrier, observed.snapshot.barrier); + assert!(bytes.len() <= MAX_BODY); + assert!(decoded.snapshot.load.is_none()); + } } #[test] -fn fleet_cut_wire_requires_explicit_coherent_publication() { - use crate::raft::host::Publication; - let mut view = observation(); - let valid = encode(&view).unwrap(); - let ordinary = crate::raft::inspection_wire::decode( - &crate::raft::inspection_wire::encode(&view.snapshot).unwrap(), - ) - .unwrap(); - assert!(ordinary.publication.is_none()); - let node = u32::from_be_bytes(valid[4..8].try_into().unwrap()) as usize; - let group = u32::from_be_bytes(valid[8..12].try_into().unwrap()) as usize; - let start = 20 + node + group; - assert!(decode(&valid[..start]).is_err()); - // Tags 2 and 3 are the distinct materialized and extent publications. - for tag in [4, 255] { - let mut malformed = valid.clone(); - malformed[start] = tag; - assert!(decode(&malformed).is_err()); - } - for offset in [1, 9, 17, 25, 45] { - let mut malformed = valid.clone(); - malformed[start + offset..start + offset + 8].fill(0); - assert!(decode(&malformed).is_err()); - } - let mut contradictory = valid[..start].to_vec(); - contradictory.push(0); - assert!(decode(&contradictory).is_err()); - view.snapshot.publication = Some(Publication::Pending); +fn fleet_configuration_commitment_is_order_independent_but_role_exact() { + let observed = observation(false); + let original = &observed.snapshot.configuration; + let mut reordered = original.clone(); + reordered.voters = vec![3, 1, 2]; assert_eq!( - decode(&encode(&view).unwrap()) + crate::raft::host::group_membership_digest(&observed.snapshot.membership, original) + .unwrap(), + crate::raft::host::group_membership_digest(&observed.snapshot.membership, &reordered) .unwrap() - .snapshot - .publication, - Some(Publication::Pending) ); - view.snapshot.view.member.as_mut().unwrap().published = None; - view.snapshot.publication = Some(Publication::Empty); assert_eq!( - decode(&encode(&view).unwrap()) + crate::raft::host::FleetMembershipConfiguration::capture(original).unwrap(), + crate::raft::host::FleetMembershipConfiguration::capture(&reordered).unwrap() + ); + reordered.voters = vec![1, 2]; + reordered.learners = vec![3]; + assert_ne!( + crate::raft::host::group_membership_digest(&observed.snapshot.membership, original) + .unwrap(), + crate::raft::host::group_membership_digest(&observed.snapshot.membership, &reordered) .unwrap() - .snapshot - .publication, - Some(Publication::Empty) ); - view.snapshot.publication = None; - assert!(encode(&view).is_err()); } #[test] -fn fleet_quorum_wire_refuses_bounds_trailing_bytes_and_impossible_barriers() { - let valid = encode(&observation()).unwrap(); - for end in 0..valid.len() { - assert!(decode(&valid[..end]).is_err()); - } - assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); - let mut trailing = valid.clone(); - trailing.push(0); - assert!(decode(&trailing).is_err()); - for offset in [0, 4, 8] { - let mut changed = valid.clone(); - changed[offset..offset + 4].copy_from_slice(&u32::MAX.to_be_bytes()); - assert!(decode(&changed).is_err()); - } - for barrier in [0_u64, 3, u64::MAX] { - let mut changed = valid.clone(); - changed[12..20].copy_from_slice(&barrier.to_be_bytes()); - assert!(decode(&changed).is_err()); - } - let mut wrong_process = observation(); - wrong_process.node.node.identity.process = ProcessGeneration::new([99; 16]).unwrap(); - assert!(encode(&wrong_process).is_err()); - let mut wrong_engine = observation(); - wrong_engine.node.node.storage_instance = EngineInstanceId::new([99; 16]).unwrap(); - assert!(encode(&wrong_engine).is_err()); - let mut bare = observation(); - bare.snapshot.membership = Some(GroupMembership::initial( - bare.snapshot.configuration.as_ref().unwrap(), - )); - assert!(encode(&bare).is_err()); +fn fleet_quorum_body_rejects_mixed_boot_root_barrier_and_configuration() { + let mut observed = observation(true); + observed.snapshot.replica.process = ProcessGeneration::new([99; 16]).unwrap(); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.barrier.index = LogIndex::new(6); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.barrier.term = Term::new(1); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.configuration.voters = vec![1, 4]; + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.status.leader = Some(4); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.status.applied.index = LogIndex::new(4); + assert!(encode(&observed).is_err()); + observed = observation(true); + observed.snapshot.engine = EngineInstanceId::new([98; 16]).unwrap(); + assert!(encode(&observed).is_err()); } #[test] -fn extent_publication_round_trip_keeps_domain_in_the_existing_wire_bound() { - use crate::application::consensus::ImageCommitment; - use crate::raft::host::{FleetImageState, FleetPublishedImage, Publication}; - use cellar_ltx::internal::{ExtentRootReference, ImageRoot}; - let commitment = ImageCommitment::Extents( - ExtentRootReference::new(ImageRoot::from_bytes([7; 32]), 144).unwrap(), - ); - let image = FleetPublishedImage::new_state( - LogPosition { - index: LogIndex::new(1), - term: Term::new(1), - }, - FleetImageState::new(1, 1, 0, 4096, 4096, commitment).unwrap(), - ) - .unwrap(); - let statement = Publication::Image(image.clone()); - assert_eq!(publication::encode(&statement).len(), 85); - assert_eq!( - publication::decode(&publication::encode(&statement)).unwrap(), - statement - ); - let mut observed = observation(); - observed.snapshot.publication = Some(statement); - let decoded = decode(&encode(&observed).unwrap()).unwrap(); - assert_eq!( - decoded.snapshot.publication, - Some(Publication::Image(image)) - ); - for invalid in [vec![3], vec![3; 84], vec![3; 86]] { - assert!(publication::decode(&invalid).is_err()); +fn fleet_quorum_body_refuses_legacy_single_image_shapes_missing_fields_and_lengths() { + let original = encode(&observation(true)).unwrap(); + for len in [0, 11, 12, original.len() - 1] { + assert!(decode(&original[..len]).is_err()); + } + let mut extra = original.clone(); + extra.push(0); + assert!(decode(&extra).is_err()); + assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); + let node_len = u32::from_be_bytes(original[4..8].try_into().unwrap()) as usize; + let body: serde_json::Value = serde_json::from_slice(&original[12 + node_len..]).unwrap(); + for field in [ + "root", + "membership", + "configuration", + "barrier_index", + "barrier_term", + "leader", + "load", + ] { + let mut changed = body.clone(); + changed.as_object_mut().unwrap().remove(field); + assert!(serde_json::from_value::(changed).is_err()); } + let mut old = body.clone(); + old["image"] = serde_json::json!(null); + assert!(serde_json::from_value::(old).is_err()); + let mut old_root = body; + old_root["root"]["root"] + .as_object_mut() + .unwrap() + .remove("last_mutation_index"); + assert!(serde_json::from_value::(old_root).is_err()); } diff --git a/crates/cellar/durability/raft/fleet_stop/closure.rs b/crates/cellar/durability/raft/fleet_stop/closure.rs index 4e6314a5b..81c488edc 100644 --- a/crates/cellar/durability/raft/fleet_stop/closure.rs +++ b/crates/cellar/durability/raft/fleet_stop/closure.rs @@ -142,7 +142,8 @@ impl Sink { .as_ref() .map_err(|_| FleetCatalogError::Unconfirmed)?; if raft.startup_incomplete - || !closed.unfinished.is_empty() + || closed.mutation_outcomes + != crate::application::consensus::MutationOutcomes::NoUnresolvedLocalProducers || closed.node.node.bytes() != selector.boot.node || closed.node.disk.bytes() != selector.boot.disk || closed.node.process.bytes() != selector.boot.process diff --git a/crates/cellar/durability/raft/fleet_stop/records.rs b/crates/cellar/durability/raft/fleet_stop/records.rs index 5588bf575..7c59aee0d 100644 --- a/crates/cellar/durability/raft/fleet_stop/records.rs +++ b/crates/cellar/durability/raft/fleet_stop/records.rs @@ -180,6 +180,13 @@ pub(crate) struct Guard { } impl Guard { + /// New group attachments must order behind this exact locally synced boot. + /// Quiescence, stop acceptance and fencing close this admission as well as + /// business traffic; only a committed resume reopens the same boot. + pub(crate) fn admits_group_start(&self, boot: &BootBinding) -> bool { + self.boot == *boot && matches!(&self.phase, Phase::Open | Phase::Resumed { .. }) + } + pub fn accepted(&self) -> Option<&StopSelector> { match &self.phase { Phase::StopAccepted { selector } => Some(selector), diff --git a/crates/cellar/durability/raft/fleet_stop_transport.rs b/crates/cellar/durability/raft/fleet_stop_transport.rs index 5c78251a3..64fc020db 100644 --- a/crates/cellar/durability/raft/fleet_stop_transport.rs +++ b/crates/cellar/durability/raft/fleet_stop_transport.rs @@ -21,6 +21,9 @@ const REQUEST_PURPOSE: &[u8] = b"cellar-fleet-stop-request-1"; const RESPONSE_PURPOSE: &[u8] = b"cellar-fleet-stop-response-1"; const MAX_BODY: usize = 4096; +// SDK request/reply retention is separate from the transport's actual I/O banks. +pub(crate) const STOP_REQUEST_BYTES: usize = MAX_BODY + 4096; + #[derive(Serialize, Deserialize)] #[serde(deny_unknown_fields)] struct RequestHead { diff --git a/crates/cellar/durability/raft/group_host/archive.rs b/crates/cellar/durability/raft/group_host/archive.rs new file mode 100644 index 000000000..29a4078a1 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/archive.rs @@ -0,0 +1,101 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Startup opens preprovisioned archive identities. A missing identity or +//! counter is corruption, never permission to create a replacement archive. +use super::*; + +pub(super) struct Pools { + io: CellGroupArchivePool, + owners: CellGroupArchiveUploadPool, + tails: CellGroupArchiveTailPool, + checkpoints: CellGroupCheckpointPool, +} + +impl Pools { + pub fn new( + options: Option, + checkpoint: Option, + ) -> anyhow::Result> { + options + .map(|o| { + Ok(Self { + io: CellGroupArchivePool::new(o.io)?, + owners: CellGroupArchiveUploadPool::new(o.uploads)?, + tails: CellGroupArchiveTailPool::new(o.tails)?, + checkpoints: checkpoint.ok_or_else(|| { + anyhow::anyhow!("archive requires complete group checkpoints") + })?, + }) + }) + .transpose() + } + + pub async fn register( + &self, + prepared: &Prepared, + controller: &CellGroupController, + log: &super::super::storage::ReplicaLog, + directory: &std::path::Path, + limits: CellGroupNodeLimits, + options: &super::super::host::RaftHostOptions, + ) -> anyhow::Result { + // All verifier groups share separate node-wide physical inventories. + // Serving pins or voter images cannot satisfy independent archive replay. + // The checkpoint receiver creates anonymous image spools in this + // node-owned directory. Its receive API expects an existing parent; + // initialize and sync it before any archive job can publish admission. + let scratch_directory = directory.join("archive-spools"); + let scratch = scratch_directory.clone(); + let filesystem = crate::asyncrt::fs(); + log.domain() + .blocking(move || { + cellar_ltx::host::create_dir_all_durable(filesystem.as_ref(), &scratch) + })? + .await??; + let base = directory.join("archive-verifier"); + let images = + resources::Images::open(log, base.clone(), options, &prepared.namespaces).await?; + for assignment in prepared + .groups + .iter() + .filter(|a| a.placement == RaftGroupPlacement::Replica) + { + let namespace = &prepared.namespaces[assignment.namespace]; + namespace + .directory + .open_group_archive(&log.domain(), &self.io, &assignment.birth) + .await?; + let route = assignment.birth.route().clone(); + let path = ManifestPageStore::group_pages_path(&base, route.history); + let backend = images.apply(namespace.directory.namespace_config(), options)?; + let store = log + .domain() + .blocking(move || { + Ok::<_, anyhow::Error>(CellGroupArchiveStore::new( + route, + resources::pages(&path, limits)?, + backend, + limits.manifest, + limits.entries, + limits.max_effect_rows, + )?) + })? + .await??; + controller + .archive(CellGroupArchiveRegistration { + directory: namespace.directory.clone(), + birth: assignment.birth.clone(), + io: self.io.clone(), + owners: self.owners.clone(), + tails: self.tails.clone(), + checkpoints: self.checkpoints.clone(), + store, + scratch: log.snapshot_scratch(), + scratch_directory: scratch_directory.clone(), + })? + .response() + .await?; + } + Ok(images.observer()) + } +} diff --git a/crates/cellar/durability/raft/group_host/mod.rs b/crates/cellar/durability/raft/group_host/mod.rs new file mode 100644 index 000000000..ac735a281 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/mod.rs @@ -0,0 +1,399 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Canonical startup uses namespace capabilities and physical group placement. +//! Logical cells are enrolled through the controller, never through node config. +mod archive; +mod registration; +mod resources; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "../host_startup_tests.rs"] +mod startup_tests; +#[cfg(test)] +pub(crate) mod test_support; + +use super::cell_groups::*; +use crate::application::consensus::{RaftCells, RaftClass, VersionKeys}; +use cellar_logic::cell_group::GroupHistory; +use std::{path::PathBuf, sync::Arc}; + +/// An explicit, immutable initial range plan. Retrying preserves every birth +/// and the directory operation identity; startup never allocates a group per cell. +pub struct RaftNamespaceInitialization { + pub operation: cellar_logic::consensus::ProposalId, + pub groups: Vec, +} + +pub struct RaftNamespaceConfig { + pub namespace: super::catalog::NamespaceConfig, + pub class: String, + /// None opens an existing namespace only. A published initialization is + /// idempotent; it cannot replace a later topology or recovery incarnation. + pub initialize: Option, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RaftGroupPlacement { + Replica, + QueryRouter, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct RaftGroupAssignment { + pub history: GroupHistory, + pub placement: RaftGroupPlacement, +} + +/// Explicit shared capacities. Physical SQL, archive and image admissions are +/// node-wide; manifest directories additionally have per-group disk ceilings. +#[derive(Clone, Copy)] +pub struct CellGroupNodeLimits { + pub namespaces: CellGroupHostLimits, + pub controller: CellGroupControllerLimits, + pub sql: CellSqlLimits, + pub provisioning: CellGroupProvisionLimits, + pub directory: DirectoryLimits, + pub dispatch: CellDispatchLimits, + pub manifest: ManifestLimits, + pub entries: crate::application::consensus::group_protocol::GroupEntryLimits, + pub manifest_pages: usize, + pub manifest_bytes: u64, + pub retained_roots: usize, + pub max_effect_rows: u32, + pub checkpoints: Option, + /// Bindings must have been durably provisioned in the namespace catalog. + /// Missing archive identity/counters are never initialized during restart. + pub archive: Option, +} + +#[derive(Clone, Copy)] +pub struct CellGroupNodeArchiveOptions { + pub io: CellGroupArchiveLimits, + pub uploads: CellGroupArchiveUploadLimits, + pub tails: CellGroupArchiveTailLimits, +} + +pub struct RaftCellsConfig { + pub infrastructure: crate::daemon::NodeRaftConfig, + pub catalog: super::catalog::CatalogOptions, + pub namespaces: Vec, + pub groups: Vec, + pub peers: Vec, + pub query_origins: Vec, + pub command_origins: Vec, + pub host: super::host::RaftHostOptions, + pub shared: CellGroupNodeLimits, +} + +pub(crate) struct Namespace { + directory: RangeDirectory, + class: Arc, +} + +pub(crate) struct Assignment { + namespace: usize, + birth: CellGroupBirth, + placement: RaftGroupPlacement, + transport: super::transport::GroupRoute, +} + +pub(crate) struct Prepared { + namespaces: Vec, + peers: Vec, + groups: Vec, + query_origins: Vec, + command_origins: Vec, +} + +pub(crate) use registration::prepare; + +/// Own infrastructure while assembling resources. Any error or cancelled startup +/// schedules joined closure on the same domain; SQL outlives its physical work. +struct Startup { + service: Option, + sql: Option, + controller: Option, +} + +struct Infrastructure { + log: super::storage::ReplicaLog, + runtime: CellGroupRuntime, + transport: super::transport::RaftTransport, +} + +impl Startup { + fn cleanup( + &mut self, + ) -> Result>, crate::asyncrt::DomainClosed> { + let Some(service) = self.service.take() else { + return Ok(None); + }; + let domain = service.log.domain(); + let sql = self.sql.take(); + let controller = self.controller.take(); + domain + .spawn_cleanup(async move { + if let Some(controller) = controller { + let _ = controller.close().await; + } else { + let _ = service.runtime.close().await; + service.transport.close().await; + if let Some(sql) = sql { + let _ = sql.close().await; + } + } + let _ = service.log.close().await; + }) + .map(Some) + } +} + +impl Drop for Startup { + fn drop(&mut self) { + if let Ok(Some(task)) = self.cleanup() { + task.detach(); + } + } +} + +pub(crate) async fn start( + service: crate::daemon::NodeRaftService, + prepared: Prepared, + directory: PathBuf, + options: super::host::RaftHostOptions, + limits: CellGroupNodeLimits, + keys: VersionKeys, +) -> anyhow::Result { + let crate::daemon::NodeRaftService { + log, + runtime, + events, + transport, + } = service; + let mut startup = Startup { + service: Some(Infrastructure { + log: log.clone(), + runtime: runtime.clone(), + transport: transport.clone(), + }), + sql: None, + controller: None, + }; + // This initialization owns several nested catalog/recovery futures. Keep + // that state off callers' native stacks, including nested node startup. + let result = Box::pin(async { + let sql = CellGroupSqlPool::start(&log, limits.sql)?; + startup.sql = Some(sql.clone()); + let checkpoint = limits + .checkpoints + .map(CellGroupCheckpointPool::new) + .transpose()?; + if let Some(pool) = &checkpoint { + transport.enable_group_checkpoints(pool.clone())?; + } + let images = + resources::Images::open(&log, directory.clone(), &options, &prepared.namespaces) + .await?; + let mut appliers = Vec::new(); + for assignment in &prepared.groups { + match assignment.placement { + RaftGroupPlacement::QueryRouter => { + // Publish every route together after all local roots attach. + } + RaftGroupPlacement::Replica => { + let namespace = &prepared.namespaces[assignment.namespace]; + let permit = namespace + .directory + .bootstrap_group_member(&assignment.birth, &log) + .await?; + let group = permit.group().clone(); + let pages = ManifestPageStore::group_pages_path( + &directory, + assignment.birth.route().history, + ); + let backend = images.apply(namespace.directory.namespace_config(), &options)?; + let (applier, current) = log + .domain() + .blocking(move || { + let pages = resources::pages(&pages, limits)?; + Ok::<_, anyhow::Error>(CellGroupApplier::recover( + group, + pages, + backend, + limits.manifest, + limits.entries, + limits.max_effect_rows, + )?) + })? + .await??; + runtime.attach(permit, current).await?; + + appliers.push(applier); + } + } + } + // Refuse incomplete inventories before any collector can consider unpinned + // files. Every retained group must have an explicitly recovered assignment. + for group in log.cell_groups()? { + anyhow::ensure!( + appliers + .iter() + .any(|a| a.history() == group.identity().scope), + "retained group is absent from the node placement inventory" + ); + } + // No physical peer or query route is reachable while a later local group + // still waits on its permanent claim or physical recovery. The attached + // owners retain bounded Ready work until the controller starts draining it. + for assignment in &prepared.groups { + match assignment.placement { + RaftGroupPlacement::Replica => { + transport.install_physical_group(assignment.transport.clone())? + } + RaftGroupPlacement::QueryRouter => { + transport.install_cell_query_group(assignment.transport.clone())? + } + } + } + let controller = CellGroupController::start( + runtime.clone(), + events, + transport.clone(), + sql.clone(), + appliers, + limits.controller, + ) + .await?; + startup.controller = Some(controller.clone()); + let admission = Arc::new(crate::application::consensus::admission::Admission::default()); + controller.bind_public_admission( + admission.clone(), + crate::asyncrt::services().application_work_registry(), + )?; + let namespaces = prepared + .namespaces + .iter() + .map(|namespace| { + Ok(CellGroupProvisionNamespace { + directory: namespace.directory.clone(), + images: images.apply(namespace.directory.namespace_config(), &options)?, + }) + }) + .collect::>>()?; + controller.configure_provisioning(CellGroupProvisioning { + directory: directory.clone(), + filesystem: crate::asyncrt::fs(), + namespaces, + manifest: limits.manifest, + entries: limits.entries, + max_effect_rows: limits.max_effect_rows, + max_pages: limits.manifest_pages, + max_page_bytes: limits.manifest_bytes, + max_retained_roots: limits.retained_roots, + peers: prepared.peers.clone(), + peer_revision: limits.provisioning.peer_revision, + max_preparations: limits.provisioning.max_preparations, + max_preparation_bytes: limits.provisioning.max_preparation_bytes, + max_parallel_preparations: limits.provisioning.max_parallel_preparations, + max_waiters: limits.provisioning.max_waiters, + })?; + let archive = archive::Pools::new(limits.archive, checkpoint)?; + let archive_images = if let Some(archive) = archive { + Some( + archive + .register(&prepared, &controller, &log, &directory, limits, &options) + .await?, + ) + } else { + None + }; + let mut registrations = Vec::new(); + let mut effects = Vec::new(); + let mut dispatchers = Vec::new(); + for namespace in &prepared.namespaces { + let dispatcher = CellGroupDispatcher::from_domain( + log.domain(), + namespace.directory.clone(), + namespace.class.clone(), + keys.clone(), + limits.dispatch, + )? + .with_s3(&controller)?; + let config = namespace.directory.namespace_config(); + let storage = CellGroupNamespaceStorage::new(images.sql(config, &options)?)?; + if namespace.class.has_system_commands() { + let resources = storage.clone(); + effects.push(CellGroupEffectRegistration::new( + dispatcher.clone(), + options + .effects + .ok_or_else(|| anyhow::anyhow!("effects require a scheduler"))?, + move |cell| Ok(resources.for_cell(cell)), + )?); + } + dispatchers.push((dispatcher.clone(), storage.clone())); + registrations.push(CellGroupHostRegistration { + dispatcher, + storage, + }); + } + if !effects.is_empty() { + controller.configure_effects(effects)?; + } + let host = CellGroupHost::bind( + &runtime, + &sql, + &transport, + controller.clone(), + registrations, + limits.namespaces, + )?; + let resources = crate::application::consensus::group_client::resources::Resources::new( + sql, + &transport, + images.observer(), + archive_images, + ); + let service = RaftCells::from_shared( + host, + controller.clone(), + runtime, + log, + admission, + resources, + transport.clone(), + &options, + )?; + // Admission/work accounting is bound before gateways or grants become + // externally reachable. Returned capabilities own these weak registrations. + for (dispatcher, storage) in dispatchers { + service.retain_gateways( + transport.install_cell_query_gateway( + dispatcher.clone(), + controller.clone(), + storage.clone(), + )?, + transport.install_cell_command_gateway(dispatcher, controller.clone(), storage)?, + )?; + } + for origin in prepared.query_origins { + transport.grant_query_origin(origin)?; + } + for origin in prepared.command_origins { + transport.grant_command_origin(origin)?; + } + startup.service.take(); + startup.sql.take(); + startup.controller.take(); + Ok(service) + }) + .await; + if result.is_err() { + // An explicit startup error is a physical join boundary even when the + // caller retains log/runtime/transport clones. Cancellation detaches + // this same cleanup task, whose dependencies remain node-owned. + if let Some(task) = startup.cleanup()? { + task.await?; + } + } + result +} diff --git a/crates/cellar/durability/raft/group_host/registration.rs b/crates/cellar/durability/raft/group_host/registration.rs new file mode 100644 index 000000000..e1cd453fb --- /dev/null +++ b/crates/cellar/durability/raft/group_host/registration.rs @@ -0,0 +1,196 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Catalog observation resolves group placement. It does not bypass member +//! claims, create per-cell births or grant a destination leader authority. +use super::*; +use crate::raft::catalog::{Catalog, CatalogError}; +use cellar_logic::{consensus::ReplicaIdentity, NamespaceMode}; +use std::collections::{BTreeMap, BTreeSet}; + +pub(crate) async fn prepare( + config: &mut RaftCellsConfig, + application: &dyn crate::application::Application, + store: Arc, + keys: &VersionKeys, +) -> anyhow::Result { + // Managed authority must be converted as a whole-group protocol. Never + // bootstrap an old managed namespace through the unmanaged permit path. + anyhow::ensure!( + config.host.managed.is_none() + && config + .namespaces + .iter() + .all(|n| n.namespace.managed_fleet.is_none()), + CatalogError::ManagedAdmissionRequired + ); + config.host.validate()?; + config.infrastructure.transport.validate()?; + config.shared.entries.validate()?; + config + .shared + .provisioning + .validate(config.shared.controller.max_groups)?; + anyhow::ensure!( + !config.namespaces.is_empty() + && config.namespaces.len() <= config.shared.namespaces.max_namespaces + && config.groups.len() <= config.shared.controller.max_groups + && config.shared.controller.max_groups <= config.infrastructure.log.max_groups + && config.shared.controller.max_groups == config.host.max_groups + && config.shared.dispatch.max_input_bytes == config.host.max_input_bytes + && config.shared.entries.envelope == config.host.execution.envelope + && config.shared.manifest.max_cells + <= config.infrastructure.consensus.max_cells_per_group, + "incoherent shared host capacities" + ); + anyhow::ensure!( + config.shared.archive.is_none() || config.shared.checkpoints.is_some(), + "archive requires checkpoint admission" + ); + anyhow::ensure!( + config.shared.controller.checkpoints.is_none() || config.shared.checkpoints.is_some(), + "checkpoint policy requires physical source admission" + ); + // Until these consumers are converted they cannot silently start a legacy + // placement/controller or publish a misleading managed node inventory. + anyhow::ensure!( + config.host.controller.is_none() + && config.host.node_directory.is_none() + && config.host.inspection_origins.is_empty(), + "whole-group placement and node directory integration is required" + ); + let mut definitions = Vec::new(); + let mut seen = BTreeSet::new(); + for registration in &config.namespaces { + anyhow::ensure!( + seen.insert(( + registration.namespace.cluster, + registration.namespace.namespace + )), + "duplicate namespace capability" + ); + anyhow::ensure!( + registration.namespace.mode == NamespaceMode::Raft, + "namespace is not Raft" + ); + anyhow::ensure!( + !application.supports_cell(®istration.class) + && !application.supports_workflow(®istration.class), + crate::application::consensus::RegistrationError::UnsupportedFactory( + registration.class.clone() + ) + ); + let class = application.resolve_raft_class(®istration.class)?; + class.validate_shared_host(&config.host)?; + anyhow::ensure!( + class.schema_digest() == registration.namespace.schema_digest, + "compiled namespace schema mismatch" + ); + anyhow::ensure!( + registration.namespace.image_storage + != super::super::catalog::RaftImageStorage::Extents + || config.host.extents.is_some(), + "extent namespace requires node storage" + ); + anyhow::ensure!( + registration.namespace.minimum_durability != cellar_logic::consensus::Durability::S3 + || config.shared.archive.is_some(), + "S3 minimum requires an archive owner" + ); + definitions.push(class); + } + let mut peers = BTreeMap::new(); + for peer in &config.peers { + anyhow::ensure!( + peers.insert(peer.node, peer).is_none(), + "duplicate physical node endpoint" + ); + } + let catalog = Catalog::new(store, config.catalog.clone())?; + let mut namespaces = Vec::new(); + for (registration, class) in config.namespaces.drain(..).zip(definitions) { + let namespace = if registration.initialize.is_some() { + catalog.open_namespace(registration.namespace).await? + } else { + catalog.existing_namespace(registration.namespace).await? + }; + namespace + .require_version_keys(keys, u64::try_from(crate::asyncrt::wall_ms())?) + .await?; + let directory = RangeDirectory::new(namespace, config.shared.directory)?; + if let Some(initialization) = registration.initialize { + anyhow::ensure!( + !initialization.groups.is_empty() + && initialization.groups.len() <= config.shared.directory.max_groups as usize, + "invalid initial range count" + ); + let mut routes = Vec::new(); + for spec in initialization.groups { + let request = directory.prepare_group_birth(spec)?; + directory.reserve_group_birth(&request).await?; + routes.push(request.route().clone()); + } + let mutation = directory.prepare_initialization(initialization.operation, &routes)?; + directory.publish(&mutation).await?; + } + directory.observe().await?; + namespaces.push(Namespace { directory, class }); + } + let mut groups = Vec::new(); + let mut seen = BTreeSet::new(); + for assignment in &config.groups { + anyhow::ensure!( + seen.insert(assignment.history), + "duplicate physical group placement" + ); + let namespace = namespaces + .iter() + .position(|n| { + let c = n.directory.namespace_config(); + c.cluster == assignment.history.cluster + && c.namespace == assignment.history.namespace + }) + .ok_or_else(|| anyhow::anyhow!("group is outside namespace capabilities"))?; + let birth = namespaces[namespace] + .directory + .materialize_selected_history(assignment.history) + .await?; + let mut members = Vec::new(); + for member in birth.voters().iter().chain(birth.learners()) { + let endpoint = peers + .get(&member.node) + .ok_or_else(|| anyhow::anyhow!("missing trusted group peer endpoint"))?; + members.push(super::super::transport::RaftPeer { + replica: ReplicaIdentity { + scope: assignment.history, + member: member.member, + node: endpoint.node, + disk: endpoint.disk, + process: endpoint.process, + }, + label: endpoint.label.clone(), + server_name: endpoint.server_name.clone(), + control_address: endpoint.control_address, + data_address: endpoint.data_address, + query_address: endpoint.query_address, + }); + } + groups.push(Assignment { + namespace, + placement: assignment.placement, + transport: super::super::transport::GroupRoute { + scope: assignment.history, + schema_digest: birth.route().schema_digest, + revision: birth.route().epoch.get(), + peers: members, + }, + birth, + }); + } + Ok(Prepared { + namespaces, + peers: std::mem::take(&mut config.peers), + groups, + query_origins: std::mem::take(&mut config.query_origins), + command_origins: std::mem::take(&mut config.command_origins), + }) +} diff --git a/crates/cellar/durability/raft/group_host/resources.rs b/crates/cellar/durability/raft/group_host/resources.rs new file mode 100644 index 000000000..7dae99d59 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/resources.rs @@ -0,0 +1,209 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One serving image inventory and one independent archive-verifier inventory +//! per node. Namespace/cell handles never open another physical store. +use super::*; +use crate::raft::{ + catalog::{NamespaceConfig, RaftImageStorage}, + execution::PrivateOptions, + host::RaftHostOptions, + storage::ReplicaLog, +}; +use cellar_ltx::{ + generation::{GenerationStore, ReadServices}, + internal::NodeExtentStore, + stream_snapshot::ScratchPool, +}; +use rand::RngCore; +use std::path::Path; + +#[derive(Clone)] +pub(super) struct Images { + directory: PathBuf, + materialized: Option, + extents: Option, + cache: Option, + accounting: Option, +} + +impl Images { + pub async fn open( + log: &ReplicaLog, + directory: PathBuf, + options: &RaftHostOptions, + namespaces: &[Namespace], + ) -> anyhow::Result { + let materialized = namespaces.iter().any(|n| { + n.directory.namespace_config().image_storage == RaftImageStorage::Materialized + }); + let extents = namespaces + .iter() + .any(|n| n.directory.namespace_config().image_storage == RaftImageStorage::Extents); + let limits = options.images; + let extent_options = options.extents; + let scratch = log.snapshot_scratch(); + let fs = crate::asyncrt::fs(); + log.domain() + .blocking(move || { + let accounting = + materialized.then(cellar_ltx::generation::GenerationAccounting::default); + let materialized = materialized + .then(|| { + GenerationStore::open_with_scratch( + directory.join("images"), + fs.clone(), + limits, + accounting.as_ref(), + scratch, + ) + }) + .transpose()?; + let (extents, cache) = if extents { + let options = extent_options.ok_or_else(|| { + anyhow::anyhow!("extent namespace requires node inventory") + })?; + ( + Some(NodeExtentStore::open_serialized( + directory.join("extent-inventory"), + fs, + options.inventory, + )?), + Some(ScratchPool::new(options.cache_scratch)?), + ) + } else { + (None, None) + }; + Ok(Self { + directory, + materialized, + extents, + cache, + accounting, + }) + })? + .await? + } + + pub fn observer(&self) -> crate::application::consensus::group_client::resources::Images { + crate::application::consensus::group_client::resources::Images { + materialized: self.accounting.clone(), + extents: self.extents.as_ref().map(NodeExtentStore::observer), + } + } + + pub fn sql( + &self, + namespace: &NamespaceConfig, + options: &RaftHostOptions, + ) -> anyhow::Result { + let directory = self + .directory + .join("sql") + .join(hex::encode(namespace.cluster.bytes())) + .join(hex::encode(namespace.namespace.bytes())); + Ok(match namespace.image_storage { + RaftImageStorage::Materialized => CellSqlOptions::Materialized(PrivateOptions { + directory, + host: host(), + sqlite_vfs: None, + images: self + .materialized + .clone() + .ok_or_else(|| anyhow::anyhow!("missing materialized inventory"))?, + query_services: services(), + application_digest: namespace.schema_digest, + limits: options.execution, + }), + RaftImageStorage::Extents => CellSqlOptions::Extents(CellSqlRootOptions { + host: host(), + sqlite_vfs: None, + images: self + .extents + .clone() + .ok_or_else(|| anyhow::anyhow!("missing extent inventory"))?, + query_services: services(), + application_digest: namespace.schema_digest, + limits: options.execution, + cache_directory: directory, + cache_pool: self + .cache + .clone() + .ok_or_else(|| anyhow::anyhow!("missing extent cache admission"))?, + cache: options + .extents + .ok_or_else(|| anyhow::anyhow!("missing extent cache limits"))? + .cache, + }), + }) + } + + pub fn apply( + &self, + namespace: &NamespaceConfig, + options: &RaftHostOptions, + ) -> anyhow::Result { + Ok(match self.sql(namespace, options)? { + CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { + images: o.images, + services: o.query_services, + }, + CellSqlOptions::Extents(o) => CellGroupApplyImages::Extents { + images: o.images, + services: o.query_services, + limits: options.images.apply, + cache_directory: o.cache_directory, + cache_pool: o.cache_pool, + cache: o.cache, + }, + }) + } +} + +pub(super) fn pages(path: &Path, limits: CellGroupNodeLimits) -> anyhow::Result { + Ok(ManifestPageStore::open( + path, + crate::asyncrt::fs(), + limits.manifest_pages, + limits.manifest_bytes, + limits.retained_roots, + )?) +} + +fn services() -> ReadServices { + ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|b| crate::asyncrt::rng("cell-group-image-read").fill_bytes(b)), + } +} + +fn host() -> cellar_ltx::LtxHost { + let fs = crate::asyncrt::fs(); + let age = fs.clone(); + let read = fs.clone(); + cellar_ltx::LtxHost::new( + crate::asyncrt::wall_ms, + move |path| { + Ok(std::time::Duration::from_millis( + crate::asyncrt::wall_ms() + .saturating_sub(age.metadata(path)?.modified_unix_millis) + .max(0) as u64, + )) + }, + move |path| { + let fs = read.clone(); + async move { + crate::asyncrt::try_blocking(move || fs.read(&path)) + .map_err(|e| std::io::Error::other(e.to_string()))? + .await + .map_err(|e| std::io::Error::other(e.to_string()))? + } + }, + |job| async move { + crate::asyncrt::try_blocking(job) + .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string()))? + .await + .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string())) + }, + ) + .with_filesystem(fs) +} diff --git a/crates/cellar/durability/raft/group_host/test_support.rs b/crates/cellar/durability/raft/group_host/test_support.rs new file mode 100644 index 000000000..a4a895b31 --- /dev/null +++ b/crates/cellar/durability/raft/group_host/test_support.rs @@ -0,0 +1,63 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Explicit bounded startup capacities shared by the native host regressions. +use super::*; + +pub(crate) fn limits() -> CellGroupNodeLimits { + CellGroupNodeLimits { + namespaces: CellGroupHostLimits { + max_namespaces: 1, + namespace_bytes: 16 * 1024 * 1024, + }, + controller: CellGroupControllerLimits { + max_groups: 2, + max_parallel_apply: 2, + retry_interval_ms: 5, + max_commands: 2, + max_command_bytes: 64 * 1024, + checkpoints: None, + }, + sql: CellSqlLimits { + workers: 2, + queue_per_worker: 4, + max_pending_cells: 8, + group_pending_cells: 8, + node_bytes: 512 * 1024 * 1024, + group_bytes: 512 * 1024 * 1024, + }, + provisioning: CellGroupProvisionLimits { + max_preparations: 1, + max_preparation_bytes: 512 * 1024 * 1024, + max_parallel_preparations: 1, + max_waiters: 2, + peer_revision: 1, + }, + directory: DirectoryLimits::default(), + dispatch: CellDispatchLimits { + max_cached_routes: 32, + route_cache_bytes: 256 * 1024, + max_invocations: 8, + retained_bytes: 1024 * 1024, + max_input_bytes: 1024, + }, + manifest: ManifestLimits { + max_cells: 32, + max_depth: 16, + }, + entries: crate::application::consensus::group_protocol::GroupEntryLimits::default(), + manifest_pages: 1024, + manifest_bytes: 16 * 1024 * 1024, + retained_roots: 16, + max_effect_rows: 100, + checkpoints: None, + archive: None, + } +} + +pub(crate) fn host_options() -> crate::raft::host::RaftHostOptions { + crate::raft::host::RaftHostOptions { + max_groups: 2, + max_input_bytes: 1024, + ..Default::default() + } +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs index 0c486fb3e..2b23b3d73 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/mod.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/mod.rs @@ -9,6 +9,8 @@ use crate::raft::cell_groups::{ use std::sync::Weak; mod codec; mod receive; +#[cfg(all(test, not(cellar_internal_tests)))] +mod route_tests; pub(crate) mod seed; mod send; mod workers; @@ -19,7 +21,9 @@ pub use seed::{CellGroupSeedError, CellGroupSeedSession}; pub(super) struct Endpoint { applier: CellGroupApplier, route: Arc>, - active: AtomicBool, + active: Arc, + owner: Arc<()>, + registration: Arc<()>, } /// Retain with the group's controller registration. Dropping this handle revokes @@ -104,10 +108,17 @@ impl RaftTransport { if registry.group_checkpoints.len() >= state.options.max_groups { return Err(TransportError::Overloaded); } + let owner = registry + .physical_owners + .get(&history) + .cloned() + .ok_or(TransportError::UnknownGroup)?; let endpoint = Arc::new(Endpoint { applier, route, - active: AtomicBool::new(true), + active: Arc::new(AtomicBool::new(true)), + owner, + registration: Arc::new(()), }); registry .group_checkpoints @@ -137,13 +148,30 @@ impl State { .ok_or(TransportError::Refused)?; if !endpoint.active.load(Ordering::Acquire) || !registry - .physical_groups + .physical_owners .get(history) - .is_some_and(|route| Arc::ptr_eq(route, &endpoint.route)) + .is_some_and(|owner| Arc::ptr_eq(owner, &endpoint.owner)) { return Err(TransportError::StalePeer); } - Ok((endpoint, pool)) + let route = registry + .physical_groups + .get(history) + .ok_or(TransportError::UnknownGroup)?; + // Keep one factory/physical owner across advisory refreshes, while each + // admitted stream captures one exact immutable route generation. + let selected = if Arc::ptr_eq(route, &endpoint.route) { + endpoint + } else { + Arc::new(Endpoint { + applier: endpoint.applier.clone(), + route: route.clone(), + active: endpoint.active.clone(), + owner: endpoint.owner.clone(), + registration: endpoint.registration.clone(), + }) + }; + Ok((selected, pool)) } fn checkpoint_current( @@ -152,9 +180,29 @@ impl State { wire: &Route, ) -> Result, TransportError> { let (_, target) = wire.decode_group()?; - let (current, _) = self.checkpoint_endpoint(&target.scope)?; - if !Arc::ptr_eq(¤t, endpoint) { - return Err(TransportError::StalePeer); + { + let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; + self.admit(®istry)?; + let current = registry + .group_checkpoints + .get(&target.scope) + .and_then(Weak::upgrade) + .ok_or(TransportError::Refused)?; + // Per-chunk validation allocates no selected endpoint or route. + // Drop this guard before peer authorization takes the registry lock. + if !endpoint.active.load(Ordering::Acquire) + || !Arc::ptr_eq(¤t.registration, &endpoint.registration) + || !registry + .physical_owners + .get(&target.scope) + .is_some_and(|owner| Arc::ptr_eq(owner, &endpoint.owner)) + || !registry + .physical_groups + .get(&target.scope) + .is_some_and(|route| Arc::ptr_eq(route, &endpoint.route)) + { + return Err(TransportError::StalePeer); + } } self.authorize_physical(wire) } diff --git a/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs b/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs new file mode 100644 index 000000000..a286f529a --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/route_tests.rs @@ -0,0 +1,211 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Real engine/registration ownership, not quorum or network-stream qualification. +use super::*; +use crate::raft::transport::shared_fixture::Fixture; +use crate::raft::{cell_groups::*, storage::CellGroupBootstrap}; +use cellar_logic::{ + cell_group::{CellRange, RouteEpoch}, + consensus::{DiskLineage, GroupId, IncarnationId, MemberId, ProcessGeneration}, +}; + +async fn fixture() -> anyhow::Result<( + Fixture, + GroupRoute, + CellGroupApplier, + tempfile::TempDir, +)> { + let history = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([31; 16])?, + group: GroupId::new([32; 16])?, + incarnation: IncarnationId::new([33; 16])?, + }; + let fixture = Fixture::start( + TransportOptions::default(), + vec![CellGroupBootstrap { + route: cellar_logic::cell_group::GroupRoute { + history, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [10; 32], + schema_digest: [9; 32], + }, + member: MemberId::new(1)?, + voters: (1..=3).map(MemberId::new).collect::, _>>()?, + learners: vec![], + }], + ) + .await?; + let peers = (1..=3) + .map(|member| { + Ok(RaftPeer { + replica: ReplicaIdentity { + scope: history, + member: MemberId::new(member)?, + node: NodeId::new([member as u8; 16])?, + disk: DiskLineage::new([member as u8 + 1; 16])?, + process: ProcessGeneration::new([member as u8 + 2; 16])?, + }, + label: format!("node-{member}"), + server_name: "localhost".into(), + control_address: if member == 1 { + fixture.transport.control_address() + } else { + format!("127.0.0.1:{}", 42000 + member * 3).parse()? + }, + data_address: if member == 1 { + fixture.transport.data_address() + } else { + format!("127.0.0.1:{}", 42001 + member * 3).parse()? + }, + query_address: if member == 1 { + fixture.transport.query_address() + } else { + format!("127.0.0.1:{}", 42002 + member * 3).parse()? + }, + }) + }) + .collect::>>()?; + let group = fixture.log.cell_group(&history)?; + let directory = tempfile::tempdir()?; + let base = directory.path().to_owned(); + let (applier, current) = fixture + .domain + .blocking(move || { + let images = cellar_ltx::generation::GenerationStore::open( + base.join("images"), + Arc::new(cellar_ltx::DirectFileSystem), + Default::default(), + )?; + let pages = ManifestPageStore::open( + &base.join("pages"), + Arc::new(cellar_ltx::DirectFileSystem), + 1024, + 16 * 1024 * 1024, + 16, + )?; + Ok::<_, anyhow::Error>(CellGroupApplier::recover( + group, + pages, + CellGroupApplyImages::Materialized { + images, + services: cellar_ltx::generation::ReadServices { + now_unix_millis: Arc::new(crate::asyncrt::wall_ms), + fill_random: Arc::new(|bytes| bytes.fill(1)), + }, + }, + ManifestLimits::default(), + crate::application::consensus::group_protocol::GroupEntryLimits::default(), + 100, + )?) + })? + .await??; + drop(current); + // Keep the directory until every real page/image owner has been released. + fixture + .transport + .enable_group_checkpoints(CellGroupCheckpointPool::new(CellGroupCheckpointLimits { + max_sources: 1, + max_retained_bytes: 32 * 1024 * 1024, + max_cells: 64, + max_transfer_bytes: 32 * 1024 * 1024, + max_chunk_bytes: 4093, + })?)?; + Ok(( + fixture, + GroupRoute { + scope: history, + schema_digest: [9; 32], + revision: 1, + peers, + }, + applier, + directory, + )) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn advisory_refresh_keeps_factory_and_physical_owner_but_revokes_old_stream( +) -> anyhow::Result<()> { + let (fixture, route, applier, _directory) = fixture().await?; + let registration = fixture.transport.register_physical_group(route.clone())?; + let checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + let state = &fixture.transport.handle.0; + let (old, _) = state.checkpoint_endpoint(&route.scope)?; + let owner = state.registry.lock().unwrap().physical_owners[&route.scope].clone(); + let mut refreshed = route.clone(); + refreshed.revision += 1; + fixture + .transport + .install_physical(refreshed, false, Some(owner))?; + let wire = Route::new_group( + &route.peers[1].replica, + &route.peers[1].label, + &route.peers[0].replica, + &route.peers[0].label, + route.schema_digest, + )?; + assert!(matches!( + state.checkpoint_current(&old, &wire), + Err(TransportError::StalePeer) + )); + let (current, _) = state.checkpoint_endpoint(&route.scope)?; + assert!(Arc::ptr_eq(&old.registration, ¤t.registration)); + assert!(state.checkpoint_current(¤t, &wire).is_ok()); + drop(registration); + assert!(!current.active.load(Ordering::Acquire)); + assert!(!state + .registry + .lock() + .unwrap() + .physical_groups + .contains_key(&route.scope)); + assert!(!state + .registry + .lock() + .unwrap() + .physical_owners + .contains_key(&route.scope)); + assert!(state.checkpoint_endpoint(&route.scope).is_err()); + drop((checkpoint, applier, old, current)); + fixture.close().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn explicit_route_replacement_is_not_owned_by_old_registration() -> anyhow::Result<()> { + let (fixture, route, applier, _directory) = fixture().await?; + let old_registration = fixture.transport.register_physical_group(route.clone())?; + let old_checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + let state = &fixture.transport.handle.0; + let old_owner = state.registry.lock().unwrap().physical_owners[&route.scope].clone(); + let mut replacement = route.clone(); + replacement.revision += 1; + fixture + .transport + .install_physical_group(replacement.clone())?; + let checkpoint = fixture + .transport + .register_group_checkpoint(applier.clone())?; + assert!(!old_checkpoint.endpoint.active.load(Ordering::Acquire)); + let mut stale_refresh = replacement.clone(); + stale_refresh.revision += 1; + assert!(matches!( + fixture + .transport + .install_physical(stale_refresh, false, Some(old_owner)), + Err(TransportError::StalePeer) + )); + drop(old_registration); + drop(old_checkpoint); + let (current, _) = state.checkpoint_endpoint(&route.scope)?; + assert_eq!(*current.route, replacement); + fixture.transport.remove_physical_group(&route.scope)?; + assert!(!current.active.load(Ordering::Acquire)); + drop((checkpoint, applier, current)); + fixture.close().await +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs index cb6d54f69..350bfb5f7 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/client.rs @@ -12,6 +12,44 @@ impl RaftTransport { directory: RangeDirectory, birth: CellGroupBirth, deadline: u64, + ) -> Result { + self.prepare_seed(local, source, directory, birth, deadline, None, None) + .await + } + + /// Seed an admitted managed learner under this exact installed physical + /// boot. A fresh catalog reservation and live quorum grant are both required. + pub async fn prepare_managed_group_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + host: crate::raft::fleet_catalog::FleetHostAuthorization, + expected: crate::raft::fleet_catalog::FleetPermitAuthorization, + ) -> Result { + self.prepare_seed( + local, + source, + directory, + birth, + deadline, + Some(host), + Some(expected), + ) + .await + } + + async fn prepare_seed( + &self, + local: ReplicaIdentity, + source: RaftPeer, + directory: RangeDirectory, + birth: CellGroupBirth, + deadline: u64, + host: Option, + expected: Option, ) -> Result { let state = self.handle.0.clone(); let log = state.runtime.shared()?.log().clone(); @@ -46,6 +84,36 @@ impl RaftTransport { }); // Refuse before remote or local mutations if the node has no receiver pool. let pool = self.group_checkpoint_pool()?; + let managed = if let Some(host) = &host { + let authorization = state + .seed_process_until(&process, deadline, async { + Ok::<_, TransportError>( + directory + .authorize_group_learner_seed( + &birth, + &log, + &local, + &source.replica, + host, + deadline, + ) + .await, + ) + }) + .await??; + if expected.as_ref().is_none_or(|expected| { + expected.permit() != authorization.permit() + || expected.target_boot() != authorization.target_boot() + }) { + return Err(TransportError::Refused.into()); + } + Some(grant::Managed { + authorization, + boot: host.boot_binding().map_err(CatalogError::from)?, + }) + } else { + None + }; let now = state.domain.mono_ms().map_err(|_| TransportError::Closed)?; let timeout_ms = deadline .checked_sub(now) @@ -57,7 +125,7 @@ impl RaftTransport { .map_err(|_| TransportError::Closed)? .checked_add(1) .ok_or(TransportError::Closed)?; - let request = Request { + let mut request = Request { route: Route::new_group( &local, state.auth.source(), @@ -87,6 +155,10 @@ impl RaftTransport { .1 .export_keying_material([0; 32], EXPORT_LABEL, None) .map_err(|e| TransportError::Tls(e.to_string()))?; + request.timeout_ms = deadline + .checked_sub(state.domain.mono_ms().map_err(|_| TransportError::Closed)?) + .filter(|ms| *ms > 0) + .ok_or(TransportError::Timeout)?; let wire = snapshots::Wire { auth: &state.auth, exporter: &exporter, @@ -132,7 +204,14 @@ impl RaftTransport { return Err(TransportError::Protocol.into()); } let root = checkpoint::seed_marker(&snapshot, &local, barrier)?; + checkpoint::seed_profile( + &snapshot, + managed.as_ref().map(|m| m.authorization.permit()), + )?; + checkpoint::seed_source(&snapshot, &source.replica)?; let grant = Arc::new(Grant { + source: source.replica.clone(), + managed, local, root, snapshot, @@ -145,10 +224,35 @@ impl RaftTransport { _request: Arc::new(guard), }); let group = state - .domain - .run(directory.stage_group_learner(&birth, &log, grant.clone())) - .await - .map_err(|_| TransportError::Closed)??; + .seed_process_until(&grant.process, deadline, async { + Ok::<_, TransportError>( + state + .domain + .run(async { + match &host { + Some(host) => { + directory + .stage_managed_group_learner( + &birth, + &log, + grant.clone(), + host, + ) + .await + } + None => { + directory + .stage_group_learner(&birth, &log, grant.clone()) + .await + } + } + }) + .await + .map_err(|_| TransportError::Closed)?, + ) + }) + .await??; + grant.check_log(&log)?; group .consensus_claimed .compare_exchange(false, true, Ordering::AcqRel, Ordering::Acquire) @@ -158,6 +262,7 @@ impl RaftTransport { let (command, ready) = oneshot::channel(); let (response, result) = oneshot::channel(); let exchange = Exchange { + host, state: state.clone(), stream, exporter, diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs index 727ffe6ef..8eb6c4302 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant.rs @@ -1,10 +1,29 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; +use crate::raft::{ + fleet_catalog::FleetPermitAuthorization, + fleet_stop::{BootBinding, Guard}, + member_state::FleetGroupBinding, +}; + +pub(super) struct Managed { + pub authorization: FleetPermitAuthorization, + pub boot: BootBinding, +} + +pub(super) fn check_boot(guard: Option<&Guard>, expected: &BootBinding) -> Result<(), LogError> { + if guard.is_none_or(|guard| !guard.admits_group_start(expected)) { + return Err(LogError::WrongIdentity); + } + Ok(()) +} /// Only a verified TLS response constructs this capability. It remains live /// through queued physical installation, including cancelled callers. pub(crate) struct Grant { + pub(super) managed: Option, pub(super) local: ReplicaIdentity, + pub(super) source: ReplicaIdentity, pub(super) root: GroupManifest, pub(super) snapshot: ::raft::prelude::Snapshot, pub(super) deadline: u64, @@ -22,6 +41,10 @@ impl Grant { &self.local } + pub(crate) fn source(&self) -> &ReplicaIdentity { + &self.source + } + pub(crate) fn snapshot(&self) -> &::raft::prelude::Snapshot { &self.snapshot } @@ -31,6 +54,21 @@ impl Grant { owner: &Arc<()>, history: GroupHistory, local: &ReplicaIdentity, + ) -> Result<(), LogError> { + self.check_target(owner, history, local)?; + if *self.process.borrow() { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + // Source liveness authorizes submission. After actual engine sync succeeds, + // only the destination and original operation bounds govern local completion. + fn check_target( + &self, + owner: &Arc<()>, + history: GroupHistory, + local: &ReplicaIdentity, ) -> Result<(), LogError> { if !Arc::ptr_eq(owner, &self.owner) || history != self.local.scope @@ -38,7 +76,6 @@ impl Grant { || local.node != self.local.node || local.disk != self.local.disk || local.process != self.local.process - || *self.process.borrow() || *self.stop.borrow() || !self.domain.mono_ms().is_ok_and(|now| now < self.deadline) { @@ -47,9 +84,57 @@ impl Grant { Ok(()) } + pub(crate) fn check_lifecycle(&self, guard: Option<&Guard>) -> Result<(), LogError> { + match &self.managed { + Some(managed) => check_boot(guard, &managed.boot), + None => Ok(()), + } + } + + pub(crate) fn initial_membership( + &self, + conf: &::raft::prelude::ConfState, + ) -> Result { + checkpoint::seed_initial_membership(&self.snapshot, conf) + .map_err(|e| LogError::Invalid(e.to_string())) + } + + pub(crate) fn deadline(&self) -> u64 { + self.deadline + } + + pub(crate) fn managed_binding(&self) -> Option<&FleetGroupBinding> { + self.managed + .as_ref() + .map(|m| m.authorization.permit().binding()) + } + + pub(crate) fn check_authorization( + &self, + authorization: &FleetPermitAuthorization, + ) -> Result<(), LogError> { + if self.managed.as_ref().is_none_or(|managed| { + managed.authorization.permit() != authorization.permit() + || authorization.minimum_trust_revision() + < managed.authorization.minimum_trust_revision() + || authorization.target_boot() != Some(&managed.boot) + }) { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + pub(crate) fn check_log(&self, log: &ReplicaLog) -> Result<(), LogError> { + self.check_target_log(log)?; + if *self.process.borrow() { + return Err(LogError::WrongIdentity); + } + Ok(()) + } + + pub(super) fn check_target_log(&self, log: &ReplicaLog) -> Result<(), LogError> { let options = log.limits(); - self.check( + self.check_target( &log.owner(), self.local.scope, &ReplicaIdentity { @@ -59,7 +144,8 @@ impl Grant { process: options.process, member: self.local.member, }, - ) + )?; + self.check_lifecycle(log.lifecycle_guard()?.as_ref()) } pub(crate) fn initial_configuration( @@ -79,8 +165,17 @@ impl Grant { { return Err(LogError::WrongIdentity); } + checkpoint::seed_profile( + &self.snapshot, + self.managed.as_ref().map(|m| m.authorization.permit()), + ) + .map_err(|e| LogError::Invalid(e.to_string()))?; checkpoint::seed_birth(&self.snapshot, &bootstrap.route, &conf) .map_err(|e| LogError::Invalid(e.to_string()))?; Ok(conf) } } + +#[cfg(test)] +#[path = "grant_tests.rs"] +mod tests; diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs new file mode 100644 index 000000000..ccf7949d1 --- /dev/null +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/grant_tests.rs @@ -0,0 +1,77 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Local guard transition fixtures do not establish a sync or seed authority. +use super::*; +use crate::raft::fleet_stop::{LifecyclePurpose, LifecycleRecord, LifecycleUpdate, StopSelector}; + +fn boot() -> BootBinding { + BootBinding { + catalog: [1; 32], + fleet: [2; 16], + node: [3; 16], + disk: [4; 16], + engine: [5; 16], + boot: [6; 16], + process: [7; 16], + epoch: 1, + join_key: crate::peer_probe::PeerProbeSigner::from_secret([8; 32]).public_key_bytes(), + } +} + +#[test] +fn managed_seed_guard_requires_exact_open_boot() { + let boot = boot(); + assert!(check_boot(None, &boot).is_err()); + let open = LifecycleRecord::empty() + .change(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline: 1000, + }) + .unwrap(); + check_boot(open.guard.as_ref(), &boot).unwrap(); + for part in 0..3 { + let mut changed = boot.clone(); + match part { + 0 => changed.process = [9; 16], + 1 => changed.engine = [10; 16], + _ => changed.boot = [11; 16], + } + assert!(check_boot(open.guard.as_ref(), &changed).is_err()); + } +} + +#[test] +fn managed_seed_guard_refuses_quiescence_and_fence() { + let boot = boot(); + let open = LifecycleRecord::empty() + .change(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline: 1000, + }) + .unwrap(); + let selector = StopSelector { + purpose: LifecyclePurpose::Maintenance, + boot: boot.clone(), + epoch: 2, + operation: [12; 16], + finalization: [13; 16], + request: [14; 16], + }; + let quiescing = open + .change(LifecycleUpdate::Quiesce { + selector: selector.clone(), + deadline: 1000, + }) + .unwrap(); + assert!(check_boot(quiescing.guard.as_ref(), &boot).is_err()); + let fenced = quiescing + .change(LifecycleUpdate::Fence { + boot: boot.clone(), + epoch: 2, + operation: selector.operation, + request: [15; 16], + deadline: 1000, + }) + .unwrap(); + assert!(check_boot(fenced.guard.as_ref(), &boot).is_err()); +} diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs index 6c2d15895..1c3727d66 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/install.rs @@ -127,6 +127,25 @@ impl Exchange { if prepared.snapshot() != self.grant.snapshot() { return Err(TransportError::Authentication.into()); } + if let Some(host) = &self.host { + let current = state + .seed_process_until(&self.grant.process, self.grant.deadline, async { + Ok::<_, TransportError>( + self.directory + .authorize_group_learner_seed( + &self.birth, + log, + self.grant.local(), + self.grant.source(), + host, + self.grant.deadline, + ) + .await, + ) + }) + .await??; + self.grant.check_authorization(¤t)?; + } let current = prepared.manifest().clone(); let mut write = GroupWrite::new(&self.group); let metadata = prepared.snapshot().get_metadata(); @@ -142,10 +161,37 @@ impl Exchange { // actual persistence finishes. Caller cancellation/transport close cannot // turn an in-flight fence into an unowned or successful empty bootstrap. log.persist_cells(self.operation, vec![write]).await?; - let permit = self - .directory - .bootstrap_joined_group_member(&self.birth, self.grant.local.member, log) - .await?; + self.grant.check_target_log(log)?; + // The engine checked source liveness immediately before entering its + // fence. A later source restart cannot revoke that completed durable + // result. Still require this destination's exact live boot, transport, + // original deadline, and freshly authorized catalog bootstrap. + let permit = state + .seed_until(self.grant.deadline, async { + Ok::<_, TransportError>(match &self.host { + Some(host) => { + self.directory + .bootstrap_managed_joined_group_member( + &self.birth, + self.grant.local.member, + log, + host, + ) + .await + } + None => { + self.directory + .bootstrap_joined_group_member( + &self.birth, + self.grant.local.member, + log, + ) + .await + } + }) + }) + .await??; + self.grant.check_target_log(log)?; drop(self.lease); Ok((permit, current)) } diff --git a/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs index b7146292d..f90fcb588 100644 --- a/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs +++ b/crates/cellar/durability/raft/group_snapshot_transport/seed/mod.rs @@ -139,6 +139,7 @@ struct ReceiveAdmission { } struct Exchange { + host: Option, state: Arc, stream: Stream, exporter: [u8; 32], diff --git a/crates/cellar/durability/raft/host.rs b/crates/cellar/durability/raft/host.rs index f676aad40..ae0b9e153 100644 --- a/crates/cellar/durability/raft/host.rs +++ b/crates/cellar/durability/raft/host.rs @@ -1,119 +1,71 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Node-owned command producers and Raft/SQL effect dispatch. Waiters never own -//! a submitted producer, its private image or its progress through publication. +//! Physical host configuration and observations shared by the cell-group runtime. +//! The only production dispatcher/startup is group_host and its node controller. pub use super::host_archive::RaftArchiveOptions; -use super::{ - archive::{ArchiveError, CellArchive}, - budget::{Budget, Permit}, - cache::GroupLog, - catalog::*, - completion::*, - execution::*, - host_sql::{self as sql, Action, ImageView, Pool, Value}, - mailbox, - query::{Minimum, PendingRead, Phase as ReadPhase, ReadId}, - runtime::{RaftRuntime, RuntimeEvents}, - transport::{GroupRoute, RaftPeer, RaftTransport}, -}; -use crate::application::consensus::client::{ - self, Binding, ControlRequest, Kind, NodeKind, NodeRequest, Request, Response, -}; -use crate::application::consensus::{command_request_digest, OperationKind, RaftCells, RaftClass}; +use super::{catalog::*, completion::*, execution::*}; +use crate::application::consensus::RaftClass; use cellar_logic::consensus::*; -use cellar_logic::raft_command::CommandState; -use cellar_ltx::generation::{GenerationLimits, ImageId, ReadServices}; -use futures_util::{future::BoxFuture, stream::FuturesUnordered, StreamExt}; -use rand::RngCore; -use std::collections::{BTreeMap, BTreeSet, VecDeque}; -use std::path::PathBuf; -use std::sync::{ - atomic::{AtomicBool, Ordering}, - Arc, Mutex, -}; -use tokio::sync::{mpsc, Notify}; - -#[path = "host_image_limits.rs"] -mod image_limits; +use cellar_ltx::generation::GenerationLimits; +use std::sync::{Arc, Mutex}; #[path = "host_extents.rs"] mod extents; +#[path = "host_image_limits.rs"] +mod image_limits; pub use extents::{RaftExtentCacheLimits, RaftExtentLimits, RaftExtentOptions}; - -#[cfg(all(test, not(cellar_internal_tests)))] -#[path = "host_startup_tests.rs"] -mod startup_tests; - -#[cfg(test)] -use super::storage_fault; - -#[path = "host_archive_status.rs"] -mod archive_status; - -#[path = "host_admission.rs"] -mod admission; - #[path = "host_controller.rs"] mod controller; -#[path = "host_drain.rs"] -mod drain; -#[path = "host_effects.rs"] -mod effect_scheduler; pub use controller::{RaftControllerOptions, RaftControllerStatus}; #[path = "host_directory.rs"] mod directory; +pub use directory::RaftNodeDirectoryOptions; #[path = "host_enrollment.rs"] mod enrollment; pub(crate) use enrollment::NamespaceRegistration; pub use enrollment::{RaftManagedHostOptions, RaftNamespaceRegistration}; -#[path = "host_inspection.rs"] -mod inspection; #[path = "host_leader_load.rs"] mod load; -#[path = "host_readiness.rs"] -mod readiness; pub(crate) use load::GroupResourceObserver; pub use load::RaftInspectionScanOptions; -#[path = "host_fleet_admission.rs"] -mod fleet_admission; -#[path = "host_fleet.rs"] -mod fleet_control; #[path = "host_fleet_observation.rs"] mod fleet_observation; +pub(crate) use fleet_observation::group_membership_digest; pub use fleet_observation::{FleetMembershipConfiguration, FleetMembershipObservation}; #[path = "host_fleet_cut.rs"] mod fleet_cut; -pub(crate) use fleet_cut::Publication; -pub use fleet_cut::{FleetImageState, FleetPublishedCut, FleetPublishedImage}; -#[path = "host_fleet_inspection.rs"] -mod fleet_inspection; pub use super::fleet_inspection_wire::RaftPeerFleetMembershipObservation; -#[path = "host_initial.rs"] -mod initial; +pub(crate) use fleet_cut::{group_prefix, Publication}; +pub use fleet_cut::{FleetImageState, FleetPublishedCut, FleetPublishedImage}; #[path = "host_node.rs"] -mod node_control; -#[path = "host_registration.rs"] -mod registration; -use node_control::reply_node; -pub use node_control::{ +mod node; +use super::group_host::RaftCellsConfig; +pub use super::transport_options::RaftPeerNode; +pub use node::{ RaftHostEnrollmentObservation, RaftHostObservation, RaftHostStartupBounds, RaftPeerNodeObservation, }; -#[path = "host_fleet_execution.rs"] -mod fleet_execution; -#[path = "host_fleet_stop.rs"] -mod fleet_stop; -#[path = "host_membership.rs"] -mod membership; -#[path = "host_node_inspection.rs"] -mod node_inspection; -pub use directory::RaftNodeDirectoryOptions; -#[path = "host_seed.rs"] -mod seed; -#[path = "host_snapshot.rs"] -mod snapshots; -#[path = "host_standby.rs"] -mod standby; + +// These storage-observation regressions remain enabled while their owning +// generic cache/archive primitives are retained. No predecessor dispatcher runs. +#[cfg(test)] +use super::host_sql::ImageView; +#[cfg(test)] +use super::storage_fault; +#[cfg(test)] +#[path = "host_admission.rs"] +mod admission; +#[cfg(test)] +#[path = "host_archive_status.rs"] +mod archive_status; +#[cfg(test)] +#[path = "host_inspection.rs"] +mod inspection; +#[cfg(test)] +#[path = "host_readiness.rs"] +mod readiness; +#[cfg(test)] +use super::archive::ArchiveError; #[derive(Clone, Debug)] pub struct RaftHostOptions { @@ -320,57 +272,6 @@ impl RaftHostOptions { } } -use super::transport_options::NamespaceQueryOrigin; -pub use super::transport_options::RaftPeerNode; - -/// Local participation is explicit. A router never creates or attaches a -/// member and never initializes a SQLite cell image. -#[derive(Clone, Copy, Debug, PartialEq, Eq)] -pub enum RaftCellPlacement { - Replica, - /// Register an eligible local standby without creating a group or image. - /// A retained handle gains local admission only after authorized learner - /// initialization in this node's existing engine and SQL pool. - Standby, - /// Resume a stable joined member or seed a new committed learner from the - /// specified trusted peer. A claimed member with lost state is refused. - /// Fresh managed learners require authorized activation on a `Standby`; - /// this placement can still resume their intact retained engines. - Joining { - member: MemberId, - source: MemberId, - source_node: NodeId, - }, - /// Seed a fresh learner from an exact S3 manifest selected by a live peer - /// read barrier. Existing stable joined state resumes without reseeding. - /// Fresh managed learners use authorized `Standby` activation instead. - JoiningFromS3 { - member: MemberId, - source: MemberId, - source_node: NodeId, - }, - QueryRouter { - leader_hint: Option, - }, -} - -#[derive(Clone, Debug)] -pub struct RaftCellRegistration { - pub placement: RaftCellPlacement, - pub namespace: NamespaceConfig, - pub class: String, - pub birth: BirthProposal, -} - -pub struct RaftCellsConfig { - pub infrastructure: crate::daemon::NodeRaftConfig, - pub catalog: CatalogOptions, - pub cells: Vec, - pub peers: Vec, - pub query_origins: Vec, - pub host: RaftHostOptions, -} - pub struct RaftCellsServices { pub listeners: crate::daemon::NodeRaftListeners, pub tls: super::transport_options::RaftTls, @@ -378,3565 +279,3 @@ pub struct RaftCellsServices { pub application: Arc, pub versions: super::version::VersionKeys, } - -pub(crate) struct Registration { - pub placement: RaftCellPlacement, - pub namespace: NamespaceCatalog, - pub birth: GroupBirth, - pub class: Arc, -} - -pub(crate) struct Registrations { - pub cells: Vec, - pub namespaces: Vec, - pub peers: Vec, - pub origins: Vec, -} - -pub(crate) async fn start( - service: crate::daemon::NodeRaftService, - registrations: Registrations, - directory: PathBuf, - label: String, - options: RaftHostOptions, - versions: super::version::VersionKeys, -) -> anyhow::Result { - let log = service.log.clone(); - let runtime = service.runtime.clone(); - let transport = service.transport.clone(); - let result = Box::pin(initialize( - service, - registrations, - directory, - label, - options, - versions, - )) - .await; - if result.is_err() { - // This includes preflight and pool-construction errors before the - // initialized-cell cleanup exists. Retained handles must observe real - // closure; dropping the last caller is not a physical join boundary. - let _ = runtime.close().await; - transport.close().await; - let _ = log.close().await; - } - result -} - -async fn initialize( - service: crate::daemon::NodeRaftService, - registrations: Registrations, - directory: PathBuf, - label: String, - mut options: RaftHostOptions, - versions: super::version::VersionKeys, -) -> anyhow::Result { - let Registrations { - cells: registrations, - namespaces: declared_namespaces, - peers, - origins, - } = registrations; - options.validate()?; - for registration in ®istrations { - registration.class.validate_host(&options)?; - } - let snapshot_limits = service - .transport - .snapshot_limits(options.execution.max_image_bytes)?; - anyhow::ensure!( - (!registrations.is_empty() || options.managed.is_some()) - && registrations.len() <= options.max_groups, - "invalid registered group count" - ); - let domain = service.log.domain(); - let budget = Budget::new(options.max_pending_bytes, options.max_requests); - let archive_capture_budget = Budget::new( - options.archive.limits.max_capture_bytes * options.archive.max_producers, - options.archive.max_producers, - ); - let query_pins = Budget::new(0, options.max_query_pins); - let mut pool = Pool::start( - domain.clone(), - options.blocking_workers, - options.worker_queue, - options.max_groups, - super::host_sql::Budgets { - business: budget.clone(), - archive_capture: archive_capture_budget.clone(), - query_pins: query_pins.clone(), - }, - service.log.performance(), - )?; - let mut cells = BTreeMap::new(); - let mut standbys = BTreeMap::new(); - let mut bindings = BTreeMap::new(); - let mut routes = Vec::new(); - let mut archive_owner = None; - let mut node_directory = None; - let mut extent_owner = None; - let peer_count = peers.len(); - let mut peers: BTreeMap<_, _> = peers.into_iter().map(|peer| (peer.node, peer)).collect(); - let initialized = async { - let needs_extents = declared_namespaces.iter().any(|registration| { - registration.namespace.config().image_storage == RaftImageStorage::Extents - }) || registrations.iter().any(|registration| { - registration.namespace.config().image_storage == RaftImageStorage::Extents - && !matches!( - registration.placement, - RaftCellPlacement::QueryRouter { .. } - ) - }); - extent_owner = extents::Owner::open( - &service.log, - directory.clone(), - ltx_host(), - options.extents, - needs_extents, - ) - .await?; - anyhow::ensure!( - peer_count == peers.len() && peer_count <= service.transport.peer_limit(), - "invalid peer discovery count" - ); - let authorizations = enrollment::prepare( - &mut options, - &service, - ®istrations, - &declared_namespaces, - &peers, - &label, - ) - .await?; - let mut grants = BTreeSet::new(); - for origin in origins { - anyhow::ensure!( - grants.insert((origin.cluster, origin.namespace, origin.peer.node)), - "duplicate namespace query origin" - ); - service.transport.grant_query_origin(origin)?; - } - node_directory = directory::prepare( - options.node_directory.as_ref(), - ®istrations, - &declared_namespaces, - options.managed.as_mut().map(|m| &mut m.authorization), - &mut peers, - &service, - &label, - ) - .await?; - let namespaces: BTreeSet<_> = registrations - .iter() - .map(|registration| { - ( - registration.namespace.config().cluster, - registration.namespace.config().namespace, - ) - }) - .chain(declared_namespaces.iter().map(|registration| { - ( - registration.namespace.config().cluster, - registration.namespace.config().namespace, - ) - })) - .collect(); - let mut grants = BTreeSet::new(); - for origin in &options.inspection_origins { - anyhow::ensure!( - namespaces.contains(&(origin.cluster, origin.namespace)) - && peers.get(&origin.peer.node) == Some(&origin.peer) - && grants.insert((origin.cluster, origin.namespace, origin.peer.node)), - "inspection origin must match a unique registered namespace and trusted peer" - ); - service.transport.grant_inspection_origin(origin.clone())?; - } - for (index, registration) in registrations.into_iter().enumerate() { - let Registration { - mut placement, - namespace, - birth, - class, - } = registration; - namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; - anyhow::ensure!( - namespace.config().schema_digest == class.schema_digest(), - "registered schema disagrees with namespace" - ); - let scope = birth.scope().clone(); - let worker = index % options.blocking_workers; - let cell_directory = directory - .join(hex::encode(scope.namespace.bytes())) - .join(hex::encode(scope.group.bytes())) - .join(hex::encode(scope.incarnation.bytes())); - let rooted = match namespace.config().image_storage { - RaftImageStorage::Materialized => None, - RaftImageStorage::Extents - if matches!(placement, RaftCellPlacement::QueryRouter { .. }) => - { - None - } - RaftImageStorage::Extents => Some( - extent_owner - .as_ref() - .ok_or_else(|| anyhow::anyhow!("missing node extent owner"))? - .cell_storage(&cell_directory), - ), - }; - let storage = sql::CellStorage { - directory: cell_directory, - host: ltx_host(), - services: read_services(), - images: options.images, - rooted, - snapshot_scratch: service.log.snapshot_scratch(), - execution: options.execution, - }; - if placement == RaftCellPlacement::Standby { - anyhow::ensure!( - options.enable_snapshots, - "standby activation requires snapshots" - ); - anyhow::ensure!( - !birth - .voters() - .iter() - .chain(birth.learners()) - .any(|m| m.node == service.log.limits().node), - "standby is an initial member" - ); - anyhow::ensure!( - !standbys.contains_key(&scope) - && !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate standby registration" - ); - namespace.require_node_storage(&service.log).await?; - let existing = match service.log.group(&scope) { - Ok(group) => Some(group), - Err(super::storage::LogError::NotFound) => None, - Err(error) => return Err(error.into()), - }; - let resume = existing.as_ref().is_some_and(|group| { - group.read().is_ok_and(|cache| { - cache.membership.permits(group.identity()) - && super::member_state::members(&cache.conf_state) - .contains(&group.identity().member.get()) - }) - }); - standbys.insert( - scope.clone(), - standby::Dormant { - namespace: namespace.clone(), - birth: birth.clone(), - class: class.clone(), - storage: storage.clone(), - worker, - failed: existing.is_some() && !resume, - }, - ); - if resume { - let group = existing - .as_ref() - .ok_or_else(|| anyhow::anyhow!("standby resume state missing"))?; - let source = birth - .voters() - .first() - .ok_or_else(|| anyhow::anyhow!("standby birth has no voter"))?; - placement = RaftCellPlacement::Joining { - member: group.identity().member, - source: source.member, - source_node: source.node, - }; - } else { - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope, - class, - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(None), - leader_hint: None, - }, - ); - continue; - } - } - if let RaftCellPlacement::QueryRouter { leader_hint } = placement { - let scope = birth.scope().clone(); - let local = service.log.limits().node; - anyhow::ensure!( - !birth - .voters() - .iter() - .chain(birth.learners()) - .any(|member| member.node == local), - "query router is an initial group member" - ); - anyhow::ensure!( - matches!( - service.log.group(&scope), - Err(super::storage::LogError::NotFound) - ), - "query router has local member state" - ); - anyhow::ensure!( - leader_hint.is_none_or(|hint| birth - .voters() - .iter() - .any(|member| member.member == hint)), - "invalid query leader hint" - ); - anyhow::ensure!( - !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate registered cell" - ); - let routes = birth - .voters() - .iter() - .chain(birth.learners()) - .map(|member| { - let peer = peers - .get(&member.node) - .ok_or_else(|| anyhow::anyhow!("missing query peer route"))?; - Ok(RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: peer.node, - member: member.member, - disk: peer.disk, - process: peer.process, - }, - label: peer.label.clone(), - server_name: peer.server_name.clone(), - control_address: peer.control_address, - data_address: peer.data_address, - query_address: peer.query_address, - }) - }) - .collect::>>()?; - service.transport.install_query_group(GroupRoute { - scope: scope.clone(), - schema_digest: class.schema_digest(), - revision: 1, - peers: routes, - })?; - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope, - class, - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(None), - leader_hint, - }, - ); - continue; - } - anyhow::ensure!( - !cells.contains_key(&scope) - && !bindings.contains_key(&(scope.namespace, scope.cell.clone())), - "duplicate registered cell" - ); - // Member bootstrap has large archive and seed futures. Keep their - // inline state out of the node enrollment/discovery poll frame. - let permit = Box::pin(async { - let permit = match placement { - RaftCellPlacement::Replica => { - if birth.managed_owner().is_some() { - let authorization = authorizations.get(&scope).ok_or_else(|| { - anyhow::anyhow!("managed initial host authorization required") - })?; - namespace - .bootstrap_managed_member(&birth, &service.log, authorization) - .await? - } else { - // Keep disaster-only archive reconstruction out of every - // ordinary managed startup poll frame. - Box::pin(async { - if let Some(permit) = - namespace.disaster_seed_permit(&birth, &service.log).await? - { - anyhow::ensure!( - options.enable_snapshots, - "disaster recovery requires snapshots" - ); - let (source, reference) = - birth.disaster_source().ok_or_else(|| { - anyhow::anyhow!("missing disaster source") - })?; - let archive = namespace - .cell_archive( - source, - storage.host.clone(), - options.archive.limits, - )? - .with_scratch_directory( - service.log.limits().directory.clone(), - ) - .with_snapshot_scratch(service.log.snapshot_scratch()); - let archive = match &storage.rooted { - Some(rooted) => { - archive.with_root_store(rooted.store.clone()) - } - None => archive, - }; - let bytes = archive.read_memory_bound(); - let reservation = - service.transport.reserve_disaster_seed(bytes)?; - let set = archive - .with_read_retention(reservation.clone()) - .recovery_set_at(reference) - .await?; - let output = pool - .call( - worker, - scope.clone(), - Action::SealDisasterSeed { - received: Box::new(super::snapshot::DisasterSeed { - permit, - set, - schema: class.schema_digest(), - limits: snapshot_limits, - reservation, - }), - storage: storage.clone(), - }, - None, - None, - ) - .await?; - let Value::Seeded(seed) = output.value else { - anyhow::bail!("disaster SQL seed response") - }; - service.log.create_seeded(*seed).await?; - } - Ok::<(), anyhow::Error>(()) - }) - .await?; - namespace.bootstrap_member(&birth, &service.log).await? - } - } - RaftCellPlacement::Joining { - member, - source, - source_node, - } - | RaftCellPlacement::JoiningFromS3 { - member, - source, - source_node, - } => { - let from_s3 = matches!(placement, RaftCellPlacement::JoiningFromS3 { .. }); - anyhow::ensure!( - options.enable_snapshots, - "joining requires snapshot recovery" - ); - match namespace - .resume_joined(&birth, &service.log, member) - .await? - { - Some(permit) => permit, - None => { - let peer = peers.get(&source_node).ok_or_else(|| { - anyhow::anyhow!("missing seed source endpoint") - })?; - peer.validate_endpoint()?; - let local = service.log.limits(); - let identity = ReplicaIdentity { - scope: scope.clone(), - node: local.node, - member, - disk: local.disk, - process: local.process, - }; - let source = RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: peer.node, - member: source, - disk: peer.disk, - process: peer.process, - }, - label: peer.label.clone(), - server_name: peer.server_name.clone(), - control_address: peer.control_address, - data_address: peer.data_address, - query_address: peer.query_address, - }; - let limits = service - .transport - .snapshot_limits(options.execution.max_image_bytes)?; - let action = if from_s3 { - let archive = namespace - .cell_archive( - &birth, - storage.host.clone(), - options.archive.limits, - )? - .with_scratch_directory( - service.log.limits().directory.clone(), - ) - .with_snapshot_scratch(service.log.snapshot_scratch()); - let archive = match &storage.rooted { - Some(rooted) => { - archive.with_root_store(rooted.store.clone()) - } - None => archive, - }; - let received = service - .transport - .seed_from_archive( - identity, - source, - &birth, - class.schema_digest(), - limits, - archive, - ) - .await?; - Action::SealArchiveSeed { - received: Box::new(received), - storage: storage.clone(), - } - } else { - let received = service - .transport - .seed_from_peer( - identity, - source, - &birth, - class.schema_digest(), - limits, - storage.rooted.as_ref().map(|root| root.store.clone()), - ) - .await?; - Action::SealSeed { - received: Box::new(received), - storage: storage.clone(), - } - }; - let output = - pool.call(worker, scope.clone(), action, None, None).await?; - let Value::Seeded(seed) = output.value else { - anyhow::bail!("SQL seed response") - }; - // Managed fresh learners use registered standby activation - // with an opaque catalog authorization. Retained joined - // state above resumes without creating a new membership. - standby::check_seed_guard( - seed.membership(), - seed.local(), - None, - false, - )?; - service - .log - .performance() - .measure( - super::performance::Operation::RecoveryInstall, - service.log.create_seeded(*seed), - ) - .await?; - namespace - .resume_joined(&birth, &service.log, member) - .await? - .ok_or_else(|| { - anyhow::anyhow!("seed state missing after fence") - })? - } - } - } - RaftCellPlacement::QueryRouter { .. } | RaftCellPlacement::Standby => { - unreachable!("non-replica handled above") - } - }; - Ok::<_, anyhow::Error>(permit) - }) - .await?; - let group = service.log.group(&scope)?; - let extent_store = storage.rooted.as_ref().map(|root| root.store.clone()); - let output = pool - .call( - worker, - scope.clone(), - Action::Initialize { - group: group.clone(), - class: class.clone(), - storage, - }, - None, - None, - ) - .await?; - let Value::Initialized(image) = output.value else { - anyhow::bail!("SQL initialization response") - }; - routes.push(membership::route( - &group, - &birth, - &peers, - class.schema_digest(), - &label, - &service.transport, - )?); - bindings.insert( - (scope.namespace, scope.cell.clone()), - Binding { - scope: scope.clone(), - class: class.clone(), - minimum: namespace.minimum_durability(), - member: client::LocalMember::new(Some(group.identity().member)), - leader_hint: None, - }, - ); - let archive = namespace - .cell_archive(&birth, ltx_host(), options.archive.limits)? - .with_scratch_directory(service.log.limits().directory.clone()) - .with_snapshot_scratch(service.log.snapshot_scratch()) - .with_publication_coordination( - super::host_archive::publication_owner(group.identity().node_identity()), - options.archive.publication_stale_ms, - 64 * 1024 * 1024, - )?; - cells.insert( - scope.clone(), - Cell::recovered( - group, - birth.clone(), - archive, - image, - worker, - namespace.minimum_durability(), - class.schema_digest(), - )? - .with_extents(extent_store), - ); - permit.attach(&service.runtime).await?; - } - if !cells.is_empty() || !standbys.is_empty() || options.managed.is_some() { - let mut nonce = [0; 16]; - crate::asyncrt::rng("raft-archive-verifier").fill_bytes(&mut nonce); - let owner = super::host_archive::Owner::open( - directory.join("archive-verifier"), - ltx_host(), - ImageId::new(nonce)?, - options.images.apply, - pool.image_accounting(), - service.log.snapshot_scratch(), - super::host_archive::PhysicalOptions { - max_jobs: options.archive.max_blocking_jobs, - performance: service.log.performance(), - }, - ) - .await?; - for cell in cells.values_mut() { - cell.archive = cell - .archive - .clone() - .with_cancellation(owner.cancellation.subscribe()); - } - archive_owner = Some(owner); - } - // The host cannot drain runtime events until initialization completes. - // Publishing earlier lets peer traffic retain the ordinary request - // budget behind that drain and reject later group attachments. Keep - // routes private until every local group and the archive owner are ready. - for route in routes { - service.transport.install_group(route)?; - } - Ok::<_, anyhow::Error>(()) - } - .await; - if let Err(error) = initialized { - let _ = service.runtime.close().await; - service.transport.close().await; - let _ = pool.join().await; - let _ = service.log.close().await; - // A failed join can still own sealed roots in cells or standby storage. - // Release their final directory locks on a joined blocking owner too. - let cleanup = service.log.domain().blocking(move || { - drop((cells, standbys, extent_owner, archive_owner)); - }); - let cleanup = match cleanup { - Ok(job) => job.await.map_err(anyhow::Error::from), - Err(error) => Err(error.into()), - }; - return match cleanup { - Ok(()) => Err(error), - Err(cleanup) => { - Err(error.context(format!("physical startup cleanup failed: {cleanup}"))) - } - }; - } - let (sender, requests) = mailbox::channel(options.max_requests); - let (control_sender, controls) = mailbox::channel(options.max_control_requests); - let control_budget = Budget::new(options.max_control_bytes, options.max_control_requests); - let seed_archive_work = Budget::new(0, options.max_control_requests); - let (close, closes) = mpsc::unbounded_channel(); - let closed = Arc::new(AtomicBool::new(false)); - let closing = Arc::new(AtomicBool::new(false)); - let terminal = Arc::new(client::Terminal { - result: Mutex::new(None), - changed: Notify::new(), - }); - let controller_bindings = if options.controller.is_some() { - bindings.values().cloned().collect() - } else { - Vec::new() - }; - let has_effects = bindings - .values() - .any(|b| b.class.alarm_limits().is_some() || b.class.outbox_definition().is_some()); - let handle = Arc::new(client::Handle { - performance: service.log.performance(), - requests: sender, - control_requests: control_sender, - control_budget: control_budget.clone(), - close, - domain: domain.clone(), - closed: closed.clone(), - closing: closing.clone(), - admission: Arc::new(crate::application::consensus::admission::Admission::default()), - terminal: terminal.clone(), - budget: budget.clone(), - archive_capture_budget, - query_pins, - sql: Some(pool.resource_observer()), - extents: extent_owner.as_ref().map(|owner| owner.store.observer()), - groups: load::GroupResourceObserver::default(), - controller: Mutex::new(None), - fleet_restart_service: Mutex::new(None), - fleet_movement_service: Mutex::new(None), - transport: service.transport.resource_observer(), - max_input_bytes: options.max_input_bytes, - bindings: client::Bindings::new(bindings, options.max_groups), - work: crate::asyncrt::services().application_work_registry(), - }); - let installation = (|| -> anyhow::Result<()> { - if options.managed.is_some() { - crate::asyncrt::services() - .raft_lifecycle() - .bind_startup_admission(&handle.admission)?; - } - service.transport.install_query_gateway(handle.clone())?; - Ok(()) - })(); - if let Err(error) = installation { - let _ = service.runtime.close().await; - service.transport.close().await; - let _ = pool.join().await; - let _ = service.log.close().await; - return Err(error); - } - let local = handle.clone(); - let crate::daemon::NodeRaftService { - log, - runtime, - events, - transport, - } = service; - domain.spawn_detached("raft-cell-host", async move { - struct Closed(Arc, Arc); - - impl Drop for Closed { - fn drop(&mut self) { - self.0.store(true, Ordering::Release); - self.1.changed.notify_waiters(); - } - } - let _closed = Closed(closed, terminal.clone()); - let mut rng = crate::asyncrt::rng("raft-cell-commands"); - let mut read_nonce = [0; 16]; - rng.fill_bytes(&mut read_nonce); - let mut driver = Driver { - drain: None, - effect_scheduler: options - .effects - .filter(|_| has_effects) - .map(effect_scheduler::Scheduler::new), - controller: options - .controller - .as_ref() - .map(|options| controller::Controller::new(options.clone(), controller_bindings)), - leader_load: Some(load::Scanner::new(options.inspection_scan.clone())), - directory: node_directory, - peers, - label, - archive_owner, - extents: extent_owner, - log, - runtime, - events, - transport, - pool, - cells, - standbys, - declared_namespaces, - history_registration: None, - directory_path: directory, - activating: None, - active: BTreeSet::new(), - applying: BTreeSet::new(), - uncertain: BTreeMap::new(), - jobs: FuturesUnordered::new(), - options, - rng, - closing, - versions, - reads: BTreeMap::new(), - read_nonce, - read_sequence: 0, - budget, - control_budget, - seed_archive_work, - local, - snapshot_limits, - snapshot_source: None, - checkpoint_cursor: None, - maintenance_at: 0, - admission_at: 0, - admission_waiters: VecDeque::new(), - }; - driver.run(requests, controls, closes).await; - let result = driver.shutdown().await; - if let Ok(mut target) = terminal.result.lock() { - *target = Some(result); - } - terminal.changed.notify_waiters(); - })?; - Ok(RaftCells { handle }) -} - -fn ltx_host() -> cellar_ltx::LtxHost { - let fs = crate::asyncrt::fs(); - let age = fs.clone(); - let read = fs.clone(); - cellar_ltx::LtxHost::new( - crate::asyncrt::wall_ms, - move |path| { - Ok(std::time::Duration::from_millis( - crate::asyncrt::wall_ms() - .saturating_sub(age.metadata(path)?.modified_unix_millis) - .max(0) as u64, - )) - }, - move |path| { - let fs = read.clone(); - async move { - crate::asyncrt::try_blocking(move || fs.read(&path)) - .map_err(|e| std::io::Error::other(e.to_string()))? - .await - .map_err(|e| std::io::Error::other(e.to_string()))? - } - }, - |job| async move { - crate::asyncrt::try_blocking(job) - .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string()))? - .await - .map_err(|e| cellar_ltx::HostTaskError::new(e.to_string())) - }, - ) - .with_filesystem(fs) -} - -fn read_services() -> ReadServices { - ReadServices { - now_unix_millis: Arc::new(crate::asyncrt::wall_ms), - fill_random: Arc::new(|bytes| crate::asyncrt::rng("raft-image-read").fill_bytes(bytes)), - } -} - -fn progress( - group: &GroupLog, - image: Option<&ImageView>, - archived: LogIndex, -) -> Result, RuntimeError> { - let cache = group.read()?; - if cache.applied.image.as_ref() != image.map(|image| &image.image) { - return Ok(None); - } - let mut progress = ReplicaProgress::checkpoint( - group.identity().clone(), - cache.applied.position.index, - archived, - ) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - progress - .persist(LogIndex::new(cache.last_index())) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - progress - .commit(LogIndex::new(cache.hard_state.commit)) - .map_err(|e| RuntimeError::Invalid(e.to_string()))?; - Ok(Some(progress)) -} - -#[derive(Clone, Copy, PartialEq, Eq)] -enum Phase { - AwaitAdmission, - Start, - AwaitStart, - Staging, - BeforeSubmit, - AwaitSubmit, - Marking, - Proposing, - Waiting, - Finish, - AwaitFinish, - Discard, - Retiring, -} - -struct Pending { - allow_new: bool, - // Unknown until the callback's physical preview is verified. Keep this - // bound after proposal submission takes `prepared`, through proof/reply. - archive_bytes: Option, - // Only transient admission deferral is timed here, until grant/refusal. - // Dropping a cancelled/retired waiter records abandonment, never a grant. - admission_wait: Option, - system: bool, - proof: Option<(LogPosition, [u8; 32])>, - request: Request, - options: client::CommandOptions, - contract: crate::application::consensus::OperationContract, - input: Option>, - identity: Option, - prepared: Option, - result: Option, - duplicate: bool, - phase: Phase, - _bytes: Option, -} - -struct Cell { - extents: Option, - birth: GroupBirth, - route_revision: LogIndex, - configuration: Option, - retired: bool, - archive: CellArchive, - archived: LogIndex, - archive_retained: admission::Retained, - archive_observation: archive_status::Observation, - archive_retry_at: u64, - group: GroupLog, - schema_digest: [u8; 32], - worker: usize, - core: CommandState, - image: Option, - pending: Option, - sequence: u64, - status_busy: bool, - prefer_delivery: bool, - sql_busy: bool, - apply: Option, - publication: Option, - apply_ticket: Option, - apply_retry_at: u64, - failed: bool, - incoming: Option, - source_request: Option, - apply_inflight: bool, - checkpoint_busy: bool, - checkpoint_retry_at: u64, - archive_checkpoint_at: u64, -} - -enum StatusPurpose { - Start, - Submit, - Finish, - Poll, - Observe(Box), -} - -enum SqlPurpose { - EffectScan(u64), - Seed(ReadId), - Checkpoint, - SnapshotSource(Box<::raft::prelude::Message>), - SnapshotPrepare(ApplyTicket), - SnapshotPublish(ApplyTicket), - CaptureArchive, - Stage, - Mark, - Discard, - Retire, - Prepare(RuntimeEvent), - Publish(ApplyTicket), - Query(ReadId), -} - -enum Job { - MessageDelivery { - scope: GroupScope, - member: u64, - result: Result<(), super::transport::TransportError>, - }, - FleetMovementServiceStopped, - FleetRestartServiceStopped, - FleetAction(Box), - PeerFleetStop(Box), - PeerFleetInspection(Box), - LocalFleetInspection(Box), - FleetAdmission(Box), - NodeInspection(Box), - HistoryRegistered(Box), - Drained(Box), - Effect(Box), - Controller(Box), - LeaderLoad(Box), - Inspection(Box), - PeerInspection(Box), - Directory(Box), - Standby(Box), - Leadership { - request: Box, - result: Result, - }, - SeedArchive { - id: ReadId, - result: Box, ArchiveError>>, - }, - Configured { - scope: GroupScope, - result: Result, - }, - SnapshotSent { - lease: u64, - result: Result<(), super::transport::TransportError>, - }, - SnapshotFetched { - ticket: ApplyTicket, - result: Box< - Result, - >, - }, - SnapshotAuthorized { - ticket: ApplyTicket, - result: Result<(), RuntimeError>, - }, - SnapshotAbort { - ticket: ApplyTicket, - result: Result<(), RuntimeError>, - }, - Checkpoint { - scope: GroupScope, - result: Result<(), RuntimeError>, - }, - Archived(Vec), - ArchiveBatchJoined, - Routed { - request: Box, - result: Result, - }, - ReadStarted { - id: ReadId, - attempt: u64, - result: Result<(), RuntimeError>, - }, - ReadSource { - id: ReadId, - result: Result, - }, - ReadLeader { - id: ReadId, - check: u64, - result: Result, - }, - Status { - scope: GroupScope, - sequence: u64, - purpose: StatusPurpose, - result: Result, - }, - Sql { - scope: GroupScope, - purpose: SqlPurpose, - result: Box>, - work: Option>, - }, - SnapshotEncoded { - scope: GroupScope, - purpose: SqlPurpose, - result: Box>, - }, - Proposed { - scope: GroupScope, - identity: CommandIdentity, - result: Result, - }, - Installed { - scope: GroupScope, - result: Result<(), RuntimeError>, - }, - Hint, - ExtentGc(anyhow::Result), -} - -struct Driver { - maintenance_at: u64, - admission_at: u64, - admission_waiters: VecDeque, - drain: Option, - effect_scheduler: Option, - controller: Option, - leader_load: Option, - directory: Option, - standbys: BTreeMap, - declared_namespaces: Vec, - history_registration: Option>, - directory_path: PathBuf, - activating: Option, - peers: BTreeMap, - label: String, - snapshot_limits: super::snapshot::SnapshotLimits, - snapshot_source: Option, - checkpoint_cursor: Option, - archive_owner: Option, - extents: Option, - local: Arc, - log: super::storage::ReplicaLog, - runtime: RaftRuntime, - events: RuntimeEvents, - transport: RaftTransport, - pool: Pool, - cells: BTreeMap, - active: BTreeSet, - applying: BTreeSet, - uncertain: BTreeMap, - jobs: FuturesUnordered>, - options: RaftHostOptions, - rng: crate::asyncrt::rng::Stream, - closing: Arc, - versions: super::version::VersionKeys, - reads: BTreeMap, - read_nonce: [u8; 16], - read_sequence: u64, - budget: Arc, - control_budget: Arc, - seed_archive_work: Arc, -} - -fn reply(request: &mut Request, result: Result) { - reply_charged(request, result, None); -} - -fn reply_charged( - request: &mut Request, - result: Result, - bytes: Option, -) { - if let Some(reply) = request.reply.take() { - let _ = reply.send(result.map(|response| client::ChargedResponse { - response, - _bytes: bytes, - })); - } -} - -fn failure(error: ExecutionError) -> CommandFailure { - match error { - ExecutionError::Refused(failure) => failure, - ExecutionError::Registration(_) => CommandFailure::UnsupportedCapability, - _ => CommandFailure::Unavailable, - } -} - -impl Driver { - fn finish_read(&mut self, id: &ReadId, result: Result) { - if let Some(mut read) = self.reads.remove(id) { - if result.is_err() { - read.request.control.cancel(); - } - if let Some(timer) = read.archive_wait.take() { - timer.finish(result.is_ok()); - } - reply(&mut read.request, result); - } - } - - fn pump_reads(&mut self) { - let now = crate::asyncrt::mono_ms(); - for id in self.reads.keys().copied().collect::>() { - let Some(read) = self.reads.get_mut(&id) else { - continue; - }; - let scope = read.request.scope.clone(); - let Some(cell) = self.cells.get(&scope) else { - self.finish_read(&id, Err(CommandFailure::Unavailable)); - continue; - }; - if read.expired() || cell.failed || cell.retired { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - continue; - } - if cell.incoming.is_some() { - continue; - } - if now < read.retry_at { - continue; - } - match read.phase { - ReadPhase::Source => { - read.phase = ReadPhase::Probing; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadSource { - id, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::StartBarrier => { - let Some(check) = read.leader_check.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.leader_check = check; - read.phase = ReadPhase::CheckingLeader; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadLeader { - id, - check, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::Barrier if !read.checking_leader => { - let Some(check) = read.leader_check.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.leader_check = check; - read.checking_leader = true; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadLeader { - id, - check, - result: runtime.status(&scope).await, - } - })); - } - ReadPhase::Image => { - if matches!(&read.operation, super::query::ReadOperation::Fleet { .. }) { - self.finish_fleet_observation(id); - continue; - } - let metadata_only = matches!( - &read.operation, - super::query::ReadOperation::Seed { - kind: super::transport::seeds::SeedKind::S3, - .. - } - ); - if !metadata_only - && (cell.sql_busy || cell.apply.is_some() || cell.publication.is_some()) - { - continue; - } - if let super::query::ReadOperation::Seed { - target, - kind, - charge, - } = &read.operation - { - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - continue; - }; - if !seed::valid_target(cell, target, &self.peers) { - self.finish_read(&id, Err(CommandFailure::Unavailable)); - continue; - } - let ready = !cell.apply_inflight - && cell.group.read().is_ok_and(|cache| { - cache.applied.position.index >= *barrier - && cache.applied.image.as_ref() - == cell.image.as_ref().map(|image| &image.image) - }); - if !ready { - continue; - } - if *kind == super::transport::seeds::SeedKind::S3 { - // One owner spans all metadata retries and waiting - // for the archive to cover this exact ReadIndex. - read.archive_wait.get_or_insert_with(|| { - self.local - .performance - .timer(super::performance::Operation::RecoveryArchiveWait) - }); - let archive = cell.archive.clone().with_read_retention(charge.clone()); - let Some(work) = self.seed_archive_work.acquire(0) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - }; - read.phase = ReadPhase::Executing; - self.jobs.push(Box::pin(async move { - // Ownership survives expiry/removal of the read waiter. - let _work = work; - Job::SeedArchive { - id, - result: Box::new(archive.recovery_metadata().await), - } - })); - continue; - } - let reservation = match self.transport.reserve_snapshot_source() { - Ok(reservation) => reservation, - Err(_) => { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - continue; - } - }; - read.phase = ReadPhase::Executing; - self.sql( - &scope, - SqlPurpose::Seed(id), - Action::CaptureSeed { - limits: self.snapshot_limits, - reservation, - }, - ); - continue; - } - let Some(image) = cell.image.as_ref() else { - continue; - }; - let candidate = cell.group.read().ok().and_then(|cache| { - (cache.applied.image.as_ref() == Some(&image.image)).then(|| { - CommittedVersion { - scope: scope.clone(), - position: cache.applied.position, - } - }) - }); - let Some(candidate) = candidate else { - continue; - }; - let Some(minimum) = read.minimum.clone() else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - continue; - }; - match minimum.satisfied(&candidate, &scope) { - Ok(false) => continue, - Err(error) => { - self.finish_read(&id, Err(error)); - continue; - } - Ok(true) => (), - } - read.phase = ReadPhase::Executing; - let super::query::ReadOperation::Query { contract, input } = &read.operation - else { - continue; - }; - let action = Action::Query { - contract: contract.clone(), - input: input.clone(), - minimum, - control: read.request.control.clone(), - source_deadline: read.source_deadline, - }; - self.sql(&scope, SqlPurpose::Query(id), action); - } - _ => (), - } - } - } - - fn on_read_source(&mut self, id: ReadId, result: Result) { - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if read.phase != ReadPhase::Probing { - return; - } - let now = crate::asyncrt::mono_ms(); - match result { - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - read.phase = ReadPhase::Source; - read.retry_at = now.saturating_add(self.options.poll_ms); - } - Ok(status) - if self - .cells - .get(&read.request.scope) - .is_some_and(|cell| status.replica == *cell.group.identity()) => - { - if let Some(deadline) = super::query::recent_source_deadline( - status.source_observed_ms, - now, - self.options.stale_source_timeout_ms, - ) { - read.source_deadline = Some(deadline); - read.phase = ReadPhase::Image; - } else { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - } - } - _ => self.finish_read(&id, Err(CommandFailure::VersionUnavailable)), - } - } - - fn on_read_leader( - &mut self, - id: ReadId, - check: u64, - result: Result, - ) { - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if check != read.leader_check - || !matches!(read.phase, ReadPhase::CheckingLeader | ReadPhase::Barrier) - { - return; - } - read.checking_leader = false; - read.retry_at = crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - let status = match result { - Ok(status) - if self - .cells - .get(&read.request.scope) - .is_some_and(|cell| status.replica == *cell.group.identity()) => - { - status - } - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - if read.phase == ReadPhase::CheckingLeader { - read.phase = ReadPhase::StartBarrier; - } - return; - } - _ => { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - return; - } - }; - let leader = (status.term, status.leader, status.is_leader); - if read.phase == ReadPhase::Barrier { - // RawNode may clear a pending read on role change, or its reply - // may be lost without any leadership change. Retry the same - // invocation after a checked observation and bounded delay. The - // original deadline remains fixed; status never supplies proof. - if read.leader != Some(leader) || crate::asyncrt::mono_ms() >= read.barrier_retry_at { - read.phase = ReadPhase::StartBarrier; - } - return; - } - if status.leader.is_none() - || status.applied.term != status.term - || (status.is_leader && !status.protocol_ready) - { - read.phase = ReadPhase::StartBarrier; - return; - } - let Some(attempt) = read.barrier_attempt.checked_add(1) else { - self.finish_read(&id, Err(CommandFailure::Overloaded)); - return; - }; - read.barrier_attempt = attempt; - read.leader = Some(leader); - read.phase = ReadPhase::Barrier; - read.barrier_retry_at = crate::asyncrt::mono_ms() - .saturating_add(self.options.poll_ms.saturating_mul(4).clamp(100, 1000)); - let scope = read.request.scope.clone(); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::ReadStarted { - id, - attempt, - result: runtime.read_index(&scope, id.to_vec()).await, - } - })); - } - - fn status(&mut self, scope: &GroupScope, purpose: StatusPurpose) { - let Some(cell) = self.cells.get_mut(scope) else { - return; - }; - if cell.status_busy { - if let StatusPurpose::Observe(mut request) = purpose { - reply(&mut request, Err(CommandFailure::Overloaded)); - } - return; - } - let Some(sequence) = cell.sequence.checked_add(1) else { - self.fail(scope); - return; - }; - cell.sequence = sequence; - cell.status_busy = true; - let runtime = self.runtime.clone(); - let scope = scope.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime.status(&scope).await; - Job::Status { - scope, - sequence, - purpose, - result, - } - })); - } - - fn sql(&mut self, scope: &GroupScope, purpose: SqlPurpose, action: Action) { - let Some(cell) = self.cells.get_mut(scope) else { - return; - }; - cell.sql_busy = true; - let work = match &purpose { - SqlPurpose::Query(id) | SqlPurpose::Seed(id) => { - self.reads.get(id).map(|read| read.request.work.clone()) - } - _ => cell.pending.as_ref().map(|p| p.request.work.clone()), - }; - let cancel = match &purpose { - SqlPurpose::Query(id) | SqlPurpose::Seed(id) => { - self.reads.get(id).map(|read| read.request.control.clone()) - } - SqlPurpose::Stage => cell - .pending - .as_ref() - .map(|pending| pending.request.control.clone()), - _ => None, - }; - let encoding_work = work.clone(); - let job = self - .pool - .call(cell.worker, scope.clone(), action, work, cancel); - let scope = scope.clone(); - self.jobs.push(Box::pin(async move { - Job::Sql { - scope, - purpose, - result: Box::new(job.await), - work: encoding_work, - } - })); - } - - fn finish_pending(&mut self, scope: &GroupScope, result: Result) { - self.finish_pending_charged(scope, result, None); - } - - fn finish_pending_charged( - &mut self, - scope: &GroupScope, - result: Result, - bytes: Option, - ) { - if let Some(cell) = self.cells.get_mut(scope) { - if let Some(mut pending) = cell.pending.take() { - let result = if pending.request.control.cancelled() - && (result.is_ok() - || pending.duplicate && matches!(result, Err(CommandFailure::Unavailable))) - { - Err( - if pending.request.control.submitted() || pending.duplicate { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::Unavailable - }, - ) - } else { - result - }; - reply_charged(&mut pending.request, result, bytes); - } - } - self.active.remove(scope); - self.admission_waiters.retain(|queued| queued != scope); - self.admission_at = 0; - } - - #[track_caller] - fn fail(&mut self, scope: &GroupScope) { - if let Some(cell) = self.cells.get_mut(scope) { - if !cell.failed { - tracing::error!(group = ?scope.group, location = %std::panic::Location::caller(), "Raft host closed group admission after a failure"); - } - cell.failed = true; - cell.core.close(); - if let Some(pending) = cell.pending.as_mut() { - if pending.request.control.submitted() { - self.uncertain.insert(scope.clone(), pending.options.id); - } - } - } - let result = self - .cells - .get(scope) - .and_then(|c| c.pending.as_ref()) - .map(|p| { - if p.request.control.submitted() { - CommandFailure::Ambiguous { - command: p.options.id, - } - } else { - CommandFailure::CorruptState - } - }) - .unwrap_or(CommandFailure::CorruptState); - self.finish_pending(scope, Err(result)); - } - - fn refuse(&mut self, scope: &GroupScope, error: CommandFailure) { - if let Some(pending) = self - .cells - .get_mut(scope) - .and_then(|cell| cell.pending.as_mut()) - { - if pending.request.control.submitted() { - reply( - &mut pending.request, - Err(CommandFailure::Ambiguous { - command: pending.options.id, - }), - ); - pending.phase = Phase::Waiting; - } else { - reply(&mut pending.request, Err(error)); - pending.phase = Phase::Discard; - } - } - } - - fn on_routed_query(&mut self, mut request: Request) { - let scope = request.scope.clone(); - let Ok(binding) = self - .local - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .and_then(|binding| { - if binding.scope == scope { - Ok(binding) - } else { - Err(CommandFailure::CommandConflict) - } - }) - else { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - }; - let Kind::Query { - contract, - input, - options, - } = std::mem::replace(&mut request.kind, Kind::Status) - else { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - }; - if contract.kind != OperationKind::Query { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let minimum = if let client::QueryConsistency::AtLeast(token) = &options.consistency { - match u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| super::version::VersionError::Expired) - .and_then(|now| { - self.versions - .verify(token, &scope, binding.class.schema_digest(), now) - }) { - Ok(version) => Some(version), - Err(error) => { - let failure = if error == super::version::VersionError::WrongHistory { - CommandFailure::WrongIncarnation - } else { - CommandFailure::VersionUnavailable - }; - reply(&mut request, Err(failure)); - return; - } - } - } else { - None - }; - let context = super::query_route::RouteContext { - member: binding.member(), - leader_hint: binding.leader_hint, - local: crate::application::consensus::RaftCellHandle { - handle: self.local.clone(), - binding, - }, - runtime: self.runtime.clone(), - transport: self.transport.clone(), - keys: self.versions.clone(), - control: request.control.clone(), - minimum, - }; - self.jobs.push(Box::pin(async move { - Job::Routed { - request: Box::new(request), - result: context.run(contract, input, options).await, - } - })); - } - - fn on_request(&mut self, mut request: Request) { - self.reconcile_membership(&request.scope); - let scope = request.scope.clone(); - if matches!(&request.kind, Kind::Inspect | Kind::InspectPeer(_)) { - if self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::Unavailable)); - } else if !request.control.cancelled() { - match request.kind { - Kind::InspectPeer(peer) if peer != self.log.limits().node => { - self.start_peer_inspection(request, peer) - } - _ => self.start_inspection(request), - } - } - return; - } - let system = matches!(&request.kind, Kind::SystemCommand { .. }); - let admission = if matches!( - &request.kind, - Kind::Command { .. } | Kind::SystemCommand { .. } - ) { - match self.archive_admission() { - admission::Decision::Granted if !self.admission_waiters.is_empty() => { - admission::Decision::Deferred - } - decision => decision, - } - } else { - admission::Decision::Refused - }; - if self.closing.load(Ordering::Acquire) - || self - .cells - .get(&scope) - .is_some_and(|cell| cell.failed || cell.retired) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if request.control.cancelled() { - return; - } - if matches!( - &request.kind, - Kind::ActivateLearner(_) | Kind::ActivateManaged { .. } | Kind::ActivateInitial(_) - ) { - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::ActivateLearner(activation) => { - self.activate_learner(request, activation, None) - } - Kind::ActivateManaged { - activation, - authorization, - } => self.activate_learner(request, activation, Some(*authorization)), - Kind::ActivateInitial(authorization) => { - self.activate_initial(request, *authorization) - } - _ => reply(&mut request, Err(CommandFailure::CorruptState)), - } - return; - } - if matches!(&request.kind, Kind::Seed { .. }) { - if let Kind::Seed { - target, - kind, - charge, - } = std::mem::replace(&mut request.kind, Kind::Status) - { - self.start_seed(request, target, kind, charge); - } - return; - } - if matches!(&request.kind, Kind::Query { options, .. } if !matches!(options.routing, client::QueryRouting::Local)) - { - self.on_routed_query(request); - return; - } - if matches!( - &request.kind, - Kind::BeginRepair { .. } | Kind::AdvanceRepair { .. } | Kind::Fleet(_) - ) { - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::BeginRepair { plan } => self.begin_repair(request, plan, None), - Kind::AdvanceRepair { - id, - step, - expected_revision, - } => self.advance_repair(request, id, step, expected_revision, None), - Kind::Fleet(action) => self.fleet_configuration(request, *action), - _ => reply(&mut request, Err(CommandFailure::CorruptState)), - } - return; - } - if matches!(&request.kind, Kind::AddLearner { .. }) { - if let Kind::AddLearner { - member, - expected_revision, - } = std::mem::replace(&mut request.kind, Kind::Status) - { - self.add_learner(request, member, expected_revision); - } - return; - } - let Some(cell) = self.cells.get_mut(&scope) else { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - }; - if cell.failed || self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if cell.incoming.is_some() - && matches!( - &request.kind, - Kind::Command { .. } | Kind::SystemCommand { .. } - ) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if request.control.cancelled() { - return; - } - match std::mem::replace(&mut request.kind, Kind::Status) { - Kind::ReplicationStatus => { - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime - .replication_status(&scope) - .await - .map(Response::Replication); - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::TransferLeadership(transfer) => { - if cell.pending.is_some() || cell.configuration.is_some() || cell.incoming.is_some() - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if !request.control.begin_submission() { - return; - } - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = if request.control.cancelled() { - Err(RuntimeError::Closed) - } else { - runtime - .transfer_leadership(&scope, transfer) - .await - .map(|()| Response::LeadershipRequested) - }; - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::RestartLeadership(mut check) => { - if check.transfer.is_some() - && (cell.pending.is_some() - || cell.configuration.is_some() - || cell.incoming.is_some()) - { - reply(&mut request, Err(CommandFailure::Unavailable)); - return; - } - if !request.control.begin_submission() { - return; - } - check.deadline = check.deadline.min(request.control.deadline()); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = if request.control.cancelled() { - Err(RuntimeError::Closed) - } else { - runtime - .restart_leadership(&scope, check) - .await - .map(|()| Response::LeadershipRequested) - }; - Job::Leadership { - request: Box::new(request), - result, - } - })); - } - Kind::Inspect - | Kind::InspectPeer(_) - | Kind::AddLearner { .. } - | Kind::ActivateLearner(_) - | Kind::ActivateManaged { .. } - | Kind::ActivateInitial(_) - | Kind::Fleet(_) - | Kind::Seed { .. } - | Kind::BeginRepair { .. } - | Kind::AdvanceRepair { .. } => { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)) - } - Kind::Status => self.status(&scope, StatusPurpose::Observe(Box::new(request))), - Kind::RepairStatus => { - let result = cell - .group - .membership() - .map_err(|_| CommandFailure::CorruptState) - .and_then(|membership| { - membership - .repair_status() - .map_err(|_| CommandFailure::CorruptState) - }) - .map(Response::Repair); - reply(&mut request, result); - } - Kind::FleetStatus => { - let result = cell - .group - .membership() - .map(|membership| Response::Fleet(membership.fleet_status().map(Box::new))) - .map_err(|_| CommandFailure::CorruptState); - reply(&mut request, result); - } - Kind::ObserveFleet(authorization) => { - self.start_fleet_observation(request, *authorization); - } - Kind::Query { - contract, - input, - options, - } => { - if contract.kind != OperationKind::Query { - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let minimum = if let client::QueryConsistency::AtLeast(token) = &options.consistency - { - let decoded = u64::try_from(crate::asyncrt::wall_ms()) - .map_err(|_| super::version::VersionError::Expired) - .and_then(|now| { - self.versions.verify(token, &scope, cell.schema_digest, now) - }); - match decoded { - Ok(version) => Some(version), - Err(error) => { - let failure = if error == super::version::VersionError::WrongHistory { - CommandFailure::WrongIncarnation - } else { - CommandFailure::VersionUnavailable - }; - reply(&mut request, Err(failure)); - return; - } - } - } else { - None - }; - let Some(sequence) = self.read_sequence.checked_add(1) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - self.read_sequence = sequence; - let mut id = [0; 32]; - id[..8].copy_from_slice(b"CLRRD001"); - id[8..24].copy_from_slice(&self.read_nonce); - id[24..].copy_from_slice(&sequence.to_be_bytes()); - self.reads.insert( - id, - PendingRead::new(request, contract, input, &options, minimum), - ); - } - Kind::Command { - contract, - input, - options, - } - | Kind::SystemCommand { - contract, - input, - options, - } => { - // Background SQL does not consume the one business-command slot. - // Keep the bounded request until the physical worker completes; - // pump() rechecks its original deadline before staging. - if cell.pending.is_some() { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - } - cell.pending = Some(Pending { - allow_new: admission == admission::Decision::Granted, - archive_bytes: None, - admission_wait: (admission == admission::Decision::Deferred).then(|| { - self.local - .performance - .timer(super::performance::Operation::ArchiveAdmissionWait) - }), - system, - proof: None, - request, - options, - contract, - input: Some(input), - identity: None, - prepared: None, - result: None, - duplicate: false, - phase: if admission == admission::Decision::Deferred { - Phase::AwaitAdmission - } else { - Phase::Start - }, - _bytes: None, - }); - let sql_idle = !cell.sql_busy; - self.active.insert(scope.clone()); - if admission == admission::Decision::Deferred { - self.admission_waiters.push_back(scope); - } else if sql_idle { - self.status(&scope, StatusPurpose::Start); - } - } - } - } - - fn on_status( - &mut self, - scope: GroupScope, - sequence: u64, - purpose: StatusPurpose, - result: Result, - ) { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.status_busy = false; - let accepted = result.and_then(|status| { - if status.replica != *cell.group.identity() - || cell.group.read()?.hard_state.term > status.term.get() - { - return Err(RuntimeError::NotReady); - } - if let Some(progress) = progress(&cell.group, cell.image.as_ref(), cell.archived)? { - cell.core - .observe_progress(progress) - .map_err(|_| RuntimeError::StaleCompletion)?; - } - let change = cell - .core - .observe_leadership(status.leadership_observation(sequence)?) - .map_err(|_| RuntimeError::StaleCompletion)?; - if change.retired.is_some() { - if let Some(pending) = cell.pending.as_mut() { - let error = if pending.request.control.submitted() { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::NotLeader { - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - } - }; - reply(&mut pending.request, Err(error)); - } - } - cell.archive_observation.observe( - status.committed, - cell.archived, - crate::asyncrt::mono_ms(), - ); - Ok(status) - }); - if let StatusPurpose::Observe(mut request) = purpose { - let result = accepted.and_then(|status| { - let now = self - .log - .domain() - .mono_ms() - .map_err(|_| RuntimeError::Closed)?; - let registered_queries = self - .local - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .map_err(|_| RuntimeError::Closed)? - .class - .has_queries(); - let (role, queries) = readiness::capture( - cell, - &status, - now, - self.options.stale_source_timeout_ms, - self.local.admission.is_draining(), - registered_queries, - )?; - Ok(Response::Status(client::CellStatus { - role, - queries, - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - term: status.term, - command_ready: role == crate::application::consensus::RaftReplicaRole::Leader - && cell.pending.is_none() - && cell.incoming.is_none() - && cell.core.authority().is_some() - && status.protocol_ready - && !status.awaiting_apply - && !status.awaiting_ready - && status.persisted == status.committed - && status.committed == status.applied.index, - version: cell.image.as_ref().map(|image| CommittedVersion { - scope: scope.clone(), - position: image.position, - }), - persisted: status.persisted, - committed: status.committed, - applied: status.applied, - checkpoint: status.checkpoint, - archived: cell.archived, - archive: cell.archive_observation.capture( - status.committed, - cell.archived, - now, - cell.archive_retained.bytes(&cell.group, cell.archived)?, - self.archive_owner - .as_ref() - .is_some_and(|owner| owner.inflight.contains(&scope)), - ), - membership_revision: status.membership_revision, - member_high_water: status.member_high_water, - })) - }); - reply( - &mut request, - result.map_err(|_| CommandFailure::Unavailable), - ); - return; - } - let status = match accepted { - Ok(status) => status, - Err(_) => { - if matches!(purpose, StatusPurpose::Start | StatusPurpose::Submit) { - self.refuse(&scope, CommandFailure::Unavailable); - } - return; - } - }; - let Some(pending) = cell.pending.as_mut() else { - return; - }; - if pending.request.control.cancelled() && !pending.request.control.submitted() { - pending.phase = Phase::Discard; - return; - } - match purpose { - StatusPurpose::Start => { - // A reader or archive capture may acquire SQL while status is - // in flight. Keep one bounded command and recheck authority - // after that work physically completes. - if cell.sql_busy { - pending.phase = Phase::Start; - return; - } - let Some(authority) = cell.core.authority().cloned() else { - if status.is_leader { - pending.phase = Phase::AwaitStart; - return; - } - self.refuse( - &scope, - CommandFailure::NotLeader { - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - }, - ); - return; - }; - let mut bytes = [0; 16]; - self.rng.fill_bytes(&mut bytes); - let Ok(proposal) = ProposalId::new(bytes) else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let identity = match cell.core.admit( - &authority, - pending.options.id, - proposal, - pending.options.durability, - ) { - Ok(identity) => identity, - Err(Refusal::LeaderNotReady) => { - pending.phase = Phase::AwaitStart; - return; - } - Err(_) => { - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - }; - pending.identity = Some(identity.clone()); - let Some(input) = pending.input.take() else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let mut preview = [0; 16]; - self.rng.fill_bytes(&mut preview); - let Ok(preview) = ImageId::new(preview) else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - let durability = match cell.core.receipt_durability(&identity) { - Ok(value) => value, - Err(_) => { - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - }; - let mut random = [0; 32]; - self.rng.fill_bytes(&mut random); - let attempt = CommandAttempt { - request: CommandRequest { - id: pending.options.id, - operation: pending.contract.operation.clone(), - contract_digest: pending.contract.digest, - request_digest: command_request_digest(&input), - durability, - retry_until_ms: pending.options.retry_until_ms, - }, - proposal, - preview, - inputs: crate::storage::CommandInputs { - now_unix_millis: crate::asyncrt::wall_ms().max(0) as u64, - random, - }, - }; - pending.phase = Phase::Staging; - let allow_new = pending.allow_new; - let system = pending.system; - self.sql( - &scope, - SqlPurpose::Stage, - Action::Stage { - identity, - status: Box::new(status), - attempt, - input, - allow_new, - system, - }, - ); - } - StatusPurpose::Submit => { - let Some(identity) = pending.identity.clone() else { - self.refuse(&scope, CommandFailure::Unavailable); - return; - }; - if !pending.request.control.begin_submission() { - pending.phase = Phase::Discard; - return; - } - if let Err(reason) = cell.core.mark_submitted(&identity) { - pending.request.control.refuse_preflight(); - if reason == Refusal::LeaderNotReady { - // Ready persistence or protocol-only work may be in - // flight between two status requests. Keep the frozen - // private result; retry the authority check on a tick, - // under the original attempt deadline, before any RPC. - tracing::debug!(?reason, "Raft private submission preflight deferred"); - pending.phase = Phase::AwaitSubmit; - return; - } - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - pending.phase = Phase::Marking; - self.sql(&scope, SqlPurpose::Mark, Action::Submit { identity }); - } - StatusPurpose::Finish => { - let (Some(identity), Some(receipt)) = - (pending.identity.as_ref(), pending.result.as_ref()) - else { - return; - }; - let result = if pending.duplicate { - cell.core.recover_receipt( - identity, - receipt.receipt().command(), - CommittedVersion { - scope: scope.clone(), - position: receipt.position(), - }, - receipt.receipt().minimum_durability(), - ) - } else { - cell.core.verify_receipt( - identity, - receipt.receipt().command(), - receipt.position(), - receipt.receipt().minimum_durability(), - ) - }; - if let Err(reason) = result { - if reason == Refusal::LeaderNotReady { - pending.phase = Phase::AwaitFinish; - return; - } - self.refuse(&scope, CommandFailure::Unavailable); - return; - } - if cell.archived >= receipt.position().index { - if let Some((position, entry_digest)) = pending.proof { - if cell - .core - .observe_proof(&OutputProof::Raft(Box::new(RaftProof { - command: identity.clone(), - position, - entry_digest, - stage: ProofStage::Archived, - }))) - .is_err() - { - self.fail(&scope); - return; - } - } - } - match cell.core.finish(identity) { - Ok(permit) => { - let token = u64::try_from(crate::asyncrt::wall_ms()) - .ok() - .and_then(|now| { - self.versions - .issue(permit.version(), cell.schema_digest, now) - .ok() - }); - let Some(version_token) = token else { - let id = permit.command(); - self.finish_pending( - &scope, - Err(CommandFailure::Ambiguous { command: id }), - ); - return; - }; - let Some(bytes) = self.budget.acquire(receipt.receipt().result().len()) - else { - let id = permit.command(); - self.finish_pending( - &scope, - Err(CommandFailure::Ambiguous { command: id }), - ); - return; - }; - let response = client::CommandReceipt { - id: permit.command(), - version: permit.version().clone(), - durability: permit.durability(), - result: receipt.receipt().result().to_vec(), - version_token, - }; - self.finish_pending_charged( - &scope, - Ok(Response::Command(response)), - Some(bytes), - ); - } - Err(Refusal::Unproven) => pending.phase = Phase::AwaitFinish, - Err(_) => self.refuse(&scope, CommandFailure::Unavailable), - } - } - StatusPurpose::Poll => { - if let Some(identity) = pending.identity.clone() { - if identity.authority.term < status.term { - if let Ok(barrier) = cell.core.begin_retirement(&identity) { - pending.phase = Phase::Retiring; - self.sql( - &scope, - SqlPurpose::Retire, - Action::Retire { identity, barrier }, - ); - } - return; - } - } - if pending.result.is_some() { - pending.phase = Phase::Finish; - } - } - StatusPurpose::Observe(_) => (), - } - } - - fn on_sql( - &mut self, - scope: GroupScope, - purpose: SqlPurpose, - result: Result, - work: Option>, - ) { - let result = match result { - Ok(sql::Output { - value: - Value::SnapshotCapture { - capture, - reservation, - }, - _bytes, - }) => { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.sql_busy = false; - if !matches!( - &purpose, - SqlPurpose::Seed(_) | SqlPurpose::Checkpoint | SqlPurpose::SnapshotSource(_) - ) { - self.fail(&scope); - return; - } - self.encode_snapshot(scope, purpose, capture, reservation, work, _bytes); - return; - } - result => result, - }; - if let SqlPurpose::EffectScan(sequence) = purpose { - self.finish_effect_scan(scope, sequence, result); - return; - } - if let SqlPurpose::Seed(id) = purpose { - self.seed_sql(scope, id, result); - return; - } - if matches!( - &purpose, - SqlPurpose::Checkpoint - | SqlPurpose::SnapshotSource(_) - | SqlPurpose::SnapshotPrepare(_) - | SqlPurpose::SnapshotPublish(_) - ) { - self.on_snapshot_sql(scope, purpose, result); - return; - } - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - cell.sql_busy = false; - // A scan while this cell was busy cannot predict when SQL finishes. - // Wake archive selection for newly idle retained work instead of - // waiting a maintenance interval and missing the collection window. - if !matches!(purpose, SqlPurpose::CaptureArchive) - && cell - .group - .applied() - .is_ok_and(|applied| applied.position.index > cell.archived) - { - if let Some(owner) = self.archive_owner.as_mut() { - if !owner.inflight.contains(&scope) { - owner.scan_at = 0; - } - } - } - if let SqlPurpose::CaptureArchive = purpose { - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - if self.closing.load(Ordering::Acquire) { - owner.inflight.remove(&scope); - return; - } - match result { - Ok(sql::Output { - value: Value::ArchiveCut(Some(cut)), - _bytes, - }) => { - if owner.pending.is_empty() { - owner.publish_at = crate::asyncrt::mono_ms() - .saturating_add(super::archive::bundle::COALESCE_MS); - } - owner.pending.push(super::host_archive::Captured { - scope, - archive: cell.archive.clone(), - cut: *cut, - reservation: _bytes, - }); - } - _ => { - owner.inflight.remove(&scope); - owner.scan_at = 0; - cell.archive_retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.archive.retry_ms); - } - } - return; - } - if let SqlPurpose::Query(id) = purpose { - let Some(mut read) = self.reads.remove(&id) else { - return; - }; - if read.expired() { - read.request.control.cancel(); - reply(&mut read.request, Err(CommandFailure::VersionUnavailable)); - return; - } - if matches!( - &result, - Err(ExecutionError::Refused(CommandFailure::VersionUnavailable)) - ) && !self.closing.load(Ordering::Acquire) - { - read.phase = ReadPhase::Image; - read.retry_at = crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - self.reads.insert(id, read); - return; - } - let mut bytes = None; - let result = match result { - Ok(sql::Output { - value: Value::Queried(value), - _bytes, - }) => { - bytes = _bytes; - if read.minimum.as_ref().is_some_and(|minimum| { - minimum.satisfied(&value.version, &scope) == Ok(true) - }) { - u64::try_from(crate::asyncrt::wall_ms()) - .ok() - .and_then(|now| { - self.versions - .issue(&value.version, cell.schema_digest, now) - .ok() - }) - .map(|version_token| { - Response::Query(client::QueryResult { - result: value.result, - version: value.version, - version_token, - served_by: cell.group.identity().member, - }) - }) - .ok_or(CommandFailure::VersionUnavailable) - } else { - Err(CommandFailure::CorruptState) - } - } - Err(error) => Err(failure(error)), - _ => Err(CommandFailure::CorruptState), - }; - if !self.closing.load(Ordering::Acquire) { - reply_charged(&mut read.request, result, bytes); - } - return; - } - let sql::Output { value, _bytes } = match result { - Ok(result) => result, - Err(error) => { - tracing::debug!(event = "raft_sql_failure", cell = %scope.cell, - operation = match &purpose { SqlPurpose::Stage => "stage", SqlPurpose::Prepare(_) => "apply", SqlPurpose::Publish(_) => "publish", _ => "lifecycle" }, - error = ?error); - if let ( - SqlPurpose::Prepare(event), - ExecutionError::Refused(CommandFailure::Overloaded), - ) = (&purpose, &error) - { - let Some(retry) = event.clone_apply() else { - self.fail(&scope); - return; - }; - cell.apply = Some(retry); - cell.apply_retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - self.applying.insert(scope); - return; - } - if matches!( - purpose, - SqlPurpose::Discard | SqlPurpose::Prepare(_) | SqlPurpose::Publish(_) - ) { - tracing::error!(group = ?scope.group, ?error, "Raft image operation failed"); - self.fail(&scope); - } else { - self.refuse(&scope, failure(error)); - } - return; - } - }; - match (purpose, value) { - (SqlPurpose::Stage, Value::Staged(outcome)) => { - self.admission_at = 0; - if let Some(pending) = cell.pending.as_mut() { - pending._bytes = _bytes; - match outcome { - StageOutcome::Prepared(prepared) => { - pending.archive_bytes = Some(prepared.archive_bytes()); - pending.prepared = Some(prepared); - pending.phase = Phase::BeforeSubmit; - } - StageOutcome::Duplicate(receipt) => { - pending.request.control.observe_committed(); - pending.result = Some(receipt); - pending.duplicate = true; - pending.phase = Phase::Finish; - } - } - } - } - (SqlPurpose::Mark, Value::Submitted) => { - let Some(pending) = cell.pending.as_mut() else { - return; - }; - let (Some(prepared), Some(identity)) = - (pending.prepared.take(), pending.identity.clone()) - else { - self.fail(&scope); - return; - }; - pending.phase = Phase::Proposing; - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let result = runtime - .propose(&scope, prepared.context(), prepared.bytes().to_vec()) - .await; - Job::Proposed { - scope, - identity, - result, - } - })); - } - (SqlPurpose::Discard, Value::Discarded) => { - if let Some(identity) = cell.pending.as_ref().and_then(|p| p.identity.as_ref()) { - if cell.core.discard_completed(identity).is_err() { - self.fail(&scope); - return; - } - } - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - (SqlPurpose::Retire, Value::Retired(barrier)) => { - if let Some(identity) = cell - .pending - .as_ref() - .and_then(|pending| pending.identity.as_ref()) - { - if cell.core.retirement_completed(identity, barrier).is_err() { - self.fail(&scope); - return; - } - } - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - (SqlPurpose::Prepare(_), Value::Prepared(completion)) => { - cell.apply_ticket = Some(completion.ticket.clone()); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - Job::Installed { - scope, - result: runtime.complete_apply(completion).await, - } - })); - } - (SqlPurpose::Publish(ticket), Value::Published { view, receipt }) => { - cell.image = Some(view); - // The runtime can deliver the next committed entry while this - // image is being published. Publication consumed the scheduler - // slot, so restore it for that queued entry after SQL joins. - cell.apply_inflight = cell.apply.is_some(); - if cell.apply_inflight { - self.applying.insert(scope.clone()); - } - cell.apply_ticket = None; - // New durable work invalidates an earlier empty inventory. - // Keep per-cell failure backoff and the single producer; only - // idle scanning should wait for the maintenance interval. - if let Some(owner) = &mut self.archive_owner { - owner.scan_at = 0; - } - if let Ok(Some(progress)) = - progress(&cell.group, cell.image.as_ref(), cell.archived) - { - let _ = cell.core.observe_progress(progress); - } - if let Some((identity, receipt)) = receipt { - if cell.pending.as_ref().and_then(|p| p.identity.as_ref()) == Some(&identity) { - let proven = cell - .core - .bind_entry(&identity, ticket.position(), ticket.entry_digest()) - .and_then(|()| { - for stage in [ - ProofStage::Committed, - ProofStage::Applied, - ProofStage::Queryable, - ] { - cell.core.observe_proof(&OutputProof::Raft(Box::new( - RaftProof { - command: identity.clone(), - entry_digest: ticket.entry_digest(), - position: ticket.position(), - stage, - }, - )))?; - } - Ok(()) - }); - if proven.is_ok() { - if let Some(pending) = cell.pending.as_mut() { - pending.request.control.observe_committed(); - pending.proof = Some((ticket.position(), ticket.entry_digest())); - pending.result = Some(receipt); - pending.phase = Phase::Finish; - } - } else { - self.refuse(&scope, CommandFailure::Unavailable); - } - } - } - } - _ => self.fail(&scope), - } - } - - fn on_event(&mut self, event: RuntimeEvent) { - match &event.kind { - RuntimeEventKind::Message { replica, message } => { - if message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot { - self.send_snapshot(&replica.scope, message.clone()); - return; - } - match self.transport.submit(&replica.scope, message.clone()) { - Ok(delivery) => { - let scope = replica.scope.clone(); - let member = message.to; - let permit = event._permit.clone(); - self.jobs.push(Box::pin(async move { - let result = delivery.accepted().await; - // Keep the runtime's send boundary owned until the - // transport accepts or refuses this exact message. - drop(permit); - Job::MessageDelivery { - scope, - member, - result, - } - })); - } - Err(_) => { - let runtime = self.runtime.clone(); - let scope = replica.scope.clone(); - let member = message.to; - let snapshot = - message.get_msg_type() == ::raft::prelude::MessageType::MsgSnapshot; - self.jobs.push(Box::pin(async move { - let _ = runtime.admit_unreachable(&scope, member).await; - if snapshot { - let _ = runtime.admit_snapshot_status(&scope, member, false).await; - } - Job::Hint - })); - } - } - } - RuntimeEventKind::Apply { ticket, .. } => { - let scope = ticket.replica().scope.clone(); - if let Some(cell) = self.cells.get_mut(&scope) { - if cell.apply.is_some() { - self.fail(&scope); - } else { - cell.archive_observation.observe( - ticket.position().index, - cell.archived, - crate::asyncrt::mono_ms(), - ); - cell.apply_inflight = true; - cell.apply = Some(event); - self.applying.insert(scope); - } - } - } - RuntimeEventKind::Applied { ticket, .. } => { - let scope = ticket.replica().scope.clone(); - if let Some(cell) = self.cells.get_mut(&scope) { - let protocol = cell.group.read().ok().is_some_and(|cache| { - cache - .entries - .get(&ticket.position().index.get()) - .is_some_and(|entry| { - (entry.get_entry_type() != raft::prelude::EntryType::EntryNormal - || entry.data.is_empty()) - && entry.index == ticket.position().index.get() - && entry.term == ticket.position().term.get() - && super::completion::entry_digest(entry).ok() - == Some(ticket.entry_digest()) - }) - }); - if protocol && ticket.replica() == cell.group.identity() { - if cell.image.is_some() { - if let Some(owner) = &mut self.archive_owner { - owner.scan_at = 0; - } - } - } else if cell.apply_ticket.as_ref() == Some(ticket) { - cell.publication = Some(ticket.clone()); - self.applying.insert(scope); - } else { - self.fail(&scope); - } - } - } - RuntimeEventKind::Failed { replica, error } => { - tracing::error!(group = ?replica.scope.group, ?error, "Raft runtime failed"); - self.fail(&replica.scope); - } - RuntimeEventKind::Snapshot { - ticket, - snapshot, - source, - } => self.receive_snapshot(ticket.clone(), snapshot.clone(), *source), - RuntimeEventKind::SnapshotInstalled { ticket, .. } => self.snapshot_installed(ticket), - RuntimeEventKind::SnapshotAborted { ticket, .. } => self.snapshot_aborted(ticket), - RuntimeEventKind::Checkpointed { ticket } => { - if let Some(cell) = self.cells.get_mut(&ticket.replica().scope) { - cell.checkpoint_busy = false; - cell.archive_checkpoint_at = crate::asyncrt::mono_ms() - .saturating_add(self.options.archive_checkpoint_ms); - } - } - RuntimeEventKind::ReadIndex { - replica, - index, - context, - } => { - let Ok(id) = ::try_from(context.as_slice()) else { - return; - }; - let Some(read) = self.reads.get_mut(&id) else { - return; - }; - if read.request.scope == replica.scope - && self - .cells - .get(&replica.scope) - .is_some_and(|cell| replica == cell.group.identity()) - && matches!( - read.phase, - ReadPhase::StartBarrier | ReadPhase::CheckingLeader | ReadPhase::Barrier - ) - { - if index.get() == 0 { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)); - } else { - read.minimum = Some(Minimum::ReadIndex(*index)); - read.phase = ReadPhase::Image; - } - } - } - _ => (), - } - } - - fn pump(&mut self) { - self.pump_directory(); - self.pump_admission(); - // Inventory scans are node maintenance, not per-frame work. Repeating - // them for every heartbeat makes N groups cost O(N²) per tick. - let now = crate::asyncrt::mono_ms(); - if now >= self.maintenance_at { - self.maintenance_at = now.saturating_add(self.options.poll_ms); - self.pump_extent_gc(); - self.pump_membership(); - self.pump_snapshots(); - } - for scope in self.applying.clone() { - let Some(cell) = self.cells.get_mut(&scope) else { - continue; - }; - if cell.failed { - self.applying.remove(&scope); - continue; - } - if cell - .incoming - .as_ref() - .is_some_and(|incoming| !incoming.allows_apply()) - { - continue; - } - if cell.sql_busy || cell.apply_retry_at > crate::asyncrt::mono_ms() { - continue; - } - if let Some(ticket) = cell.publication.take() { - self.sql( - &scope, - SqlPurpose::Publish(ticket.clone()), - Action::Publish { ticket }, - ); - self.applying.remove(&scope); - } else if let Some(event) = cell.apply.take() { - let mut bytes = [0; 16]; - self.rng.fill_bytes(&mut bytes); - let Ok(image) = ImageId::new(bytes) else { - self.fail(&scope); - continue; - }; - let Some(retry) = event.clone_apply() else { - self.fail(&scope); - continue; - }; - self.sql( - &scope, - SqlPurpose::Prepare(retry), - Action::Prepare { event, image }, - ); - self.applying.remove(&scope); - } - } - for scope in self.active.clone() { - let Some(cell) = self.cells.get_mut(&scope) else { - continue; - }; - if cell.failed || cell.sql_busy || cell.status_busy || cell.incoming.is_some() { - continue; - } - let Some(pending) = cell.pending.as_mut() else { - self.active.remove(&scope); - continue; - }; - if pending.request.control.cancelled() && !pending.request.control.submitted() { - pending.phase = Phase::Discard; - } - match pending.phase { - Phase::Start => self.status(&scope, StatusPurpose::Start), - Phase::BeforeSubmit => self.status(&scope, StatusPurpose::Submit), - Phase::Finish => self.status(&scope, StatusPurpose::Finish), - Phase::Discard => { - if let Some(identity) = pending.identity.clone() { - self.sql(&scope, SqlPurpose::Discard, Action::Discard { identity }); - } else { - self.finish_pending(&scope, Err(CommandFailure::Unavailable)); - } - } - _ => (), - } - } - self.pump_archive(); - self.pump_reads(); - self.pump_leader_load(); - self.pump_controller(); - self.publish_controller_resources(); - self.pump_effects(); - self.pump_node_drain(); - } - - fn pump_archive(&mut self) { - if self.closing.load(Ordering::Acquire) { - return; - } - let now = crate::asyncrt::mono_ms(); - // Already captured cuts must be published even when every slot is - // occupied. A partial batch's deadline cannot be renewed by slow SQL. - let publish = self.archive_owner.as_ref().is_some_and(|owner| { - !owner.pending.is_empty() - && (owner.pending.len() - >= self - .options - .archive - .max_batch_groups - .min(self.options.archive.max_producers) - || now >= owner.publish_at) - }); - if publish { - self.publish_archive_batch(); - } - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - if owner.inflight.len() >= self.options.archive.max_producers { - return; - } - if now < owner.scan_at { - return; - } - let eligible = |cell: &Cell| { - !cell.failed - && !cell.sql_busy - && cell.archive_retry_at <= now - && cell - .group - .applied() - .is_ok_and(|applied| applied.position.index > cell.archived) - }; - let mut captures = Vec::new(); - while owner.inflight.len() < self.options.archive.max_producers { - let next = super::host_archive::next_scope( - &self.cells, - owner.cursor.as_ref(), - |scope, cell| !owner.inflight.contains(scope) && eligible(cell), - ) - .cloned(); - let Some(scope) = next else { - owner.scan_at = now.saturating_add(self.options.poll_ms); - break; - }; - owner.inflight.insert(scope.clone()); - // Advance once per selected cell, independently of how completed - // captures are partitioned into compatible upload batches. - owner.cursor = Some(scope.clone()); - captures.push(scope); - } - let limits = self.options.archive.limits; - for scope in captures { - self.sql( - &scope, - SqlPurpose::CaptureArchive, - Action::CaptureArchive { limits }, - ); - } - } - - fn publish_archive_batch(&mut self) { - let Some(owner) = self.archive_owner.as_mut() else { - return; - }; - while !owner.pending.is_empty() { - let mut pending = vec![owner.pending.remove(0)]; - while pending.len() < self.options.archive.max_batch_groups { - let next = owner - .pending - .iter() - .position(|candidate| pending[0].archive.can_batch_with(&candidate.archive)); - let Some(next) = next else { - break; - }; - pending.push(owner.pending.remove(next)); - } - // Each cut has one bounded completion channel. The physical batch - // remains in jobs through joined completion, while a finished cell - // can release its own slot without waiting for another history. - let mut senders = Vec::with_capacity(pending.len()); - for capture in &pending { - let scope = capture.scope.clone(); - let position = capture.cut.position(); - let (send, receive) = tokio::sync::oneshot::channel(); - senders.push(Some(send)); - self.jobs.push(Box::pin(async move { - let completed = receive.await.unwrap_or(super::host_archive::Completed { - scope, - position, - result: Err(super::archive::ArchiveError::Unconfirmed), - }); - Job::Archived(vec![completed]) - })); - } - let verifier = owner.verifier.clone(); - let physical = owner.physical.clone(); - let prepared = self.transport.prepared_snapshot_cache(); - let timer = self - .local - .performance - .timer(super::performance::Operation::ArchivePublicationBatch); - self.jobs.push(Box::pin(async move { - let succeeded = super::host_archive::publish( - pending, - verifier, - prepared, - physical, - move |number, completed| { - if let Some(sender) = senders.get_mut(number).and_then(Option::take) { - let _ = sender.send(completed); - } - }, - ) - .await; - timer.finish(succeeded); - Job::ArchiveBatchJoined - })); - } - } - - fn on_job(&mut self, job: Job) { - match job { - Job::MessageDelivery { - scope, - member, - result, - } => { - if let Err(error) = result { - tracing::debug!(?error, "Raft message was not accepted"); - let runtime = self.runtime.clone(); - self.jobs.push(Box::pin(async move { - let _ = runtime.admit_unreachable(&scope, member).await; - Job::Hint - })); - } - } - Job::FleetMovementServiceStopped | Job::FleetRestartServiceStopped => (), - Job::FleetAction(completed) => self.finish_fleet_action(*completed), - Job::PeerFleetStop(completed) => self.finish_peer_fleet_stop(*completed), - Job::FleetAdmission(completed) => self.finish_fleet_admission(*completed), - Job::HistoryRegistered(completed) => self.finish_history_registration(*completed), - Job::ExtentGc(result) => self.extent_gc_completed(result), - Job::Drained(completed) => self.finish_node_drain(*completed), - Job::Controller(completed) => self.finish_controller(*completed), - Job::Effect(completed) => self.finish_effect(*completed), - Job::LeaderLoad(completed) => self.finish_leader_load(*completed), - Job::Inspection(completed) => self.finish_inspection(*completed), - Job::PeerInspection(completed) => self.finish_peer_inspection(*completed), - Job::NodeInspection(completed) => self.finish_node_inspection(*completed), - Job::PeerFleetInspection(completed) => self.finish_peer_fleet_inspection(*completed), - Job::LocalFleetInspection(completed) => self.finish_local_fleet_inspection(*completed), - Job::Directory(observation) => self.on_directory(*observation), - Job::Standby(completed) => self.on_standby_job(*completed), - Job::Leadership { - mut request, - result, - } => { - let result = if request.control.cancelled() || self.closing.load(Ordering::Acquire) - { - Err(CommandFailure::Unavailable) - } else { - result.map_err(|error| match error { - RuntimeError::NotLeader => CommandFailure::NotLeader { leader: None }, - RuntimeError::Overloaded | RuntimeError::NotReady => { - CommandFailure::Overloaded - } - _ => CommandFailure::Unavailable, - }) - }; - reply(&mut request, result); - } - Job::SeedArchive { id, result } => self.seed_archive(id, *result), - Job::Configured { scope, result } => self.configured(scope, result), - job @ (Job::SnapshotSent { .. } - | Job::SnapshotFetched { .. } - | Job::SnapshotAuthorized { .. } - | Job::SnapshotAbort { .. } - | Job::Checkpoint { .. }) => self.on_snapshot_job(job), - Job::ArchiveBatchJoined => (), - Job::Archived(completed) => { - self.admission_at = 0; - if let Some(owner) = self.archive_owner.as_mut() { - owner.scan_at = 0; - for result in &completed { - owner.inflight.remove(&result.scope); - } - } - for super::host_archive::Completed { - scope, - position, - result, - } in completed - { - use super::performance::Counter; - if let Err(error) = &result { - // Expected publication contention is common. Keep the - // typed cause available for diagnosis without warning - // on each advisory deferral or changing retry policy. - tracing::debug!( - event = "raft_archive_publication_failure", - node = ?self.log.limits().node, - group = ?scope.group, - index = position.index.get(), - error = ?error, - "Raft archive publication did not confirm the cut" - ); - } - self.local.performance.increment( - if result.is_ok() { - Counter::ArchivePublicationCutSucceeded - } else { - Counter::ArchivePublicationCutFailed - }, - 1, - ); - let Some(cell) = self.cells.get_mut(&scope) else { - if result.is_ok() { - self.local - .performance - .increment(Counter::ArchiveCoverageMissingCell, 1); - } - continue; - }; - if result.is_ok() && position.index > cell.archived { - // The proof names the captured local cut, even if another - // replica has already archived a later cut of this history. - // Never import that replica's later applied progress. - match progress(&cell.group, cell.image.as_ref(), position.index) { - Ok(Some(progress)) => { - if cell.core.observe_progress(progress).is_ok() { - self.local.performance.increment( - Counter::ArchiveCoverageIndexAdvance, - position.index.get() - cell.archived.get(), - ); - self.local - .performance - .increment(Counter::ArchiveCoverageAdvanced, 1); - cell.archived = position.index; - // The confirmed cut resolves this wait. Re-run - // the normal fresh authority/output checks now; - // it is not itself permission to send a result. - if let Some(pending) = cell.pending.as_mut() { - if pending.phase == Phase::AwaitFinish - && pending.result.as_ref().is_some_and(|receipt| { - receipt.position().index <= cell.archived - }) - { - pending.phase = Phase::Finish; - } - } - } else { - self.local - .performance - .increment(Counter::ArchiveCoverageRejected, 1); - } - } - Ok(None) => self - .local - .performance - .increment(Counter::ArchiveCoverageImagePending, 1), - Err(_) => self - .local - .performance - .increment(Counter::ArchiveCoverageRejected, 1), - } - } else if result.is_ok() { - self.local - .performance - .increment(Counter::ArchiveCoverageUnchanged, 1); - } - // A confirmed cut may already have newer work waiting. Keep it - // eligible for the next fair scan; back off failures or a proof - // that the local state machine could not accept. - let now = crate::asyncrt::mono_ms(); - if let Err(error) = &result { - cell.archive_observation.failed(error); - } - cell.archive_retry_at = if result.is_ok() && position.index <= cell.archived { - cell.archive_observation.confirmed(cell.archived, now); - now - } else { - now.saturating_add(self.options.archive.retry_ms) - }; - } - } - Job::Routed { - mut request, - result, - } => { - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - reply(&mut request, Err(CommandFailure::VersionUnavailable)); - } else { - match result { - Ok(output) => reply_charged( - &mut request, - Ok(Response::Query(output.result)), - Some(output.bytes), - ), - Err(error) => reply(&mut request, Err(error)), - } - } - } - Job::ReadSource { id, result } => self.on_read_source(id, result), - Job::ReadLeader { id, check, result } => self.on_read_leader(id, check, result), - Job::ReadStarted { - id, - attempt, - result, - } => { - if let Some(read) = self.reads.get_mut(&id) { - // A barrier event can arrive before the RPC acknowledgement. - // That late reply cannot regress an executing read. - if read.phase == ReadPhase::Barrier && read.barrier_attempt == attempt { - match result { - Ok(()) => (), - Err(RuntimeError::NotReady | RuntimeError::Overloaded) => { - read.phase = ReadPhase::StartBarrier; - read.checking_leader = false; - read.retry_at = - crate::asyncrt::mono_ms().saturating_add(self.options.poll_ms); - } - Err(_) => { - self.finish_read(&id, Err(CommandFailure::VersionUnavailable)) - } - } - } - } - } - Job::Status { - scope, - sequence, - purpose, - result, - } => self.on_status(scope, sequence, purpose, result), - Job::Sql { - scope, - purpose, - result, - work, - } => self.on_sql(scope, purpose, *result, work), - Job::SnapshotEncoded { - scope, - purpose, - result, - } => match purpose { - SqlPurpose::Seed(id) => self.seed_encoded(scope, id, *result), - SqlPurpose::Checkpoint | SqlPurpose::SnapshotSource(_) => { - self.on_snapshot_encoded(scope, purpose, *result) - } - _ => self.fail(&scope), - }, - Job::Installed { scope, result } => { - if let Err(error) = result { - tracing::error!(group = ?scope.group, ?error, "Raft durable apply completion failed"); - self.fail(&scope); - } - } - Job::Hint => (), - Job::Proposed { - scope, - identity, - result, - } => { - let Some(cell) = self.cells.get_mut(&scope) else { - return; - }; - if cell.pending.as_ref().and_then(|p| p.identity.as_ref()) != Some(&identity) { - return; - } - // Publication or a newer-term retirement may finish before this - // RPC reply. It cannot put that producer back into Waiting. - let waiting = cell - .pending - .as_ref() - .is_some_and(|p| p.phase == Phase::Proposing); - if waiting { - if let Some(pending) = cell.pending.as_mut() { - pending.phase = Phase::Waiting; - } - } - match result { - Ok(accepted) => { - if accepted.replica != identity.authority.replica { - self.fail(&scope); - } else { - // A retired gate may refuse this late observation. - // Only the verified committed receipt can release it. - let _ = cell.core.bind_entry( - &identity, - accepted.position, - accepted.entry_digest, - ); - } - } - Err(_) if waiting => self.refuse(&scope, CommandFailure::Unavailable), - Err(_) => (), - } - } - } - } - - async fn run( - &mut self, - mut requests: mailbox::Receiver, - mut controls: mailbox::Receiver, - mut closes: mpsc::UnboundedReceiver<()>, - ) { - let mut requests_open = true; - let mut controls_open = true; - let mut closes_open = true; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let mut poll_at = now.saturating_add(self.options.poll_ms); - loop { - if self.closing.load(Ordering::Acquire) { - return; - } - // Continue an admitted command before another administrative - // observer can occupy its serialized authority-status check. - self.pump(); - // Give admitted control work a bounded turn even while business - // traffic is continuously ready. The main select still services - // business, completions and consensus between these small batches. - for _ in 0..4 { - match controls.try_recv() { - Ok(request) => self.on_control(request), - Err(mpsc::error::TryRecvError::Empty) => break, - Err(mpsc::error::TryRecvError::Disconnected) => { - controls_open = false; - break; - } - } - } - self.pump(); - let wake_at = self - .archive_owner - .as_ref() - .filter(|owner| !owner.pending.is_empty()) - .map_or(poll_at, |owner| poll_at.min(owner.publish_at)); - let Ok(timer) = self.log.domain().sleep_until(wake_at) else { - return; - }; - crate::asyncrt::select! { - request=controls.recv(),if controls_open=>match request{Some(request)=>self.on_control(request),None=>controls_open=false}, - request=requests.recv(),if requests_open=>match request{Some(request)=>self.on_request(request),None=>requests_open=false}, - close=closes.recv(),if closes_open=>if close.is_some(){return;}else{closes_open=false;}, - event=self.events.recv()=>match event{Some(event)=>self.on_event(event),None=>return}, - job=self.jobs.next(),if !self.jobs.is_empty()=>if let Some(job)=job{self.on_job(job)}, - _=timer=>{let Ok(now)=self.log.domain().mono_ms() else{return;}; - if now < poll_at { continue; } - poll_at=now.saturating_add(self.options.poll_ms); - for scope in self.active.clone(){ - let purpose=self.cells.get(&scope).filter(|cell|!cell.status_busy).and_then(|cell|cell.pending.as_ref()).and_then(|pending|match pending.phase { - Phase::AwaitStart=>Some(StatusPurpose::Start),Phase::AwaitSubmit=>Some(StatusPurpose::Submit),Phase::AwaitFinish=>Some(StatusPurpose::Finish),Phase::Waiting=>Some(StatusPurpose::Poll),_=>None, - }); - if let Some(purpose)=purpose {self.status(&scope,purpose);} - }}, - } - } - } - - async fn shutdown(&mut self) -> Result { - self.closing.store(true, Ordering::Release); - RaftCells { - handle: self.local.clone(), - } - .stop_fleet_movement_service(); - RaftCells { - handle: self.local.clone(), - } - .stop_fleet_restart_service(); - if let Some(control) = self.history_registration.take() { - control.cancel(); - } - self.snapshot_source.take(); - if let Some(controller) = &self.controller { - controller.stop(); - } - self.close_configurations(); - if let Some(owner) = &mut self.archive_owner { - let _ = owner.cancellation.send(true); - owner.pending.clear(); - } - for (_, mut read) in std::mem::take(&mut self.reads) { - read.request.control.cancel(); - reply(&mut read.request, Err(CommandFailure::Unavailable)); - } - let mut unfinished = self - .uncertain - .iter() - .map(|(scope, id)| client::UnfinishedCommand { - scope: scope.clone(), - id: *id, - }) - .collect::>(); - for (scope, cell) in &mut self.cells { - cell.core.close(); - if let Some(pending) = cell.pending.as_mut() { - let submitted = pending.request.control.submitted(); - if submitted { - unfinished.push(client::UnfinishedCommand { - scope: scope.clone(), - id: pending.options.id, - }); - } - let error = if submitted { - CommandFailure::Ambiguous { - command: pending.options.id, - } - } else { - CommandFailure::Unavailable - }; - reply(&mut pending.request, Err(error)); - } - } - self.pool.stop(); - let runtime = self.runtime.close().await; - self.transport.close().await; - while self.jobs.next().await.is_some() {} - let sql = self.pool.join().await; - self.archive_owner.take(); - let node = NodeIdentity { - node: self.log.limits().node, - disk: self.log.limits().disk, - process: self.log.limits().process, - }; - let instance = self.log.storage_instance(); - let observations = (|| -> Result<_, CommandFailure> { - let mut groups = Vec::new(); - let mut groups_with_archive_lag = 0; - let mut bytes = 0usize; - for (scope, cell) in &self.cells { - let cache = cell - .group - .read() - .map_err(|_| CommandFailure::CorruptState)?; - let committed = LogIndex::new(cache.hard_state.commit); - if cell.archived < committed { - groups_with_archive_lag += 1; - } - let charge = scope - .cell - .len() - .checked_add(2048) - .ok_or(CommandFailure::CorruptState)?; - if groups.len() < self.options.close_report.max_groups - && bytes - .checked_add(charge) - .is_some_and(|n| n <= self.options.close_report.max_bytes) - { - bytes += charge; - groups.push(super::operations::RaftGroupCloseReport { - scope: scope.clone(), - member: cell.group.identity().member, - committed, - applied: cache.applied.position, - archived: cell.archived, - retired: cell.retired, - failed: cell.failed, - }); - } - } - Ok((groups, groups_with_archive_lag)) - })(); - let groups_total = self - .log - .group_count() - .map_err(|_| CommandFailure::CorruptState); - let attached = self.cells.len(); - self.cells.clear(); - self.standbys.clear(); - let log = self.log.close().await; - let closed_extents = if let Some(extents) = self.extents.take() { - match self.log.domain().blocking(move || drop(extents)) { - Ok(job) => job.await.is_ok(), - Err(_) => false, - } - } else { - true - }; - if runtime.is_err() || sql.is_err() || log.is_err() || !closed_extents { - return Err(CommandFailure::CorruptState); - } - if self.local.admission.is_poisoned() { - return Err(CommandFailure::Unavailable); - } - let (groups, groups_with_archive_lag) = observations?; - let groups_total = groups_total?; - let groups_with_unknown_archive = groups_total - .checked_sub(attached) - .ok_or(CommandFailure::CorruptState)?; - let omitted_groups = groups_total - groups.len(); - Ok(client::RaftCloseReport { - node, - storage_instance: instance.map_err(|_| CommandFailure::CorruptState)?, - groups, - groups_total, - omitted_groups, - groups_with_archive_lag, - groups_with_unknown_archive, - unfinished, - }) - } -} diff --git a/crates/cellar/durability/raft/host_admission.rs b/crates/cellar/durability/raft/host_admission.rs index 3771b2027..95b09e9bd 100644 --- a/crates/cellar/durability/raft/host_admission.rs +++ b/crates/cellar/durability/raft/host_admission.rs @@ -1,10 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Retained history refuses new work; temporary reservations defer callbacks. -use super::{Driver, Phase}; use crate::raft::{ cache::{CachedGroup, GroupLog}, - performance::{Counter as MeasuredCounter, Operation as MeasuredOperation}, storage::LogError, }; use cellar_logic::consensus::LogIndex; @@ -80,127 +78,3 @@ impl Retained { Ok(usage) } } - -#[derive(Clone, Copy, PartialEq, Eq)] -pub(super) enum Decision { - Granted, - Deferred, - Refused, -} - -impl Driver { - pub(super) fn archive_admission(&mut self) -> Decision { - let timer = self - .local - .performance - .timer(MeasuredOperation::ArchiveAdmissionCheck); - let decision = self.check_archive_admission(); - // This measures completed policy evaluation, including refusal and - // deferral. Decision counters retain the reason; success is not a grant. - timer.finish(true); - decision - } - - fn check_archive_admission(&mut self) -> Decision { - let reservation = self - .options - .execution - .max_image_bytes - .saturating_add(self.options.execution.envelope.max_payload_bytes as u64) - .saturating_add(self.options.execution.envelope.max_header_bytes as u64) - .saturating_add(1024); - let mut bytes = reservation; - let mut entries = 1u64; - let mut reserved_bytes = 0u64; - let mut reserved_entries = 0u64; - for cell in self.cells.values_mut() { - let Ok(usage) = cell.archive_retained.observe(&cell.group, cell.archived) else { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionObservationRefused, 1); - return Decision::Refused; - }; - bytes = bytes.saturating_add(usage.bytes); - entries = entries.saturating_add(usage.entries); - if let Some(pending) = cell.pending.as_ref().filter(|pending| pending.allow_new) { - reserved_bytes = - reserved_bytes.saturating_add(pending.archive_bytes.unwrap_or(reservation)); - reserved_entries = reserved_entries.saturating_add(1); - } - // Record the first encountered limit. These count decisions, - // including waiter rechecks, rather than terminal client outcomes. - if bytes > self.options.archive.max_unarchived_bytes { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionRetainedBytesRefused, 1); - return Decision::Refused; - } - if entries > self.options.archive.max_unarchived_entries as u64 { - self.local - .performance - .increment(MeasuredCounter::ArchiveAdmissionRetainedEntriesRefused, 1); - return Decision::Refused; - } - } - let (decision, counter) = - if bytes.saturating_add(reserved_bytes) > self.options.archive.max_unarchived_bytes { - ( - Decision::Deferred, - MeasuredCounter::ArchiveAdmissionReservedBytesDeferred, - ) - } else if entries.saturating_add(reserved_entries) - > self.options.archive.max_unarchived_entries as u64 - { - ( - Decision::Deferred, - MeasuredCounter::ArchiveAdmissionReservedEntriesDeferred, - ) - } else { - (Decision::Granted, MeasuredCounter::ArchiveAdmissionGranted) - }; - self.local.performance.increment(counter, 1); - decision - } - - pub(super) fn pump_admission(&mut self) { - let now = crate::asyncrt::mono_ms(); - if self.admission_waiters.is_empty() || now < self.admission_at { - return; - } - self.admission_at = now.saturating_add(self.options.poll_ms); - // These waiters retain the existing bounded request/input permits. - // They acquire no execution reservation and run no callback until - // granted. FIFO order and a bounded turn avoid starving other work. - for _ in 0..4 { - let Some(scope) = self.admission_waiters.pop_front() else { - break; - }; - let Some(pending) = self - .cells - .get(&scope) - .and_then(|cell| cell.pending.as_ref()) - else { - continue; - }; - if pending.phase != Phase::AwaitAdmission || pending.request.control.cancelled() { - continue; - } - let decision = self.archive_admission(); - if decision == Decision::Deferred { - self.admission_waiters.push_front(scope); - break; - } - if let Some(pending) = self - .cells - .get_mut(&scope) - .and_then(|cell| cell.pending.as_mut()) - { - pending.allow_new = decision == Decision::Granted; - if let Some(timer) = pending.admission_wait.take() { - timer.finish(pending.allow_new); - } - pending.phase = Phase::Start; - } - } - } -} diff --git a/crates/cellar/durability/raft/host_admission_tests.rs b/crates/cellar/durability/raft/host_admission_tests.rs index f21d53fda..1f73d1d5d 100644 --- a/crates/cellar/durability/raft/host_admission_tests.rs +++ b/crates/cellar/durability/raft/host_admission_tests.rs @@ -392,7 +392,28 @@ async fn cached_accounting_refuses_a_poisoned_group_cache() -> anyhow::Result<() .await? ); let result = retained.observe(&group, LogIndex::new(0)); - log.close().await?; + let closed = log.close().await; + assert!( + matches!(closed, Err(LogError::Corrupt(_))), + "poisoned cache must not report a healthy engine close: {closed:?}" + ); + assert!(matches!(log.storage_instance(), Err(LogError::Closed))); + // A typed corruption error still joins physical engine teardown. Reopen + // before shutting down the domain or dropping retained old group handles. + let mut resumed = options(root.path()); + resumed.mode = LogOpenMode::Resume; + resumed.process = ProcessGeneration::new([9; 16])?; + let reopened = domain + .run(ReplicaLog::open( + resumed, + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + assert_eq!( + reopened.group(&bootstrap(1).scope)?.identity().process, + ProcessGeneration::new([9; 16])? + ); + reopened.close().await?; domain.shutdown().await; assert!(matches!(result, Err(LogError::Corrupt(_)))); Ok(()) diff --git a/crates/cellar/durability/raft/host_controller.rs b/crates/cellar/durability/raft/host_controller.rs index 27f310919..7e88fa498 100644 --- a/crates/cellar/durability/raft/host_controller.rs +++ b/crates/cellar/durability/raft/host_controller.rs @@ -1,21 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Owned reconciliation. Inspection supplies hints; ordinary revision-bound -//! host/runtime operations retain every quorum, seeding and progress check. -use super::*; -use crate::application::consensus::client::RaftCellHandle; -use crate::raft::inspection::{InspectionError, InspectionSnapshot}; -use crate::raft::member_state::{RepairPhase, RepairPlan, RepairStep}; -use cellar_logic::raft_leadership::LeaderPolicy; -use cellar_logic::raft_placement::PlacementPolicy; -use tokio::sync::watch; - -#[path = "controller_action.rs"] -mod action; -#[path = "placement.rs"] -mod policy; -#[path = "controller_probe.rs"] -mod probe; +//! Explicit automatic-placement policy. Runtime integration belongs to groups. +use super::RaftHostOptions; +use cellar_logic::{raft_leadership::LeaderPolicy, raft_placement::PlacementPolicy}; const PROBE_MEMORY: usize = 1024 * 1024; @@ -97,405 +84,3 @@ pub struct RaftControllerStatus { pub transfer_attempts: u64, pub completed_repairs: u64, } - -struct FailedProbe { - peer: RaftPeerNode, - since: u64, - last: u64, -} - -struct History { - binding: Binding, - candidate_cursor: Option, - pull_cursor: Option, - epoch: u64, - failures: BTreeMap, - leader_term: Option, - gained_at: u64, - seeding: Option<(ProposalId, u64)>, - last_completed: Option, -} - -pub(super) struct Controller { - options: RaftControllerOptions, - status: RaftControllerStatus, - histories: Vec, - inventory: usize, - active: BTreeSet, - cursor: usize, - active_cursor: Option, - priority_turn: bool, - intent: Option<(usize, RepairPlan)>, - started: Option, - last_transfer: u64, - next_at: u64, - sequence: u64, - epoch: u64, - busy: Option, - stop: watch::Sender, - memory: Arc, -} - -pub(super) enum Completed { - Observed { - sequence: u64, - epoch: u64, - history: usize, - result: Result, CommandFailure>, - }, - Acted { - sequence: u64, - epoch: u64, - history: usize, - result: Result<(), CommandFailure>, - }, -} - -impl Controller { - pub(super) fn new(options: RaftControllerOptions, bindings: Vec) -> Self { - let (stop, _) = watch::channel(false); - let histories = bindings - .into_iter() - .map(|binding| History { - binding, - candidate_cursor: None, - pull_cursor: None, - epoch: 0, - failures: BTreeMap::new(), - leader_term: None, - gained_at: 0, - seeding: None, - last_completed: None, - }) - .collect(); - Self { - status: RaftControllerStatus { - enabled: !options.start_paused, - recovering: true, - probe_rounds: 0, - repair_requests: 0, - activation_requests: 0, - transfer_attempts: 0, - completed_repairs: 0, - }, - memory: Budget::new(options.metadata_bytes, 1), - options, - histories, - inventory: 0, - active: BTreeSet::new(), - cursor: 0, - active_cursor: None, - priority_turn: false, - intent: None, - started: None, - last_transfer: 0, - next_at: 0, - sequence: 0, - epoch: 1, - busy: None, - stop, - } - } - - pub(super) fn stop(&self) { - self.stop.send_replace(true); - } - - fn next_sequence(&mut self) -> Option { - match self.sequence.checked_add(1) { - Some(sequence) => { - self.sequence = sequence; - Some(sequence) - } - None => { - self.status.enabled = false; - self.stop.send_replace(true); - None - } - } - } - - fn history(&mut self, index: usize, now: u64) -> Option<&mut History> { - let history = self.histories.get_mut(index)?; - if history.epoch != self.epoch { - history.epoch = self.epoch; - history.failures.clear(); - history.leader_term = None; - history.gained_at = now; - history.seeding = None; - } - Some(history) - } - - fn next_history(&mut self) -> Option { - if self.histories.is_empty() { - return None; - } - self.priority_turn = !self.priority_turn; - if self.priority_turn && !self.active.is_empty() { - let after = self - .active_cursor - .map_or(std::ops::Bound::Unbounded, std::ops::Bound::Excluded); - let index = self - .active - .range((after, std::ops::Bound::Unbounded)) - .next() - .or_else(|| self.active.first()) - .copied()?; - self.active_cursor = Some(index); - return Some(index); - } - let index = self.cursor; - self.cursor = (self.cursor + 1) % self.histories.len(); - Some(index) - } -} - -impl Driver { - pub(super) fn publish_controller_resources(&self) { - let sample = self.controller.as_ref().and_then(|controller| { - Some(( - self.log.domain().mono_ms().ok()?, - crate::application::consensus::RaftControllerResources { - enabled: controller.status.enabled, - recovering: controller.status.recovering, - busy: controller.busy.is_some(), - metadata: crate::application::consensus::RaftBudgetUsage::capture( - &controller.memory, - ), - probe_rounds: controller.status.probe_rounds, - repair_requests: controller.status.repair_requests, - activation_requests: controller.status.activation_requests, - transfer_attempts: controller.status.transfer_attempts, - completed_repairs: controller.status.completed_repairs, - sample_age_ms: 0, - }, - )) - }); - if let Ok(mut source) = self.local.controller.lock() { - *source = sample; - } - } - - pub(super) fn controller_control(&mut self, mut request: NodeRequest, enabled: Option) { - let result = (|| { - let controller = self - .controller - .as_mut() - .ok_or(CommandFailure::UnsupportedCapability)?; - if let Some(enabled) = enabled { - if enabled != controller.status.enabled { - let now = self - .log - .domain() - .mono_ms() - .map_err(|_| CommandFailure::Unavailable)?; - let epoch = controller - .epoch - .checked_add(1) - .ok_or(CommandFailure::Unavailable)?; - controller.stop.send_replace(true); - let (stop, _) = watch::channel(false); - controller.stop = stop; - controller.epoch = epoch; - controller.status.enabled = enabled; - controller.status.recovering = true; - controller.inventory = 0; - controller.active.clear(); - controller.last_transfer = now; - controller.next_at = now.saturating_add(controller.options.interval_ms); - } - } - Ok(Response::Controller(controller.status.clone())) - })(); - reply_node(&mut request, result); - } - - pub(super) fn pump_controller(&mut self) { - if self.local.admission.application_paused() { - if let Some(controller) = &self.controller { - controller.stop(); - } - return; - } - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Some(controller) = self.controller.as_mut() else { - return; - }; - if !controller.status.enabled || controller.busy.is_some() || now < controller.next_at { - return; - } - if controller.started.is_none() { - controller.started = Some(now); - controller.last_transfer = now; - } - // Reconstruct existing durable plans in bounded batches before any new - // automatic plan. Directory state cannot supply or clear this intent. - if controller.status.recovering { - let end = (controller.inventory + 4).min(controller.histories.len()); - for index in controller.inventory..end { - let scope = &controller.histories[index].binding.scope; - let group = match self.log.group(scope) { - Ok(group) => group, - Err(super::super::storage::LogError::NotFound) => continue, - Err(_) => { - controller.next_at = now.saturating_add(controller.options.interval_ms); - return; - } - }; - let active = match group - .membership() - .and_then(|membership| membership.repair_status()) - { - Ok(status) => status.is_some_and(|status| status.phase.active()), - Err(_) => { - controller.next_at = now.saturating_add(controller.options.interval_ms); - return; - } - }; - if active { - controller.active.insert(index); - } - } - controller.inventory = end; - if end != controller.histories.len() { - return; - } - controller.status.recovering = false; - if let Some((index, _)) = controller.intent.as_ref() { - controller.active.insert(*index); - } - } - let Some(index) = controller.next_history() else { - return; - }; - let options = controller.options.clone(); - let Some(memory) = controller - .memory - .acquire(options.probes_per_group * PROBE_MEMORY) - else { - return; - }; - let Some(sequence) = controller.next_sequence() else { - return; - }; - let epoch = controller.epoch; - let stop = controller.stop.subscribe(); - let Some(history) = controller.history(index, now) else { - return; - }; - let binding = history.binding.clone(); - let cursor = if binding.member().is_some() { - &mut history.candidate_cursor - } else { - &mut history.pull_cursor - }; - let setup = probe::targets( - &binding.scope, - &self.cells, - &self.peers, - self.log.limits().node, - *cursor, - options.probes_per_group, - ); - let (targets, next) = match setup { - Ok(setup) => setup, - Err(_) => { - controller.next_at = now.saturating_add(options.interval_ms); - return; - } - }; - *cursor = next; - controller.busy = Some(sequence); - controller.next_at = now.saturating_add(options.interval_ms); - let Some(count) = controller.status.probe_rounds.checked_add(1) else { - controller.status.enabled = false; - controller.stop.send_replace(true); - return; - }; - controller.status.probe_rounds = count; - let work = probe::Work { - local: RaftCellHandle { - handle: self.local.clone(), - binding, - }, - transport: self.transport.clone(), - runtime: self.runtime.clone(), - domain: self.log.domain(), - options, - targets, - memory, - }; - let performance = self.local.performance.clone(); - self.jobs.push(Box::pin(async move { - let timer = performance.timer(crate::raft::performance::Operation::ControllerProbe); - let result = action::until_stopped(stop, work.run()).await; - timer.finish(result.is_ok()); - Job::Controller(Box::new(Completed::Observed { - sequence, - epoch, - history: index, - result: result.map(Box::new), - })) - })); - } - - pub(super) fn finish_controller(&mut self, completed: Completed) { - let (sequence, epoch, index) = match &completed { - Completed::Observed { - sequence, - epoch, - history, - .. - } - | Completed::Acted { - sequence, - epoch, - history, - .. - } => (*sequence, *epoch, *history), - }; - let Some(controller) = self.controller.as_mut() else { - return; - }; - if controller.busy != Some(sequence) { - return; - } - controller.busy = None; - if !controller.status.enabled - || controller.epoch != epoch - || self.closing.load(Ordering::Acquire) - || self.local.admission.application_paused() - { - return; - } - let observed = match completed { - Completed::Observed { - result: Ok(observed), - .. - } => observed, - Completed::Acted { - result: Err(error), .. - } => { - tracing::debug!(%error, "automatic operation needs committed-state reconciliation"); - return; - } - _ => return, - }; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let context = policy::Context { - cells: &self.cells, - peers: &self.peers, - local: self.log.limits().node, - rng: &mut self.rng, - }; - let decision = policy::decide(controller, index, *observed, now, context); - if let Ok(Some(action)) = decision { - self.start_controller_action(index, action, now); - } - } -} diff --git a/crates/cellar/durability/raft/host_directory.rs b/crates/cellar/durability/raft/host_directory.rs index 28c140f40..d9f65a6fb 100644 --- a/crates/cellar/durability/raft/host_directory.rs +++ b/crates/cellar/durability/raft/host_directory.rs @@ -1,27 +1,6 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Owned exact-key discovery. Directory observations refresh endpoints only; -//! existing quorum authority survives missing metadata or an object-store outage. -use super::super::fleet_catalog::{FleetHostAuthorization, FleetTrustReader, FleetTrustSnapshot}; -use super::*; -use std::time::Duration; - -const FLEET_READ_BYTES: usize = 4 * 1024 * 1024; - -#[derive(Clone)] -enum DiscoveryNode { - Configured(RaftPeerNode), - Enrolled(Box), -} - -impl DiscoveryNode { - fn node(&self) -> NodeId { - match self { - Self::Configured(peer) => peer.node, - Self::Enrolled(node) => node.node(), - } - } -} +//! Explicit bounds for the physical-node directory owner. #[derive(Clone, Debug)] pub struct RaftNodeDirectoryOptions { @@ -54,421 +33,3 @@ impl RaftNodeDirectoryOptions { Ok(()) } } - -pub(super) struct Directory { - options: RaftNodeDirectoryOptions, - registrations: Vec, - trusted: Vec, - enrollment: Option, - rooted: Option, - namespace: usize, - node: usize, - next_at: u64, - busy: bool, - own_revision: u64, -} - -fn remaining(domain: &crate::asyncrt::DomainToken, deadline: u64) -> anyhow::Result { - let millis = deadline - .checked_sub(domain.mono_ms()?) - .filter(|ms| *ms > 0) - .ok_or_else(|| anyhow::anyhow!("node directory original deadline"))?; - Ok(Duration::from_millis(millis)) -} - -pub(super) async fn prepare( - options: Option<&RaftNodeDirectoryOptions>, - cells: &[Registration], - declared: &[NamespaceRegistration], - mut enrollment: Option<&mut super::super::fleet_catalog::FleetHostAuthorization>, - peers: &mut BTreeMap, - service: &crate::daemon::NodeRaftService, - label: &str, -) -> anyhow::Result> { - let Some(options) = options else { - return Ok(None); - }; - options.validate()?; - anyhow::ensure!( - !declared.is_empty() - || cells - .iter() - .any(|cell| !matches!(cell.placement, RaftCellPlacement::QueryRouter { .. })), - "node directory requires stable replica or standby storage" - ); - let local = service.log.limits(); - let own = peers - .get(&local.node) - .ok_or_else(|| anyhow::anyhow!("node directory requires an explicit self endpoint"))? - .clone(); - anyhow::ensure!( - own.node == local.node - && own.disk == local.disk - && own.process == local.process - && own.label == label, - "node directory self endpoint disagrees with engine owner" - ); - for peer in peers.values() { - peer.validate_endpoint()?; - } - let mut namespaces = BTreeMap::new(); - for (namespace, class) in cells - .iter() - .map(|c| (&c.namespace, &c.class)) - .chain(declared.iter().map(|c| (&c.namespace, &c.class))) - { - namespace.require_mode(cellar_logic::NamespaceMode::Raft)?; - anyhow::ensure!( - namespace.config().schema_digest == class.schema_digest(), - "registered schema disagrees with namespace" - ); - namespaces - .entry(namespace.config().cluster) - .or_insert_with(|| namespace.clone()); - } - let domain = service.log.domain(); - let deadline = domain.mono_ms()?.saturating_add(options.timeout_ms); - let boot = if enrollment.is_some() { - let mut id = [0; 16]; - crate::asyncrt::rng("fleet-host-boot").fill_bytes(&mut id); - Some(cellar_logic::consensus::ProposalId::new(id)?) - } else { - None - }; - let mut registrations = Vec::with_capacity(namespaces.len()); - let mut own_revision = 0; - let mut rooted = None; - for namespace in namespaces.into_values() { - let registration = namespace - .register_node_storage(&service.log, remaining(&domain, deadline)?) - .await?; - if let Some(authorization) = enrollment.as_mut() { - let boot = boot.ok_or_else(|| anyhow::anyhow!("missing managed boot identity"))?; - let join_key = crate::asyncrt::services() - .peer_probe_signer()? - .public_key_bytes(); - let (bound, update) = if authorization.state() - == crate::raft::fleet_catalog::FleetNodeState::RestartPending - { - authorization - .bind_restart_boot( - &service.log, - boot, - join_key, - deadline, - remaining(&domain, deadline)?, - ) - .await? - } else { - // Preserve the original ordinary-boot poll path. Polling a - // combined maintenance wrapper adds stack even when boxed. - let bound = authorization - .bind_boot( - service.log.storage_instance()?, - local.process, - boot, - join_key, - remaining(&domain, deadline)?, - ) - .await?; - let update = crate::raft::fleet_stop::LifecycleUpdate::Boot { - boot: bound.boot_binding()?, - deadline, - }; - (bound, update) - }; - **authorization = bound; - crate::asyncrt::timeout( - remaining(&domain, deadline)?, - service.log.update_lifecycle(update), - ) - .await??; - if let Some(selector) = service - .log - .lifecycle_guard()? - .and_then(|g| g.quiescing().cloned()) - { - crate::asyncrt::services() - .raft_lifecycle() - .install_startup_maintenance(&selector)?; - } - let trust = authorization - .trust_snapshot(remaining(&domain, deadline)?) - .await?; - service.transport.install_fleet_trust(trust.clone())?; - anyhow::ensure!( - trust.nodes().len() <= service.transport.peer_limit(), - "fleet exceeds transport peer bound" - ); - rooted = Some(trust); - } - let publication = registration - .publication(own.clone(), remaining(&domain, deadline)?) - .await?; - let view = publication.publish(remaining(&domain, deadline)?).await?; - own_revision = own_revision.max(view.revision()); - service.transport.accept_node_view(view, &own)?; - // Cold managed members have only a self hint. Resolve exact rooted - // process views before constructing their current membership routes; - // periodic discovery then refreshes restarts without static addresses. - if let Some(trust) = &rooted { - for node in trust - .nodes() - .iter() - .filter(|node| node.node() != local.node) - { - let result = registration - .observe_fleet_node(node, remaining(&domain, deadline)?) - .await; - match result { - Ok(Some(view)) => { - let peer = view.peer().clone(); - service.transport.accept_node_view(view, &peer)?; - peers.insert(peer.node, peer); - } - Ok(None) => (), - Err(error) => tracing::debug!(%error, "startup node view unavailable"), - } - } - } - registrations.push(registration); - } - Ok(Some(Directory { - options: options.clone(), - registrations, - trusted: rooted.as_ref().map_or_else( - || { - peers - .values() - .cloned() - .map(DiscoveryNode::Configured) - .collect() - }, - trusted_nodes, - ), - enrollment: enrollment - .as_deref() - .map(FleetHostAuthorization::trust_reader), - rooted, - namespace: 0, - node: 0, - next_at: domain.mono_ms()?, - busy: false, - own_revision, - })) -} - -enum DiscoveryError { - Catalog(CatalogError), - Transport(super::super::transport::TransportError), -} - -struct PeerObservation { - trusted: DiscoveryNode, - result: Result, DiscoveryError>, -} - -pub(super) struct Observation { - peers: Vec, - trust: Option>, - _metadata: Option, -} - -fn trusted_nodes(trust: &FleetTrustSnapshot) -> Vec { - trust - .nodes() - .iter() - .filter(|node| node.permits_transport()) - .cloned() - .map(|node| DiscoveryNode::Enrolled(Box::new(node))) - .collect() -} - -impl Directory { - pub(super) fn authorization(&self, node: NodeId) -> Option<&FleetHostAuthorization> { - self.rooted.as_ref()?.node(node) - } - - pub(super) fn rooted_revision(&self) -> Option { - self.rooted.as_ref().map(FleetTrustSnapshot::revision) - } -} - -impl Driver { - pub(super) fn pump_directory(&mut self) { - let Some(directory) = self.directory.as_mut() else { - return; - }; - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - if directory.busy || now < directory.next_at { - return; - } - let metadata = if directory.enrollment.is_some() { - let Some(permit) = self.control_budget.acquire(FLEET_READ_BYTES) else { - directory.next_at = now.saturating_add(directory.options.poll_ms); - return; - }; - Some(permit) - } else { - None - }; - directory.busy = true; - directory.next_at = now.saturating_add(directory.options.poll_ms); - let deadline = now.saturating_add(directory.options.timeout_ms); - let count = directory.options.batch_nodes.min( - directory - .registrations - .len() - .saturating_mul(directory.trusted.len()), - ); - let mut batch = Vec::with_capacity(count); - for _ in 0..count { - batch.push(( - directory.registrations[directory.namespace].clone(), - directory.trusted[directory.node].clone(), - )); - directory.node += 1; - if directory.node == directory.trusted.len() { - directory.node = 0; - directory.namespace = (directory.namespace + 1) % directory.registrations.len(); - } - } - let transport = self.transport.clone(); - let domain = self.log.domain(); - let enrollment = directory.enrollment.clone(); - let previous = directory.rooted.clone(); - self.jobs.push(Box::pin(async move { - let trust = if let Some(enrollment) = enrollment { - Some( - transport - .seed_control_until(deadline, async { - let timeout = remaining(&domain, deadline) - .map_err(|_| super::super::transport::TransportError::Timeout)?; - Ok(enrollment.refresh(previous.as_ref(), timeout).await) - }) - .await - .map_err(|error| error.to_string()) - .and_then(|result| result.map_err(|error| error.to_string())), - ) - } else { - None - }; - let observations = - futures_util::future::join_all(batch.into_iter().map(|(registration, trusted)| { - let transport = transport.clone(); - let domain = domain.clone(); - async move { - let result = transport - .seed_control_until(deadline, async { - let timeout = remaining(&domain, deadline).map_err(|_| { - super::super::transport::TransportError::Timeout - })?; - Ok(match &trusted { - DiscoveryNode::Configured(peer) => { - registration.observe_node(peer, timeout).await - } - DiscoveryNode::Enrolled(node) => { - registration.observe_fleet_node(node, timeout).await - } - }) - }) - .await - .map_err(DiscoveryError::Transport) - .and_then(|result| result.map_err(DiscoveryError::Catalog)); - PeerObservation { trusted, result } - } - })) - .await; - Job::Directory(Box::new(Observation { - peers: observations, - trust, - _metadata: metadata, - })) - })); - } - - pub(super) fn on_directory(&mut self, observation: Observation) { - if let Some(directory) = self.directory.as_mut() { - directory.busy = false; - } - if let Some(result) = observation.trust { - match result { - Ok(trust) => { - let local = self.log.limits(); - if trust - .node(local.node) - .is_none_or(|node| !node.permits_transport()) - { - self.closing.store(true, Ordering::Release); - return; - } - match self.transport.install_fleet_trust(trust.clone()) { - Ok(()) => { - if let Some(directory) = &mut self.directory { - if directory.rooted_revision() != Some(trust.revision()) { - directory.rooted = Some(trust.clone()); - directory.trusted = trusted_nodes(&trust); - directory.node = directory - .node - .min(directory.trusted.len().saturating_sub(1)); - } - } - self.peers.retain(|node, peer| { - trust - .node(*node) - .is_some_and(|entry| entry.matches_peer(peer)) - }); - } - Err(error) => tracing::debug!(%error, "Raft fleet trust cut refused"), - } - } - Err(error) => tracing::debug!(%error, "Raft fleet trust refresh unavailable"), - } - } - for observation in observation.peers { - let view = match observation.result { - Ok(Some(view)) => view, - // No observation supplies authority or invalidates a live - // quorum. Keep the last accepted routes and retry boundedly. - Ok(None) => continue, - Err(DiscoveryError::Catalog(error)) => { - tracing::debug!(node = ?observation.trusted.node(), %error, "Raft node directory observation refused"); - continue; - } - Err(DiscoveryError::Transport(error)) => { - tracing::debug!(node = ?observation.trusted.node(), %error, "Raft node directory observation cancelled"); - continue; - } - }; - let peer = view.peer().clone(); - let local = self.log.limits(); - let newer_boot = self.directory.as_ref().is_some_and(|directory| { - peer.process != local.process && view.revision() > directory.own_revision - }); - if peer.node == local.node - && (newer_boot || self.log.storage_instance().ok() != Some(view.storage_instance())) - { - // The shared closing flag immediately fences retained handles; - // the host loop then joins every owned service and physical job. - self.closing.store(true, Ordering::Release); - return; - } - let trusted = match &observation.trusted { - DiscoveryNode::Configured(peer) => peer, - // The transport's installed rooted cut independently checks - // registration identity and storage pin before accepting this. - DiscoveryNode::Enrolled(_) => &peer, - }; - match self.transport.accept_node_view(view, trusted) { - Ok(()) => { - self.peers.insert(peer.node, peer); - } - Err(error) => { - tracing::debug!(node = ?peer.node, %error, "Raft node process refresh refused") - } - } - } - } -} diff --git a/crates/cellar/durability/raft/host_effects.rs b/crates/cellar/durability/raft/host_effects.rs index 489a5965e..8532015e2 100644 --- a/crates/cellar/durability/raft/host_effects.rs +++ b/crates/cellar/durability/raft/host_effects.rs @@ -295,7 +295,7 @@ impl Driver { let mut options = client::CommandOptions::new(scan.next, retry_until); options.durability = due.minimum(); options.timeout = std::time::Duration::from_millis(owner.options.operation_timeout_ms); - let local = crate::application::consensus::RaftCellHandle { + let local = crate::application::consensus::client::RaftCellHandle { handle: self.local.clone(), binding, }; diff --git a/crates/cellar/durability/raft/host_enrollment.rs b/crates/cellar/durability/raft/host_enrollment.rs index 1336489f6..2051329d8 100644 --- a/crates/cellar/durability/raft/host_enrollment.rs +++ b/crates/cellar/durability/raft/host_enrollment.rs @@ -84,86 +84,3 @@ pub(crate) struct NamespaceRegistration { pub namespace: NamespaceCatalog, pub class: Arc, } - -pub(super) async fn prepare( - options: &mut RaftHostOptions, - service: &crate::daemon::NodeRaftService, - cells: &[Registration], - namespaces: &[NamespaceRegistration], - peers: &BTreeMap, - label: &str, -) -> anyhow::Result> { - let Some(managed) = &mut options.managed else { - anyhow::ensure!(namespaces.is_empty(), "undeclared managed namespaces"); - return Ok(BTreeMap::new()); - }; - managed.authorization = managed - .authorization - .refresh(std::time::Duration::from_secs(5)) - .await?; - managed.validate()?; - anyhow::ensure!( - namespaces.len() == managed.namespaces.len(), - "incomplete managed namespace preparation" - ); - for (prepared, declared) in namespaces.iter().zip(&managed.namespaces) { - anyhow::ensure!( - prepared.namespace.config() == &declared.namespace - && prepared.class.name() == declared.class - && prepared.class.schema_digest() == declared.namespace.schema_digest, - "managed namespace preparation mismatch" - ); - } - let auth = &managed.authorization; - let local = service.log.limits(); - let own = peers - .get(&local.node) - .ok_or_else(|| anyhow::anyhow!("managed host self endpoint missing"))?; - anyhow::ensure!( - auth.node() == local.node - && auth.disk() == local.disk - && auth.label() == label - && own.label == auth.label() - && own.server_name == auth.server_name() - && auth.instance_matches(service.log.storage_instance()?), - "managed host physical identity mismatch" - ); - managed - .authorization - .check_declared_namespaces(namespaces, std::time::Duration::from_secs(5)) - .await?; - - let mut authorizations = BTreeMap::new(); - for cell in cells { - let scope = cell.birth.scope(); - let authorization = auth - .group_authorization(scope, std::time::Duration::from_secs(5)) - .await?; - anyhow::ensure!( - authorization.check_birth(&cell.namespace, &cell.birth), - "managed history birth/schema mismatch" - ); - match service.log.group(scope) { - Ok(group) => anyhow::ensure!( - group - .membership()? - .fleet_status() - .is_some_and(|status| status.binding == *authorization.binding()), - "managed host requires committed group enrollment" - ), - Err(super::super::storage::LogError::NotFound) => anyhow::ensure!( - cell.placement == RaftCellPlacement::Replica - && cell.birth.managed_owner().is_some() - && authorization.allows_managed_genesis() - || matches!( - cell.placement, - RaftCellPlacement::Standby | RaftCellPlacement::QueryRouter { .. } - ), - "managed host cannot bootstrap an unguarded member" - ), - Err(error) => return Err(error.into()), - } - authorizations.insert(scope.clone(), authorization); - } - Ok(authorizations) -} diff --git a/crates/cellar/durability/raft/host_extents.rs b/crates/cellar/durability/raft/host_extents.rs index 57c42447f..ee2ba5201 100644 --- a/crates/cellar/durability/raft/host_extents.rs +++ b/crates/cellar/durability/raft/host_extents.rs @@ -4,11 +4,7 @@ //! The node supplies physical policy; immutable namespaces select commitments. use super::*; -use cellar_ltx::{ - internal::{ExtentRootReference, NodeExtentStore, EXTENT_BYTES}, - stream_snapshot::{ScratchLimits, ScratchPool}, -}; -use std::path::Path; +use cellar_ltx::{internal::EXTENT_BYTES, stream_snapshot::ScratchLimits}; pub use cellar_ltx::internal::{ CacheLimits as RaftExtentCacheLimits, NodeExtentLimits as RaftExtentLimits, @@ -66,139 +62,3 @@ impl RaftExtentOptions { Ok(()) } } - -pub(super) struct Owner { - pub store: NodeExtentStore, - cache: ScratchPool, - options: RaftExtentOptions, - next_gc: u64, - busy: bool, - failed: bool, -} - -impl Owner { - /// Startup is still parked. Rebuild retention from every durable engine - /// group, including unregistered/closed groups, before allowing any GC. - pub async fn open( - log: &super::super::storage::ReplicaLog, - directory: PathBuf, - host: cellar_ltx::LtxHost, - options: Option, - needed: bool, - ) -> anyhow::Result> { - if !needed && options.is_none() { - return Ok(None); - } - let options = options - .ok_or_else(|| anyhow::anyhow!("extent namespace requires node extent storage"))?; - let groups = log.groups()?; - let owner = log - .domain() - .blocking(move || { - let store = NodeExtentStore::open_serialized( - directory.join("extent-inventory"), - host.filesystem(), - options.inventory, - )?; - let mut retained = BTreeMap::<[u8; 32], ExtentRootReference>::new(); - for group in groups { - let applied = group.applied()?; - let snapshot_index = group.read()?.snapshot.get_metadata().index; - for (role, revision, image) in [ - (1, applied.position.index.get(), applied.image), - ( - 2, - snapshot_index, - super::super::snapshot::retained_image(&group)?, - ), - ] { - let Some(image) = image else { - continue; - }; - image.validate()?; - let crate::application::consensus::ImageCommitment::Extents(reference) = - image.commitment - else { - continue; - }; - store.retain_reference( - super::super::rooted::marker::reference_key(&group, role), - revision, - reference, - )?; - retained.insert(reference.root().bytes(), reference); - } - } - store.retain_only(&retained.into_values().collect::>())?; - store.reclaim(options.gc_batch_files)?; - Ok::<_, anyhow::Error>(Self { - store, - cache: ScratchPool::new(options.cache_scratch)?, - options, - next_gc: 0, - busy: false, - failed: false, - }) - })? - .await??; - Ok(Some(owner)) - } - - pub fn cell_storage(&self, directory: &Path) -> sql::RootStorage { - sql::RootStorage { - store: self.store.clone(), - cache_pool: self.cache.clone(), - cache: self.options.cache, - cache_directory: directory.join("extent-cache"), - } - } -} - -impl Driver { - pub(super) fn pump_extent_gc(&mut self) { - let Some(owner) = &mut self.extents else { - return; - }; - let now = crate::asyncrt::mono_ms(); - if owner.busy || owner.failed { - return; - } - let pressure = owner.store.usage().is_ok_and(|usage| { - usage.root_extents >= owner.options.inventory.max_root_extents.saturating_mul(3) / 4 - || usage.roots >= owner.options.inventory.max_roots.saturating_mul(3) / 4 - || usage.bytes >= owner.options.inventory.max_bytes.saturating_mul(3) / 4 - }); - if now < owner.next_gc && !pressure { - return; - } - owner.busy = true; - owner.next_gc = now.saturating_add(owner.options.gc_interval_ms); - let store = owner.store.clone(); - let limit = owner.options.gc_batch_files; - let job = self.log.domain().blocking(move || store.reclaim(limit)); - self.jobs.push(Box::pin(async move { - let result = match job { - Ok(job) => job - .await - .map_err(anyhow::Error::from) - .and_then(|value| value.map_err(anyhow::Error::from)), - Err(error) => Err(error.into()), - }; - Job::ExtentGc(result) - })); - } - - pub(super) fn extent_gc_completed(&mut self, result: anyhow::Result) { - if let Some(owner) = &mut self.extents { - owner.busy = false; - match result { - Ok(removed) if removed == owner.options.gc_batch_files => owner.next_gc = 0, - Ok(_) => (), - Err(error) => { - owner.failed = true; - tracing::error!(event = "raft_extent_gc_failed", %error); - } - } - } - } -} diff --git a/crates/cellar/durability/raft/host_fleet_cut.rs b/crates/cellar/durability/raft/host_fleet_cut.rs index cf3a876b6..f362c04b1 100644 --- a/crates/cellar/durability/raft/host_fleet_cut.rs +++ b/crates/cellar/durability/raft/host_fleet_cut.rs @@ -1,5 +1,5 @@ -//! Fixed publication cuts pair the durable configuration with its SQL image. -//! A data position can precede configuration-only applied progress. +//! Fixed complete-group roots pair durable membership with all cell state. +//! Authenticated mutation summaries distinguish data from protocol-only apply. use super::*; use crate::application::consensus::ImageCommitment; use crate::raft::cache::{CommittedImage, DurableApply}; @@ -287,34 +287,42 @@ fn marker_state(image: &CommittedImage) -> Result, storage_instance: EngineInstanceId, - birth: [u8; 32], - schema: [u8; 32], - revision: LogIndex, - applied: LogPosition, - image: Option, + root: crate::raft::cell_groups::GroupManifest, + configuration_digest: [u8; 32], } impl FleetPublishedCut { - /// Re-establish a stored historical prefix from an actual current quorum - /// publication. Serialized receipt material alone cannot construct a cut. + pub(crate) fn capture_group( + membership: &FleetMembershipObservation, + storage_instance: EngineInstanceId, + ) -> Self { + Self { + binding: membership.status().binding.clone(), + replica: membership.replica().clone(), + storage_instance, + root: membership.root().clone(), + configuration_digest: membership.configuration_digest(), + } + } + + /// Stored journal data is historical material, never an independently minted + /// proof. Re-establish its prefix through an actual current quorum cut. pub(crate) fn checked_prefix( &self, - applied: LogPosition, - revision: LogIndex, - image: Option, + root: crate::raft::cell_groups::GroupManifest, + configuration_digest: [u8; 32], ) -> Result { + root.check().map_err(|_| CommandFailure::CorruptState)?; let prefix = Self { - applied, - revision, - image, + root, + configuration_digest, ..self.clone() }; if !self.covers(&prefix)? { @@ -323,33 +331,11 @@ impl FleetPublishedCut { Ok(prefix) } - pub(crate) fn capture( - membership: &FleetMembershipObservation, - storage_instance: EngineInstanceId, - publication: &Publication, - ) -> Option { - let image = match publication { - Publication::Pending => return None, - Publication::Empty => None, - Publication::Image(image) => Some(image.clone()), - }; - Some(Self { - binding: membership.status().binding.clone(), - replica: membership.replica().clone(), - storage_instance, - birth: membership.birth_digest(), - schema: membership.schema_digest(), - revision: membership.membership_revision(), - applied: membership.applied(), - image, - }) - } - pub fn binding(&self) -> &FleetGroupBinding { &self.binding } - pub fn replica(&self) -> &ReplicaIdentity { + pub fn replica(&self) -> &ReplicaIdentity { &self.replica } @@ -358,83 +344,77 @@ impl FleetPublishedCut { } pub fn birth_digest(&self) -> [u8; 32] { - self.birth + self.root.route().birth_digest } pub fn schema_digest(&self) -> [u8; 32] { - self.schema + self.root.route().schema_digest } pub fn membership_revision(&self) -> LogIndex { - self.revision + LogIndex::new(self.root.membership_revision()) } - /// Includes configuration-only progress even when the SQL data position - /// has not advanced. The image state was checked against this durable cut. pub fn applied(&self) -> LogPosition { - self.applied + self.root.position() } - /// None is proven canonical empty state, not missing publication metadata. - pub fn image(&self) -> Option<&FleetPublishedImage> { - self.image.as_ref() + /// The complete group commitment includes the route, every cell's image, + /// enrollment/receipt summary and recovery lineage. Empty is an explicit + /// zero-cell manifest; missing/pending publication cannot construct a cut. + pub fn root(&self) -> &crate::raft::cell_groups::GroupManifest { + &self.root + } + + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest } - /// Compare committed prefix coverage at the same exact configuration. - /// Physical replicas may differ. Configuration changes require a fresh - /// verification attempt; inconsistent same-position images are corruption. pub fn covers(&self, minimum: &Self) -> Result { if self.binding != minimum.binding - || self.birth != minimum.birth - || self.schema != minimum.schema - || self.revision != minimum.revision + || self.membership_revision() != minimum.membership_revision() + || self.configuration_digest != minimum.configuration_digest + || self.root.route() != minimum.root.route() { return Err(CommandFailure::CommandConflict); } - if self - .image - .as_ref() - .zip(minimum.image.as_ref()) - .is_some_and(|(a, b)| !a.state.same_representation(&b.state)) - { - return Err(CommandFailure::CorruptState); - } - if self.applied.index < minimum.applied.index { - return Ok(false); - } - if self.applied.term < minimum.applied.term - || self.applied.index == minimum.applied.index - && self.applied.term != minimum.applied.term - { + group_prefix(&self.root, &minimum.root) + } +} + +pub(crate) fn group_prefix( + current: &crate::raft::cell_groups::GroupManifest, + minimum: &crate::raft::cell_groups::GroupManifest, +) -> Result { + current + .check() + .and_then(|()| minimum.check()) + .map_err(|_| CommandFailure::CorruptState)?; + if current.route() != minimum.route() + || current.membership_revision() != minimum.membership_revision() + { + return Err(CommandFailure::CommandConflict); + } + let (a, b) = (current.position(), minimum.position()); + if a.index < b.index { + return Ok(false); + } + if a.term < b.term + || a.index == b.index && current != minimum + || !current.preserves_recovery(minimum) + { + return Err(CommandFailure::CorruptState); + } + if current.root() != minimum.root() { + let mutation = current + .root() + .last_mutation() + .ok_or(CommandFailure::CorruptState)?; + if mutation.index <= b.index || mutation.term < b.term { return Err(CommandFailure::CorruptState); } - match (&self.image, &minimum.image) { - (None, None) => Ok(true), - (None, Some(_)) => Err(CommandFailure::CorruptState), - (Some(current), prior) => { - if !consistent_positions(current.position, minimum.applied) - || prior.as_ref().is_some_and(|prior| { - !consistent_positions(current.position, prior.position) - }) - { - return Err(CommandFailure::CorruptState); - } - // Live and restored replicas can publish exactly the same - // state at different protocol-only positions. Changed state - // must instead follow the entire minimum applied prefix. - if prior - .as_ref() - .is_some_and(|prior| current.state == prior.state) - { - return Ok(true); - } - if current.position.index <= minimum.applied.index { - return Err(CommandFailure::CorruptState); - } - Ok(true) - } - } } + Ok(true) } fn consistent_positions(a: LogPosition, b: LogPosition) -> bool { diff --git a/crates/cellar/durability/raft/host_fleet_cut_tests.rs b/crates/cellar/durability/raft/host_fleet_cut_tests.rs index b3539fb32..ae09a52ea 100644 --- a/crates/cellar/durability/raft/host_fleet_cut_tests.rs +++ b/crates/cellar/durability/raft/host_fleet_cut_tests.rs @@ -1,3 +1,5 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + use super::*; fn position(index: u64, term: u64) -> LogPosition { @@ -19,45 +21,52 @@ fn marker() -> CommittedImage { } } -fn image(index: u64, term: u64, digest: u8) -> FleetPublishedImage { - FleetPublishedImage::new( - position(index, term), - ImageState { - epoch: 1, - txid: TXID(index), - checksum: 0, - page_size: 4096, - length: 4096, - digest: [digest; 32], +fn root( + index: u64, + term: u64, + mutation: u64, + digest: u8, +) -> crate::raft::cell_groups::GroupManifest { + use cellar_logic::cell_group::*; + let route = GroupRoute { + history: GroupHistory { + cluster: ClusterId::new([1; 16]).unwrap(), + namespace: NamespaceId::new([2; 16]).unwrap(), + group: GroupId::new([3; 16]).unwrap(), + incarnation: IncarnationId::new([4; 16]).unwrap(), }, - ) - .unwrap() + range: CellRange::all(), + epoch: RouteEpoch::new(1).unwrap(), + birth_digest: [8; 32], + schema_digest: [9; 32], + }; + let mut value = + serde_json::to_value(crate::raft::cell_groups::GroupManifest::genesis(route).unwrap()) + .unwrap(); + value["index"] = serde_json::json!(index); + value["term"] = serde_json::json!(term); + if mutation > 0 { + value["root"] = serde_json::json!({"digest":vec![digest;32],"height":2,"cells":2,"last_mutation_index":mutation,"last_mutation_term":1,"next_due_unix_millis":null}); + } + serde_json::from_value(value).unwrap() } -fn cut(node: u8, index: u64, term: u64, image: Option) -> FleetPublishedCut { - let scope = GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:cut".into(), - group: GroupId::new([3; 16]).unwrap(), - incarnation: IncarnationId::new([4; 16]).unwrap(), - }; +fn cut(node: u8, index: u64, term: u64, mutation: u64, digest: u8) -> FleetPublishedCut { + let root = root(index, term, mutation, digest); + let history = root.route().history; FleetPublishedCut { - binding: FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &scope, [6; 32]) + binding: FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &history, [6; 32]) .unwrap(), replica: ReplicaIdentity { - scope, + scope: history, member: MemberId::new(node as u64).unwrap(), node: NodeId::new([node; 16]).unwrap(), disk: DiskLineage::new([node + 10; 16]).unwrap(), process: ProcessGeneration::new([node + 20; 16]).unwrap(), }, storage_instance: EngineInstanceId::new([node + 30; 16]).unwrap(), - birth: [8; 32], - schema: [9; 32], - revision: LogIndex::new(3), - applied: position(index, term), - image, + root, + configuration_digest: [10; 32], } } @@ -124,8 +133,8 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { published.digest(), Err(CommandFailure::UnsupportedCapability) ); - let mut rooted_cut = cut(1, 5, 2, Some(published)); - let flat_cut = cut(2, 5, 2, Some(image(2, 1, 7))); + let rooted_cut = cut(1, 5, 2, 2, 7); + let flat_cut = cut(2, 5, 2, 2, 8); assert_eq!( rooted_cut.covers(&flat_cut), Err(CommandFailure::CorruptState) @@ -134,10 +143,6 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { flat_cut.covers(&rooted_cut), Err(CommandFailure::CorruptState) ); - let mut later = rooted_cut.clone(); - later.applied = position(8, 3); - rooted_cut.image.as_mut().unwrap().position = position(4, 2); - assert!(later.covers(&rooted_cut).unwrap()); assert_eq!( Publication::capture(&durable, Some((&marker(), position(2, 1)))), Err(CommandFailure::CorruptState) @@ -146,70 +151,77 @@ fn publication_preserves_extent_domain_and_refuses_materialized_substitution() { #[test] fn fixed_publication_cut_covers_configuration_only_progress_and_later_writes() { - let minimum = cut(1, 5, 2, Some(image(2, 1, 7))); - let protocol_only = cut(2, 7, 3, Some(image(2, 1, 7))); - assert!(protocol_only.covers(&minimum).unwrap()); - assert!(!minimum.covers(&protocol_only).unwrap()); - let later = cut(3, 9, 3, Some(image(8, 3, 8))); + let minimum = cut(1, 5, 2, 2, 7); + let protocol = cut(2, 7, 3, 2, 7); + assert!(protocol.covers(&minimum).unwrap()); + assert!(!minimum.covers(&protocol).unwrap()); + let mut later = cut(3, 9, 3, 8, 8); + let mut value = serde_json::to_value(later.root()).unwrap(); + value["root"]["last_mutation_term"] = serde_json::json!(3); + later.root = serde_json::from_value(value).unwrap(); assert!(later.covers(&minimum).unwrap()); assert_eq!(minimum.applied(), position(5, 2)); - assert_eq!(minimum.image().unwrap().position(), position(2, 1)); - let mut restored = minimum.clone(); - restored.image.as_mut().unwrap().position = restored.applied; - assert!(protocol_only.covers(&restored).unwrap()); - assert!(restored.covers(&minimum).unwrap()); + assert_eq!(minimum.root().root().last_mutation(), Some(position(2, 1))); + assert_eq!(minimum.root().root().cells(), 2); } #[test] fn fixed_publication_cut_rejects_foreign_configuration_and_inconsistent_prefix() { - let minimum = cut(1, 5, 2, Some(image(2, 1, 7))); - let mut changed = cut(2, 7, 3, Some(image(2, 1, 7))); - changed.revision = LogIndex::new(6); - assert!(matches!( + let minimum = cut(1, 5, 2, 2, 7); + let mut changed = cut(2, 7, 3, 2, 7); + changed.configuration_digest = [99; 32]; + assert_eq!( changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - changed.revision = minimum.revision; - changed.birth = [99; 32]; - assert!(matches!( + ); + let mut changed = minimum.clone(); + let mut value = serde_json::to_value(&changed.root).unwrap(); + value["birth"] = serde_json::json!(vec![99; 32]); + changed.root = serde_json::from_value(value).unwrap(); + assert_eq!( changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - let mut foreign = minimum.clone(); - foreign.binding = FleetGroupBinding::new( + ); + changed = minimum.clone(); + changed.binding = FleetGroupBinding::new( IncarnationId::new([5; 16]).unwrap(), - &foreign.replica.scope, + &changed.replica.scope, [99; 32], ) .unwrap(); - assert!(matches!( - foreign.covers(&minimum), + assert_eq!( + changed.covers(&minimum), Err(CommandFailure::CommandConflict) - )); - for inconsistent in [ - cut(2, 5, 1, Some(image(2, 1, 7))), - cut(2, 7, 3, Some(image(2, 1, 99))), - cut(2, 7, 3, Some(image(4, 2, 8))), - ] { - assert!(matches!( + ); + for inconsistent in [cut(2, 5, 1, 2, 7), cut(2, 7, 3, 2, 99), cut(2, 7, 3, 4, 8)] { + assert_eq!( inconsistent.covers(&minimum), Err(CommandFailure::CorruptState) - )); + ); } + let mut changed = minimum.clone(); + let mut value = serde_json::to_value(&changed.root).unwrap(); + value["root"]["cells"] = serde_json::json!(3); + changed.root = serde_json::from_value(value).unwrap(); + assert_eq!(changed.covers(&minimum), Err(CommandFailure::CorruptState)); } #[test] fn canonical_empty_cut_is_distinct_from_missing_publication() { - let empty = cut(1, 5, 2, None); - assert!(cut(2, 7, 3, None).covers(&empty).unwrap()); - assert!(cut(2, 9, 3, Some(image(8, 3, 7))).covers(&empty).unwrap()); - assert!(matches!( - cut(2, 7, 3, Some(image(4, 2, 7))).covers(&empty), + let empty = cut(1, 5, 2, 0, 0); + assert_eq!(empty.root().root().cells(), 0); + assert!(cut(2, 7, 3, 0, 0).covers(&empty).unwrap()); + let mut later = cut(2, 9, 3, 8, 7); + let mut value = serde_json::to_value(&later.root).unwrap(); + value["root"]["last_mutation_term"] = serde_json::json!(3); + later.root = serde_json::from_value(value).unwrap(); + assert!(later.covers(&empty).unwrap()); + assert_eq!( + cut(2, 7, 3, 4, 7).covers(&empty), Err(CommandFailure::CorruptState) - )); - assert!(!empty.covers(&cut(2, 9, 3, Some(image(8, 3, 7)))).unwrap()); + ); + assert!(!empty.covers(&later).unwrap()); assert_eq!( - Publication::Empty, Publication::capture( &DurableApply { position: position(5, 2), @@ -217,7 +229,8 @@ fn canonical_empty_cut_is_distinct_from_missing_publication() { }, None ) - .unwrap() + .unwrap(), + Publication::Empty ); assert_ne!(Publication::Empty, Publication::Pending); } diff --git a/crates/cellar/durability/raft/host_fleet_execution.rs b/crates/cellar/durability/raft/host_fleet_execution.rs deleted file mode 100644 index e35789ffb..000000000 --- a/crates/cellar/durability/raft/host_fleet_execution.rs +++ /dev/null @@ -1,187 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! All nested work belongs to the host job set. Cancellation drops its waiter; -//! committed configuration and physical initialization retain their owners. -use super::*; -use crate::raft::fleet_catalog::FleetCatalogError; -use crate::raft::transport::fleet_actions::Delivery; -use std::time::Duration; - -pub(super) struct Completed { - request: Box, - result: Result<(), FleetCatalogError>, - _memory: Option, -} - -impl Driver { - pub(super) fn start_restart_service( - &mut self, - mut request: NodeRequest, - mut work: crate::raft::fleet_catalog::RestartServiceWork, - ) { - let result = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid( - "no managed restart service origin", - )) - .and_then(|managed| work.admit(&managed.authorization)); - if result.is_ok() { - self.jobs.push(Box::pin(async move { - work.run().await; - Job::FleetRestartServiceStopped - })); - } - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } - - pub(super) fn start_movement_service( - &mut self, - mut request: NodeRequest, - mut work: crate::raft::fleet_catalog::MovementServiceWork, - ) { - let result = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid( - "no managed movement service origin", - )) - .and_then(|managed| work.admit(&managed.authorization)); - if result.is_ok() { - self.jobs.push(Box::pin(async move { - work.run().await; - Job::FleetMovementServiceStopped - })); - } - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } - - pub(super) fn start_peer_fleet_action( - &mut self, - mut request: NodeRequest, - call: client::PeerFleetActionCall, - ) { - if call.command.target().node == self.log.limits().node.bytes() { - let origin = self - .log - .lifecycle_guard() - .ok() - .flatten() - .map(|guard| guard.boot); - if let Some(origin) = origin { - self.start_local_fleet_action( - request, - client::LocalFleetActionCall { - command: call.command, - minimum_revision: call.minimum_revision, - origin, - }, - ); - } else { - reply_node( - &mut request, - Ok(Response::FleetExecutionAction(Err( - FleetCatalogError::Closed, - ))), - ); - } - return; - } - let delivery = if self.options.managed.is_some() { - self.transport.request_fleet_action( - call.command, - call.minimum_revision, - request.control.deadline(), - ) - } else { - Err(FleetCatalogError::Invalid("no managed movement origin")) - }; - let delivery: Delivery = match delivery { - Ok(delivery) => delivery, - Err(error) => { - reply_node(&mut request, Ok(Response::FleetExecutionAction(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel movement origin without assuming rollback"; - _ = request.control.cancellation() => Err(FleetCatalogError::Unconfirmed), - result = delivery.receive() => result, - }; - Job::FleetAction(Box::new(Completed { request: Box::new(request), result, _memory: None })) - })); - } - - pub(super) fn start_local_fleet_action( - &mut self, - mut request: NodeRequest, - call: client::LocalFleetActionCall, - ) { - let setup = (|| { - call.command.validate()?; - let host = self - .options - .managed - .as_ref() - .ok_or(FleetCatalogError::Invalid("no managed receiver"))? - .authorization - .clone(); - let boot = self - .log - .lifecycle_guard() - .map_err(|_| FleetCatalogError::Closed)? - .ok_or(FleetCatalogError::Closed)? - .boot; - if &boot != call.command.target() { - return Err(FleetCatalogError::Conflict); - } - let memory = self - .control_budget - .acquire(128 * 1024) - .ok_or(FleetCatalogError::Overloaded)?; - Ok((host, memory)) - })(); - let (host, memory) = match setup { - Ok(setup) => setup, - Err(error) => { - reply_node(&mut request, Ok(Response::FleetExecutionAction(Err(error)))); - return; - } - }; - let gateway = RaftCells { - handle: self.local.clone(), - }; - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let work = async { - let remaining = request.control.deadline().checked_sub(domain.mono_ms().map_err(|_| FleetCatalogError::Closed)?) - .filter(|ms| *ms > 0).map(Duration::from_millis).ok_or(FleetCatalogError::Unconfirmed)?; - host.execute_fleet_command(call.command, call.origin, call.minimum_revision, &gateway, remaining).await - }; - let result = crate::asyncrt::select_biased! { "cancel receiver waiter while physical owners retain admitted work"; - _ = request.control.cancellation() => Err(FleetCatalogError::Unconfirmed), - result = work => result, - }; - Job::FleetAction(Box::new(Completed { request: Box::new(request), result, _memory: Some(memory) })) - })); - } - - pub(super) fn finish_fleet_action(&mut self, completed: Completed) { - let Completed { - mut request, - result, - _memory, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = if request.control.before(request.control.deadline()) { - result - } else { - Err(FleetCatalogError::Unconfirmed) - }; - reply_node(&mut request, Ok(Response::FleetExecutionAction(result))); - } -} diff --git a/crates/cellar/durability/raft/host_fleet_inspection.rs b/crates/cellar/durability/raft/host_fleet_inspection.rs deleted file mode 100644 index 0f7c2ddd3..000000000 --- a/crates/cellar/durability/raft/host_fleet_inspection.rs +++ /dev/null @@ -1,237 +0,0 @@ -//! Remote quorum metadata owns nested SDK calls through cancellation and join. -use super::*; -use crate::application::consensus::client::PeerFleetRequest; -use crate::raft::fleet_inspection_wire::{History, LocalObservation, RESPONSE_BYTES}; -use crate::raft::inspection::{InspectionError, InspectionSnapshot}; -use crate::raft::transport::fleet_inspections::Received; -use std::time::Duration; - -pub(super) struct PeerCompleted { - request: Box, - call: PeerFleetRequest, - bytes: Permit, - result: Result, -} - -pub(super) struct LocalCompleted { - request: Box, - bytes: Permit, - result: Result<(Box, LogIndex), CommandFailure>, - peer: Option, -} - -impl Driver { - pub(super) fn start_peer_fleet_inspection( - &mut self, - mut request: NodeRequest, - call: PeerFleetRequest, - ) { - if call.node == self.log.limits().node { - self.start_fleet_inspection( - request, - History::new(&call.authorization), - call.minimum_revision, - Some(call), - ); - return; - } - let admitted = (|| { - if self.options.managed.is_none() { - return Err(InspectionError::Refused); - } - let bytes = self - .control_budget - .acquire(RESPONSE_BYTES) - .ok_or(InspectionError::Overloaded)?; - let delivery = self.transport.inspect_fleet_membership( - call.node, - &call.authorization, - call.minimum_revision, - request.control.deadline(), - )?; - Ok((bytes, delivery)) - })(); - let (bytes, delivery) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply_node(&mut request, Ok(Response::PeerFleetObservation(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel remote quorum observation before delivery"; - _ = request.control.cancellation() => Err(InspectionError::Unavailable), - result = delivery.receive() => result, - }; - Job::PeerFleetInspection(Box::new(PeerCompleted { - request: Box::new(request), call, bytes, result, - })) - })); - } - - pub(super) fn finish_peer_fleet_inspection(&mut self, completed: PeerCompleted) { - let PeerCompleted { - mut request, - call, - bytes, - result, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|received| { - self.transport.validate_fleet_node_observation( - &received.observation.node, - call.minimum_revision, - )?; - let age = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(request.started)) - .ok_or(InspectionError::Unavailable)?; - received - .observation - .into_peer(&call.authorization, age) - .map(Box::new) - .map_err(|_| InspectionError::Refused) - }); - node_control::reply_node_charged( - &mut request, - Ok(Response::PeerFleetObservation(result)), - Some(bytes), - ); - } - - pub(super) fn start_local_fleet_inspection( - &mut self, - request: NodeRequest, - history: History, - minimum_revision: u64, - ) { - self.start_fleet_inspection(request, history, minimum_revision, None); - } - - fn start_fleet_inspection( - &mut self, - mut request: NodeRequest, - history: History, - minimum_revision: u64, - peer: Option, - ) { - let admitted = (|| { - let host = self - .options - .managed - .as_ref() - .ok_or(CommandFailure::UnsupportedCapability)? - .authorization - .clone(); - let scope = history - .scope() - .map_err(|_| CommandFailure::CommandConflict)?; - let cell = self - .cells - .get(&scope) - .ok_or(CommandFailure::UnsupportedCapability)?; - if cell.failed || cell.retired { - return Err(CommandFailure::Unavailable); - } - let gateway = RaftCells { - handle: self.local.clone(), - }; - let handle = gateway.cell(scope.namespace, &scope.cell)?; - if handle.scope() != &scope { - return Err(CommandFailure::CommandConflict); - } - let bytes = self - .control_budget - .acquire(RESPONSE_BYTES) - .ok_or(CommandFailure::Overloaded)?; - Ok((host, scope, handle, bytes)) - })(); - let (host, scope, handle, bytes) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply_node(&mut request, Err(error)); - return; - } - }; - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let work = async { - let remaining = || { - let now = domain.mono_ms().map_err(|_| CommandFailure::Unavailable)?; - request.control.deadline().checked_sub(now) - .filter(|ms| *ms > 0 && *ms <= 60_000) - .map(Duration::from_millis).ok_or(CommandFailure::Unavailable) - }; - let authorization = host.group_authorization_at_least(&scope, minimum_revision, remaining()?).await - .map_err(|error| match error { - crate::raft::fleet_catalog::FleetCatalogError::Overloaded - | crate::raft::fleet_catalog::FleetCatalogError::Limit => CommandFailure::Overloaded, - _ => CommandFailure::Unavailable, - })?; - if !history.matches(&authorization) { return Err(CommandFailure::CommandConflict); } - let proof = handle.observe_fleet_membership(authorization, remaining()?).await?; - let snapshot = handle.inspect_snapshot(remaining()?).await?; - if !proof.matches_snapshot(&snapshot) { return Err(CommandFailure::Unavailable); } - Ok((snapshot, proof.barrier())) - }; - let result = crate::asyncrt::select_biased! { "cancel nested quorum wait before retaining metadata"; - _ = request.control.cancellation() => Err(CommandFailure::Unavailable), - result = work => result, - }; - Job::LocalFleetInspection(Box::new(LocalCompleted { - request: Box::new(request), bytes, result, peer, - })) - })); - } - - pub(super) fn finish_local_fleet_inspection(&mut self, completed: LocalCompleted) { - let LocalCompleted { - mut request, - bytes, - result, - peer, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|(snapshot, barrier)| { - let node = self.node_observation()?; - if snapshot.view.node.identity != node.node.identity - || snapshot.view.node.storage_instance != node.node.storage_instance - { - return Err(CommandFailure::Unavailable); - } - let local = LocalObservation { - node: Box::new(node), - snapshot, - barrier, - }; - if let Some(call) = peer { - self.transport - .validate_fleet_node_observation(&local.node, call.minimum_revision) - .map_err(|_| CommandFailure::Unavailable)?; - let age = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(request.started)) - .ok_or(CommandFailure::Unavailable)?; - Ok(Response::PeerFleetObservation( - local - .into_peer(&call.authorization, age) - .map(Box::new) - .map_err(|_| InspectionError::Refused), - )) - } else { - Ok(Response::LocalFleetObservation(Box::new(local))) - } - }); - node_control::reply_node_charged(&mut request, result, Some(bytes)); - } -} diff --git a/crates/cellar/durability/raft/host_fleet_observation.rs b/crates/cellar/durability/raft/host_fleet_observation.rs index 49e5aceb8..5b2ad5d1f 100644 --- a/crates/cellar/durability/raft/host_fleet_observation.rs +++ b/crates/cellar/durability/raft/host_fleet_observation.rs @@ -1,14 +1,15 @@ -//! Fresh safe-ReadIndex membership observations, independent of SQL images. -//! Catalog and group lifecycles must still prove complete inventory separately. -use super::*; -use crate::raft::fleet_catalog::FleetGroupAuthorization; -use crate::raft::member_state::{ - FleetMembershipStatus, GroupMembership, JoiningMember, RepairStatus, -}; +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -// Includes the bounded membership ledger, its fleet status, all four voter/ -// learner sets and reply retention. Acquired before reading/cloning the ledger. -const OBSERVATION_BYTES: usize = 64 * 1024; +//! Fresh physical-group metadata. No application cell is selected or aliased. +use crate::raft::{ + cell_groups::GroupManifest, + fleet_catalog::FleetGroupAuthorization, + member_state::{FleetMembershipStatus, GroupMembership, JoiningMember, RepairStatus}, +}; +use cellar_logic::{ + cell_group::GroupHistory, + consensus::{CommandFailure, LogIndex, LogPosition, MemberId, ReplicaIdentity}, +}; /// The exact configuration at a quorum-backed membership observation. Lists /// retain joint-consensus roles; outgoing voters are never silently omitted. @@ -42,13 +43,15 @@ impl FleetMembershipConfiguration { self.auto_leave } - fn capture(conf: &::raft::prelude::ConfState) -> Result { + pub(crate) fn capture(conf: &::raft::prelude::ConfState) -> Result { let ids = |values: &[u64]| { - values + let mut ids = values .iter() .copied() .map(|id| MemberId::new(id).map_err(|_| CommandFailure::CorruptState)) - .collect::, _>>() + .collect::, _>>()?; + ids.sort_unstable(); + Ok::<_, CommandFailure>(ids) }; Ok(Self { voters: ids(&conf.voters)?, @@ -60,107 +63,90 @@ impl FleetMembershipConfiguration { } } -/// An opaque observation minted after this invocation's safe ReadIndex and -/// local durable apply. It binds the physical process, exact history and full -/// configuration. Retaining it supplies no lease, complete inventory, archive -/// coverage or permission to remove a node or release a reservation. +/// A fresh safe-ReadIndex observation of the complete durable group root and +/// configuration. Its process/history bindings grant no current health, lease, +/// placement, archive coverage, or permission to release physical debt. #[derive(Clone, Debug)] pub struct FleetMembershipObservation { - replica: ReplicaIdentity, - birth_digest: [u8; 32], - schema_digest: [u8; 32], - barrier: LogIndex, - applied: LogPosition, + replica: ReplicaIdentity, + root: GroupManifest, + barrier: LogPosition, membership: GroupMembership, configuration: FleetMembershipConfiguration, status: FleetMembershipStatus, + configuration_digest: [u8; 32], } impl FleetMembershipObservation { - pub(crate) fn matches_snapshot( - &self, - snapshot: &crate::raft::inspection::InspectionSnapshot, - ) -> bool { - snapshot.membership.as_ref() == Some(&self.membership) - && snapshot.configuration.as_ref().is_some_and(|conf| { - FleetMembershipConfiguration::capture(conf).as_ref() == Ok(&self.configuration) - }) - && snapshot - .view - .member - .as_ref() - .is_some_and(|m| m.applied.index >= self.barrier) - } - - // Only the purpose-specific authenticated quorum RPC calls this decoder. - // An ordinary inspection snapshot alone cannot establish a read barrier. - pub(crate) fn from_quorum_snapshot( - snapshot: &crate::raft::inspection::InspectionSnapshot, + // Only the authenticated quorum RPC (or the same local owner) invokes this + // constructor. Decoded status inspection cannot mint a quorum observation. + pub(crate) fn check_quorum_group( + snapshot: &crate::raft::fleet_inspection_wire::GroupSnapshot, authorization: &FleetGroupAuthorization, - barrier: LogIndex, - ) -> Result { - snapshot.validate()?; - let view = &snapshot.view; - let member = view.member.as_ref().ok_or(CommandFailure::Unavailable)?; - let membership = snapshot + ) -> Result<(), CommandFailure> { + snapshot + .validate() + .map_err(|_| CommandFailure::CorruptState)?; + let route = snapshot.root.route(); + let status = snapshot .membership - .as_ref() - .ok_or(CommandFailure::Unavailable)?; - let configuration = snapshot - .configuration - .as_ref() - .ok_or(CommandFailure::Unavailable)?; - let status = membership .fleet_status() .ok_or(CommandFailure::UnsupportedCapability)?; - if member.applied.index < barrier - || barrier.get() == 0 - || &status.binding != authorization.binding() - || !authorization.matches_scope(&view.scope) - || authorization.history_digests() != (view.birth_digest, view.schema_digest) + if &status.binding != authorization.binding() + || !authorization.matches_scope(&route.history) + || authorization.history_digests() != (route.birth_digest, route.schema_digest) + || !authorization.matches_replica(&snapshot.replica, &snapshot.membership) { return Err(CommandFailure::CommandConflict); } - let replica = ReplicaIdentity { - scope: view.scope.clone(), - member: member.member, - node: view.node.identity.node, - disk: view.node.identity.disk, - process: view.node.identity.process, - }; - if !authorization.matches_replica(&replica, membership) { - return Err(CommandFailure::CommandConflict); - } + Ok(()) + } + + pub(crate) fn from_quorum_group( + snapshot: crate::raft::fleet_inspection_wire::GroupSnapshot, + authorization: &FleetGroupAuthorization, + ) -> Result { + Self::check_quorum_group(&snapshot, authorization)?; + let status = snapshot + .membership + .fleet_status() + .ok_or(CommandFailure::UnsupportedCapability)?; + let configuration = FleetMembershipConfiguration::capture(&snapshot.configuration)?; + let configuration_digest = + group_membership_digest(&snapshot.membership, &snapshot.configuration)?; Ok(Self { - replica, - birth_digest: view.birth_digest, - schema_digest: view.schema_digest, - barrier, - applied: member.applied, - membership: membership.clone(), - configuration: FleetMembershipConfiguration::capture(configuration)?, + replica: snapshot.replica, + root: snapshot.root, + barrier: snapshot.barrier, + membership: snapshot.membership, + configuration, status, + configuration_digest, }) } - pub fn replica(&self) -> &ReplicaIdentity { + pub fn replica(&self) -> &ReplicaIdentity { &self.replica } + pub fn root(&self) -> &GroupManifest { + &self.root + } + pub fn birth_digest(&self) -> [u8; 32] { - self.birth_digest + self.root.route().birth_digest } pub fn schema_digest(&self) -> [u8; 32] { - self.schema_digest + self.root.route().schema_digest } pub fn barrier(&self) -> LogIndex { - self.barrier + self.barrier.index } pub fn applied(&self) -> LogPosition { - self.applied + self.root.position() } pub(crate) fn member_high_water(&self) -> u64 { @@ -179,8 +165,10 @@ impl FleetMembershipObservation { &self.status } - /// Only dynamically joined bindings have a committed disk here. Original - /// members retain their immutable birth identity and separate engine pins. + pub fn configuration_digest(&self) -> [u8; 32] { + self.configuration_digest + } + pub fn joined_member( &self, member: MemberId, @@ -195,111 +183,21 @@ impl FleetMembershipObservation { .repair_status() .map_err(|_| CommandFailure::CorruptState) } - - fn capture( - cell: &Cell, - authorization: &FleetGroupAuthorization, - barrier: LogIndex, - ) -> Result, CommandFailure> { - let cache = cell - .group - .read() - .map_err(|_| CommandFailure::CorruptState)?; - if cache.applied.position.index < barrier - || cell.route_revision < cache.membership.revision() - { - return Ok(None); - } - cache - .membership - .validate(&cache.conf_state, cache.applied.position.index.get()) - .and_then(|()| cache.membership.validate_scope(cell.birth.scope())) - .map_err(|_| CommandFailure::CorruptState)?; - let status = cache - .membership - .fleet_status() - .ok_or(CommandFailure::UnsupportedCapability)?; - if &status.binding != authorization.binding() - || !authorization.matches_birth(&cell.birth, cell.schema_digest) - { - return Err(CommandFailure::CommandConflict); - } - Ok(Some(Self { - replica: cell.group.identity().clone(), - birth_digest: cell.birth.digest(), - schema_digest: cell.schema_digest, - barrier, - applied: cache.applied.position, - membership: cache.membership.clone(), - configuration: FleetMembershipConfiguration::capture(&cache.conf_state)?, - status, - })) - } } -impl Driver { - pub(super) fn start_fleet_observation( - &mut self, - mut request: Request, - authorization: FleetGroupAuthorization, - ) { - let valid = self.cells.get(&request.scope).is_some_and(|cell| { - !cell.failed - && !cell.retired - && authorization.matches_scope(&request.scope) - && authorization.matches_birth(&cell.birth, cell.schema_digest) - }); - if !valid { - reply(&mut request, Err(CommandFailure::CommandConflict)); - return; - } - let Some(charge) = self.control_budget.acquire(OBSERVATION_BYTES) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let Some(id) = self.next_read_id() else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - self.reads - .insert(id, PendingRead::fleet(request, authorization, charge)); - } - - pub(super) fn finish_fleet_observation(&mut self, id: ReadId) { - use crate::raft::query::ReadOperation; - let outcome = (|| { - let read = self.reads.get(&id).ok_or(CommandFailure::CorruptState)?; - let ReadOperation::Fleet { authorization, .. } = &read.operation else { - return Err(CommandFailure::CorruptState); - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { - return Err(CommandFailure::CorruptState); - }; - let cell = self - .cells - .get(&read.request.scope) - .ok_or(CommandFailure::Unavailable)?; - FleetMembershipObservation::capture(cell, authorization, *barrier) - })(); - if matches!(outcome, Ok(None)) { - return; - } - let Some(mut read) = self.reads.remove(&id) else { - return; - }; - let ReadOperation::Fleet { charge, .. } = read.operation else { - reply(&mut read.request, Err(CommandFailure::CorruptState)); - return; - }; - let result = if read.request.control.cancelled() { - Err(CommandFailure::VersionUnavailable) - } else { - match outcome { - Ok(Some(observation)) => Ok(Response::FleetObservation(Box::new(observation))), - Err(error) => Err(error), - Ok(None) => Err(CommandFailure::CorruptState), - } - }; - reply_charged(&mut read.request, result, Some(charge)); - } +pub(crate) fn group_membership_digest( + membership: &GroupMembership, + configuration: &::raft::prelude::ConfState, +) -> Result<[u8; 32], CommandFailure> { + use sha2::{Digest, Sha256}; + let ledger = serde_json::to_vec(membership).map_err(|_| CommandFailure::CorruptState)?; + let conf = crate::raft::manifest::encode_conf(configuration) + .map_err(|_| CommandFailure::CorruptState)?; + let mut digest = Sha256::new(); + digest.update(b"cellar-fleet-group-membership-1\0"); + digest.update((ledger.len() as u64).to_be_bytes()); + digest.update(ledger); + digest.update((conf.len() as u64).to_be_bytes()); + digest.update(conf); + Ok(digest.finalize().into()) } diff --git a/crates/cellar/durability/raft/host_image_limits.rs b/crates/cellar/durability/raft/host_image_limits.rs index 3dbc17a41..23f375d74 100644 --- a/crates/cellar/durability/raft/host_image_limits.rs +++ b/crates/cellar/durability/raft/host_image_limits.rs @@ -27,9 +27,10 @@ impl RaftCellsConfig { options: super::RaftExtentOptions, ) -> anyhow::Result { anyhow::ensure!( - self.cells + self.namespaces .iter() - .all(|cell| cell.namespace.image_storage == super::RaftImageStorage::Extents), + .all(|namespace| namespace.namespace.image_storage + == super::RaftImageStorage::Extents), "extent image policy requires extent namespaces" ); self.host.extents = Some(options); diff --git a/crates/cellar/durability/raft/host_image_limits_tests.rs b/crates/cellar/durability/raft/host_image_limits_tests.rs index ddc5d3df8..65e6eb87b 100644 --- a/crates/cellar/durability/raft/host_image_limits_tests.rs +++ b/crates/cellar/durability/raft/host_image_limits_tests.rs @@ -20,7 +20,10 @@ fn config() -> RaftCellsConfig { transport: Default::default(), }, catalog: Default::default(), - cells: vec![], + namespaces: vec![], + groups: vec![], + command_origins: vec![], + shared: crate::raft::group_host::test_support::limits(), peers: vec![], query_origins: vec![], host: Default::default(), @@ -158,31 +161,18 @@ fn extent_profile_refuses_inadequate_shared_inventory() { #[test] fn extent_profile_requires_an_explicit_namespace_choice() { let mut configured = config(); - let scope = GroupScope { - cluster: ClusterId::new([1; 16]).unwrap(), - namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:policy".into(), - group: GroupId::new([4; 16]).unwrap(), - incarnation: IncarnationId::new([5; 16]).unwrap(), - }; - configured.cells.push(RaftCellRegistration { - placement: RaftCellPlacement::Replica, - namespace: NamespaceConfig::new( - scope.cluster, - scope.namespace, - cellar_logic::NamespaceMode::Raft, - [6; 32], - ), - class: "Counter".into(), - birth: BirthProposal { - scope, - voters: vec![InitialMember { - member: MemberId::new(1).unwrap(), - node: NodeId::new([1; 16]).unwrap(), - }], - learners: vec![], - }, - }); + configured + .namespaces + .push(crate::raft::group_host::RaftNamespaceConfig { + namespace: NamespaceConfig::new( + cellar_logic::consensus::ClusterId::new([1; 16]).unwrap(), + cellar_logic::consensus::NamespaceId::new([2; 16]).unwrap(), + cellar_logic::NamespaceMode::Raft, + [6; 32], + ), + class: "Counter".into(), + initialize: None, + }); assert!(configured .with_extent_image_limit(10_000_000_000, Default::default()) .is_err()); diff --git a/crates/cellar/durability/raft/host_inspection.rs b/crates/cellar/durability/raft/host_inspection.rs index 240a21d94..9a0c8d7d1 100644 --- a/crates/cellar/durability/raft/host_inspection.rs +++ b/crates/cellar/durability/raft/host_inspection.rs @@ -1,348 +1,8 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Capture bounded registered-history metadata through reserved control work. +//! Exact physical image publication observations, independent of a dispatcher. use super::*; use crate::raft::cache::CachedGroup; -use crate::raft::inspection::{ - CellInspection, InspectionError, InspectionSnapshot, MemberInspection, NodeInspection, - ReplicaOccupancy, RESPONSE_RESERVATION, -}; - -pub(super) struct Completed { - request: Box, - started: u64, - result: Result, -} - -pub(super) struct PeerCompleted { - request: Box, - started: u64, - bytes: Permit, - result: Result, -} - -impl Driver { - pub(super) fn start_peer_inspection(&mut self, mut request: Request, peer: NodeId) { - let admitted = (|| { - if self.options.node_profile.is_none() || self.directory.is_none() { - return Err(InspectionError::Refused); - } - let endpoint = self - .peers - .get(&peer) - .cloned() - .ok_or(InspectionError::Refused)?; - let binding = self - .local - .bindings - .get(&(request.scope.namespace, request.scope.cell.clone())) - .map_err(|_| InspectionError::Unavailable)?; - let birth = self - .cells - .get(&request.scope) - .map(|cell| cell.birth.digest()) - .or_else(|| { - self.standbys - .get(&request.scope) - .map(|cell| cell.birth.digest()) - }) - .ok_or(InspectionError::Refused)?; - let bytes = self - .control_budget - .acquire(RESPONSE_RESERVATION) - .ok_or(InspectionError::Overloaded)?; - let started = self - .log - .domain() - .mono_ms() - .map_err(|_| InspectionError::Unavailable)?; - let delivery = self.transport.inspect_peer( - request.scope.clone(), - binding.class.schema_digest(), - birth, - endpoint, - request.control.deadline(), - )?; - Ok((started, bytes, delivery)) - })(); - let (started, bytes, delivery) = match admitted { - Ok(admitted) => admitted, - Err(error) => { - reply(&mut request, Ok(Response::PeerInspection(Err(error)))); - return; - } - }; - self.jobs.push(Box::pin(async move { - let result = crate::asyncrt::select_biased! { "cancel an inspection before retaining its response"; - _ = request.control.cancellation() => Err(InspectionError::Unavailable), - result = delivery.receive() => result, - }; - Job::PeerInspection(Box::new(PeerCompleted { request: Box::new(request), started, bytes, result })) - })); - } - - pub(super) fn finish_peer_inspection(&mut self, completed: PeerCompleted) { - let PeerCompleted { - mut request, - started, - bytes, - result, - } = completed; - if request.control.cancelled() || self.closing.load(Ordering::Acquire) { - return; - } - let result = result.and_then(|mut received| { - let elapsed = self - .log - .domain() - .mono_ms() - .ok() - .and_then(|now| now.checked_sub(started)) - .ok_or(InspectionError::Unavailable)?; - if let Some(member) = received.snapshot.view.member.as_mut() { - member.sample_age_ms = member - .sample_age_ms - .checked_add(elapsed) - .ok_or(InspectionError::Unavailable)?; - } - if let Some(load) = received.snapshot.view.leader_load.as_mut() { - load.sample_age_ms = load - .sample_age_ms - .checked_add(elapsed) - .ok_or(InspectionError::Unavailable)?; - } - Ok(received.snapshot) - }); - reply_charged( - &mut request, - Ok(Response::PeerInspection(result)), - Some(bytes), - ); - } - - pub(super) fn start_inspection(&mut self, request: Request) { - if self.options.node_profile.is_none() { - let mut request = request; - reply(&mut request, Err(CommandFailure::UnsupportedCapability)); - return; - } - let scope = request.scope.clone(); - if self - .cells - .get(&scope) - .is_some_and(|cell| !cell.failed && !cell.retired) - { - let runtime = self.runtime.clone(); - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let (started, result) = match domain.mono_ms() { - Ok(started) => (started, runtime.status(&scope).await), - Err(_) => (0, Err(RuntimeError::Closed)), - }; - Job::Inspection(Box::new(Completed { - request: Box::new(request), - started, - result, - })) - })); - } else { - self.reply_inspection(request, None); - } - } - - pub(super) fn finish_inspection(&mut self, completed: Completed) { - let Completed { - request, - started, - result, - } = completed; - let mut request = *request; - if request.control.cancelled() { - return; - } - match result { - Ok(status) => self.reply_inspection(request, Some((started, status))), - Err(error) => reply( - &mut request, - Err(match error { - RuntimeError::Overloaded | RuntimeError::NotReady => CommandFailure::Overloaded, - _ => CommandFailure::Unavailable, - }), - ), - } - } - - fn reply_inspection(&self, mut request: Request, status: Option<(u64, GroupStatus)>) { - let Some(bytes) = self.control_budget.acquire(RESPONSE_RESERVATION) else { - reply(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let result = self - .capture_inspection(&request.scope, status) - .map(|snapshot| Response::Inspection(Box::new(snapshot))); - reply_charged(&mut request, result, Some(bytes)); - } - - fn capture_inspection( - &self, - scope: &GroupScope, - status: Option<(u64, GroupStatus)>, - ) -> Result { - let node = self.capture_node_profile()?; - let groups = node.groups; - let leader_load = self.log.domain().mono_ms().ok().and_then(|now| { - self.leader_load.as_ref()?.view( - now, - load::Population { - cells: self.cells.len(), - groups: groups as usize, - }, - self.activating.is_some(), - node.profile.leader_capacity, - ) - }); - let mut snapshot = InspectionSnapshot { - view: CellInspection { - scope: scope.clone(), - schema_digest: [0; 32], - birth_digest: [0; 32], - node, - occupancy: ReplicaOccupancy::Failed, - member: None, - leader_load, - }, - membership: None, - configuration: None, - publication: None, - }; - self.fill_inspection(scope, status, &mut snapshot)?; - Ok(snapshot) - } - - pub(super) fn capture_node_profile(&self) -> Result { - let mut profile = self - .options - .node_profile - .as_ref() - .ok_or(CommandFailure::UnsupportedCapability)? - .clone(); - profile.draining |= self.local.admission.application_paused(); - let groups = self - .log - .group_count_with_pending(self.activating.as_ref()) - .map_err(|_| CommandFailure::Unavailable)?; - let groups = u32::try_from(groups).map_err(|_| CommandFailure::Overloaded)?; - let reserved_bytes = profile - .bytes_per_group - .checked_mul(u64::from(groups)) - .ok_or(CommandFailure::Overloaded)?; - let local = self.log.limits(); - Ok(NodeInspection { - identity: NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }, - storage_instance: self - .log - .storage_instance() - .map_err(|_| CommandFailure::Unavailable)?, - profile, - groups, - reserved_bytes, - }) - } - - fn fill_inspection( - &self, - scope: &GroupScope, - status: Option<(u64, GroupStatus)>, - snapshot: &mut InspectionSnapshot, - ) -> Result<(), CommandFailure> { - if let Some(cell) = self.cells.get(scope) { - snapshot.view.schema_digest = cell.schema_digest; - snapshot.view.birth_digest = cell.birth.digest(); - let member = cell.group.identity().member; - let cache = cell.group.read().map_err(|_| CommandFailure::Unavailable)?; - if cell.failed { - return Ok(()); - } - if cell.retired - || !crate::raft::member_state::members(&cache.conf_state).contains(&member.get()) - { - snapshot.view.occupancy = ReplicaOccupancy::Retired(member); - return Ok(()); - } - let (started, status) = status.ok_or(CommandFailure::Overloaded)?; - if status.replica != *cell.group.identity() - || status.term.get() != cache.hard_state.term - || status.membership_revision != cache.membership.revision() - || status.committed.get() != cache.hard_state.commit - || status.applied != cache.applied.position - { - return Err(CommandFailure::Overloaded); - } - let published = cell.image.as_ref().map(|image| image.position); - if cell - .image - .as_ref() - .is_some_and(|image| !valid_published_position(&cache, image)) - { - return Err(CommandFailure::Overloaded); - } - let sample_age_ms = self - .log - .domain() - .mono_ms() - .map_err(|_| CommandFailure::Unavailable)? - .checked_sub(started) - .ok_or(CommandFailure::Unavailable)?; - snapshot.view.occupancy = ReplicaOccupancy::Member(member); - snapshot.view.member = Some(MemberInspection { - member, - term: status.term, - leader: status.leader.and_then(|id| MemberId::new(id).ok()), - is_leader: status.is_leader, - membership_revision: cache.membership.revision(), - member_high_water: cache.membership.high_water(), - committed: status.committed, - applied: status.applied, - published, - repair: cache - .membership - .repair_status() - .map_err(|_| CommandFailure::CorruptState)?, - sample_age_ms, - }); - snapshot.membership = Some(cache.membership.clone()); - snapshot.configuration = Some(cache.conf_state.clone()); - snapshot.publication = Some(Publication::capture( - &cache.applied, - cell.image - .as_ref() - .map(|image| (&image.image, image.position)), - )?); - return Ok(()); - } - if let Some(dormant) = self.standbys.get(scope) { - snapshot.view.schema_digest = dormant.class.schema_digest(); - snapshot.view.birth_digest = dormant.birth.digest(); - snapshot.view.occupancy = if dormant.failed { - ReplicaOccupancy::Failed - } else if self.activating.as_ref() == Some(scope) { - ReplicaOccupancy::Initializing - } else { - match self.log.group(scope) { - Ok(group) => ReplicaOccupancy::Occupied(group.identity().member), - Err(super::super::storage::LogError::NotFound) => ReplicaOccupancy::Standby, - Err(_) => return Err(CommandFailure::Unavailable), - } - }; - return Ok(()); - } - Err(CommandFailure::UnsupportedCapability) - } -} fn valid_published_position(cache: &CachedGroup, image: &ImageView) -> bool { let position = image.position; diff --git a/crates/cellar/durability/raft/host_leader_load.rs b/crates/cellar/durability/raft/host_leader_load.rs index 706dc0677..cbb83f90b 100644 --- a/crates/cellar/durability/raft/host_leader_load.rs +++ b/crates/cellar/durability/raft/host_leader_load.rs @@ -3,9 +3,8 @@ //! Node-wide leadership observations, collected in bounded owned batches. //! A count is published only after every attached history was sampled. use super::*; -use crate::application::consensus::{RaftGroupResources, RaftReplicaRole}; +use crate::application::consensus::RaftGroupResources; use crate::raft::inspection::LeaderLoad; -use std::ops::Bound::{Excluded, Unbounded}; /// Metadata only: retaining an SDK handle cannot retain a group or engine. #[derive(Default)] @@ -284,157 +283,6 @@ impl Scanner { } } -impl Driver { - pub(super) fn pump_leader_load(&mut self) { - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Ok(groups) = self.log.group_count() else { - return; - }; - let population = Population { - cells: self.cells.len(), - groups, - }; - let Some(scanner) = self.leader_load.as_mut() else { - return; - }; - let begins = scanner.begin(now, population, self.activating.is_some()); - if let Ok(mut source) = self.local.groups.0.lock() { - *source = scanner - .cached - .as_ref() - .filter(|cached| cached.population == population && self.activating.is_none()) - .map(|cached| (cached.started, cached.resources)); - } - if !begins { - return; - } - let Some(cycle) = scanner.cycle.as_ref() else { - return; - }; - let after = cycle.cursor.as_ref().map_or(Unbounded, Excluded); - let batch: Vec<_> = self - .cells - .range((after, Unbounded)) - .take(scanner.options.groups_per_batch) - .map(|(_, cell)| cell.group.identity().clone()) - .collect(); - if batch.is_empty() { - scanner.failed(now); - return; - } - let sequence = cycle.sequence; - let deadline = cycle.deadline; - scanner.busy = true; - let runtime = self.runtime.clone(); - let domain = self.log.domain(); - self.jobs.push(Box::pin(async move { - let observed = batch.into_iter().map(|identity| { - let runtime = runtime.clone(); - let domain = domain.clone(); - async move { - let result = match domain.sleep_until(deadline) { - Ok(timer) => crate::asyncrt::select_biased! { "the original complete-scan deadline precedes a late status"; - _ = timer => Err(RuntimeError::NotReady), - result = runtime.status(&identity.scope) => result, - }, - Err(_) => Err(RuntimeError::Closed), - }; - (identity, result) - } - }); - Job::LeaderLoad(Box::new(Completed { sequence, result: futures_util::future::join_all(observed).await })) - })); - } - - pub(super) fn finish_leader_load(&mut self, completed: Completed) { - let Ok(now) = self.log.domain().mono_ms() else { - return; - }; - let Ok(groups) = self.log.group_count() else { - return; - }; - let population = Population { - cells: self.cells.len(), - groups, - }; - if let Some(scanner) = self.leader_load.as_mut() { - scanner.accept( - completed, - now, - population, - self.activating.is_some(), - |status| { - let cell = self.cells.get(&status.replica.scope)?; - if cell.group.identity() != &status.replica { - return None; - } - let registered = self - .local - .bindings - .get(&( - status.replica.scope.namespace, - status.replica.scope.cell.clone(), - )) - .ok()? - .class - .has_queries(); - let (role, queries) = readiness::capture( - cell, - status, - now, - self.options.stale_source_timeout_ms, - self.local.admission.is_draining(), - registered, - ) - .ok()?; - let apply_lag = status - .committed - .get() - .checked_sub(status.applied.index.get())?; - let archive_lag = status.committed.get().saturating_sub(cell.archived.get()); - Some(RaftGroupResources { - leaders: usize::from(role == RaftReplicaRole::Leader), - voters: usize::from(role == RaftReplicaRole::Voter), - learners: usize::from(role == RaftReplicaRole::Learner), - removed: usize::from(role == RaftReplicaRole::Removed), - unknown: usize::from(role == RaftReplicaRole::Unknown), - coherent_images: usize::from(queries.coherent_image), - historical_queries: usize::from(queries.historical), - recent_source_queries: usize::from(queries.recent_source), - failed: usize::from(cell.failed), - retired: usize::from(cell.retired), - sql_busy: usize::from(cell.sql_busy), - pending_commands: usize::from(cell.pending.is_some()), - incoming_seeds: usize::from(cell.incoming.is_some()), - applying: usize::from( - cell.apply.is_some() - || cell.publication.is_some() - || cell.apply_inflight, - ), - archive_pending: usize::from(archive_lag > 0), - archive_current: usize::from(cell.archived.get() > 0 && archive_lag == 0), - archive_unconfirmed: usize::from(cell.archived.get() == 0), - apply_lag_entries: apply_lag, - max_apply_lag_entries: apply_lag, - archive_lag_entries: archive_lag, - max_archive_lag_entries: archive_lag, - ..Default::default() - }) - }, - ); - if let Ok(mut source) = self.local.groups.0.lock() { - *source = scanner - .cached - .as_ref() - .filter(|cached| cached.population == population && self.activating.is_none()) - .map(|cached| (cached.started, cached.resources)); - } - } - } -} - #[cfg(test)] #[path = "host_leader_load_tests.rs"] mod tests; diff --git a/crates/cellar/durability/raft/host_node.rs b/crates/cellar/durability/raft/host_node.rs index 6902989d8..314fd7c53 100644 --- a/crates/cellar/durability/raft/host_node.rs +++ b/crates/cellar/durability/raft/host_node.rs @@ -50,7 +50,10 @@ pub struct RaftHostObservation { pub fleet_trust_revision: Option, pub declared_namespaces: u32, pub registered_histories: u32, + /// Registered groups unavailable for application work, including quarantine. pub dormant_histories: u32, + /// Pending physical attachment/provisioning operations. This is independent + /// of durable engine occupancy and is not limited to one business cell. pub pending_history_registrations: u32, /// Physical engine slots not attached as running cells, including retained /// or retired histories. This count never claims inventory completeness. @@ -72,191 +75,3 @@ pub struct RaftPeerNodeObservation { /// residence and retention by an unpolled caller. pub sample_age_ms: u64, } - -impl Driver { - pub(super) fn on_control(&mut self, request: ControlRequest) { - let mut request = match request { - ControlRequest::Cell(request) => { - self.on_request(*request); - return; - } - ControlRequest::Node(request) => *request, - }; - if matches!(&request.kind, NodeKind::Drain) { - self.start_node_drain(request); - return; - } - if self.closing.load(Ordering::Acquire) || request.control.cancelled() { - reply_node(&mut request, Err(CommandFailure::Unavailable)); - return; - } - match std::mem::replace(&mut request.kind, NodeKind::ControllerStatus) { - NodeKind::StartRestartService(work) => self.start_restart_service(request, *work), - NodeKind::StartMovementService(work) => self.start_movement_service(request, *work), - NodeKind::PeerFleetAction(call) => self.start_peer_fleet_action(request, *call), - NodeKind::LocalFleetAction(call) => self.start_local_fleet_action(request, *call), - NodeKind::PeerFleetStop(call) => self.start_peer_fleet_stop(request, *call), - NodeKind::InspectPeerFleet(call) => self.start_peer_fleet_inspection(request, *call), - NodeKind::ObserveFleet { - history, - minimum_revision, - } => self.start_local_fleet_inspection(request, *history, minimum_revision), - NodeKind::InspectPeer { - node, - minimum_revision, - physical, - } => self.start_node_inspection(request, node, minimum_revision, physical), - NodeKind::RegisterHistory(authorization) => { - self.start_history_registration(request, *authorization) - } - NodeKind::Observe => { - let result = self.node_observation().map(|o| Response::Node(Box::new(o))); - reply_node(&mut request, result); - } - NodeKind::ObservePhysical => { - let Some(bytes) = self - .control_budget - .acquire(super::super::physical_inventory::RESPONSE_BYTES) - else { - reply_node(&mut request, Err(CommandFailure::Overloaded)); - return; - }; - let result = self - .physical_node_observation() - .map(|(node, inventory)| Response::PhysicalNode(Box::new((node, inventory)))); - reply_node_charged(&mut request, result, Some(bytes)); - } - NodeKind::ControllerStatus => self.controller_control(request, None), - NodeKind::SetControllerEnabled(enabled) => { - self.controller_control(request, Some(enabled)) - } - NodeKind::Drain => reply_node(&mut request, Err(CommandFailure::CorruptState)), - } - } - - pub(super) fn node_observation(&self) -> Result { - let node = self.capture_node_profile()?; - let physical = self - .log - .group_count() - .map_err(|_| CommandFailure::Unavailable)?; - let unattached = physical - .checked_sub(self.cells.len()) - .ok_or(CommandFailure::CorruptState)?; - let bounded = |n| u32::try_from(n).map_err(|_| CommandFailure::Overloaded); - let dormant = self - .standbys - .keys() - .filter(|scope| !self.cells.contains_key(*scope)) - .count(); - let current_enrollment = self - .directory - .as_ref() - .and_then(|d| d.authorization(node.identity.node)) - .map(RaftHostEnrollmentObservation::capture); - Ok(RaftHostObservation { - node, - fleet_boot: self - .log - .lifecycle_guard() - .map_err(|_| CommandFailure::Unavailable)? - .map(|guard| guard.boot), - startup_bounds: self.startup_bounds()?, - startup_enrollment: self - .options - .managed - .as_ref() - .map(|managed| RaftHostEnrollmentObservation::capture(&managed.authorization)), - current_enrollment, - fleet_trust_revision: self - .directory - .as_ref() - .and_then(directory::Directory::rooted_revision), - declared_namespaces: bounded(self.declared_namespaces.len())?, - registered_histories: bounded( - self.cells - .len() - .checked_add(dormant) - .ok_or(CommandFailure::Overloaded)?, - )?, - dormant_histories: bounded(dormant)?, - pending_history_registrations: u32::from(self.history_registration.is_some()), - unattached_engine_histories: bounded(unattached)?, - seed_workers: bounded( - self.seed_archive_work - .usage() - .1 - .checked_add( - self.transport - .resource_observer() - .capture() - .ok_or(CommandFailure::Unavailable)? - .snapshot_source - .items, - ) - .ok_or(CommandFailure::Overloaded)?, - )?, - }) - } - - pub(super) fn physical_node_observation( - &self, - ) -> Result { - let inventory = self - .log - .physical_inventory(self.activating.as_ref()) - .map_err(|_| CommandFailure::Unavailable)?; - let node = self.node_observation()?; - // Engine sync can insert a group between the two lock acquisitions. - // Refuse a mixed cut rather than silently omitting an occupied slot. - inventory - .validate(&node) - .map_err(|_| CommandFailure::Overloaded)?; - Ok((node, inventory)) - } - - fn startup_bounds(&self) -> Result { - let runtime = self.runtime.limits(); - let log = self.log.limits(); - let buffers = [ - self.options.max_pending_bytes, - self.options.max_control_bytes, - runtime.max_request_bytes, - runtime.max_event_bytes, - runtime.max_ready_bytes, - runtime.max_apply_bytes, - log.max_pending_bytes, - log.max_cache_bytes, - ]; - let bounded_buffer_bytes = buffers - .into_iter() - .try_fold(0u64, |sum, bytes| { - sum.checked_add(u64::try_from(bytes).ok()?) - }) - .ok_or(CommandFailure::CorruptState)?; - let count = |n| u32::try_from(n).map_err(|_| CommandFailure::CorruptState); - Ok(RaftHostStartupBounds { - host_groups: count(self.options.max_groups)?, - consensus_groups: count(runtime.max_groups)?, - engine_groups: count(log.max_groups)?, - bounded_buffer_bytes, - }) - } -} - -pub(super) fn reply_node(request: &mut NodeRequest, result: Result) { - reply_node_charged(request, result, None); -} - -pub(super) fn reply_node_charged( - request: &mut NodeRequest, - result: Result, - bytes: Option, -) { - if let Some(reply) = request.reply.take() { - let _ = reply.send(result.map(|response| client::ChargedResponse { - response, - _bytes: bytes, - })); - } -} diff --git a/crates/cellar/durability/raft/host_readiness.rs b/crates/cellar/durability/raft/host_readiness.rs index 26f69303e..b5c94720e 100644 --- a/crates/cellar/durability/raft/host_readiness.rs +++ b/crates/cellar/durability/raft/host_readiness.rs @@ -2,7 +2,7 @@ //! Conservative local observations, separate from per-invocation proof gates. use super::*; -use crate::application::consensus::{RaftQueryReadiness, RaftReplicaRole}; +use crate::application::consensus::RaftReplicaRole; fn role(cache: &super::super::cache::CachedGroup, status: &GroupStatus) -> RaftReplicaRole { if cache.membership.revision() != status.membership_revision @@ -38,45 +38,3 @@ fn coherent_image( cache.coherent_image_position(&image.image, image.position) == Some(applied) }) } - -pub(super) fn capture( - cell: &Cell, - status: &GroupStatus, - now: u64, - source_timeout_ms: u64, - draining: bool, - registered_queries: bool, -) -> Result<(RaftReplicaRole, RaftQueryReadiness), RuntimeError> { - let cache = cell.group.read()?; - let role = role(&cache, status); - let coherent_image = coherent_image(&cache, cell.image.as_ref(), status.applied); - let historical = registered_queries - && coherent_image - && !matches!(role, RaftReplicaRole::Removed | RaftReplicaRole::Unknown) - && !draining - && !cell.failed - && !cell.retired - && cell.incoming.is_none() - && !cell.sql_busy - && cell.apply.is_none() - && cell.publication.is_none(); - let source_age_ms = status.source_observed_ms.and_then(|at| now.checked_sub(at)); - let recent_source = historical - && super::super::query::recent_source_deadline( - status.source_observed_ms, - now, - source_timeout_ms, - ) - .is_some(); - Ok(( - role, - RaftQueryReadiness { - registered_queries, - coherent_image, - historical, - recent_source, - source_age_ms, - source_timeout_ms, - }, - )) -} diff --git a/crates/cellar/durability/raft/host_seed.rs b/crates/cellar/durability/raft/host_seed.rs index 779ce01d4..2615b2c29 100644 --- a/crates/cellar/durability/raft/host_seed.rs +++ b/crates/cellar/durability/raft/host_seed.rs @@ -4,7 +4,6 @@ //! archive is never a substitute for this request's live-quorum authorization. use super::*; -use crate::raft::query::ReadOperation; use crate::raft::transport::seeds::SourceSeed; pub(super) fn valid_target( @@ -102,11 +101,8 @@ impl Driver { let Some(mut read) = self.reads.remove(&id) else { return; }; - let ReadOperation::Seed { target, .. } = &read.operation else { - reply(&mut read.request, Err(CommandFailure::CorruptState)); - return; - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.clone() else { + let target = &read.target; + let Some(barrier) = read.minimum else { reply(&mut read.request, Err(CommandFailure::CorruptState)); return; }; @@ -170,15 +166,13 @@ impl Driver { return; }; let scope = read.request.scope.clone(); - let ReadOperation::Seed { target, kind, .. } = &read.operation else { - self.finish_read(&id, Err(CommandFailure::CorruptState)); - return; - }; - let Some(Minimum::ReadIndex(barrier)) = read.minimum.as_ref() else { + let target = &read.target; + let kind = read.kind; + let Some(barrier) = read.minimum.as_ref() else { self.finish_read(&id, Err(CommandFailure::CorruptState)); return; }; - if *kind != crate::raft::transport::seeds::SeedKind::S3 + if kind != crate::raft::transport::seeds::SeedKind::S3 || read.expired() || self .cells diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs index d82633ce5..c8c7d8a9f 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/admission.rs @@ -54,6 +54,10 @@ impl Drop for CellLease { } impl Admission { + pub fn limits(&self) -> CellSqlLimits { + self.limits + } + pub fn new(limits: CellSqlLimits, lease: Arc) -> Arc { Arc::new(Self { lease, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs index 78bc870c3..ed9f7bc26 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/mod.rs @@ -7,6 +7,7 @@ mod apply; pub(in crate::raft::host_sql) use apply::ApplyRequest; pub use apply::CellGroupPreparedApply; mod query; +pub(crate) use query::ReadRetention; mod worker; pub use query::{CellSqlEffectScan, CellSqlQueryResult, CellSqlReceiptLookup}; @@ -35,7 +36,7 @@ use std::sync::{ use tokio::sync::{oneshot, Notify}; pub(super) use worker::{Cells, Request}; -#[derive(Clone, Copy)] +#[derive(Clone, Copy, Debug, serde::Serialize)] pub struct CellSqlLimits { pub workers: usize, pub queue_per_worker: usize, @@ -330,7 +331,10 @@ pub enum CellSqlReconciliation { Refused(CommandFailure), } +#[derive(Clone, Copy, Debug, serde::Serialize)] pub struct CellSqlUsage { + pub limits: CellSqlLimits, + pub closing: bool, /// Independent progress budget; not included in SQL producer retention. pub apply_retained_bytes: usize, pub apply_retained_operations: usize, @@ -459,6 +463,8 @@ impl CellGroupSqlPool { let resources = self.inner.resources.capture().ok_or_else(unavailable)?; let (apply_retained_bytes, apply_retained_operations) = self.inner.apply_admission.usage(); Ok(CellSqlUsage { + limits: self.inner.admission.limits(), + closing: self.inner.closing.load(Ordering::Acquire), apply_retained_bytes, apply_retained_operations, active_cells, diff --git a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs index c84e6af12..e9598bf78 100644 --- a/crates/cellar/durability/raft/host_sql/cell_groups/query.rs +++ b/crates/cellar/durability/raft/host_sql/cell_groups/query.rs @@ -30,7 +30,23 @@ enum Callback { }, } +/// Keeps upstream admission alive on the physical reader after async timeout. +/// The SQL owner does not interpret or release that admission early. +pub(crate) struct ReadRetention { + _owner: Arc, +} + +impl ReadRetention { + pub(crate) fn new(owner: Arc) -> Self { + Self { _owner: owner } + } +} + pub(super) struct Query { + _retention: Option, + // SQL can outlive both its response deadline and the controller's reply. + // Node drain must still observe the physical callback until it exits. + _work: crate::host_services::ApplicationWorkGuard, options: CellSqlOptions, authority: CellReadAuthority, callback: Callback, @@ -57,11 +73,18 @@ impl CellSqlEffectScan { pub struct CellSqlQueryResult { result: Vec, + replica: cellar_logic::consensus::ReplicaIdentity, observation: crate::raft::version::CellVersionObservation, _bytes: Arc, } impl CellSqlQueryResult { + pub fn replica( + &self, + ) -> &cellar_logic::consensus::ReplicaIdentity { + &self.replica + } + pub fn result(&self) -> &[u8] { &self.result } @@ -195,12 +218,25 @@ impl CellGroupSqlPool { /// image and version witness. The existing query owner enforces the original /// deadline, cancellation, read-only SQL, output budget and joined cleanup. pub async fn query_registered( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + class: Arc, + contract: crate::application::consensus::OperationContract, + input: Vec, + ) -> Result { + self.query_registered_retained(options, authority, class, contract, input, None) + .await + } + + pub(crate) async fn query_registered_retained( &self, options: CellSqlOptions, authority: CellReadAuthority, class: Arc, mut contract: crate::application::consensus::OperationContract, input: Vec, + retention: Option, ) -> Result { if contract.operation.len() > 128 { return Err(ExecutionError::Registration( @@ -212,7 +248,7 @@ impl CellGroupSqlPool { return Err(overloaded()); } match self - .query_callback( + .query_callback_retained( options, authority, Callback::Registered { @@ -220,6 +256,7 @@ impl CellGroupSqlPool { contract, input: input.into_boxed_slice(), }, + retention, ) .await? { @@ -272,6 +309,17 @@ impl CellGroupSqlPool { options: CellSqlOptions, authority: CellReadAuthority, callback: Callback, + ) -> Result { + self.query_callback_retained(options, authority, callback, None) + .await + } + + async fn query_callback_retained( + &self, + options: CellSqlOptions, + authority: CellReadAuthority, + callback: Callback, + retention: Option, ) -> Result { self.admit()?; let domain = &self.inner.workers[0].domain; @@ -301,7 +349,17 @@ impl CellGroupSqlPool { let deadline = authority.deadline(); let cancelled = Arc::new(AtomicBool::new(false)); let _cancel = Cancel(cancelled.clone()); + let work = domain + .run(async { + crate::asyncrt::services() + .application_work_registry() + .enter() + }) + .await + .map_err(|_| unavailable())?; let query = Query { + _retention: retention, + _work: work, options, authority, callback, @@ -480,6 +538,7 @@ pub(super) fn run( return Err(overloaded()); } drop(database); + let replica = query.authority.replica().clone(); drop(query.authority); Bytes::shrink(&mut query.bytes, result.retained_bytes() + 4096)?; Ok(match result { @@ -489,6 +548,7 @@ pub(super) fn run( }), Output::Query(result) => CellSqlOutcome::Queried(CellSqlQueryResult { result, + replica, observation, _bytes: query.bytes, }), diff --git a/crates/cellar/durability/raft/host_startup_tests.rs b/crates/cellar/durability/raft/host_startup_tests.rs index 9a6d95dae..eb378b9ed 100644 --- a/crates/cellar/durability/raft/host_startup_tests.rs +++ b/crates/cellar/durability/raft/host_startup_tests.rs @@ -2,8 +2,14 @@ //! A startup error owns infrastructure until actual file-lock release. #![allow(clippy::disallowed_methods)] -use super::storage_fault as fault; use super::*; +use crate::raft::storage_fault as fault; +use crate::raft::{ + catalog::{Catalog, CatalogOptions, InitialMember, NamespaceConfig}, + completion::RuntimeError, + transport::{GroupRoute as PhysicalRoute, RaftPeer, RaftTransport}, + transport_options::RaftPeerNode, +}; use crate::raft::{ options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, storage::ReplicaLog, @@ -11,6 +17,12 @@ use crate::raft::{ version::{VersionKeys, VersionSigningKey}, }; use anyhow::Context as _; +use cellar_logic::{ + cell_group::{CellId, CellRange, GroupHistory, RouteEpoch}, + consensus::*, +}; +use std::collections::BTreeMap; +use std::sync::atomic::Ordering; use std::time::Duration; #[tokio::test(flavor = "multi_thread", worker_threads = 4)] @@ -32,7 +44,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow )) .await??; let (runtime, events) = - RaftRuntime::start_parked(log.clone(), RaftRuntimeOptions::default()).await?; + CellGroupRuntime::start(log.clone(), RaftRuntimeOptions::default()).await?; let tls = RaftTls::new( vec![include_bytes!("../../transport/tcp/fixtures/ca.der") .to_vec() @@ -45,7 +57,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow ) .into(), )?; - let transport = RaftTransport::start( + let transport = RaftTransport::start_shared( runtime.clone(), Arc::new(crate::peer_auth::PeerAuth::new([8; 32], "startup")?), tls, @@ -64,15 +76,16 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow let release = gate.arm(fault::HOLD_LOCK_DROP); let mut startup = Box::pin(domain.run(start( service, - Registrations { - cells: vec![], + Prepared { namespaces: vec![], + groups: vec![], peers: vec![], - origins: vec![], + query_origins: vec![], + command_origins: vec![], }, root.path().join("cells"), - "startup".into(), - RaftHostOptions::default(), + test_support::host_options(), + test_support::limits(), VersionKeys::new( VersionSigningKey::new([31; 32], u64::MAX)?, vec![], @@ -102,7 +115,7 @@ async fn failed_preflight_joins_infrastructure_with_retained_handles() -> anyhow }?; let admission_closed = matches!( log.storage_instance(), - Err(super::super::storage::LogError::Closed) + Err(crate::raft::storage::LogError::Closed) ); if !returned_early { // This is deliberately before cleanup through the retained handles. @@ -144,8 +157,8 @@ struct StartupNode { root: tempfile::TempDir, domain: crate::asyncrt::ExecutionDomain, log: ReplicaLog, - runtime: RaftRuntime, - events: Option, + runtime: CellGroupRuntime, + events: Option, transport: RaftTransport, } @@ -165,7 +178,7 @@ impl StartupNode { Arc::new(cellar_ltx::DirectFileSystem), )) .await??; - let (runtime, events) = RaftRuntime::start_parked( + let (runtime, events) = CellGroupRuntime::start( log.clone(), RaftRuntimeOptions { max_requests: 1, @@ -187,7 +200,7 @@ impl StartupNode { ) .into(), )?; - let transport = RaftTransport::start( + let transport = RaftTransport::start_shared( runtime.clone(), Arc::new(crate::peer_auth::PeerAuth::new( [8; 32], @@ -275,65 +288,141 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() class.schema_digest(), )) .await?; + let directory = RangeDirectory::new(namespace.clone(), DirectoryLimits::default())?; + let key = crate::application::consensus::cell_routing_key( + namespace.config().namespace, + &CellId::new("Counter:startup-boundary").map_err(|_| anyhow::anyhow!("cell id"))?, + ); + let (a, b) = CellRange::all() + .split(key) + .map_err(|_| anyhow::anyhow!("ranges"))?; + let mut requests = Vec::new(); + for (id, range) in [(1, a), (2, b)] { + let request = directory.prepare_group_birth(CellGroupBirthSpec { + history: GroupHistory { + cluster: namespace.config().cluster, + namespace: namespace.config().namespace, + group: GroupId::new([id; 16])?, + incarnation: IncarnationId::new([9; 16])?, + }, + range, + epoch: RouteEpoch::new(1).map_err(|_| anyhow::anyhow!("epoch"))?, + creation: ProposalId::new([id; 16])?, + voters: (1..=3) + .map(|member| { + Ok(InitialMember { + member: MemberId::new(member)?, + node: NodeId::new([member as u8; 16])?, + }) + }) + .collect::>()?, + learners: vec![], + })?; + local + .domain + .run(directory.reserve_group_birth(&request)) + .await??; + requests.push(request); + } + let initialization = directory.prepare_initialization( + ProposalId::new([33; 16])?, + &requests + .iter() + .map(|r| r.route().clone()) + .collect::>(), + )?; + local + .domain + .run(directory.publish(&initialization)) + .await??; let mut births = Vec::new(); - for id in 1..=2 { + for request in requests { births.push( - namespace - .resolve_birth(BirthProposal { - scope: GroupScope { - cluster: namespace.config().cluster, - namespace: namespace.config().namespace, - cell: format!("Counter:startup-{id}"), - group: GroupId::new([id; 16])?, - incarnation: IncarnationId::new([9; 16])?, - }, - voters: (1..=3) - .map(|member| { - Ok(InitialMember { - member: MemberId::new(member)?, - node: NodeId::new([member as u8; 16])?, - }) - }) - .collect::>()?, - learners: vec![], - }) - .await?, + local + .domain + .run(directory.materialize_selected_history(request.route().history)) + .await??, ); } - let first = births[0].scope().clone(); - let second = births[1].scope().clone(); - namespace - .bootstrap_member(&births[0], &remote.log) - .await? - .attach(&remote.runtime) - .await - .context("attach remote startup member")?; - remote.transport.install_group(membership::route( - &remote.log.group(&first)?, - &births[0], - &peers, - class.schema_digest(), - "node-2", - &remote.transport, - )?)?; + let first = births[0].route().history; + let second = births[1].route().history; + let assignment = |birth: &CellGroupBirth| -> anyhow::Result { + Ok(Assignment { + namespace: 0, + placement: RaftGroupPlacement::Replica, + birth: birth.clone(), + transport: PhysicalRoute { + scope: birth.route().history, + schema_digest: birth.route().schema_digest, + revision: birth.route().epoch.get(), + peers: birth + .voters() + .iter() + .map(|member| { + let peer = &peers[&member.node]; + RaftPeer { + replica: ReplicaIdentity { + scope: birth.route().history, + member: member.member, + node: peer.node, + disk: peer.disk, + process: peer.process, + }, + label: peer.label.clone(), + server_name: peer.server_name.clone(), + control_address: peer.control_address, + data_address: peer.data_address, + query_address: peer.query_address, + } + }) + .collect(), + }, + }) + }; + let mut remote = remote; + let remote_prepared = Prepared { + namespaces: vec![Namespace { + directory: directory.clone(), + class: class.clone(), + }], + groups: vec![assignment(&births[0])?], + peers: peers.values().cloned().collect(), + query_origins: vec![], + command_origins: vec![], + }; + let remote_service = remote.service(); + let remote_host = remote + .domain + .run(start( + remote_service, + remote_prepared, + remote.root.path().join("cells"), + test_support::host_options(), + test_support::limits(), + VersionKeys::new( + VersionSigningKey::new([31; 32], u64::MAX)?, + vec![], + Duration::from_secs(86400), + )?, + )) + .await??; // Hold the second local member's permanent claim after the first member // has attached. Its real peer keeps sending while no host drains events. store.hold_nth_claim.store(2, Ordering::Release); store.claim_hang.store(true, Ordering::Release); let release = ReleaseClaim(store.clone()); - let registrations = Registrations { - cells: births - .into_iter() - .map(|birth| Registration { - placement: RaftCellPlacement::Replica, - namespace: namespace.clone(), - birth, - class: class.clone(), - }) - .collect(), + let registrations = Prepared { + namespaces: vec![Namespace { + directory: directory.clone(), + class: class.clone(), + }], + groups: births + .iter() + .map(assignment) + .collect::>()?, peers: peers.into_values().collect(), - namespaces: vec![], - origins: vec![], + query_origins: vec![], + command_origins: vec![], }; let service = local.service(); let directory = local.root.path().join("cells"); @@ -344,8 +433,8 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() service, registrations, directory, - "node-1".into(), - RaftHostOptions::default(), + test_support::host_options(), + test_support::limits(), VersionKeys::new( VersionSigningKey::new([31; 32], u64::MAX)?, vec![], @@ -368,7 +457,10 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() responses.push( tokio::time::timeout( Duration::from_secs(3), - remote.transport.submit(&first, message)?.accepted(), + remote + .transport + .submit_physical(&first, message)? + .accepted(), ) .await?, ); @@ -390,13 +482,13 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() .as_ref() .map_err(|error| anyhow::anyhow!("startup failed: {error:#}"))?; for scope in [&first, &second] { - anyhow::ensure!(host.cell(scope.namespace, &scope.cell).is_ok()); + anyhow::ensure!(host.group(*scope).is_ok()); // The active host's background inspection shares this fixture's // single request slot. Read-only overload is temporary contention, // not evidence that startup failed to attach the group. tokio::time::timeout(Duration::from_secs(3), async { loop { - match local.runtime.status(scope).await { + match local.runtime.status(*scope).await { Err(RuntimeError::Overloaded) => { tokio::time::sleep(Duration::from_millis(10)).await; } @@ -405,8 +497,8 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() } }) .await - .with_context(|| format!("startup status deadline for {}", scope.cell))? - .with_context(|| format!("inspect attached startup group {}", scope.cell))?; + .with_context(|| format!("startup status deadline for {:?}", scope.group))? + .with_context(|| format!("inspect attached startup group {:?}", scope.group))?; } let mut message = Message { from: 2, @@ -419,7 +511,7 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() // The active host's one-item runtime queue also serves inspection. // Retry only explicit refusal and still require successful admission. loop { - let result = match remote.transport.submit(&first, message.clone()) { + let result = match remote.transport.submit_physical(&first, message.clone()) { Ok(delivery) => delivery.accepted().await, Err(error) => Err(error), }; @@ -438,6 +530,7 @@ async fn peer_traffic_waits_until_all_local_groups_attach() -> anyhow::Result<() if let Ok(host) = started { host.close().await?; } + remote_host.close().await?; remote.close().await?; third.close().await?; local.close().await?; diff --git a/crates/cellar/durability/raft/inspection.rs b/crates/cellar/durability/raft/inspection.rs index cdbf0b778..7bead86ff 100644 --- a/crates/cellar/durability/raft/inspection.rs +++ b/crates/cellar/durability/raft/inspection.rs @@ -133,6 +133,19 @@ pub struct CellInspection { pub leader_load: Option, } +/// Progress captured with one physical group's complete quorum-selected root. +/// This never substitutes a business cell identity or grants leader authority. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GroupInspection { + pub history: cellar_logic::cell_group::GroupHistory, + pub schema_digest: [u8; 32], + pub birth_digest: [u8; 32], + pub node: NodeInspection, + pub member: Option, + /// None means no complete node-wide leadership scan was captured. + pub leader_load: Option, +} + // The transport uses the validated ledger/configuration pair to encode a strict // bounded response. The public SDK returns only the observation above. #[derive(Clone, Debug)] diff --git a/crates/cellar/durability/raft/inspection_transport.rs b/crates/cellar/durability/raft/inspection_transport.rs index 8e42e3f8d..e4f5f9597 100644 --- a/crates/cellar/durability/raft/inspection_transport.rs +++ b/crates/cellar/durability/raft/inspection_transport.rs @@ -4,7 +4,6 @@ //! transport task registry; dropping a caller cancels its read-only RPC. use super::snapshots::{read_bulk_head, Wire}; use super::*; -use crate::application::consensus::{CommandFailure, RaftCells}; use crate::raft::inspection::{InspectionError, InspectionSnapshot}; use crate::raft::inspection_identity::{InspectionRoute, PhysicalNode}; use crate::raft::inspection_wire as body; @@ -440,158 +439,6 @@ impl State { validate_response(route, &snapshot)?; Ok(snapshot) } - - pub(super) async fn receive_inspection( - &self, - stream: &mut S, - exporter: &[u8; 32], - prefix: [u8; frame::PREFIX], - ) -> Result<(), TransportError> { - let _node = self - .inspection_metadata - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?; - let head = self - .bounded( - None, - read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), - ) - .await?; - let wire = Wire { - auth: &self.auth, - exporter, - options: &self.options, - max_body: body::MAX_BODY, - }; - let request: RequestHead = wire.decode(REQUEST_MAGIC, REQUEST_PURPOSE, &head, None)?; - if frame::decimal(&request.nonce)? == 0 || !(1..=60_000).contains(&request.timeout_ms) { - return Err(TransportError::Protocol); - } - let (scope, source, target) = request.route.decode()?; - let local = self.runtime.log().limits(); - if target.identity - != (NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }) - || target.storage_instance - != self - .runtime - .log() - .storage_instance() - .map_err(|_| TransportError::Closed)? - || request.route.target_label() != self.auth.source() - { - return Err(TransportError::StalePeer); - } - let authorized = (|| { - let mut registry = self.registry.lock().map_err(|_| TransportError::Closed)?; - self.admit(®istry)?; - let grant = registry - .inspection_origins - .get(&(scope.cluster, scope.namespace, source.identity.node)) - .cloned() - .ok_or(TransportError::Refused)?; - let peer = grant.peer.as_ref().ok_or(TransportError::Refused)?; - registry.check_node_view(peer)?; - let view = registry - .node_views - .get(&peer.node) - .ok_or(TransportError::Refused)?; - if source.identity - != (NodeIdentity { - node: peer.node, - disk: peer.disk, - process: peer.process, - }) - || source.storage_instance != view.storage_instance() - || request.route.source_label() != peer.label - { - return Err(TransportError::StalePeer); - } - let budget = registry.inspection_peer_budget(peer.node, self.options.max_peers)?; - Ok::<_, TransportError>((grant, budget)) - })(); - let (grant, peer_budget) = match authorized { - Ok(authorized) => authorized, - Err(TransportError::Refused | TransportError::UnknownGroup) => { - let reply = ReplyHead { - route: request.route, - nonce: request.nonce, - status: 2, - digest: Sha256::digest([]).into(), - }; - let response = - wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &[])?; - return self.bounded(None, write_head(stream, &response)).await; - } - Err(error) => return Err(error), - }; - let _peer = peer_budget - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?; - let deadline = self - .domain - .mono_ms() - .map_err(|_| TransportError::Closed)? - .checked_add(request.timeout_ms) - .ok_or(TransportError::Limit)?; - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|h| h.upgrade()) - .ok_or(TransportError::Refused)?; - let binding = handle - .bindings - .get(&(scope.namespace, scope.cell.clone())) - .map_err(|_| TransportError::UnknownGroup)?; - if binding.scope != scope || binding.class.schema_digest() != request.route.schema() { - return Err(TransportError::Refused); - } - let cell = RaftCells { handle } - .cell(scope.namespace, &scope.cell) - .map_err(|_| TransportError::UnknownGroup)?; - let epoch = grant.stop.subscribe(); - self.seed_process_until(&epoch, deadline, async { - let now = self.domain.mono_ms().map_err(|_| TransportError::Closed)?; - let duration = std::time::Duration::from_millis( - deadline - .checked_sub(now) - .filter(|ms| *ms > 0) - .ok_or(TransportError::Timeout)?, - ); - let result = cell.inspect_snapshot(duration).await; - if *epoch.borrow() { - return Err(TransportError::StalePeer); - } - if *self.stop.borrow() { - return Err(TransportError::Closed); - } - let (status, bytes) = match result { - Ok(snapshot) if snapshot.view.birth_digest == request.route.birth() => { - (0, body::encode(&snapshot)?) - } - Err(CommandFailure::Overloaded) => (1, vec![]), - _ => (2, vec![]), - }; - let reply = ReplyHead { - route: request.route, - nonce: request.nonce, - status, - digest: Sha256::digest(&bytes).into(), - }; - let response = - wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &bytes)?; - write_head(stream, &response).await?; - stream.write_all(&bytes).await?; - stream.flush().await?; - Ok(()) - }) - .await - } } #[cfg(test)] diff --git a/crates/cellar/durability/raft/member_fleet.rs b/crates/cellar/durability/raft/member_fleet.rs index 45f6d7d9c..dc47deff3 100644 --- a/crates/cellar/durability/raft/member_fleet.rs +++ b/crates/cellar/durability/raft/member_fleet.rs @@ -2,16 +2,15 @@ //! only ordered, committed changes can consume or fence a permit sequence. use super::*; use ::raft::prelude::ConfChangeSingle; -use cellar_logic::consensus::{ - ClusterId, GroupId, GroupScope, IncarnationId, NamespaceId, ProposalId, -}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::{ClusterId, GroupId, IncarnationId, NamespaceId, ProposalId}; use sha2::{Digest, Sha256}; #[cfg(test)] #[path = "member_fleet_tests.rs"] mod tests; -/// Exact fleet and cell-history identity. Encoding this descriptor does not +/// Exact fleet and physical group-history identity. Encoding this descriptor does not /// enroll a group; enrollment requires a rooted catalog authorization. #[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] @@ -20,7 +19,6 @@ pub struct FleetGroupBinding { catalog: [u8; 32], cluster: [u8; 16], namespace: [u8; 16], - cell: String, group: [u8; 16], incarnation: [u8; 16], } @@ -28,10 +26,9 @@ pub struct FleetGroupBinding { impl FleetGroupBinding { pub fn new( fleet: IncarnationId, - scope: &GroupScope, + scope: &GroupHistory, catalog: [u8; 32], ) -> Result { - scope.validate().map_err(|_| invalid())?; if catalog == [0; 32] { return Err(invalid()); } @@ -40,7 +37,6 @@ impl FleetGroupBinding { catalog, cluster: scope.cluster.bytes(), namespace: scope.namespace.bytes(), - cell: scope.cell.clone(), group: scope.group.bytes(), incarnation: scope.incarnation.bytes(), }) @@ -54,12 +50,11 @@ impl FleetGroupBinding { self.catalog } - pub fn scope(&self) -> Result { + pub fn scope(&self) -> Result { self.validate()?; - Ok(GroupScope { + Ok(GroupHistory { cluster: ClusterId::new(self.cluster).map_err(|_| invalid())?, namespace: NamespaceId::new(self.namespace).map_err(|_| invalid())?, - cell: self.cell.clone(), group: GroupId::new(self.group).map_err(|_| invalid())?, incarnation: IncarnationId::new(self.incarnation).map_err(|_| invalid())?, }) @@ -68,11 +63,11 @@ impl FleetGroupBinding { pub fn key(&self) -> Result { self.validate()?; let bytes = serde_json::to_vec(&( + "cellar-cell-group-fleet-binding-1", self.fleet, self.catalog, self.cluster, self.namespace, - &self.cell, self.group, self.incarnation, )) @@ -94,16 +89,16 @@ impl FleetGroupBinding { NamespaceId::new(self.namespace).map_err(|_| invalid())?; GroupId::new(self.group).map_err(|_| invalid())?; IncarnationId::new(self.incarnation).map_err(|_| invalid())?; - if !cellar_logic::cell::valid_cell_scope(&self.cell) { - return Err(invalid()); - } Ok(()) } - pub(crate) fn matches_scope(&self, scope: &GroupScope) -> bool { + pub(crate) fn matches_scope(&self, scope: &S) -> bool + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); self.cluster == scope.cluster.bytes() && self.namespace == scope.namespace.bytes() - && self.cell == scope.cell && self.group == scope.group.bytes() && self.incarnation == scope.incarnation.bytes() } @@ -374,6 +369,23 @@ impl GroupMembership { Ok(initial) } + /// A non-serving seed base preserves whether the source was fleet guarded + /// at genesis or enrolled later. The real full snapshot must still succeed + /// this base; this projection grants no startup or membership authority. + pub(crate) fn seed_initial(&self, conf: &ConfState) -> Result { + let initial = Self::initial(conf); + if !self.same_birth(&initial) { + return Err(invalid()); + } + match &self.fleet { + Some(state) => match state.origin { + EnrollmentOrigin::Genesis => Self::managed_initial(conf, state.binding.clone()), + EnrollmentOrigin::Committed => Ok(initial), + }, + None => Ok(initial), + } + } + pub fn fleet_status(&self) -> Option { self.fleet.as_ref().map(|state| FleetMembershipStatus { binding: state.binding.clone(), @@ -462,7 +474,10 @@ impl GroupMembership { ) } - pub(crate) fn validate_scope(&self, scope: &GroupScope) -> Result<(), LogError> { + pub(crate) fn validate_scope(&self, scope: &S) -> Result<(), LogError> + where + for<'a> GroupHistory: From<&'a S>, + { if self .fleet .as_ref() @@ -473,12 +488,15 @@ impl GroupMembership { Ok(()) } - pub(crate) fn admit_scoped( + pub(crate) fn admit_scoped( &self, change: &ConfChangeV2, conf: &ConfState, - scope: &GroupScope, - ) -> Result<(), LogError> { + scope: &S, + ) -> Result<(), LogError> + where + for<'a> GroupHistory: From<&'a S>, + { self.validate_scope(scope)?; let binding = match self.intent(change)?.fleet { Some(FleetIntent::Enroll { binding }) => Some(binding), diff --git a/crates/cellar/durability/raft/member_fleet_tests.rs b/crates/cellar/durability/raft/member_fleet_tests.rs index 5ccc37389..bf2e814b0 100644 --- a/crates/cellar/durability/raft/member_fleet_tests.rs +++ b/crates/cellar/durability/raft/member_fleet_tests.rs @@ -1,5 +1,56 @@ use super::*; +#[test] +fn fleet_group_binding_refuses_cell_scoped_records() { + let (_, _, history, binding) = fixture(); + let mut record = serde_json::to_value(&binding).unwrap(); + assert!(record.get("cell").is_none()); + record["cell"] = serde_json::json!("Counter:legacy"); + assert!(serde_json::from_value::(record).is_err()); + let mut foreign = history; + foreign.incarnation = IncarnationId::new([91; 16]).unwrap(); + let other = + FleetGroupBinding::new(IncarnationId::new([5; 16]).unwrap(), &foreign, [90; 32]).unwrap(); + assert_ne!(binding.key().unwrap(), other.key().unwrap()); + assert!(!binding.matches_scope(&foreign)); +} + +#[test] +fn group_membership_retains_its_managed_guard_and_permit_sequence() { + let (state, conf, history, binding) = fixture(); + let (permit, _) = permit(binding); + let mut change = ConfChangeV2::default(); + state.bind_fleet_fence(&mut change, &conf, &permit).unwrap(); + state.admit_group(&change, &conf, &history).unwrap(); + let mut foreign = history; + foreign.group = GroupId::new([92; 16]).unwrap(); + assert!(state.admit_group(&change, &conf, &foreign).is_err()); + let next = state + .applied(&change, &conf, &conf, LogIndex::new(3)) + .unwrap(); + let restored: GroupMembership = + serde_json::from_slice(&serde_json::to_vec(&next).unwrap()).unwrap(); + restored.validate_scope(&history).unwrap(); + assert_eq!(restored.fleet_status().unwrap().consumed_sequence, 1); + assert!(restored + .bind_fleet_fence(&mut ConfChangeV2::default(), &conf, &permit) + .is_err()); + let replica = cellar_logic::consensus::ReplicaIdentity { + scope: history, + member: MemberId::new(1).unwrap(), + node: NodeId::new([61; 16]).unwrap(), + disk: DiskLineage::new([62; 16]).unwrap(), + process: cellar_logic::consensus::ProcessGeneration::new([63; 16]).unwrap(), + }; + assert!(restored.permits_group(&replica)); + assert!( + !restored.permits_group(&cellar_logic::consensus::ReplicaIdentity { + scope: foreign, + ..replica + }) + ); +} + #[test] fn fleet_genesis_origin_is_explicit_and_cannot_be_retrofitted() { let (_, conf, scope, binding) = fixture(); @@ -59,11 +110,10 @@ fn fleet_genesis_fences_first_permit_before_any_business_write() { assert_ne!(unused.expected_revision(), permit.expected_revision()); } -fn fixture() -> (GroupMembership, ConfState, GroupScope, FleetGroupBinding) { - let scope = GroupScope { +fn fixture() -> (GroupMembership, ConfState, GroupHistory, FleetGroupBinding) { + let scope = GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:fleet".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }; diff --git a/crates/cellar/durability/raft/member_state.rs b/crates/cellar/durability/raft/member_state.rs index 07cf8a25d..e0e11a106 100644 --- a/crates/cellar/durability/raft/member_state.rs +++ b/crates/cellar/durability/raft/member_state.rs @@ -137,7 +137,10 @@ impl GroupMembership { .any(|binding| binding.disk == disk.bytes()) } - pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { + pub(crate) fn permits(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool + where + for<'a> cellar_logic::cell_group::GroupHistory: From<&'a S>, + { if self.validate_scope(&replica.scope).is_err() { return false; } @@ -148,7 +151,7 @@ impl GroupMembership { &self, replica: &cellar_logic::consensus::ReplicaIdentity, ) -> bool { - self.fleet.is_none() && self.permits_binding(replica) + self.validate_scope(&replica.scope).is_ok() && self.permits_binding(replica) } fn permits_binding(&self, replica: &cellar_logic::consensus::ReplicaIdentity) -> bool { @@ -314,16 +317,13 @@ impl GroupMembership { &self, change: &ConfChangeV2, conf: &ConfState, + history: &cellar_logic::cell_group::GroupHistory, ) -> Result<(), LogError> { let intent = self.intent(change)?; - if self.fleet.is_some() - || self.repair.is_some() - || intent.fleet.is_some() - || intent.repair.is_some() - { + if self.fleet.is_none() && (self.repair.is_some() || intent.repair.is_some()) { return Err(invalid()); } - self.admit(change, conf) + self.admit_scoped(change, conf, history) } pub(crate) fn admit(&self, change: &ConfChangeV2, conf: &ConfState) -> Result<(), LogError> { diff --git a/crates/cellar/durability/raft/mod.rs b/crates/cellar/durability/raft/mod.rs index a8ce22a04..46745f5e5 100644 --- a/crates/cellar/durability/raft/mod.rs +++ b/crates/cellar/durability/raft/mod.rs @@ -17,9 +17,11 @@ pub(crate) mod execution; mod filesystem; pub(crate) mod fleet_catalog; pub(crate) mod fleet_stop; +pub(crate) mod group_host; pub(crate) mod host; mod host_archive; mod host_sql; +pub(crate) mod restart_leadership; pub(crate) use host_sql::ResourceObserver as SqlResourceObserver; pub(crate) mod fleet_inspection_wire; pub(crate) mod inspection; @@ -37,7 +39,6 @@ pub(crate) mod physical_inventory; mod query; mod query_identity; -mod query_route; mod query_wire; pub(crate) mod receipt_maintenance; mod receipts; diff --git a/crates/cellar/durability/raft/node_inspection_transport.rs b/crates/cellar/durability/raft/node_inspection_transport.rs index 6896f27e6..b7c3fc5fd 100644 --- a/crates/cellar/durability/raft/node_inspection_transport.rs +++ b/crates/cellar/durability/raft/node_inspection_transport.rs @@ -47,6 +47,48 @@ pub(crate) struct Received { } impl RaftTransport { + pub(crate) fn install_group_fleet_gateway( + &self, + cells: &RaftCells, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let mut gateway = state + .fleet_gateway + .lock() + .map_err(|_| TransportError::Closed)?; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if gateway + .as_ref() + .and_then(std::sync::Weak::upgrade) + .is_some() + || cells.is_closed() + || !cells.handle.domain.same_owner(&state.domain) + || !cells.uses_transport(self) + { + return Err(TransportError::Refused); + } + *gateway = Some(Arc::downgrade(&cells.handle)); + Ok(()) + } + + /// Read the exact complete trust cut already installed in this transport. + /// A separate catalog refresh cannot acknowledge installed routing state. + pub(crate) fn installed_fleet_node( + &self, + node: NodeId, + ) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + registry + .fleet_trust + .as_ref() + .and_then(|trust| trust.node(node)) + .cloned() + .ok_or(TransportError::Refused) + } + pub(crate) fn inspect_fleet_node( &self, node: NodeId, @@ -459,7 +501,7 @@ impl State { .checked_add(request.timeout_ms) .ok_or(TransportError::Limit)?; let handle = self - .query_gateway + .fleet_gateway .lock() .map_err(|_| TransportError::Closed)? .as_ref() @@ -471,7 +513,7 @@ impl State { .filter(|ms| *ms > 0) .ok_or(TransportError::Timeout)?; let result = if request.physical { - match (RaftCells { handle }) + match RaftCells::from_internal_handle(handle) .observe_physical_node(std::time::Duration::from_millis(remaining)) .await { @@ -482,7 +524,7 @@ impl State { Err(error) => Err(error), } } else { - match (RaftCells { handle }) + match RaftCells::from_internal_handle(handle) .observe_node(std::time::Duration::from_millis(remaining)) .await { diff --git a/crates/cellar/durability/raft/node_inspection_wire.rs b/crates/cellar/durability/raft/node_inspection_wire.rs index 91a1c1610..21f2fe25c 100644 --- a/crates/cellar/durability/raft/node_inspection_wire.rs +++ b/crates/cellar/durability/raft/node_inspection_wire.rs @@ -162,6 +162,7 @@ struct Body { declared_namespaces: u32, registered_histories: u32, dormant_histories: u32, + #[serde(rename = "pending_group_registrations")] pending_history_registrations: u32, unattached_engine_histories: u32, seed_workers: u32, @@ -211,7 +212,6 @@ pub(crate) fn validate(view: &RaftHostObservation) -> Result<(), TransportError> || !(1..=64).contains(&view.declared_namespaces) || view.registered_histories > 4096 || view.dormant_histories > view.registered_histories - || view.pending_history_registrations > 1 || view.node.groups > 4097 || view.unattached_engine_histories > view.node.groups || view diff --git a/crates/cellar/durability/raft/node_inspection_wire_tests.rs b/crates/cellar/durability/raft/node_inspection_wire_tests.rs index 6b14d72c2..216b650c7 100644 --- a/crates/cellar/durability/raft/node_inspection_wire_tests.rs +++ b/crates/cellar/durability/raft/node_inspection_wire_tests.rs @@ -84,7 +84,7 @@ fn node_metadata_rejects_missing_invalid_and_unbounded_fields() { "startup", "current", "profile", - "pending_history_registrations", + "pending_group_registrations", "boot", "bounds", "seed_workers", @@ -122,3 +122,27 @@ fn node_metadata_rejects_missing_invalid_and_unbounded_fields() { assert!(decode(&serde_json::to_vec(&changed).unwrap()).is_err()); assert!(decode(&vec![0; MAX_BODY + 1]).is_err()); } + +#[test] +fn node_metadata_preserves_concurrent_group_admission_debt() { + let mut observed = view(); + observed.pending_history_registrations = 7; + let bytes = encode(&observed).unwrap(); + let decoded = decode(&bytes).unwrap(); + assert_eq!(decoded.pending_history_registrations, 7); + assert_eq!(decoded.node.groups, 0); + assert_eq!(decoded.registered_histories, 0); + // A durable engine scan with zero slots cannot erase admitted provisioning + // debt. The old single-history counter is rejected by the hard cutover. + let mut old: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + let n = old + .as_object_mut() + .unwrap() + .remove("pending_group_registrations") + .unwrap(); + old["pending_history_registrations"] = n; + assert!(decode(&serde_json::to_vec(&old).unwrap()).is_err()); + let mut overflow: serde_json::Value = serde_json::from_slice(&bytes).unwrap(); + overflow["pending_group_registrations"] = serde_json::json!(u64::from(u32::MAX) + 1); + assert!(decode(&serde_json::to_vec(&overflow).unwrap()).is_err()); +} diff --git a/crates/cellar/durability/raft/operations.rs b/crates/cellar/durability/raft/operations.rs index ab395b80e..ce5488069 100644 --- a/crates/cellar/durability/raft/operations.rs +++ b/crates/cellar/durability/raft/operations.rs @@ -75,13 +75,59 @@ pub struct RaftCloseReport { /// joining the complete domain. Missing startup/close proof remains explicit. #[derive(Clone, Debug, PartialEq, Eq)] pub struct RaftNodeShutdownReport { - pub result: Result, + pub result: + Result, pub startup_incomplete: bool, /// Best-effort handoff under the original Node shutdown deadline. Missing /// evidence is not a graceful drain, even when physical closure succeeded. pub drain: Option>, } +/// Preserve the joined closure failure, including both controller and log +/// failures. Cloned waiters share one failure identity; admission errors remain +/// comparable by value. This error never resolves an individual mutation. +#[derive(Clone, Debug)] +pub enum RaftNodeShutdownError { + Admission(CommandFailure), + Closure(Arc), +} + +impl From for RaftNodeShutdownError { + fn from(error: CommandFailure) -> Self { + Self::Admission(error) + } +} + +impl PartialEq for RaftNodeShutdownError { + fn eq(&self, other: &Self) -> bool { + match (self, other) { + (Self::Admission(a), Self::Admission(b)) => a == b, + (Self::Closure(a), Self::Closure(b)) => Arc::ptr_eq(a, b), + _ => false, + } + } +} + +impl Eq for RaftNodeShutdownError {} + +impl std::fmt::Display for RaftNodeShutdownError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Admission(error) => std::fmt::Display::fmt(error, f), + Self::Closure(error) => std::fmt::Display::fmt(error, f), + } + } +} + +impl std::error::Error for RaftNodeShutdownError { + fn source(&self) -> Option<&(dyn std::error::Error + 'static)> { + Some(match self { + Self::Admission(error) => error, + Self::Closure(error) => error.as_ref(), + }) + } +} + /// Each registered local replica receives at most one bounded handoff turn. /// Waiting for admitted group work and inspecting a successor share that turn's /// deadline; a busy group cannot renew its budget or delay every other group. @@ -101,7 +147,7 @@ impl RaftNodeShutdownReport { pub(crate) fn cli_summary(&self) -> serde_json::Value { match &self.result { Ok(report) => { - serde_json::json!({"owners_closed":true,"startup_incomplete":self.startup_incomplete,"groups":report.groups_total,"omitted_group_details":report.omitted_groups,"groups_with_archive_lag":report.groups_with_archive_lag,"groups_with_unknown_archive":report.groups_with_unknown_archive,"unfinished_commands":report.unfinished.len(),"retain_replica_disks":true,"handoff":self.drain.as_ref().map(|result|match result {Ok(drain)=>serde_json::json!({"groups_examined":drain.groups_examined,"attempts":drain.handoff_attempts,"accepted":drain.handoffs_accepted,"leadership_released":drain.leadership_released,"unavailable_groups":drain.unavailable_groups,"deadline_elapsed":drain.deadline_elapsed}),Err(error)=>serde_json::json!({"error":error.to_string()})})}) + serde_json::json!({"owners_closed":true,"startup_incomplete":self.startup_incomplete,"groups":report.groups_total,"omitted_group_details":report.omitted_groups,"groups_with_archive_lag":report.groups_with_archive_lag,"groups_with_unknown_archive":report.groups_with_unknown_archive,"mutation_outcomes":report.mutation_outcomes,"retained_calls_at_shutdown":report.retained_calls_at_shutdown,"retain_replica_disks":true,"handoff":self.drain.as_ref().map(|result|match result {Ok(drain)=>serde_json::json!({"groups_examined":drain.groups_examined,"attempts":drain.handoff_attempts,"accepted":drain.handoffs_accepted,"leadership_released":drain.leadership_released,"unavailable_groups":drain.unavailable_groups,"deadline_elapsed":drain.deadline_elapsed}),Err(error)=>serde_json::json!({"error":error.to_string()})})}) } Err(error) => { serde_json::json!({"owners_closed":false,"startup_incomplete":self.startup_incomplete,"error":error.to_string(),"retain_replica_disks":true}) @@ -258,7 +304,10 @@ impl NodeLifecycle { pub(crate) fn resource_usage( &self, - ) -> Result, CommandFailure> { + ) -> Result< + Option, + CommandFailure, + > { Ok(self .sample_service()? .map(|service| service.resource_usage())) @@ -312,7 +361,7 @@ impl NodeLifecycle { Ok(state) => state, Err(_) => { return Some(RaftNodeShutdownReport { - result: Err(CommandFailure::Unavailable), + result: Err(CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }) @@ -333,13 +382,16 @@ impl NodeLifecycle { None => None, }; RaftNodeShutdownReport { - result: service.close().await, + result: service + .close() + .await + .map_err(RaftNodeShutdownError::Closure), startup_incomplete: false, drain, } } None => RaftNodeShutdownReport { - result: Err(CommandFailure::Unavailable), + result: Err(CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }, diff --git a/crates/cellar/durability/raft/physical_inventory.rs b/crates/cellar/durability/raft/physical_inventory.rs index 737a4478d..ebbd1c62c 100644 --- a/crates/cellar/durability/raft/physical_inventory.rs +++ b/crates/cellar/durability/raft/physical_inventory.rs @@ -158,7 +158,7 @@ pub(crate) fn scope_digest(scope: &GroupScope) -> Result<[u8; 32], LogError> { Ok(Sha256::digest(bytes).into()) } -fn group_digest(history: &GroupHistory) -> [u8; 32] { +pub(crate) fn group_digest(history: &GroupHistory) -> [u8; 32] { let mut digest = Sha256::new(); digest.update(b"cellar-cell-group-physical-history-1\0"); digest.update(history.cluster.bytes()); diff --git a/crates/cellar/durability/raft/query.rs b/crates/cellar/durability/raft/query.rs index 9dade0421..9f24a9a37 100644 --- a/crates/cellar/durability/raft/query.rs +++ b/crates/cellar/durability/raft/query.rs @@ -1,12 +1,9 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. -//! Request-owned read constraints. A fresh ReadIndex result is correlated to -//! one invocation; a cached commit watermark can never replace that barrier. +//! Internal SQL minimum checks and the remaining seed barrier owner. Business +//! reads are admitted only through the shared physical-group controller. use crate::application::consensus::client::Request; -use crate::application::consensus::{ - OperationContract, QueryConsistency, QueryOptions, StaleReadPolicy, -}; use cellar_logic::consensus::{CommandFailure, CommittedVersion, GroupScope, LogIndex, Term}; pub(crate) type ReadId = [u8; 32]; @@ -48,8 +45,6 @@ impl Minimum { #[derive(Clone, Copy, PartialEq, Eq)] pub(crate) enum Phase { - Source, - Probing, StartBarrier, CheckingLeader, Barrier, @@ -57,28 +52,13 @@ pub(crate) enum Phase { Executing, } -pub(crate) enum ReadOperation { - Fleet { - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - charge: crate::raft::budget::Permit, - }, - Query { - contract: OperationContract, - input: Vec, - }, - Seed { - target: cellar_logic::consensus::ReplicaIdentity, - kind: crate::raft::transport::seeds::SeedKind, - charge: std::sync::Arc, - }, -} - pub(crate) struct PendingRead { pub request: Request, - pub operation: ReadOperation, - pub minimum: Option, + pub target: cellar_logic::consensus::ReplicaIdentity, + pub kind: crate::raft::transport::seeds::SeedKind, + pub charge: std::sync::Arc, + pub minimum: Option, pub phase: Phase, - pub source_deadline: Option, pub retry_at: u64, pub leader: Option<(Term, Option, bool)>, pub checking_leader: bool, @@ -89,81 +69,19 @@ pub(crate) struct PendingRead { } impl PendingRead { - pub fn fleet( - request: Request, - authorization: crate::raft::fleet_catalog::FleetGroupAuthorization, - charge: crate::raft::budget::Permit, - ) -> Self { - Self::with_options( - request, - ReadOperation::Fleet { - authorization, - charge, - }, - &QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }, - None, - ) - } - - pub fn new( - request: Request, - contract: OperationContract, - input: Vec, - options: &QueryOptions, - minimum: Option, - ) -> Self { - Self::with_options( - request, - ReadOperation::Query { contract, input }, - options, - minimum, - ) - } - pub fn seed( request: Request, target: cellar_logic::consensus::ReplicaIdentity, kind: crate::raft::transport::seeds::SeedKind, charge: std::sync::Arc, ) -> Self { - Self::with_options( - request, - ReadOperation::Seed { - target, - kind, - charge, - }, - &QueryOptions { - consistency: QueryConsistency::Linearizable, - ..Default::default() - }, - None, - ) - } - - fn with_options( - request: Request, - operation: ReadOperation, - options: &QueryOptions, - minimum: Option, - ) -> Self { - let (phase, minimum) = match options.consistency { - QueryConsistency::Linearizable => (Phase::StartBarrier, None), - QueryConsistency::AtLeast(_) => (Phase::Image, minimum.map(Minimum::Version)), - QueryConsistency::Stale if options.stale_policy == StaleReadPolicy::RecentSource => { - (Phase::Source, Some(Minimum::Any)) - } - QueryConsistency::Stale => (Phase::Image, Some(Minimum::Any)), - }; Self { request, - operation, - minimum, - phase, - source_deadline: None, + target, + kind, + charge, + minimum: None, + phase: Phase::StartBarrier, retry_at: 0, leader: None, checking_leader: false, @@ -176,8 +94,5 @@ impl PendingRead { pub fn expired(&self) -> bool { self.request.control.cancelled() - || self - .source_deadline - .is_some_and(|deadline| !self.request.control.before(deadline)) } } diff --git a/crates/cellar/durability/raft/query_cell_transport.rs b/crates/cellar/durability/raft/query_cell_transport.rs index 02500bda7..0433ba82a 100644 --- a/crates/cellar/durability/raft/query_cell_transport.rs +++ b/crates/cellar/durability/raft/query_cell_transport.rs @@ -2,16 +2,15 @@ //! Shared cell reads reuse the peer query lane; namespace grants remain read-only. use super::*; -use crate::application::consensus::{CommandFailure, RaftCells}; +use crate::application::consensus::CommandFailure; use crate::raft::cell_groups::{ - CellDispatchError, CellGroupDispatcher, CellGroupRuntime, CellGroupSqlPool, CellQueryResponse, - CellRoutingLimits, CellSqlOptions, + CellDispatchError, CellGroupController, CellGroupDispatcher, CellGroupNamespaceStorage, + CellQueryResponse, CellRoutingLimits, }; use crate::raft::query_identity::QueryRoute; use crate::raft::query_wire::{ self as wire, cell::{CellQueryCall, CellQueryValue}, - QueryCall, }; use crate::raft::version::{CellVersionObservation, CellVersionToken, VersionKeys}; use cellar_logic::{ @@ -42,10 +41,9 @@ impl Drop for Registration { pub(super) struct Gateway { accepting: AtomicBool, - runtime: CellGroupRuntime, - sql: CellGroupSqlPool, + controller: CellGroupController, dispatcher: CellGroupDispatcher, - options: CellSqlOptions, + storage: CellGroupNamespaceStorage, } /// Retains the same node/peer query admission through consumption of the result. @@ -104,17 +102,20 @@ impl RaftTransport { pub fn install_cell_query_gateway( &self, dispatcher: CellGroupDispatcher, - sql: CellGroupSqlPool, - options: CellSqlOptions, + controller: CellGroupController, + storage: CellGroupNamespaceStorage, ) -> Result { let state = &self.handle.0; let runtime = state.runtime.shared()?; - if !dispatcher.uses_domain(&state.domain) - || !sql.uses_domain(&state.domain) - || !sql.belongs_to_engine(&runtime.log().owner()) - { + if !dispatcher.uses_domain(&state.domain) || !controller.belongs_to(runtime) { return Err(TransportError::Identity); } + let (namespace, _, _) = dispatcher + .host_binding() + .map_err(|_| TransportError::Identity)?; + storage + .validate_namespace(&namespace) + .map_err(|_| TransportError::Identity)?; let key = dispatcher.namespace(); let mut gateways = state .cell_query_gateways @@ -131,10 +132,9 @@ impl RaftTransport { } let inner = Arc::new(Gateway { accepting: AtomicBool::new(true), - runtime: runtime.clone(), - sql, + controller, dispatcher, - options, + storage, }); gateways.insert(key, Arc::downgrade(&inner)); Ok(CellQueryGateway { @@ -341,7 +341,7 @@ impl State { ) .await; // A fully authenticated application refusal consumed a complete reply; - // retain the peer connection just as the legacy query lane does. + // retain the peer connection for the next authenticated request. let failed = matches!(&result, Err(error) if !matches!(error, TransportError::Query(_))); let _ = request .reply @@ -355,7 +355,6 @@ impl State { } pub(super) enum IncomingCall { - Legacy(QueryCall), Shared(CellQueryCall), Command(crate::raft::query_wire::command::CellCommandCall), Enrollment(crate::raft::query_wire::enrollment::CellEnrollmentCall), @@ -383,16 +382,11 @@ impl IncomingCall { crate::raft::query_wire::command::CellCommandCall::decode(bytes, limit)?, )); } - if route.is_shared() { - Ok(Self::Shared(CellQueryCall::decode(bytes, limit)?)) - } else { - Ok(Self::Legacy(QueryCall::decode(bytes, limit)?)) - } + Ok(Self::Shared(CellQueryCall::decode(bytes, limit)?)) } pub(super) fn timeout_ms(&self) -> u64 { match self { - Self::Legacy(call) => call.options.timeout.as_millis() as u64, Self::Shared(call) => call.timeout_ms, Self::Command(call) => call.timeout_ms, Self::Enrollment(call) => call.timeout_ms, @@ -409,30 +403,6 @@ impl IncomingCall { Self::Command(call) => state.execute_cell_command(route, call).await, Self::Enrollment(call) => state.execute_cell_enrollment(route, call).await, Self::Receipt(call) => state.execute_cell_receipt(route, call).await, - Self::Legacy(call) => { - let (_, target) = route.decode()?; - let handle = state - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|h| h.upgrade()) - .ok_or(TransportError::Refused)?; - let cell = RaftCells { handle } - .cell(target.scope.namespace, &target.scope.cell) - .map_err(|_| TransportError::Refused)?; - if cell.scope() != &target.scope - || cell.binding.class.schema_digest() != route.schema() - { - return Err(TransportError::StalePeer); - } - let result = cell.query(call.contract, call.input, call.options).await; - if result.as_ref().is_ok_and(|v| v.served_by != target.member) { - return Err(TransportError::Identity); - } - wire::encode_result(&result, state.options.max_query_response_bytes) - .map(super::query_replies::Reply::plain) - } Self::Shared(call) => { let (_, target, cell) = route.decode_shared()?; let gateway = state @@ -448,7 +418,7 @@ impl IncomingCall { let result = gateway .execute(call, target.scope, cell, route.schema()) .await - .map_err(query_failure); + .map_err(|e| query_failure(&e)); wire::cell::encode_cell_result(&result, state.options.max_query_response_bytes) .map(super::query_replies::Reply::plain) } @@ -463,40 +433,34 @@ impl Gateway { group: GroupHistory, cell: CellHistory, schema: [u8; 32], - ) -> Result { - let epoch = call.route_epoch; - let birth = call.birth_digest; - let mut query = self.dispatcher.query( - cell, - call.request, - CellRoutingLimits { - timeout: Duration::from_millis(call.timeout_ms), - max_attempts: 1, - }, - None, - )?; - query.begin_current().await?; - let selected = query - .attempt() - .ok_or(CellDispatchError::Invalid("missing query route"))? - .route(); - if selected.history != group - || selected.schema_digest != schema - || selected.epoch.get() != epoch - || selected.birth_digest != birth - { - return Err(CellDispatchError::Invalid("remote query route")); - } - query - .execute(&self.runtime, &self.sql, self.options.clone()) - .await + ) -> Result> { + let options = self.storage.for_cell(&cell); + let queued = (|| { + let mut query = self.dispatcher.query( + cell, + call.request, + CellRoutingLimits { + timeout: Duration::from_millis(call.timeout_ms), + max_attempts: 1, + }, + None, + )?; + query.bind_remote_route(crate::raft::cell_groups::RemoteMutationRoute { + group, + schema, + epoch: call.route_epoch, + birth: call.birth_digest, + }); + self.controller.query(query, options) + })(); + queued.map_err(Arc::new)?.response().await } } -fn query_failure(error: CellDispatchError) -> CommandFailure { +fn query_failure(error: &CellDispatchError) -> CommandFailure { use crate::raft::{execution::ExecutionError, version::VersionError}; match error { - CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure, + CellDispatchError::Execution(ExecutionError::Refused(failure)) => failure.clone(), CellDispatchError::Runtime(RuntimeError::Overloaded) => CommandFailure::Overloaded, CellDispatchError::Runtime(RuntimeError::DeadlineExceeded | RuntimeError::NotReady) => { CommandFailure::VersionUnavailable diff --git a/crates/cellar/durability/raft/query_cell_wire.rs b/crates/cellar/durability/raft/query_cell_wire.rs index 78dde0a0a..327604540 100644 --- a/crates/cellar/durability/raft/query_cell_wire.rs +++ b/crates/cellar/durability/raft/query_cell_wire.rs @@ -42,6 +42,13 @@ impl CellQueryCall { body.extend_from_slice(&request.contract.digest); match &request.consistency { CellQueryConsistency::Stale => body.push(0), + CellQueryConsistency::RecentSource { max_age_ms } => { + if !(1..=3_600_000).contains(max_age_ms) { + return Err(TransportError::Protocol); + } + body.push(3); + body.extend_from_slice(&max_age_ms.to_be_bytes()); + } CellQueryConsistency::Linearizable => body.push(1), CellQueryConsistency::AtLeast(token) => { body.push(2); @@ -93,6 +100,13 @@ impl CellQueryCall { CellVersionToken::from_encoded(&body.string(4096)?) .map_err(|_| TransportError::Authentication)?, ), + 3 => { + let max_age_ms = u64::from_be_bytes(body.array()?); + if !(1..=3_600_000).contains(&max_age_ms) { + return Err(TransportError::Protocol); + } + CellQueryConsistency::RecentSource { max_age_ms } + } _ => return Err(TransportError::Protocol), }; let timeout_ms = u64::from_be_bytes(body.array()?); diff --git a/crates/cellar/durability/raft/query_identity.rs b/crates/cellar/durability/raft/query_identity.rs index d454b2c99..646562b91 100644 --- a/crates/cellar/durability/raft/query_identity.rs +++ b/crates/cellar/durability/raft/query_identity.rs @@ -4,7 +4,6 @@ //! The complete target history remains authenticated in each request/reply. use super::transport::TransportError; -use crate::application::consensus::protocol::WireScope; use cellar_logic::cell_group::{CellHistory, CellId, GroupHistory}; use cellar_logic::consensus::{ ClusterId, DiskLineage, GroupId, IncarnationId, MemberId, NamespaceId, NodeId, NodeIdentity, @@ -43,11 +42,10 @@ impl Endpoint { } } -// Marker-1 hard cutover: no reader for the former untagged query history. +// Marker-1 hard cutover: only physical-group plus logical-cell query histories. #[derive(Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(tag = "kind", content = "identity", deny_unknown_fields)] enum History { - Cell(WireScope), Shared { group: [[u8; 16]; 4], cell: String, @@ -74,41 +72,6 @@ pub(super) struct QueryRoute { } impl QueryRoute { - pub fn new( - source: NodeIdentity, - source_label: &str, - target: &ReplicaIdentity, - target_label: &str, - schema: [u8; 32], - ) -> Result { - let route = Self { - purpose: Purpose::Query, - scope: History::Cell(WireScope::from(&target.scope)), - schema, - source: Endpoint::new(source, source_label, None), - target: Endpoint::new(target.node_identity(), target_label, Some(target.member)), - }; - route.decode()?; - Ok(route) - } - - /// Request orientation only. A reversed response route is compared with - /// the exact outstanding request; it cannot decode as another request. - pub fn decode(&self) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { - if self.requires_command_grant() { - return Err(TransportError::Identity); - } - let History::Cell(scope) = &self.scope else { - return Err(TransportError::Identity); - }; - let scope = scope - .clone() - .decode() - .map_err(|_| TransportError::Identity)?; - scope.validate().map_err(|_| TransportError::Identity)?; - self.endpoints(scope) - } - fn endpoints(&self, scope: S) -> Result<(NodeIdentity, ReplicaIdentity), TransportError> { if self.source.member.is_some() || self.schema == [0; 32] { return Err(TransportError::Identity); @@ -176,10 +139,7 @@ impl QueryRoute { group, cell, incarnation, - } = &self.scope - else { - return Err(TransportError::Identity); - }; + } = &self.scope; let scope = GroupHistory { cluster: ClusterId::new(group[0]).map_err(|_| TransportError::Identity)?, namespace: NamespaceId::new(group[1]).map_err(|_| TransportError::Identity)?, @@ -230,16 +190,8 @@ impl QueryRoute { self.is_command() || self.is_enrollment() || self.is_receipt() } - pub fn is_shared(&self) -> bool { - matches!(self.scope, History::Shared { .. }) - } - pub fn validate(&self) -> Result<(), TransportError> { - if self.is_shared() { - self.decode_shared()?; - } else { - self.decode()?; - } + self.decode_shared()?; Ok(()) } diff --git a/crates/cellar/durability/raft/query_route.rs b/crates/cellar/durability/raft/query_route.rs deleted file mode 100644 index 9f0f0a8a6..000000000 --- a/crates/cellar/durability/raft/query_route.rs +++ /dev/null @@ -1,194 +0,0 @@ -// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. - -//! Bounded read attempts. Every destination receives the original contract, -//! input and consistency; attempts consume one shared invocation deadline. - -use super::budget::Permit; -use super::query_wire::QueryCall; -use super::runtime::RaftRuntime; -use super::transport::{RaftTransport, TransportError}; -use crate::application::consensus::client::AttemptControl; -use crate::application::consensus::{ - CommandFailure, CommittedVersion, MemberId, OperationContract, QueryOptions, QueryResult, - QueryRouting, RaftCellHandle, VersionKeys, -}; -use std::{collections::BTreeSet, sync::Arc, time::Duration}; - -pub(super) struct RouteContext { - pub local: RaftCellHandle, - pub member: Option, - pub leader_hint: Option, - pub runtime: RaftRuntime, - pub transport: RaftTransport, - pub keys: VersionKeys, - pub control: Arc, - pub minimum: Option, -} - -pub(super) struct RoutedOutput { - pub result: QueryResult, - pub bytes: Permit, -} - -enum Attempt { - Success(RoutedOutput), - Retry(CommandFailure), - Fatal(CommandFailure), -} - -impl RouteContext { - pub async fn run( - self, - contract: OperationContract, - input: Vec, - mut options: QueryOptions, - ) -> Result { - let QueryRouting::Replicas { - members, - attempt_timeout, - leader_fallback, - } = std::mem::take(&mut options.routing) - else { - return Err(CommandFailure::UnsupportedConsistency); - }; - if members.is_empty() - || members.len() > 8 - || attempt_timeout.as_millis() == 0 - || attempt_timeout > Duration::from_secs(60) - { - return Err(CommandFailure::UnsupportedConsistency); - } - let mut tried = BTreeSet::new(); - let mut last = CommandFailure::VersionUnavailable; - for member in members { - if !tried.insert(member) { - return Err(CommandFailure::UnsupportedConsistency); - } - match self - .attempt(member, &contract, &input, &options, attempt_timeout) - .await - { - Attempt::Success(result) => return Ok(result), - Attempt::Fatal(error) => return Err(error), - Attempt::Retry(error) => last = error, - } - } - if leader_fallback && !self.control.cancelled() { - // This is a routing hint only. The destination still obtains the - // requested read proof; a stale leader hint grants no authority. - let leader = if self.member.is_some() { - let status = self.runtime.status(self.local.scope()); - let status = crate::asyncrt::select! { _ = self.control.cancellation() => return Err(CommandFailure::VersionUnavailable), value = status => value }; - status - .ok() - .and_then(|status| status.leader) - .and_then(|id| MemberId::new(id).ok()) - } else { - self.leader_hint - }; - if let Some(member) = leader.filter(|member| tried.insert(*member)) { - match self - .attempt(member, &contract, &input, &options, attempt_timeout) - .await - { - Attempt::Success(result) => return Ok(result), - Attempt::Fatal(error) => return Err(error), - Attempt::Retry(error) => last = error, - } - } - } - if self.control.cancelled() { - Err(CommandFailure::VersionUnavailable) - } else { - Err(last) - } - } - - async fn attempt( - &self, - member: MemberId, - contract: &OperationContract, - input: &[u8], - options: &QueryOptions, - timeout: Duration, - ) -> Attempt { - if self.control.cancelled() { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - } - let domain = &self.local.handle.domain; - let Ok(now) = domain.mono_ms() else { - return Attempt::Fatal(CommandFailure::Unavailable); - }; - let deadline = self - .control - .deadline() - .min(now.saturating_add(timeout.as_millis() as u64)); - let Some(remaining) = deadline.checked_sub(now).filter(|ms| *ms > 0) else { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - }; - let Ok(timer) = domain.sleep_until(deadline) else { - return Attempt::Fatal(CommandFailure::Unavailable); - }; - let mut options = options.clone(); - options.routing = QueryRouting::Local; - options.timeout = Duration::from_millis(remaining); - let query = async { - let result = if Some(member) == self.member { - self.local - .query(contract.clone(), input.to_vec(), options) - .await - } else { - let delivery = self.transport.query( - self.local.scope(), - member, - QueryCall { - contract: contract.clone(), - input: input.to_vec(), - options, - }, - self.keys.clone(), - deadline, - )?; - delivery.receive().await?.value - }; - Ok::<_, TransportError>(result) - }; - let result = crate::asyncrt::select! { - _ = self.control.cancellation() => return Attempt::Fatal(CommandFailure::VersionUnavailable), - _ = timer => return Attempt::Retry(CommandFailure::VersionUnavailable), - result = query => result, - }; - if self.control.cancelled() { - return Attempt::Fatal(CommandFailure::VersionUnavailable); - } - match result { - Ok(Ok(result)) => { - if result.served_by != member - || result.version.scope != *self.local.scope() - || self - .minimum - .as_ref() - .is_some_and(|minimum| result.version.covers(minimum) != Ok(true)) - { - return Attempt::Fatal(CommandFailure::CorruptState); - } - let Some(bytes) = self.local.handle.budget.acquire(result.result.len()) else { - return Attempt::Retry(CommandFailure::Overloaded); - }; - Attempt::Success(RoutedOutput { result, bytes }) - } - Ok(Err(error @ (CommandFailure::WrongIncarnation | CommandFailure::CorruptState))) => { - Attempt::Fatal(error) - } - Ok(Err(error)) => Attempt::Retry(error), - Err( - TransportError::Authentication - | TransportError::Identity - | TransportError::Protocol - | TransportError::Replay, - ) => Attempt::Fatal(CommandFailure::Unavailable), - Err(TransportError::Overloaded) => Attempt::Retry(CommandFailure::Overloaded), - Err(_) => Attempt::Retry(CommandFailure::VersionUnavailable), - } - } -} diff --git a/crates/cellar/durability/raft/query_transport.rs b/crates/cellar/durability/raft/query_transport.rs index dc6ada81b..35ac13bc1 100644 --- a/crates/cellar/durability/raft/query_transport.rs +++ b/crates/cellar/durability/raft/query_transport.rs @@ -4,58 +4,17 @@ //! occupy the control or LTX connection required to establish its barrier. use super::*; -use crate::application::consensus::{client, CommandFailure, QueryResult, VersionKeys}; +use crate::application::consensus::client; use crate::raft::query_identity::QueryRoute; -use crate::raft::query_wire::{self as wire, QueryCall}; -use cellar_logic::consensus::{MemberId, NodeIdentity}; +use crate::raft::query_wire as wire; +use cellar_logic::consensus::NodeIdentity; use sha2::{Digest, Sha256}; -pub(crate) struct QueryResponse { - pub value: Result, - _node: Permit, - _peer: Permit, -} - -pub(crate) struct QueryDelivery { - result: Option>>, - cancel: watch::Sender, -} - -impl QueryDelivery { - pub async fn receive(mut self) -> Result { - self.result - .take() - .ok_or(TransportError::Closed)? - .await - .map_err(|_| TransportError::Closed)? - } -} - -impl Drop for QueryDelivery { - fn drop(&mut self) { - self.cancel.send_replace(true); - } -} - pub(super) enum SendQuery { - Legacy(Box), Shared(Box), Write(Box), } -pub(super) struct LegacySendQuery { - scope: GroupScope, - route: Arc, - peer: RaftPeer, - call: QueryCall, - deadline: u64, - keys: VersionKeys, - cancel: watch::Receiver, - reply: oneshot::Sender>, - node: Permit, - peer_bytes: Permit, -} - pub(super) struct OriginGrant { pub(super) revision: u64, pub(super) peer: Option, @@ -267,98 +226,6 @@ impl RaftTransport { *gateway = Some(Arc::downgrade(&handle)); Ok(()) } - - pub(in crate::raft) fn query( - &self, - scope: &GroupScope, - member: MemberId, - call: QueryCall, - keys: VersionKeys, - deadline: u64, - ) -> Result { - let state = &self.handle.0; - let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; - state.admit(®istry)?; - if state.domain.mono_ms().map_err(|_| TransportError::Closed)? >= deadline { - return Err(TransportError::Timeout); - } - let route = registry - .groups - .get(scope) - .or_else(|| registry.query_groups.get(scope)) - .ok_or(TransportError::UnknownGroup)? - .clone(); - let local_node = state.runtime.log().limits().node; - let peer = route - .peers - .iter() - .find(|p| p.replica.member == member && p.replica.node != local_node) - .ok_or(TransportError::Identity)? - .clone(); - let pool = registry - .pools - .get(&peer.replica.node) - .ok_or(TransportError::StalePeer)? - .clone(); - drop(registry); - let charge = reservation( - call.input - .len() - .checked_add(4096) - .ok_or(TransportError::Limit)?, - &state.options, - )?; - let node = state - .outgoing - .query - .acquire(charge) - .ok_or(TransportError::Overloaded)?; - let peer_bytes = pool - .outgoing - .query - .acquire(charge) - .ok_or(TransportError::Overloaded)?; - // Validate/size the bounded representation before queueing, but retain - // only captured inputs. The sender encodes again with remaining time. - drop(call.encode(state.options.max_query_request_bytes)?); - let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; - state.admit(®istry)?; - if !registry - .groups - .get(scope) - .or_else(|| registry.query_groups.get(scope)) - .is_some_and(|r| Arc::ptr_eq(r, &route)) - || !registry - .pools - .get(&peer.replica.node) - .is_some_and(|p| Arc::ptr_eq(p, &pool)) - { - return Err(TransportError::StalePeer); - } - let (reply, result) = oneshot::channel(); - let (cancel, cancelled) = watch::channel(false); - pool.query - .try_send(SendQuery::Legacy(Box::new(LegacySendQuery { - scope: scope.clone(), - route, - peer, - call, - deadline, - keys, - cancel: cancelled, - reply, - node, - peer_bytes, - }))) - .map_err(|e| match e { - mpsc::error::TrySendError::Full(_) => TransportError::Overloaded, - mpsc::error::TrySendError::Closed(_) => TransportError::Closed, - })?; - Ok(QueryDelivery { - result: Some(result), - cancel, - }) - } } pub(super) fn reservation( @@ -398,63 +265,11 @@ impl State { if wire.requires_command_grant() { return self.authorize_cell_command(wire); } - if wire.is_shared() { - return self.authorize_cell_query(wire); - } - let (source, target) = wire.decode()?; - let registry = self.registry.lock().map_err(|_| TransportError::Closed)?; - self.admit(®istry)?; - let route = registry - .groups - .get(&target.scope) - .ok_or(TransportError::UnknownGroup)?; - let group = self - .runtime - .log() - .group(&target.scope) - .map_err(|_| TransportError::UnknownGroup)?; - if *group.identity() != target - || wire.target_label() != self.auth.source() - || wire.schema() != route.schema_digest - { - return Err(TransportError::StalePeer); - } - let member_peer = route.peers.iter().find(|peer| { - peer.replica.node_identity() == source && peer.label == wire.source_label() - }); - let cache = group - .read() - .map_err(|error| TransportError::Runtime(error.into()))?; - if !cache.membership.permits(&target) { - return Err(TransportError::StalePeer); - } - let conf = &cache.conf_state; - let present = |member| { - conf.voters - .iter() - .chain(&conf.voters_outgoing) - .chain(&conf.learners) - .chain(&conf.learners_next) - .any(|id| *id == member) - }; - if !present(target.member.get()) { - return Err(TransportError::StalePeer); - } - let implicit = member_peer.filter(|peer| { - present(peer.replica.member.get()) && cache.membership.permits(&peer.replica) - }); - self.authorize_query_origin( - ®istry, - target.scope.cluster, - target.scope.namespace, - source, - wire.source_label(), - implicit, - ) + self.authorize_cell_query(wire) } - /// Shared and temporary legacy queries have exactly the same read-only - /// origin grant, process binding, revocation and peer-pool permission path. + /// Every cell read uses the same read-only origin grant, process binding, + /// revocation and peer-pool permission path. pub(super) fn authorize_query_origin( &self, registry: &Registry, @@ -495,28 +310,6 @@ impl State { Ok((pool, permission)) } - fn current_query_route( - &self, - scope: &GroupScope, - route: &Arc, - peer: &RaftPeer, - ) -> bool { - if self.runtime.log().group(scope).is_ok() { - return self.current_route(scope, route, peer); - } - self.registry.lock().is_ok_and(|registry| { - self.admit(®istry).is_ok() - && registry - .query_groups - .get(scope) - .is_some_and(|view| Arc::ptr_eq(view, route)) - && registry - .pools - .get(&peer.replica.node) - .is_some_and(|pool| pool.peer.same_host(peer)) - }) - } - pub(super) async fn query_until( &self, pool: &Pool, @@ -725,30 +518,6 @@ impl State { } let request = crate::asyncrt::select! { _ = stop.changed() => return, _ = peer_stop.changed() => return, request = requests.recv() => match request { Some(request) => request, None => return } }; let failed = match request { - SendQuery::Legacy(mut request) => { - let result = self - .query_until( - &pool, - request.deadline, - request.cancel.clone(), - None, - self.send_legacy_query(&mut request, &mut connection), - ) - .await; - let failed = result.is_err(); - let LegacySendQuery { - reply, - node, - peer_bytes, - .. - } = *request; - let _ = reply.send(result.map(|value| QueryResponse { - value, - _node: node, - _peer: peer_bytes, - })); - failed - } SendQuery::Write(request) => { self.send_cell_write(&pool, *request, &mut connection).await } @@ -763,56 +532,6 @@ impl State { } } - async fn send_legacy_query( - &self, - request: &mut LegacySendQuery, - connection: &mut Option, - ) -> Result, TransportError> { - if !self.current_query_route(&request.scope, &request.route, &request.peer) { - return Err(TransportError::StalePeer); - } - let remaining = request - .deadline - .checked_sub(self.domain.mono_ms().map_err(|_| TransportError::Closed)?) - .filter(|ms| *ms > 0) - .ok_or(TransportError::Timeout)?; - request.call.options.timeout = request - .call - .options - .timeout - .min(std::time::Duration::from_millis(remaining)); - let body = request.call.encode(self.options.max_query_request_bytes)?; - let local = self.runtime.log().limits(); - let route = QueryRoute::new( - NodeIdentity { - node: local.node, - disk: local.disk, - process: local.process, - }, - self.auth.source(), - &request.peer.replica, - &request.peer.label, - request.route.schema_digest, - )?; - let body = self - .exchange_query(connection, &RaftPeerNode::from(&request.peer), route, body) - .await?; - if !self.current_query_route(&request.scope, &request.route, &request.peer) { - return Err(TransportError::StalePeer); - } - let now = - u64::try_from(crate::asyncrt::wall_ms()).map_err(|_| TransportError::Authentication)?; - wire::decode_result( - &body, - &request.scope, - request.route.schema_digest, - &request.keys, - now, - self.options.max_query_response_bytes, - request.peer.replica.member, - ) - } - pub(super) async fn exchange_query( &self, connection: &mut Option, diff --git a/crates/cellar/durability/raft/query_wire.rs b/crates/cellar/durability/raft/query_wire.rs index ec84c03bf..181048d95 100644 --- a/crates/cellar/durability/raft/query_wire.rs +++ b/crates/cellar/durability/raft/query_wire.rs @@ -8,14 +8,12 @@ use super::transport::TransportError; use super::transport_frame::{FrameHead, PREFIX}; use super::transport_options::TransportOptions; use crate::application::consensus::{ - CommandFailure, OperationContract, OperationKind, QueryConsistency, QueryOptions, QueryResult, - QueryRouting, StaleReadPolicy, VersionKeys, VersionToken, + CommandFailure, OperationContract, OperationKind, VersionKeys, }; use crate::peer_auth::PeerAuth; -use cellar_logic::consensus::{Durability, GroupScope, MemberId}; +use cellar_logic::consensus::{Durability, MemberId}; use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; -use std::time::Duration; #[path = "query_cell_wire.rs"] pub(super) mod cell; @@ -44,176 +42,6 @@ fn read_durability(body: &mut Reader<'_>) -> Result pub(super) const EXPORT_LABEL: &[u8] = b"EXPORTER-cellar-raft-query-1"; const REQUEST: &[u8] = b"cellar-raft-query-request-1"; const RESPONSE: &[u8] = b"cellar-raft-query-response-1"; -const MAX_TOKEN: usize = 1878; - -pub(super) struct QueryCall { - pub contract: OperationContract, - pub input: Vec, - pub options: QueryOptions, -} - -impl QueryCall { - pub fn encode(&self, limit: usize) -> Result, TransportError> { - if self.contract.kind != OperationKind::Query - || self.contract.operation.is_empty() - || self.contract.operation.len() > 128 - || self.contract.operation.chars().any(char::is_control) - || self.options.timeout.as_millis() == 0 - || self.options.timeout > Duration::from_secs(3600) - || !matches!(self.options.routing, QueryRouting::Local) - { - return Err(TransportError::Protocol); - } - let mut body = b"CLRQRQ01".to_vec(); - string(&mut body, &self.contract.operation)?; - body.extend_from_slice(&self.contract.digest); - match &self.options.consistency { - QueryConsistency::Stale => body.push(0), - QueryConsistency::AtLeast(token) => { - body.push(1); - string(&mut body, token.as_str())?; - } - QueryConsistency::Linearizable => body.push(2), - } - body.push(match self.options.stale_policy { - StaleReadPolicy::RecentSource => 0, - StaleReadPolicy::DisconnectedHistorical => 1, - }); - body.extend_from_slice(&(self.options.timeout.as_millis() as u64).to_be_bytes()); - let length = u32::try_from(self.input.len()).map_err(|_| TransportError::Limit)?; - if body - .len() - .checked_add(4) - .and_then(|n| n.checked_add(self.input.len())) - .is_none_or(|n| n > limit) - { - return Err(TransportError::Limit); - } - body.extend_from_slice(&length.to_be_bytes()); - body.extend_from_slice(&self.input); - Ok(body) - } - - pub fn decode(bytes: &[u8], limit: usize) -> Result { - if bytes.len() > limit { - return Err(TransportError::Limit); - } - let mut body = Reader(bytes); - if body.take(8)? != b"CLRQRQ01" { - return Err(TransportError::Protocol); - } - let operation = body.string(128)?; - if operation.is_empty() || operation.chars().any(char::is_control) { - return Err(TransportError::Protocol); - } - let contract = OperationContract { - operation, - kind: OperationKind::Query, - digest: body.array()?, - }; - let consistency = match body.byte()? { - 0 => QueryConsistency::Stale, - 1 => QueryConsistency::AtLeast( - VersionToken::from_encoded(&body.string(MAX_TOKEN)?) - .map_err(|_| TransportError::Protocol)?, - ), - 2 => QueryConsistency::Linearizable, - _ => return Err(TransportError::Protocol), - }; - let stale_policy = match body.byte()? { - 0 => StaleReadPolicy::RecentSource, - 1 => StaleReadPolicy::DisconnectedHistorical, - _ => return Err(TransportError::Protocol), - }; - let millis = u64::from_be_bytes(body.array()?); - if millis == 0 || millis > 3_600_000 { - return Err(TransportError::Protocol); - } - let length = u32::from_be_bytes(body.array()?) as usize; - let input = body.take(length)?.to_vec(); - if !body.0.is_empty() { - return Err(TransportError::Protocol); - } - Ok(Self { - contract, - input, - options: QueryOptions { - consistency, - stale_policy, - timeout: Duration::from_millis(millis), - routing: QueryRouting::Local, - }, - }) - } -} - -pub(super) fn encode_result( - result: &Result, - limit: usize, -) -> Result, TransportError> { - let mut body = b"CLRQRS01".to_vec(); - match result { - Ok(result) => { - body.push(0); - string(&mut body, result.version_token.as_str())?; - let length = u32::try_from(result.result.len()).map_err(|_| TransportError::Limit)?; - if body - .len() - .checked_add(4) - .and_then(|n| n.checked_add(result.result.len())) - .is_none_or(|n| n > limit) - { - return Err(TransportError::Limit); - } - body.extend_from_slice(&length.to_be_bytes()); - body.extend_from_slice(&result.result); - } - Err(failure) => body.push(failure_code(failure)), - } - if body.len() > limit { - return Err(TransportError::Limit); - } - Ok(body) -} - -pub(super) fn decode_result( - bytes: &[u8], - scope: &GroupScope, - schema: [u8; 32], - keys: &VersionKeys, - now_ms: u64, - limit: usize, - served_by: MemberId, -) -> Result, TransportError> { - if bytes.len() > limit { - return Err(TransportError::Limit); - } - let mut body = Reader(bytes); - if body.take(8)? != b"CLRQRS01" { - return Err(TransportError::Protocol); - } - let result = match body.byte()? { - 0 => { - let version_token = VersionToken::from_encoded(&body.string(MAX_TOKEN)?) - .map_err(|_| TransportError::Authentication)?; - let version = keys - .verify(&version_token, scope, schema, now_ms) - .map_err(|_| TransportError::Authentication)?; - let length = u32::from_be_bytes(body.array()?) as usize; - Ok(QueryResult { - result: body.take(length)?.to_vec(), - version, - version_token, - served_by, - }) - } - status => Err(decode_failure(status)?), - }; - if !body.0.is_empty() { - return Err(TransportError::Protocol); - } - Ok(result) -} fn failure_code(failure: &CommandFailure) -> u8 { match failure { diff --git a/crates/cellar/durability/raft/query_wire_tests.rs b/crates/cellar/durability/raft/query_wire_tests.rs index c9353392f..4eeb65ef0 100644 --- a/crates/cellar/durability/raft/query_wire_tests.rs +++ b/crates/cellar/durability/raft/query_wire_tests.rs @@ -1,15 +1,17 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. use super::*; -use crate::application::consensus::{CommittedVersion, VersionSigningKey}; +use crate::application::consensus::VersionSigningKey; +use crate::raft::cell_groups::CellQueryConsistency; +use cellar_logic::cell_group::{CellHistory, CellId, GroupHistory}; use cellar_logic::consensus::*; +use std::time::Duration; -fn identity(member: u64) -> ReplicaIdentity { +fn identity(member: u64) -> ReplicaIdentity { ReplicaIdentity { - scope: GroupScope { + scope: GroupHistory { cluster: ClusterId::new([1; 16]).unwrap(), namespace: NamespaceId::new([2; 16]).unwrap(), - cell: "Counter:one".into(), group: GroupId::new([3; 16]).unwrap(), incarnation: IncarnationId::new([4; 16]).unwrap(), }, @@ -20,12 +22,22 @@ fn identity(member: u64) -> ReplicaIdentity { } } +fn cell_history() -> CellHistory { + CellHistory { + cluster: identity(1).scope.cluster, + namespace: identity(1).scope.namespace, + cell: CellId::new("Counter:one").unwrap(), + incarnation: IncarnationId::new([5; 16]).unwrap(), + } +} + fn route() -> Route { - Route::new( + Route::new_shared( identity(1).node_identity(), "node-1", &identity(2), "node-2", + &cell_history(), [9; 32], ) .unwrap() @@ -40,62 +52,94 @@ fn keys() -> VersionKeys { .unwrap() } -fn call(consistency: QueryConsistency) -> QueryCall { - QueryCall { - contract: OperationContract { - operation: "Get".into(), - kind: OperationKind::Query, - digest: [11; 32], - }, - input: vec![1, 2, 3], - options: QueryOptions { - consistency, - timeout: Duration::from_millis(500), - ..Default::default() - }, - } +fn result_bytes() -> Vec { + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); + let mut bytes = b"CLGCQS01\0".to_vec(); + string(&mut bytes, token.as_str()).unwrap(); + bytes.extend_from_slice(&3u32.to_be_bytes()); + bytes.extend_from_slice(&[8, 7, 6]); + bytes } -fn result() -> QueryResult { - let version = CommittedVersion { - scope: identity(1).scope, - position: LogPosition { - index: LogIndex::new(7), - term: Term::new(2), - }, - }; - QueryResult { - version_token: keys().issue(&version, [9; 32], 1000).unwrap(), - result: vec![8, 7, 6], - version, - served_by: MemberId::new(2).unwrap(), +// Canonical superseded values are test inputs, never production readers. +fn legacy_scope() -> GroupScope { + let group = identity(1).scope; + GroupScope { + cluster: group.cluster, + namespace: group.namespace, + cell: "Counter:one".into(), + group: group.group, + incarnation: group.incarnation, } } +fn legacy_route() -> serde_json::Value { + let mut value = serde_json::to_value(route()).unwrap(); + value["scope"] = serde_json::json!({ + "kind": "Cell", + "identity": crate::application::consensus::protocol::WireScope::from(&legacy_scope()), + }); + value +} + +fn legacy_request() -> Vec { + let mut body = b"CLRQRQ01".to_vec(); + string(&mut body, "Get").unwrap(); + body.extend_from_slice(&[11; 32]); + body.extend_from_slice(&[0, 0]); // Stale, RecentSource. + body.extend_from_slice(&500u64.to_be_bytes()); + body.extend_from_slice(&3u32.to_be_bytes()); + body.extend_from_slice(&[1, 2, 3]); + body +} + +fn legacy_reply() -> Vec { + let token = keys() + .issue( + &CommittedVersion { + scope: legacy_scope(), + position: LogPosition { + index: LogIndex::new(7), + term: Term::new(2), + }, + }, + [9; 32], + 1000, + ) + .unwrap(); + let mut body = b"CLRQRS01\0".to_vec(); + string(&mut body, token.as_str()).unwrap(); + body.extend_from_slice(&3u32.to_be_bytes()); + body.extend_from_slice(&[8, 7, 6]); + body +} + #[test] fn query_wire_preserves_explicit_constraints() { + let token = keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap(); for consistency in [ - QueryConsistency::Stale, - QueryConsistency::AtLeast(result().version_token), - QueryConsistency::Linearizable, + CellQueryConsistency::Stale, + CellQueryConsistency::RecentSource { max_age_ms: 5000 }, + CellQueryConsistency::AtLeast(token), + CellQueryConsistency::Linearizable, ] { - for stale_policy in [ - StaleReadPolicy::RecentSource, - StaleReadPolicy::DisconnectedHistorical, - ] { - let mut call = call(consistency.clone()); - call.options.stale_policy = stale_policy; - let bytes = call.encode(4096).unwrap(); - let decoded = QueryCall::decode(&bytes, 4096).unwrap(); - assert_eq!(decoded.contract, call.contract); - assert_eq!(decoded.input, call.input); - assert_eq!(decoded.options.consistency, consistency); - assert_eq!(decoded.options.timeout, call.options.timeout); - assert_eq!(decoded.options.stale_policy, stale_policy); - let mut trailing = bytes; - trailing.push(0); - assert!(QueryCall::decode(&trailing, 4096).is_err()); - } + let call = cell_call(consistency); + let bytes = call.encode(4096).unwrap(); + let decoded = cell::CellQueryCall::decode(&bytes, 4096).unwrap(); + assert_eq!(decoded.encode(4096).unwrap(), bytes); + assert_eq!(decoded.request.contract, call.request.contract); + assert_eq!(decoded.request.input, call.request.input); + assert_eq!(decoded.request.invocation, call.request.invocation); + assert_eq!(decoded.timeout_ms, call.timeout_ms); + assert_eq!(decoded.route_epoch, call.route_epoch); + assert_eq!(decoded.birth_digest, call.birth_digest); + let mut trailing = bytes; + trailing.push(0); + assert!(cell::CellQueryCall::decode(&trailing, 4096).is_err()); } } @@ -103,7 +147,9 @@ fn query_wire_preserves_explicit_constraints() { fn query_wire_binds_channel_request_route_and_payload() { let auth = PeerAuth::new([17; 32], "node-1").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Linearizable).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); let request = RequestHead::new(route(), 7, &body); let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); assert!(read_request(&head, &auth, &[3; 32], &options).is_ok()); @@ -113,8 +159,7 @@ fn query_wire_binds_channel_request_route_and_payload() { modified[index] ^= 1; assert!(request.verify_body(&modified).is_err()); } - let value = result(); - let reply = encode_result(&Ok(value.clone()), 4096).unwrap(); + let reply = result_bytes(); let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); assert_eq!( read_response(&response, &auth, &[3; 32], &request, &head.tag, &options).unwrap(), @@ -122,21 +167,21 @@ fn query_wire_binds_channel_request_route_and_payload() { ); assert!(read_response(&response, &auth, &[4; 32], &request, &head.tag, &options).is_err()); assert!(read_response(&response, &auth, &[3; 32], &request, &[5; 32], &options).is_err()); - let other_request = RequestHead::new(route(), 8, &body); assert!(read_response( &response, &auth, &[3; 32], - &other_request, + &RequestHead::new(route(), 8, &body), &head.tag, &options ) .is_err()); - let other_route = Route::new( + let other_route = Route::new_shared( identity(1).node_identity(), "node-1", &identity(3), "node-3", + &cell_history(), [9; 32], ) .unwrap(); @@ -149,42 +194,28 @@ fn query_wire_binds_channel_request_route_and_payload() { &options ) .is_err()); - assert_eq!( - decode_result( - &reply, - &value.version.scope, - [9; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) + let member = MemberId::new(2).unwrap(); + let cell = cell_history(); + let value = cell::decode_cell_result(&reply, &cell, [9; 32], &keys(), 1000, 4096, member) .unwrap() - .unwrap(), - value + .unwrap(); + assert_eq!(value.observation, cell_observation()); + assert_eq!( + value.token, + keys() + .issue_cell(&cell_observation(), [9; 32], 1000) + .unwrap() ); - assert!(decode_result( - &reply, - &value.version.scope, - [10; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) - .is_err()); - let mut wrong = value.version.scope.clone(); + assert_eq!(value.result, [8, 7, 6]); + assert_eq!(value.served_by, member); + assert!( + cell::decode_cell_result(&reply, &cell, [10; 32], &keys(), 1000, 4096, member).is_err() + ); + let mut wrong = cell; wrong.incarnation = IncarnationId::new([99; 16]).unwrap(); - assert!(decode_result( - &reply, - &wrong, - [9; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) - .is_err()); + assert!( + cell::decode_cell_result(&reply, &wrong, [9; 32], &keys(), 1000, 4096, member).is_err() + ); } #[test] @@ -195,14 +226,23 @@ fn query_origin_binds_node_process_without_voting_identity() { process: ProcessGeneration::new([93; 16]).unwrap(), }; let target = identity(2); - let route = Route::new(origin, "ingress", &target, "node-2", [9; 32]).unwrap(); - assert_eq!(route.decode().unwrap(), (origin, target.clone())); - assert!(route.reversed().decode().is_err()); + let route = Route::new_shared( + origin, + "ingress", + &target, + "node-2", + &cell_history(), + [9; 32], + ) + .unwrap(); + assert_eq!( + route.decode_shared().unwrap(), + (origin, target.clone(), cell_history()) + ); + assert!(route.reversed().decode_shared().is_err()); let encoded = serde_json::to_value(&route).unwrap(); assert!(encoded["source"]["member"].is_null()); assert_eq!(encoded["target"]["member"], "2"); - // Even an authenticated query origin cannot reuse its route as the - // member-bearing route required for a consensus message. assert!( serde_json::from_value::(encoded.clone()).is_err() ); @@ -216,31 +256,33 @@ fn query_origin_binds_node_process_without_voting_identity() { modified["target"]["member"] = member; assert!(serde_json::from_value::(modified) .unwrap() - .decode() + .decode_shared() .is_err()); } let mut voting_origin = encoded.clone(); voting_origin["source"]["member"] = serde_json::json!("1"); assert!(serde_json::from_value::(voting_origin) .unwrap() - .decode() + .decode_shared() .is_err()); for endpoint in ["source", "target"] { for field in ["node", "disk", "process"] { let mut modified = encoded.clone(); - modified[endpoint][field] = serde_json::to_value([0_u8; 16]).unwrap(); + modified[endpoint][field] = serde_json::to_value([0u8; 16]).unwrap(); assert!(serde_json::from_value::(modified) .unwrap() - .decode() + .decode_shared() .is_err()); } } let auth = PeerAuth::new([17; 32], "ingress").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Linearizable).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Linearizable) + .encode(4096) + .unwrap(); let request = RequestHead::new(route, 7, &body); let head = request_head(&auth, &[3; 32], &request, &body, &options).unwrap(); - let reply = encode_result(&Ok(result()), 4096).unwrap(); + let reply = result_bytes(); let response = response_head(&auth, &[3; 32], &request, &head.tag, &reply, &options).unwrap(); let mut changed_origins = Vec::new(); for field in 0..3 { @@ -255,7 +297,7 @@ fn query_origin_binds_node_process_without_voting_identity() { changed_origins.push((origin, "other-ingress")); for (changed, label) in changed_origins { let other = RequestHead::new( - Route::new(changed, label, &target, "node-2", [9; 32]).unwrap(), + Route::new_shared(changed, label, &target, "node-2", &cell_history(), [9; 32]).unwrap(), 7, &body, ); @@ -267,7 +309,7 @@ fn query_origin_binds_node_process_without_voting_identity() { fn query_wire_refuses_size_replay_and_consensus_domain_confusion() { let auth = PeerAuth::new([17; 32], "node-1").unwrap(); let options = TransportOptions::default(); - let body = call(QueryConsistency::Stale).encode(4096).unwrap(); + let body = cell_call(CellQueryConsistency::Stale).encode(4096).unwrap(); assert!(request_head( &auth, &[3; 32], @@ -304,14 +346,14 @@ fn query_wire_refuses_size_replay_and_consensus_domain_confusion() { prefix = head.prefix; prefix[12..].copy_from_slice(&u64::MAX.to_be_bytes()); assert!(parse_prefix(&prefix, &options, false).is_err()); - let mut call = call(QueryConsistency::Stale); - call.options.timeout = Duration::ZERO; + let mut call = cell_call(CellQueryConsistency::Stale); + call.timeout_ms = 0; assert!(call.encode(4096).is_err()); - call.options.timeout = Duration::from_millis(500); - call.contract.kind = OperationKind::Command; + call.timeout_ms = 500; + call.request.contract.kind = OperationKind::Command; assert!(call.encode(4096).is_err()); - call.contract.kind = OperationKind::Query; - call.input = vec![0; 4096]; + call.request.contract.kind = OperationKind::Query; + call.request.input = vec![0; 4096]; assert!(call.encode(4096).is_err()); } @@ -327,23 +369,13 @@ fn query_wire_failures_remain_typed() { CommandFailure::StorageFull, CommandFailure::Unavailable, ] { - let bytes = encode_result(&Err(failure.clone()), 4096).unwrap(); - assert_eq!( - decode_result( - &bytes, - &identity(1).scope, - [9; 32], - &keys(), - 1000, - 4096, - MemberId::new(2).unwrap() - ) - .unwrap(), - Err(failure) + let bytes = cell::encode_cell_result(&Err(failure.clone()), 4096).unwrap(); + assert!( + matches!(cell::decode_cell_result(&bytes, &cell_history(), [9; 32], &keys(), 1000, 4096, MemberId::new(2).unwrap()).unwrap(), Err(actual) if actual == failure) ); - assert!(decode_result( + assert!(cell::decode_cell_result( &bytes, - &identity(1).scope, + &cell_history(), [9; 32], &keys(), 1000, @@ -354,11 +386,27 @@ fn query_wire_failures_remain_typed() { } } +#[test] +fn query_hard_cutover_refuses_legacy_route_request_and_reply() { + assert!(serde_json::from_value::(legacy_route()).is_err()); + assert!(cell::CellQueryCall::decode(&legacy_request(), 4096).is_err()); + assert!(cell::decode_cell_result( + &legacy_reply(), + &cell_history(), + [9; 32], + &keys(), + 1000, + 4096, + MemberId::new(2).unwrap() + ) + .is_err()); +} + fn cell_observation() -> crate::raft::version::CellVersionObservation { use cellar_logic::cell_group::*; crate::raft::version::CellVersionObservation { version: CellVersion { - history: CellHistory::try_from(&identity(1).scope).unwrap(), + history: cell_history(), sequence: MutationSequence::new(7).unwrap(), image: CellImage { epoch: 1, @@ -368,7 +416,7 @@ fn cell_observation() -> crate::raft::version::CellVersionObservation { }, }, witness: CommitWitness { - group: GroupHistory::from(&identity(1).scope), + group: identity(1).scope, position: LogPosition { index: LogIndex::new(100), term: Term::new(9), @@ -382,7 +430,11 @@ fn cell_call(consistency: crate::raft::cell_groups::CellQueryConsistency) -> cel cell::CellQueryCall { request: crate::raft::cell_groups::CellQueryRequest { invocation: ProposalId::new([91; 16]).unwrap(), - contract: call(QueryConsistency::Stale).contract, + contract: OperationContract { + operation: "Get".into(), + kind: OperationKind::Query, + digest: [11; 32], + }, input: vec![1, 2, 3], consistency, }, @@ -416,17 +468,12 @@ fn shared_query_wire_preserves_cell_constraints_and_refuses_other_payloads() { for end in 0..bytes.len() { assert!(cell::CellQueryCall::decode(&bytes[..end], 4096).is_err()); } - assert!(QueryCall::decode(&bytes, 4096).is_err()); let mut trailing = bytes.clone(); trailing.push(0); assert!(cell::CellQueryCall::decode(&trailing, 4096).is_err()); assert!(cell::CellQueryCall::decode(&bytes, bytes.len() - 1).is_err()); } - assert!(cell::CellQueryCall::decode( - &call(QueryConsistency::Stale).encode(4096).unwrap(), - 4096 - ) - .is_err()); + assert!(cell::CellQueryCall::decode(&legacy_request(), 4096).is_err()); let mut call = cell_call(CellQueryConsistency::Stale); call.request.contract.kind = OperationKind::Command; assert!(call.encode(4096).is_err()); @@ -444,10 +491,10 @@ fn shared_query_wire_preserves_cell_constraints_and_refuses_other_payloads() { #[test] fn shared_query_route_binds_physical_group_logical_cell_and_process() { use crate::raft::cell_groups::CellQueryConsistency; - use cellar_logic::cell_group::{CellId, GroupHistory}; + use cellar_logic::cell_group::CellId; let old = identity(2); let target = ReplicaIdentity { - scope: GroupHistory::from(&old.scope), + scope: old.scope, node: old.node, disk: old.disk, process: old.process, @@ -460,12 +507,8 @@ fn shared_query_route_binds_physical_group_logical_cell_and_process() { route.decode_shared().unwrap(), (source, target.clone(), cell.clone()) ); - assert!(route.decode().is_err()); assert!(route.reversed().decode_shared().is_err()); - assert!(Route::new(source, "node-1", &old, "node-2", [9; 32]) - .unwrap() - .decode_shared() - .is_err()); + assert!(serde_json::from_value::(legacy_route()).is_err()); let value = serde_json::to_value(&route).unwrap(); assert!(value["source"]["member"].is_null()); assert!(serde_json::from_value::(value.clone()).is_err()); @@ -593,14 +636,14 @@ fn shared_query_result_authenticates_cell_progress_not_group_log_index() { member ) .is_err()); - assert!(decode_result( - &bytes, - &identity(1).scope, + assert!(cell::decode_cell_result( + &legacy_reply(), + &observation.version.history, [9; 32], &keys(), 1000, 4096, - member + member, ) .is_err()); for end in 0..bytes.len() { @@ -693,10 +736,9 @@ fn shared_command_wire_preserves_inputs_and_refuses_query_payloads() { #[test] fn shared_command_purpose_is_authenticated_and_cannot_reuse_read_reply() { - use cellar_logic::cell_group::GroupHistory; let old = identity(2); let target = ReplicaIdentity { - scope: GroupHistory::from(&old.scope), + scope: old.scope, member: old.member, node: old.node, disk: old.disk, @@ -987,7 +1029,7 @@ fn shared_enrollment_wire_preserves_captured_identity_and_inputs() { fn shared_enrollment_purpose_cannot_be_reinterpreted_as_command_or_read() { let old = identity(2); let target = ReplicaIdentity { - scope: cellar_logic::cell_group::GroupHistory::from(&old.scope), + scope: old.scope, member: old.member, node: old.node, disk: old.disk, @@ -1300,7 +1342,7 @@ fn shared_receipt_reply_distinguishes_unknown_from_empty_committed_result() { fn shared_receipt_purpose_cannot_authorize_mutation() { let old = identity(2); let target = ReplicaIdentity { - scope: cellar_logic::cell_group::GroupHistory::from(&old.scope), + scope: old.scope, member: old.member, node: old.node, disk: old.disk, diff --git a/crates/cellar/durability/raft/records.rs b/crates/cellar/durability/raft/records.rs index 391f8115d..8bcbb83c9 100644 --- a/crates/cellar/durability/raft/records.rs +++ b/crates/cellar/durability/raft/records.rs @@ -177,10 +177,10 @@ impl MappingRecord { .as_ref() .is_none_or(|root| root.route() != genesis.route()) || cache.applied.image.is_some() - || cache.membership.fleet_status().is_some() { return Err(LogError::WrongIdentity); } + cache.membership.validate_scope(&history)?; if genesis.recovery_request().is_some() { let current = cache .group_manifest diff --git a/crates/cellar/durability/raft/replica_leadership.rs b/crates/cellar/durability/raft/replica_leadership.rs index c3adf17c6..c8e672ea6 100644 --- a/crates/cellar/durability/raft/replica_leadership.rs +++ b/crates/cellar/durability/raft/replica_leadership.rs @@ -44,76 +44,16 @@ impl Replica { ) -> Result<(), RuntimeError> { let progress = self.replication_status()?; let cache = self.log.read()?; - let voters = &cache.conf_state.voters; - if check.through.get() == 0 - || self.node.raft.pending_conf_index > self.node.raft.raft_log.applied - || self.node.raft.lead_transferee.is_some() - || progress.term != check.expected_term - || progress.membership_revision != check.expected_revision - || progress.committed < check.through - || voters.len() != 3 - || !cache.conf_state.voters_outgoing.is_empty() - || !cache.conf_state.learners.is_empty() - || !cache.conf_state.learners_next.is_empty() - || cache.conf_state.auto_leave - || check.survivors[0] == check.survivors[1] - || check - .survivors - .iter() - .any(|member| !voters.contains(&member.get())) - || check.survivors.iter().any(|member| { - progress - .members - .iter() - .find(|p| p.member == *member) - .is_none_or(|p| { - !p.voter - || p.matched < check.through - || (*member != self.identity().member && !p.recent_active) - }) - }) - { - return Err(RuntimeError::NotReady); - } - if check.all_voters - && voters.iter().any(|id| { - progress - .members - .iter() - .find(|p| p.member.get() == *id) - .is_none_or(|p| { - !p.voter - || p.matched < check.through - || (*id != self.identity().member.get() && !p.recent_active) - }) - }) - { - return Err(RuntimeError::NotReady); - } - let local = self.identity().member; - match &check.transfer { - Some((target, selector)) => { - if check.survivors.contains(&local) - || !check.survivors.contains(target) - || selector.boot.node != self.identity().node.bytes() - || selector.boot.disk != self.identity().disk.bytes() - || selector.boot.process != self.identity().process.bytes() - { - return Err(RuntimeError::NotReady); - } - } - None if !check.survivors.contains(&local) => return Err(RuntimeError::NotReady), - None => (), - } + let transfer = super::super::restart_leadership::check_progress( + &progress, + &cache.conf_state, + &check, + self.node.raft.pending_conf_index > self.node.raft.raft_log.applied, + self.node.raft.lead_transferee.is_some(), + )?; drop(cache); - if let Some((target, _)) = check.transfer { - // The existing owner also requires this destination to be caught up - // to the latest commit, and refuses repair/joint/active transfers. - self.transfer_leadership(LeadershipTransfer { - target, - expected_term: check.expected_term, - expected_revision: check.expected_revision, - })?; + if let Some(transfer) = transfer { + self.transfer_leadership(transfer)?; } Ok(()) } diff --git a/crates/cellar/durability/raft/restart_leadership.rs b/crates/cellar/durability/raft/restart_leadership.rs new file mode 100644 index 000000000..5e73e5156 --- /dev/null +++ b/crates/cellar/durability/raft/restart_leadership.rs @@ -0,0 +1,107 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! One restart-readiness contract, evaluated by the serialized physical owner. +//! Replication observations are inputs; only the owner can admit a transfer. +use super::completion::{ + LeadershipTransfer, ReplicationStatus, RestartLeadershipCheck, RuntimeError, +}; +use super::storage::ReplicaLog; +use ::raft::prelude::ConfState; + +#[cfg(test)] +#[path = "restart_leadership_tests.rs"] +mod tests; + +pub(crate) fn check_gate( + log: &ReplicaLog, + now: u64, + check: &RestartLeadershipCheck, +) -> Result<(), RuntimeError> { + if now >= check.deadline { + return Err(RuntimeError::Closed); + } + if let Some((_, selector)) = &check.transfer { + if log.lifecycle_guard()?.as_ref().and_then(|g| g.quiescing()) != Some(selector) { + return Err(RuntimeError::NotReady); + } + } + Ok(()) +} + +pub(crate) fn check_progress( + progress: &ReplicationStatus, + configuration: &ConfState, + check: &RestartLeadershipCheck, + pending_configuration: bool, + transferring: bool, +) -> Result, RuntimeError> { + let voters = &configuration.voters; + if check.through.get() == 0 + || pending_configuration + || transferring + || progress.term != check.expected_term + || progress.membership_revision != check.expected_revision + || progress.committed < check.through + || voters.len() != 3 + || !configuration.voters_outgoing.is_empty() + || !configuration.learners.is_empty() + || !configuration.learners_next.is_empty() + || configuration.auto_leave + || check.survivors[0] == check.survivors[1] + || check + .survivors + .iter() + .any(|member| !voters.contains(&member.get())) + || check.survivors.iter().any(|member| { + progress + .members + .iter() + .find(|p| p.member == *member) + .is_none_or(|p| { + !p.voter + || p.matched < check.through + || (*member != progress.replica.member && !p.recent_active) + }) + }) + { + return Err(RuntimeError::NotReady); + } + if check.all_voters + && voters.iter().any(|id| { + progress + .members + .iter() + .find(|p| p.member.get() == *id) + .is_none_or(|p| { + !p.voter + || p.matched < check.through + || (*id != progress.replica.member.get() && !p.recent_active) + }) + }) + { + return Err(RuntimeError::NotReady); + } + let local = progress.replica.member; + match &check.transfer { + Some((target, selector)) => { + if check.survivors.contains(&local) + || !check.survivors.contains(target) + || selector.boot.node != progress.replica.node.bytes() + || selector.boot.disk != progress.replica.disk.bytes() + || selector.boot.process != progress.replica.process.bytes() + { + return Err(RuntimeError::NotReady); + } + } + None if !check.survivors.contains(&local) => return Err(RuntimeError::NotReady), + None => (), + } + Ok(check + .transfer + .as_ref() + .map(|(target, _)| LeadershipTransfer { + target: *target, + expected_term: check.expected_term, + expected_revision: check.expected_revision, + })) +} diff --git a/crates/cellar/durability/raft/restart_leadership_tests.rs b/crates/cellar/durability/raft/restart_leadership_tests.rs new file mode 100644 index 000000000..503545dae --- /dev/null +++ b/crates/cellar/durability/raft/restart_leadership_tests.rs @@ -0,0 +1,240 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Metadata checks grant no physical group, quorum or lifecycle authority. +use super::*; +use crate::raft::completion::MemberProgress; +use crate::raft::fleet_stop::{BootBinding, LifecyclePurpose, LifecycleUpdate, StopSelector}; +use cellar_logic::cell_group::GroupHistory; +use cellar_logic::consensus::*; + +fn progress() -> ReplicationStatus { + ReplicationStatus { + replica: ReplicaIdentity { + scope: GroupHistory { + cluster: ClusterId::new([4; 16]).unwrap(), + namespace: NamespaceId::new([5; 16]).unwrap(), + group: GroupId::new([6; 16]).unwrap(), + incarnation: IncarnationId::new([7; 16]).unwrap(), + }, + node: NodeId::new([1; 16]).unwrap(), + disk: DiskLineage::new([2; 16]).unwrap(), + process: ProcessGeneration::new([3; 16]).unwrap(), + member: MemberId::new(1).unwrap(), + }, + term: Term::new(8), + membership_revision: LogIndex::new(9), + committed: LogIndex::new(10), + members: (1..=3) + .map(|id| MemberProgress { + member: MemberId::new(id).unwrap(), + matched: LogIndex::new(10), + voter: true, + recent_active: true, + }) + .collect(), + } +} + +fn configuration() -> ConfState { + ConfState { + voters: vec![1, 2, 3], + ..Default::default() + } +} + +fn check() -> RestartLeadershipCheck { + RestartLeadershipCheck { + deadline: 1000, + expected_term: Term::new(8), + expected_revision: LogIndex::new(9), + through: LogIndex::new(10), + survivors: [MemberId::new(1).unwrap(), MemberId::new(2).unwrap()], + all_voters: true, + transfer: None, + } +} + +#[test] +fn group_restart_observation_requires_third_voter_matched_and_active() { + let mut progress = progress(); + let conf = configuration(); + let mut check = check(); + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); + progress.members[2].matched = LogIndex::new(9); + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + check.all_voters = false; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); + check.all_voters = true; + progress.members[2].matched = LogIndex::new(10); + progress.members[2].recent_active = false; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + progress.members[2].recent_active = true; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Ok(None) + )); +} + +#[test] +fn group_restart_readiness_rejects_changed_term_revision_and_configuration() { + for scenario in 0..8 { + let mut progress = progress(); + let mut conf = configuration(); + let mut check = check(); + let mut pending = false; + let mut transferring = false; + match scenario { + 0 => progress.term = Term::new(9), + 1 => progress.membership_revision = LogIndex::new(10), + 2 => conf.voters_outgoing = vec![1, 2, 3], + 3 => conf.learners = vec![4], + 4 => pending = true, + 5 => transferring = true, + 6 => check.survivors[1] = check.survivors[0], + _ => progress.committed = LogIndex::new(9), + } + assert!( + matches!( + check_progress(&progress, &conf, &check, pending, transferring), + Err(RuntimeError::NotReady) + ), + "accepted stale readiness scenario {scenario}" + ); + } +} + +fn selector(boot: BootBinding) -> StopSelector { + StopSelector { + purpose: LifecyclePurpose::Maintenance, + boot, + epoch: 2, + operation: [11; 16], + finalization: [12; 16], + request: [13; 16], + } +} + +#[test] +fn group_restart_transfer_binds_source_process_and_survivor_roles() { + let progress = progress(); + let conf = configuration(); + let mut check = check(); + let boot = BootBinding { + catalog: [14; 32], + fleet: [15; 16], + node: progress.replica.node.bytes(), + disk: progress.replica.disk.bytes(), + engine: [16; 16], + boot: [17; 16], + process: progress.replica.process.bytes(), + epoch: 1, + join_key: [18; 32], + }; + check.survivors = [MemberId::new(2).unwrap(), MemberId::new(3).unwrap()]; + check.transfer = Some((MemberId::new(2).unwrap(), selector(boot))); + let transfer = check_progress(&progress, &conf, &check, false, false) + .unwrap() + .unwrap(); + assert_eq!(transfer.target, MemberId::new(2).unwrap()); + assert_eq!(transfer.expected_term, progress.term); + check.transfer.as_mut().unwrap().1.boot.process = [19; 16]; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); + check.transfer.as_mut().unwrap().1.boot.process = progress.replica.process.bytes(); + check.survivors[0] = progress.replica.member; + assert!(matches!( + check_progress(&progress, &conf, &check, false, false), + Err(RuntimeError::NotReady) + )); +} + +#[tokio::test] +async fn group_restart_gate_requires_exact_synced_quiescence_and_rejects_fence( +) -> anyhow::Result<()> { + use crate::asyncrt::ExecutionDomain; + use crate::raft::options::{LogOpenMode, ReplicaLogOptions}; + let root = tempfile::tempdir()?; + let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + root.path().join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([2; 16])?, + ProcessGeneration::new([3; 16])?, + LogOpenMode::Create, + ), + std::sync::Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + let deadline = log.domain().mono_ms()? + 5000; + let boot = BootBinding { + catalog: [14; 32], + fleet: [15; 16], + node: log.limits().node.bytes(), + disk: log.limits().disk.bytes(), + process: log.limits().process.bytes(), + engine: log.storage_instance()?.bytes(), + boot: [17; 16], + epoch: 1, + join_key: crate::peer_probe::PeerProbeSigner::from_secret([18; 32]).public_key_bytes(), + }; + log.update_lifecycle(LifecycleUpdate::Boot { + boot: boot.clone(), + deadline, + }) + .await?; + let selector = selector(boot.clone()); + let mut check = check(); + check.deadline = deadline; + check.transfer = Some((MemberId::new(2)?, selector.clone())); + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + log.update_lifecycle(LifecycleUpdate::Quiesce { + selector: selector.clone(), + deadline, + }) + .await?; + check_gate(&log, log.domain().mono_ms()?, &check)?; + check.transfer.as_mut().unwrap().1.request = [20; 16]; + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + check.transfer.as_mut().unwrap().1 = selector.clone(); + log.update_lifecycle(LifecycleUpdate::Fence { + boot, + epoch: selector.epoch, + operation: selector.operation, + request: [21; 16], + deadline, + }) + .await?; + assert!(matches!( + check_gate(&log, log.domain().mono_ms()?, &check), + Err(RuntimeError::NotReady) + )); + assert!(matches!( + check_gate(&log, deadline, &check), + Err(RuntimeError::Closed) + )); + log.close().await?; + domain.shutdown().await; + Ok(()) +} diff --git a/crates/cellar/durability/raft/runtime.rs b/crates/cellar/durability/raft/runtime.rs index 9c2af1240..a8edaee06 100644 --- a/crates/cellar/durability/raft/runtime.rs +++ b/crates/cellar/durability/raft/runtime.rs @@ -1158,29 +1158,13 @@ impl Worker { // catalog cancellation CAS was lost. Read-only observation // on a surviving leader needs no local quiescence. let gate = match &action { - GroupAction::RestartLeadership(check) - if self - .domain - .mono_ms() - .map_or(true, |now| now >= check.deadline) => - { - Err(RuntimeError::Closed) - } - GroupAction::RestartLeadership(check) => match &check.transfer { - Some((_, selector)) => self - .log - .lifecycle_guard() - .map_err(RuntimeError::from) - .and_then(|guard| { - if guard.as_ref().and_then(|g| g.quiescing()) == Some(selector) - { - Ok(()) - } else { - Err(RuntimeError::NotReady) - } - }), - None => Ok(()), - }, + GroupAction::RestartLeadership(check) => self + .domain + .mono_ms() + .map_err(|_| RuntimeError::Closed) + .and_then(|now| { + super::restart_leadership::check_gate(&self.log, now, check) + }), _ => Ok(()), }; let result = if new_read && pending_reads >= self.options.max_pending_reads { diff --git a/crates/cellar/durability/raft/seed_transport.rs b/crates/cellar/durability/raft/seed_transport.rs index 4721a84a7..b6a7c4cf0 100644 --- a/crates/cellar/durability/raft/seed_transport.rs +++ b/crates/cellar/durability/raft/seed_transport.rs @@ -69,7 +69,6 @@ impl SourceSeed { } use super::snapshots::{read_bulk_head, Wire}; -use crate::application::consensus::RaftCells; use sha2::{Digest, Sha256}; use std::sync::Arc; @@ -576,117 +575,6 @@ impl State { .map_err(|_| TransportError::Closed)?; crate::asyncrt::select! {_=timer=>Err(TransportError::Timeout),result=self.bounded(None,future)=>result} } - - pub(super) async fn receive_seed( - &self, - stream: &mut S, - exporter: &[u8; 32], - prefix: [u8; frame::PREFIX], - ) -> Result<(), TransportError> { - let metadata = Arc::new( - self.seed_metadata - .acquire(META_BYTES) - .ok_or(TransportError::Overloaded)?, - ); - let head = self - .bounded( - None, - read_bulk_head(stream, Some(prefix), REQUEST_MAGIC, 0, &self.options), - ) - .await?; - let wire = Wire { - auth: &self.auth, - exporter, - options: &self.options, - max_body: SNAPSHOT_META_BYTES, - }; - let request: SeedRequest = wire.decode(REQUEST_MAGIC, REQUEST_PURPOSE, &head, None)?; - if frame::decimal(&request.nonce)? == 0 - || request.timeout_ms == 0 - || request.timeout_ms > 3_600_000 - { - return Err(TransportError::Protocol); - } - self.authorize_route(&request.route)?; - let (joining, source) = request.route.decode()?; - let group = self - .runtime - .log() - .group(&source.scope) - .map_err(|_| TransportError::UnknownGroup)?; - if !group - .read() - .map_err(|e| TransportError::Runtime(e.into()))? - .conf_state - .learners - .contains(&joining.member.get()) - { - return Err(TransportError::Refused); - } - let handle = self - .query_gateway - .lock() - .map_err(|_| TransportError::Closed)? - .as_ref() - .and_then(|h| h.upgrade()) - .ok_or(TransportError::Refused)?; - let gateway = RaftCells { handle }; - let cell = gateway - .cell(source.scope.namespace, &source.scope.cell) - .map_err(|_| TransportError::Refused)?; - if cell.scope() != &source.scope - || cell.binding.class.schema_digest() != request.route.schema() - { - return Err(TransportError::Identity); - } - let deadline = self - .domain - .mono_ms() - .map_err(|_| TransportError::Closed)? - .saturating_add(request.timeout_ms); - let seed = cell.prepare_seed( - joining, - request.kind, - metadata.clone(), - std::time::Duration::from_millis(request.timeout_ms), - ); - let mut extra = [0; 1]; - let result=self.seed_until(deadline,async { - crate::asyncrt::select! {result=seed=>Ok(result),read=stream.read(&mut extra)=>{let _=read?;Err(TransportError::Closed)}} - }).await?; - self.authorize_route(&request.route)?; - let (status, barrier, lease, body) = match result { - Ok(seed) if seed.kind() == request.kind => { - (0, seed.barrier.get(), seed.lease(), seed.body()?) - } - Ok(_) => return Err(TransportError::Protocol), - Err(crate::application::consensus::CommandFailure::Overloaded) => (1, 0, 0, vec![]), - Err(_) => (2, 0, 0, vec![]), - }; - let reply = SeedReply { - route: request.route.reversed(), - request: head.tag, - nonce: request.nonce, - status, - kind: request.kind, - barrier: barrier.to_string(), - lease: lease.to_string(), - digest: Sha256::digest(&body).into(), - }; - let response = wire.encode(RESPONSE_MAGIC, RESPONSE_PURPOSE, &head.tag, &reply, &body)?; - let result = self - .seed_io(deadline, async { - write_head(stream, &response).await?; - stream.write_all(&body).await?; - stream.flush().await?; - Ok(()) - }) - .await; - if result.is_err() && lease != 0 { - self.finish_seed_source(lease); - } - result - } } #[path = "seed_archive.rs"] diff --git a/crates/cellar/durability/raft/storage.rs b/crates/cellar/durability/raft/storage.rs index 601fed081..f58fcffe5 100644 --- a/crates/cellar/durability/raft/storage.rs +++ b/crates/cellar/durability/raft/storage.rs @@ -261,6 +261,12 @@ struct Request { _permit: Permit, } +struct ManagedCells { + binding: super::member_state::FleetGroupBinding, + allow_genesis: bool, + boot: super::fleet_stop::BootBinding, +} + enum Work { NodeFence(Reply), Lifecycle(Box, Reply), @@ -272,6 +278,7 @@ enum Work { CreateCells( CellGroupBootstrap, Option>, + Option>, Reply>, ), RecoverCells( @@ -675,8 +682,10 @@ impl ReplicaLog { bootstrap: CellGroupBootstrap, ) -> Result, LogError> { bootstrap.configuration()?; - self.request(16 * 1024, |reply| Work::CreateCells(bootstrap, None, reply)) - .await + self.request(16 * 1024, |reply| { + Work::CreateCells(bootstrap, None, None, reply) + }) + .await } pub(crate) async fn stage_cell_group( @@ -686,7 +695,52 @@ impl ReplicaLog { ) -> Result, LogError> { grant.check_log(self)?; self.request(96 * 1024, |reply| { - Work::CreateCells(bootstrap, Some(grant), reply) + Work::CreateCells(bootstrap, Some(grant), None, reply) + }) + .await + } + + /// The catalog-selected initial membership and fleet guard are persisted + /// in the same first engine sync. An existing unguarded history refuses. + pub(crate) async fn create_managed_cell_group( + &self, + bootstrap: CellGroupBootstrap, + authorization: &super::fleet_catalog::FleetGroupAuthorization, + boot: &super::fleet_stop::BootBinding, + ) -> Result, LogError> { + let conf = bootstrap.configuration()?; + if !authorization.matches_scope(&bootstrap.route.history) + || authorization.history_digests() + != (bootstrap.route.birth_digest, bootstrap.route.schema_digest) + { + return Err(LogError::WrongIdentity); + } + let binding = authorization.binding().clone(); + boot.validate()?; + if boot.fleet != binding.fleet() + || boot.catalog != binding.catalog() + || boot.node != self.limits().node.bytes() + || boot.disk != self.limits().disk.bytes() + || boot.engine != self.storage_instance()?.bytes() + || boot.process != self.limits().process.bytes() + || self + .lifecycle_guard()? + .is_none_or(|guard| !guard.admits_group_start(boot)) + { + return Err(LogError::WrongIdentity); + } + super::member_state::GroupMembership::managed_initial(&conf, binding.clone())?; + self.request(24 * 1024, |reply| { + Work::CreateCells( + bootstrap, + None, + Some(Box::new(ManagedCells { + binding, + allow_genesis: authorization.allows_managed_genesis(), + boot: boot.clone(), + })), + reply, + ) }) .await } @@ -1421,7 +1475,19 @@ impl Worker { &mut self, bootstrap: CellGroupBootstrap, seed: Option<&super::transport::CellGroupSeedGrant>, + managed: Option, ) -> Result, LogError> { + if seed.is_some() && managed.is_some() { + return Err(LogError::WrongIdentity); + } + if managed.as_ref().is_some_and(|managed| { + self.lifecycle + .guard + .as_ref() + .is_none_or(|guard| !guard.admits_group_start(&managed.boot)) + }) { + return Err(LogError::WrongIdentity); + } let conf = if let Some(seed) = seed { seed.check( &self.owner, @@ -1434,6 +1500,7 @@ impl Worker { member: bootstrap.member, }, )?; + seed.check_lifecycle(self.lifecycle.guard.as_ref())?; seed.initial_configuration(&bootstrap)? } else { bootstrap.configuration()? @@ -1449,10 +1516,57 @@ impl Worker { if self.mappings.get(&group.key) != Some(&expected) { return Err(LogError::WrongIdentity); } + let cache = group.read()?; + let binding = managed + .as_ref() + .map(|m| &m.binding) + .or_else(|| seed.and_then(|grant| grant.managed_binding())); + let staged = seed + .map(|seed| seed.initial_membership(&conf)) + .transpose()?; + match (binding, cache.membership.fleet_status()) { + (Some(binding), Some(status)) if binding == &status.binding => { + if seed.is_none() + && (!super::member_state::members(&cache.conf_state) + .contains(&bootstrap.member.get()) + || !cache.membership.permits_group(group.identity())) + { + return Err(LogError::WrongIdentity); + } + } + (Some(_), None) + if staged.as_ref() == Some(&cache.membership) + && cache.group_seed.is_none() + && cache.applied.position.index.get() == 1 + && cache.group_manifest.as_ref().is_some_and(|root| { + root.root().cells() == 0 && root.position().index.get() == 1 + }) => + { + () + } + (None, None) => (), + _ => return Err(LogError::WrongIdentity), + } return Ok(group.clone()); } let mapping = MappingRecord::new_cells(self.root.next_key, &bootstrap, &conf)?; - let mut cache = genesis(conf); + let mut cache = genesis(conf.clone()); + if let Some(managed) = managed { + if !managed.allow_genesis { + return Err(LogError::WrongIdentity); + } + let binding = managed.binding; + if !binding.matches_scope(&bootstrap.route.history) { + return Err(LogError::WrongIdentity); + } + cache.membership = + super::member_state::GroupMembership::managed_initial(&conf, binding)?; + } + if let Some(seed) = seed { + // Preserve the source's real genesis/enrollment origin. The learner + // remains outside this birth configuration until authenticated install. + cache.membership = seed.initial_membership(&conf)?; + } cache.group_manifest = Some( super::cell_groups::GroupManifest::genesis(bootstrap.route) .map_err(|error| LogError::Invalid(error.to_string()))?, @@ -1740,6 +1854,7 @@ impl Worker { self.mappings[&write.group.key].history()?, write.group.identity(), )?; + grant.check_lifecycle(self.lifecycle.guard.as_ref())?; } } for publication in &mut publications { @@ -1899,8 +2014,8 @@ impl Worker { let _ = reply.send(result); fatal } - Work::CreateCells(bootstrap, seed, reply) => { - let result = self.create_cells(bootstrap, seed.as_deref()); + Work::CreateCells(bootstrap, seed, binding, reply) => { + let result = self.create_cells(bootstrap, seed.as_deref(), binding.map(|b| *b)); let fatal = result.as_ref().err().filter(|error| error.fatal()).cloned(); if fatal.is_some() { self.closed.store(true, Ordering::Release); diff --git a/crates/cellar/durability/raft/storage_genesis_tests.rs b/crates/cellar/durability/raft/storage_genesis_tests.rs index 1e49f6d66..021d322fd 100644 --- a/crates/cellar/durability/raft/storage_genesis_tests.rs +++ b/crates/cellar/durability/raft/storage_genesis_tests.rs @@ -47,7 +47,11 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho )) .await??; let initial = bootstrap(5); - let binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [90; 32])?; + let binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; let guarded = log .create_managed_group(initial.clone(), binding.clone()) .await?; @@ -55,7 +59,11 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho assert_eq!(before.fleet_status().unwrap().changed_at.get(), 1); let bare = bootstrap(7); log.create_group(bare.clone()).await?; - let bare_binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &bare.scope, [90; 32])?; + let bare_binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&bare.scope), + [90; 32], + )?; assert!(matches!( log.create_managed_group(bare.clone(), bare_binding).await, Err(LogError::WrongIdentity) @@ -65,13 +73,20 @@ async fn managed_genesis_reopen_never_retrofits_bare_or_rebound_slots() -> anyho .membership()? .fleet_status() .is_none()); - let wrong = FleetGroupBinding::new(IncarnationId::new([8; 16])?, &initial.scope, [90; 32])?; + let wrong = FleetGroupBinding::new( + IncarnationId::new([8; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; assert!(matches!( log.create_managed_group(initial.clone(), wrong).await, Err(LogError::WrongIdentity) )); - let other_catalog = - FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [91; 32])?; + let other_catalog = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [91; 32], + )?; assert!(matches!( log.create_managed_group(initial.clone(), other_catalog) .await, @@ -108,7 +123,11 @@ async fn managed_genesis_publication_waits_for_actual_engine_sync() -> anyhow::R )) .await??; let initial = bootstrap(5); - let binding = FleetGroupBinding::new(IncarnationId::new([6; 16])?, &initial.scope, [90; 32])?; + let binding = FleetGroupBinding::new( + IncarnationId::new([6; 16])?, + &GroupHistory::from(&initial.scope), + [90; 32], + )?; let release = gate.arm(fault::HOLD_SYNC); let owned = log.clone(); let proposal = initial.clone(); diff --git a/crates/cellar/durability/raft/storage_lifecycle_tests.rs b/crates/cellar/durability/raft/storage_lifecycle_tests.rs index 390392a25..697ba1aea 100644 --- a/crates/cellar/durability/raft/storage_lifecycle_tests.rs +++ b/crates/cellar/durability/raft/storage_lifecycle_tests.rs @@ -67,6 +67,13 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept }) .await?; let mut gate = stop(&binding, 2); + assert!(log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); + let mut other_boot = binding.clone(); + other_boot.boot = [99; 16]; + assert!(!log + .lifecycle_guard()? + .unwrap() + .admits_group_start(&other_boot)); gate.purpose = LifecyclePurpose::Maintenance; let mut selector = gate.clone(); selector.finalization = [13; 16]; @@ -93,6 +100,7 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept deadline: deadline(&log)?, }) .await?; + assert!(!log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); let mut foreign = gate.clone(); foreign.request = [15; 16]; assert!(matches!( @@ -113,6 +121,7 @@ async fn fleet_lifecycle_maintenance_stop_requires_exact_gate_and_retains_accept .await?; } assert_eq!(log.lifecycle_guard()?.unwrap().accepted(), Some(&selector)); + assert!(!log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); assert!(matches!( log.update_lifecycle(LifecycleUpdate::Fence { boot: binding, @@ -790,6 +799,7 @@ async fn fleet_lifecycle_resume_requires_exact_gate_and_survives_reopen() -> any log.lifecycle_guard()?.unwrap().resumed(), Some((&resume, &gate)) ); + assert!(log.lifecycle_guard()?.unwrap().admits_group_start(&binding)); assert!(matches!( log.update_lifecycle(LifecycleUpdate::Fence { boot: binding.clone(), diff --git a/crates/cellar/durability/raft/transport.rs b/crates/cellar/durability/raft/transport.rs index d7a18f245..737d743c9 100644 --- a/crates/cellar/durability/raft/transport.rs +++ b/crates/cellar/durability/raft/transport.rs @@ -16,6 +16,9 @@ use runtime_adapter::TransportRuntime; mod group_snapshots; #[path = "transport_shared.rs"] mod shared; +#[cfg(all(test, not(cellar_internal_tests)))] +#[path = "transport_shared_fixture.rs"] +mod shared_fixture; use super::transport_frame::{self as frame, Disposition, FrameHead, RequestHeader, Route}; use super::transport_options::{NamespaceQueryOrigin, RaftPeerNode, RaftTls, TransportOptions}; use crate::asyncrt::DomainToken; @@ -303,6 +306,9 @@ struct Registry { tasks: usize, groups: BTreeMap>, physical_groups: BTreeMap>>, + // One bounded lifecycle token per physical route. Advisory process/member + // refreshes keep it; explicit replacement/removal starts another lifetime. + physical_owners: BTreeMap>, group_checkpoint_pool: Option, group_checkpoints: BTreeMap>, query_groups: BTreeMap>, @@ -370,6 +376,8 @@ struct State { cell_query_gateways: Mutex>>, query_gateway: Mutex>>, + fleet_gateway: + Mutex>>, registry: Mutex, outgoing: Banks, incoming: Banks, @@ -528,6 +536,7 @@ impl RaftTransport { local_data, local_query, query_gateway: Mutex::new(None), + fleet_gateway: Mutex::new(None), cell_command_gateways: Mutex::new(BTreeMap::new()), cell_query_gateways: Mutex::new(BTreeMap::new()), registry: Mutex::new(Registry { @@ -535,6 +544,7 @@ impl RaftTransport { tasks: 0, groups: BTreeMap::new(), physical_groups: BTreeMap::new(), + physical_owners: BTreeMap::new(), group_checkpoint_pool: None, group_checkpoints: BTreeMap::new(), query_groups: BTreeMap::new(), @@ -1068,6 +1078,9 @@ impl State { if let Ok(mut gateway) = self.query_gateway.lock() { gateway.take(); } + if let Ok(mut gateway) = self.fleet_gateway.lock() { + gateway.take(); + } if let Ok(mut gateways) = self.cell_command_gateways.lock() { gateways.clear(); } @@ -1330,22 +1343,6 @@ impl State { Ok(()) }) .await?; - if &prefix[..4] == inspections::REQUEST_MAGIC { - if lane != Lane::Control - || sequence != 0 - || !self.inspection_enabled.load(Ordering::Acquire) - { - return Err(TransportError::Protocol); - } - let exporter = stream - .get_ref() - .1 - .export_keying_material([0; 32], inspections::EXPORT_LABEL, None) - .map_err(|error| TransportError::Tls(error.to_string()))?; - return self - .receive_inspection(&mut stream, &exporter, prefix) - .await; - } if &prefix[..4] == node_inspections::REQUEST_MAGIC { if lane != Lane::Control || sequence != 0 @@ -1410,17 +1407,6 @@ impl State { .receive_fleet_inspection(&mut stream, &exporter, prefix) .await; } - if &prefix[..4] == seeds::REQUEST_MAGIC { - if lane != Lane::Data || sequence != 0 { - return Err(TransportError::Protocol); - } - let exporter = stream - .get_ref() - .1 - .export_keying_material([0; 32], seeds::EXPORT_LABEL, None) - .map_err(|error| TransportError::Tls(error.to_string()))?; - return self.receive_seed(&mut stream, &exporter, prefix).await; - } if &prefix[..4] == group_snapshots::seed::REQUEST_MAGIC { if lane != Lane::Data || sequence != 0 { return Err(TransportError::Protocol); diff --git a/crates/cellar/durability/raft/transport_fleet.rs b/crates/cellar/durability/raft/transport_fleet.rs index b284238a6..d9ceb2121 100644 --- a/crates/cellar/durability/raft/transport_fleet.rs +++ b/crates/cellar/durability/raft/transport_fleet.rs @@ -20,6 +20,38 @@ impl Registry { } impl RaftTransport { + /// Resolve an endpoint only from an installed rooted trust cut and its + /// authenticated current node view. Discovery supplies no membership grant. + pub(crate) fn managed_group_peer( + &self, + scope: &GroupHistory, + node: NodeId, + minimum_revision: u64, + ) -> Result { + let state = &self.handle.0; + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + let trust = registry + .fleet_trust + .as_ref() + .ok_or(TransportError::Refused)?; + if trust.cluster() != scope.cluster || trust.revision() < minimum_revision { + return Err(TransportError::StalePeer); + } + let view = registry + .node_views + .get(&node) + .ok_or(TransportError::StalePeer)?; + let enrolled = trust.node(node).ok_or(TransportError::Refused)?; + if view.cluster() != scope.cluster + || enrolled.storage_instance() != Some(view.storage_instance()) + || !enrolled.matches_peer(view.peer()) + { + return Err(TransportError::StalePeer); + } + Ok(view.peer().clone()) + } + /// Routing may omit a permanently quarantined member. Its Raft vote and /// physical placement still belong to the durable configuration until a /// quorum commits removal. Missing discovery alone is never sufficient. @@ -40,13 +72,17 @@ impl RaftTransport { /// Check a rooted admission against current trust and actual durable /// configuration. Exclusion may continue the exact reserved destination /// boot; it cannot substitute another process or admit fresh placement. - pub(crate) fn check_fleet_placement( + pub(crate) fn check_fleet_placement( &self, - scope: &GroupScope, + scope: &S, members: &[NodeId], target: &JoiningMember, authorization: &FleetPermitAuthorization, - ) -> Result<(), TransportError> { + ) -> Result<(), TransportError> + where + for<'a> GroupHistory: From<&'a S>, + { + let scope = GroupHistory::from(scope); let state = &self.handle.0; let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; @@ -77,7 +113,7 @@ impl RaftTransport { if !destination.permits_transport() || (!destination.eligibility().receive_replicas && !continuing) || destination.boot_binding().ok().as_ref() != Some(original_boot) - || !authorization.matches_scope(scope) + || !authorization.matches_scope::(&scope) || destination.storage_instance().is_none() { return Err(TransportError::StalePeer); diff --git a/crates/cellar/durability/raft/transport_query_tests.rs b/crates/cellar/durability/raft/transport_query_tests.rs new file mode 100644 index 000000000..0f67ff3bd --- /dev/null +++ b/crates/cellar/durability/raft/transport_query_tests.rs @@ -0,0 +1,248 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Actual synced shared engines and transport grants. These authentication +//! observations grant no serving, quorum or logical-cell admission authority. +use super::*; +use crate::raft::storage::CellGroupBootstrap; +use cellar_logic::cell_group::{CellHistory, CellId, CellRange, GroupHistory, RouteEpoch}; + +use super::super::shared_fixture::Fixture; + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn query_grants_are_scoped_revocable_and_never_voting_authority() -> anyhow::Result<()> { + use crate::raft::query_identity::QueryRoute; + let first = GroupHistory { + cluster: ClusterId::new([1; 16])?, + namespace: NamespaceId::new([2; 16])?, + group: GroupId::new([3; 16])?, + incarnation: IncarnationId::new([4; 16])?, + }; + let mut second = first; + second.namespace = NamespaceId::new([5; 16])?; + second.group = GroupId::new([6; 16])?; + let groups = [&first, &second] + .into_iter() + .map(|scope| { + Ok(CellGroupBootstrap { + route: cellar_logic::cell_group::GroupRoute { + history: *scope, + range: CellRange::all(), + epoch: RouteEpoch::new(1)?, + birth_digest: [10; 32], + schema_digest: [9; 32], + }, + member: MemberId::new(1)?, + voters: (1..=3).map(MemberId::new).collect::, _>>()?, + learners: vec![], + }) + }) + .collect::>>()?; + let fixture = Fixture::start( + TransportOptions { + max_query_origins: 2, + ..Default::default() + }, + groups, + ) + .await?; + for scope in [&first, &second] { + let peers = (1..=3) + .map(|member| { + Ok(RaftPeer { + replica: ReplicaIdentity { + scope: *scope, + node: NodeId::new([member as u8; 16])?, + member: MemberId::new(member)?, + disk: DiskLineage::new([member as u8 + 1; 16])?, + process: ProcessGeneration::new([member as u8 + 2; 16])?, + }, + label: format!("node-{member}"), + server_name: "localhost".into(), + control_address: if member == 1 { + fixture.transport.control_address() + } else { + format!("127.0.0.1:{}", 40000 + member * 3).parse()? + }, + data_address: if member == 1 { + fixture.transport.data_address() + } else { + format!("127.0.0.1:{}", 40001 + member * 3).parse()? + }, + query_address: if member == 1 { + fixture.transport.query_address() + } else { + format!("127.0.0.1:{}", 40002 + member * 3).parse()? + }, + }) + }) + .collect::>>()?; + fixture.transport.install_physical_group(GroupRoute { + scope: *scope, + schema_digest: [9; 32], + revision: 1, + peers, + })?; + } + fixture + .domain + .run(async { + let peer = RaftPeerNode { + node: NodeId::new([4; 16])?, + disk: DiskLineage::new([9; 16])?, + process: ProcessGeneration::new([10; 16])?, + label: "node-4".into(), + server_name: "localhost".into(), + control_address: "127.0.0.1:41001".parse()?, + data_address: "127.0.0.1:41002".parse()?, + query_address: "127.0.0.1:41003".parse()?, + }; + let origin = NodeIdentity { + node: peer.node, + disk: peer.disk, + process: peer.process, + }; + let route = |scope: &GroupHistory, source| -> anyhow::Result<_> { + Ok(QueryRoute::new_shared( + source, + "node-4", + fixture.log.cell_group(scope).unwrap().identity(), + "node-1", + &CellHistory { + cluster: scope.cluster, + namespace: scope.namespace, + cell: CellId::new("Counter:one")?, + incarnation: IncarnationId::new([20; 16])?, + }, + [9; 32], + )?) + }; + let first_route = route(&first, origin)?; + let second_route = route(&second, origin)?; + let state = &fixture.transport.handle.0; + assert!(matches!( + state.authorize_query_route(&first_route), + Err(TransportError::Refused) + )); + let mut grant = NamespaceQueryOrigin { + cluster: first.cluster, + namespace: first.namespace, + revision: 1, + peer: peer.clone(), + }; + fixture.transport.grant_query_origin(grant.clone())?; + let (pool, permission) = state.authorize_query_route(&first_route)?; + let permission = permission.expect("nonmember requires an explicit grant"); + assert!(matches!( + state.authorize_query_route(&second_route), + Err(TransportError::Refused) + )); + let mut wrong_process = origin; + wrong_process.process = ProcessGeneration::new([11; 16])?; + assert!(matches!( + state.authorize_query_route(&route(&first, wrong_process)?), + Err(TransportError::Refused) + )); + let mut other = grant.clone(); + other.namespace = second.namespace; + fixture.transport.grant_query_origin(other.clone())?; + let (_, other_permission) = state.authorize_query_route(&second_route)?; + let fake_member = ReplicaIdentity { + scope: first, + node: origin.node, + disk: origin.disk, + process: origin.process, + member: MemberId::new(99)?, + }; + assert!(state + .authorize_physical(&Route::new_group( + &fake_member, + "node-4", + fixture.log.cell_group(&first)?.identity(), + "node-1", + [9; 32] + )?) + .is_err()); + let (_cancel, cancel) = watch::channel(false); + let deadline = state.domain.mono_ms()?.saturating_add(1000); + let waiting = state.query_until( + &pool, + deadline, + cancel.clone(), + Some(permission.clone()), + std::future::pending::>(), + ); + tokio::pin!(waiting); + assert!(futures_util::poll!(&mut waiting).is_pending()); + fixture + .transport + .revoke_query_origin(first.cluster, first.namespace, peer.node, 2)?; + assert!(matches!( + crate::asyncrt::timeout(std::time::Duration::from_secs(1), waiting).await?, + Err(TransportError::Refused) + )); + assert!(*permission.borrow()); + assert!( + !*pool.stop.borrow(), + "another namespace still owns this physical pool" + ); + assert!(!*other_permission.unwrap().borrow()); + assert!(matches!( + fixture.transport.grant_query_origin(grant.clone()), + Err(TransportError::StalePeer) + )); + let mut third = grant.clone(); + third.namespace = NamespaceId::new([7; 16])?; + assert!(matches!( + fixture.transport.grant_query_origin(third), + Err(TransportError::Overloaded) + )); + grant.revision = 3; + fixture.transport.grant_query_origin(grant.clone())?; + assert!(!*state + .authorize_query_route(&first_route)? + .1 + .unwrap() + .borrow()); + assert!( + matches!( + state + .query_until( + &pool, + deadline, + cancel, + Some(permission), + std::future::ready(Ok::<(), TransportError>(())) + ) + .await, + Err(TransportError::Refused) + ), + "regrant cannot revive an old in-flight authorization" + ); + let mut replacement = grant.clone(); + replacement.revision = 4; + replacement.peer.process = wrong_process.process; + assert!( + matches!( + fixture.transport.grant_query_origin(replacement.clone()), + Err(TransportError::StalePeer) + ), + "other namespace still binds the old physical process" + ); + fixture.transport.revoke_query_origin( + second.cluster, + second.namespace, + peer.node, + 2, + )?; + fixture.transport.grant_query_origin(replacement)?; + assert!(*pool.stop.borrow()); + assert!(state.authorize_query_route(&first_route).is_err()); + assert!(state + .authorize_query_route(&route(&first, wrong_process)?) + .is_ok()); + assert_eq!(fixture.log.cell_groups()?.len(), 2); + Ok::<_, anyhow::Error>(()) + }) + .await??; + fixture.close().await +} diff --git a/crates/cellar/durability/raft/transport_shared.rs b/crates/cellar/durability/raft/transport_shared.rs index 150c4fd46..029d0e4b9 100644 --- a/crates/cellar/durability/raft/transport_shared.rs +++ b/crates/cellar/durability/raft/transport_shared.rs @@ -56,26 +56,28 @@ impl RaftTransport { /// Install already-authorized physical membership. This creates no group, /// grants no serving authority and changes no committed configuration. pub fn install_physical_group(&self, route: PhysicalRoute) -> Result<(), TransportError> { - self.install_physical(route, false).map(|_| ()) + self.install_physical(route, false, None).map(|_| ()) } pub(crate) fn register_physical_group( &self, route: PhysicalRoute, ) -> Result { - let route = self.install_physical(route, true)?; + let (route, owner) = self.install_physical(route, true, None)?; Ok(PhysicalRegistration { state: Arc::downgrade(&self.handle.0), route, + owner, checkpoint: None, }) } - fn install_physical( + pub(super) fn install_physical( &self, route: PhysicalRoute, exclusive: bool, - ) -> Result, TransportError> { + refresh: Option>, + ) -> Result<(Arc, Arc<()>), TransportError> { let state = &self.handle.0; let runtime = state.runtime.shared()?; let group = runtime @@ -123,6 +125,15 @@ impl RaftTransport { } let mut registry = state.registry.lock().map_err(|_| TransportError::Closed)?; state.admit(®istry)?; + if let Some(expected) = &refresh { + if !registry + .physical_owners + .get(&route.scope) + .is_some_and(|owner| Arc::ptr_eq(owner, expected)) + { + return Err(TransportError::StalePeer); + } + } for peer in &route.peers { registry.check_node_view(&RaftPeerNode::from(peer))?; } @@ -134,7 +145,11 @@ impl RaftTransport { return Err(TransportError::Identity); } if **old == route { - return Ok(old.clone()); + let owner = registry + .physical_owners + .get(&route.scope) + .ok_or(TransportError::Closed)?; + return Ok((old.clone(), owner.clone())); } if old.revision >= route.revision || old.schema_digest != route.schema_digest { return Err(TransportError::StalePeer); @@ -183,7 +198,11 @@ impl RaftTransport { } let starts = state.prepare_pools(&mut registry, &physical)?; let route = Arc::new(route); - group_snapshots::revoke(&mut registry, &route.scope); + if refresh.is_none() { + group_snapshots::revoke(&mut registry, &route.scope); + } + let owner = refresh.unwrap_or_else(|| Arc::new(())); + registry.physical_owners.insert(route.scope, owner.clone()); registry.physical_groups.insert(route.scope, route.clone()); registry.prune_pools(); drop(registry); @@ -192,12 +211,155 @@ impl RaftTransport { drop(PhysicalRegistration { state: Arc::downgrade(state), route, + owner, checkpoint: None, }); } return Err(error); } - Ok(route) + Ok((route, owner)) + } + + /// Refresh only advisory managed peer routes. Committed membership supplies + /// member/disk bindings; a current rooted node view supplies the endpoint. + /// Missing discovery keeps known voters usable and never removes Raft debt. + pub(crate) fn reconcile_managed_physical( + &self, + history: &GroupHistory, + ) -> Result<(), TransportError> { + let state = &self.handle.0; + let runtime = state.runtime.shared()?; + let (old, owner) = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + ( + registry + .physical_groups + .get(history) + .cloned() + .ok_or(TransportError::UnknownGroup)?, + registry + .physical_owners + .get(history) + .cloned() + .ok_or(TransportError::UnknownGroup)?, + ) + }; + let group = runtime + .log() + .cell_group(history) + .map_err(|_| TransportError::UnknownGroup)?; + // Never hold a group read guard while taking the registry lock. A queued + // writer can otherwise deadlock against registry->group authorization. + let (binding, members) = { + let cache = group + .read() + .map_err(|e| TransportError::Runtime(e.into()))?; + let Some(fleet) = cache.membership.fleet_status() else { + return Ok(()); + }; + let current = crate::raft::member_state::members(&cache.conf_state); + if current.len() == old.peers.len() + && old.peers.iter().all(|peer| { + current.contains(&peer.replica.member.get()) + && cache.membership.permits_group(&peer.replica) + }) + { + return Ok(()); + } + if !current.contains(&group.identity().member.get()) { + return Err(TransportError::StalePeer); + } + let members = current + .into_iter() + .map(|id| { + let member = cellar_logic::consensus::MemberId::new(id) + .map_err(|_| TransportError::Identity)?; + let joined = cache + .membership + .joined(member) + .map_err(|e| TransportError::Runtime(e.into()))?; + Ok((member, joined.map(|(binding, _)| binding))) + }) + .collect::, TransportError>>()?; + (fleet.binding, members) + }; + let peers = { + let registry = state.registry.lock().map_err(|_| TransportError::Closed)?; + state.admit(®istry)?; + if !registry + .physical_owners + .get(history) + .is_some_and(|current| Arc::ptr_eq(current, &owner)) + { + return Err(TransportError::StalePeer); + } + // An advisory view by itself cannot expand a managed route. Until + // trust arrives, leave existing routing permissions unchanged. + let Some(trust) = ®istry.fleet_trust else { + return Ok(()); + }; + if trust.fleet() != binding.fleet() + || trust.catalog() != binding.catalog() + || trust.cluster() != history.cluster + { + return Err(TransportError::Refused); + } + let mut peers = Vec::with_capacity(members.len()); + for (member, joined) in members { + let prior = old.peers.iter().find(|peer| peer.replica.member == member); + let peer = if let Some(joined) = joined { + let Some(view) = registry.node_views.get(&joined.node) else { + continue; + }; + let node = view.peer(); + if node.disk != joined.disk || registry.check_node_view(node).is_err() { + continue; + } + RaftPeer { + replica: ReplicaIdentity { + scope: *history, + member, + node: node.node, + disk: node.disk, + process: node.process, + }, + label: node.label.clone(), + server_name: node.server_name.clone(), + control_address: node.control_address, + data_address: node.data_address, + query_address: node.query_address, + } + } else { + // Initial member identities come from the authorized birth + // route, never from guessing an ID from node discovery. + let Some(prior) = prior else { + continue; + }; + prior.clone() + }; + peers.push(peer); + } + peers + }; + if peers == old.peers { + return Ok(()); + } + let revision = old + .revision + .checked_add(1) + .ok_or(TransportError::StalePeer)?; + self.install_physical( + PhysicalRoute { + scope: *history, + schema_digest: old.schema_digest, + revision, + peers, + }, + false, + Some(owner), + )?; + Ok(()) } pub fn remove_physical_group(&self, history: &GroupHistory) -> Result<(), TransportError> { @@ -207,6 +369,7 @@ impl RaftTransport { state.admit(®istry)?; group_snapshots::revoke(&mut registry, history); registry.physical_groups.remove(history); + registry.physical_owners.remove(history); registry.prune_pools(); Ok(()) } @@ -217,10 +380,20 @@ impl RaftTransport { &self, history: &GroupHistory, message: Message, + ) -> Result { + self.submit_physical_inner(history, message, true) + } + + fn submit_physical_inner( + &self, + history: &GroupHistory, + message: Message, + discover: bool, ) -> Result { let state = &self.handle.0; let runtime = state.runtime.shared()?; if message.get_msg_type() == MessageType::MsgSnapshot { + self.reconcile_managed_physical(history)?; return self.submit_group_checkpoint(history, message); } let lane = frame::classify(&message)?; @@ -243,14 +416,21 @@ impl RaftTransport { if message.from != local.member.get() { return Err(TransportError::Identity); } - let peer = route + let peer = match route .peers .iter() .find(|p| p.replica.member.get() == message.to) - // A committed learner can precede discovery of its process route. - // Treat that interval as unavailable delivery, not node corruption. - .ok_or(TransportError::StalePeer)? - .clone(); + { + Some(peer) => peer.clone(), + None if discover => { + // Only a missing route needs discovery on ordinary traffic. + // Retry once, before allocating/charging a serialized frame. + drop(registry); + self.reconcile_managed_physical(history)?; + return self.submit_physical_inner(history, message, false); + } + None => return Err(TransportError::StalePeer), + }; if peer.replica.node == local.node { return Err(TransportError::Identity); } @@ -414,6 +594,7 @@ impl Registry { pub(crate) struct PhysicalRegistration { state: std::sync::Weak, route: Arc, + owner: Arc<()>, pub(crate) checkpoint: Option, } @@ -424,12 +605,13 @@ impl Drop for PhysicalRegistration { }; if let Ok(mut registry) = state.registry.lock() { if registry - .physical_groups + .physical_owners .get(&self.route.scope) - .is_some_and(|r| Arc::ptr_eq(r, &self.route)) + .is_some_and(|owner| Arc::ptr_eq(owner, &self.owner)) { group_snapshots::revoke(&mut registry, &self.route.scope); registry.physical_groups.remove(&self.route.scope); + registry.physical_owners.remove(&self.route.scope); registry.prune_pools(); } }; diff --git a/crates/cellar/durability/raft/transport_shared_fixture.rs b/crates/cellar/durability/raft/transport_shared_fixture.rs new file mode 100644 index 000000000..bd8506f5f --- /dev/null +++ b/crates/cellar/durability/raft/transport_shared_fixture.rs @@ -0,0 +1,85 @@ +// Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. + +//! Synced shared engines and real TLS listeners for transport ownership tests. +//! Groups are parked: these fixtures grant no quorum or serving authority. +use super::*; +use crate::raft::{ + cell_groups::{CellGroupRuntime, CellGroupRuntimeEvents}, + options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}, + storage::{CellGroupBootstrap, ReplicaLog}, +}; +use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; + +pub(super) struct Fixture { + pub(super) domain: crate::asyncrt::ExecutionDomain, + pub(super) log: ReplicaLog, + pub(super) runtime: CellGroupRuntime, + pub(super) transport: RaftTransport, + _events: CellGroupRuntimeEvents, + _directory: tempfile::TempDir, +} + +impl Fixture { + pub(super) async fn start( + options: TransportOptions, + groups: Vec, + ) -> anyhow::Result { + let directory = tempfile::tempdir()?; + let domain = crate::asyncrt::ExecutionDomain::new(tokio::runtime::Handle::current()); + let log = domain + .run(ReplicaLog::open( + ReplicaLogOptions::new( + directory.path().join("log"), + NodeId::new([1; 16])?, + DiskLineage::new([2; 16])?, + ProcessGeneration::new([3; 16])?, + LogOpenMode::Create, + ), + Arc::new(cellar_ltx::DirectFileSystem), + )) + .await??; + for group in groups { + log.create_cell_group(group).await?; + } + let (runtime, events) = + CellGroupRuntime::start(log.clone(), RaftRuntimeOptions::default()).await?; + let tls = RaftTls::new( + vec![include_bytes!("../../transport/tcp/fixtures/ca.der") + .to_vec() + .into()], + vec![include_bytes!("../../transport/tcp/fixtures/server.der") + .to_vec() + .into()], + rustls::pki_types::PrivatePkcs8KeyDer::from( + include_bytes!("../../transport/tcp/fixtures/key.der").to_vec(), + ) + .into(), + )?; + let transport = RaftTransport::start_shared( + runtime.clone(), + Arc::new(PeerAuth::new([8; 32], "node-1")?), + tls, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + TcpListener::bind("127.0.0.1:0").await?, + options, + ) + .await?; + Ok(Self { + domain, + log, + runtime, + transport, + _events: events, + _directory: directory, + }) + } + + pub(super) async fn close(self) -> anyhow::Result<()> { + self.transport.close().await; + self.runtime.close().await?; + self.log.close().await?; + self.domain.shutdown().await; + Ok(()) + } +} diff --git a/crates/cellar/durability/raft/transport_tests.rs b/crates/cellar/durability/raft/transport_tests.rs index 43ce4aa33..3e1bf9d09 100644 --- a/crates/cellar/durability/raft/transport_tests.rs +++ b/crates/cellar/durability/raft/transport_tests.rs @@ -7,6 +7,8 @@ mod directory; mod pipeline; #[path = "transport_pipeline_regression_tests.rs"] mod pipeline_regressions; +#[path = "transport_query_tests.rs"] +mod queries; use crate::raft::options::{LogOpenMode, RaftRuntimeOptions, ReplicaLogOptions}; use crate::raft::storage::ReplicaLog; use cellar_logic::consensus::{DiskLineage, ProcessGeneration}; @@ -316,235 +318,6 @@ async fn poisoned_registry_still_waits_for_resource_owners() -> anyhow::Result<( fixture.close().await } -#[tokio::test(flavor = "multi_thread", worker_threads = 4)] -async fn query_grants_are_scoped_revocable_and_never_voting_authority() -> anyhow::Result<()> { - use crate::raft::{query_identity::QueryRoute, storage::GroupBootstrap}; - let first = GroupScope { - cluster: ClusterId::new([1; 16])?, - namespace: NamespaceId::new([2; 16])?, - cell: "Counter:one".into(), - group: GroupId::new([3; 16])?, - incarnation: IncarnationId::new([4; 16])?, - }; - let mut second = first.clone(); - second.namespace = NamespaceId::new([5; 16])?; - second.group = GroupId::new([6; 16])?; - let groups = [&first, &second] - .into_iter() - .map(|scope| { - Ok(GroupBootstrap { - scope: scope.clone(), - member: MemberId::new(1)?, - birth_digest: [10; 32], - voters: (1..=3).map(MemberId::new).collect::, _>>()?, - learners: vec![], - }) - }) - .collect::>>()?; - let fixture = Fixture::start( - TransportOptions { - max_query_origins: 2, - ..Default::default() - }, - groups, - ) - .await?; - for scope in [&first, &second] { - let peers = (1..=3) - .map(|member| { - Ok(RaftPeer { - replica: ReplicaIdentity { - scope: scope.clone(), - node: NodeId::new([member as u8; 16])?, - member: MemberId::new(member)?, - disk: DiskLineage::new([member as u8 + 1; 16])?, - process: ProcessGeneration::new([member as u8 + 2; 16])?, - }, - label: format!("node-{member}"), - server_name: "localhost".into(), - control_address: if member == 1 { - fixture.transport.control_address() - } else { - format!("127.0.0.1:{}", 40000 + member * 3).parse()? - }, - data_address: if member == 1 { - fixture.transport.data_address() - } else { - format!("127.0.0.1:{}", 40001 + member * 3).parse()? - }, - query_address: if member == 1 { - fixture.transport.query_address() - } else { - format!("127.0.0.1:{}", 40002 + member * 3).parse()? - }, - }) - }) - .collect::>>()?; - fixture.transport.install_group(GroupRoute { - scope: scope.clone(), - schema_digest: [9; 32], - revision: 1, - peers, - })?; - } - fixture - .domain - .run(async { - let peer = RaftPeerNode { - node: NodeId::new([4; 16])?, - disk: DiskLineage::new([9; 16])?, - process: ProcessGeneration::new([10; 16])?, - label: "node-4".into(), - server_name: "localhost".into(), - control_address: "127.0.0.1:41001".parse()?, - data_address: "127.0.0.1:41002".parse()?, - query_address: "127.0.0.1:41003".parse()?, - }; - let origin = NodeIdentity { - node: peer.node, - disk: peer.disk, - process: peer.process, - }; - let route = |scope: &GroupScope, source| { - QueryRoute::new( - source, - "node-4", - fixture.log.group(scope).unwrap().identity(), - "node-1", - [9; 32], - ) - }; - let first_route = route(&first, origin)?; - let second_route = route(&second, origin)?; - let state = &fixture.transport.handle.0; - assert!(matches!( - state.authorize_query_route(&first_route), - Err(TransportError::Refused) - )); - let mut grant = NamespaceQueryOrigin { - cluster: first.cluster, - namespace: first.namespace, - revision: 1, - peer: peer.clone(), - }; - fixture.transport.grant_query_origin(grant.clone())?; - let (pool, permission) = state.authorize_query_route(&first_route)?; - let permission = permission.expect("nonmember requires an explicit grant"); - assert!(matches!( - state.authorize_query_route(&second_route), - Err(TransportError::Refused) - )); - let mut wrong_process = origin; - wrong_process.process = ProcessGeneration::new([11; 16])?; - assert!(matches!( - state.authorize_query_route(&route(&first, wrong_process)?), - Err(TransportError::Refused) - )); - let mut other = grant.clone(); - other.namespace = second.namespace; - fixture.transport.grant_query_origin(other.clone())?; - let (_, other_permission) = state.authorize_query_route(&second_route)?; - let fake_member = ReplicaIdentity { - scope: first.clone(), - node: origin.node, - disk: origin.disk, - process: origin.process, - member: MemberId::new(99)?, - }; - assert!(state - .authorize_route(&Route::new( - &fake_member, - "node-4", - fixture.log.group(&first)?.identity(), - "node-1", - [9; 32] - )?) - .is_err()); - let (_cancel, cancel) = watch::channel(false); - let deadline = state.domain.mono_ms()?.saturating_add(1000); - let waiting = state.query_until( - &pool, - deadline, - cancel.clone(), - Some(permission.clone()), - std::future::pending::>(), - ); - tokio::pin!(waiting); - assert!(futures_util::poll!(&mut waiting).is_pending()); - fixture - .transport - .revoke_query_origin(first.cluster, first.namespace, peer.node, 2)?; - assert!(matches!( - crate::asyncrt::timeout(std::time::Duration::from_secs(1), waiting).await?, - Err(TransportError::Refused) - )); - assert!(*permission.borrow()); - assert!( - !*pool.stop.borrow(), - "another namespace still owns this physical pool" - ); - assert!(!*other_permission.unwrap().borrow()); - assert!(matches!( - fixture.transport.grant_query_origin(grant.clone()), - Err(TransportError::StalePeer) - )); - let mut third = grant.clone(); - third.namespace = NamespaceId::new([7; 16])?; - assert!(matches!( - fixture.transport.grant_query_origin(third), - Err(TransportError::Overloaded) - )); - grant.revision = 3; - fixture.transport.grant_query_origin(grant.clone())?; - assert!(!*state - .authorize_query_route(&first_route)? - .1 - .unwrap() - .borrow()); - assert!( - matches!( - state - .query_until( - &pool, - deadline, - cancel, - Some(permission), - std::future::ready(Ok::<(), TransportError>(())) - ) - .await, - Err(TransportError::Refused) - ), - "regrant cannot revive an old in-flight authorization" - ); - let mut replacement = grant.clone(); - replacement.revision = 4; - replacement.peer.process = wrong_process.process; - assert!( - matches!( - fixture.transport.grant_query_origin(replacement.clone()), - Err(TransportError::StalePeer) - ), - "other namespace still binds the old physical process" - ); - fixture.transport.revoke_query_origin( - second.cluster, - second.namespace, - peer.node, - 2, - )?; - fixture.transport.grant_query_origin(replacement)?; - assert!(*pool.stop.borrow()); - assert!(state.authorize_query_route(&first_route).is_err()); - assert!(state - .authorize_query_route(&route(&first, wrong_process)?) - .is_ok()); - assert_eq!(fixture.log.groups()?.len(), 2); - Ok::<_, anyhow::Error>(()) - }) - .await??; - fixture.close().await -} - #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn command_grants_bound_tombstones_without_changing_read_permission() -> anyhow::Result<()> { use crate::raft::transport_options::NamespaceCommandOrigin; diff --git a/crates/cellar/fleet/management/raft/connection.rs b/crates/cellar/fleet/management/raft/connection.rs index e7ea6d349..895a6866e 100644 --- a/crates/cellar/fleet/management/raft/connection.rs +++ b/crates/cellar/fleet/management/raft/connection.rs @@ -32,6 +32,8 @@ pub struct RaftFleetNamespace { /// Must match the existing namespace Root; opening never changes its mode. #[serde(default = "materialized")] pub image_storage: crate::raft::catalog::RaftImageStorage, + /// Explicit bounded physical-group directory discovery and operation limits. + pub group_directory: crate::raft::cell_groups::DirectoryLimits, } fn materialized() -> crate::raft::catalog::RaftImageStorage { @@ -121,7 +123,9 @@ impl RaftFleetControl { _ = timer => return Err(FleetCatalogError::Unconfirmed.into()), result = adapter.existing_namespace(namespace_config) => result.map_err(namespace_failure)?, }; - namespaces.push(opened); + namespaces.push( + RangeDirectory::new(opened, entry.group_directory).map_err(namespace_failure)?, + ); } catalog .check_inventory(&root, &namespaces, turn.remaining()?) diff --git a/crates/cellar/fleet/management/raft/lifecycle/mod.rs b/crates/cellar/fleet/management/raft/lifecycle/mod.rs index dee52d410..b76827ca0 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/mod.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/mod.rs @@ -5,13 +5,12 @@ mod records; pub(super) mod verification; mod waiting; use super::*; -use crate::raft::archive::{ArchiveLimits, ArchiveVerifier}; +use crate::raft::fleet_catalog::FleetGroupArchiveVerifier; use crate::raft::fleet_catalog::{ FleetFinalizationAdmission, FleetRemovalAdmission, FleetRemovalAttempt, FleetRemovalRequest, FleetSnapshot, }; use cellar_logic::consensus::ProcessGeneration; -use cellar_ltx::LtxHost; pub use records::*; use verification::{Advance, Key, Session, Verification}; pub use waiting::{ @@ -138,15 +137,10 @@ fn request_from(identity: &FleetRequestIdentity, catalog: &str) -> RaftFleetNode impl RaftFleetControl { /// Explicit replay services for one bounded lifecycle proof session. The - /// caller owns the private GenerationStore and its configured disk/memory - /// limits. This neither creates a node nor starts a background controller. - pub fn with_removal_verifier( - mut self, - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, - ) -> Result { - self.removal_verification = Some(Arc::new(Verification::new(host, verifier, limits)?)); + /// caller owns private group replay inventories and shared checkpoint, + /// archive, tail and scratch pools. This starts no background controller. + pub fn with_removal_verifier(mut self, verifier: FleetGroupArchiveVerifier) -> Result { + self.removal_verification = Some(Arc::new(Verification::new(verifier))); Ok(self) } @@ -284,7 +278,9 @@ impl RaftFleetControl { }); } let session = lease.session.as_mut().ok_or(FleetCatalogError::Closed)?; - let advance = runtime.advance(self, session, observer, &turn).await?; + let advance = runtime + .advance(self, session, observer, &root, &turn) + .await?; if advance != Advance::Exhausted { if advance == Advance::Waiting { let current = self.catalog.read(turn.remaining()?).await?; @@ -520,7 +516,9 @@ impl RaftFleetControl { }); } let session = lease.session.as_mut().ok_or(FleetCatalogError::Closed)?; - let advance = runtime.advance(self, session, observer, &turn).await?; + let advance = runtime + .advance(self, session, observer, &root, &turn) + .await?; if advance != Advance::Exhausted { if advance == Advance::Waiting { let current = self.catalog.read(turn.remaining()?).await?; @@ -726,7 +724,6 @@ impl RaftFleetControl { namespace: hex::encode(m.group.namespace.bytes()), group: hex::encode(m.group.group.bytes()), incarnation: hex::encode(m.group.incarnation.bytes()), - cell: m.group.cell, target: hex::encode(m.target.bytes()), peak_engine_slots: m.peak.engine_slots, peak_durable_bytes: m.peak.durable_bytes, diff --git a/crates/cellar/fleet/management/raft/lifecycle/records.rs b/crates/cellar/fleet/management/raft/lifecycle/records.rs index 470a29dbe..d1ab80779 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/records.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/records.rs @@ -240,7 +240,6 @@ pub struct RaftFleetEvacuationDebt { pub namespace: String, pub group: String, pub incarnation: String, - pub cell: String, pub target: String, pub peak_engine_slots: u32, pub peak_durable_bytes: u64, diff --git a/crates/cellar/fleet/management/raft/lifecycle/verification.rs b/crates/cellar/fleet/management/raft/lifecycle/verification.rs index 697b05eb1..223c33acf 100644 --- a/crates/cellar/fleet/management/raft/lifecycle/verification.rs +++ b/crates/cellar/fleet/management/raft/lifecycle/verification.rs @@ -2,21 +2,18 @@ //! At most one in-memory complete-inventory walk. Restart repeats its proofs. use super::*; -use crate::raft::archive::{ArchiveError, ArchiveLimits, ArchiveVerifier}; use crate::raft::fleet_catalog::{ - FleetEvacuationArchive, FleetEvacuationVerification, FleetHistoryState, + FleetEvacuationArchive, FleetEvacuationVerification, FleetGroupArchiveVerifier, + FleetHistoryState, }; -use cellar_logic::consensus::{MemberId, NodeId}; -use cellar_ltx::LtxHost; +use cellar_logic::consensus::NodeId; use std::sync::{ atomic::{AtomicBool, Ordering}, Mutex, }; pub(in super::super) struct Verification { - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, + archive: FleetGroupArchiveVerifier, session: Mutex>, busy: AtomicBool, } @@ -93,21 +90,12 @@ impl Verification { Ok(Some(value)) } - pub(super) fn new( - host: LtxHost, - verifier: ArchiveVerifier, - limits: ArchiveLimits, - ) -> Result { - limits - .validate() - .map_err(|_| RaftFleetFailure::invalid("removal archive bounds"))?; - Ok(Self { - host, - verifier, - limits, + pub(super) fn new(archive: FleetGroupArchiveVerifier) -> Self { + Self { + archive, session: Mutex::new(None), busy: AtomicBool::new(false), - }) + } } pub(super) fn lease(&self, key: &Key) -> Result> { @@ -140,6 +128,7 @@ impl Verification { control: &RaftFleetControl, session: &mut Session, observer: &RaftCells, + cut: &FleetSnapshot, turn: &Turn<'_>, ) -> Result { session.waiting = None; @@ -158,87 +147,56 @@ impl Verification { let namespace = control .namespaces .iter() - .find(|n| n.config().namespace == scope.namespace) - .ok_or(FleetCatalogError::InventoryIncomplete)?; - let birth = turn - .run(namespace.birth(&scope.cell)) - .await? - .map_err(FleetCatalogError::from)? + .find(|n| n.namespace_config().namespace == scope.namespace) .ok_or(FleetCatalogError::InventoryIncomplete)?; - if birth.scope() != &scope || birth.digest() != history.birth_digest() { - return Err(FleetCatalogError::Conflict.into()); - } - // Bind archive I/O to the management owner's execution domain - // even for SDK calls made outside an HTTP/domain task. - let archive = turn - .run(async { namespace.cell_archive(&birth, self.host.clone(), self.limits) }) - .await? - .map_err(FleetCatalogError::from)?; - let metadata = turn - .run(archive.recovery_metadata()) - .await? - .map_err(archive_error)? - .ok_or(FleetCatalogError::Unconfirmed)?; - let configuration = metadata.configuration().map_err(archive_error)?; - if configuration.voters.len() != 3 - || !configuration.voters_outgoing.is_empty() - || !configuration.learners.is_empty() - || !configuration.learners_next.is_empty() - { - return Err(FleetCatalogError::Unconfirmed.into()); - } - let membership = metadata.membership(); - let mut nodes = Vec::with_capacity(3); - for member in configuration.voters { - let member = MemberId::new(member) - .map_err(|_| FleetCatalogError::Corrupt("archive voter"))?; - let node = match membership - .joined(member) - .map_err(|_| FleetCatalogError::Corrupt("archive binding"))? - { - Some((binding, _)) => binding.node, - None => birth - .voters() - .iter() - .chain(birth.learners()) - .find(|v| v.member == member) - .map(|v| v.node) - .ok_or(FleetCatalogError::InventoryIncomplete)?, - }; - nodes.push(node); - } - let nodes: [NodeId; 3] = nodes + // Rooted current membership only selects candidates; the + // walk independently authenticates all three live survivors. + let (voters, _) = turn + .run(control.catalog.settled_group_voters(cut, &history)) + .await??; + let nodes: [NodeId; 3] = voters + .into_keys() + .collect::>() .try_into() .map_err(|_| FleetCatalogError::InventoryIncomplete)?; - let reference = metadata.reference(); + let fixed = match session + .walk + .freeze_live(observer, nodes, turn.remaining()?) + .await + { + Ok(fixed) => fixed, + Err(error) => return waiting(session, &history, error), + }; + let registry = turn + .run(self.archive.open(&control.domain, namespace, &history)) + .await??; + let hold = session + .walk + .acquire_archive_hold(®istry, turn.remaining()?) + .await?; + let receiver = turn + .run(self.archive.receiver( + &control.domain, + fixed.root().route(), + namespace.namespace_config().image_storage, + )) + .await??; let result = session .walk .verify_live( observer, nodes, FleetEvacuationArchive { - archive: &archive, - reference: &reference, - verifier: &self.verifier, + registry: ®istry, + hold: &hold, + tails: self.archive.tails(), + receiver, }, turn.remaining()?, ) .await; if let Err(error) = result { - let Some(blocker) = RaftFleetVerificationBlocker::observation( - &error, - RaftFleetVerificationStage::HistoryQuorum, - Some( - history - .binding() - .key() - .map_err(|_| FleetCatalogError::Corrupt("waiting history key"))?, - ), - ) else { - return Err(error.into()); - }; - session.waiting = Some(blocker); - return Ok(Advance::Waiting); + return waiting(session, &history, error); } } _ => return Err(FleetCatalogError::Unconfirmed.into()), @@ -251,14 +209,23 @@ impl Verification { } } -fn archive_error(error: ArchiveError) -> FleetCatalogError { - match error { - ArchiveError::Invalid(reason) => FleetCatalogError::Invalid(reason), - ArchiveError::Corrupt(reason) => FleetCatalogError::Corrupt(reason), - ArchiveError::Conflict => FleetCatalogError::Conflict, - ArchiveError::Limit => FleetCatalogError::Limit, - ArchiveError::Unconfirmed | ArchiveError::Timeout => FleetCatalogError::Unconfirmed, - ArchiveError::Stopped => FleetCatalogError::Closed, - error => FleetCatalogError::Storage(error.into()), - } +fn waiting( + session: &mut Session, + history: &crate::raft::fleet_catalog::FleetHistory, + error: FleetCatalogError, +) -> Result { + let Some(blocker) = RaftFleetVerificationBlocker::observation( + &error, + RaftFleetVerificationStage::HistoryQuorum, + Some( + history + .binding() + .key() + .map_err(|_| FleetCatalogError::Corrupt("waiting history key"))?, + ), + ) else { + return Err(error.into()); + }; + session.waiting = Some(blocker); + Ok(Advance::Waiting) } diff --git a/crates/cellar/fleet/management/raft/mod.rs b/crates/cellar/fleet/management/raft/mod.rs index f67894981..8bc13ae4e 100644 --- a/crates/cellar/fleet/management/raft/mod.rs +++ b/crates/cellar/fleet/management/raft/mod.rs @@ -12,7 +12,7 @@ mod nodes; mod records; use crate::application::consensus::RaftCells; use crate::asyncrt::{DomainToken, ExecutionDomain}; -use crate::raft::catalog::NamespaceCatalog; +use crate::raft::catalog::RangeDirectory; use crate::raft::fleet_catalog::{ FleetCatalog, FleetCatalogError, FleetCursor, FleetRequestIdentity, FleetTopologyIntentRequest, FleetTopologyOperation, FleetTopologyPhase, FleetTopologyPolicy, @@ -46,7 +46,7 @@ type Result = std::result::Result; pub struct RaftFleetControl { catalog: FleetCatalog, catalog_id: String, - namespaces: Arc<[NamespaceCatalog]>, + namespaces: Arc<[RangeDirectory]>, domain: DomainToken, node_observer: Option, removal_verification: Option>, @@ -56,7 +56,7 @@ impl RaftFleetControl { pub fn new( domain: &ExecutionDomain, catalog: FleetCatalog, - namespaces: Vec, + namespaces: Vec, ) -> Result { if namespaces.len() > 64 { return Err(RaftFleetFailure::invalid("too many managed namespaces")); @@ -233,7 +233,6 @@ impl RaftFleetControl { namespace: hex::encode(m.group.namespace.bytes()), group: hex::encode(m.group.group.bytes()), incarnation: hex::encode(m.group.incarnation.bytes()), - cell: m.group.cell, source: hex::encode(m.source.bytes()), target: hex::encode(m.target.bytes()), admitted_revision: m.admitted_revision, diff --git a/crates/cellar/fleet/management/raft/records.rs b/crates/cellar/fleet/management/raft/records.rs index 82b739cc9..df567853c 100644 --- a/crates/cellar/fleet/management/raft/records.rs +++ b/crates/cellar/fleet/management/raft/records.rs @@ -168,7 +168,6 @@ pub struct RaftFleetMovement { pub namespace: String, pub group: String, pub incarnation: String, - pub cell: String, pub source: String, pub target: String, pub admitted_revision: u64, diff --git a/crates/cellar/fleet/management/raft_stop.rs b/crates/cellar/fleet/management/raft_stop.rs index a3d2d5cc7..8d0d75ff2 100644 --- a/crates/cellar/fleet/management/raft_stop.rs +++ b/crates/cellar/fleet/management/raft_stop.rs @@ -1,7 +1,7 @@ // Copyright 2026 Cellar Project Authors. Licensed under Apache-2.0. //! Bounded stop observations, without per-group identities or deletion authority. -use crate::application::consensus::RaftNodeShutdownReport; +use crate::application::consensus::{MutationOutcomes, RaftNodeShutdownReport}; use anyhow::{ensure, Result}; use serde::{Deserialize, Serialize}; @@ -34,7 +34,10 @@ pub struct RaftStopOwners { pub omitted_group_details: usize, pub groups_with_archive_lag: usize, pub groups_with_unknown_archive: usize, - pub unfinished_commands: usize, + pub mutation_outcomes: MutationOutcomes, + /// Calls, including queries, retained when public admission was sealed. + /// This is not a count of unresolved business mutations. + pub retained_calls_at_shutdown: usize, } /// Accepted transfer requests and observed old-leader releases are separate. @@ -84,7 +87,8 @@ impl From<&RaftNodeShutdownReport> for RaftStopProof { omitted_group_details: owners.omitted_groups, groups_with_archive_lag: owners.groups_with_archive_lag, groups_with_unknown_archive: owners.groups_with_unknown_archive, - unfinished_commands: owners.unfinished.len(), + mutation_outcomes: owners.mutation_outcomes, + retained_calls_at_shutdown: owners.retained_calls_at_shutdown, }, handoff, } @@ -105,11 +109,8 @@ impl RaftStopProof { ); ensure!( owners.omitted_group_details <= owners.groups - && owners.groups_with_unknown_archive <= owners.omitted_group_details - && owners.unfinished_commands - <= owners - .groups - .saturating_sub(owners.groups_with_unknown_archive) + && (owners.mutation_outcomes != MutationOutcomes::NoUnresolvedLocalProducers + || owners.retained_calls_at_shutdown == 0) && owners .groups_with_archive_lag .checked_add(owners.groups_with_unknown_archive) @@ -126,10 +127,7 @@ impl RaftStopProof { } = handoff { ensure!( - *groups_examined - <= owners - .groups - .saturating_sub(owners.groups_with_unknown_archive) + *groups_examined <= owners.groups && *unavailable_groups <= *groups_examined && *accepted <= *groups_examined && *accepted <= *attempts @@ -156,7 +154,7 @@ impl RaftStopProof { deadline_elapsed: false, .. }, - } => owners.unfinished_commands == 0, + } => owners.mutation_outcomes == MutationOutcomes::NoUnresolvedLocalProducers, _ => false, } } diff --git a/crates/cellar/fleet/management/raft_stop_tests.rs b/crates/cellar/fleet/management/raft_stop_tests.rs index b96d58e4d..5ad3bd304 100644 --- a/crates/cellar/fleet/management/raft_stop_tests.rs +++ b/crates/cellar/fleet/management/raft_stop_tests.rs @@ -14,7 +14,8 @@ fn closed() -> RaftStopProof { omitted_group_details: 999_999, groups_with_archive_lag: 800_000, groups_with_unknown_archive: 100_000, - unfinished_commands: 0, + mutation_outcomes: MutationOutcomes::NoUnresolvedLocalProducers, + retained_calls_at_shutdown: 0, }, handoff: RaftStopHandoff::Observed { groups_examined: 10, @@ -61,7 +62,8 @@ fn raft_stop_refuses_incomplete_work_and_unknown_handoff() { } let mut proof = closed(); if let RaftStopProof::Closed { owners, .. } = &mut proof { - owners.unfinished_commands = 1; + owners.mutation_outcomes = MutationOutcomes::Unobserved; + owners.retained_calls_at_shutdown = 8; } assert!(!proof.complete()); let mut proof = closed(); @@ -86,6 +88,12 @@ fn raft_stop_refuses_incomplete_work_and_unknown_handoff() { #[test] fn raft_stop_refuses_fabricated_identity_counts_and_fields() { let valid = serde_json::to_value(closed()).unwrap(); + let mut invalid = valid.clone(); + invalid["owners"]["retained_calls_at_shutdown"] = serde_json::json!(8); + assert!(serde_json::from_value::(invalid) + .unwrap() + .validate() + .is_err()); for key in ["node", "disk", "process", "engine"] { let mut invalid = valid.clone(); invalid["owners"][key] = serde_json::json!(vec![0; 16]); @@ -116,3 +124,36 @@ fn raft_stop_refuses_fabricated_identity_counts_and_fields() { assert!(serde_json::from_value::(invalid).is_err()); assert!(serde_json::from_str::("{}").is_err()); } + +#[test] +fn shared_group_stop_never_infers_outcomes_from_call_count() { + for count in [0, 1, 8, 1_000_000] { + let mut proof = closed(); + if let RaftStopProof::Closed { owners, .. } = &mut proof { + owners.groups = 1; + owners.omitted_group_details = 0; + owners.groups_with_archive_lag = 0; + owners.groups_with_unknown_archive = 1; + owners.retained_calls_at_shutdown = count; + owners.mutation_outcomes = MutationOutcomes::Unobserved; + } + if let RaftStopProof::Closed { handoff, .. } = &mut proof { + *handoff = RaftStopHandoff::Observed { + groups_examined: 1, + attempts: 0, + accepted: 0, + leadership_released: 0, + unavailable_groups: 1, + deadline_elapsed: false, + }; + } + assert!(proof.validate().is_ok()); + assert!(!proof.complete()); + let mut value = serde_json::to_value(proof).unwrap(); + value["owners"] + .as_object_mut() + .unwrap() + .remove("mutation_outcomes"); + assert!(serde_json::from_value::(value).is_err()); + } +} diff --git a/crates/cellar/node/lifecycle.rs b/crates/cellar/node/lifecycle.rs index 385a9b685..755faa07d 100644 --- a/crates/cellar/node/lifecycle.rs +++ b/crates/cellar/node/lifecycle.rs @@ -205,7 +205,7 @@ mod tests { let outcome = NodeOutcome::Stopped(ShutdownReport { raft: Some(Box::new( crate::application::consensus::RaftNodeShutdownReport { - result: Err(crate::application::consensus::CommandFailure::Unavailable), + result: Err(crate::application::consensus::CommandFailure::Unavailable.into()), startup_incomplete: true, drain: None, }, diff --git a/crates/cellar/node/mod.rs b/crates/cellar/node/mod.rs index acb7d5ca6..2c8224d72 100644 --- a/crates/cellar/node/mod.rs +++ b/crates/cellar/node/mod.rs @@ -102,15 +102,16 @@ pub struct NodeRaftListeners { } /// Infrastructure handles owned by the embedded node's execution domain. The -/// host dispatcher consumes events, routes messages and supplies verified image -/// completions. Transport admission itself is never a write acknowledgement. -/// Recovered groups remain parked until catalog/bootstrap admission attaches them. +/// group controller consumes events, routes messages and supplies verified +/// whole-group manifest completions. Transport admission itself is never a write +/// acknowledgement. Recovered groups remain parked until their exact catalog +/// member permit and complete selected root authorize attachment. #[doc(hidden)] pub struct NodeRaftService { - pub log: crate::application::consensus::verification::ReplicaLog, - pub runtime: crate::application::consensus::verification::RaftRuntime, - pub events: crate::application::consensus::verification::RuntimeEvents, - pub transport: crate::application::consensus::verification::RaftTransport, + pub log: crate::raft::storage::ReplicaLog, + pub runtime: crate::raft::cell_groups::CellGroupRuntime, + pub events: crate::raft::cell_groups::CellGroupRuntimeEvents, + pub transport: crate::raft::transport::RaftTransport, } /// Let an admitted HTTP request finish, then close the transport even when a diff --git a/crates/cellar/node/raft_startup.rs b/crates/cellar/node/raft_startup.rs index 0d5992449..a7a0aa501 100644 --- a/crates/cellar/node/raft_startup.rs +++ b/crates/cellar/node/raft_startup.rs @@ -49,177 +49,26 @@ impl Startup<'_> { crate::application::consensus::RaftCells, Option, )> { - use crate::application::consensus::{RaftCellPlacement, RegistrationError}; anyhow::ensure!( config.infrastructure.log.directory == self.data_dir.join("raft"), "Raft log must use this node's raft directory" ); - config.host.validate()?; - anyhow::ensure!( - config.host.extents.is_some() - || config.host.managed.as_ref().is_none_or(|managed| { - managed.namespaces.iter().all(|entry| { - entry.namespace.image_storage - != crate::application::consensus::RaftImageStorage::Extents - }) - }), - "managed extent namespace requires node extent storage" - ); - config.infrastructure.transport.validate()?; - config - .infrastructure - .transport - .snapshot_limits(config.host.execution.max_image_bytes)?; - anyhow::ensure!( - (!config.cells.is_empty() || config.host.managed.is_some()) - && config.cells.len() <= config.host.max_groups, - "invalid registered Raft cell count" - ); - let mut definitions = Vec::new(); - for cell in &config.cells { - anyhow::ensure!( - cell.birth.voters.len() <= 32 && cell.birth.learners.len() <= 32, - "registered birth membership exceeds the catalog bound" - ); - if let Some(managed) = &config.host.managed { - anyhow::ensure!( - managed - .namespaces - .iter() - .any(|n| n.namespace == cell.namespace && n.class == cell.class), - "cell is outside managed host capabilities" - ); - } - anyhow::ensure!( - cell.namespace.image_storage - != crate::application::consensus::RaftImageStorage::Extents - || matches!(cell.placement, RaftCellPlacement::QueryRouter { .. }) - || config.host.extents.is_some(), - "extent namespace requires node extent storage" - ); - anyhow::ensure!( - !services.application.supports_cell(&cell.class) - && !services.application.supports_workflow(&cell.class), - RegistrationError::UnsupportedFactory(cell.class.clone()) - ); - let class = services.application.resolve_raft_class(&cell.class)?; - class.validate_host(&config.host)?; - anyhow::ensure!( - cell.namespace.mode == cellar_logic::NamespaceMode::Raft - && cell.namespace.schema_digest == class.schema_digest(), - "Raft namespace mode/schema disagrees with registration" - ); - anyhow::ensure!( - cellar_logic::cell::valid_cell_scope(&cell.birth.scope.cell) - && cell - .birth - .scope - .cell - .split_once(':') - .is_some_and(|(name, _)| name == class.name()), - "Raft cell scope must name its registered class" - ); - definitions.push(class); - } - let mut declared_definitions = Vec::new(); - if let Some(managed) = &config.host.managed { - anyhow::ensure!( - managed.authorization.node() == config.infrastructure.log.node - && managed.authorization.disk() == config.infrastructure.log.disk, - "managed startup identity mismatch" - ); - for entry in &managed.namespaces { - anyhow::ensure!( - !services.application.supports_cell(&entry.class) - && !services.application.supports_workflow(&entry.class), - RegistrationError::UnsupportedFactory(entry.class.clone()) - ); - let class = services.application.resolve_raft_class(&entry.class)?; - class.validate_host(&config.host)?; - anyhow::ensure!( - class.schema_digest() == entry.namespace.schema_digest, - "managed compiled schema mismatch" - ); - declared_definitions.push(class); - } - } - let (performance, registrations, declared_namespaces) = self + let (performance, prepared) = self .domain .run(async { let performance = crate::raft::performance::Registry::new(self.domain.clone()); let store = crate::raft::object_metrics::wrap(services.store, performance.clone()); - let catalog = crate::raft::catalog::Catalog::new(store, config.catalog)?; - let mut declared_namespaces = Vec::new(); - if let Some(managed) = &mut config.host.managed { - managed.authorization = managed - .authorization - .refresh(std::time::Duration::from_secs(5)) - .await?; - if managed.authorization.state() - == crate::raft::fleet_catalog::FleetNodeState::RestartPending - { - crate::asyncrt::services() - .raft_lifecycle() - .hold_startup_maintenance(); - } - for (entry, class) in managed.namespaces.iter().zip(declared_definitions) { - let namespace = catalog.existing_namespace(entry.namespace.clone()).await?; - namespace - .require_version_keys( - &services.versions, - u64::try_from(crate::asyncrt::wall_ms())?, - ) - .await?; - declared_namespaces - .push(crate::raft::host::NamespaceRegistration { namespace, class }); - } - } - let managed = config.host.managed.is_some(); - let mut registrations = Vec::new(); - for (cell, class) in config.cells.into_iter().zip(definitions) { - let namespace = match cell.placement { - RaftCellPlacement::Replica if !managed => { - catalog.open_namespace(cell.namespace).await? - } - _ => catalog.existing_namespace(cell.namespace).await?, - }; - let now_ms = u64::try_from(crate::asyncrt::wall_ms())?; - namespace - .require_version_keys(&services.versions, now_ms) - .await?; - let birth = - match cell.placement { - RaftCellPlacement::Replica if !managed => { - namespace.resolve_birth(cell.birth).await? - } - _ => { - let mut expected = cell.birth; - expected.voters.sort_by_key(|member| member.member); - expected.learners.sort_by_key(|member| member.member); - let birth = - namespace.birth(&expected.scope.cell).await?.ok_or_else( - || anyhow::anyhow!("existing member/route birth missing"), - )?; - anyhow::ensure!( - birth.scope() == &expected.scope - && birth.voters() == expected.voters.as_slice() - && birth.learners() == expected.learners.as_slice(), - "member/route disagrees with existing birth" - ); - birth - } - }; - registrations.push(crate::raft::host::Registration { - placement: cell.placement, - namespace, - birth, - class, - }); - } - Ok::<_, anyhow::Error>((performance, registrations, declared_namespaces)) + let prepared = crate::raft::group_host::prepare( + &mut config, + services.application.as_ref(), + store, + &services.versions, + ) + .await?; + Ok::<_, anyhow::Error>((performance, prepared)) }) .await??; - let service = self + let infrastructure = self .infrastructure( config.infrastructure, services.listeners, @@ -227,34 +76,17 @@ impl Startup<'_> { Some(performance), ) .await?; - let log = service.log.clone(); - let runtime = service.runtime.clone(); - let transport = service.transport.clone(); - let stop = config - .host - .managed - .as_ref() - .map(|managed| { - Ok::<_, anyhow::Error>(crate::raft::fleet_stop::Services { - authorization: managed.authorization.clone(), - log: log.clone(), - signer: crate::asyncrt::services().peer_probe_signer()?, - }) - }) - .transpose()?; + let log = infrastructure.log.clone(); + let runtime = infrastructure.runtime.clone(); + let transport = infrastructure.transport.clone(); let result = self .domain - .run(crate::raft::host::start( - service, - crate::raft::host::Registrations { - cells: registrations, - namespaces: declared_namespaces, - peers: config.peers, - origins: config.query_origins, - }, + .run(crate::raft::group_host::start( + infrastructure, + prepared, self.data_dir.join("raft-cells"), - self.node.to_owned(), config.host, + config.shared, services.versions, )) .await; @@ -263,13 +95,11 @@ impl Startup<'_> { Err(error) => Err(error.into()), }; if result.is_err() { - // Domain entry/cancellation can fail before the host's owned - // startup guard runs. Join the infrastructure here as well. let _ = runtime.close().await; transport.close().await; let _ = log.close().await; } - result.map(|service| (service, stop)) + result.map(|service| (service, None)) } pub(super) async fn infrastructure( @@ -279,7 +109,9 @@ impl Startup<'_> { tls: crate::application::consensus::verification::RaftTls, performance: Option>, ) -> anyhow::Result { - use crate::application::consensus::verification::{RaftRuntime, RaftTransport, ReplicaLog}; + use crate::raft::{ + cell_groups::CellGroupRuntime, storage::ReplicaLog, transport::RaftTransport, + }; anyhow::ensure!( config.log.directory == self.data_dir.join("raft"), "Raft log must use this node's raft directory" @@ -289,14 +121,14 @@ impl Startup<'_> { let log = ReplicaLog::open_observed(config.log, crate::asyncrt::fs(), performance) .await?; let (runtime, events) = - match RaftRuntime::start_parked(log.clone(), config.consensus).await { + match CellGroupRuntime::start(log.clone(), config.consensus).await { Ok(runtime) => runtime, Err(error) => { let _ = log.close().await; return Err(anyhow::Error::new(error)); } }; - let transport = match RaftTransport::start( + let transport = match RaftTransport::start_shared( runtime.clone(), self.auth.clone(), tls, diff --git a/crates/cellar/tests/native_cell_groups_catalog.rs b/crates/cellar/tests/native_cell_groups_catalog.rs index e8e4df190..8d5ab6ef2 100644 --- a/crates/cellar/tests/native_cell_groups_catalog.rs +++ b/crates/cellar/tests/native_cell_groups_catalog.rs @@ -15,6 +15,11 @@ mod directory; #[allow(dead_code)] mod fault; +#[tokio::test] +async fn group_import_freeze_retains_pending_and_fences_topology() -> anyhow::Result<()> { + directory::group_import_freeze().await +} + #[tokio::test] async fn directory_inventory_pages_bind_the_exact_topology_cut() -> anyhow::Result<()> { directory::inventory_pages().await diff --git a/crates/cellar/tests/native_cell_groups_execution.rs b/crates/cellar/tests/native_cell_groups_execution.rs index 9d397f488..14a13eb4d 100644 --- a/crates/cellar/tests/native_cell_groups_execution.rs +++ b/crates/cellar/tests/native_cell_groups_execution.rs @@ -181,6 +181,16 @@ async fn membership_group_configuration_waits_for_sync_and_joins_close() -> Resu runtime_fixture::membership_sync().await } +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_quorum_waits_for_durable_floor_before_its_barrier() -> Result<()> { + runtime_fixture::membership_quorum_floor().await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn membership_follower_quorum_preserves_group_root_without_leader_authority() -> Result<()> { + runtime_fixture::membership_quorum_followers().await +} + #[tokio::test(flavor = "multi_thread", worker_threads = 3)] async fn checkpoint_controller_compacts_materialized_groups_during_writes() -> Result<()> { runtime_fixture::automatic_checkpoints(false, false).await @@ -1392,3 +1402,33 @@ async fn retained_log_materialized_refuses_new_callbacks_and_reclaims() -> Resul async fn retained_log_extents_refuse_new_callbacks_and_reclaim() -> Result<()> { Box::pin(runtime_fixture::retained_log_admission(true)).await } + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_follower_linearizable_reads_bind_live_quorum() -> Result<()> { + Box::pin(runtime_fixture::follower_linearizable(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_follower_linearizable_reads_bind_live_quorum() -> Result<()> { + Box::pin(runtime_fixture::follower_linearizable(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_recent_source_reads_keep_physical_deadline() -> Result<()> { + Box::pin(runtime_fixture::recent_source_reads(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_recent_source_reads_keep_physical_deadline() -> Result<()> { + Box::pin(runtime_fixture::recent_source_reads(true)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn shared_remote_follower_reads_preserve_constraints() -> Result<()> { + Box::pin(runtime_fixture::remote_follower_reads(false)).await +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 3)] +async fn rooted_remote_follower_reads_preserve_constraints() -> Result<()> { + Box::pin(runtime_fixture::remote_follower_reads(true)).await +} diff --git a/crates/cellar/tests/support/cell_group_controller_archive.rs b/crates/cellar/tests/support/cell_group_controller_archive.rs index b1dc65702..782b8c30e 100644 --- a/crates/cellar/tests/support/cell_group_controller_archive.rs +++ b/crates/cellar/tests/support/cell_group_controller_archive.rs @@ -125,6 +125,7 @@ pub(super) async fn registration_minimum( route.schema_digest, ); config.minimum_durability = minimum; + config.image_storage = fixture.cluster.directory.namespace_config().image_storage; let namespace = fixture .cluster .domain diff --git a/crates/cellar/tests/support/cell_group_crash_fixture.rs b/crates/cellar/tests/support/cell_group_crash_fixture.rs index f4eeed434..059633146 100644 --- a/crates/cellar/tests/support/cell_group_crash_fixture.rs +++ b/crates/cellar/tests/support/cell_group_crash_fixture.rs @@ -121,8 +121,10 @@ pub(super) async fn restore_catalog( objects: Vec<(String, Vec)>, route: &GroupRoute, domain: &ExecutionDomain, + image_storage: RaftImageStorage, ) -> Result<(RangeDirectory, CellGroupBirth)> { - let (directory, birth, _) = restore_catalog_store(objects, route, domain).await?; + let (directory, birth, _) = + restore_catalog_store(objects, route, domain, image_storage).await?; Ok((directory, birth)) } @@ -130,6 +132,7 @@ pub(super) async fn restore_catalog_store( objects: Vec<(String, Vec)>, route: &GroupRoute, domain: &ExecutionDomain, + image_storage: RaftImageStorage, ) -> Result<( RangeDirectory, CellGroupBirth, @@ -140,14 +143,14 @@ pub(super) async fn restore_catalog_store( store.put(&ObjectPath::from(name), bytes.into()).await?; } let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; - let namespace = domain - .run(catalog.open_namespace(NamespaceConfig::new( - route.history.cluster, - route.history.namespace, - cellar_logic::NamespaceMode::Raft, - route.schema_digest, - ))) - .await??; + let mut config = NamespaceConfig::new( + route.history.cluster, + route.history.namespace, + cellar_logic::NamespaceMode::Raft, + route.schema_digest, + ); + config.image_storage = image_storage; + let namespace = domain.run(catalog.existing_namespace(config)).await??; let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; let birth = domain .run(directory.materialize_selected_group(route)) diff --git a/crates/cellar/tests/support/cell_group_directory.rs b/crates/cellar/tests/support/cell_group_directory.rs index 8d9578bc3..fe778c6b0 100644 --- a/crates/cellar/tests/support/cell_group_directory.rs +++ b/crates/cellar/tests/support/cell_group_directory.rs @@ -24,6 +24,96 @@ struct Fixture { limits: DirectoryLimits, } +pub(super) async fn group_import_freeze() -> Result<()> { + let f = Fixture::new(DirectoryLimits::default()).await?; + let mut selected = ranges(3); + f.reserve(&mut selected).await?; + let genesis = f + .directory + .prepare_initialization(operation(71000), &selected)?; + f.run(f.directory.publish(&genesis)).await?; + let before = f.run(f.directory.observe()).await?; + let mut children = split(&selected[1], 71001); + f.reserve(&mut children).await?; + let stale = + f.directory + .prepare_replacement(&before, operation(71002), &selected[1..2], &children)?; + let pending = + f.directory + .prepare_group_birth(birth_spec(&assigned(71003, CellRange::all(), 2)))?; + f.run(f.directory.reserve_group_birth(&pending)).await?; + let delayed = + f.directory + .prepare_group_birth(birth_spec(&assigned(71004, CellRange::all(), 2)))?; + f.store + .birth_inventory_cas_hang + .store(true, Ordering::Release); + f.store + .hold_next_birth_inventory_cas + .store(true, Ordering::Release); + let writer = f.run(f.directory.reserve_group_birth(&delayed)); + let freezer = async { + let reached = tokio::time::timeout(Duration::from_secs(5), async { + while !f.store.birth_inventory_cas_waiting.load(Ordering::Acquire) { + tokio::task::yield_now().await; + } + }) + .await; + // Lose the reply after the actual namespace freeze CAS. The original + // operation must reconcile; no successful CAS is synthesized. + f.store + .fail_birth_inventory_after + .store(true, Ordering::Release); + let frozen = if reached.is_ok() { + f.run(f.directory.freeze_group_births(operation(71005))) + .await + } else { + Err(CatalogError::Unconfirmed) + }; + f.store + .birth_inventory_cas_hang + .store(false, Ordering::Release); + f.store.birth_inventory_cas_resume.notify_waiters(); + frozen + }; + let (delayed_result, frozen) = tokio::join!(writer, freezer); + anyhow::ensure!(matches!(delayed_result, Err(CatalogError::BirthsFrozen))); + let frozen = frozen?; + anyhow::ensure!(frozen.len() == 6 && !frozen.is_empty()); + anyhow::ensure!(frozen.operation() == operation(71005)); + anyhow::ensure!(matches!( + f.run(f.directory.publish(&stale)).await, + Err(CatalogError::BirthsFrozen) + )); + let new = f + .directory + .prepare_group_birth(birth_spec(&assigned(71006, CellRange::all(), 2)))?; + anyhow::ensure!(matches!( + f.run(f.directory.reserve_group_birth(&new)).await, + Err(CatalogError::BirthsFrozen) + )); + anyhow::ensure!(matches!( + f.run(f.directory.freeze_group_births(operation(71007))) + .await, + Err(CatalogError::Conflict(_)) + )); + let after = f.run(f.directory.observe()).await?; + anyhow::ensure!(after.digest() != before.digest() && after.groups() == 3); + for route in &selected { + // Lazy selected births remain live; no cell is chosen to stand in for + // the physical group. Their voters still need real bootstrap/quorum. + let birth = f.run(f.directory.materialize_selected_group(route)).await?; + anyhow::ensure!(birth.route() == route); + } + let reopened = f.reopen().await?; + let retried = f + .run(reopened.freeze_group_births(operation(71005))) + .await?; + anyhow::ensure!(retried.len() == frozen.len() && retried.operation() == frozen.operation()); + anyhow::ensure!(f.store.listing_calls.load(Ordering::Acquire) == 0); + Ok(()) +} + impl Fixture { async fn new(limits: DirectoryLimits) -> Result { Self::with_schema(limits, route().schema_digest).await @@ -632,6 +722,23 @@ pub async fn birth_recovery() -> Result<()> { assert_eq!(retry.route(), request.route()); let birth = f.run(restarted.materialize_group_birth(&retry)).await?; assert_eq!(birth.route(), route); + let selected = f + .run(restarted.materialize_selected_history(route.history)) + .await?; + assert_eq!(selected.route(), route); + let mut wrong = route.history; + wrong.incarnation = IncarnationId::new([77; 16])?; + assert!(matches!( + f.run(restarted.materialize_selected_history(wrong)).await, + Err(CatalogError::Fenced) + )); + wrong = route.history; + wrong.namespace = NamespaceId::new([77; 16])?; + assert!(matches!( + f.run(restarted.materialize_selected_history(wrong)).await, + Err(CatalogError::Conflict(_)) + )); + assert_eq!(birth.creation(), request.creation()); assert_eq!(birth.voters(), request.voters()); assert!(birth.learners().is_empty()); @@ -677,6 +784,11 @@ pub async fn birth_recovery() -> Result<()> { f.run(restarted.materialize_group_birth(&retry)).await, Err(CatalogError::Fenced) )); + assert!(matches!( + f.run(restarted.materialize_selected_history(route.history)) + .await, + Err(CatalogError::Fenced) + )); assert_eq!( f.store.inner.get(&birth_key).await?.bytes().await?.as_ref(), request.encode() diff --git a/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs index 694acce7e..e9451801f 100644 --- a/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs +++ b/crates/cellar/tests/support/cell_group_dispatch_cache_execution.rs @@ -26,7 +26,7 @@ pub(crate) async fn cached_dispatch_execution() -> Result<()> { ); let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let gateways = gateways(&fixture, class.clone()).await?; + let gateways = gateways(&fixture, &controllers, class.clone()).await?; let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; let mut warm = invocation( &dispatch, diff --git a/crates/cellar/tests/support/cell_group_host.rs b/crates/cellar/tests/support/cell_group_host.rs index 8b36856e2..e37aaa695 100644 --- a/crates/cellar/tests/support/cell_group_host.rs +++ b/crates/cellar/tests/support/cell_group_host.rs @@ -280,6 +280,13 @@ pub(crate) async fn many_cells(rooted: bool) -> Result<()> { } pub(crate) async fn query_shutdown(rooted: bool) -> Result<()> { + for expired in [false, true] { + held_query_shutdown(rooted, expired).await?; + } + Ok(()) +} + +async fn held_query_shutdown(rooted: bool, expired: bool) -> Result<()> { let (entered, started) = tokio::sync::oneshot::channel(); let entered = Mutex::new(Some(entered)); let (release, held) = std::sync::mpsc::sync_channel(1); @@ -326,13 +333,35 @@ pub(crate) async fn query_shutdown(rooted: bool) -> Result<()> { ); idle(&host).await?; let pool = f.appliers[&(0, route.history)].0.clone(); - let call = handle.query( - query(&class, CellQueryConsistency::Linearizable)?, - routing(), - )?; + let mut limits = routing(); + if expired { + limits.timeout = Duration::from_millis(600); + } + let call = handle.query(query(&class, CellQueryConsistency::Linearizable)?, limits)?; tokio::time::timeout(Duration::from_secs(5), started).await??; assert_eq!(host.usage().controller.retained_commands, 1); - drop(call); + if expired { + let error = call + .response() + .await + .err() + .context("original read deadline")?; + assert!(matches!( + error.as_ref(), + CellDispatchError::Runtime(RuntimeError::DeadlineExceeded) + )); + } else { + drop(call); + } + assert_eq!(host.usage().controller.retained_commands, 1); + assert!( + f.cluster + .domain + .services() + .application_work() + .pending_events + > 0 + ); let refused = handle .command(command(&class, 1, retry_until)?, routing()) .err() @@ -357,6 +386,14 @@ pub(crate) async fn query_shutdown(rooted: bool) -> Result<()> { tokio::time::timeout(Duration::from_secs(5), host.close()).await??; assert_eq!(pool.usage()?.running_workers, 0); assert_eq!(host.usage().controller.retained_commands, 0); + assert_eq!( + f.cluster + .domain + .services() + .application_work() + .pending_events, + 0 + ); for c in &controllers { c.close().await?; } diff --git a/crates/cellar/tests/support/cell_group_log_reclaim.rs b/crates/cellar/tests/support/cell_group_log_reclaim.rs index 403d993a4..0f7acb684 100644 --- a/crates/cellar/tests/support/cell_group_log_reclaim.rs +++ b/crates/cellar/tests/support/cell_group_log_reclaim.rs @@ -395,14 +395,22 @@ async fn run(rooted: bool, store: Arc) -> Result< Ok(Sha256::digest(input).to_vec()) })?, ); - let cluster = Box::pin(Cluster::with_store_log_limits( + let cluster = Box::pin(Cluster::with_store_birth( RaftRuntimeOptions::default(), tempfile::tempdir()?, Arc::new(DirectFileSystem), class.schema_digest(), store, - Durability::Quorum, - log_limits(), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum: Durability::Quorum, + log: log_limits(), + }, )) .await?; let (mut fixture, route, a, b) = Box::pin(registered::setup_cluster( diff --git a/crates/cellar/tests/support/cell_group_membership.rs b/crates/cellar/tests/support/cell_group_membership.rs index e48f61898..b67adeb41 100644 --- a/crates/cellar/tests/support/cell_group_membership.rs +++ b/crates/cellar/tests/support/cell_group_membership.rs @@ -87,6 +87,35 @@ pub(crate) async fn membership_lifecycle(rooted: bool) -> Result<()> { fixture.runtimes[0].campaign(second).await?; controller::wait_for(async || Ok(fixture.runtimes[0].status(second).await?.ready)).await?; let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + // Administrative observations address the physical group, with no chosen + // cell. Every invocation obtains its own current-term ReadIndex barrier. + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observation = fixture.runtimes[0] + .quorum_membership(route.history, deadline) + .await?; + assert_eq!(observation.replica().scope, route.history); + assert_eq!(observation.root().route(), &route); + assert_eq!(observation.configuration().voters, vec![1, 2, 3]); + assert!(observation.barrier().index.get() > 0); + assert!(observation.root().position().index >= observation.barrier().index); + assert!(observation.membership().revision() <= observation.barrier().index); + assert!(fixture.runtimes[0].usage().read_pins.items > 0); + drop(observation); + let expired = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() }) + .await?; + assert!(matches!( + fixture.runtimes[0] + .quorum_membership(route.history, expired) + .await, + Err(RuntimeError::DeadlineExceeded) + )); let original = fixture.runtimes[0] .execution_authority(route.history, a.clone()) .await?; @@ -111,6 +140,21 @@ pub(crate) async fn membership_lifecycle(rooted: bool) -> Result<()> { .is_err()); assert_eq!(group.last_index()?, last); let entered = configure(&fixture, &route, demote, &[]).await?; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observation = fixture.runtimes[0] + .quorum_membership(route.history, deadline) + .await?; + assert_eq!(observation.membership().revision(), entered.position.index); + assert!(observation.barrier().index >= entered.position.index); + assert_eq!(observation.configuration().voters, vec![1, 2]); + assert_eq!(observation.configuration().voters_outgoing, vec![1, 2, 3]); + assert_eq!(observation.configuration().learners_next, vec![3]); + assert_eq!(observation.root().root(), root); + drop(observation); for voter in &fixture.cluster.voters { let log = voter.log.cell_group(&route.history)?; let conf = log.initial_state()?.conf_state; @@ -289,6 +333,36 @@ pub(crate) async fn membership_lifecycle(rooted: bool) -> Result<()> { original.cell() ); + let progress = fixture.runtimes[0] + .replication_status(route.history) + .await?; + assert_eq!(progress.replica.scope, route.history); + assert_eq!(progress.replica.member, MemberId::new(1)?); + for (term, revision) in [ + ( + Term::new(progress.term.get() + 1), + progress.membership_revision, + ), + ( + progress.term, + LogIndex::new(progress.membership_revision.get() + 1), + ), + ] { + assert!(matches!( + fixture.runtimes[0] + .transfer_leadership( + route.history, + cellar::application::consensus::LeadershipTransfer { + target: MemberId::new(2)?, + expected_term: term, + expected_revision: revision + } + ) + .await, + Err(RuntimeError::Invalid(_)) + )); + } + // Move leadership using the real TimeoutNow protocol. The fixture does // not advance election ticks or inject votes to manufacture this handoff. assert!(fixture.runtimes[0] @@ -468,3 +542,166 @@ pub(crate) async fn membership_sync() -> Result<()> { } Ok(()) } + +pub(crate) async fn membership_quorum_floor() -> Result<()> { + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, _, _) = Box::pin(registered::setup(rooted, &class)).await?; + fixture.connect_tcp().await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let runtime = fixture.runtimes[0].clone(); + controller::wait_for(async || Ok(runtime.status(route.history).await?.ready)).await?; + let group = fixture.cluster.voters[0].log.cell_group(&route.history)?; + let before = group.group_manifest()?.context("before")?; + let request = CellGroupMembershipChange::new( + &route, + &group.membership()?, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + )?; + let gate = fixture.cluster.voters[0].gate.clone(); + let release = gate.arm(fault::HOLD_SYNC); + let accepted = runtime.configure(route.history, request).await?; + tokio::time::timeout(Duration::from_secs(5), gate.entered.notified()).await?; + assert_eq!(group.group_manifest()?.as_ref(), Some(&before)); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let read = + runtime.quorum_membership_after(route.history, accepted.position.index, deadline); + tokio::pin!(read); + assert!(tokio::time::timeout(Duration::from_millis(30), &mut read) + .await + .is_err()); + assert!(runtime.usage().read_pins.items > 0); + drop(release); + let observed = tokio::time::timeout(Duration::from_secs(10), &mut read).await??; + assert_eq!(observed.membership().revision(), accepted.position.index); + assert!(observed.barrier().index >= accepted.position.index); + assert!(observed.root().position().index >= observed.barrier().index); + assert_eq!(observed.configuration().learners_next, vec![3]); + drop(observed); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + let unreachable = LogIndex::new(accepted.position.index.get() + 10_000); + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 30 }) + .await?; + assert!(matches!( + runtime + .quorum_membership_after(route.history, unreachable, deadline) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} + +pub(crate) async fn membership_quorum_followers() -> Result<()> { + use raft::prelude::{Entry, Message, MessageType}; + for rooted in [false, true] { + let class = RaftClass::new("Counter", 1, [81; 32])?; + let (mut fixture, route, cell, _) = Box::pin(registered::setup(rooted, &class)).await?; + fixture.connect_tcp().await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + controller::wait_for(async || Ok(fixture.runtimes[0].status(route.history).await?.ready)) + .await?; + let accepted = configure_at( + &fixture, + 0, + &route, + change( + 3, + ConfChangeType::AddLearnerNode, + ConfChangeTransition::Explicit, + ), + &[], + ) + .await?; + let expected = fixture.cluster.voters[0] + .log + .cell_group(&route.history)? + .group_manifest()? + .context("selected group")?; + assert_eq!(expected.root().cells(), 2); + + let mut context = b"cellar-group-membership-read-1\0".to_vec(); + context.extend_from_slice(&2u64.to_be_bytes()); + context.extend_from_slice(&[97; 16]); + context.extend_from_slice(&1u64.to_be_bytes()); + let status = fixture.runtimes[1].status(route.history).await?; + let forged = Message { + msg_type: MessageType::MsgReadIndexResp, + from: 3, + to: 2, + term: status.term.get(), + index: expected.position().index.get(), + entries: vec![Entry { + data: context.into(), + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!(matches!( + fixture.runtimes[1].step(route.history, forged).await, + Err(RuntimeError::Invalid(_)) + )); + + for node in 0..3 { + let runtime = &fixture.runtimes[node]; + let deadline = fixture + .cluster + .domain + .run(async { cellar::asyncrt::mono_ms() + 10_000 }) + .await?; + let observed = runtime + .quorum_membership_after(route.history, accepted.position.index, deadline) + .await?; + assert_eq!(observed.replica().member.get(), node as u64 + 1); + assert_eq!( + observed.engine(), + fixture.cluster.voters[node].log.storage_instance()? + ); + assert_eq!(observed.root(), &expected); + assert!(observed.barrier().index >= accepted.position.index); + assert_eq!(observed.barrier().term, observed.status().term); + assert_eq!(observed.status().leader, Some(1)); + assert_eq!(observed.configuration().learners_next, vec![3]); + if node != 0 { + assert!(!observed.status().ready); + // A metadata barrier cannot become local leader/execution or + // membership-transfer authority on either follower. + assert!(matches!( + runtime.replication_status(route.history).await, + Err(RuntimeError::NotLeader) + )); + assert!(matches!( + runtime + .execution_authority(route.history, cell.clone()) + .await, + Err(RuntimeError::NotLeader) + )); + } + drop(observed); + controller::wait_for(async || Ok(runtime.usage().read_pins.items == 0)).await?; + } + for controller in controllers { + controller.close().await?; + } + fixture.close().await?; + } + Ok(()) +} diff --git a/crates/cellar/tests/support/cell_group_membership_crash.rs b/crates/cellar/tests/support/cell_group_membership_crash.rs index 2d7e6dc85..a54fbf85e 100644 --- a/crates/cellar/tests/support/cell_group_membership_crash.rs +++ b/crates/cellar/tests/support/cell_group_membership_crash.rs @@ -13,6 +13,7 @@ use std::path::Path; #[serde(deny_unknown_fields)] struct Oracle { rooted: bool, + image_storage: RaftImageStorage, joint: bool, source_directory: String, expected: GroupManifest, @@ -97,6 +98,7 @@ pub(crate) async fn membership_process_fixture() -> Result<()> { &root.join("oracle.json"), &serde_json::to_vec(&Oracle { rooted, + image_storage: fixture.cluster.directory.namespace_config().image_storage, joint: cut == "joint", source_directory, expected, @@ -162,7 +164,8 @@ async fn recover(root: &Path, rooted: bool, joint: bool) -> Result<()> { let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let route = oracle.expected.route().clone(); let (directory, birth) = - crash_fixture::restore_catalog(oracle.objects, &route, &domain).await?; + crash_fixture::restore_catalog(oracle.objects, &route, &domain, oracle.image_storage) + .await?; let mut nodes = Vec::new(); let mut peers = Vec::new(); for n in 1..=3u8 { diff --git a/crates/cellar/tests/support/cell_group_outbox_effects.rs b/crates/cellar/tests/support/cell_group_outbox_effects.rs index 86d1d93f8..3e58063c9 100644 --- a/crates/cellar/tests/support/cell_group_outbox_effects.rs +++ b/crates/cellar/tests/support/cell_group_outbox_effects.rs @@ -320,7 +320,11 @@ async fn run_image( class.schema_digest(), store, ClusterBirthOptions { - image_storage: RaftImageStorage::Materialized, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, group, minimum, log: ClusterLogLimits::default(), diff --git a/crates/cellar/tests/support/cell_group_provision.rs b/crates/cellar/tests/support/cell_group_provision.rs index dca961c76..6b9c43fa6 100644 --- a/crates/cellar/tests/support/cell_group_provision.rs +++ b/crates/cellar/tests/support/cell_group_provision.rs @@ -20,15 +20,21 @@ async fn assignment( incarnation: IncarnationId::new([number; 16])?, }; let catalog = Catalog::new(store.clone(), CatalogOptions::default())?; + let mut namespace_config = NamespaceConfig::new( + scope.cluster, + scope.namespace, + cellar_logic::NamespaceMode::Raft, + old.schema_digest, + ); + namespace_config.image_storage = if f.cluster.voters[0].extents.is_some() { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }; let namespace = f .cluster .domain - .run(catalog.open_namespace(NamespaceConfig::new( - scope.cluster, - scope.namespace, - cellar_logic::NamespaceMode::Raft, - old.schema_digest, - ))) + .run(catalog.open_namespace(namespace_config)) .await??; let directory = RangeDirectory::new(namespace, DirectoryLimits::default())?; let birth = directory.prepare_group_birth(CellGroupBirthSpec { @@ -83,6 +89,7 @@ async fn settings( f: &RuntimeCluster, a: &CellHistory, node: usize, + directories: &[RangeDirectory], ) -> Result { let images = match f.cell_options_at(a, node).await? { CellSqlOptions::Materialized(o) => CellGroupApplyImages::Materialized { @@ -107,7 +114,14 @@ async fn settings( Ok(CellGroupProvisioning { directory: f.cluster._root.path().join(format!("provision-{node}")), filesystem: Arc::new(DirectFileSystem), - images, + namespaces: directories + .iter() + .cloned() + .map(|directory| CellGroupProvisionNamespace { + directory, + images: images.clone(), + }) + .collect(), manifest: ManifestLimits { max_cells: 32, max_depth: 16, @@ -127,13 +141,14 @@ async fn settings( } fn pages_path(base: &std::path::Path, h: GroupHistory) -> std::path::PathBuf { - base.join(format!( - "{}-{}-{}-{}", - hex::encode(h.cluster.bytes()), - hex::encode(h.namespace.bytes()), - hex::encode(h.group.bytes()), - hex::encode(h.incarnation.bytes()) - )) + // Independent expectation for the canonical startup layout. The restart + // drill below must reopen these selected pages, not a second empty store. + base.join("groups") + .join(hex::encode(h.cluster.bytes())) + .join(hex::encode(h.namespace.bytes())) + .join(hex::encode(h.group.bytes())) + .join(hex::encode(h.incarnation.bytes())) + .join("pages") } pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { @@ -146,9 +161,9 @@ pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { let (directory, route, store) = assignment(&f, 31).await?; let (foreign, _, _) = assignment(&f, 31).await?; let (other, other_route, _) = assignment(&f, 41).await?; - let mut invalid = settings(&f, &a, 0).await?; + let mut invalid = settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?; let directory_before = invalid.directory.clone(); - match &mut invalid.images { + match &mut invalid.namespaces[0].images { CellGroupApplyImages::Extents { cache_directory, .. } => *cache_directory = "relative-cache".into(), @@ -164,12 +179,44 @@ pub(crate) async fn provisioning_coalesces_and_serves() -> Result<()> { !directory_before.exists(), "invalid configuration performs no filesystem work" ); + let duplicate = settings(&f, &a, 0, &[directory.clone(), directory.clone()]).await?; + assert!(matches!( + controllers[0].configure_provisioning(duplicate), + Err(CellGroupControllerError::Invalid(_)) + )); + let mut too_small = settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?; + too_small.max_preparation_bytes = 1; + assert!(matches!( + controllers[0].configure_provisioning(too_small), + Err(CellGroupControllerError::Invalid(_)) + )); + assert!( + !directory_before.exists(), + "invalid namespace profiles perform no filesystem work" + ); for (node, c) in controllers.iter().enumerate() { - c.configure_provisioning(settings(&f, &a, node).await?)?; + c.configure_provisioning( + settings(&f, &a, node, &[directory.clone(), other.clone()]).await?, + )?; } assert!(controllers[0] - .configure_provisioning(settings(&f, &a, 0).await?) + .configure_provisioning(settings(&f, &a, 0, &[directory.clone(), other.clone()]).await?) .is_err()); + let error = controllers[0] + .provision(foreign.clone(), route.clone())? + .response() + .await + .unwrap_err(); + assert!(matches!( + error.as_ref(), + CellGroupControllerError::Invalid("unconfigured namespace owner") + )); + assert!(matches!( + f.cluster.voters[0].log.cell_group(&route.history), + Err(LogError::NotFound) + )); + assert_eq!(controllers[0].usage().preparation_bytes, 0); + assert!(controllers[0].usage().provision_configuration_bytes > 0); store.hold_group_head_reads.store(true, Ordering::Release); let first = controllers[0].provision(directory.clone(), route.clone())?; wait_for(async || Ok(store.active_group_head_reads.load(Ordering::Acquire) == 1)).await?; @@ -283,7 +330,7 @@ pub(crate) async fn provisioning_retries_retained_owner() -> Result<()> { let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; let (directory, route, store) = assignment(&f, 31).await?; - let mut options = settings(&f, &a, 0).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; let path = pages_path(&options.directory, route.history); let faults = physical::Faults::new(path.clone(), None); options.filesystem = Arc::new(physical::FaultFileSystem { @@ -347,7 +394,7 @@ pub(crate) async fn provisioning_joins_cancelled_close() -> Result<()> { let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; let (directory, route, _) = assignment(&f, 31).await?; - let mut options = settings(&f, &a, 0).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; let faults = physical::Faults::new(pages_path(&options.directory, route.history), None); let entered = Arc::new(AtomicBool::new(false)); let reached = entered.clone(); @@ -408,7 +455,7 @@ pub(crate) async fn provisioning_recovers_selected_roots() -> Result<()> { let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; let (directory, route, _) = assignment(&f, 31).await?; for (n, c) in controllers.iter().enumerate() { - c.configure_provisioning(settings(&f, &a, n).await?)?; + c.configure_provisioning(settings(&f, &a, n, &[directory.clone()]).await?)?; c.provision(directory.clone(), route.clone())? .response() .await?; @@ -528,7 +575,7 @@ async fn reopen_and_verify( ) .await?; assert_eq!(c.usage().groups, 0); - let options = settings(f, cell, n).await?; + let options = settings(f, cell, n, &[directory.clone()]).await?; let marker = f.cluster.voters[n] .log .cell_group(&route.history)? @@ -612,7 +659,7 @@ pub(crate) async fn provisioning_refusals_preserve_healthy_group() -> Result<()> let (mut f, _, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut f, controller::limits()).await?; let (directory, route, _) = assignment(&f, 31).await?; - let mut options = settings(&f, &a, 0).await?; + let mut options = settings(&f, &a, 0, &[directory.clone()]).await?; let incorrect_endpoint = std::net::TcpListener::bind("127.0.0.1:0")?; if missing_member { options.peers.pop(); diff --git a/crates/cellar/tests/support/cell_group_query_origins.rs b/crates/cellar/tests/support/cell_group_query_origins.rs index baa2ddd3b..0b0a1c07c 100644 --- a/crates/cellar/tests/support/cell_group_query_origins.rs +++ b/crates/cellar/tests/support/cell_group_query_origins.rs @@ -117,7 +117,7 @@ pub(crate) async fn nonmember_queries() -> Result<()> { )?); let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let owners = gateways(&fixture, class.clone()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; let router = Router::start(&fixture).await?; let discovery = fixture.tcp_route(route.history)?; router @@ -211,6 +211,25 @@ pub(crate) async fn nonmember_queries() -> Result<()> { .is_err()); drop(denied); assert_eq!(calls.load(Ordering::SeqCst), 0); + let mut denied_strong = invocation( + &dispatch, + &class, + &a, + CellQueryConsistency::Linearizable, + Duration::from_secs(5), + b"", + ) + .await?; + assert!(denied_strong + .execute_remote(&router.transport, MemberId::new(2)?) + .await + .is_err()); + drop(denied_strong); + assert_eq!( + calls.load(Ordering::SeqCst), + 0, + "strong reads also require the exact namespace/process grant" + ); router.grant(&fixture, 2)?; let mut query = invocation( &dispatch, @@ -222,15 +241,19 @@ pub(crate) async fn nonmember_queries() -> Result<()> { ) .await?; let first_attempt = query.attempt().context("first strong attempt")?.clone(); - assert!(matches!( - query - .execute_remote(&router.transport, MemberId::new(2)?) - .await, - Err(CellDispatchError::Transport(TransportError::Query( - CommandFailure::VersionUnavailable - ))) - )); - assert_eq!(calls.load(Ordering::SeqCst), 0); + // A granted follower obtains its own correlated quorum read. This + // router still has no local replica and cannot supply that authority. + let follower = query + .execute_remote(&router.transport, MemberId::new(2)?) + .await?; + assert_eq!(follower.result(), a.cell.as_str().as_bytes()); + assert_eq!(follower.observation().version.history, a); + assert_eq!(follower.served_by(), MemberId::new(2)?); + let follower_observation = follower.observation().clone(); + assert_eq!(router.transport.usage().query_items, 1); + drop(follower); + assert_eq!(router.transport.usage().query_items, 0); + assert_eq!(calls.load(Ordering::SeqCst), 1); query.retry().await?; assert_eq!( query.attempt().context("leader retry")?.deadline_ms(), @@ -242,6 +265,9 @@ pub(crate) async fn nonmember_queries() -> Result<()> { .await?; assert_eq!(reply.result(), a.cell.as_str().as_bytes()); assert_eq!(reply.observation().version.history, a); + assert_eq!(reply.observation(), &follower_observation); + assert_eq!(reply.served_by(), MemberId::new(1)?); + assert_eq!(calls.load(Ordering::SeqCst), 2); let token = reply.token().clone(); assert_eq!(router.transport.usage().query_items, 1); drop(reply); @@ -267,7 +293,7 @@ pub(crate) async fn nonmember_queries() -> Result<()> { cell.cell.as_str().as_bytes() ); } - assert_eq!(calls.load(Ordering::SeqCst), 3); + assert_eq!(calls.load(Ordering::SeqCst), 4); // All discovery routes share the same three physical peer pools and // bounded route admission. They never allocate local group storage. let mut second = discovery.clone(); @@ -359,7 +385,7 @@ pub(crate) async fn nonmember_query_revocation(replace_route: bool) -> Result<() )?); let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let owners = gateways(&fixture, class.clone()).await?; + let owners = gateways(&fixture, &controllers, class.clone()).await?; let router = Router::start(&fixture).await?; let mut discovery = fixture.tcp_route(route.history)?; router diff --git a/crates/cellar/tests/support/cell_group_receipt_lookup.rs b/crates/cellar/tests/support/cell_group_receipt_lookup.rs index a21761f41..0848c4f5c 100644 --- a/crates/cellar/tests/support/cell_group_receipt_lookup.rs +++ b/crates/cellar/tests/support/cell_group_receipt_lookup.rs @@ -91,7 +91,9 @@ pub(crate) async fn receipt_lookup_ingress() -> Result<()> { let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - fixture.cell_options(&a).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, )?; let router = Router::start(&fixture).await?; router @@ -402,7 +404,9 @@ pub(crate) async fn receipt_lookup_cancelled() -> Result<()> { let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - fixture.cell_options(&a).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, )?; let router = Router::start(&fixture).await?; router diff --git a/crates/cellar/tests/support/cell_group_remote_commands.rs b/crates/cellar/tests/support/cell_group_remote_commands.rs index d59e6de45..e77bdbf97 100644 --- a/crates/cellar/tests/support/cell_group_remote_commands.rs +++ b/crates/cellar/tests/support/cell_group_remote_commands.rs @@ -73,14 +73,16 @@ pub(crate) async fn remote_commands() -> Result<()> { fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[1].clone(), - options.clone() + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, ), Err(TransportError::Identity) )); let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - options, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new(options)?, )?; let router = Router::start(&fixture).await?; router @@ -151,7 +153,9 @@ pub(crate) async fn remote_commands() -> Result<()> { let follower_gateway = fixture.transports[1].install_cell_command_gateway( dispatch.clone(), controllers[1].clone(), - fixture.cell_options_at(&a, 1).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&a, 1).await?, + )?, )?; let mut follower = invocation(&dispatch, &class, &a, retry_until, b"once").await?; let refusal = follower @@ -253,7 +257,9 @@ pub(crate) async fn remote_command_lifecycle(revoke: bool) -> Result<()> { let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - fixture.cell_options(&a).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&a).await?, + )?, )?; let router = Router::start(&fixture).await?; router diff --git a/crates/cellar/tests/support/cell_group_remote_enrollment.rs b/crates/cellar/tests/support/cell_group_remote_enrollment.rs index a90ea3693..7b3b6c742 100644 --- a/crates/cellar/tests/support/cell_group_remote_enrollment.rs +++ b/crates/cellar/tests/support/cell_group_remote_enrollment.rs @@ -68,7 +68,9 @@ pub(crate) async fn remote_enrollment() -> Result<()> { let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - fixture.cell_options(&cell).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&cell).await?, + )?, )?; let router = Router::start(&fixture).await?; router @@ -98,7 +100,9 @@ pub(crate) async fn remote_enrollment() -> Result<()> { let follower_gateway = fixture.transports[1].install_cell_command_gateway( dispatch.clone(), controllers[1].clone(), - fixture.cell_options_at(&cell, 1).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&cell, 1).await?, + )?, )?; let mut follower = invocation(&dispatch, &cell, 215, Duration::from_secs(15)).await?; let error = follower @@ -264,7 +268,9 @@ pub(crate) async fn remote_enrollment_lifecycle(mode: u8) -> Result<()> { let gateway = fixture.transports[0].install_cell_command_gateway( dispatch.clone(), controllers[0].clone(), - fixture.cell_options(&cell).await?, + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options(&cell).await?, + )?, )?; let router = Router::start(&fixture).await?; router diff --git a/crates/cellar/tests/support/cell_group_remote_queries.rs b/crates/cellar/tests/support/cell_group_remote_queries.rs index 7c6b7a6a3..8aac775e4 100644 --- a/crates/cellar/tests/support/cell_group_remote_queries.rs +++ b/crates/cellar/tests/support/cell_group_remote_queries.rs @@ -63,16 +63,18 @@ pub(super) async fn invocation( pub(super) async fn gateways( fixture: &RuntimeCluster, + controllers: &[CellGroupController], class: Arc, ) -> Result> { let mut gateways = Vec::new(); - let group = fixture.cluster.birth.route().history; let cell = history(fixture.cluster.birth.route(), "query-options")?; for n in 0..3 { gateways.push(fixture.transports[n].install_cell_query_gateway( dispatcher(fixture, class.clone(), keys(190)?)?, - fixture.appliers[&(n, group)].0.clone(), - fixture.cell_options_at(&cell, n).await?, + controllers[n].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + fixture.cell_options_at(&cell, n).await?, + )?, )?); } Ok(gateways) @@ -110,23 +112,27 @@ pub(crate) async fn remote_queries() -> Result<()> { )?); let (mut fixture, route, a, b) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let mut gateways = gateways(&fixture, class.clone()).await?; + let mut gateways = gateways(&fixture, &controllers, class.clone()).await?; let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; assert_eq!(gateways[0].namespace(), (a.cluster, a.namespace)); let options = fixture.cell_options(&a).await?; assert!(matches!( fixture.transports[0].install_cell_query_gateway( dispatch.clone(), - fixture.appliers[&(0, route.history)].0.clone(), - options.clone() + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, ), Err(TransportError::Refused) )); assert!(matches!( fixture.transports[0].install_cell_query_gateway( dispatch.clone(), - fixture.appliers[&(1, route.history)].0.clone(), - options.clone() + controllers[1].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new( + options.clone() + )?, ), Err(TransportError::Identity) )); @@ -290,8 +296,8 @@ pub(crate) async fn remote_queries() -> Result<()> { assert_eq!(calls.load(Ordering::SeqCst), before); let replacement = fixture.transports[0].install_cell_query_gateway( dispatch.clone(), - fixture.appliers[&(0, route.history)].0.clone(), - options, + controllers[0].clone(), + cellar::application::consensus::verification::cell_groups::CellGroupNamespaceStorage::new(options)?, )?; absent.retry().await?; assert_eq!( @@ -405,7 +411,7 @@ pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { )?); let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; - let mut gateways = gateways(&fixture, class.clone()).await?; + let mut gateways = gateways(&fixture, &controllers, class.clone()).await?; let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; let timeout = if expire { Duration::from_millis(600) @@ -497,3 +503,83 @@ pub(crate) async fn remote_query_lifecycle(expire: bool) -> Result<()> { } Ok(()) } + +/// The selected follower, not the ingress node, executes all three read modes +/// over the actual authenticated query and consensus TLS lanes. +pub(crate) async fn remote_follower_reads(rooted: bool) -> Result<()> { + let calls = Arc::new(AtomicUsize::new(0)); + let count = calls.clone(); + let class = Arc::new(RaftClass::new("Counter", 1, [81; 32])?.query( + "Get", + [82; 32], + move |context, input| { + count.fetch_add(1, Ordering::SeqCst); + let QueryVersion::Cell(observation) = context.version() else { + anyhow::bail!("physical group observation"); + }; + assert_eq!(context.identity(), &observation.version.history); + let value: i64 = context.query_row("SELECT n FROM value", [], |row| row.get(0))?; + let mut output = value.to_le_bytes().to_vec(); + output.extend_from_slice(input); + Ok(output) + }, + )?); + let (mut fixture, route, a, _) = Box::pin(registered::setup(rooted, &class)).await?; + let controllers = controller::start_with_limits(&mut fixture, controller::limits()).await?; + let gateways = gateways(&fixture, &controllers, class.clone()).await?; + let dispatch = dispatcher(&fixture, class.clone(), keys(190)?)?; + assert_eq!( + fixture.runtimes[2].status(route.history).await?.leader, + Some(1) + ); + assert!(!fixture.runtimes[2].status(route.history).await?.ready); + let mut minimum: Option = None; + let mut observation = None; + for index in 0..3_u8 { + let consistency = match index { + 0 => CellQueryConsistency::Linearizable, + 1 => CellQueryConsistency::RecentSource { max_age_ms: 5000 }, + _ => CellQueryConsistency::AtLeast(minimum.clone().context("first token")?), + }; + let mut query = dispatch.query( + a.clone(), + CellQueryRequest { + invocation: ProposalId::new([210 + index; 16])?, + contract: class.contract("Get").context("Get")?, + input: vec![index], + consistency, + }, + CellRoutingLimits { + timeout: Duration::from_secs(5), + max_attempts: 1, + }, + None, + )?; + query.begin().await?; + let response = query + .execute_remote(&fixture.transports[1], MemberId::new(3)?) + .await?; + let mut expected = 0_i64.to_le_bytes().to_vec(); + expected.push(index); + assert_eq!(response.result(), expected); + assert_eq!(response.served_by(), MemberId::new(3)?); + assert_eq!(response.observation().version.history, a); + assert_eq!(response.observation().version.sequence.get(), 1); + if let Some(expected) = &observation { + assert_eq!(response.observation(), expected); + } + observation = Some(response.observation().clone()); + minimum = Some(response.token().clone()); + assert_eq!(fixture.transports[1].usage().query_items, 1); + drop(response); + assert_eq!(fixture.transports[1].usage().query_items, 0); + drop(query); + assert_eq!(dispatch.usage(), (0, 0)); + } + assert_eq!(calls.load(Ordering::SeqCst), 3); + for controller in controllers { + controller.close().await?; + } + drop(gateways); + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_runtime.rs b/crates/cellar/tests/support/cell_group_runtime.rs index 406a92f44..0b01dadcd 100644 --- a/crates/cellar/tests/support/cell_group_runtime.rs +++ b/crates/cellar/tests/support/cell_group_runtime.rs @@ -856,7 +856,7 @@ pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { sql::submission_rejections(rooted).await } -pub(super) use sql::{coherent_reads, query_lifecycle}; +pub(super) use sql::{coherent_reads, follower_linearizable, query_lifecycle, recent_source_reads}; pub(super) use sql::apply_images; @@ -880,7 +880,7 @@ pub(super) use sql::{ enrollment_requires_registration, owned_enrollment, owned_enrollment_failure, }; -pub(super) use sql::{remote_queries, remote_query_lifecycle}; +pub(super) use sql::{remote_follower_reads, remote_queries, remote_query_lifecycle}; pub(super) use sql::{nonmember_queries, nonmember_query_revocation}; @@ -913,7 +913,9 @@ pub(super) use sql::{ automatic_checkpoints, checkpoint_during_apply, controller_checkpoint_shutdown, }; -pub(super) use sql::{membership_lifecycle, membership_sync}; +pub(super) use sql::{ + membership_lifecycle, membership_quorum_floor, membership_quorum_followers, membership_sync, +}; pub(super) use sql::{seed_minority, seed_source}; diff --git a/crates/cellar/tests/support/cell_group_s3_commands.rs b/crates/cellar/tests/support/cell_group_s3_commands.rs index aca64d51a..91ae792c1 100644 --- a/crates/cellar/tests/support/cell_group_s3_commands.rs +++ b/crates/cellar/tests/support/cell_group_s3_commands.rs @@ -158,14 +158,25 @@ impl Harness { let (mut fixture, route, cell, _) = if let Some(Resources::Retention(retention)) = resources { let filesystem = Arc::new(DirectFileSystem); - let cluster = Box::pin(Cluster::with_store_retention( + let cluster = Box::pin(Cluster::with_store_birth( RaftRuntimeOptions::default(), tempfile::tempdir()?, filesystem.clone(), class.schema_digest(), store, - minimum, - retention, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum, + log: ClusterLogLimits { + retention, + ..Default::default() + }, + }, )) .await?; Box::pin(registered::setup_cluster(rooted, cluster, filesystem)).await? @@ -177,7 +188,11 @@ impl Harness { class.schema_digest(), store, ClusterBirthOptions { - image_storage: RaftImageStorage::Materialized, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, group: GroupId::new([3; 16])?, minimum, log: ClusterLogLimits::default(), diff --git a/crates/cellar/tests/support/cell_group_s3_receipts.rs b/crates/cellar/tests/support/cell_group_s3_receipts.rs index cf70c5c74..ec6b472b7 100644 --- a/crates/cellar/tests/support/cell_group_s3_receipts.rs +++ b/crates/cellar/tests/support/cell_group_s3_receipts.rs @@ -263,7 +263,9 @@ pub(crate) async fn s3_receipt_minimums(rooted: bool, store: Arc Result<()> { &root.join("oracle.json"), &serde_json::to_vec(&Oracle { rooted, + image_storage: fixture.cluster.directory.namespace_config().image_storage, source_root: fixture.cluster._root.path().to_path_buf(), cut: cut.clone(), staged, @@ -276,7 +278,8 @@ async fn recover(root: &Path, rooted: bool, cut: &str) -> Result<()> { let local = seed_install::target(route.history)?; let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); let (directory, birth) = - crash_fixture::restore_catalog(oracle.objects, &route, &domain).await?; + crash_fixture::restore_catalog(oracle.objects, &route, &domain, oracle.image_storage) + .await?; let base = root.join("learner"); let log = domain .run(ReplicaLog::open( diff --git a/crates/cellar/tests/support/cell_group_seed_persistence.rs b/crates/cellar/tests/support/cell_group_seed_persistence.rs index ed1af4f72..1d44f092e 100644 --- a/crates/cellar/tests/support/cell_group_seed_persistence.rs +++ b/crates/cellar/tests/support/cell_group_seed_persistence.rs @@ -13,6 +13,7 @@ enum Cut { StreamingRevocation, QueuedRevocation, EnteredRevocation, + EnteredExpiry, SyncError, AfterSyncError, } @@ -25,6 +26,7 @@ pub(crate) async fn seed_persistence(rooted: bool) -> Result<()> { for cut in [ Cut::QueuedRevocation, Cut::EnteredRevocation, + Cut::EnteredExpiry, Cut::SyncError, Cut::AfterSyncError, ] { @@ -108,12 +110,23 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { captured, seed_receiver::Fault::Valid, ))?; + // The expiry cut must complete inside the existing peer's five-second + // disconnect guard. Other cuts keep their original ten-second budget. + let deadline = reads::deadline( + &fixture, + if cut == Cut::EnteredExpiry { + 2000 + } else { + 10_000 + }, + ) + .await?; let session = Box::pin(transport.prepare_group_seed( local.clone(), peer.clone(), fixture.cluster.directory.clone(), fixture.cluster.birth.clone(), - reads::deadline(&fixture, 10_000).await?, + deadline, )) .await?; let group = session.group().clone(); @@ -144,7 +157,7 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { let release = gate.arm(match cut { Cut::StreamingRevocation => fault::HOLD_DEPENDENCY_PREFIX, - Cut::QueuedRevocation | Cut::EnteredRevocation => fault::HOLD_SYNC, + Cut::QueuedRevocation | Cut::EnteredRevocation | Cut::EnteredExpiry => fault::HOLD_SYNC, Cut::SyncError => fault::FAIL_SYNC, Cut::AfterSyncError => fault::FAIL_AFTER_SYNC, }); @@ -171,7 +184,10 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { let mut early_result = None; if matches!( cut, - Cut::StreamingRevocation | Cut::QueuedRevocation | Cut::EnteredRevocation + Cut::StreamingRevocation + | Cut::QueuedRevocation + | Cut::EnteredRevocation + | Cut::EnteredExpiry ) { if cut == Cut::QueuedRevocation { controller::wait_for(async || Ok(queued(&log)? == 1)).await?; @@ -194,10 +210,20 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { .is_err(), "no receipt before the engine fence" ); - restarted_source = Some( - seed_receiver::revoke_source(&fixture, &mut controllers, &transport, &peer, &route) - .await?, - ); + if cut == Cut::EnteredExpiry { + // Keep the actual fence held through the original operation deadline. + // Its eventual durable result must not grant a late bootstrap reply. + fixture + .cluster + .domain + .run(async { cellar::asyncrt::sleep_until(deadline).await }) + .await?; + } else { + restarted_source = Some( + seed_receiver::revoke_source(&fixture, &mut controllers, &transport, &peer, &route) + .await?, + ); + } if cut == Cut::StreamingRevocation { assert!( gate.dependency_cut @@ -259,6 +285,19 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { assert_eq!(installed.marker(), &expected); assert!(installed.pin_cell(&a.cell).is_some() && installed.pin_cell(&b.cell).is_some()); } + Cut::EnteredExpiry => { + assert!( + matches!( + result, + Err(CellGroupSeedError::Log(LogError::WrongIdentity)) + | Err(CellGroupSeedError::Transport(TransportError::Timeout)) + ), + "completed sync cannot extend the original deadline: {:?}", + result.err() + ); + assert_eq!(group.group_manifest()?.as_ref(), Some(&expected)); + assert_eq!(group.checkpoint_manifest()?.as_ref(), Some(&expected)); + } Cut::SyncError | Cut::AfterSyncError => assert!( matches!(result, Err(CellGroupSeedError::Log(LogError::Io(_)))), "sync must report uncertainty: {:?}", @@ -348,7 +387,9 @@ async fn run(rooted: bool, cut: Cut) -> Result<()> { let selected = group.group_manifest()?.context("reopened root")?; match cut { Cut::StreamingRevocation | Cut::QueuedRevocation => assert_eq!(selected, staged), - Cut::EnteredRevocation | Cut::AfterSyncError => assert_eq!(selected, expected), + Cut::EnteredRevocation | Cut::EnteredExpiry | Cut::AfterSyncError => { + assert_eq!(selected, expected) + } Cut::SyncError => assert!(selected == staged || selected == expected), } let (pages, images) = seed_install::stores(&base, rooted) diff --git a/crates/cellar/tests/support/cell_group_seed_receiver.rs b/crates/cellar/tests/support/cell_group_seed_receiver.rs index 4c6fa75a7..ed232a9bc 100644 --- a/crates/cellar/tests/support/cell_group_seed_receiver.rs +++ b/crates/cellar/tests/support/cell_group_seed_receiver.rs @@ -543,6 +543,14 @@ pub(super) async fn revoke_source( peer: &RaftPeer, route: &GroupRoute, ) -> Result { + let namespace = fixture.cluster.directory.namespace_config(); + anyhow::ensure!( + route.history.cluster == namespace.cluster + && route.history.namespace == namespace.namespace + && route.schema_digest == namespace.schema_digest + && peer.replica.scope == route.history, + "source revocation route is outside the fixture namespace", + ); controllers.remove(0).close().await?; let new_process = ProcessGeneration::new([99; 16])?; let options = ReplicaLogOptions::new( @@ -567,12 +575,7 @@ pub(super) async fn revoke_source( .run(async { let catalog = Catalog::new(fixture.cluster.store.clone(), CatalogOptions::default())?; let namespace = catalog - .open_namespace(NamespaceConfig::new( - route.history.cluster, - route.history.namespace, - cellar_logic::NamespaceMode::Raft, - route.schema_digest, - )) + .existing_namespace(fixture.cluster.directory.namespace_config().clone()) .await?; let registration = namespace .register_node_storage(&restarted, Duration::from_secs(5)) diff --git a/crates/cellar/tests/support/cell_group_seed_retry.rs b/crates/cellar/tests/support/cell_group_seed_retry.rs index d2a738525..1d9b9bec0 100644 --- a/crates/cellar/tests/support/cell_group_seed_retry.rs +++ b/crates/cellar/tests/support/cell_group_seed_retry.rs @@ -149,8 +149,13 @@ async fn retry(root: &Path, rooted: bool, cut: &str) -> Result<()> { } let route = oracle.expected.route().clone(); let domain = ExecutionDomain::new(tokio::runtime::Handle::current()); - let (directory, birth, store) = - crash_fixture::restore_catalog_store(oracle.objects.clone(), &route, &domain).await?; + let (directory, birth, store) = crash_fixture::restore_catalog_store( + oracle.objects.clone(), + &route, + &domain, + oracle.image_storage, + ) + .await?; let mut f = Box::pin(restore_source( root, &oracle, diff --git a/crates/cellar/tests/support/cell_group_sql.rs b/crates/cellar/tests/support/cell_group_sql.rs index b25fb2127..fe9c2246c 100644 --- a/crates/cellar/tests/support/cell_group_sql.rs +++ b/crates/cellar/tests/support/cell_group_sql.rs @@ -496,7 +496,10 @@ pub(super) async fn submission_rejections(rooted: bool) -> Result<()> { #[path = "cell_group_sql_reads.rs"] mod reads; -pub(crate) use reads::{coherent as coherent_reads, lifecycle as query_lifecycle}; +pub(crate) use reads::{ + coherent as coherent_reads, follower_linearizable, lifecycle as query_lifecycle, + recent_source_reads, +}; #[path = "cell_group_sql_apply.rs"] pub(super) mod apply; @@ -589,7 +592,7 @@ pub(crate) use controller_enrollment::{ #[path = "cell_group_remote_queries.rs"] mod remote_queries; -pub(crate) use remote_queries::{remote_queries, remote_query_lifecycle}; +pub(crate) use remote_queries::{remote_follower_reads, remote_queries, remote_query_lifecycle}; #[path = "cell_group_query_origins.rs"] mod query_origins; @@ -650,7 +653,9 @@ pub(crate) use checkpoint_fairness::concurrent_checkpoints; #[path = "cell_group_membership.rs"] mod membership; -pub(crate) use membership::{membership_lifecycle, membership_sync}; +pub(crate) use membership::{ + membership_lifecycle, membership_quorum_floor, membership_quorum_followers, membership_sync, +}; #[path = "cell_group_seed.rs"] mod seed; diff --git a/crates/cellar/tests/support/cell_group_sql_reads.rs b/crates/cellar/tests/support/cell_group_sql_reads.rs index d2fe2f57f..b91e25b4e 100644 --- a/crates/cellar/tests/support/cell_group_sql_reads.rs +++ b/crates/cellar/tests/support/cell_group_sql_reads.rs @@ -391,3 +391,187 @@ pub(crate) async fn lifecycle(rooted: bool) -> Result<()> { assert_eq!(pool.usage()?.running_workers, 0); fixture.close().await } + +/// A follower must obtain its own correlated quorum barrier, apply through it, +/// and retain only that invocation's authority across SQL work. +pub(crate) async fn follower_linearizable(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 1)?; + let options = fixture.cell_options_at(&a, 1).await?; + let authority = selected(&mut fixture, 1, &a, CellReadConsistency::Linearizable).await?; + let barrier = authority.barrier().context("follower correlated barrier")?; + assert!(authority.applied().index >= barrier.index); + assert_eq!(authority.cell().version.history, a); + let output = value(&pool, options.clone(), authority).await?; + assert_eq!( + output.replica(), + fixture.cluster.voters[1] + .log + .cell_group(&route.history)? + .identity() + ); + drop(output); + + // No response from a live quorum: a previous successful read cannot be reused. + assert!(matches!( + fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::Linearizable, + deadline(&fixture, 50).await?, + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + fixture.drive(None).await?; + + let mut forged = b"cellar-cell-forward-read-1\0".to_vec(); + forged.extend_from_slice(&3_u64.to_be_bytes()); + forged.extend_from_slice(&[41; 16]); + forged.extend_from_slice(&1_u64.to_be_bytes()); + let term = fixture.runtimes[0].status(route.history).await?.term.get(); + let message = Message { + msg_type: MessageType::MsgReadIndex, + from: 2, + to: 1, + term, + entries: vec![Entry { + data: forged.into(), + ..Default::default() + }] + .into(), + ..Default::default() + }; + assert!( + fixture.runtimes[0] + .step(route.history, message) + .await + .is_err(), + "forwarded read context must name its authenticated origin" + ); + assert!( + fixture.runtimes[0] + .read_index( + route.history, + b"cellar-cell-forward-read-1\0forged".to_vec(), + ) + .await + .is_err(), + "public contexts cannot enter the private cell-read namespace" + ); + + let retained = selected(&mut fixture, 1, &a, CellReadConsistency::Linearizable).await?; + super::reconciliation::transfer(&mut fixture, 0, 2).await?; + assert!(pool + .query(options, retained, |_, _| panic!( + "revoked follower authority" + )) + .await + .is_err()); + pool.close().await?; + fixture.close().await +} + +/// Source-age bounds follow the physical query pin, including entered callbacks. +pub(crate) async fn recent_source_reads(rooted: bool) -> Result<()> { + let (mut fixture, route, a, _) = enrolled_sql_cells(rooted).await?; + let pool = pool(&fixture, 1)?; + let options = fixture.cell_options_at(&a, 1).await?; + // A fresh actual leader heartbeat, not an advisory catalog refresh. + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + fixture.drive(None).await?; + } + // This read needs no network round. The generic selected() helper pumps + // peer traffic concurrently and may admit another heartbeat after capture. + let authority = fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::RecentSource { max_age_ms: 100 }, + deadline(&fixture, 5000).await?, + ) + .await?; + let expiry = authority.deadline(); + let (entered, started) = tokio::sync::oneshot::channel(); + let (release, blocked) = std::sync::mpsc::sync_channel(1); + let reader = pool.clone(); + let held_options = options.clone(); + let pending = fixture.cluster.domain.spawn(async move { + reader + .query(held_options, authority, move |db, _| { + let _ = entered.send(()); + blocked.recv_timeout(Duration::from_secs(5))?; + let value: i64 = db.query_row("SELECT n FROM value", [], |row| row.get(0))?; + Ok(value.to_le_bytes().to_vec()) + }) + .await + })?; + tokio::time::timeout(Duration::from_secs(5), started).await??; + fixture + .cluster + .domain + .run(async { cellar::asyncrt::sleep_until(expiry).await }) + .await?; + assert!( + pending.await?.is_err(), + "old source observation cannot authorize a late reply" + ); + assert_eq!( + pool.usage()?.retained_operations, + 1, + "expired waiter retains entered physical SQL" + ); + release.send(())?; + tokio::time::timeout(Duration::from_secs(5), async { + while pool.usage()?.retained_operations != 0 { + tokio::task::yield_now().await; + } + Ok::<_, anyhow::Error>(()) + }) + .await??; + // A stale-term heartbeat cannot refresh the current source observation. + let term = fixture.runtimes[1].status(route.history).await?.term.get(); + assert!(term > 1); + fixture.runtimes[1] + .step( + route.history, + Message { + msg_type: MessageType::MsgHeartbeat, + from: 1, + to: 2, + term: term - 1, + ..Default::default() + }, + ) + .await?; + assert!(matches!( + fixture.runtimes[1] + .read_authority( + route.history, + a.clone(), + CellReadConsistency::RecentSource { max_age_ms: 100 }, + deadline(&fixture, 30).await? + ) + .await, + Err(RuntimeError::DeadlineExceeded) + )); + // Historical reads are a separate explicit policy, never the fallback above. + let authority = selected(&mut fixture, 1, &a, CellReadConsistency::Stale).await?; + drop(value(&pool, options.clone(), authority).await?); + for _ in 0..2 { + fixture.runtimes[0].tick().await?; + fixture.drive(None).await?; + } + let authority = selected( + &mut fixture, + 1, + &a, + CellReadConsistency::RecentSource { max_age_ms: 1000 }, + ) + .await?; + drop(value(&pool, options, authority).await?); + pool.close().await?; + fixture.close().await +} diff --git a/crates/cellar/tests/support/cell_group_sql_registered.rs b/crates/cellar/tests/support/cell_group_sql_registered.rs index b1493ce95..fbafd7f40 100644 --- a/crates/cellar/tests/support/cell_group_sql_registered.rs +++ b/crates/cellar/tests/support/cell_group_sql_registered.rs @@ -170,11 +170,22 @@ pub(super) async fn setup_in( root: tempfile::TempDir, filesystem: Arc, ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { - let cluster = Cluster::with_storage_schema( + let cluster = Cluster::with_store_birth( RaftRuntimeOptions::default(), root, filesystem.clone(), class.schema_digest(), + Arc::new(object_store::memory::InMemory::new()), + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum: Durability::Quorum, + log: ClusterLogLimits::default(), + }, ) .await?; setup_cluster(rooted, cluster, filesystem).await @@ -195,13 +206,22 @@ pub(super) async fn setup_store_minimum( minimum: Durability, ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { let filesystem = Arc::new(DirectFileSystem); - let cluster = Cluster::with_store_minimum( + let cluster = Cluster::with_store_birth( RaftRuntimeOptions::default(), tempfile::tempdir()?, filesystem.clone(), class.schema_digest(), store, - minimum, + ClusterBirthOptions { + group: GroupId::new([3; 16])?, + image_storage: if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + minimum, + log: ClusterLogLimits::default(), + }, ) .await?; setup_cluster(rooted, cluster, filesystem).await @@ -227,6 +247,15 @@ pub(super) async fn setup_cluster_named( filesystem: Arc, names: [&str; 2], ) -> Result<(RuntimeCluster, GroupRoute, CellHistory, CellHistory)> { + assert_eq!( + cluster.directory.namespace_config().image_storage, + if rooted { + RaftImageStorage::Extents + } else { + RaftImageStorage::Materialized + }, + "fixture backend must match its immutable namespace capability", + ); let mut fixture = Box::pin(RuntimeCluster::from_cluster(cluster, 256, 256, 1_000_000)).await?; let route = fixture.cluster.birth.route().clone(); if rooted { diff --git a/crates/cellar/tests/support/raft_docker_binding.rs b/crates/cellar/tests/support/raft_docker_binding.rs index 67f9f7439..9b00f78f4 100644 --- a/crates/cellar/tests/support/raft_docker_binding.rs +++ b/crates/cellar/tests/support/raft_docker_binding.rs @@ -4,6 +4,7 @@ //! provisioning boundary. This does not claim container shutdown evidence. use super::*; use anyhow::Context; +use cellar::application::consensus::verification::cell_groups::RangeDirectory; use cellar::docker::{ContainerId, Docker}; use object_store::ObjectStore; use serde_json::json; @@ -36,7 +37,10 @@ async fn binding(local_observer: bool) -> Result<()> { catalog.clone(), Docker::new(&socket), "native-binding", - vec![capacity_tests::namespace(&f).await?], + vec![RangeDirectory::new( + capacity_tests::namespace(&f).await?, + Default::default(), + )?], )?; let node = NodeId::new(boot.node)?; let container = ContainerId::new(id)?; @@ -107,7 +111,10 @@ async fn binding(local_observer: bool) -> Result<()> { Fixture::open_catalog(&f.domain, f.store.clone())?, Docker::new(&socket), "native-binding", - vec![capacity_tests::namespace(&f).await?], + vec![RangeDirectory::new( + capacity_tests::namespace(&f).await?, + Default::default(), + )?], )?; reopened .bind_node(node, &container, TIMEOUT) diff --git a/docs/catalog.json b/docs/catalog.json index 587a67d82..d3ad798d9 100644 --- a/docs/catalog.json +++ b/docs/catalog.json @@ -637,6 +637,102 @@ ], "category": "Design and evidence" }, + { + "path": "design/multi-raft/cell-groups/README.md", + "title": "Cell group delivery package", + "summary": "Shared cell-group delivery status, execution commands and completion boundaries.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/design.md", + "title": "Cell group design", + "summary": "Physical group consensus over independent SQLite cells, identities and hard cutover.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/handoff.md", + "title": "Cell-group implementation handoff", + "summary": "Current implementation checkpoint, evidence transfer and next executor steps.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/implementation-plan.md", + "title": "Implementation order", + "summary": "Dependency order and status of the sixteen cell-group implementation tasks.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/progress.md", + "title": "Cell-group implementation progress", + "summary": "Source-pinned cell-group implementation checkpoints and qualification limits.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/protocols.md", + "title": "Durable protocols and crash cuts", + "summary": "Durable enrollment, apply, topology, hibernation and disaster-recovery protocols.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/qualification.md", + "title": "Qualification and evidence contract", + "summary": "Failure oracles, workload profiles and evidence required for release.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, + { + "path": "design/multi-raft/cell-groups/remaining-work.md", + "title": "Remaining cell-group work and priorities", + "summary": "Prioritize the remaining cell-group MVP, merge gates and full architecture deliverables.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, { "path": "design/multi-raft/control-window-performance.md", "title": "Multi-Raft control-window performance", @@ -649,6 +745,18 @@ ], "category": "Design and evidence" }, + { + "path": "design/multi-raft/density-decision.md", + "title": "Shared cell groups: architecture decision", + "summary": "TiKV investigation and the decision to share physical Raft groups across whole cells.", + "roles": [ + "builder", + "platform", + "operations", + "team" + ], + "category": "Design and evidence" + }, { "path": "design/multi-raft/disaster-recovery.md", "title": "Zero-quorum disaster recovery", @@ -1224,85 +1332,6 @@ "team" ], "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/README.md", - "title": "Cell group delivery package", - "summary": "Shared cell-group delivery status, execution commands and completion boundaries.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/design.md", - "title": "Cell group design", - "summary": "Physical group consensus over independent SQLite cells, identities and hard cutover.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/handoff.md", - "title": "Cell-group implementation handoff", - "summary": "Current implementation checkpoint, evidence transfer and next executor steps.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/implementation-plan.md", - "title": "Implementation order", - "summary": "Dependency order and status of the sixteen cell-group implementation tasks.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/progress.md", - "title": "Cell-group implementation progress", - "summary": "Source-pinned cell-group implementation checkpoints and qualification limits.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/protocols.md", - "title": "Durable protocols and crash cuts", - "summary": "Durable enrollment, apply, topology, hibernation and disaster-recovery protocols.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/cell-groups/qualification.md", - "title": "Qualification and evidence contract", - "summary": "Failure oracles, workload profiles and evidence required for release.", - "roles": [ - "platform", - "operations" - ], - "category": "Design and evidence" - }, - { - "path": "design/multi-raft/density-decision.md", - "title": "Shared cell groups: architecture decision", - "summary": "TiKV investigation and the decision to share physical Raft groups across whole cells.", - "roles": [ - "platform" - ], - "category": "Design and evidence" } ], "visuals": [ diff --git a/docs/design/multi-raft/application-contract.md b/docs/design/multi-raft/application-contract.md index 6ffd75713..0d73d7866 100644 --- a/docs/design/multi-raft/application-contract.md +++ b/docs/design/multi-raft/application-contract.md @@ -169,29 +169,31 @@ Keep separate command and query contract digests. A follower can hold the correc Perform current authentication and authorization at ingress and destination. Authorization derived from a stale database image is itself stale; operations requiring latest revocation checks must use a fresh authoritative check. Version tokens prove requested history, not access permission. -- The current SDK exposes a diagnostic `CommittedVersion` in `version` and an opaque authenticated observation in `version_token`. -- `AtLeast` accepts the opaque token and validates it on the destination before any index comparison. -- Tokens bind cluster, namespace, cell, group, incarnation, application digest, index, term, issuance and expiry. -- They survive ordinary leadership changes; cross-history comparisons fail. -- The returned position names the cut captured with the pinned image, including verified protocol-only coverage of that image. - -- `QueryOptions::routing` defaults to local execution. -- `QueryRouting::Replicas` accepts up to eight distinct member IDs in priority order, a per-attempt timeout between one millisecond and sixty seconds, and an explicit leader-fallback flag. -- The total query deadline is fixed on initial admission. -- Each attempt preserves its contract/input, consistency/token and stale policy. -- A fallback uses an untried leader hint and obtains the same read proof at its destination. -- `QueryResult::served_by` reports the responding member. -- A forwarding origin need not have compatible query code; the destination must check it before execution. - -- Each `RaftCellRegistration` explicitly selects `RaftCellPlacement::Replica` or `QueryRouter { leader_hint }`. -- A routing-only binding requires an existing namespace/birth and cannot create a member or SQLite image. -- It rejects command, local-query and local-status calls; choose explicit replica routing. -- Its optional leader hint is discovery metadata and never supplies a read barrier. -- `RaftCellsConfig::query_origins` installs trusted namespace read grants for exact cluster/namespace/node/disk/process endpoints. -- These grants do not authorize Raft messages or change membership. -- Live revocation cancels old query waits; revision tombstones prevent stale updates/regrants from reviving those requests. -- The embedding supplies trusted grants on restart; MR-11 distributes current process views and implements bounded dynamic placement. -- See the [read audit](read-audit.md) for scope and [counter HTTP contract](https://github.com/crabbuild/cellar/blob/8f18c29a0f28977eb8a4cae51848982ad7d43739/examples/raft-counter/README.md) for the native/Axum consistency, token and bounded-routing headers. +The canonical shared SDK query result carries CellVersion, CommitWitness, +CellVersionToken and the serving NodeIdentity. The logical version remains +separate from physical group term/index. Tokens authenticate the complete logical +history, schema, version, witness, issuance and expiry. They prove no access +permission. AtLeast validates the token before comparing the selected cell's +version; another history or schema refuses. Linearizable needs a fresh safe +ReadIndex and the corresponding applied root. Stale supplies neither a quorum +barrier nor a wall-clock age promise. + +QueryOptions supplies consistency, timeout and a bounded max_attempts value. The +invocation's original deadline is fixed before returning its waiter; retries keep +the same contract, captured input and requested token. QueryRouting::Replicas and +the per-cell QueryRouter registration path are retired. Group discovery supplies +no member, image, read barrier or write authority. Complete public SDK, ingress +and managed lifecycle qualification remains tracked by the +[cell-group implementation record](cell-groups/progress.md). + +RaftCellsConfig::query_origins installs trusted namespace read grants for exact +cluster/namespace/node/disk/process endpoints. Every authenticated cell route +binds a physical group and a separate logical cell. A grant does not authorize +Raft messages, commands, enrollment or receipt mutations. Revocation cancels old +query waits; revision tombstones prevent stale updates or regrants from reviving +those requests. The embedding supplies trusted grants on restart. The +[read audit](read-audit.md) retains historical qualification scope; its old SDK +fixtures must be converted before they can qualify the shared runtime. - `RaftCellsServices::versions` supplies explicit managed Ed25519 signing and verification keys. - Configured token lifetime is between one millisecond and seven days. diff --git a/docs/design/multi-raft/cell-groups/README.md b/docs/design/multi-raft/cell-groups/README.md index 4465f92c4..0542e0a5b 100644 --- a/docs/design/multi-raft/cell-groups/README.md +++ b/docs/design/multi-raft/cell-groups/README.md @@ -2,24 +2,20 @@ # Cell group delivery package -Status: executable design and implementation plan delivered on 2026-10-29. -CG-01 through CG-06 have passed their current scoped native evidence gates at -e32115aa. The unchanged gate refresh passes 123 commands: 110 exact native cases, -nineteen pure-logic tests and the shared repository checks. Fresh retained records -replace the unavailable historical raw artifacts. CG-06's complete archive, -local reclamation and fresh-history recovery boundary also passes both actual -Docker/RustFS all-voter-disks-deleted drills with permanent launch revocation, -fresh-quorum activation, exact receipts, new writes and retained restart. -CG-08 due effects and safe cache lifetimes now pass the complete native gate and -same-source/executable RustFS case. CG-07 SDK/fleet cutover has started in the spare checkout and remains unqualified. See the latest [progress](progress.md) and [handoff](handoff.md). -CG-07 synchronous shared-command admission now passes its focused native and lint -checkpoint, preserving the original deadline and node-owned producer lifetime. -The namespace-host component also passes its ten-command native/lint checkpoint, -including eight cells in one group and physical query shutdown in both image forms. -The production SDK/fleet has not yet cut over; topology, hibernation, managed S3 -recovery and deployment/release qualification remain tracked in later tasks. -See [implementation progress](progress.md) and the generated task index. -Package validation alone does not qualify implementation or a release. +Purpose: Shared cell-group delivery status, execution commands and completion boundaries. + +Status: executable design and implementation plan delivered on 2026-10-29; +end-to-end implementation remains in progress. CG-01–06 and CG-08 have passed +recorded component gates. CG-07 canonical SDK/fleet cutover is IN_PROGRESS; +CG-09–16 remain required. PR #160 is a draft checkpoint until its public/build +and current-source backend gates pass. + +Start with [remaining work and priorities](remaining-work.md) for the MVP order +and merge criteria. [Progress](progress.md) records source-scoped validation; +[handoff](handoff.md) identifies the current implementation and continuation. +Historical Docker/RustFS results cover their recorded local component boundaries, +including all-voter-disk loss and effects. They do not qualify the latest combined +SDK/fleet runtime, independent hosts, sustained load or release readiness. This delivery is the executable design and implementation plan. End-to-end implementation and hard cutover remain tracked as separate task and release gates. See the [executor handoff](handoff.md) for the current diff --git a/docs/design/multi-raft/cell-groups/design.md b/docs/design/multi-raft/cell-groups/design.md index c2203ae95..26b220808 100644 --- a/docs/design/multi-raft/cell-groups/design.md +++ b/docs/design/multi-raft/cell-groups/design.md @@ -2,6 +2,8 @@ # Cell group design +Purpose: Physical group consensus over independent SQLite cells, identities and hard cutover. + Status: target contract for CG-01 through CG-16; implementation in progress. See [implementation progress](progress.md); no runtime cutover is qualified yet. Cellar baseline: `1be27d234d4674d4d014785c60cecbfe394bdab9`. diff --git a/docs/design/multi-raft/cell-groups/handoff.md b/docs/design/multi-raft/cell-groups/handoff.md index cf7bbf4ec..405b95c8e 100644 --- a/docs/design/multi-raft/cell-groups/handoff.md +++ b/docs/design/multi-raft/cell-groups/handoff.md @@ -2,6 +2,87 @@ # Cell-group implementation handoff +Purpose: Current implementation checkpoint, evidence transfer and next executor steps. + +The user explicitly requests updating PR #160 and scoping remaining work. +Publish this as a draft checkpoint; do not merge. The previous publication hold +is superseded by that request. Preserve/exclude the user's eight diagram files +and both MR progress basenames. Keep the PR checkout and isolated effects +checkout aligned after publication. All CG-01–16 requirements remain intact. + +Start with [remaining work and priorities](remaining-work.md). The MVP order is +public query routing and consumer ports; managed directory/startup and whole-group +lifecycle; exact-reference S3 learner activation/archive holds; then all original +public, build and current-source RustFS/DR/load gates. The five public cases and +CG-09–16 follow-ups are explicitly listed there. No TiKV performance parity or +combined system/MVP completion is claimed. + +Latest source adds safe follower ReadIndex, source-generation fencing, captured +recent-source expiry through physical SQL completion, SDK source-age defaults and +actual selecting-replica metadata. New TLS cases retain namespace/process grants, +logical observations, responder identity and physical permits. Fifteen focused +cases pass on router-fixed-05; the source-age fixture also passes forty repetitions. +The complete native execution suite passes: 173 passed, zero failed and two +ignored, in 408.03 seconds on the normal default stack. +Core-fixed-01 has identical production/example Rust: unit 1048 passed, one failed, +two ignored (old managed inspection protocol); two retained effects smoke runs +pass using the SQLite development object store. Strict Clippy fails with 234 +diagnostics. The overall native Cargo command still fails on 34 old performance +API errors. These component results do not qualify the public package or current +RustFS/managed/recovery/load orchestration. + +Read shared_read_public_work in the ignored continuation for exact source hashes, +immutable executables, retained failing baselines, backup and owned handles. +Finish every owned job before staging Rust. An observation timeout is not failure; +never restart a live build/test merely because its output has not arrived. +Historical sections below describe scoped checkpoints; this newest section governs. + +Latest unpublished WIP fixes retained-preparation reuse and the completion phase +of an already synced seed. Boundary probes reproduced the previous three native +failures: repeated sync of the same retained page root, and a source-process +watch rejecting completion after persistence succeeds. The probes are removed. +The original LOCK, no-extra-sync, queued-revocation, uncertainty, recovery and +fresh-quorum assertions remain. An added entered-fence expiry case preserves the +original deadline and checks cold recovery of the actual completed write. + +CG-07-seed-retained-fixed-02 passes all three formerly failing native cases, +including both image forms and the added entered-fence expiry cut. Only that new +case has a two-second operation deadline inside the unchanged peer five-second +guard; original cases retain ten seconds. Its complete native execution suite +passes: 167 passed, zero failed and two ignored in 452.06 seconds. All owned +build/test/check jobs are terminal. The full WIP, source inputs and immutable +executables are preserved under the normal checkout-specific qualification target. +Fixed-01's production/library and example Rust are identical to fixed-02. Its +normal unit target and effects binary build; original two-run effects smoke and +five static checks pass. Full library: 1048 passed, one failed, two ignored +(retired inspection protocol). Strict Clippy: 235 diagnostics, including 215 +dead_code, with none in the six production files changed for these fixes. +The overall native Cargo command still fails on fifteen counter-binary API errors. + +The earlier gateway-backend-fixed-03 checkpoint has 164 passed, three failed and +two ignored; all 21 targeted gateway/cold cases and 39 lifecycle cases pass. +Fixed-02 resolves those remaining three failures and passes the complete native +suite. This component result does not replace the failing package/library and +strict Clippy gates. + +Canonical archive startup creates/syncs its scratch parent before archive work. +The effects example retries typed busy/overload with the same command identity, +input and original deadline. Eight retained runs pass at the earlier fixed-02 +source and two more at gateway-backend-fixed-03. These are SQLite development +object-store smoke runs, not RustFS, managed recovery or load qualification. +Fixture backend declarations and cold oracles retain exact immutable namespace +metadata. Runtime/domain/schema/backend guards remain unchanged. + +Next port public counter/performance binaries and real managed shared inspection +cancellation, +finish canonical managed directory/startup and exact-reference S3 learner +activation. All five original CG-07 public gates, refreshed real Docker-fenced +RustFS recovery/load qualification, CG-09 through CG-16 and the combined hard +cutover remain required. Strict Clippy and the old inspection cancellation test +are open. At that earlier checkpoint the WIP remained unpublished; no final +system/MVP completion was claimed. +Historical sections below describe scoped checkpoints, not combined qualification. + The executable design and implementation plan is delivered. This handoff scopes the remaining end-to-end implementation and hard cutover, including the full task and release gates, for subsequent execution. @@ -16,8 +97,10 @@ node resources. A group spans voter nodes; it is not all cells on one node. Reuse codex/shared-cell-groups and [PR 160](https://github.com/crabbuild/cellar/pull/160). Keep the isolated effects checkout and PR checkout aligned at each published checkpoint. Preserve the user's untracked diagram and both excluded MR progress basenames. -The current goal remains incomplete: the production SDK/fleet still uses its -old host and no final combined release or load envelope is qualified. +The current goal remains incomplete: the SDK now uses the shared host, while +managed startup, remaining public consumers and the final combined release/load +envelope are unqualified. Historical sections below describe earlier checkpoints; +the newest progress section governs current implementation and validation. CG-01 through CG-06 are DONE at their scoped native boundaries. Their refreshed records remain under .cellar/cell-groups-evidence/CG-01-06-gates-e32115aa-02/. @@ -62,14 +145,332 @@ the public SDK/fleet cutover and all original CG-07 acceptance cases remain open Next implement the canonical namespace/class registry and shared host with logical cell handles, then complete managed whole-group lifecycle and accounting. -CG-07 is IN_PROGRESS in the spare PR checkout. Its first canonical node service -and startup changes now select CellGroupRuntime and shared transport. They are -uncommitted and intentionally incomplete: the old host/SDK/fleet consumers still -need replacement before that checkout can compile. Preserve both dirty Rust -files and the user's diagrams. Do not restore legacy startup or add a mode flag. +The latest managed routing/repair work converts Begin/Promote/Finish to actual +physical-group controls (Abort is also implemented). Rooted history, permit/plan, +phase/revision and full occupant placement checks precede normal configuration; +actual durable apply plus a fresh quorum precede success. Native RawNode catch-up +and original deadline/uncertain-result rules remain. At that historical checkpoint Activate was still the old per-cell consumer. +The later peer learner work has converted it; S3 exact-reference transfer and +current-source qualification remain open, without peer fallback for S3. + +Managed route discovery projects committed joining bindings through rooted node +views. Configuration apply/checkpoint install trigger reconciliation; ordinary +known-peer traffic and cell writes keep their hot path. One missing-target retry +is bounded before frame admission. Stable physical/checkpoint factory lifetimes +survive advisory refresh; individual transfers retain exact route generations. +Explicit replacement/removal still revokes old owners and stale refreshes. New +factory views are selected once per transfer, not per chunk. Two new parked-engine +registration regressions are uncompiled/unexecuted. The existing query-auth test +body is byte-identical after moving its fixture to a reusable sibling. + +Final CG-07-managed-routing-advance-04 retains 4726 inputs (SHA256 +bfdbc3c3ff638d7cfd20b8034626e825e33c9fd09e3e9be78beb8713e6f14053) +and the same 36 inherited library errors, with no new route/repair diagnostic or +non-vendor warning. Intermediate type errors and their fixes remain recorded. +The standard unit-test compile attempt, CG-07-managed-routing-test-01, stopped at +that ordinary-library failure before test modules compiled; it ran no tests. +Read the newest progress section and exact owned handles before editing. Full +managed startup, all five original gates, fresh native/RustFS/load qualification, +archive hold retirement and CG-09–16 remain open. Do not publish unqualified WIP. + +The latest managed-genesis implementation replaces the receiver's old cell +registration/activation with activate_managed_initial_group. Catalog I/O enters +the installed execution domain even when polled elsewhere. The SDK rechecks its +namespace capability, rooted birth and actual initial voter plus synced boot, +then delegates whole-group recovery/attachment to the existing provisioning +owner. It returns only the actual durable fleet guard, never a quorum/serving +certificate. Original deadlines, late-reply suppression, owned physical joins, +retained-preparation retry and current-roster/shared-engine failure checks remain. +Attached retries validate without reopening pages or adopting the existing +registration lifecycle. Immutable namespace/directory/backend profiles share the +node image inventories and a single preparation/parallel bank. Their caches and +fixed peer metadata are charged within the original total. Canonical startup +configures these explicit resources; the managed startup gate is not lifted. + +The first two locked builds report 36 errors rather than 40, retiring all four +genesis receiver errors with no direct new code diagnostics or non-vendor +warnings. Final capture CG-07-managed-genesis-advance-04 includes original +unconsumed-reply expiry, the existing 60-second maximum and typed catalog-error +preservation. Its terminal result also has 36 remaining errors and no direct +new diagnostic/non-vendor warning. All 4723 final source inputs are retained +(SHA256 25b5a30439963dea58e34d16208b4f2572f37d6fa257c1c35dc2c8afca4ae3c6), +and all captured Rust inputs match at validation closure. All four compiler +handles are terminal. Final formatting, layout, dependency isolation, +documentation and diff checks pass; all owned validation processes are terminal. +Static review retains all 43 original assertions plus seven new profile refusal +assertions, without executing or compiling them. Read its review-resolution.json +and the newest progress section before editing; no Rust edit while an owned +build/check is live. Existing provisioning +behavioral assertions remain, with namespace profile/backend registration and +new early-owner/budget/duplicate refusals. The tests are still uncompiled and +unexecuted. Next use these profiles/owners for managed peer/S3 learner activation, +real group repair and trusted target routes, then remaining legacy retirement, +node-directory/managed startup and all five original public gates. No managed +native/RustFS qualification or full-goal completion is claimed. + +The latest path correction makes canonical startup, controller provisioning and +independent archive replay share ManifestPageStore::group_pages_path. Serving +owners use groups/////pages under the +same node base; the verifier still has an independent base. Existing physical +fault/restart test bodies are byte-identical outside their independent expected +path helper. Their original assertions remain unexecuted and uncompiled. The +workspace-volume locked library check still has the same 40 errors, with no new +primary path-code error or non-vendor warning. CG-07-provision-path-advance-01 +retains 4722 source inputs (SHA256 +87cdfd5273f790446ac1753f5593bdfe88d3e5ee8f0a6fb2a095298e5496bad3). +Final formatting, layout, dependency isolation, documentation and diff checks +pass; all owned validation processes are terminal. The new path is implemented +but unqualified. Continue namespace-bound image +profiles and existing controller-owned managed activation/registration, followed +by real repair/genesis consumers, old caller retirement and the original gates. + +The newest seed implementation adds prepare_managed_group_seed on the existing +shared complete-checkpoint stream. Fresh root admission binds the active target +reservation and the selected source's member/node/disk/process boot. Authenticated +metadata must contain the exact Seeding permit/plan; revisions cannot regress and +catalog time consumes the original deadline. The serialized engine checks its +last synced destination boot at stage and before install fence. Late delivery +cannot erase durable receipt/debt or report late startup success. Managed joined +bootstrap/revalidation requires the actual receipt, permanent claim, rooted Live +group and exact running boot. Both genesis-guarded and later-enrolled source +origins retain normal successor checks. Unmanaged entry points check namespace +ownership. Catalog preflight/staging/joined delivery use the original transport +deadline plus process/shutdown cancellation; catalog and handshake time is deducted +before forwarding, and ownership reads stay inside directory operation bounds. +All three checks report the same 40 outstanding +legacy errors, with no direct new seed/storage/bootstrap/source-auth diagnostic +or non-vendor warning. All 4722 captured source inputs are retained and all Rust +inputs matched the final CG-07-managed-seed-advance-03 at its validation point. The workspace volume is writable +again; the reviewed build/checks return to its normal checkout-specific target. +The final formatting, layout, dependency, documentation and diff checks pass; +all owned validation processes are terminal. +Five new descriptive profile/origin/guard regressions are unexecuted and grant no +physical/quorum proof. No managed/native/RustFS qualification follows. Read the +newest progress section and invariant-source-review.json. Next use existing +controller provision/registration ownership for managed learner activation, add +actual group repair begin/advance and managed genesis consumers, finish managed +S3 learner restore, remove old seed/inspection/host callers and pass all five +original CG-07 cases. Hold retirement, refreshed native/backend gates and CG-09–16 +remain open. The subsequent path correction described above unifies provisioning +and canonical startup. Its one image-backend profile still needs immutable +namespace-bound profiles sharing node admissions. Retain/extract the common +seed_transport deadline/process/shutdown helpers when removing old seed RPCs. +Do not enable managed canonical startup or publish before qualification. + +The newest query cutover removes retired per-cell SDK query admission/types, +host query execution, routing loop and request/reply body codec. QueryRoute now +accepts only a physical group plus logical cell, refusing the old Cell variant. +Current query/command/enrollment/receipt framing remains purpose separated. Common +MAC/framing, exchange, origin permission and deadline/cancellation/revocation regions +are byte-identical to baseline; the source review is not executed qualification. +Wire security oracles now use logical tokens/physical routes. The original grant +oracle uses synced shared engines/start_shared transport and retains all 18 +assertions; a new regression requires old route/request/reply refusal. Tests are +unexecuted. The sequential locked check falls from 65 to 40 errors, with no primary +query-path error or non-vendor warning. Remaining host.rs errors concern predecessor +startup services. All 4719 captured source inputs are retained and all Rust inputs +still match the reviewed final CG-07-query-cutover-advance-03 capture. All three +sequential checks report 40 errors. Review restored exact returned-token equality +in the ported wire oracle before the final formatted source; tests remain unrun. +Format/layout, native dependency, +documentation and diff checks pass; all owned validation jobs are terminal. +The workspace target remains EPERM; +only this checkout's ignored temporary system-disk target is reused. No new tests, +Clippy, RustFS or managed qualification is claimed. See the newest progress section. +Next finish actual complete-group managed repair/learner seed and genesis owners, +remove old seed/inspection/host consumers, install directory/startup ownership and +pass all five original CG-07 cases. Convert old public read/Axum fixtures without +weakening oracles. Hold retirement, refreshed native/backend gates, CG-09–16 and +the release matrix remain open. Do not publish unqualified WIP or mark the goal done. + +The newest action work routes fleet execution through the installed shared SDK, +with synchronous original-deadline/control admission, exact synced boot checks, +internal observers and unchanged authenticated remote receiver checks. Restart +handoff/resume now address GroupHistory. Actual serialized group owners recheck +RF3 term/revision/committed and survivor matched/active progress, plus the exact +synced quiescence selector for transfer. Expiry is tightened to the original call; +late entered work stays uncertain. Obsolete per-cell fleet action dispatch is +removed and restart reporting carries physical histories. The first diagnostic +falls from 90 to 70 errors; after fixing stale report types the final rebuilt +library reports 65. All captured Rust inputs match; no primary diagnostic is in +the converted action/readiness/handoff/resume paths. Four new regressions are +unexecuted and no runtime/backend qualification follows. Format/layout, native +dependency, documentation and diff checks pass; all owned jobs are terminal. Read +CG-07-group-actions-advance-01/-02 and the newest progress section. The workspace +volume still returned EPERM after cleanup; the final rebuild uses a fresh ignored +checkout-specific system-disk target, as disclosed and recorded in environment.json. +Preserve the original evidence and user diagrams. Next remove legacy query +consumers while retaining shared authenticated framing/security oracles; finish +actual whole-group managed learner/repair/genesis and installed directory/startup. +All five original CG-07 cases, explicit hold retirement, refreshed native/backend +qualification and CG-09–16 remain required. The goal is incomplete. + +The newest fleet decision core uses physical GroupHistory throughout reservation, +topology and isolated repair, with GroupRepairPlacement results and no cell-name +identity. One replica costs one engine slot while full image/staging/memory debt +remains charged. Namespace-qualified group IDs stay distinct; two current +incarnations cannot coexist in topology, and a new incarnation cannot reuse or +erase old reservation debt. Docker recovery now takes RangeDirectory handles and +checks rooted group authorization before repair observations. All 96 enabled core +tests and strict all-target Clippy pass, including three new identity/debt cases; +the first missing-import build failure is retained. Eight binaries and 4718 source +inputs are hashed under CG-07-group-model-advance-02. The runtime diagnostic falls +from 100 to 90 errors, with no primary error/warning in the converted paths and no +captured Rust input changes. Format/layout, dependency, documentation and diff +checks pass; all owned validation jobs are terminal. No public SDK, managed, +Docker or RustFS qualification follows from pure core tests. Preserve shared TLS query/command/enrollment/receipt framing while +removing the old per-cell query consumers, then complete group action RPCs, +directory ownership, startup and all original CG-07 gates. Persistent hold +retirement, refreshed native/backend gates and CG-09–16 remain unfinished. + +The newest implementation replaces both managed removal verifier callers with +complete group replay and explicit private physical inventories/shared pools. +A persistent acquire journal chooses one intent before S3 mutation, restores the +same fixed cut after restart, and reconciles its exact allocation after lost +responses. A new intent needs a fresh registry proof that the previous revision +can no longer commit. Receiver admission precedes page scans and follows physical +ownership through cancellation. Format passes. The locked diagnostic falls from +112 to 100 errors, with no primary error/warning in the new paths. All captured +Rust inputs still match. Read CG-07-managed-archive-advance-01 and the newest +progress section. Format/layout, dependency isolation, documentation and diff +checks pass in the corrected toolchain environment; all owned jobs are terminal. +Five new regressions and Clippy are unexecuted while the library fails. Persistent hold retirement remains explicit and +unfinished; do not release on drop or borrow a publisher hold. Next close remaining +group models/action RPCs/legacy consumers/directory/startup, convert old verifier +fixtures, and pass all original CG-07 and refreshed native/backend gates. CG-09–16 +and full release qualification remain open. No runtime changes are published. + +The newest work cuts fleet quorum RPCs over to the actual shared SDK gateway +and complete physical group roots. Published cuts commit full membership plus +configuration; authenticated subtree mutation summaries distinguish data changes +from protocol-only progress. Fleet archive verification now uses complete group +checkpoint/tail replay and exact persistent holds. Live receipt payloads are +hash-addressed under the existing page bound while index rows and batch byte +ceilings remain unchanged. Per-cell fleet quorum requests/read variants and the +old host RPC implementation are removed. The first library check has 120 errors; +the second and final checks each have 112, with no primary error in the new +metadata/coverage/summary/receipt modules. The final check includes configuration +order and descriptor ownership review. All 4714 source inputs are captured and +all captured Rust inputs still match. Read CG-07-fleet-root-advance-01/-02/-03 +source captures and the newest progress section. Format/layout, dependency, +documentation and diff checks pass. Regressions and Clippy are unexecuted while +the library fails. All owned jobs are terminal. No native/backend qualification +or publishable runtime is claimed. +Next implement the owned hold-acquire journal and full replay receivers for both +managed verification adapters; do not reintroduce a single-image archive or fake +scope. Then group placement/repair, action RPCs, node directory, managed startup +and all original CG-07 gates. The changed marker-1 root shape needs fresh full +qualification; earlier CG-06/08 backend evidence remains historical/source-bound. + +The newest SDK work adds physical node inventory and authenticated inspection +from the real shared engine/roster, preserving pending provisioning debt, +quarantined/unattached slots and original deadline/budget ownership. Transport +routing uses a weak shared-SDK gateway and the actual installed trust cut. Local +node profiles can now be configured; managed startup is still gated. The first +sequential diagnostic reports 124 errors with no primary error in the new node +metadata modules. The second locked check also reports 124 errors with all +captured Rust bytes unchanged. Follower-safe membership metadata is implemented: its distinct +ReadIndex context binds the origin member and accepts replies only from the +current leader, while execution/seed contexts and follower execution refusal +remain unchanged. The new two-image, three-engine regression is unexecuted while +the library fails. Read CG-07-node-inventory-advance-01/-02 captures and the newest +progress section. Next convert the whole-group fleet observation/publication +codec and complete-root archive/removal proofs, then repair/restart RPCs, the +installed directory owner, managed startup and all five original CG-07 cases. +This is local unqualified implementation, not a native/backend release gate. + +The newest SDK work adds guarded physical-group enrollment and permit fencing +with typed refused/uncertain outcomes and a fresh quorum observation after durable +apply. Minimum membership reads defer ReadIndex until apply and re-arm after a +configuration overtakes the barrier, retaining their original budget/deadline. +A fixed control-service registry joins actual movement/restart worker lifetimes +before consensus/storage shutdown; internal observers do not retain the last +external SDK lifetime. Startup checks installed host/boot/lifecycle identity. +Managed canonical startup remains gated. The first diagnostic reports 165 +errors; the final captured check reports 157 and no direct diagnostic in the new control/read/service modules. The new +SDK now retains the real transport and sends exact-boot stop/fence RPCs on its +existing authenticated lane, with original deadline and control admission +captured before returning the waiter. The receiver owns admitted mutations and +no cell/group alias is chosen. Captures are CG-07-physical-controls-advance-01 and +-02. The new real-sync and service ownership +regressions are unrun while the library is failing; read the source capture under +CG-07-physical-controls-advance-01 and newest progress section. Do not treat this +as native or RustFS qualification. Next implement the physical RPC/inspection +owners and whole-group archive/removal proofs, then managed startup and all five +original CG-07 acceptance cases. + +The latest spare work replaces namespace import's cell inventory with an +admitted physical-group set and an immutable frozen topology cut. It settles +unselected pending births without dropping debt, verifies whole-group quorum +coverage and every selected range, and performs exact-owner adoption plus a +revision-advancing topology unlock. Managed restart birth settlement and the +maintenance/resource/operator inventories now take physical directories. Public +movement/debt views omit a cell field; standalone connection requires explicit +group_directory limits. The first locked diagnostic has 194 errors; its joined +method mismatch and unused re-exports were corrected before the follow-up. +The second diagnostic has 177 errors and no own warning; the import modules +emit no direct diagnostic. Resource/movement/topology/repair models now use +GroupHistory, with final capture CG-07-group-import-advance-03. The new real-CAS +freeze regression remains unrun until the canonical crate builds. Read the three +captures and newest progress section. No +new managed runtime, RustFS, load or release qualification is claimed. Continue +with physical SDK/control RPC owners and whole-group archive/removal proofs, +then managed startup and all five original CG-07 acceptance cases. + +Review found that several admitted candidates can share a logical range and +epoch. The fleet range-debt key now includes physical group/incarnation identity; +the directory retains exclusive selected-range ownership. A new invariant test +covers distinct pending keys at the same range/epoch. The third sequential check +reported 174 errors with no own warning before that fix; the final reviewed +source capture is CG-07-group-import-advance-04. The final locked library check reports 174 remaining integration errors; +all captured Rust bytes remain unchanged. The new admission, freeze, import and +inventory modules emit no direct diagnostic, which is not qualification. Both +new regressions remain unexecuted until the canonical library builds. + +The newest spare work converts fleet bindings/birth records to physical group +histories and ranges, adds guarded managed genesis and exact local-boot startup +ordering, and adds a fresh ReadIndex membership observation independent of cells. +Inventory traverses complete physical range pages and retained reservation debt. +Earlier cell-bearing records refuse under the new marker-1 protocol identity. +The first diagnostic reported 202 errors and overlapped formatting; it is retained +without qualification. The latest sequential check reports 198 remaining errors +after the later inventory, quorum, boot, typed-error and runtime-history edits. +All new core assertions remain unexecuted, managed startup is +still gated, and old import/cancellation/inspection/movement/restart consumers +must be converted. The separate primary directory pager passes all seven checks +at 66d5ea6a, including nineteen native catalog cases. Its reviewed baseline +passes all four post-gate checks; PR 160 and both checkouts are at c54307c5. +All WIP and diagrams are preserved; managed cutover changes remain unqualified. See the newest progress section and ignored +CG-07-managed-group-advance-01 through -04 records before editing. + +The spare SDK is now a group_client component with close, drain and resource +owners. Node shutdown/stop proofs use complete group counts and explicit producer +quiescence; one unfinished command per group is no longer assumed. Transfer uses +actual owner progress with term/revision fencing. Namespace stores are released +by the controller's joined owner, independently of response cancellation. The +latest spare library check has 114 remaining fleet/query/predecessor-host errors; +no canonical acceptance case is qualified. See the newest progress section. +The minimal closed-manifest-pin fix passes twelve checks in the primary checkout +and is published on PR 160 at 110b7a4e (runtime 51d136fb). Both checkouts share +that base with all WIP and eight diagrams preserved. Keep its source-scoped evidence separate from the +unqualified cutover. Its recovery regression retains closed log/group handles +through fresh recovery in both image forms. + +CG-07 is IN_PROGRESS in the spare PR checkout. Canonical startup now assembles +namespace registrations, whole-group placements, selected-root recovery, shared +SQL/image resources, controller archival/effects and logical SDK handles. Public +admission and remote ingress are being connected to the same physical owners. +These changes are uncommitted and intentionally incomplete: the first locked +library check reported 132 unconverted SDK/fleet/query/shutdown integration +errors; the second reports 117 after the physical-query and native-effect edits. +No executable or acceptance case is qualified by either run. Preserve all +dirty/new Rust files and the user's diagrams. Do not restore +legacy startup, reinterpret GroupScope.cell or add a mode flag. See the latest +progress section and the diagnostic artifacts in +.cellar/cell-groups-evidence/CG-07-canonical-startup-advance-01/. The exact source state and remaining work are recorded in the spare checkout's .cellar/cell-groups-evidence/CG-07-cutover-progress.json. -Its reviewed source preflight still matches. +The previous source preflight is not evidence for these new runtime inputs. Read plan.py show CG-07 and the existing production callers before editing. 1. Replace canonical host/startup/registration with group placement and cell diff --git a/docs/design/multi-raft/cell-groups/implementation-plan.md b/docs/design/multi-raft/cell-groups/implementation-plan.md index 2aa267431..2665a0cfe 100644 --- a/docs/design/multi-raft/cell-groups/implementation-plan.md +++ b/docs/design/multi-raft/cell-groups/implementation-plan.md @@ -2,6 +2,8 @@ # Implementation order +Purpose: Dependency order and status of the sixteen cell-group implementation tasks. + Generated from `tasks.json`; print again with `plan.py index` after status changes. Every implementation task is pending until source-bound evidence passes. diff --git a/docs/design/multi-raft/cell-groups/progress.md b/docs/design/multi-raft/cell-groups/progress.md index 56238635d..70f8621b5 100644 --- a/docs/design/multi-raft/cell-groups/progress.md +++ b/docs/design/multi-raft/cell-groups/progress.md @@ -2,6 +2,8 @@ # Cell-group implementation progress +Purpose: Source-pinned cell-group implementation checkpoints and qualification limits. + The executable design and implementation plan is delivered. Remaining end-to-end implementation and hard cutover work is specified in the [executor handoff](handoff.md). @@ -10,6 +12,1143 @@ CG-07 is IN_PROGRESS; CG-09 through CG-16 remain TODO. Intermediate contracts do not qualify the public shared runtime. No production legacy/shared mode flag is being added. +## CG-07 shared reads and PR #160 MVP scope + +PR #160 publishes the canonical shared SDK/fleet work as a draft checkpoint. +[Remaining work and priorities](remaining-work.md) orders the MVP deliverables +and names the original public merge gates. CG-07 and the end-to-end goal remain +incomplete; all CG-09–16 requirements remain in scope. + +The counter port exposed missing public read behavior. Two real follower/SQLite +cases reproduce NotReady before production changes in +CG-07-shared-read-follower-before-01, in both image forms. Core-fixed-01 adds +origin-bound safe ReadIndex forwarding in a separate private context domain. +Selection requires current-term quorum and contiguous local apply through the +correlated barrier. Source generations revoke retained follower proofs on a +term/leader change, failure or shutdown. Seed and metadata contexts retain their +own admission rules; read proofs confer no mutation or seed authority. + +Recent-source reads now have an explicit age constraint in the SDK and wire. +Only valid current-term leader traffic refreshes follower source observations. +The captured expiry caps the physical SQL deadline, including entered callbacks; +later traffic cannot renew an old proof. Default SDK stale reads use the injected +host setting; disconnected historical reads require explicit selection. Results +expose the actual selecting member and node. Namespace/process grants, bounded +attempts, original invocation identity and joined physical cleanup remain enforced. + +The source-age fixture initially pumped fresh leader traffic while selecting its +first proof. Tagged probes reproduced that race; fixed-03 selects the non-barrier +proof directly, preserving every expiry assertion and deadline. Forty repetitions +pass, twenty per image form. Debug probes are removed. Fixed-04 adds actual TLS +follower queries with minimum-version, linearizable and recent-source constraints, +exact observations/responder metadata and transport-permit release. These are +three engines in one process, not independent-host qualification. + +Fixed-04's full execution run has 172 passed, one failed and two ignored. Its only +failure is an older router oracle expecting an authorized follower linearizable +read to refuse. Fixed-05 updates that oracle to check exact successful follower +and leader observations, callback counts and released permits, retaining namespace/ +process denial, global retry bounds and the router's absence of local replicas. +All fifteen focused cases pass on fixed-05. The complete native execution suite passes: 173 passed, zero failed and two +ignored, in 408.03 seconds on the normal default stack. +Source manifest SHA256: +64fa92f41bb3e29fb7873ebf0ae5a532ba4c007049c6e1ae47ffcb6039e2f565. + +Core-fixed-01's unit and effects targets build. Its full library suite has 1048 +passed, one failed and two ignored; the failed inspection-cancellation fixture +still sends the retired per-cell protocol. The original two-run effects smoke +passes using the SQLite development object store. Strict library Clippy fails +with 234 diagnostics, including 214 dead_code. Production and example Rust are +unchanged through fixed-05; only integration fixtures changed after core-fixed-01. +Format, layout, native dependency, documentation and diff checks pass at fixed-04; +publication checks cover the final documentation and router fixture separately. + +The overall fixed-05 native Cargo command fails on 34 unconverted performance- +binary API errors despite emitting the fresh native executable used above. Counter +and performance consumer ports, public replica routing, managed directory/startup, +exact-reference S3 learner activation and all five public acceptance cases remain +open. No full-package, strict Clippy, current-source managed RustFS/DR/load, release +or MVP pass is claimed. Historical real RustFS results retain their earlier scope. + +## CG-07 retained preparation and completed seed installation + +The preceding gateway/backend checkpoint (CG-07-gateway-backend-fixed-03) passes +all 21 targeted cases and has 164 passed, three failed and two ignored in the +full native execution suite. Its normal unit-test target and effects executable +build. The original two-run effects smoke and 39 focused lifecycle cases pass on +that same source. Format, layout, native dependency, documentation and diff checks +pass. This effects smoke uses the local SQLite-backed development object store; +it is not current RustFS or managed disaster-recovery qualification. + +The three remaining native failures are reproduced independently in +CG-07-seed-retained-probe-01. Tagged boundary probes show that seed persistence +succeeds before a post-sync source watch rejects completion in both image forms. +The retained-owner retry adds exactly one sync of the same page directory. These +probes have been removed and the original assertions remain unchanged. + +CG-07-seed-retained-fixed-01 separates source liveness before submission from +local completion after successful engine persistence. The queued engine still +checks the complete grant and exact managed boot before entering its fence. +After successful persistence, completion retains destination engine/member/disk/ +process identity, current boot, shutdown and original deadline checks, followed +by fresh catalog bootstrap. It no longer requires the source process to remain +alive after the bytes and receipt have been durably installed. This grants no +quorum or serving authority. Sync errors remain uncertain and fail closed. + +Provisioning retries now reuse an already verified physical closure only with +the same live page and engine owners, exact selected marker, complete checkpoint +retention and no pending/quarantined publication. A changed selection recovers +on the retained applier. Marker equality cannot clear uncertainty. The original +LOCK and no-additional-sync assertions remain. The seed persistence cases also +hold an entered fence through its unchanged original deadline and require a +refused late response while cold recovery preserves the actual synced root. + +Fixed-01 builds the normal unit target and effects executable; its original +retained-disk smoke passes both runs. The full ordinary library suite has 1048 +passed, one failed and two ignored. Its remaining inspection-cancellation test +still sends the retired per-cell protocol and must be ported to the actual shared +managed protocol. Format, layout, native dependency, docs and diff checks pass. +Strict library Clippy reports the same 235 diagnostics (215 dead_code); none have +a primary span in the six production files changed for these two fixes. + +The new expiry cut initially held work past the fixture peer's unchanged +five-second disconnect guard. Fixed-02 gives only that new case a two-second +operation deadline; all other cases retain their original ten-second deadline. +It passes retained-owner reuse and both complete seed-persistence cases, including +all queued/entered revocation, expiry, pre/post-sync error and cold-reopen cuts. +Production/library and example Rust match fixed-01 byte for byte; only the new +expiry fixture deadline changes. The complete native execution suite now passes: +167 passed, zero failed and two ignored, in 452.06 seconds on the normal default +stack. Its three focused regressions also pass. Capture source manifest SHA256: +1a60ff1d210b4f7c3432fabc6f4a669a88539d71271bf6faaed391f6e07c29d6. +The overall native Cargo command still fails on fifteen unconverted counter-binary +API errors even when it emits this fresh native test executable. No full-package, +strict Clippy, managed/RustFS, load or release pass is claimed. All owned jobs are +terminal; the WIP, source inputs, immutable executables and passing/failed evidence +are preserved in the checkout-specific target's qualification-evidence directory. +CG-07 and the end-to-end goal remain incomplete; CG-09 through CG-16 are required. + +The next critical MVP work is the public counter/performance and actual shared +managed inspection cutover, followed by canonical managed directory/startup and +exact-reference S3 learner activation. Preserve all original public acceptance +behaviors, then refresh real Docker-fenced RustFS recovery and replication/ +reclamation under load on the combined source before publication. + +## CG-07 canonical S3 scratch ownership and gateway backend declarations + +Canonical S3 enrollment's Image(Io(NotFound)) failure is now diagnosed and fixed. +The checkpoint receiver's anonymous materialized image spool required a parent +that canonical startup never created. An unchanged source-12 executable reproduced +it on fresh data. Creating only the three archive-spools parents made both original +smoke runs pass. A tagged image-reader probe emitted no failed-image path, and is +removed; cellar-ltx generation.rs is restored byte for byte to its prior source. +Canonical archive startup now creates and syncs this node-owned directory on its +execution domain before registering any archive owner. S3 proof, scratch quotas, +original timeouts and all physical receipt/fencing checks remain. + +The fix also exposed intermittent typed Execution(Busy) in the finite effects +example. Its caller now backs off only busy/overload under the same command ID, +contract, input and original deadline, including ambiguous outcomes. It does not +claim rollback or create a fresh business command. CG-07-canonical-s3-fixed-02 +builds the normal library unit target and effects binary. The original two-run +retained-disk smoke passes, and three additional original two-run trials pass: +eight retained runs total with executable SHA256 +4106a1a1c819f1335e62dffa91417646498bbf19821778baa8e18d69d3797995. +Capture source manifest SHA256: +231fa577c83a1d06ccfe1694e6e8a0e45ef5d259988684b9dc45b21f4e59cce1. +This is the public SDK/effects path with a SQLite-backed development object store, +not RustFS, managed fleet, zero-quorum recovery or a load envelope. All 39 prior +focused boundary cases also pass with the scratch fix (fixed-01); the subsequent +fixed-02 change is example retry policy only. + +The sixteen native gateway failures were caused by extent fixture setup declaring +a materialized namespace. Registered, custom-retention, effects and reclamation +fixtures now declare the actual backend at birth. Archive reopen forwards the +same immutable backend; a new fixture assertion refuses a contradictory setup. +The read-only RangeDirectory namespace configuration accessor is public for +embedding resources; it neither refreshes routing nor grants startup authority. +Production schema/backend/domain/runtime identity guards remain unchanged. + +A standalone original gateway case reproduces on the immutable source-11 binary. +CG-07-gateway-backend-fixed-01 emits a fresh native execution test artifact while +its overall Cargo command still fails on fifteen old counter binary API errors. +All sixteen original failing gateway cases now pass, retaining every original +assertion and both backend variants. Source manifest SHA256: +a0460dd39dc55b4401073c370110c64198ded0030dbfb40ece11ecb07018497e. +The later fixed-03 full shared execution suite has 164 passed, three failed and +two ignored; its 21 targeted cases pass. Retained-owner retry, seed revocation and the +old library inspection cancellation test remain open until new evidence proves +otherwise. No combined/full-package or strict Clippy pass is claimed. + +Next fix remaining owner/seed regressions, port public counter/performance callers +and managed shared-protocol cancellation, finish managed node-directory/startup +and exact-reference S3 learner activation, then all five original public gates. +Current-source real Docker-fenced RustFS qualification, persistent hold retirement, +CG-09 through CG-16 and combined load/release gates remain required. The full goal +is still incomplete and this WIP remains unpublished. + +## CG-07 predecessor retirement and executable qualification + +The unpublished cutover now compiles the production library and its ordinary +unit-test target. The previous 32 integration errors are resolved by removing +the predecessor per-cell host dispatcher, its managed birth/registration API +and inbound per-cell seed/inspection dispatch. Used host options, observation +contracts and storage-policy regression assertions remain. Unreachable +predecessor helpers still require the CG-16 deletion sweep; no warning +suppression or compatibility dispatcher was added. + +Canonical startup now joins physical infrastructure on explicit failure even +with retained caller handles. Cancellation schedules the same owned cleanup. +Peer/query routes are installed only after every local root attaches and the +retained inventory is checked. Both existing startup tests use real shared-group +engines, directory births and TLS peers. A default-stack overflow in permanent +claim revalidation was reproduced, traced through the macOS crash stack and +fixed by boxing catalog work before constructing nested timeout futures, with +an additional heap boundary around canonical initialization. Startup boxing +alone was insufficient (test-05); the catalog boundary fixes the original test +(test-07), and both original tests pass without diagnostic probes (test-11). +No stack limit, timeout, resource ceiling or assertion was relaxed. + +The existing poisoned-accounting regression requires close to return its +corruption error, retains old handles, and reopens the physical engine before +dropping them. This verifies release while preserving the corruption oracle. +The old storage-family tests do not prove managed startup or serving authority. + +The effects executable now builds with explicit namespace capabilities/group +placement, separate logical cell histories, enrollment initializers and public +request/error values. Development provisioning persists one-use archive intents +before committing counters; retained startup only opens them. A fresh harmless +S3 command after observed settlement replaces the advisory archive-position wait. +This executable is not yet qualified: its local SQLite-backed object-store smoke +first refused missing initializers (test-11), then, after that fix, failed S3 +enrollment during checkpoint capture with Image(Io(NotFound)) (test-12). +The failed run's voter disks, catalog, creation intents and logs are retained. +Do not reduce durability or recreate archive counters to pass this case. + +Current evidence: + +- CG-07-legacy-retirement-test-11 retains 4730 source inputs (manifest SHA256 + a48f8733addb9559a02e1b4e61398b7432edfc304393afb5034aeb731326aa70). + All 39 focused cases pass, including both startup tests, 30 host policy cases, + four low-level storage genesis, two route-lifetime and one query-auth case. +- The full ordinary library suite has 1048 passed, one failed and two ignored. + The failure sends the retired per-cell inspection protocol; its cancellation + oracle must be ported to the real shared managed protocol, not suppressed. +- A fresh native execution artifact was emitted before Cargo failed on 15 old-API + counter executable errors. The compiler event (fresh:false), source hashes and + immutable executable are retained. Its full suite has 148 passed, 19 failed and + two ignored. Sixteen failures stop at gateway Identity validation; the other + failures are retained-page-owner retry and seed persistence WrongIdentity in + both image modes. These are open regressions, not passing qualification. +- Test-12 changes only example initializers. Library and native-fixture Rust + sources match test-11 byte for byte. The normal locked unit build and effects + executable build pass. It retains 4730 inputs (manifest SHA256 + af3055a1e8692bf0a384f2c31d1baaf6a65a13c1a070c10f6701c47116b12e16). + Format, layout, native dependency, documentation and diff checks pass. + Strict library Clippy fails with 235 diagnostics (215 dead_code and 20 other + lints); the complete classification is in static/clippy-audit.json. No Clippy + or combined release pass is claimed. + +Next resolve canonical S3 enrollment's missing checkpoint image using the actual +smoke repro. Diagnose gateway identity without weakening namespace/backend/domain +checks; existing extent fixtures may advertise a materialized namespace. Preserve +all owner reuse and seed revocation assertions. Port the counter example to many +cells in one explicit group, then the remaining executable consumers. Complete +managed node-directory/startup integration, exact-reference S3 learner transfer +and all five original CG-07 public gates. Refresh real Docker-fenced RustFS +qualification on current source before publication. CG-09 through CG-16 and final +combined/load gates remain required. No new RustFS backend or recovery drill ran +in this checkpoint; historical scoped CG-06 evidence remains separate. + +## CG-07 controller-owned managed peer learner activation + +The Activate movement receiver now selects its physical history and enters the +same node provisioning manager as initial startup. Its public SDK call uses the +installed execution domain, original bounded deadline, exact namespace owner, +rooted birth and current physical boot. Pre-submission refusals remain distinct +from outcomes after controller submission. Its returned local membership receipt +is descriptive; promotion still requires actual catch-up and fresh quorum. + +Managed staging compares the expected permit and target boot with the fresh +catalog admission before creating a mapping. The source exchange still requires +its own fresh-quorum grant. Source endpoints come from rooted trust and current +node views. After complete checkpoint verification and persistence, attachment +uses actual committed membership and physical bindings, without a cell alias. +S3 learner activation explicitly refuses before submission; the exact-reference +archive transfer remains unimplemented and never falls back to a peer. + +The manager saves the applier/page owner before installation. Cancelled, expired +or lost replies retain that physical preparation and its capacity. The parallel +permit spans recovery and transfer. A completed install whose response was lost +is resumed using the durable seed receipt and permanent claim, with selected-root +recovery on the same page owner. It is not seeded again or given blank genesis. +Entered work is joined before closing transport/runtime dependencies. Retries +coalesce only for the same directory, route, boot, exact permit and source. + +Locked library capture CG-07-managed-learner-advance-01 reports 32 remaining +legacy errors, removing all four errors in the Activate receiver. No new primary +error or non-vendor warning appears in the delivered learner implementation. +This does not establish successful runtime compilation or execution. The capture +contains 4728 source inputs (manifest SHA256 +a4291efcce24441e5ebd48cea89ad3dae028a3dec322396b254b4b663492bea9). +No managed lifecycle, native race, backend or load test ran on this source. +Final format, module layout, native dependency, documentation and diff checks +passed. All owned compiler and static-check processes are terminal. + +Next remove the remaining legacy host/catalog/seed/inspection consumers while +retaining common physical deadline/process/shutdown services. Finish managed +node-directory/startup and exact-reference S3 learner transfer, then execute all +five original CG-07 public cases and the real RustFS recovery drill on current +source. Persistent archive hold retirement, CG-09–16 and the complete release +matrix remain open. The complete goal remains active and WIP is unpublished. + +## CG-07 managed group repair controls and peer-route lifetimes + +Begin, Promote and Finish movement consumers now call physical RaftGroupHandle +repair APIs. The controls require the rooted group and exact admitted permit, +check immutable birth/schema, actual current occupants and target boot/trust/ +physical-host separation, then use ordinary configuration proposals. Exact +permit/plan/phase and expected revision govern retries; aborted/fenced/later +permits cannot masquerade as progress. Successful mutation requires durable +apply and a fresh post-apply quorum, with original deadline and closure checks. +Actual RawNode readiness and catch-up enforcement is unchanged. Abort is also a +real group control. Learner Activate still uses its old consumer and remains a +required conversion; no S3 fallback or fabricated cell alias is added. + +Managed advisory routing can resolve committed joining-member/disk bindings +through rooted node views. It changes no configuration or serving authority. +Discovery lag leaves known voters usable and unresolved peers unavailable; +initial identities are taken from the authorized birth route, not guessed from +node IDs. Ordinary cell apply does not scan routes. Reconciliation runs after +configuration apply/checkpoint install and ordinary frames retry once for an +unknown target, before frame allocation/admission. Existing queue, peer, snapshot +and physical worker limits are unchanged. + +A bounded per-group lifetime token separates physical registration from advisory +route generations. Refresh and node-view changes retain the existing checkpoint +factory. Each transfer still captures an exact immutable route and rejects old +generations; per-chunk checking allocates no selected endpoint. Explicit route +replacement/removal revokes the old factory, and a stale refresh or old guard +cannot adopt/remove its successor. Group/cache guards are released before the +registry lock, and peer authorization runs after its validation guard drops. +These are implementation invariants, not executed qualification claims. + +Two new registration/generation regressions use parked real synced engines, +filesystem stores and transport/TLS listeners. They cover factory continuity, +old selection rejection, physical-owner cleanup and explicit successor retention. +They do not establish a fresh quorum, an active network checkpoint stream or +RustFS behavior. The existing query authentication test body is byte-identical; +only its shared fixture moved to a sibling file, retaining embedded TLS paths. + +Final library capture CG-07-managed-routing-advance-04 has the same 36 inherited +errors as the prior goal turn, including four old activation errors. No primary +error or non-vendor warning is added by the delivered route/repair code. The +intermediate control build's five type/import errors were corrected; its real +stream is retained with the other captures. Final source includes 4726 inputs +(manifest SHA256 +bfdbc3c3ff638d7cfd20b8034626e825e33c9fd09e3e9be78beb8713e6f14053), +with every captured Rust input matching at closure. The standard locked unit-test +compile (--lib --no-run) also stopped at the same 36 ordinary-library errors +before test modules compiled. No tests ran. All builds use the required restored +workspace-volume target. No managed/native/backend/load qualification follows, +and the WIP remains unpublished with the full goal incomplete. +Final format, module layout, native dependency, documentation and diff checks +all passed. All owned compile and static-check processes are terminal. + +Next finish controller-owned managed peer/S3 learner staging, complete-image/ +receipt install and registration using the existing immutable namespace profiles. +Preserve exact-reference S3 semantics and original fencing/retained debt. Then +remove remaining legacy host/catalog/seed/inspection consumers, retain common +physical transport deadline/process services, finish node-directory/managed +startup and all five original CG-07 public cases. Qualify these controls, trusted +route discovery and both regression modules using actual native/race/RustFS +harnesses. Persistent archive hold retirement, CG-09–16 and the full release +matrix remain required. + +## CG-07 controller-owned managed genesis and namespace profiles + +The restart-genesis receiver now calls activate_managed_initial_group on the +canonical SDK, replacing its retired cell lookup/registration/activation path. +The invocation enters its installed execution domain for catalog I/O, even if +polled on another runtime. It uses the exact registered namespace capability, actual rooted +Live birth, admitted initial voter and last-synced local boot. It reauthorizes +before submission and after attachment, checks the selected root and actual +membership, and returns a descriptive persisted fleet guard. That receipt grants +no fresh quorum, seed, serving or physical-release authority. Existing quorum +readiness and permanence/lost-disk checks remain separate and unchanged. + +The controller owns managed initial preparation, applier, selected pages, roster +reservation and transport attachment. Running attempts retain their original +deadline; another waiter cannot extend it. A completed attempt can be retried +with a fresh invocation while reusing its retained physical owner. Entered work +is joined after cancellation/expiry. Real late attachment remains registered, +but delivery cannot report late success. Both queued delivery and consuming an +unread successful reply check its original deadline; managed admission retains +the existing 60-second maximum. An attached retry performs fresh boot/catalog +checks under shared preparation/parallel admission without reopening its page +store, reattaching consensus or adopting another registration's lifecycle. +Current roster/quarantine and shared-engine fatal-error checks remain in force. +Typed startup refusal and unconfirmed outcomes stay distinct; catalog pressure, +conflict, corruption and owning causes are preserved at the receiving boundary. + +Provisioning now has immutable CellGroupProvisionNamespace profiles. Each binds +an exact directory owner and declared image backend; duplicate namespaces, +foreign owners and representation mismatches refuse. Canonical startup clones +its single node image inventories into these profiles and configures explicit +CellGroupProvisionLimits. All namespaces share one preparation bank and parallel +limit. Fixed peer metadata and advertised directory-cache capacities are charged +once against the same total byte ceiling, then subtracted from available active +preparation capacity. No resource limit is enlarged. Usage/metrics expose that +configuration retention separately, and existing joined shutdown drops it after +physical work has joined. + +Existing provisioning fixtures register their actual namespace owners and image +representation for both backends. Their coalescing, pressure, retained-owner, +joined-cancellation, missing-page restart and healthy-group assertions remain. +Additional checks refuse duplicate profiles, insufficient configuration budget +and an unconfigured foreign owner before its first materialization. These test +modules have not compiled or executed while the runtime library fails. They +provide no managed lifecycle, sync, quorum or backend qualification yet. + +The first two locked workspace-volume library builds reduce errors from 40 to +36, removing all four retired genesis receiver errors without new primary +startup/provision/profile diagnostics or non-vendor warnings. The final reviewed +capture is CG-07-managed-genesis-advance-04, also terminal with the same 36 +remaining errors. All four real compiler streams and formatter results are +retained. The final 4723 source inputs have manifest SHA256 +25b5a30439963dea58e34d16208b4f2572f37d6fa257c1c35dc2c8afca4ae3c6; +all captured Rust inputs match at this validation point. Static review records +11 unchanged native invariant owners, including permanent claims, serialized +boot checks, RawNode catch-up, complete-root attachment and common physical +transport cancellation. This is not executed qualification. The restored +workspace-volume target is used for all four captures. Formatting, module +layout, native dependency isolation, documentation and diff checks pass. All +owned validation processes are terminal. Static assertion review retains all +43 original provisioning assertions (with explicit profile arguments) and seven +additional refusals; the tests are still uncompiled and unexecuted. Latest +origin/main is an ancestor of the working branch. Canonical managed startup +remains gated and unqualified WIP is not published. + +Next finish controller-owned managed learner activation, exact-reference S3 +restore, real group repair begin/advance and trusted replacement peer routes. +Retire the remaining old host/catalog/seed/inspection consumers while retaining +common physical transport deadline/process services. Installed node-directory +ownership, managed startup sequencing and all five original CG-07 public cases +remain open, followed by current-source native/race/RustFS qualification, explicit +archive hold retirement, CG-09–16 and the complete release matrix. + +## CG-07 shared physical page path for provisioning and restart + +Canonical startup, controller provisioning and independent archive replay now +use ManifestPageStore::group_pages_path. Serving owners use the same configured +node base and groups/////pages inventory. +The archive verifier retains its independent base. This removes provisioning's +flat path, which could reopen a different page store after canonical restart. +The ordinary exclusive lock, selected-root validation and failure behavior stay +unchanged; no flat-path reader or migration is added. + +The existing provisioning fault and restart fixtures now independently expect +that canonical directory. Static comparison confirms every test body and +assertion outside the path helper is byte-identical, including missing-page +refusal, retained-owner retry and joined cancellation. Both materialized and +extent cases remain. These tests have not executed or compiled while the runtime +library fails; static retention is not physical recovery qualification. + +The workspace-volume locked build is terminal with the same 40 error-message/path +pairs as the managed-seed capture, with no primary diagnostic or non-vendor +warning in this change. CG-07-provision-path-advance-01 captures 4722 source inputs +(manifest SHA256 +87cdfd5273f790446ac1753f5593bdfe88d3e5ee8f0a6fb2a095298e5496bad3). +All captured Rust inputs match at this validation point. Formatting, module +layout, native dependency isolation, documentation and diff checks pass. All +owned validation processes are terminal. The new layout and managed peer-seed +implementation remain unqualified and unpublished. + +Next configure immutable namespace-bound provisioning profiles using shared node +image admissions, wire controller-owned managed learner activation/registration, +and add real group repair/genesis consumers. Remove remaining legacy callers +while retaining shared transport deadline/process services. Managed canonical +startup, all five original CG-07 gates, physical race tests, fresh RustFS, explicit +archive hold retirement, CG-09–16 and the release matrix remain required. + +## CG-07 managed complete-group peer seed admission and fencing + +The shared transport now has prepare_managed_group_seed, using the same complete +checkpoint stream, receiver pools and retained engine work as unmanaged seeding. +The destination must have a fresh active physical reservation for its exact boot. +The selected source must be a rooted seed owner with the exact current member, +node, disk and process boot, and a voter in the authenticated checkpoint. The +source descriptor must carry this exact consumed permit/repair in Seeding phase. +Subsequent authorization cannot regress its original catalog revision. Catalog and handshake time are deducted before forwarding the remaining original +timeout. Review also put catalog preflight, staging and joined-permit delivery +under the existing transport deadline, process revocation and shutdown wrapper. +Namespace ownership reads count inside directory operation bounds. Queued engine +work retains its grant even after the metadata waiter times out. + +Managed staging and installation retain the permanent member/engine claim and +check the exact last-synced local boot in the serialized storage worker. A queued +quiescence, stop or fence refuses further installation. Once accepted, physical +work remains owned through actual sync; expiry/close afterward suppresses late +success without erasing a durable receipt. Managed joined-member bootstrap and +permit revalidation require that receipt, the permanent claim, the rooted Live +group and the current exact boot. An old active permit is not restart authority. +Unmanaged joined/seed entry points also check actual namespace ownership. + +The staged base preserves the source's real enrollment origin. Genesis guards +remain genesis guards; an imported group's committed enrollment is installed by +ordinary snapshot successor validation. Normal membership successors, recovery +lineage, checkpoint encoding and RawNode promotion checks are unchanged, as +recorded in static source review. A staged learner remains outside the birth +configuration and cannot acquire startup/serving authority from this projection. +No marker bump, old reader, cell alias, no-op action or resource increase is added. + +The previous goal turn removed the legacy query path and reduced the diagnostic +to 40 errors. All three managed-seed captures also report 40. The reviewed final +CG-07-managed-seed-advance-03 includes deadline composition and has the same error-message/path multiset as the query +capture, with no primary error or non-vendor warning in the new seed, storage, +joined-bootstrap or source-authorization code. Its 4722 source inputs are bound +by manifest SHA256 +7746dd2c6a4902499291d91f15f3bbcb09bebb1bbc534b16ec5985cf09efa0d5; +all captured Rust inputs matched at that validation point. The first capture used the temporary target. +Workspace-volume writes are now restored; the reviewed build and follow-up checks +return to the normal checkout-specific target required by AGENTS.md. +Formatting, module layout, native dependency isolation, documentation and diff +checks pass for the final capture; all owned validation processes are terminal. + +Five new profile/origin/boot-guard regressions remain unexecuted while the library +fails. Their metadata/guard fixtures grant no catalog, sync, quorum or installation +proof. No new runtime, Clippy, managed lifecycle or RustFS qualification is claimed. +This implements the lower managed peer-seed boundary; SDK/controller-owned learner +activation and managed canonical startup are still gated. Next wire this path into +the existing provision/registration owner, add group repair begin/advance and +managed genesis consumers, finish managed S3 learner restore and remove old +seed/inspection/host callers. All five original CG-07 cases, physical race/native/ +RustFS qualification, explicit hold retirement, CG-09–16 and the release matrix +remain required. Unqualified WIP is not published and the goal stays incomplete. + +## CG-07 query hard cutover and retained authentication + +The retired per-cell SDK no longer admits business queries. Its query options, +routing/result types, pending query operation, host execution branch and remote +routing loop are removed. The old query_route module and request/reply body codec +are deleted. QueryRoute accepts only a physical-group plus logical-cell identity; +the Cell history variant and its reader are gone. Current cell queries, commands, +enrollments and receipt lookups keep their separate purpose-bound bodies on the +same authenticated peer query lane. Marker 1 remains unchanged, with no old reader, +translation, production switch or increased resource bound. + +The common framing/MAC helpers, exchange, origin-grant validator and deadline/ +cancellation/revocation controller are byte-identical to their baseline regions. +This is recorded as static source review, not executed security qualification. +Existing query codec oracles now use logical cell tokens and physical group routes, +preserving channel/exporter, digest, sequence, response correlation, process, +non-voting origin, bounds and typed-failure checks. The original grant test moves +to real synced shared engines and start_shared transport with all 18 assertions +retained: namespace/process isolation, tombstone pressure, in-flight revocation, +regrant refusal and physical pool retention/retirement. Its parked runtimes grant +no serving or quorum proof. A new regression supplies complete superseded route, +request and reply inputs and requires refusal. These tests remain unexecuted. + +The previous goal turn implemented shared fleet actions and restart readiness, +reducing the locked library diagnostic from 90 to 65 errors. The first check +reports 40 errors; the reviewed, formatted final capture +CG-07-query-cutover-advance-03 also reports 40, with no primary error or non-vendor +warning in the converted query code. Review restored exact token equality in a +ported wire oracle before the final capture; the tests remain unexecuted. All +three sequential captures and their real streams are retained. Two remaining +host.rs errors concern unchanged predecessor startup service types, rather than +query execution. Its 4719 source inputs are retained (manifest SHA256 +86d951269719dd948dcd2e0849eb9efa7c0d0d63fde87edb2fcf501b21db56dc), +and every captured Rust input still matches. Formatting, module layout, native +dependency isolation, documentation and diff checks pass; all owned validation +jobs are terminal. The system-disk target remains +checkout-specific and ignored because the normal workspace-volume probe still +returns EPERM. No other checkout artifacts were reused or cleaned. + +This is hard-cutover implementation progress, not runtime qualification. No new +runtime tests, Clippy or RustFS drill ran while the canonical library fails. +Earlier core/native/backend records retain their original source scope. Remaining +legacy SDK/Axum/read examples and low-level SQL fixtures must be converted without +weakening their behavioral oracles. Complete-group managed repair/learner seeding, +registration/genesis, old seed/inspection callers, installed directory ownership +and managed startup are the critical next work. All five original CG-07 public +gates, explicit persistent hold retirement, refreshed native/backend qualification, +CG-09 through CG-16 and the complete release matrix remain open. + +## CG-07 shared fleet actions and restart readiness + +Fleet movement/restart action dispatch now uses the installed shared SDK gateway. +Local execution checks the exact synced boot and installed runtime owner; remote +execution retains the existing authenticated action lane, rooted receiver checks +and exact target. Invocation captures its original deadline and control admission +before polling, with the existing 128 KiB reservation. Internal observers do not +retain the external SDK lifetime. Late delivery or an interrupted entered action +remains unconfirmed; it does not establish rollback. Quiescing hosts can perform +their restart handoff, while fenced/stopped boots refuse action admission. + +Restart handoff and resume callers now address complete GroupHistory values. The +actual group owner checks term, membership revision, stable RF3 configuration, +committed floor and each required survivor's matched/active progress before a +transfer. The runtime rechecks the exact synced quiescence selector after queueing +and tightens its evidence expiry to the original call deadline. The shared and +remaining legacy owners use one extracted predicate, preserving the existing +checks and actual caught-up leadership transfer. Replication metadata alone +grants no quorum, health or seed authority. Restart progress reports carry the +physical history. The obsolete per-cell fleet action dispatch modules are removed. + +The first locked library check reports 70 errors, down from the prior 90. The +reviewed final source fixes five remaining restart-report type mismatches and +reports 65 errors. There is no primary diagnostic in the new action, shared-owner, +restart-gate or converted handoff/resume paths. All 4719 captured source inputs +are retained in CG-07-group-actions-advance-02 (manifest SHA256 +248e011a2bdaa4ec19a2f93ee2b80cae44f14da43d06b70517bf76684edd6f40), +and every captured Rust input still matches. This is compiler progress, not runtime +qualification. Four new readiness/identity/quiescence regressions are unexecuted +while the library fails. No new runtime tests, Clippy or RustFS qualification ran. +The previous 96 enabled pure-core test results remain scoped to their capture. +Formatting, module layout, native dependency isolation, documentation and diff +checks pass. All owned build/formatter/check jobs are terminal. + +After the user cleaned the workspace volume, a new lock open and independent +checkout-specific probes still returned EPERM there. The failed formatter attempt +is retained. The existing repository formatter completed directly; the final +locked rebuild used a fresh ignored checkout-specific target on the system disk. +No other checkout's artifacts were reused or cleaned. The capture's environment +record documents this temporary deviation from the normal workspace-volume rule. + +The next critical cutover removes the old query routing/body codec while preserving +the authenticated framing, node/process and grant/revocation checks shared by +cell queries, commands, enrollments and receipts. Complete-group learner seeding +already has authenticated grants and joined-member bootstrap; managed repair, +registration/genesis and activation must use those real owners. Installed directory +ownership, managed startup and all five original CG-07 public cases remain open. +Persistent hold retirement, refreshed native/backend gates and CG-09 through CG-16 +are still required. No unqualified runtime changes are published. + +## CG-07 physical-group resource, repair and topology decisions + +The pure fleet reservation ledger, topology view/targets/moves and isolated-repair +view/results now identify complete GroupHistory values. Cell names no longer enter +these models. A physical replica still costs one engine slot; its combined image, +staging and memory footprint stays charged under the unchanged resource rules. +Resource reservations retain exact namespace, group and incarnation identity. +Old-incarnation reservations cannot be reused or silently erased by a new history. +Topology rejects two current incarnations of one namespace/group ID, while equal +opaque group IDs in different namespaces remain distinct. The isolated repair +result is GroupRepairPlacement, carrying the same revision/member fences for every +cell. Its deterministic tie-break uses physical history, with no cell alias. + +The Docker failure-recovery adapter now supplies complete RangeDirectory handles. +Before sampling isolated-repair resources it checks current rooted group +authorization. This neither infers failure from absence nor grants membership, +release, recovery or deletion authority. Existing physical debt, exact boot, +capacity/domain checks, work bounds and RF3 assertions remain unchanged. + +The previous goal turn made implementation progress on complete-group replay and +persistent acquire journals, reducing the captured library diagnostic to 100 +errors with static checks passing. This turn changes the authoritative group +models. Its first test compilation caught a missing BTreeSet import in a new +regression; that failed attempt is retained. The corrected capture passes all 96 +enabled cellar-logic tests and strict all-target Clippy. Three new executed +regressions cover namespace-qualified group IDs, conflicting current incarnations +and retained old-incarnation debt. No existing planner oracle was weakened. + +The eight executed pure-core binaries are retained and hashed on the workspace +volume. Capture .cellar/cell-groups-evidence/CG-07-group-model-advance-02/ binds +4718 source inputs (manifest SHA256 +e974a366a32ad5982c07d1de70ae79f250934d69c8e9b98cd9319c14a1691b62). +The locked runtime check now reports 90 errors, down from 100, with no primary +error or warning in the converted fleet models or Docker adapter. All captured +Rust inputs still match. Pure-core results do not qualify public SDK/fleet startup, +Docker isolation, RustFS or managed lifecycle execution. The library and all five +original CG-07 public gates still need to pass. Formatting, layout, dependency, +documentation and diff checks pass; all owned validation jobs are terminal. + +Investigation confirms that the old per-cell query transport shares framing with +logical-cell queries, commands, enrollments and receipts. Keep that authenticated +shared infrastructure while removing legacy query routing/codecs/host consumers. +Then finish group-native learner/restart/leadership control RPCs, installed node +directory ownership and managed startup. Explicit persistent hold retirement after +fencing/joining, old fixture cutover and refreshed native/backend gates remain +required. CG-09 through CG-16 and the complete feature-off/load/crash/release matrix +remain open. No unqualified runtime changes are published. + +## CG-07 managed complete-group replay and durable acquire ownership + +Automatic failed-node removal and management lifecycle verification now use one +shared FleetGroupArchiveVerifier contract. It opens the exact namespace-owned +physical group archive, selects survivor candidates from rooted fleet membership, +freezes the complete quorum-proven cut, acquires independent persistent retention +and restores the full checkpoint plus contiguous tail. The former per-cell birth, +cell archive and single-image verifier factories are removed from both callers. +Public configuration now supplies explicit private materialized/extent image +inventories, page bounds and node-shared archive/checkpoint/tail/scratch pools. + +An acquire journal is keyed by the exact verification basis and pending group, +independently of later business writes. Before the registry sees a mutation, the +fleet catalog conditionally stores the complete intent and observes its winner. +Concurrent controllers send only that persisted winner. Restart restores its +original fixed cut even before a live receipt was appended. Unknown responses +retain the same allocation; a fresh registry read must prove its revision was +consumed without its hold before a conditional journal successor is prepared. +An independently confirmed hold still reconciles after unrelated allocations. +Each method retains its original deadline and rechecks rooted basis before and +after work. No publisher-owned hold, new allocation on timeout, one-cell alias, +format bump or compatibility reader is used. + +Receiver admission now precedes the blocking page-inventory scan. The existing +checkpoint byte/item bank reserves receiver workspace plus bounded page inventory +metadata. Its reservation follows the entered worker, receiver and recovered +closure; cancellation cannot release it while physical work still owns it. +Pressure and limit failures remain typed. No configured quotas are increased. +Persistent holds are deliberately not released on drop or timeout. Explicit +terminal retirement still needs a fence for every user and joined physical work; +the existing per-registry hold ceiling is unchanged. + +The previous goal turn completed the captured 112-error integration diagnostic, +static checks and preservation of the prior complete-root work. This turn changes +the authoritative implementation. Formatting passes. The locked library check +now reports 100 errors, down from 112; no primary error or warning is reported in +the new verifier, acquire journal, admission code or converted lifecycle callers. +This is a failing compiler diagnostic, not qualification of those modules. +The capture under .cellar/cell-groups-evidence/CG-07-managed-archive-advance-01/ +binds 4718 source inputs (manifest SHA256 +87c1ddfc914865321c39cd67a7aff912c55044180340fb7fd7811bb85e1d4d3b). +All captured Rust inputs still match. A dependency-check attempt lacked Cargo on +PATH; its failing stderr is retained separately. With the required toolchain +environment, format/layout, native dependency isolation, documentation and diff +checks all pass. All owned formatter, compiler and static jobs are terminal. +No tests or Clippy ran because the library still fails. Origin was fetched and remains +81cbbc2f72f302751dd471cf06cf51394146ac63, with the branch zero commits behind. +Five new regressions cover lost-response allocation reconciliation, consumed +revision and foreign-purpose refusal, and receiver inventory pressure/lifetimes. +They remain unexecuted while the canonical library fails. No native, managed, +RustFS, load or release qualification is claimed for this work. + +Next finish the remaining group-native topology/repair consumers, authenticated +learner/restart/leadership RPCs and installed node-directory owner, and convert +legacy query/seed/host consumers without restoring the one-cell runtime. Managed +startup and all five original CG-07 public cases remain required, including the +hard cutover of old verifier fixtures. Then qualify explicit hold retirement and +refresh complete native/backend gates for the changed marker-1 root shape. +CG-09 through CG-16 and the combined feature-off/load/crash/release matrix remain +open; the original end-to-end goal is not complete. + +## CG-07 complete-group fleet publication and archive receipts + +Fleet membership RPCs now use the shared SDK gateway and actual physical group +owner. Both local and remote paths reauthorize the exact rooted fleet history, +retain the original deadline and use fresh safe ReadIndex plus durable apply. +The body carries the complete GroupManifest, full membership/configuration and +same-owner progress. No cell, SQL image alias or per-cell query is selected. +Published cuts contain a complete root and exact membership commitment; the +zero-cell root is explicit and missing publication cannot construct a cut. +Configuration set order is canonicalized without changing joint roles. Delivery +age extends once through the original SDK invocation. Metadata descriptors move +their membership ledger into the returned proof rather than cloning it across +response retention; complete RPC/capture work reserves the existing 1 MiB +physical metadata envelope from the unchanged configured control bank. + +Manifest page references now authenticate the maximum physical cell mutation +position throughout each subtree. Updates recompute it along existing AVL paths; +metadata reads obtain it directly from the selected root without scanning cells. +Full tree verification recomputes the summary from actual pages and leaves. +Complete-root prefix comparison distinguishes metadata-only progress from later +cell changes, rejects a conflicting root at the same position, and rejects a +changed root whose latest mutation predates the fixed cut. It also checks exact +route, configuration commitment and recovery lineage. Physical group positions +remain distinct from logical cell versions and original receipt witnesses. + +Fleet archive coverage now independently restores every checkpoint dependency +and replays the contiguous group tail under an exact persistent archive hold. +It compares the complete root and configuration commitment. A locator or uploaded +single image cannot supply this proof. The caller must own the hold and replay +receiver, preserve its original deadline, and retain entered physical work. +The fleet verifier suppresses late delivery and keeps pressure, timeout and +conflict failures distinct. This proof supplies no current health, membership, +write acknowledgement or physical release authority. + +Receipt index rows retain their original 4 KiB bound. Complete live evidence is +stored in an immutable, content-hashed object under the existing catalog page +bound; one append batch retains the original aggregate encoded-byte ceiling. +Readers require the exact object/hash and revalidate the full root/hold shape. +Orphan objects or index pages grant no receipt. Marker-1 readers refuse the old +inline/single-image shape; no fallback reader, format bump or resource quota +increase is added. The superseded per-cell fleet observation requests, read +variant and host RPC implementation are removed. Other legacy adapters remain +in the unfinished cutover. + +The previous goal turn made progress on physical node inventory and follower +quorum metadata, with a captured 124-error library diagnostic. This turn changes +the authoritative implementation. The first locked check reports 120 errors; +the second and final checks each report 112, with no primary diagnostic in the +new group metadata, complete-root coverage, manifest-summary or receipt modules. +The final check includes configuration-order review and captures 4714 source +inputs (manifest SHA256 +fb11c46a225c6859ef9ac482740721b2efbea260fa4dbf762265e4d907ec008b). +Every captured Rust input still matches the current source. This is a failing +compiler diagnostic, not qualification of those modules. Captures and actual +streams are under +.cellar/cell-groups-evidence/CG-07-fleet-root-advance-01/, -02/ and -03/. +New codec, complete-root prefix, authenticated subtree mutation and bounded +receipt-shape regressions are unexecuted while the canonical library fails. +No native, managed, RustFS, load or release qualification is claimed. Existing +CG-06/08 backend results remain scoped to their original artifacts; the new +marker-1 manifest shape requires fresh qualification before publication. +Formatting, module layout, native dependency isolation, documentation and diff +checks all pass. No new tests or Clippy ran because the library still fails. +The final compiler owner and watcher, all formatter jobs and the static runner +are terminal; no owned build or backend job remains live. + +Critical next work is the managed verifier's owned archive-hold intent journal +and complete replay-receiver integration in both automatic removal and management +control. Do not borrow a publisher's disposable hold or retry an uncertain acquire +with a new allocation. Then finish group-native placement/repair models, +learner/restart/leadership RPCs, installed node-directory ownership and managed +startup. All five original CG-07 acceptance cases, CG-09 through CG-16 and the +combined feature-off/load/crash/release matrix remain required. + +## CG-07 physical SDK inventory and follower quorum metadata + +The SDK now obtains local and authenticated remote node observations from the +shared controller and actual cell-group engine catalog. A complete engine scan +retains unattached/quarantined histories and checks that the roster and durable +slot count did not change during capture. Pending attachment/provisioning debt is +reported independently of durable slots; a zero engine count cannot erase it. +Startup bounds come from the actual controller/runtime/log capacities. Current +fleet enrollment is the exact trust cut installed in the transport, not a separate +catalog refresh. The SDK keeps original invocation deadlines and response +reservations; internal metadata observers do not retain external node lifetime. + +The transport has a weak gateway to the real shared SDK and validates its runtime +and execution-domain owner. Node probes no longer upgrade an old cell gateway or +invent a cell identity. The marker-1 node body requires the new +pending_group_registrations field and refuses the old single-history spelling. +Its new regression preserves concurrent debt independently of empty disk slots, +refuses the old body and checks scalar overflow. Local profile observation can +now be configured; managed startup, node-directory publication and mutation +controllers remain gated until their whole-group protocols are complete. + +Review found that the prior shared metadata reader required a local leader, +which cannot supply the three voter observations required by fleet recovery. +Metadata-only safe ReadIndex now forwards from followers. Its distinct context +binds the origin member and fresh invocation nonce; incoming requests must match +that authenticated member and responses must come from the current leader. +Execution/seed contexts remain local. Selection waits for the actual local +durable apply/root/configuration, records same-owner progress and rejects a term +change. The original byte/item/deadline banks and leader-only execution readiness +are unchanged. A new three-engine TCP regression covers both image forms, two +cells, joint configuration on every voter, forged nonleader replies, released +read banks and continued refusal of follower execution/transfer authority. + +The prior goal turn only reverified CG-06 status and is classified as no progress +for the full cutover. This turn changes the authoritative implementation. The +first locked diagnostic reports 124 remaining errors, down from 157. The +second locked diagnostic also reports 124 after the follower metadata changes, +with no primary diagnostic in the new inventory/read-owner/gateway modules. All +captured Rust bytes remain unchanged. Captures and real +streams are under .cellar/cell-groups-evidence/CG-07-node-inventory-advance-01/ and +-02/. Format check, layout (2026 Rust files), native dependency isolation (496 reachable +packages), documentation links and whitespace checks all pass. No new regression +has executed while the canonical library is failing. No managed runtime, RustFS, +load, or release qualification is claimed here. + +Critical next work is the authenticated whole-group fleet observation/publication +body and complete-root archive/removal proofs. Then finish repair/learner and +restart/leadership RPCs, installed node-directory ownership and managed startup, +and run all five original CG-07 acceptance cases. CG-09 through CG-16 and the +combined feature-off/release matrix remain required. Existing CG-06/08 backend +evidence stays scoped to its previously qualified source artifacts. + +## CG-07 physical SDK controls and joined service ownership + +Local SDK group handles now expose guarded fleet enrollment, exact permit +fencing and fresh physical membership observation. Configuration admission is +not completion: the call waits for its durable apply floor, then obtains a fresh +safe ReadIndex under the original invocation deadline. Exact already committed +guards/fences reconcile through a new observation. Refused means this invocation +submitted no configuration; an error after a possible send remains Unconfirmed. +It never authorizes automatic retry, reservation release or node removal. + +The bounded read owner can defer its quorum barrier until the requested minimum +is durably applied, without choosing an application cell. If a later configuration +overtakes a pending barrier, the same invocation re-arms within its original +reservation and deadline. A held minimum consumes the existing fixed node/group +read banks and does not busy-poll or enlarge a quota. The new native regression +holds actual engine sync, checks non-completion and retained admission, then +requires the real current configuration and quorum barrier in both image forms. +It also checks an unreachable floor's deadline and release of its reservations. + +The new SDK has a fixed two-slot control-service registry sharing the configured +control byte/item bank. Movement/restart workers hold their registration through +actual destructor completion; independent done signals occur after retained +worker fields are dropped. Cancelling a close waiter keeps admission closed and +does not erase the live worker. Node closure signals and joins these services +before controller/transport/storage shutdown. Internal observers retain metadata +without retaining the final external SDK lifetime, so a service cannot self-own +an otherwise abandoned node. Actual control admission is included in SDK resource +observations. Service startup refreshes the installed host and checks its exact +node/disk/engine/process boot and synced lifecycle guard; managed canonical +startup remains gated while the remaining consumers are unconverted. + +The previous goal turn made progress: physical namespace import and inventory +interfaces advanced, with an unchanged-source library diagnostic of 174 errors +and full preservation of 136 changed/new files and all eight user diagrams. +Current source is local and unqualified. The first sequential locked library +check reports 165 remaining errors; the final captured check reports 157, with no direct diagnostic in the new SDK, +read-owner or service-worker modules. Actual transport is now retained by the +new SDK, which submits persisted exact-boot lifecycle requests on the existing +authenticated stop/fence lane. It captures deadline and control admission before +returning the waiter; the receiver independently owns admitted guard mutations. +No local business group is required. A lost/expired reply remains Unconfirmed. +Formatting passes over 2025 Rust files; native dependency isolation passes with +496 reachable packages. Source captures and actual streams are retained under +.cellar/cell-groups-evidence/CG-07-physical-controls-advance-01/ and -02/. The new native and +service-lifetime regressions remain unrun until the canonical library compiles. +No RustFS, sustained-load, managed runtime or release qualification is claimed. + +Reflection: connect the physical SDK/control RPC and inspection owners, convert +whole-group archive/restart/removal verification and occupancy, and finish managed +canonical startup. Then run all five original CG-07 acceptance cases and the full +affected feature matrix. CG-07 remains IN_PROGRESS; CG-09 through CG-16 and final +combined qualification remain required. + +## CG-07 physical-group import and lifecycle inventory cutover + +The new import path accepts RangeDirectory and an opaque frozen physical-group +inventory. Shared birth reservation now enters the namespace's CAS admission +index before topology publication. Admission freeze arbitrates against disaster +recovery, then a second CAS locks the directory. The exact locked head is retained +by digest. Delayed admissions and stale topology writers cannot enter that cut; +unknown responses retain the same operation and permanent identity claims. + +Inventory settles every physical birth, including unselected pending groups. +Cancellation competes with materialization at one immutable decision key and +retains the group identity and fleet/node references. Selected lazy births stay +live; absent voter disks do not authorize imported genesis. Coverage consumes a +fresh cell-independent quorum membership capability and verifies the exact birth, +guard, stable RF3 roles, node storage pins and permanent group member/engine +claims. Handoff independently walks every frozen selected range and requires +its exact admission and coverage witness, in addition to the complete intent set. +The adopted owner is checked by actual managed birth/bootstrap effects. Topology +unlock advances the head again, so a predecessor expected head cannot reappear. + +Restart birth settlement now uses physical group decisions and exact opaque +fleet reservations. Maintenance, capacity, evacuation, resource and platform +interfaces retain RangeDirectory handles. Standalone management connection takes +explicit group directory limits; movement/debt projections no longer contain a +cell field. The marker-1 protocol adds group-import-1 with a hard cutover. + +All of this remains local and unqualified. The first sequential locked library +check reports 194 remaining integration errors, including one new joined-method +name mismatch corrected after the check ended. Its two unused predecessor +re-exports were also removed. The new native regression holds an actual admission +CAS while freeze wins, drops a freeze reply, and asserts exact inventory count, +stale topology refusal and retained reopen without object listing. It has not +executed: the canonical library still fails compilation. The follow-up check and +its exact source capture are under CG-07-group-import-advance-02. That check +ends with 177 remaining errors and no own-crate warning. The new admission, +freeze, import and inventory modules emit no direct diagnostics. Resource, +movement, topology, repair and service models were then converted to GroupHistory; +the final capture is CG-07-group-import-advance-03. None of these diagnostics +qualifies the canonical runtime or replaces the earlier directory pager checkpoint. + +Review found that several admitted candidates can share a logical range and +epoch. The fleet range-debt key now includes physical group/incarnation identity; +the directory retains exclusive selected-range ownership. A new invariant test +covers distinct pending keys at the same range/epoch. The third sequential check +reported 174 errors with no own warning before that fix; the final reviewed +source capture is CG-07-group-import-advance-04. The final locked library check reports 174 remaining integration errors; +all captured Rust bytes remain unchanged. The new admission, freeze, import and +inventory modules emit no direct diagnostic, which is not qualification. Both +new regressions remain unexecuted until the canonical library builds. + +Reflection: finish the physical SDK/control RPC owners and complete-group archive +verification used by movement, restart and removal. Then enable managed canonical +startup and run all five original CG-07 public acceptance cases. Existing per-cell +host/query/control consumers still need removal or conversion. CG-07 remains +IN_PROGRESS, and CG-09 through CG-16 and the combined release matrix remain open. + +## CG-07 group-native managed authority work + +Unqualified cutover work now removes the application cell from FleetGroupBinding. +Fleet histories carry the exact physical GroupHistory, range, route epoch and +original creation identity. Range reservations and permanent group IDs retain +the existing rooted CAS journal, node references and unknown-outcome behavior. +Namespace/class declarations are checked explicitly, without deriving a class +from a synthetic group cell. The marker stays 1; the protocol identity adds +cell-group-fleet-1 and earlier cell-bearing bindings refuse decoding. + +Managed group genesis now writes its configuration-owned fleet guard in the +first engine sync. Existing bare histories refuse retrofitting. Imported histories +can only resume an existing matching guard. Catalog bootstrap checks the exact +host/node/disk/engine/process boot, and the engine worker rechecks the synced +lifecycle guard before creating or resuming a managed group. Quiescence, accepted +stop and fencing block this work; a matching durable resume can reopen admission. +Membership envelopes retain full route binding and guarded permit sequencing +through enrollment, fence, learner repair and joint-consensus steps. Checkpoint +capture/decoding checks the guard's physical history. + +The shared runtime has a new cell-independent quorum membership observation. +It sends a fresh safe ReadIndex, waits for contiguous durable apply, and selects +one coherent configuration behind that barrier. Node/group metadata reservations +are acquired before cloning membership and remain charged through response +retention. The SDK captures the original observation deadline before polling. +This metadata grants no eligibility, archive, deletion or release lease. + +Fleet inventory now traverses physical range pages and checks both directions: +each selected route needs a rooted history, and every retained reservation must +keep its exact namespace/birth/node references, including pending births outside +the selected tree. Import and cancellation require complete group-native terminal +proofs; a predecessor cell inventory or a caller-supplied Cancelled value refuses. +The additive range pager passes its separate primary checkpoint at 66d5ea6a. +It bounds each response, pins its continuation to an immutable directory cut, +and uses authenticated tree paths without listing object-store keys. + +The first managed-authority library diagnostic ended with 202 errors in unconverted +consumers. It overlapped an owned formatter; pre/post-format snapshots and the +complete output are retained as diagnostic only. No new core authority module +emitted a diagnostic in that run, which is not qualification. The successive +sequential library checks report 205, then 198, then 198 remaining integration +errors. The last run includes typed fleet-authority failures, a definite refusal +for missing imported genesis without quarantining an otherwise healthy engine, +and public execution-domain clocks in the new external native assertions. +All new core assertions remain unexecuted. CG-07 remains IN_PROGRESS and +managed canonical startup remains closed until its real consumers are converted. + +Scope addition: native_cell_groups_catalog.rs and its existing directory fixture +cover the physical inventory pager; no original CG-07 case or oracle is replaced. +The first pager build caught a missing catalog re-export before any test executed; +that failure is retained. The corrected checkpoint passes all seven commands: +the exact new case, nineteen catalog cases, strict all-target Clippy, formatting, +layout, native dependencies and docs. No case is ignored. Native record SHA256: +8061d03d7902d21f291fb82b16bcf04c91d20716465ef5292eecd7c754dcd91a; +executable SHA256: 43f6e2f3654ca92b623b3960c04bbd443a17128a1391728f35c725bdb20b8d48. +The six-file source snapshot, exact executable and actual streams are retained +and independently backed up. Both checkouts share runtime base 66d5ea6a with all +92 preserved WIP files and eight diagrams unchanged across that fast-forward. +The reviewed source-baseline document passes all four post-gate checks and is +published with this component on PR 160 through c54307c5. Both checkouts now +share that head; managed cutover changes remain local and unqualified. +Records live under .cellar/cell-groups-evidence/CG-07-managed-group-advance-01/ +through -04/ in the spare checkout, and CG-07-directory-inventory-01/ and +CG-07-directory-inventory-fixed-02/ in the primary checkout. These are metadata +fixture results, with no new RustFS, managed runtime, load or release qualification. + +Reflection: finish group-native import/cancellation and physical inspection/RPC +consumers, connect the actual movement/restart owners and managed canonical +startup, then qualify the original five public SDK/ingress/lifecycle cases. +The full cutover, CG-09–16 and final combined release/load/crash matrix remain open. + +## CG-07 whole-group shutdown, drain and resource ownership + +The spare cutover now connects Node shutdown to the logical SDK's physical close +owner. The bounded close report contains whole GroupHistory replicas, complete +aggregate group/archive counts and separate unavailable coverage. Retained calls +are not interpreted as one unfinished command per group. Producer quiescence is +observed only behind sealed public admission, an idle application-work registry, +healthy group/provisioning state and idle physical SQL. Unobserved outcomes never +become a clean stop because a counter happens to be zero. Controller, inventory +and engine close failures remain distinct, and no report grants disk deletion. + +Shared owners expose real replication progress and recheck term, membership +revision, stable incoming voters, recent activity and catch-up on leadership +transfer. Node drain gives each physical group one bounded turn under the original +absolute shutdown deadline. Offered attempts, owner acceptance and observed +successor leadership remain separate observations, including timeout races. +No cell receives a worker or timer. Stale-term/revision assertions were added to +the real membership fixture; they are not yet executed in the uncompiled cutover. + +Resource observations now describe shared namespace/controller/runtime/SQL +admissions and actual capacities. Serving and independent archive-verifier image +stores have separate passive accounting; cached manifest counts are metadata, +not serving authority. The SDK implementation is split by responsibility under +application/consensus/group_client/. The controller's existing join owner releases +namespace image resources even when a close waiter is cancelled. Unpolled +enrollment and receipt futures defer cloning those physical resources. + +A retained-closed-metadata regression exposed cached manifest pins surviving +physical engine closure. Both image forms fail at 5e7aadb6 with directory-lock +refusal. The minimal cache/storage fix releases only derived pins after the +closed fence and physical engine join, before terminal completion; independent +readers retain their own pins. Both regressions pass with the fix. Its twelve +checks also pass full execution (165, two fixture ignores), archive data (29, +five RustFS ignores), effects (ten, one RustFS ignore), namespace host (three) +and controller-archive suites, plus all-target strict Clippy and repository checks. +The first controller target invocation used an incorrect name and was rejected +before execution; the corrected target passes. Actual streams and five executable +copies are retained under +.cellar/cell-groups-evidence/CG-07-closed-manifest-pins-fixed-01/ in the primary +checkout and its workspace-volume backup. This fix is committed at 51d136fb +and published on PR 160 through 110b7a4e, including the reviewed baseline. +Both checkouts share that base; every preserved WIP file and all eight user +diagrams match their pre-sync hashes. It does not qualify the new Node/SDK/fleet +pipeline or another RustFS drill. + +The cutover's latest locked library check still fails with 114 unconverted fleet, +query-wire and predecessor-host integration errors. Its new Node shutdown, +resource and transfer modules emit no diagnostics in that run, which is not an +executable qualification. The earlier missing import and recursive cleanup-type +graph were corrected without increasing recursion, stack, quotas or deadlines. +Layout, native dependency and documentation checks pass after the lifecycle +changes. The new lifecycle/stop-proof assertions remain unrun. CG-07 stays IN_PROGRESS; +finish whole-group fleet authority and public routing/ingress, convert their real +consumers, then run all five original acceptance cases. CG-09–16 and the final +combined release matrix remain required. + +## CG-07 canonical startup and logical SDK cutover work + +CG-07 remains IN_PROGRESS. The unpublished spare checkout now replaces the +canonical startup's per-cell birth loop with namespace capabilities, an explicit +initial range plan and whole-group assignments. Assignments resolve permanent +group reservations and recheck directory selection before bootstrap. Startup +recovers complete selected roots through the existing appliers before attaching +shared owners, and refuses retained groups missing from its placement inventory. +There is no legacy/shared startup flag. The published PR checkpoint is +110b7a4e; this incomplete cutover has not been pushed. + +The new SDK uses logical CellHistory, CellVersion and CommitWitness values and +separates administrative group handles. It owns joined closure independently of +response cancellation and explicitly reports unobserved mutation outcomes rather +than manufacturing clean shutdown or deletion proof. Materialized and extent +namespaces share node image inventories and SQL workers; independent archive +verification uses separate shared inventories. Startup opens only existing +namespace-owned archive bindings and refuses missing identities or counters. + +Public admission and application-work registration now share the Node's short +maintenance/drain barrier. Commands, enrollments and queries retain their work +leases through queued ownership, unresolved completion and undelivered replies. +Remote query ingress is being moved onto the same controller queue. Gateway +private paths derive from logical history; namespace/schema and exact controller +ownership remain checked. Native effect endpoints now use weak logical handles +instead of looking up per-cell physical bindings. + +Inspection found a further query-expiry gap: async timeout can precede physical +callback completion. The new work transfers upstream admission into that SQL +reader and tracks its physical lifetime separately. The held-query fixture now +also exercises an expired response, global admission retention, work visibility, +cancelled close and release after callback exit. Directory regressions additionally +cover history-based placement lookup, wrong namespace/incarnation and a superseded +range. These new assertions have not run while the canonical library is broken. + +The first locked library check failed with 132 compiler errors. They primarily +identify unconverted fleet, maintenance, query-wire, resource and shutdown +consumers of the old SDK types. A second check after the physical-query and +native-effect changes failed with 117 errors; the effect adapter's fourteen +old-handle errors are removed. Neither run qualifies a build or executable. +The retained streams, source manifests and new +source copies are under +`.cellar/cell-groups-evidence/CG-07-canonical-startup-advance-01/`. +This is diagnostic evidence, not qualification. Subsequent runtime edits require +another check and all five original CG-07 acceptance cases. Managed namespaces +currently refuse with ManagedAdmissionRequired before catalog writes; whole-group +managed lifecycle is unfinished. No CG task status or release gate was relaxed. + +The next critical work is actual Node shutdown/resource/drain conversion and +whole-group fleet authority, followed by public remote routing and ingress, +canonical native fixtures, Axum/admin/CLI/examples and the unchanged CG-07 gate. +CG-09 through CG-16 remain required for the end-to-end hard cutover. + ## CG-07 namespace host component CG-07 remains IN_PROGRESS. CellGroupHost now binds compiled namespace/class diff --git a/docs/design/multi-raft/cell-groups/protocols.md b/docs/design/multi-raft/cell-groups/protocols.md index 32529c3ad..c6a0d496e 100644 --- a/docs/design/multi-raft/cell-groups/protocols.md +++ b/docs/design/multi-raft/cell-groups/protocols.md @@ -2,6 +2,8 @@ # Durable protocols and crash cuts +Purpose: Durable enrollment, apply, topology, hibernation and disaster-recovery protocols. + Status: required target behavior. Names below are contracts to implement. The [design](design.md) defines identity, partitioning and storage. A catalog record is never a vote, a majority proof or infrastructure fencing evidence. diff --git a/docs/design/multi-raft/cell-groups/qualification.md b/docs/design/multi-raft/cell-groups/qualification.md index 4113ec83a..5ad0b7de9 100644 --- a/docs/design/multi-raft/cell-groups/qualification.md +++ b/docs/design/multi-raft/cell-groups/qualification.md @@ -2,6 +2,8 @@ # Qualification and evidence contract +Purpose: Failure oracles, workload profiles and evidence required for release. + Status: proposed workload contracts and release gates, not measured results. `profiles.json` freezes candidate requirements. Change a missed target only through an explicit reviewed profile revision, retaining the old run and its failure. diff --git a/docs/design/multi-raft/cell-groups/remaining-work.md b/docs/design/multi-raft/cell-groups/remaining-work.md new file mode 100644 index 000000000..1f1aa8248 --- /dev/null +++ b/docs/design/multi-raft/cell-groups/remaining-work.md @@ -0,0 +1,92 @@ + + +# Remaining cell-group work and priorities + +Purpose: Prioritize the remaining cell-group MVP, merge gates and full architecture deliverables. + +PR [#160](https://github.com/crabbuild/cellar/pull/160) contains the shared runtime +and the canonical SDK/fleet cutover in progress. It is a draft implementation +checkpoint, not a completed MVP or release. This ordering preserves the full +CG-01–16 objective in [tasks.json](tasks.json), [design](design.md) and +[qualification](qualification.md). See [progress](progress.md) for exact source +and validation boundaries and [handoff](handoff.md) for executor context. + +## Current boundary + +CG-01–06 and CG-08 have passed their recorded component gates. The implementation +now has shared group owners, cell-specific SQLite histories and receipts, complete +manifests/checkpoints, group archives/reclamation, fenced fresh-history recovery, +indexed effects, bounded caches, canonical namespace/group startup and logical +SDK handles. Managed group controls and source/boot checks are implemented in +part; combined public orchestration remains unqualified. Shared follower reads, +source-age policies and actual responder metadata advance that cutover. + +CG-07 remains IN_PROGRESS. The five original public acceptance cases are still +required, and their `native_cell_groups_public` target has not been delivered. +CG-09–16 remain TODO. Historical local Docker/RustFS recovery and effects results +do not qualify the latest combined SDK/fleet source, independent hosts, sustained +load or a release. No production legacy/shared switch or old-format migration is +part of the target. + +## P0: finish the MVP and its merge gates + +| Order | Deliverable | Required result | +| --- | --- | --- | +| 1 | Public query routing and executable consumers (CG-07) | Implement bounded explicit replica candidates, per-attempt limits and an untried leader fallback through the shared controller. Preserve one original deadline, invocation, input, token and global attempt budget. Keep namespace/process grants and fresh destination admission. Port counter, native/Axum consumers and performance/fleet runners to namespace capabilities, physical group assignment and replicated cell enrollment. Retain every original HTTP/read/retry behavior and report the actual responder. | +| 2 | Managed canonical startup and whole-group lifecycle (CG-07) | Finish node-directory/startup integration and qualify fresh/retained boots, ownership discovery, capacity, repair, drain, maintenance, stop/return/resume and accounting as complete groups with separate cell/image occupancy. Reuse CAS journals, permanent claims, exact boot fencing and physical disruption debt. Join entered work on failure/cancellation before releasing resources. | +| 3 | Managed S3 learner activation and archive holds (CG-06/07) | Implement the currently refused exact-reference S3 path without peer fallback. Verify and retain the complete manifest/image/receipt closure; preserve local owner and uncertain outcomes through actual sync. Reauthorize the destination boot and catalog, attach the actual installed member, catch up and obtain fresh quorum before promotion/success. Complete safe archive-hold retirement. A seed receipt or S3 object alone is not activation authority. | +| 4 | Public, build and backend qualification (CG-07 and MVP portions of CG-14/15/16) | Deliver all five public cases below; resolve public binary compilation, port the old inspection cancellation oracle to the managed shared protocol, and clear strict lint failures without suppressions. Run affected feature modes from SDK CI. Refresh same-source real RustFS recovery and replication/reclamation under bounded mixed load, retaining the exact binaries, ledgers, failures and resource/latency measurements. | + +Public replica routing belongs in one controller-owned invocation. Local and +remote attempts must retain the same routing session and admission; a timed-out +physical read remains owned while another permitted attempt runs. Incoming RPCs +execute locally rather than recursively forwarding. Peer discovery and leader +hints choose a destination but never supply authority. Do not implement retries +by creating fresh independent deadlines or attempt banks. + +The original public merge gates are: + +- `sdk_many_cells_share_group`: public startup/handles, independent SQLite cells, + one configured group and all replicas; observe actual group/RawNode counts. +- `managed_group_lifecycle`: repair, drain, stop/return/resume and capacity move + complete groups while preserving cell counts and physical debt. +- `ingress_cannot_cross_cell_authority`: native/Axum/admin tenant, cell and group + binding refusals, with immutable command identity across retries. +- `occupancy_one_uses_shared_path`: one-cell and multi-cell configurations use + the same host, manifest, snapshot and archive implementation. +- `feature_off_fleet_contract_unchanged`: existing non-Raft ownership/durability + modes pass their actual regression matrix. + +Run the original counter process/restart drill after its port, including late +startup failure, both HTTP hosts, remote minimum-version/linearizable reads, +permission and cross-cell token refusal, exact retained receipts and joined close. +Do not delete options or weaken assertions to make consumers compile. + +The RustFS recovery gate deletes every old voter disk behind the accepted +fencing-authority/Docker contract, then restores a new incarnation from complete +S3 evidence. Verify restored images/receipts/client floors, original S3 minima, +permanent old-launch/process refusal, fresh quorum activation, new writes and +retained restart in both image forms. Under load, verify archive coverage before +local reclamation and account for accepted, refused and ambiguous outcomes. +Local loopback qualification must remain labeled as such. + +## P1: complete the requested architecture + +These are required follow-ups, not removed scope or optional substitutes for the +full hard cutover. + +| Task | Deliverable and dependency | +| --- | --- | +| CG-09 | Irreversible source sealing and whole-cell transfer, with cancellation/seal serialization, quorum installation and atomic range publication. Preserve cell histories, receipts, effect IDs and remote recovery closure. | +| CG-10 | Split hot groups and merge compatible adjacent ranges using CG-09; recover partially completed transitions without reopening a sealed source. | +| CG-11 | Group placement, failure-domain/reserve accounting, load observations, hysteresis and hot-cell isolation over the same transition/membership owners. | +| CG-12 | Fenced group parking and authority-checked wakeup. Resume actual membership/quorum under the original request deadline; measure cold response latency rather than assuming immediate readiness. | +| CG-13 | Managed S3 recovery of complete groups and incomplete transfers/splits/merges, including zero-quorum fencing, fresh incarnations and unfinished lifecycle journals. | +| CG-14 | Reproducible process/RustFS harnesses with real independent cells/groups, source/binary/profile hashes, owned resources and complete outcome ledgers. | +| CG-15 | Full load/crash, transition, wakeup and disaster-recovery matrix. Measure the documented profiles and failure schedules; no capacity or TiKV performance-parity claim follows from correctness tests. | +| CG-16 | Delete remaining predecessor consumers/codecs, finish feature-off/full release checks, publish the unified API/runbooks and qualified operating envelope. No one-cell special path, compatibility reader or production mode toggle. | + +The execution sequence is public routing/consumer ports, managed startup and S3 +activation, the complete MVP public/backend gates, then transition/topology/wake +features and the final managed recovery/load/release matrix. Passing one layer +does not mark another layer or the end-to-end goal complete. diff --git a/docs/design/multi-raft/density-decision.md b/docs/design/multi-raft/density-decision.md index 8e2fc3435..c864f99f0 100644 --- a/docs/design/multi-raft/density-decision.md +++ b/docs/design/multi-raft/density-decision.md @@ -2,6 +2,8 @@ # Shared cell groups: architecture decision +Purpose: TiKV investigation and the decision to share physical Raft groups across whole cells. + Status: selected replacement architecture; runtime conversion and qualification are pending. This document does not claim that Cellar supports shared groups yet. It supersedes one-cell-per-group as the target architecture, while the existing diff --git a/docs/index.html b/docs/index.html index ab7639cc7..c3bb476e5 100644 --- a/docs/index.html +++ b/docs/index.html @@ -63,7 +63,7 @@

The document library

Choose your r